How Products and Services Are Evaluated in Gartner Critical Capabilities

Overview

Analysis

Assessing the range of technology products or services in a given market is a crucial business task. Every range of products and services varies in terms of core features, target market, compatibility, expandability, user experience and other areas.

As a companion to the Gartner Magic Quadrant, Gartner Critical Capabilities research delivers focused insight into providers’ product or service offerings and how these offerings rate against common client-usage scenarios. Critical Capabilities help determine which technology or service offers the best fit by providing a comparative analysis that scores competing products or services against a set of critical differentiators identified by Gartner. Thus, Critical Capabilities research shows, in graphical and tabular form, which products or services best meet the most significant usage scenarios in a given market. Critical Capabilities also help assess the comparative strengths of multiple products and services against current (and future) needs.

Gartner’s and Critical Capabilities research are closely linked. The Magic Quadrant provides a broader view of the most relevant vendors or service providers in a particular market, while the companion Critical Capabilities analysis focuses on the specific products or services from key vendors.

Magic Quadrants provide a holistic view of the set of vendor characteristics across 15 distinct criteria, focusing on vision and execution capabilities. The Magic Quadrant also goes beyond the specifics of product or service offerings. Critical Capabilities analyze a group of competing products or services based on a set of use cases that match typical client deployment scenarios. These use cases are based on the real-world problems that clients need to solve, as well as how they intend to use the technology or service in their enterprises.

The correlation between vendor positioning in a Magic Quadrant and its product or service ratings in the Critical Capabilities is limited. Only one of the 15 standard Magic Quadrant assessment criteria — Product/Service under Ability to Execute — is closely connected.

For example, a provider in the Leader’s quadrant of a Magic Quadrant could have a low-ranked product or service in the Critical Capabilities, because even leading vendors’ products or services can miss critical market needs for a given use case.Regardless of its relative position on a Magic Quadrant, a vendor may produce a tightly focused and highly targeted product or service that outranks other vendor offerings.The purpose of the Critical Capabilities is to provide an assessment framework for rating these products and services (see Figure 1).

Critical Capabilities research is an effective tool to help you better understand vendors and their product and service offerings. You can also consult with a Gartner analyst who specializes in the market to gain details and insight affecting your specific needs and mission-critical priorities.

In defining a product or service category for evaluation, the analyst first identifies the leading usage scenarios for the products or services in this market.

The analyst answers the questions:

• What needs are end users looking to fulfill when considering products or services in this market?

• What usage scenarios are vendors typically targeting with their products or services?

The analyst models all of the common end-user deployment scenarios. Next, the analyst combines some scenarios and eliminates other less significant ones to arrive at between three and seven distinct end-user scenarios that define the use cases for the specific market.

Gartner methodology requires the analyst to then identify the “critical capabilities” for a class of products or services. These attributes differentiate products and services in a class based on their quality and performance. The analyst has to establish which capabilities form the most important criteria for purchasing decisions for the defined usage scenarios. The critical capabilities typically form a small subset or a grouping of the features commonly required by a class of products and services, not the entire range of capabilities that could be assessed. These “critical” capabilities should be distinct from the multitude of important criteria that would be common among all usage scenarios, but offer little differentiation.

Some capabilities may be defined further to emphasize the differentiating factors of the capability and remove the assessment of important (but ever-present) baseline functionality. Each capability will be more or less important in every defined use case, and that importance is represented by its weighting.

Critical Capabilities research is not intended to be an exhaustive analysis of every product and service for the vendor in a market but rather, a focused overview. Vendors that appear in the Critical Capabilities report must have a product or service that could fulfill at least one of the differentiating capabilities.

Vendors that appear in the companion Magic Quadrant may or may not appear in the Critical Capabilities based on the inclusion criteria for the particular Critical Capabilities. Typically, a Magic Quadrant and its companion Critical Capabilities will share identical inclusion criteria, but not always.

Each vendor’s product or service is evaluated in terms of how well it delivers and differentiates on each capability, on a five-point scale. These ratings are displayed side by side for all vendors, allowing easy comparisons between the different sets of features.

Ratings and summary scores range from 1 to 5:

1 = Poor — Most or all defined requirements not achieved

2 = Fair — Some requirements not achieved

3 = Good — Generally meets requirements (may exceed some and not fully achieve others)

4 = Excellent — Meets or exceeds some requirements

5 = Outstanding — Significantly exceeds requirements

The analyst establishes the expected range for each capability scoring and what the characteristics of a 1 and a 5 would be. During evaluation, based on findings, an updated baseline for these expectations may be created to establish the final rating range.

Research activities include:

• Attending vendor briefings and product demonstrations

• Conducting surveys

• Reviewing the vendor’s customer feedback available to Gartner

• Interacting with industry contacts

• Discussing pertinent topics with clients and nonclients

• Consulting public sources, articles, speeches and published papers

• Gathering input from other Gartner analysts

Teams of Gartner analysts collaborate to evaluate and rate each critical capability for each product or service against the rating criteria. The resulting product or service scores are used to generate a Critical Capabilities analysis.

To determine an overall score for each product in the use cases, each capability score is multiplied by the weighting factor for that capability and use case to derive the use-case product score. The product scores for each vendor are independent and are not weighted or normalized again to produce the simple indexed ranking.

The Magic Quadrant and Critical Capabilities for a given market are typically published at the same time (or close to each other), explaining each vendor’s position and rating, and comparing its products or services. As a result, these documents provide a context for the best use of any comparison. Before publication, the research undergoes rigorous internal peer review and validation, as well as a factual review by the vendors included in the Critical Capabilities.

The update process is followed yearly during the annual refresh cycle, which defines market scope, inclusion criteria, use cases, scoring norms and weightings.

Gartner may produce off-cycle updates to its Critical Capabilities research to reflect the latest material changes to a product or service’s capability score or fit for a use case in rapidly iterating markets. These updates highlight only major changes most relevant to technology portfolio decisions and include new analysis and adjusted scoring for affected vendors.

Analysts decide whether product or service changes materially affect the capabilities or fit for use cases and update the named vendor scoring and profile if necessary.Updated capability scores and analysis feature in interactive and static Critical Capabilities.

The factors Gartner has selected do not represent all capabilities for a given product or service. Therefore, they might not represent the most important capabilities for a specific use situation or business objective. Gartner clients should use a Critical Capabilities analysis as one of several sources of input about a product or service before making a decision.

Gartner’s interactive Critical Capabilities help you gain additional insight into a market and its vendors, and provide a more detailed view of the most relevant products and services. You can use the interactive features to customize the use cases within a Critical Capabilities based on your business needs and requirements, and see which products or services fit best. You can also download all of Gartner’s data and use it as part of your procurement decision-making process.

The option to create client-customized Critical Capabilities content is intended to give you relevant insight into a market, product or service, but it does not represent Gartner’s view. Client-customized views of the Critical Capabilities are for clients’ internal use only and are subject to our Gartner Usage Policy, which is found on our policies overview page.

What is a Gartner Critical Capabilities report?

A Critical Capabilitiesreport is an essential companion to the Magic Quadrant. It is a comparative analysis that scores competing products and services against a set of critical differentiators identified by Gartner. It shows clients which products or services fit best in various use cases and provides action-oriented advice to end-user clients on the products and services they should add to their vendor shortlists for further evaluation.

How does a Gartner Critical Capabilities differ from a Magic Quadrant?

A Critical Capabilities report is a companion note to a Magic Quadrant. The Magic Quadrant positions vendors in a market and presents a vendor’s broadly assessed performance relative to others in the same market. A Critical Capabilities report discretely analyzes the specific capabilities of products and services and presents the results against multiple usage scenarios.

What does it mean if a vendor isn’t included in a Critical Capabilities but appears in the companion Magic Quadrant?

Inclusion in the Critical Capabilities and Magic Quadrant reports are based on vendors meeting the inclusion criteria that are specific to each document. A Critical Capabilities may not feature all of the vendors in its related Magic Quadrant, but it could also feature additional vendors. Consult both documents for the differences in inclusion criteria. Differences in inclusion criteria that exclude a vendor from either report may not be relevant to your specific business need. You should not consider or eliminate a vendor from your selection process based solely on its inclusion in one or both reports.

Are products and services analyzed the same way every year?

No. As the market changes and client needs and requirements evolve, so does the analysis. Changes can occur in inclusion criteria, the use-case scenarios, the capabilities themselves and Gartner’s opinion on the features needed to meet the requirements.

How often are Critical Capabilities updated?

Critical Capabilities are typically updated annually. Gartner also publishes off-cycle, event-driven updates to its Critical Capabilities to reflect material changes that significantly change a product or service’s capability score or fit for a use case. Updates may be necessary when a provider issues a new release of a product or service during the year, or as other significant events occur, such as the opening of new distribution channels. Our methodology ensures only material changes (that is, those most relevant to technology portfolio decisions) drive the publishing of event-driven updates. Refer to Gartner Critical Capabilities: Event-Driven (Off-Cycle) Updates FAQs for more details.

My usage scenario doesn’t match any in the report; can I create my own?

Yes. Gartner clients with access to the interactive version of the Critical Capabilities research can modify an existing use case to create their own, based on their own capability weightings.

Can I add my own capabilities?

Adding new capabilities requires an assessment of each vendor product or service offering against each capability to attribute a rating against a set of predefined ranges. To support this effort, you can reuse the Gartner capability rating data to support your own internal assessment process. We recommend discussing any additional capabilities you want to add with a Gartner analyst to speed your process and avoid adding capabilities that may offer little differentiation.

What is the relationship between Magic Quadrants, Critical Capabilities and Gartner Peer Insights?

Gartner Peer Insights provides ratings and reviews of software and services contributed by end-user professionals. Contributors share their firsthand experience of a vendor and product through the various stages of life cycle management, from evaluation through deployment. Gartner Peer Insights research leverages the structure of the Market Description, Vendor List, Product List and Critical Capabilities (from the related Magic Quadrant and Critical Capabilities) for its user review survey structure and the client experience.With this innovative design (analyst-developed Magic Quadrants and Critical Capabilities, complemented by Gartner Peer Insights), organizations can count on Gartner as their must-have source for a full-spectrum perspective on markets, vendors, products and services. For more on the subject, see Explore Provider Offerings More Deeply With Gartner Critical Capabilities.

What if there is a difference between how a vendor or product is rated in Gartner Peer Insights and other Gartner research, such as Magic Quadrants and Critical Capabilities?

Gartner Peer Insights research reflects the individual opinions and experiences of end users who have submitted reviews and ratings through a rigorous verification, validation and moderation process. Analysts draw on a large base of information from multiple sources to contribute to their research. Although end-user feedback is important, so is feedback and input from vendors, public sources, investors, the press and social media. Lead authors of Magic Quadrants and Critical Capabilities will get access to Peer Insights review data that has been verified, validated and moderated. Analysts will be able to see data throughout the year to review crowd sentiment. A single Gartner Peer Insights review will not affect the placement of a vendor or product in the Magic Quadrant or Critical Capabilities. The Magic Quadrant and Critical Capabilities research process is based on a structured methodology to maintain independence and objectivity, and provide unbiased advice to clients.

Do Gartner’s research analysts use Gartner Peer Insights in their research?

Yes. Gartner Peer Insights reviews are one of the sources of customer input information that may be considered by Gartner experts as part of our rigorous research process. Gartner may consider other sources of customer input information in addition to Gartner Peer Insights, such as analysts’ inquiries with Gartner end-user clients and primary research survey results. Note that while end-user feedback is important, it is just one aspect in a vast range of criteria that are considered.

Gartner Recommended Reading