Gartner: Missing the Risk & Compliance (GRC) Target
Gartner, in context of governance, risk management, and compliance (GRC) related research, is ignorant and harmful to organizations that rely on their research publications and advice.
In full disclosure, Gartner is my competitor. I have been an analyst for seventeen of my twenty-four years as a GRC professional. I spent seven years at Forrester Research, Gartner’s primary competitor, and the past ten years on my own as an independent market research analyst and advisor. Forrester I have a lot of respect for, although I wish their research on GRC related areas was deeper and evolving to keep up. Verdantix is another competitor that I have deep respect and admiration in the quality and thoroughness of their research, though they only cover a segment of the GRC market in environmental, health, and safety (EH&S). On the other hand, it is perilous to rely on Gartner’s GRC research.
My rants on Gartner are the most popular commentaries and posts that I do, but also the hardest. I am not trying to take cheap shots at a competitor. I care about this space and find the market for GRC related solutions, content, and services to be as much a passion for me as it is a career. I provide this commentary because organizations need to be wary of what and how Gartner is doing this research. Specifically, I am talking about Gartner’s GRC related research and not all their research. I have former colleagues that I deeply respect that now work for Gartner. I can’t just stay idle on their approach to their GRC related research, it would not be professional on my part.
My issues with Gartner and their approach to GRC related research run deep, these include:
- The cost of Gartner. They charge organizations tens of thousands of dollars for very basic access to their research and analysts. Solution providers that fare well in their reports pay for redistribution rights at the cost of tens of thousands of dollars. If a solution provider or organization wants a strategy day with Gartner it is typically more than $15,000 for a day of advisory. My issue here is one of context and setting the stage. One would think their research would be deep and thorough as a result. This is not the case. Obviously, organizations are willing to pay for this even though it is outrageous. But the assumption would be that there would be deep methodologies and transparency in their research at these rates. They are trying to automate, streamline, and make more money by cutting corners. Let us now unpack this further . . .
- Lack of consistency in evaluating solutions in Magic Quadrants. When it comes to several of the Magic Quadrants in GRC related areas, they are primarily asking for video demos. This does vary, as some Magic Quadrants do want live demonstrations. But the fact is that Gartner is inconsistent. For many of these Magic Quadrants they are not actually sitting behind the solution, navigating through it, and figuring it out how it works, all they want is a video submission. This makes their rankings in Magic Quadrants nothing more than a beauty contest in who can provide the best video demo of functionality that may or may not actually be there. They are not engaging solution providers on a fair playing field and validating functionality. Gartner analysts are often not actually working with these solutions they are ranking and scoring. They may fall back and state this is because they have previous experience with these solutions, but this is cutting corners. If you are publishing research ranking solutions then you should go through each solution step by step in a defined methodology and evaluation. A video submission does not cut this.
- No transparency in Magic Quadrants. When it comes to Magic Quadrants, they are what they say they are . . . MAGIC. No one but Gartner knows how solution providers are measured and scored. Forrester, on the other hand, publishes all their criteria for Waves. With Gartner no one has any idea about the criteria and scores for vendors plotted on their Magic Quadrants. For example, the Operational Risk Magic Quadrant, the only way I can imagine the solutions plotting out the way they do on this is if Gartner is weighting IT security extremely high. If it was true operational risk management capabilities across operational risk areas there is no way the solutions would plot the way they do. But no one can really determine this as Gartner will not reveal criteria or scoring. This is bad research. Evaluations should be fully transparent and allow organizations to see how solutions score on specific criteria and adjust for their own needs.
- Simplifying client reference checks. This is exacerbated by how they are streamlining client reference checks. They used to get on the phone and talk to client references and ask them the hard questions. Now there is more reliance on sending web surveys to client references. Surveys that solution providers, in some cases I am aware of, are providing pre-populated answers for their references. This is not fair. When I do reference checks I talk to clients of solution providers. Furthermore, I not only talk to the references solution providers provide, I also ask to talk to others on their teams that use the solution every day. Decision makers give glowing references, you often find a different story with the people that use a solution day in and day out. You cannot get to the dirt and issues that organizations need to understand when making purchasing decisions for solutions by sending out a survey form. Deeper conversations with stakeholders are so much more valuable than an automated survey.
- Putting a new coat of paint on the same thing. My latest issue with Gartner is their relabeling of GRC to IRM (Integrated Risk Management). From my perspective, this is just putting a new coat of paint on the same thing. To me, it makes no sense. Organizations, associations, professional service firms, solution providers, and more have invested in GRC. So, why would they do this? Perhaps to leverage their position, creating some differentiation for Gartner? But let me ask the key question – does this help the market? I see no benefit to this name change, just obfuscation. If they do not like the acronym GRC, then just fall back to ERM (enterprise risk management). As an aside, GRC is a better acronym in my opinion. By the official definition (from OCEG), GRC is an integrated capability to reliably achieve objectives [governance], while addressing uncertainty [risk management], and act with integrity [compliance]. There is a natural flow to this and puts risk management and compliance in context of governance and objectives.
Organizations are relying on Gartner to produce quality research. They are spending tens to hundreds of thousands of dollars with Gartner. Worse, they are making investment decisions in GRC solutions with licensing that can costs hundreds of thousands a year for some organizations. Gartner is failing these organizations by cutting corners and not going deep and working with these solutions first hand. Defining proprietary markets and researching them with video demos, web survey references, and opaque scoring criteria is robbery for what Gartner charges both organizations evaluating solutions as well as the solution providers themselves.
I personally wish Gartner would ask about usability. I get so many complaints about Leaders in Magic Quadrants and Forrester Waves that struggle with interfaces that are not intuitive, difficult to use, and often look like they were coded over a decade ago. I would love to see them say “in a live environment, configure this solution” then have them “demonstrate how the solution works.” This would show the front and back end of the products they are evaluating. They do a terrible job at differentiating products. For example . . . ask them to compare the workflow functionality of four products and they cannot. Ask them how the products differ when importing information and they cannot.
Gartner also has dropped very important areas of GRC related research, particularly Environmental, Health & Safety (EH&S). I am seeing more and more RFPs that are include EH&S as a primary focus of GRC yet Gartner abandoned this a few years back. Largely, Gartner appears to see GRC (or what they now call IRM) related solutions predominantly through an IT security point of view, as I reference with the Operational Risk Magic Quadrant, and is also apparent in their Vendor Risk Magic Quadrant.
Bottom Line on Gartner: Gartner’s approach to their risk and compliance research (e.g., GRC, IRM) is disloyal, dishonest, untrue, treacherous, and unfair from the part of an analyst who is supposed to be a trusted advisor to many. It’s outrageously expensive, but not just that: expensive for no value.
NOTE: While I have greater respect for Forrester, things need to evolve there as well. Forrester publishes their criteria and scoring, thus is transparent. But their criteria is at a high level and has not evolved much over the years. It also concerns me that they rank client satisfaction so low, where someone that scores a 1 out of 5 on client satisfaction can be positioned highly in a Wave while someone that scores a 5 out of 5 does not.