- The cost of Gartner. They charge organizations tens of thousands of dollars for very basic access to their research and analysts. Solution providers that fare well in their reports pay for redistribution rights at the cost of tens of thousands of dollars. If a solution provider or organization wants a strategy day with Gartner it is typically more than $15,000 for a day of advisory. My issue here is one of context and setting the stage. One would think their research would be deep and thorough as a result. This is not the case. Obviously, organizations are willing to pay for this even though it is outrageous. But the assumption would be that there would be deep methodologies and transparency in their research at these rates. They are trying to automate, streamline, and make more money by cutting corners. Let us now unpack this further . . .
- Lack of consistency in evaluating solutions in Magic Quadrants. When it comes to several of the Magic Quadrants in GRC related areas, they are primarily asking for video demos. This does vary, as some Magic Quadrants do want live demonstrations. But the fact is that Gartner is inconsistent. For many of these Magic Quadrants they are not actually sitting behind the solution, navigating through it, and figuring it out how it works, all they want is a video submission. This makes their rankings in Magic Quadrants nothing more than a beauty contest in who can provide the best video demo of functionality that may or may not actually be there. They are not engaging solution providers on a fair playing field and validating functionality. Gartner analysts are often not actually working with these solutions they are ranking and scoring. They may fall back and state this is because they have previous experience with these solutions, but this is cutting corners. If you are publishing research ranking solutions then you should go through each solution step by step in a defined methodology and evaluation. A video submission does not cut this.
- No transparency in Magic Quadrants. When it comes to Magic Quadrants, they are what they say they are . . . MAGIC. No one but Gartner knows how solution providers are measured and scored. Forrester, on the other hand, publishes all their criteria for Waves. With Gartner no one has any idea about the criteria and scores for vendors plotted on their Magic Quadrants. For example, the Operational Risk Magic Quadrant, the only way I can imagine the solutions plotting out the way they do on this is if Gartner is weighting IT security extremely high. If it was true operational risk management capabilities across operational risk areas there is no way the solutions would plot the way they do. But no one can really determine this as Gartner will not reveal criteria or scoring. This is bad research. Evaluations should be fully transparent and allow organizations to see how solutions score on specific criteria and adjust for their own needs.
- Simplifying client reference checks. This is exacerbated by how they are streamlining client reference checks. They used to get on the phone and talk to client references and ask them the hard questions. Now there is more reliance on sending web surveys to client references. Surveys that solution providers, in some cases I am aware of, are providing pre-populated answers for their references. This is not fair. When I do reference checks I talk to clients of solution providers. Furthermore, I not only talk to the references solution providers provide, I also ask to talk to others on their teams that use the solution every day. Decision makers give glowing references, you often find a different story with the people that use a solution day in and day out. You cannot get to the dirt and issues that organizations need to understand when making purchasing decisions for solutions by sending out a survey form. Deeper conversations with stakeholders are so much more valuable than an automated survey.
- Putting a new coat of paint on the same thing. My latest issue with Gartner is their relabeling of GRC to IRM (Integrated Risk Management). From my perspective, this is just putting a new coat of paint on the same thing. To me, it makes no sense. Organizations, associations, professional service firms, solution providers, and more have invested in GRC. So, why would they do this? Perhaps to leverage their position, creating some differentiation for Gartner? But let me ask the key question – does this help the market? I see no benefit to this name change, just obfuscation. If they do not like the acronym GRC, then just fall back to ERM (enterprise risk management). As an aside, GRC is a better acronym in my opinion. By the official definition (from OCEG), GRC is an integrated capability to reliably achieve objectives [governance], while addressing uncertainty [risk management], and act with integrity [compliance]. There is a natural flow to this and puts risk management and compliance in context of governance and objectives.
Gartner, in context of governance, risk management, and compliance (GRC) related research, is ignorant and harmful to organizations that rely on their research publications and advice. In full disclosure, Gartner is my competitor. I have been an analyst for seventeen of my twenty-four years as a GRC professional. I spent seven years at Forrester Research, Gartner’s primary competitor, and the past ten years on my own as an independent market research analyst and advisor. Forrester I have a lot of respect for, although I wish their research on GRC related areas was deeper and evolving to keep up. Verdantix is another competitor that I have deep respect and admiration in the quality and thoroughness of their research, though they only cover a segment of the GRC market in environmental, health, and safety (EH&S). On the other hand, it is perilous to rely on Gartner’s GRC research. My rants on Gartner are the most popular commentaries and posts that I do, but also the hardest. I am not trying to take cheap shots at a competitor. I care about this space and find the market for GRC related solutions, content, and services to be as much a passion for me as it is a career. I provide this commentary because organizations need to be wary of what and how Gartner is doing this research. Specifically, I am talking about Gartner’s GRC related research and not all their research. I have former colleagues that I deeply respect that now work for Gartner. I can’t just stay idle on their approach to their GRC related research, it would not be professional on my part. My issues with Gartner and their approach to GRC related research run deep, these include: