Does Technology Selection Need Yet Another Assessment and Evaluation Mechanism?
Research firm Horses for Sources recently introduced its HfS Blueprint methodology for business process outsourcing capabilities which they argue is an assessment methodology that is not reliant on the arbitrary viewpoint of a single analyst. HfS is obviously pitching an alternative to either Gartner’s Magic Quadrant or the Forrester Wave methodology.
Our view is that technology vendors will embrace any and all alternatives to being held hostage to a single industry analyst firm. However, for technology buying teams, while these evaluations can be helpful, they must be placed in a proper balanced context. Assessment of any technology vendor or service provider should be determined by consistency in rating among multiple analyst firms. That would include that a vendor has demonstrated a consistently positive presence over a period of time, for instance two or more assessments. Additional attractiveness will be dependent on a number of other factors, including continual evaluation of more mature technologies and/or services which traditional industry analyst firms tend to de-emphasize, along with consistency in definition of the application and/or service.
Buyer reliance on these methodologies has been waning because there are so many other additional sources of information that selection teams can tap on their own. They would include independent voices. Access to these methodologies also comes with a price that being a subscription to the individual analyst firm’s evaluation services. That is where the win-win proposition plays itself out for vendors and traditional analyst firms. Vendors woe and lobby individual analysts to be rated favorably while the analyst firms garner a steady stream of subscription revenues. Any technology product marketing executive is all too aware of how subjective individual analyst rankings can become, especially when evaluation criteria is constantly in a state of purposeful change.
According to the HfS description of its methodology, vendors and service providers will have their capabilities ranked by two high level criteria, Execution and Innovation. That is somewhat similar to the two axis of Gartner’s MQ.
The differentiators for Blueprint are described as crowdsourced inputs from four major constituencies: an Executive Council of Enterprise Buyers, Service Providers themselves, a body of advisors, consultants and other industry stakeholders, and finally, the HfS analyst team. What seems not to be described is the overall weighting applied to each evaluation body. Perhaps that will be better clarified.
Data is collected from interviews with buyers, vendor briefings, and available information. Again, we cannot determine the overall weighting applied to each of these inputs.
Supply Chain Matters applauds the availability of alternative evaluation methodology for technology selection teams and we wish HfS well in its endeavor.
However, by our view, success of this or any new methodology will be highly dependent on the weighting criteria of major constituency viewpoints, perhaps with a stronger voice for buyers and the body of advisors, consultants and other industry influencers that are described as part of this process. The more weighting that is placed external to the analyst firm, the more such methodologies can be viewed as objective and balanced.
Then again, the existence of a yet another methodology is another reference that can serve as a tie breaker for technology evaluation teams.