You’ve seen the ads: Crowds of smiling employees clapping their hands and exchanging high-fives. Why are they so happy? Because their companies had the good sense to install software from SAP. The tagline: “Companies that run SAP are 32% more profitable than companies that don’t.”
That’s an awfully big claim – associating significantly higher profits with the installation of a particular software package – but, as I’ve pointed out before (here and here), there’s no way to validate it. SAP has spent millions on the ad campaign, but it hasn’t made the research public. We get the tagline, not the facts.
Next week, though, a study examining the relative profitability of SAP customers will be made public. It’s not the SAP-sponsored study, however. It’s a study by Nucleus Research, an independent reseach house that specializes in helping companies assess the financial returns on IT investments. The headline of the Nucleus Research study is a bit different than SAP’s advertising tagline: “SAP customers are 20% less profitable than their peers.”
Nucleus Research looked at the 81 public companies that SAP lists as customers on its own website. It determined the return on equity (ROE) earned by each company, based on formal financial filings, and then compared that number with the average ROE for the company’s industry, as calculated by Hoovers. The upshot, as stated in Nucleus’s research note: “Nucleus found SAP customers had an average ROE of 12.6 percent, compared to an industry average ROE of 15.7 percent. It is interesting to note that three areas of significant focus for SAP, customer relationship management (CRM), enterprise resource planning (ERP), and supply chain management (SCM), had customers who fared quite poorly, with CRM customers achieving profitability 18 percent lower, ERP customers achieving profitability 32 percent lower, and SCM customers achieving profitability 40 percent lower than their peers.” In its note, Nucleus documents the data it collected for each of the 81 companies.
In an email exchange, I asked Nucleus’s vice president of research, Rebecca Wettemann, why her firm undertook the study. She replied, “Nothing inspired us other than a few of our clients who commented about [the SAP claims]. We do a lot of ROI investigation and since SAP opened the door to talking about profitability, it seemed reasonable to take a closer look.” She also noted that “no one paid us or commissioned this in any way.” I also asked her if she could explain the discrepancy between the Nucleus findings and the findings promoted by SAP in its ads. “It’s hard to comment on SAP’s study since they won’t share it with us,” she said.
The Nucleus Research study is a limited one, but at least the firm has clearly documented its data and analysis, so readers can draw their own conclusions. One wonders if SAP will now at long last release the documentation of its study. Until it does, Nucleus Research will have the last word in this game of strained statistics: “Despite SAP advertising claims to the contrary, factual analysis of ROE data shows the best-run companies don’t run SAP.”
Nick, not fair. Few weeks ago you pointed out that SAP and Oracle both took credit for customer profitability when they should not have. Why present other evidence which is likely just as spurious about their impact on negative corporate performance. Remember IT doesn’t matter?
amazing exercise. Nick, you must spread the word about this one, branding it like “the best run businesses do NOT run SAP” or soemthing like that and spread it. take tips from scoble. if he can spread brreeereport (or whatever it was) he can spread anything.
then, SAP will be forced to respond. make it as popular as dell hell. this will become interesting.
Vinnie, I’m just reporting the news. Even if we simply take SAP’s much-publicized claim as an example of spurious correlation, the new study still calls the claim into question. People should know about it. Nick
Nick,
“Do not trust any statistics you did not fake yourself” – Winston Churchill. I belive SAP must have some solid facts to backup this claim without which it loses a lot of credibility. I am an SAP Practitioner & have knowledge of companies getting real benefits from SAP ERP thru integration and process standardization. But it is difficult to quantify.
Best, Jagan Vaman
I hope that Nucleus captures in its reportthe objectives behind the SAP implementation, the budgets spent, the integrators employed therein and a snapshot of the SAP implementation story of the studied enterprises,to enable take a look at the qualitative dimension of the findings. Just as aircraft manufacturers can not lay claim to airline profitability (Singapore Airlines Vs American Airline) would be the best example, claims about software directly contributing to profitability the rigor that goes into traditional investments are very hard to justify objectively. Moving forward business would begin to apply more rigor in evaluating IT investments & returns as generally applied to traditional investments. Clearly time to demand more from IT – not for damning IT but for finding better business models for IT companies and more importantly to make user organizations optimize better to get better yields.
Smells like a set-up.. Aren’t you the guy who
said IT doesn’t matter? Now you quote Nucleus studies and
question SAP who may just baiting you to prove IT DOES
MATTER . You have already said
Stratascope is telling the truth, and is honest, but that you
don’t have the SAP study details. What if you did ? and
it showed independance and large enough sample of SAP and
non-SAP to be statisically relevant ? Do you REALLY think that
cherry picking 81 companies off sap.com is a good way for
Nucleus to do research ? If SAP has approx. 30,000 companies, 81
is 0.0027 sampling-not sure I would stand behind that! I am also
not sure I buy Rebecca (Nucleus Founder) on “Aw, we were sitting
around and thought we would do this study.” Does anyone believe
Nucleus is unbiased? How do they get paid to do these things? Can I get in on that ? Go find a single pro-SAP doc they have
done. I CAN find about 30 pro-oracle ones. This nucleus research
study may only prove that SAP.com doesn’t post the most
compelling customers-only those who give permission-suggesting
these companies got some value from SAP.
There are possibly greater minds at work and silence may be a
strategy and not avoidance. If Stratscope did a study with the
integrity you ascribed to its CEO, this could be a brilliant
chess move in a world of one-off claims. After all, if the data
holds out as an observation, it could serve to make both Nucleus
and you look pretty shortsighted and just plain
biased.After all, the GREAT NICK CARR is talking about
it ! What if you are proven wrong ?
Jagan: That’s a great quote – thanks. SAP, Oracle and many other vendors have a history of making broad claims in their marketing that their products bring competitive advantage/superior profitability to their clients. I have yet to see any solid evidence that those claims are true. That doesn’t mean that the products don’t bring real benefits such as increases in productivity. But such benefits don’t bring strategic gains if they end up being broadly shared by competitors.
Sadagopan: From what I’ve seen, the report doesn’t go beyond what I’ve described.
Howard Beale: You overestimate my power enormously – I think SAP has far better things to do than bait me – but thanks for the flattery. If SAP releases the study, I will certainly cover it, and link to it, in my blog. Even Stratascope, though, has already said the study does not show that using SAP software results in superior financial performance, as SAP claims in its television ads. Stratascope says that the study merely shows that companies that have purchased SAP software tend to be more profitable than companies that don’t. The Nucleus Research study calls even this claim of association (rather than causality) into question. It may be that the SAP/Stratascope study is more compelling than the Nucleus study, but that’s impossible to judge because SAP hasn’t released it.
If you’ve read my writing on this subject, you know that I sincerely believe a lot of companies have been ill-served by extravagant and unsupported claims by IT vendors, particularly about competitive advantage and superior profitability. When I see such claims being made, by any vendors, I’ll challenge them. If the vendors making the claims can prove me wrong, I would think they would do so.
By the way, I see you list your web address as USB.com. Wasn’t Howard Beale the name of the USB TV anchorman in the movie Network? Methinks I’m jousting with a shadow.
Nick,
I had written about the same two years ago on a ITToolBox.com blog. A copy of the same is also here.
Your writings are thought provoking.
I also have some viewpoints on “Does IT Matter ? “ that can be shared at the blog for the same.
In case there is a URL, kindly let me know as per your convenience (I could not find it on this webpage).
Best Regards,
Mahesh Khatri.
Kaytek.
http://www.kaytek.co.in
Amen. I’ve been saying it for years – “If there was actually any ROI in any of these enterprise implementations, we’d have heard about it.”
I was in the SAP game briefly (average skills and accomplishments) and I saw enough such that my first reaction when I saw those recent ads was “how do we know their profitability wasn’t actually BETTER than that BEFORE they put in SAP?” How else could they have afforded it, right?
I can’t wait to read the details of the research.
Luckily for me, they don’t play that ad in Australia (yet ?), the IBM ads are bad enough, having to also watch SAP ads would definitely make me install a bucket next to the couch to deposit my dinner in. Anyway, the point I want to make is: Just because the advertisement states that the SAP customers are x% more profitable, it does not have to mean it is because of the SAP implementation and usage. Could it be suggesting that these companies have good enough management to make x% more money than other companies and ALSO have the good judgement to choose SAP ? In other words, good management chooses SAP ? Sounds like the ‘no-one gets fired for choosing IBM’ train of thought. Trust me, I am no SAP fan, but I think we need to think outside of the box here and suggest that we are reading the wrong message from these ads ? Hey, if IT doesn’t matter, then ADVERTISING certainly doesn’t even warrant any attention at all !!! Why are we bothering ? SAA.
Anyone see Oracle’s ad in today’s WSJ? It has a direct call-out from the Nucleus Study showing SAP customers are 20% less profitable. The saga continues…