Back in November, I expressed some skepticism about a claim that software giant SAP made in an email newsletter it sent out. The newsletter reported that “a recent study of companies listed on NASDAQ and NYSE found that companies that run SAP are 32% more profitable than those that don’t.” The email included a link that promised “We invite you to see for yourself,” but instead of bringing you to details about the study, the link just brought you to a SAP web page with more marketing pitches. I did a considerable bit of searching but, as I reported, I was unable to find any documentation of the study – either on SAP’s site or on the site of the company that performed the study, Stratascope. But I did find that SAP is a big client of Stratascope’s. I wrote: “Now, the fact that SAP and Stratascope are cozy – and that it’s in Stratascope’s interest to make its client happy – doesn’t mean that Stratascope’s research is necessarily unsound. But it does raise questions – questions that can only be answered through a careful review of the research methodology and results.”
Although I didn’t realize it at the time, the SAP newsletter was just the start of a huge advertising campaign built around the “32% percent more profitable” study. If you’ve opened a business magazine recently, or watched any cable TV, you’ve likely seen the ads. To the best of my knowledge, though, SAP still hasn’t released the details of the research.
Today, Stratascope’s CEO, Bruce Brien, posted a comment to my November post, which I reprint here in full:
It is interesting that Mr. Carr, in his detailed research for this article, never contacted us at Stratascope Inc. The study was commissioned by SAP so it will not be published on our website. SAP owns the study. I would be more than happy to explain our methodology and calculations, as well as our auditing and quality assurance procedures that we use whenever we undertake such a study for any of our clients. All Mr. Carr had to do was ask.
I can also tell you that SAP commissioned the study without any idea what the results would show. The Ad campaign was created as a result of the study and not the other way around.
We also need to understand a little about statistics. Hypothetically, if the average non-SAP client in the study was showing a profit margin of 6%, a 32% higher margin would be only 7.92%. The numbers are not so hard to imagine in this light.
Finally, it is also clear that this is a success by association Ad campaign. There are no claims in any aspect of the campaign that imply that the client’s success is because of SAP.
Although bespoke research always makes me nervous, I’m willing to accept Brien’s assurance of the study’s integrity. Nonetheless, I continue to believe that SAP has an obligation to release the full report so that we can judge the basis and reliability of its claim that the use of SAP is associated with significantly superior profitability. In fact, Brien’s comment leads me to believe that such a disclosure is all the more important because SAP appears to be mischaracterizing the Stratascope study. Brien underscores that the research does not show that the use of SAP causes superior profitability; it’s just associated with superior profitability. Brien goes on to write that “There are no claims in any aspect of the [ad] campaign that imply that the client’s success is because of SAP.” Oh yeah? Here is a transcript of the voiceover from a SAP television ad:
The right software can make any size company more efficient, more agile, more responsive. In short, make your company more. That’s why companies that run SAP are 32% more profitable than companies that don’t.
After receiving Brien’s comment, I shot him an email, asking: “Doesn’t the ‘that’s why’ [in the SAP ad] suggest causality rather than just association?” He replied:
You are correct. I guess I didn’t pay enough attention to the voiceover. We try to make it very clear to our clients that software does not make companies perform better, that software cannot improve your bottom line and that since most software is not free it will cost you money.
If the people running the company make better decisions because of the software, then they can impact the performance of their business. This campaign was played back to us as an association type campaign where smart people who run great companies are choosing SAP and you should too.
Brien’s a straightshooter. I wish I could say the same for SAP’s marketers.