How to Determine the ROI of Your AI

Do you know how to tell who is a true pioneer? Look for the arrows in their backs. Two years of consulting within the artificial intelligence space will teach you a lot. I have arrows to prove it. Lesson one: Just because you can fund AI projects doesn’t mean you can do an AI project. Lesson two: Promising proof of concepts seldom become promising deployments. Lesson three: Deploying a successful AI capability does not mean people will actually use the capability. No matter how much effort Google and Amazon put forth into making an AI easy button, deploying artificial intelligence capabilities that provide measurable benefits is still quite hard. 

A recent study out of MIT’s Sloan Management Review concluded that although the interest in artificial intelligence in business is increasing rapidly, the success stories are harder to find: 

“Many AI initiatives fail. Seven out of 10 companies surveyed report minimal or no impact from AI so far. Among the 90% of companies that have made some investment in AI, fewer than 2 out of 5 report business gains from AI in the past three years. This number improves to 3 out of 5 when we include companies that have made significant investments in AI. Even so, this means 40% of organizations making significant investments in AI do not report business gains from AI.”

Yet, some businesses are producing pretty incredible returns: 

“Our biggest [AI] win [at Amazon] was several billion dollars in revenue, which translates to maybe $100 million in profit because the retail business is actually a relatively low-margin business.” — Charles Elkan – former head of ML at Amazon

This new frontier is still wild and full of pitfalls. While the interest and investment in artificial intelligence is undeniable, seeing return on that investment is challenging. So why is it so hard to find ROI in AI projects? And how are some businesses able to produce incredible returns?

Why Projects Fail

The traps are bountiful when embarking on an artificial intelligence project journey. From my observations in the field, these are the top reasons projects fail. 

  1. Lack of vision. Many AI projects are considered research and development efforts and have no clear objective beyond just innovation. Any AI or machine learning project must have a clear business purpose or be rooted in a problem that already has a business case worth solving. 
  2. Bad data. Many companies lack strong data competency. They don’t have a strategic way to collect data, store data, and make data accessible. Most companies produce and collect mountains of data, however the data is poorly structured or difficult to access. 
  3. Culture. Most companies do not have a culture that can readily embrace emerging technologies in their operations or a process to identify where the next wave might be coming from. Others don’t have strong data literacy or routinely make data informed decisions. Culture can quickly kill the rollout of promising AI capabilities. 
  4. Unclear use cases. Many businesses lack education on what problems are good problems for AI to solve. They either underestimate how AI can be applied to the business or overestimate the general capability to solve business relevant problems. 
  5. Patience. Businesses don’t give themselves the patience to take the time and fail your way to success. 

In fact, according to the MIT Sloan Management Review report: Winning with AI, it takes 11 or more deployments of AI for businesses to claim to see “moderate” (43%) and “substantial” (49%) benefits from their AI tools in production. Those that make it that far have a strong vision, learn from mistakes, and take the time to see great benefit. 

Finally, we all are generally bad at goal setting and measurement. For example if you don’t know what a SMART goal is, and are not using them today, you will likely run into problems finding success in AI. A top reason AI programs fail is because we have a hard time defining and measuring success. Instead of deploying an AI solution and then measure the impact, try the reverse. Build a business case around a specific problem, forecast impact, then see if the AI solution measures up. Pick problems that are easy to measure. Using personalization to increase click-through rate is much easier to measure than personalization to lift brand affinity. Business fail when they don’t understand which problems are worth solving. A former Amazon executive recently went on record stating that one AI project generated billions of dollars in revenue:

“…How do we get a win of several billion dollars in revenue? We improve a business that already has several tens of billions of dollars in revenue. There’s no secret there: Start with something big and then, if you can improve it, you get a big improvement.” — Charles Elkan – former head of ML at Amazon

How To Set Up For Success.

Do the opposite of the above why projects fail. But seriously, here are some key takeaways: 

  1. Start with the business problem and measurement, then data.
  2. Deploy AI for only very targeted use cases.
  3. Your success will depend on the data.
  4. Include all stakeholders in the project selection
  5. Deploy in many small milestones

A pro tip for picking the right problem is to pick a problem that already has a business case for solving. If a more general problem has a business case already, funding will follow. It is unlikely that you the reader (assuming you are the one who will lead an AI project) will have both the skillset and time to frivolously build business cases that have no intention of an AI solution. Instead cherry-pick problems that are already grounded in a business case. 

Then, try to solve that problem in a small way. Businesses that launch AI successfully understand that data capabilities do not solve general problems, but work great solving narrow ones. The narrow solution will contribute to the overall business case, but not solve the whole problem. 

Do’s and Dont’s for unlocking ROI

DO: include business leaders, data scientists, and engineers when selecting the problem to solve to ensure all perspectives are recognized and goals are aligned. 

DONT: begin a project without the knowledge of what AI can do.

DO: pick problems that are easy to measure. For example, a focus on thwarting equipment failure with proactive maintenance is easy to measure the before and after. If you are looking to improve customer experience, that is far more difficult to measure and hard to prove ROI on any AI spend. 

DO: test new project ideas in small pilots. Follow Andrew Ng’s advice to shoot for first AI projects with 6-12 month timeframes, not massive multi-year roll-outs. Meaning limited data, limited functionality, limited users. 

DON’T: forget about change management. Just because you deployed an AI solution does not mean people know how, or want to, use it. Having a successful rollout plan is key. 

How To Calculate ROI

There are four traditional ways to calculate return on investment:

Break-Even Analysis 

BEA illustrates how many sales are needed to recoup the overall investment. This is expressed in units sold. This type of measurement is best for market-focused projects, such as product development and entrepreneurial endeavors. Break-Even Analysis is not great when you have high, time-dependent operating costs or infrastructure benefits since it is impossible to quantify the impact on sales. 

Payback Period

Payback Period states the time period required to recoup the investment. It is expressed in months or years. This approach is best for projects with heavy upfront investment like facilities projects, or productivity projects that accumulate benefits over time. 

Net Present Value

Net Present Value states today’s value of the long-term investment by discounting for future cash flows. This allows you to see what the 10-year impact of the present project can be, expressed in today’s total dollar amount. This metric is in total dollars and demonstrates the total project value to the business. NPV gives the most complete view of return on investment. 

Internal Rate of Return

IRR may be the most traditional way to measure return. It measures the rate of return the project can deliver over the life cycle. IRR is expressed in a percentage. This is especially beneficial for projects where the company reports to investors or borrows money. The goal is to compare the IRR of a project to other long term investments, like parking money in a CD or mutual fund. This is the metric you want to use if you are fundraising with your project at the center.

Net Present Value and Internal Rate of Return are the best calculations to express ROI on AI projects from my experience. However all these calculations can be useful to express value in for different projects. 

Right now, you are likely in one of two camps. Perhaps you are interested in proceeding with an AI project and work for an organization that requires measurable return for any new innovation project. Or you are skeptical AI can drive value and are looking for ways to prove or disprove the validity. To the latter, you are right. Many AI projects fail. But they fail more often because project leaders pick the wrong problem to solve or do not know how to measure success. It is most important to proceed with caution but still proceed. Why?

There is $13,000,000,000,000 on the line. The 4th Industrial revolution, like every other revolution, there will be winners and losers. Just don’t lose on the technicalities of problem assessment or lack of measurement.