How Should You Measure The Benefits of Generative AI In Your Business?

It’s only been a year and a half since Generative AI began reshaping the world of business. While many businesses have begun experimenting with it, much of that work to date has been informal – just seeing if the technology works in one application or another, without establishing hard metrics or ROI.

Generative AI is a challenging tech, though – it’s so flexible that it’s not always clear if it’s delivering value in the way you expect. Here are some ideas you can use to measure the benefits of Generative AI for your business.

What are the core business benefits of Generative AI?

There are three main ways that GenAI can improve your business:

• Efficiency. The AI can take on tasks that a human may have done otherwise, reducing or eliminating the time required to complete the task.
• Availability. With humans, you have to work around their schedule and loading. With AI, you just call it when you need it (as long as the service is running!).
• Quality. The AI must do the same or better quality of work than a human could have completed. (This is especially true for junior roles – studies have shown that AI tends to elevate poor performers to average level, while having limited quality impact on average or excellent performers.) This includes issues like mistakes or error rate.

“Quality” can also apply to creative tasks; the AI can provide a good starting point with a starting draft or rough sketch, helping humans create a better final product than they would have otherwise.

Of these, efficiency or productivity is by far the easiest to measure, using time studies or throughput. Quality is typically more subjective, and is frequently a dimension to be controlled versus directly targeted.

Internal metrics for Generative AI

You’ll use a different set of metrics depending on who the customer is for your process. If the AI-aided process lands with your end customer, you’ll use external metrics. If the results are used internally or are primarily operations-focused, you’ll focus instead on internal metrics.

The Klarna case study provides an excellent example of using a mix of metrics to validate the impact of generative AI for their business. Here are some internal metrics they used, as well as some others to consider:

• Productivity: At it’s core, it is how much a system can produce within a specified period of time. This might focus on a single employee. At a larger scale, this might be an entire business process.
• Efficiency: Related to productivity, this focuses on the amount of resources (like time) are needed to produce an output. Motorola Chief Technology Officer Mahesh Saptharishi noted that this was a particular focus for his company, “tracking the hours it takes to complete a task with a generative AI tool and without it.” Likewise, Klarna’s legal team reported that GenAI let them complete a 1 hour contract drafting task in 10 minutes.
• Timeliness: When a resource is needed, is that resource more readily available than it is today? Or – focusing on the end of the process – does it better deliver a result when that result is expected/needed? AI will likely win out here frequently.
• Accuracy/error rate: How often is the AI right (or close enough to right) versus how often correction is needed? If used for planning or prediction, you can compare the accuracy of the AI’s estimate with the actual outcomes after the fact. Autodesk reports that they are using generative AI within the finance team for “forecasting and identifying patterns in customers’ consumption of cloud services”, allowing them to plan their own cloud expenditures.
• Revenue: Did use of AI in an area of the business – say, greater sales performance from more effective outreach – improve top line revenue?
• Cost/ROI: Given the improvement in resource position (employee time, increased revenue, etc.), how long did it take to recover your investment? For most small AI use cases, if it is accurate and repeatable, the answer should be a quick ROI.
• Profitability or profit contribution: It’s often difficult to attribute overall business profitability to AI use specifically (unless you’re, you know, an AI business), but looking at incremental contributions based on revenue, savings, or profit is always valuable. Klarna saw a $40M incremental profit improvement from their AI use.
• Employee satisfaction: Hiring is expensive and time consuming. If using AI makes work less arduous and reduces turnover, that is real value delivered to the business.
• Incremental hiring requirements: Klarna found that the AI systems did the work of 700 agents – allowing Klarna to expand their business without having to expand their staffing. In most cases, AI won’t fully replace existing humans at a business, but it may slow the rate at which you have to hire as you expand.

External metrics for Generative AI

Again, Klarna provides an excellent example of using external metrics to validate the impact of generative AI for their business. Metrics they targeted included:

• Customer satisfaction: How do your customers appreciate working with an AI versus a person? Klarna found that AI delivered similar satisfaction scores versus human agents.
• Repeat inquiries: Does the AI fully resolve the issue the first time, or must customers come back to fix their issue? Klarna found that AI decreased repeat inquiries 25%, reducing the overall customer service capacity needed.
• Time to resolution: Klarna found that issues were solved 85% faster on average than human-only methods. This allows for higher throughput, as well as lower resource needs.

Other key metrics to consider may be:

• Customer retention or engagement: AI can be more reactive and persuasive with your customers individually than a salesperson. WSJ reports that Mailchimp is A/B testing AI-assisted mailings to see if they increase reader engagement.
• Value perception: Does the addition of AI make your product more valuable to you customers? This is the path that Google, Microsoft, and other big players have been pursuing as they add AI features to their existing software products.

Planning your Generative AI implementation

Clearly, these are a lot of options to choose from, and not all of them matter for your business. The important point here is figuring out the handful that matter most to you, and focusing on them.

Then, with those in hand, spend a some time to develop a plan with the appropriate stakeholders. Key things to consider:

1. What are the tasks you will focus on?
2. What metrics make the most sense, and are valuable to you or your customers?
3. How will you measure how the tasks are done today, and over what period?
4. How will AI fit into the process?
5. What resources (time/money) are required to implement the AI-aided version?
6. Over what period will you test the AI-aided version?
7. Given your metrics, what levels are needed to be considered successful?
8. Are there any risk/compliance/privacy/safety/ethical implications that must be addressed before proceeding?

Then execute on the plan. The level of detail and formality should be proportional to your investment. Your approach will look much different if it’s low stakes and quick wins versus a more complex and high-visibility deployment.

Given how fast generative AI is shaping business – and how quickly you can see the benefits – the key is to simply get started.

Become your company’s AI expert in under 30 minutes a month by signing up for the Executive Summary newsletter by AI For Business.

If you liked this post, have future posts delivered straight to your inbox by subscribing to the AI For Business newsletter. Thank you!