How Should You Measure The Benefits of Generative AI In Your Business?

It’s only been a year and a half since Generative AI began reshaping the world of business. While many businesses have begun experimenting with it, much of that work to date has been informal – just seeing if the technology works in one application or another, without establishing hard metrics or ROI.

Generative AI is a challenging tech, though – it’s so flexible that it’s not always clear if it’s delivering value in the way you expect. Here are some ideas you can use to measure the benefits of Generative AI for your business.

What are the core business benefits of Generative AI?

There are three main ways that GenAI can improve your business:

“Quality” can also apply to creative tasks; the AI can provide a good starting point with a starting draft or rough sketch, helping humans create a better final product than they would have otherwise.

Of these, efficiency or productivity is by far the easiest to measure, using time studies or throughput. Quality is typically more subjective, and is frequently a dimension to be controlled versus directly targeted.

Internal metrics for Generative AI

You’ll use a different set of metrics depending on who the customer is for your process. If the AI-aided process lands with your end customer, you’ll use external metrics. If the results are used internally or are primarily operations-focused, you’ll focus instead on internal metrics.

The Klarna case study provides an excellent example of using a mix of metrics to validate the impact of generative AI for their business. Here are some internal metrics they used, as well as some others to consider:

External metrics for Generative AI

Again, Klarna provides an excellent example of using external metrics to validate the impact of generative AI for their business. Metrics they targeted included:

Other key metrics to consider may be:

Planning your Generative AI implementation

Clearly, these are a lot of options to choose from, and not all of them matter for your business. The important point here is figuring out the handful that matter most to you, and focusing on them.

Then, with those in hand, spend a some time to develop a plan with the appropriate stakeholders. Key things to consider:

  1. What are the tasks you will focus on?
  2. What metrics make the most sense, and are valuable to you or your customers?
  3. How will you measure how the tasks are done today, and over what period?
  4. How will AI fit into the process?
  5. What resources (time/money) are required to implement the AI-aided version?
  6. Over what period will you test the AI-aided version?
  7. Given your metrics, what levels are needed to be considered successful?
  8. Are there any risk/compliance/privacy/safety/ethical implications that must be addressed before proceeding?

Then execute on the plan. The level of detail and formality should be proportional to your investment. Your approach will look much different if it’s low stakes and quick wins versus a more complex and high-visibility deployment.

Given how fast generative AI is shaping business – and how quickly you can see the benefits – the key is to simply get started.