US retailers spent $50 million a year bidding on 1 million keywords on Google. This spending generated $500 million in sales (equivalent to ROAS, or return on ad spend of 10). They were very happy with the results and were planning to increase their spending.

But when we helped retailers analyze their performance at the keyword level, a different picture emerged. Overall performance was good, but they were spending $7 million a year on thousands of long-tail keywords with zero sales. Google’s bidding algorithm had embedded parameters that determined the acceptable cost before pausing a particular keyword. Changing the value of this one parameter saved us $7 million a year and had no impact on our sales.

Faced with millions of keywords, it’s tempting to retreat to simplified aggregate averages such as ROAS (which is what we’re here for, after all). However, looking at aggregated results is not enough to manage and optimize algorithms that operate at the atomic level. This requires new approaches to understanding algorithms and managing their performance.

Algorithms are needed for digitization

In the past, companies had sluggish mechanisms that allowed them to operate at an aggregate level, but digital technology now enables surgical, granular decision-making across the enterprise. The amount of these decisions requires automation, and algorithms define the assumptions, data, and logic that determine how micro-decisions are made.

These algorithms are increasingly prevalent in digitally-driven businesses and are used for pricing, promotions, inventory allocation, supply chain, resource scheduling, credit decisions, fraud, digital marketing, product sorting ordering, recommendations, and personalization. We make decisions in a wide range of areas. A common feature of these decisions is that they have inherent uncertainties that require trade-offs. For example, digital marketing decisions involve tradeoffs between volume and profit. Supply chain decisions involve tradeoffs between waste and availability. Resource decisions have cost-of-service tradeoffs.

Business leaders recognize the need to manage and optimize these new decision-making systems to avoid trade-offs and drive continuous improvement. But how? Metrics and KPIs are mechanisms for management control, but traditional approaches to reporting don’t work for decisions powered by these new algorithms, as explained below.

Algorithm structure

To manage a decision algorithm, we must start by understanding how it is constructed. It is helpful to decompose the properties of the algorithm into four P’s and highlight the trade-offs made with each.

Purpose: What is the purpose of the algorithm?

Define main objectives and guardrails or constraints. There is usually a trade-off between choosing a complex corporate objective (such as profit) and choosing an alternative or simpler, siled objective (such as ROAS). For example, choosing between maximizing sales and maximizing profit within ROAS guardrails has very different decision logic.

Accuracy: HHow are microdecisions personalized?

For example, if you’re a retailer bidding on millions of keywords, should you set one ROAS target for all keywords? Or millions of “personalized” keyword-level targets? or something in between? The trade-off is management. Are the human resources required to set atomic-level strategies worth the effort?

Prediction: H.How is uncertainty modeled?

This could be a simple extrapolation or a very complex AI/ML model. Additionally, predictive models can be simplified as microdecisions become more frequent. For example, a European airline, which used to set prices on a weekly basis, had developed a very sophisticated predictive model of projected fill rates. As they evolved, prices changed from weekly to every few minutes, allowing us to meet actual demand rather than requiring a forecasting model. Between interpretability, and decision-making frequency.

Policy: WAre there rules/logic/mathematics that determine the actual micro-decisions?

The trade-off here is between simple, easy-to-understand algorithms and more complex formulas that give better results but can only be understood by experts. For example, you can define keyword bids using simple rules or complex regression equations.

These examples aresatisfaction” Developed by Nobel Prize-winning economist Herb Simon. It’s a choice between finding the best solution for a simplistic world, or finding a satisfying solution for a more realistic world. There is no “perfect” algorithm. But now we have the data and the tools to change our minds and find a “good enough” solution in the real world.

As we have seen, the nature of algorithms requires new types of trade-offs at both the micro-decision level and the algorithmic level. A key role of the leader is to navigate these trade-offs on an ongoing basis, not just when the algorithm is being designed. Algorithm improvements tend to change the rules and parameters of the software. It’s more like adjusting a graphic equalizer knob than redesigning a physical plant or deploying a new IT system.

Algorithmic new metrics

Algorithms are often treated with too much respect. The algorithm is smart, so it must be correct. And in many cases, the algorithms have significantly improved on what “was done before.” But this kind of cognitive bias can leave managers in a false sense of complacency.

Metrics are important for measuring algorithm performance, but they are also important for highlighting opportunities for improvement, especially where trade-offs may not be optimal. The different nature of the data produced by these digital decision-making systems motivates new approaches.

Traditional methods for defining metrics focus on administrative control using insights that are considered discrete ad-hoc activities. But now, metrics can be designed to drive continuous improvement cycles, creating an increasingly autonomous feedback loop as reporting speeds up. This requires a shift in the mindset of managers accustomed to weekly management meetings where colleagues brief on performance.

Further complicating matters is the interdependencies created by the fragmentation, requiring performance measurements across silos. For example, in the pre-digital world, retail marketers could measure the performance of TV advertising separately from the sales performance of pricing and promotions. Currently, advertising on Google (determined by marketing algorithms) drives traffic directly to products and interacts with pricing and promotional decisions (determined by sales algorithms). Measuring the performance of each algorithm separately can be misleading. For performance management algorithms, metrics have three important adaptations:

From top-down to bottom-up metrics

Traditional: Enterprise purpose cascades into functional silos.
New approach: The low-level metrics needed to evaluate your algorithm should roll up to your company’s KPIs.

From results to input metrics

Traditional: Outcome-centric metrics that focus on aggregation and averages.

New approach: An input-centric metric that focuses on distribution and deaveraging. In the pre-digital world, the challenge was to manage ‘average’ and deal with outliers. Atomization now allows companies to take advantage of heterogeneity and take advantage of outliers.

From reports to actionable metrics

Traditional: The concept of reporting with static reports that require human review and interpretation.

new approach: action-oriented. We recognize that reducing latency from decision to insight creates opportunities to semi-automate or fully automate feedback loops.

How to manage algorithms

So what do you do? Below is a six-step approach to managing your algorithm and defining what to measure and monitor. Central to this approach is determining where the waste is by measuring fault conditions. These are important for monitoring performance, prioritizing enhancements, and understanding if actions are improving performance.

1. Define company goals.

Identifying the company’s goals that the algorithm will affect is fundamental. This could be profit, return on investment, or customer lifetime value. For example, one retailer decided that customer lifetime value was the goal of her algorithm in digital marketing.

2. Atomic level identification.

Understand the level at which micro-decisions are currently being made and what is the lowest level at which such decisions can be made. For example, one retailer was bidding on keyword level, but realized that they could bid based on geography, device, time of day, and other customer characteristics.

3. Define Success/Guardrails.

Determine acceptable performance. This is always a judgment of the business her leader and should usually lead to a discussion of risk appetite. This step sets guardrails at the micro-decision level. For example, one retailer decided that he wanted to recoup its advertising costs within six months, so he was willing to invest if he was confident of the returns.

4. Quantify fault conditions.

Understand the price of being wrong. This is an important step in assigning monetary value to performance outside the guardrails. Sometimes this is a simple measure of wasted spending, but sometimes models are needed to estimate lost opportunities. For example, a retailer identified his two main areas of waste. Underspending on keywords that have acquired a large number of profitable customers, and overspending on keywords with paybacks of 6 months or more.

5. Measure performance.

What about total waste and lost opportunities? The next step is to analyze performance at the lowest levels identified in Step 2 to understand the total business value outside the guardrails. A common mistake here is to analyze performance at the same level that you currently operate. This is usually self-fulfilling, hiding both problems and opportunities. For example, a retailer was able to identify $Xm of wasted spend and $Ym of missed customer lifetime value opportunities.

6. Understand the causes of waste.

Finally, analyze performance through the lens of the 4Ps (Objective, Accuracy, Prediction, and Policy) to understand which elements of your algorithm are the biggest contributors to waste and lost opportunities. For example, one retailer found an opportunity to improve each aspect of its bidding algorithm. Accuracy is the single biggest root cause of missed opportunities.

This is the HR of the algorithm. Understanding how to measure and manage algorithm performance can be the difference between success and failure.