Web Performance, Part 2: Lab data lies (a little)

Picture this. You run Lighthouse on the checkout page. The score is 98, all green. You ship it. A week later, someone from customer service forwards a message: "Pay just hangs on my phone."

Both things are true. Your test was real. So was the customer. They just measured two different worlds.

This post is the story of finding the gap between them. In the previous post, we learned what to measure. Now we use those metrics on a real mystery. Why did a page that scored 98 hang for a customer? Along the way, we meet the tools and the traps.

Three kinds of data

Every clue in this hunt comes from one of three places. Each place answers a different question. So we start by meeting them.

• Lab data comes from a test you run yourself in a controlled setup. You open Lighthouse or DevTools and load the page.
• Synthetic data is the same kind of test. A robot runs it on a schedule from fixed locations. (web.dev calls both of these "lab data". We split them because you use them in different ways.)
• Field data comes from real visits. It is also called real user monitoring, or RUM.

Here is how they compare:

Our 98 came from the first kind. Lab data is the one you reach for first. It is also the one that lies a little. So why did it miss our problem?

Why lab data lies (a little)

Lab data isn't wrong. It just sees a narrow slice. web.dev's guide to lab and field differences lists what a lab test leaves out:

• One device, one network, one place. Your laptop on office Wi-Fi is not the same as a customer's mid-range phone on a train.
• A cold cache. Lab tests usually load everything from scratch. Real visitors often have your JS bundle cached already. The back/forward cache can also make a return visit much faster.
• No real taps. A lab test loads the page. It doesn't scroll or tap Pay.
• One version of the page. Logged-in content, A/B tests and saved cards change what each customer sees.
• Only the load. Lab CLS covers shifts while the page loads. Field CLS covers the whole visit. For example, an image without a set height can shift the page when the customer scrolls to it.

Now look at the gap that matters for our story. The customer tapped Pay. Lighthouse never did. Lighthouse 10 builds its performance score from five metrics: FCP (10%), Speed Index (10%), LCP (25%), TBT (30%) and CLS (25%). INP is not one of them.

TBT stands for Total Blocking Time. It is a rough stand-in for INP. But web.dev says plainly that it is not a substitute. So a score of 98 can sit next to a slow Pay button. That is our first answer: Lighthouse never looked at the tap. We look at how TBT works in the flame chart section below.

The same web.dev guide gives a simple rule. Use field data to decide what to fix. Use lab data to find out why, and to test the fix before you ship.

So let's check the field. We collect how long the Pay tap takes on real visits. But field data is a pile of numbers, not one number. Our first instinct is to average them. That instinct fails.

Why averages mislead

Say we collect the Pay-tap time (INP) for 20 real visits. In the previous post, we asked why Google uses the 75th percentile and not the average. Here is the answer. Paste this into the DevTools console:

const inpMs = [
  60, 70, 70, 80, 80, 90, 90, 100, 100, 110, 110, 120, 130, 140, 150, 170, 190, 620, 780, 910,
];
 
const average = (xs) => xs.reduce((sum, x) => sum + x, 0) / xs.length;
 
// Nearest-rank percentile: sort, then take the value at rank ceil(p/100 × n).
const percentile = (xs, p) => [...xs].sort((a, b) => a - b)[Math.ceil((p / 100) * xs.length) - 1];
 
console.log('average:', Math.round(average(inpMs)), 'ms'); // 209 ms
console.log('p50:', percentile(inpMs, 50), 'ms'); // 110 ms
console.log('p75:', percentile(inpMs, 75), 'ms'); // 150 ms
console.log('p95:', percentile(inpMs, 95), 'ms'); // 780 ms

Seventeen taps are quick. The other three take between 620 and 910 ms. Those three are our hanging Pay button.

The average is 209 ms. That number describes none of these taps. It sits just above the 200 ms target, so it looks like a small problem. It hides a big one.

A percentile gives a better picture. Sort the taps from fastest to slowest. The p75 is the point where 75% of taps are as fast or faster. Here it is 150 ms, which passes the 200 ms target. The p95 is 780 ms. Anything over 500 ms is poor. So one tap in twenty is miserable.

Even p75 has a blind spot. It ignores the slowest 25% of taps. Three of those taps took more than 600 ms. So look at the whole spread of numbers, and check p95 too.

CrUX gives you that spread. It shows the share of visits that are good, need improvement and are poor. We meet CrUX below.

Now we know what the numbers say. Next, we need to find where the delay comes from. Time to hunt.

The tools

We have two suspects. The first is the network. Maybe Pay waits on a slow server. The second is the main thread. Maybe our own JS blocks the tap. Here is our toolbox:

We start where the story started: with Lighthouse.

Lighthouse and its throttling

You can run Lighthouse from DevTools, from PageSpeed Insights, or from the command line:

npm install -g lighthouse
lighthouse https://example.com/checkout

By default, Lighthouse uses simulated throttling. It loads the page once. Then it models a slower device and network. The mobile defaults are 150 ms of round-trip latency, 1.6 Mbps down, 750 Kbps up and a 4x CPU slowdown.

Some customers are slower than that. Test for your slowest common customer, not for your laptop:

lighthouse https://example.com/checkout --throttling.cpuSlowdownMultiplier=6

One more warning. Scores change from run to run. Ads, A/B tests, browser extensions and network routing all play a part. Run Lighthouse a few times and treat the score as a range.

Lighthouse gave us a 98 and no clues. DevTools shows us where to look.

The DevTools Performance panel

Open the Performance panel. Before you record anything, it already shows live LCP, CLS and INP for the page. Click around the page to capture INP too.

Then record a trace:

1. Open Capture settings. Set CPU to 4x slowdown. Use 20x to mimic a low-end phone.
2. Set the Network drop-down to something like Fast 4G.
3. Click Record. Reload the page or tap Pay. Then click Stop.

You can also compare your local numbers with real users. Go to Next steps, then Field data, then Set up. The panel then shows CrUX data next to your own numbers.

Now we can question our suspects. We start with the network.

Reading a waterfall

Suspect one: the network. Maybe the Pay tap waits for a slow server. The Network panel can tell us:

1. Tick Disable cache and pick a throttling preset.
2. Reload the page.
3. Read each row as one request. The bar's left edge is when the request started. The bar's length is how long it took.
4. Click a request and open the Timing tab. It splits the time into waiting for the server and downloading.

Look for three patterns. A long wait before the first byte means a slow server. A long download means a big file. A staircase of requests means a request chain. Each request waits for the one before it. We come back to all three later in the series.

The picture shows what each problem looks like. In our story, the checkout page shows none of them. The network is clean. So the delay must be somewhere else.

Reading a flame chart

Suspect two: the main thread. Maybe our own JS blocks the tap. For slow JS, use the Main section of the Performance panel. The x-axis is time. Each bar is an event, and a wider bar took longer. A bar stacked below another bar was called by it.

1. Find the gray bars labeled Task.
2. Look for a red corner on a task. It marks a long task, which runs for more than 50 ms.
3. Hover over a task to see how long it ran.
4. Click a task, open Bottom-Up, and choose Group by Activity. This shows which activity used the most time.

There is our culprit. In this recording, the tap on Pay lands on a task that lasts 210 ms. Inside it, the click handler runs validateCard and then sendAnalytics. The browser can't paint anything until they finish. So the Pay button looks frozen.

Why 50 ms? The browser can't interrupt a task that is already running. If a customer taps during a long task, the tap has to wait.

Lighthouse measures this as Total Blocking Time (TBT). For each long task, it counts only the time past 50 ms. Then it adds those up. In our example, that is 70 + 10 + 0 + 160 = 240 ms. The tasks run for 430 ms in total. web.dev suggests keeping TBT under 200 ms on average mobile hardware. (Real TBT only counts the time between FCP and Time to Interactive. We simplified the picture.)

TBT can flag a problem that no customer ever hits. It can also miss a problem that only a real tap causes. And one recording only proves that the problem can happen. It doesn't tell us how many customers feel it. Real taps happen in the field, so let's go there.

Field data: CrUX and PageSpeed Insights

We start with public data. CrUX is the Chrome User Experience Report. It is Google's public dataset of real Chrome visits. The easiest way to read it is PageSpeed Insights.

Paste your URL into PageSpeed Insights and read the field data first. It shows the 75th percentile of LCP, INP and CLS over the previous 28 days. Each metric is rated good, needs improvement or poor. The Lighthouse run comes below.

CrUX has limits worth knowing:

• Chrome users only, and only some of them. CrUX counts opted-in users on desktop Chrome and Android Chrome. Chrome on iOS, Android WebView apps and other Chromium browsers such as Edge don't contribute.
• Enough traffic. A page needs enough visits to appear. If it has too few, PageSpeed Insights shows data for the whole site (the origin) instead. If the origin has too few visits too, you get no field data.
• A slow window. The data covers 28 days. A fix shows up gradually.

CrUX tells us how many customers are hurt. It can't tell us which tap was slow, or why. For that, we collect our own data.

Your own RUM

CrUX is not the only way to get field data. You can collect it yourself. Several vendors sell RUM. Examples are SpeedCurve, Datadog, New Relic and Sentry. Or you can build a basic version with web-vitals and one endpoint.

On the page, send each metric when it is ready:

import { onCLS, onINP, onLCP } from 'web-vitals';
 
function sendToAnalytics({ name, value, rating, id }) {
  const body = JSON.stringify({ name, value, rating, id, page: location.pathname });
  // sendBeacon still works while the page is being closed or backgrounded.
  navigator.sendBeacon('/analytics', body);
}
 
onCLS(sendToAnalytics);
onINP(sendToAnalytics);
onLCP(sendToAnalytics);

On the server, accept the POST and store it. This Node fragment keeps samples in memory. In production, use a real database. Also serve the endpoint from the same origin to skip CORS:

import express from 'express';
 
const app = express();
app.use(express.text({ type: '*/*' })); // sendBeacon sends strings as text/plain
 
const samples = [];
 
app.post('/analytics', (req, res) => {
  samples.push(JSON.parse(req.body));
  res.sendStatus(204);
});
 
app.listen(3001);

A metric can fire more than once per page load. Use id to remove duplicates on the server. Then run the same percentile function from earlier on samples. That gives you your own p75.

Want to know why a tap was slow? Swap in the attribution build. It splits INP into the same three phases we saw in the previous post:

import { onINP } from 'web-vitals/attribution';
 
onINP(({ value, attribution }) => {
  console.log('INP', Math.round(value), 'ms on', attribution.interactionTarget);
  console.log('input delay', Math.round(attribution.inputDelay), 'ms');
  console.log('processing', Math.round(attribution.processingDuration), 'ms');
  console.log('presentation delay', Math.round(attribution.presentationDelay), 'ms');
});

In our story, the processing time would be the big part. That points straight at our own handlers: validateCard and sendAnalytics. The attribution build is about 1.5 KB larger when compressed. Only use it if you act on the data.

Setting goals

We found the culprit, and we can watch it. Now the big question is what "fixed" looks like. Don't copy numbers from a blog post, this one included. Work them out in three steps.

1. Start from your real users. Which devices, networks and regions do they use? Check your analytics. Also check the phone-versus-desktop split in PageSpeed Insights. Then set your lab profile to match. Say most visitors use mid-range Android phones on 4G. Then test with a throttled CPU and a 4G network, not on your laptop.
2. Compare with competitors. CrUX data is public. Paste a competitor's URL into PageSpeed Insights. If they have enough traffic, you can see their LCP, INP and CLS at p75.
3. Fix the worst metric first. Find the metric that is furthest from "good". Check p75, then check p95 to see how bad the tail is. Start there. One metric fixed well beats five metrics half fixed.

Here is how that looks for our checkout page. The numbers are made up:

At p75, every metric passes. That is why the headline numbers looked fine. The p95 column tells the real story. Only INP lands in the poor range, which starts above 500 ms. So the Pay tap goes first.

A performance budget for checkout

Say we fix the Pay tap. A month later, someone adds a new tracking script. The tap hangs again. Goals drift unless something guards them.

A performance budget is a set of limits on the things that affect speed. When someone crosses a limit, the build tells them. This catches a new tracking script before your customers feel it.

Here is a budget for the checkout page. The numbers are an example. Start from your own users and competitors. Then tighten the limits over time.

The 170 KB figure comes from web.dev's budgets guide. It is a default starting point for critical-path resources that are compressed and minified.

Lighthouse can enforce the lab part. Save this as budget.json:

[
  {
    "path": "/*",
    "timings": [{ "metric": "first-contentful-paint", "budget": 1800 }],
    "resourceSizes": [
      { "resourceType": "script", "budget": 170 },
      { "resourceType": "total", "budget": 600 }
    ],
    "resourceCounts": [{ "resourceType": "third-party", "budget": 8 }]
  }
]

Then pass it to the CLI:

lighthouse https://example.com/checkout --budget-path=./budget.json

The report gets a Budgets section in the Performance category. Sizes are in KB and timings are in ms. Timing budgets need Lighthouse 6 or later. To run budgets in CI, look at Lighthouse CI.

The three Core Web Vitals need real users. Set alerts for those in your RUM instead. Lab budgets catch a regression before you ship. Field numbers confirm that customers feel the difference.

The story so far

Let's look back at the trail. We started with a 98 and a hanging Pay button. Lighthouse missed the problem because it never taps. The average hid it. DevTools led us to the culprit: a click handler that blocks the main thread. CrUX and our own RUM show how many customers feel it. A budget keeps it from coming back.

But we haven't fixed anything yet. That comes next.

Key takeaways

• Lab data is for debugging. Field data is for deciding. A lab test can't see real devices, networks or taps.
• Lighthouse's performance score doesn't include INP. Measure INP in the field.
• Averages hide the tail. Use p75 for pass or fail. Check p95 for pain. Look at the whole spread of numbers.
• Use PageSpeed Insights and CrUX for field data (Chrome only, 28 days). Use web-vitals for your own RUM.
• Set goals from your real users and competitors. Fix the worst metric first. Guard it with a budget.

Next in the series

In the next post, we start fixing things from the top of the page load. We begin with TTFB and FCP. We look at why the first byte is slow. We also look at how to get the first paint on screen sooner. The slow Pay tap comes later in the series.

Further reading

• Lab and field data differences on web.dev
• Chrome UX Report (CrUX) and its methodology
• About PageSpeed Insights
• Lighthouse performance scoring and throttling
• Total Blocking Time and Find long tasks in DevTools
• Performance budgets 101 and Use Lighthouse for performance budgets
• The web-vitals library
• Web Performance Fundamentals, v2 by Todd Gardner, at master.dev