Informing HTTP Extension Design with Data
Friday, 7 August 2026
A while back, I refactored REDbot to separate out a HTTP message linting library. httplint checks about 325 aspects of HTTP messages, covering the syntax and semantics of almost 200 HTTP header fields.
Then, I got curious: what happens when you point it at Common Crawl? Its availability on AWS brings the opportunity to do a large-scale survey of how HTTP is used and abused. With the help of Claude Code, the result is cc-lint.
Pointed at over 120 million responses from nearly 50,000 sites, the resulting report is a lot to digest. For me, the most interesting aspects are reflections of how we design new protocol elements – and often, how they come up short. Collectively, what I observe below is in some ways very obvious, but also we keep on forgetting it when we design new protocol elements – i.e., “hope-based standardisation.”
So, this post serves as a reminder or wake-up call for protocol designers (depending on your perspective). It covers the tool and those observations; followups are likely to dive into specifics.
This is based upon the lightning talk I gave at the 2026 HTTP Workshop.
Background and Caveats
First, the caveats.
Common Crawl is not the whole Web; they are blocked from crawling some sites, for example; others sit behind paywalls, WAFs or geofences. Furthermore, cc-lint focuses on the Tranco top 100,000 sites, so that behaviour of very small, idiosyncratic sites (especially spam sites) doesn’t skew the results.
The corpus is overwhelmingly 200 (OK) responses to GET requests: almost no redirects, errors, conditional or range requests. And it’s response metadata only — headers, no bodies, and nothing from the request side.
Put together, what you’re looking at is the successful responses of popular sites, seen from one side. A lot of the most interesting HTTP — error handling, negotiation, cache validation — occurs in the responses that aren’t here. But it’s still over 120 million real responses, which turns out to be plenty to see some patterns very clearly.
1. Set It and Forget It
It’s very noticeable that a common deployment pattern for a successful protocol extension (like a new header field) is “fire and forget” – someone decides they need a particular behaviour, looks up the relevant protocol element, adds it to their configuration, and never revisits that decision.
This means that adoption often depends more on how much effort is involved than how important it is to get things right.
Headers that can be copied from a HOWTO and then forgotten have great deployment; HSTS is on 51% of sites, X-Frame-Options on 44%, and even X-XSS-Protection – which browsers stopped paying attention to seven years ago – is still on 29% of sites.
A header that requires crafting a site-specific policy after careful consideration fares less well; Content-Security-Policy is on 32% of sites, but there are major caveats on that which I’ll cover in a separate post. Permissions-Policy is only present on 9% of sites; Cross-Origin-Opener-Policy on 4% and Cross-Origin-Embedder-Policy on less than 1%.
So, if you want your protocol extension to be adopted and deployed – and even zombie on long after its useful life has ended – keep it simple and repeatable.
2. Make It Break
Another pattern: an extension gets deployed correctly in proportion to how loudly it fails when you get it wrong. The visible failure is its own feedback loop: operators fix what they’d otherwise never notice.
The clearest evidence is a natural experiment that the browsers have unknowingly run: the *-Report-Only headers. A policy header and its -Report-Only twin share identical syntax and are set by the same kinds of operators — but the plain header changes browser behaviour (so a mistake is visible), while the Report-Only version only sends a report somewhere (so a mistake usually isn’t). Turn enforcement off and the error rate jumps:
- Cross-Origin-Opener-Policy is malformed in 3.0% of responses; its Report-Only variant, 12.7% — about four times worse.
- Cross-Origin-Embedder-Policy climbs from 13.1% to 20.6% the same way.
The syntax is the same; only the consequence of getting it wrong differs.
So, if you want your header to be emitted correctly, make the breakage visible in some fashion.
3. Migrating to Standards isn’t Automatic
There are a number of examples where we’ve attempted to “pave the cowpaths” – create standard headers that consolidate well-understood proprietary extensions, to promote interoperability and easy switching between providers.
For example, Cache-Status is an attempt to consolidate a galaxy of proprietary cache reporting headers - X-Cache, CF-Cache-Status, X-Cache-Hits and many more. It was standardised with feedback and requirements of many cache implementers, but implementation has lagged: while there are tens of millions of instances of proprietary cache status headers in the crawl, only about 1.1 million instances of Cache-Status were seen.
Proxy-Status – the complementary header for intermediary status reporting – fares worse, with only about 7,900 instances seen, compared to tens of millions of the various proprietary intermediary status headers (e.g., x-served-by, x-timer, akamai-grn…)
Of course, adoption of standards takes time, so there’s still hope that we’ll see eventual convergence for these. However, a notable contrast is seen when there’s a forcing function. For example, Feature-Policy was fairly recently replaced by Permissions-Policy, and the transition between those two mechanisms is 87% complete in the crawl. That indicates that interoperability pressure – in this case, from browsers – matters.
4. Considering the Backlog
All of this doesn’t mean we should stop trying to extend and evolve HTTP; just that we need to be more thoughtful when we do so. Proposals that leverage these trends have a better chance of success. But what should we focus on?
To me, the ugliest part of HTTP that we haven’t yet improved is definitely the Vary header field. 26% of responses with Vary list two or more dimensions of variance – i.e., it’s not just Accept-Encoding any more. Most interestingly, a sizeable portion of sites – almost 3,000 of them – are Varying on four or more axes. Some go as high as 10, 23, or even 47 axes.
What in the world are they doing? It turns out that when the Vary numbers get higher, sites are almost invariably (yes, I went there) injecting synthetic request headers in an intermediary (reverse proxy or CDN) and then using Vary to partition the cache in a way that promotes better hit rates.
For example, their intermediary might inject CDN-User-Geo and then the site will send back Vary: CDN-User-Geo, so that they can reliably serve different content to different regions. Multiply that by up to 47 dimensions and you can get a very performant, capable system for serving personalised content.
But Vary is kind of a horrible, kludgy mechanism for doing this, since intermediaries have pretty abysmal interoperability across it – understandably, since its specification is so difficult to implement both correctly and performantly. We can do better.
Previously, I’ve proposed Availability Hints as a way to address Varys shortcomings. This data makes me think that the older Key proposal might be closer to the mark for these uses.