Testing how the (third party) cookie will crumble: lessons from the CMA’s quantitative testing guidance on Google's Privacy Sandbox

Back in 2021, the UK’s Competition and Markets Authority (CMA) opened an abuse of dominance investigation into Google’s plans to phase out third-party cookies (TPCs) following a complaint from a group of publishers. The complaint alleged that Google’s plans would have a devastating impact on publishers' ability to generate online advertising revenues (and in turn benefit Google’s own ad business). In 2022, Google agreed a set of commitments with the CMA to have the investigation shut down, which as we have written about previously gave CMA a seat at the table in Google’s plans to develop alternative technologies to TPCs – known as the “Privacy Sandbox”.

In accepting commitments, the CMA said it would keep a “close eye” on Google. Crucially, the commitments require Google to get the CMA’s approval before deprecating TPCs. In an important step in the case, last month the CMA published a guidance note addressed to ad techs, publishers, and advertisers, providing advice on how they can test the Google Privacy Sandbox tools in a way that would contribute to the CMA's assessment of whether Google has lived up to the promises it made in the commitments. Google’s current plan is to begin to deprecate TPCs from Q3 2024 and the testing phase is a critical hurdle for Google to overcome to be able to deliver on this plan.

This is the first time the CMA has published guidance on technical testing on this scale, but it will not be the last. As we explain below, the “experimentation” phase of the Sandbox saga will doubtless teach the CMA important lessons that will put into practice when it gets its new powers to administer forthcoming Strategic Market Status (SMS) regime.  And the trend of greater scrutiny of algorithms is not limited to the UK or to competition authorities, but is something that is fast becoming the new normal for digital firms. 

Crunch-time in the Sandbox saga

The Privacy Sandbox Commitments established a set of “development and implementation” criteria that Google was required to abide by in developing its plans to phase out TPCs. The criteria relate to the impacts of Google’s proposals on publishers, advertisers and focus primarily on the effectiveness of the targeting and measurement APIs proposed by Google in replicating the functionality currently supported by TPCs. At the core of the commitments is the CMA’s right to assess the effectiveness of the Privacy Sandbox changes (using a range of quantitative and qualitative evidence) before Google deprecates TPCs.

The guidance note explains how the CMA will do this. The note was developed in consultation with a range of industry participants on what kinds of “experiments” would be most effective in assessing whether Google’s proposals meet the criteria. Rather than proposing a single industry wide test, in recognition that many organisations have already invested heavily in preparing for the deprecation of TPCs, the CMA provides guidance on how businesses can run their own tests in a way that will inform the CMA’s assessment of whether Google has delivered on its promises to ensure its deprecation of TPCs does not have a negative impact on competition.

The guidance note also describes how the CMA's proposed testing approaches fit with the support Google intends to provide for testing, including implications for timing. Google has announced plans to implement two testing modes in Chrome, whereby market participants will be able to identify whether ad requests are in “treatment” or in “control” groups through Google’s experimental labels.

Finally, the guidance note outlines how market participants can report test results to the CMA in a meaningful way. It provides questions market participants should answer when submitting test results (e.g., which inventory, formats, geography, and campaign types did the testing cover, how comprehensive were any mitigations used and how much these were relied upon).

The CMA explains it will be most interested in results showing how the Privacy Sandbox might affect key metrics relevant to publishers, advertisers and competition, including revenues per impression, clicks and conversions per dollar / impression, web page latency, unique viewers and Average time spent or video completion rates.

Testing the testing system

As we have explained previously, the Privacy Sandbox case offers important lessons for the future of tech regulation in the UK and beyond and demonstrates a new normal in which the largest tech companies should expect to work hand in glove with regulators as technologies are developed. This is especially true of the testing phase, because under the SMS regime, CMA-supervised testing is likely to become common.

Under the SMS regime the CMA will have powers to force SMS firms to demonstrate compliance with the regime through running “specified demonstrations or tests” and the CMA is given the power to enter business premises to observe demonstrations or tests. It is envisaged this power could be used to require an SMS firm to “demonstrate a technical process with examples, such as how an algorithm operates, or to undertake testing or field trials of its algorithms and report the outcomes”. As part of this, the CMA can specify relevant input data, parameters and other aspects of the test or demonstration.

The CMA has invested heavily in its Digital Markets Unit (DMU), which will be given powers to administer the SMS regime. The DMU is building a team of specialist lawyers, economists but also crucially, computer scientists and other technical experts, with the objective of ensuring it can fully understand the technical aspects of how digital markets operate, and the implications for competition and contestability in these markets. 

Opening up the black box

The push to upskill on the technical front is not unique to the CMA. Last September the UK's Digital Regulation Cooperation Forum published a discussion paper on auditing algorithms, which highlighted the potential for regulators to run tests of algorithms to understand a system’s outputs or assess the design and technical nature of a system. The DRCF has said it is doing further work to understand regulators’ roles in the field of algorithmic processing.

Opening up algorithms that were formerly seen as something of a "black box" is a common theme in the new wave of digital regulation. For example, the EU Digital Services Act introduces due diligence and transparency obligations regarding algorithmic decision-making and provides for auditing of algorithms. The UK's proposed online harms regime also provides for scrutiny of algorithms. 

While it remains to be seen precisely how these various obligations will be administered and enforced, it is beyond doubt that the days of algorithms operating as a "black box" from a regulatory perspective are well and truly behind us. Digital firms need to be ready to not just explain, but demonstrate, compliance.