In this month’s edition of Cherry Picked, we cover all things incrementality measurement (and doing it properly):

💭 The state of incrementality, an interview with Mike Feldman from VaynerMedia

🌎 The power of geo-based tests to provide marketers with more transparency and control

🫣 Methodologies to unblack-box incrementality testing

Let’s dive in 👇

Incrementality has become a bit of a buzzword—used liberally and often misunderstood. And while many claim to measure it, few adhere to the data-science-backed methods necessary to actually prove it (as seen here and here). It is not attribution or new-to-brand customers but a controlled experiment comparing a holdout group (users who do not see an ad) with an experimental group (users who do see an ad). This eliminates the guesswork and proves whether ad exposure can drive measurable increases in metrics like conversions and revenue.

So why aren’t more people properly measuring incrementality?

According to Anthony Kilili, a Data Science Leader at Kroger Precision Marketing, “Retailers need expert media teams and sophisticated tools capable of rapidly and repeatedly executing randomized controlled trials and other testing methodologies.” The TL;DR? Companies likely don’t have the tools and teams to execute incrementality correctly. And it is likely why they look to third-party lift studies like Kantar or Neilsen that only provide high-level directional impact analysis and not outcome-based metrics.

Now, there are several ways to execute a data-science-backed incrementality test, like randomized control groups, geo-based testing, and intent-to-treat/ghost ads, but when it comes to measuring incrementality for large-scale advertising campaigns, especially on the Amazon DSP and with Streaming TV ads, you have to use a geo-based test. This is because its scale, ad delivery method, data accessibility, and reliability ensure clean experimental conditions within the Amazon DSP.

Why a geo-based test over other popular tests like randomized control groups?

When users are randomly split, ad spillover can occur because users in the holdout group might still see the ads in shared environments (households, public spaces, etc), contaminating the holdout group and reducing the accuracy of the results. With a geo holdout, the risk of ad spillover and contamination is minimized because ads are either shown or not shown to entire regions. This reduces the likelihood of users in the holdout group seeing the ad, ensuring a more accurate measurement of the campaign's incremental impact.

Additionally, a randomized control group test could assign users across regions, leading to geographic imbalances where market dynamics (e.g., competitive conditions, economic factors, seasonality) might vary significantly, making the results harder to generalize to the broader population. In using a geo-based test, marketers can measure incrementality at a market-wide level, capturing the broader impact of the campaign.

Why geo-based incrementality tests are so effective within the Amazon DSP:

For marketers that demand greater transparency and control, this level of customization offers the ability to intentionally pull select levers within the test. For example, a seasonal winter brand would likely want to ensure colder and warmer regions are evenly distributed amongst the two test groups to avoid skewing the results of the campaign toward a specific seasonality. This would’ve otherwise been impossible to control in a randomized test, or in a test where parameters are a black box.

Considerations for control—tradeoffs when curating your DMA groupings:

Incrementality testing can be scary. Not all brands are necessarily willing to risk 50% of their audience not seeing their ads throughout the testing period. And while DMA curation allows marketers to adjust their holdout/experimental group breakdown, there will be tradeoffs. Brands that don’t want to do an even 50/50 split and potentially want to adjust their experimental group to 60 or even 70 percent of the test, could risk encountering statistical insignificance as there isn’t enough data to compare within the holdout group. Alternatively, some brands may be more bullish to prove the effectiveness of their ads on driving incremental revenue, and could skew their holdout group to 60-70%. This would likely provide greater statistical significance and better help assess how ads are potentially cannibalizing sales.

At Gigi, we’re addressing this demand for transparent and controllable incrementality testing with our geo-based tests built specifically for Amazon advertisers. No more black box measurement and no more guesswork. Once DMA grouping parameters are confirmed, Gigi’s proprietary DMA curation engine builds holdout and experimental groups, while our DMA exclusion ensures all agreed-upon DMAs are removed from all STV campaigns. This is built directly into the Amazon DSP and is automated when building audience segmentation at a line item level—helping eliminate manual effort and time. Plus, via data collaboration, marketers can use their 1P data for their DMA creation and measurement models, allowing them to measure the incremental omnichannel impact of their STV ads with robust reporting on iROAS, lift in revenue, incremental lift percentages, and detailed statistical analysis.

Ready to see the real impact of your STV campaigns? Learn more 👉 gigico.tv/incrementality

Adam recently sat down with Mike Feldman, SVP Global Head of Retail Media at VaynerMedia, to get his take on the state of incrementality. Mike dives into how marketers should approach incrementality to drive business outcomes, from education gaps to using Amazon Marketing Cloud to bridge the gaps across retail media networks’ fragmented measurement methodologies and reporting. Let’s dive in.

The education gap in retail media remains a significant challenge, extending beyond just a gap in knowledge around incrementality measurement. One of the fundamental issues is that retail media budgets are often sourced from multiple teams, such as trade, sales, e-commerce, shopper marketing, and brand teams. Each of these groups has different objectives, KPIs, and levels of understanding, making alignment difficult. Because retail media is relatively new compared to traditional advertising channels, many brands lack a foundational understanding of how it works and how to optimize it effectively.

When it comes to incrementality measurement, the complexity increases. Incrementality is best measured by comparing control and exposed groups and calculating the difference in revenue generated. However, the lack of standardization across Retail Media Networks (RMNs) makes this process difficult. Different RMNs use varying lookback windows, control methodologies, and data extrapolation techniques, leading to inconsistent and sometimes unreliable insights.

For example, Walmart’s cash extrapolation method has long been a point of contention, as even internal teams at Walmart have questioned its validity. Other RMNs rely on loyalty card partnerships or alternative tracking mechanisms, which further complicates measurement. The result is a fragmented system where brands struggle to compare performance across different platforms, making it challenging to determine the true incremental impact of their media spend.

Given these challenges, brands need better education, alignment, and methodology standardization to ensure retail media dollars are being spent efficiently. One promising approach is geo-based DMA holdout testing, where advertising is deliberately excluded from certain geographic areas to create a natural control group. By comparing purchase behaviors in these excluded regions to those in exposed regions, brands can gain a clearer understanding of true incremental impact without relying on inconsistent RMN methodologies.

Amazon Marketing Cloud (AMC) stands out as a powerful solution for incrementality measurement due to its high-fidelity first-party data collaboration capabilities and comprehensive path-to-conversion insights. Unlike other Retail Media Networks (RMNs), AMC offers granular audience tracking and a more standardized methodology for understanding consumer behavior across multiple ad touchpoints. By leveraging AMC, brands can analyze incrementality through control and exposed groups, gaining insights into how various advertising strategies drive actual customer conversions.

Ultimately, retail media success depends on brands breaking out of traditional silos and integrating their advertising strategies across channels. Many brands still optimize search, display, and video separately, missing out on synergies that could drive higher performance. By adopting a full-funnel and omnichannel approach, leveraging first-party data, and implementing incrementality testing, brands can maximize the effectiveness of their retail media investments while minimizing wasted spend.

Want to learn more? Register for our upcoming webinar “The State of Incrementality” April 2nd at 1pm ET 👉 https://bit.ly/incrementality-webinar

We keep talking about the importance of unblack-boxing incrementality testing…but what does that actually mean?

For years, marketers have been forced to trust opaque third-party lift studies that provide little to no visibility into their methodologies. These providers dictate the test design, control group selection, and data analysis—without allowing brands to see under the hood. The result? Ambiguous, unverifiable results that can’t be properly optimized against.

Marketers should demand more control and transparency over how their incrementality tests are run. At Gigi, we’ve built a geo-based incrementality testing framework that provides full visibility into how tests are designed, executed, and analyzed. Here’s how we do it.

One of the biggest issues with traditional incrementality testing is the lack of transparency in how control groups are formed. Many third-party studies fail to define who falls into the control vs. experimental group, leading to biased results and inflated performance metrics.

At Gigi, we solve this with our DMA Creation Engine—a fully automated system that curates control and test groups based on a brand’s specific risk profile and parameters. Marketers can define how their DMA groupings are built based on a number of aspects from household inclusions to total sales inclusions across both Amazon and 1P channels (via data collaboration), adjusting the split of holdout vs experimental based on how reserved or bullish they intend to be throughout the test.

Once an incrementality test is live, analyzing results correctly is critical. At Gigi, we apply Difference-in-Differences (DiD) modeling—a rigorous statistical approach used in causal inference studies. With DiD modeling, we can ensure control and experimental groups start at similar sales levels, so the only variable is ad exposure. This also filters out external factors like seasonality and market trends to isolate the true incremental impact of ads. And confidence intervals and p-values help ensure results are statistically significant, not just a random variance. With a look at the statistical analysis of their incrementality testing, marketers can get a clearer picture of how the test performed.

To validate media investments, Gigi prioritizes several outputs for brands, including incremental iROAS, lift on revenue, incremental lift percentages and a breakdown of sales by group and channel. In doing this, brands can understand how their STV ads drive incremental omnichannel performance, and have actionable insights to scale their campaigns. For example, a brand running a multi-channel campaign could discover that 70% of its revenue lift came from Amazon, while only 30% came from their DTC site—helping them shift budgets accordingly.

“The new Omnicom would achieve unprecedented buying clout that has never been reached”.

Variety covered the Omnicom and IPG merger leading with a Mad Men —> Math Men data and technology narrative, but most of the article illustrates that increased scale presents greater leverage for Omnicom to wield for principal-based media buying with ad sellers and publishers.

Mark Reed, CEO of WPP (an Omnicom competitor), had previously spoken on principal-based media buying, saying “you can’t find it in financials,” and referred to it as “black box media models that are difficult to understand and monitor.”

It’s interesting because in WPP’s most recent earnings call, GroupM’s new CEO, Brian Lesser, said that principal-based media buying would be a focus for GroupM. He noted that they would be “introducing new proprietary trading models and next generation media products,” which would allow GroupM “to offer more performance to our clients at efficient prices using our expertise, scale, and data capabilities to redefine industry standards.”

A few thoughts on this. While principal-based media buying may seem shady, less transparent, and possibly antiquated, it does create a win-win scenario for all parties involved. Publishers receive guaranteed demand, holdcos have a high-margin business line, and brands benefit from lower CPMS across the board. If there are no real losers, then what’s the issue?

Finally, Digiday highlighted the legal ramifications of principal-based media buying and that holding companies will create separate legal entities for the purposes of executing these buys. “If the agency acts as principal and there are no clauses on transparency, kickbacks, billing differences, etc., the agency is legally entitled to those benefits/differences.”

ESPN (and Disney) ended its 35-year relationship with the MLB. The WSJ reports ESPN lowballed MLB with an offer that was less than half its current deal. Now, MLB Commissioner Rob Manfred is scrambling and trying to convince teams to “cede control of their local rights to the league office so that MLB can sell them collectively as a unified streaming package.” This will be a tough sell for big market teams like the Dodgers, who signed a 25-year, $8.35 billion local TV deal with Time Warner Cable in 2013. Regardless, national streaming rights for a major sports league are now up for grabs to the highest bidder. We expect Amazon to make a splash here — they have done so for every other major sports opportunity over the past few years - and undoubtedly want to become the clear-cut leader in live sports streaming.

Three years after Amazon paid $8.5m for MGM studios, it allegedly just paid an additional $1b for full control of the rights of the James Bond franchise. They do this after three years of tenuous negotiations with the previous rights holder, the Broccoli family. Barbara Broccoli, who previously owned all creative control, just months ago referred to Amazon as “temporary people making long-term decisions” and stated that Amazon was not a “good fit” for Bond. Puck had a somber take on the news: “A tech retailer that flexes its power by snapping up a troubled movie studio to lure shoppers—but then alienates its top creative partner to the point where she throws up her hands and sells, thus giving the tech retailer more power. As a result, the most enduring action hero in movie history has now been swallowed by the algorithm. And the franchise with Bezos-like villains is now dictated by Bezos himself.” So what’s next? Most predict Amazon will give James Bond the “Marvel” treatment: monetize all the allure out of Bond. We just hope they sign up Christopher Nolan for the next 3 feature films, as rumoured here.

Thanks for reading Cherry Picked, and we’ll see you next month!