How to run a matched market (geo) incrementality test in Canada

A matched market test answers the question every marketing budget eventually runs into: if we stopped (or doubled) this spend, how many sales would we actually lose (or gain)? It does that without cookies, device IDs or click paths, by comparing regions that get a change with similar regions that don't. Most published guides assume the US, with its 210 media markets. Canada is different, and the differences change how you design the test.

This guide walks through a practical geo test for a Canadian advertiser, from the business question to the readout. It is written for marketers and analysts at growth-stage and mid-sized companies, not data science teams, though we point to the open-source tools a data scientist would use.

What a geo incrementality test is (and isn't)

In a geo experiment you split a country into non-overlapping regions, change advertising in some of them (the test or treatment markets), leave the others alone (the control markets), and compare sales. Because the comparison happens at the level of regions, you only need aggregated sales by region and day or week. You never need to track an individual.

Google researchers formalised this approach for search advertising in a 2011 paper, Measuring Ad Effectiveness Using Geo Experiments (opens in a new tab), which estimates incremental return on ad spend from the difference between test and control regions before and during the test. Since then, two open-source tools have made the method much more accessible:

• GeoLift (opens in a new tab), an R package from Meta that uses synthetic control methods to build a weighted "virtual" control from many regions, and includes power analysis and market selection.
• Trimmed Match (opens in a new tab), a Python library from Google designed for randomized paired geo experiments, which trims poorly matched pairs so that a few unusual regions don't dominate the result (Chen, Longfils & Remy (opens in a new tab)).

A geo test is not the same as a platform "conversion lift" study, which randomizes users inside one ad platform. Lift studies are useful, but they only measure that platform, and the platform grades its own homework. A geo test measures your real sales in your own systems, and works for any channel you can switch on and off by region: search, social, YouTube, connected TV, radio, out-of-home, even direct mail.

Why Canada needs a different design

Three features of the Canadian market matter more than anything else.

1. A few very large metros. According to Statistics Canada's 2021 Census, the Toronto census metropolitan area had about 6.2 million people, Montréal about 4.3 million and Vancouver about 2.6 million, followed by Ottawa–Gatineau, Calgary and Edmonton at roughly 1.4 to 1.5 million each (Statistics Canada (opens in a new tab)). For many brands, the top three metros produce a large share of sales. Put Toronto in the test group and it can swamp everything; put it in control and you need something big enough to balance it.

2. Language and culture. Quebec behaves differently from the rest of Canada: different language, media habits, competitors, holidays and often a separate creative strategy. Matching a Quebec market to an Ontario market on historical sales may look fine statistically and still break during the test because of a Quebec-only promotion or media event. Many tests are cleaner if Quebec is handled as its own stratum or excluded.

3. Fewer, uneven regions. There are 10 provinces and 3 territories, and Statistics Canada defines dozens of census metropolitan areas and many smaller census agglomerations (definitions (opens in a new tab)). That's far fewer natural units than the US. The practical answer is to build your own regions.

Step 1: choose your geographic unit

Your unit has to satisfy two constraints at once: you must be able to target ads to it, and you must be able to attribute sales to it.

• Provinces. Easy to target everywhere and easy to map sales to, but too few and too uneven for most tests. Useful for very large national advertisers or for a quick, low-precision read.
• Census metropolitan areas and agglomerations. A good default. They reflect real commuting and media markets, and you can map customer postal codes to them using Statistics Canada geography files.
• Postal-code regions. The first three characters of a Canadian postal code (the forward sortation area, or FSA) give you hundreds of small units you can cluster into custom regions. Both Google Ads and Meta support Canadian location targeting at city, region and postal-code level; check the current options for your markets in Google's location target types (opens in a new tab) and Meta's location targeting tools before you commit to a design.

Whatever unit you choose, map every order or lead to it the same way (usually by shipping or billing postal code), and decide in advance what happens to orders with no usable location.

One more practical point: ad platforms target by the user's likely location, which is never perfect. People commute, travel and use VPNs. Larger units leak less. Clustering FSAs into contiguous regions, rather than a checkerboard of scattered postal codes, keeps spillover manageable.

Step 2: write down the question and the decision

A test without a decision attached tends to produce an interesting chart and no change. Before you touch data, write one paragraph:

• The change: for example, "pause all Meta prospecting" or "increase YouTube spend by 100%".
• The KPI: orders, revenue, new customers, qualified leads. Pick one primary metric. Revenue is noisier than orders; new customers are often the most honest measure for acquisition spend.
• The decision rule: for example, "if iROAS is below 1.5 with reasonable confidence, we cut the channel by half; if it is above 3, we scale."

Holdout tests (switching something off) are usually cheaper and clearer for always-on channels like branded search and retargeting. Scale-up tests (adding spend) are better when you want to know whether the next dollar works, which is what budget decisions actually need. We cover that distinction in iROAS vs ROAS.

Step 3: assemble the historical data

You need a clean panel: one row per region per day (or week), with the KPI and, ideally, ad spend by channel. A year or more of history is ideal because it captures seasonality; a few months can work for very stable businesses. Check for:

• Regions with very low volume, which add noise. Merge them.
• Structural breaks: a store opening, a new retailer, a regional promotion. Exclude affected regions or periods.
• Promotions coming up during the test window. A national sale is fine (it hits test and control alike); a regional one is not.

Step 4: select markets and run a power analysis

This is where the open-source tools earn their keep. Market selection searches for test groups whose historical KPI can be closely reproduced by a combination of control regions. Power analysis then simulates the test on historical data to estimate the smallest effect you could reliably detect for a given test length and set of markets.

GeoLift's market selection function, for example, ranks candidate test groups by how well the synthetic control fits and by the effect size they can detect (GeoLift methodology (opens in a new tab)). Practical rules of thumb for Canada:

• Keep Toronto, Montréal and Vancouver in the control pool unless the test is specifically about them. Their size makes them excellent contributors to a synthetic control and poor solo test markets.
• Use several mid-sized test markets rather than one large one. Spreading treatment across, say, five or six CMAs or FSA clusters reduces the risk that one local event ruins the test.
• Stratify by language: if you test in Quebec, match Quebec to Quebec.
• Don't accept a design that can only detect an implausibly large effect. If the power analysis says you can only detect a 40% lift and you expect 5%, the test will almost certainly come back "inconclusive". Change the design: more markets, longer duration, a bigger spend change or a less noisy KPI.

Why the fuss over power? Advertising effects are small relative to the natural noise in sales. In a well-known analysis of 25 large field experiments, Lewis and Rao showed how hard it is to measure advertising returns precisely, even with millions of people in the test (The Quarterly Journal of Economics, 2015 (opens in a new tab)). Geo tests face the same arithmetic, which is why design beats analysis.

Step 5: run the test cleanly

A typical test runs four to eight weeks, plus a "cool-down" of one to three weeks to catch delayed purchases. During that time:

• Implement the change precisely. Use location exclusions or separate campaigns per group. Double-check that national campaigns aren't quietly re-serving into holdout regions.
• Freeze everything else you can. No new regional promotions, no surprise budget changes in other channels that differ between groups.
• Watch for contamination. Check delivery reports by region weekly. If a platform's algorithm shifts spend unexpectedly, fix it fast and note it.
• Don't peek and stop early. Deciding to end the test the moment it looks good inflates false positives. Set the end date in advance.

Platform-side alternatives exist and are worth knowing. Google Ads offers geography-based Conversion Lift (opens in a new tab) for eligible advertisers, and Meta offers Conversion Lift studies in Ads Manager. They're convenient, but they measure one platform at a time and use the platform's own conversion data, so we prefer to run an independent geo readout alongside them when possible.

Step 6: read the result

At the end, compare what happened in the test markets with the counterfactual predicted from the controls. Report:

• Incremental conversions and incremental revenue (actual minus counterfactual), with a confidence interval.
• iROAS: incremental revenue divided by the change in spend. For a holdout test, the spend is what you would have spent in the holdout regions.
• Cost per incremental acquisition: change in spend divided by incremental new customers.
• Lift: incremental KPI as a percentage of the counterfactual.

Then compare the iROAS with what the platform reported for the same campaigns over the same period. The ratio between the two is a useful calibration factor. If the platform said 6.0 and the test says 2.0, you know to discount that platform's day-to-day ROAS by roughly two thirds when making decisions, until the next test.

Be honest about uncertainty. A result like "iROAS of 2.4, with a 90% interval of 1.1 to 3.7" is still decision-useful: it says the channel very likely pays back but you shouldn't scale it blindly. An interval spanning zero means the test couldn't separate the effect from noise. That's a finding too, especially if the platform claimed spectacular returns.

A worked example (illustrative)

Imagine a Canadian DTC brand spending on Meta prospecting nationally, outside Quebec. The question: does pausing prospecting for six weeks reduce new-customer orders enough to justify the spend?

1. Customer postal codes are mapped to CMAs and to FSA clusters for smaller centres, giving about 40 English-language regions.
2. A year of daily new-customer orders by region feeds market selection. The tool proposes six mid-sized regions (none of the three largest metros) whose combined history is closely matched by a weighted mix of the rest.
3. Power analysis on historical data suggests a six-week test can detect a lift of around 8% or larger. The team expects prospecting to drive more than that, so the design is approved.
4. Prospecting is excluded from the six regions for six weeks, then a two-week cool-down follows. All other activity continues nationally.
5. The readout compares actual new-customer orders in the six regions with the synthetic control and converts the gap into incremental orders, revenue and iROAS.

The numbers here are hypothetical; the structure is what matters. Each step produces a document someone can check, which is what makes the final answer credible to finance.

Common mistakes

• Testing too small a change. A 10% budget change will almost never be detectable. Test big changes and interpolate.
• Using revenue when orders would do. Basket-size noise can hide a real effect on order counts.
• Ignoring lag. Upper-funnel channels influence purchases weeks later. Include a cool-down and measure it.
• Running during a big season without enough history to model it.
• Treating one test as permanent truth. Results drift as creative, competition and audiences change. Re-test important channels at least once or twice a year.

From one test to a testing program

The real value comes from running tests on a calendar: one major channel per quarter, with each result feeding budget allocation (see budget allocation by marginal return) and the calibration factors your team uses every day. Over a year, you replace arguments about attribution with a small library of causal results specific to your business.

If you'd like help designing your first Canadian geo test, or a second opinion on one you've run, see our incrementality testing service or get in touch.