The test was running.The data was not.
An agency CRO team, 2022 to 2025. The build side of its testing: the installs, the variants, and the tracking that kept breaking.
For two and a half years I was one of three developers on an agency's CRO team. The strategist wrote the hypotheses and read the results. We built the variants, installed the testing tool, and kept the numbers honest.
This is the part of that work that rarely gets written up: the installs, the experiments that stopped collecting data without anyone noticing, and what the tests I built actually read, the losing one included.
Brands where I installed Convert, May to December 2024. One install was manual, so it would work with Checkout Extensibility.
Install tasks, May to December 2024
Test variants I built, alone or with another developer, for 18 tests at nine brands.
Test and build tasks, November 2022 to April 2025
Of those tests, the ones with a result on record. One lost clearly, one won on a sample too small to trust, five did not clear the bar.
Client tracking sheets
What the job was
Four of us: a strategist and three developers, of whom I was one. She wrote the hypotheses, set the goals and read the results. We built the variants, previewed and QA'd them, launched them, and when a number stopped making sense, we were the ones who had to find out why.
The tools changed under us. Google Optimize in 2022, split URLs in PageFly in 2023, VWO's free plan until its monthly session cap blocked further tests in March 2024, and Convert from May 2024, after I wrote up the technical case for it.
The program wound down in April 2025. The records it left are what this write-up is built from.
Six installs and one removal
Between May and December 2024 I installed Convert for six brands: creating each project, adding the tracking code to the theme, and setting up the goals its tests would be read against.
| Closed | Brand | What went in |
|---|---|---|
| May 2024 | A luggage brand | Convert |
| May 2024 | A water bottle brand | Convert |
| May 2024 | A supplements brand | Convert and Microsoft Clarity |
| Oct 2024 | Another DTC brand | Convert, with the store and analytics access checked first |
| Nov 2024 | A home goods brand | Convert, installed manually so it would work with Checkout Extensibility |
| Dec 2024 | A GNSS hardware maker | Convert, after asking whether the US and international stores were both in scope |
One store still carried a previous agency's Convert and Clarity code. I removed it in June 2024.
The same month, none of the agency's Convert experiences were tracking goals properly. The repair was assigned to me and closed within six days.
When the data stopped
An experiment that is not collecting data looks almost exactly like one that is. The page renders, the variant shows, the test reads as running. Three times the numbers simply stopped, and each time the cause was in the wiring, not in the variant.
A condition that matched nobody
In June 2024 the strategist found that a returns-policy test on the supplements brand was collecting no data at all. The next day I found why: the experiment was targeted by page type where it needed a page URL, so no visitor ever qualified. I changed the condition and it started tracking.
Two kinds of revenue in one checkout
The same brand sold subscriptions and one-time orders, and the strategist wanted the two read separately. In February 2025 I adapted Convert's documented revenue script to split each order's lines on whether they carry a recurring selling plan, and to send each total to its own goal.
Shopify.checkout.line_items.forEach(product => {// Following line checks if it is a subscribed product by querying a specific property of the productif (product.selling_plan_allocation && product.selling_plan_allocation.selling_plan.recurring_deliveries) {total_subs_revenue += product.line_price - product.line_level_total_discount;total_subs_products += 1;} else {total_non_subs_revenue += product.line_price - product.line_level_total_discount;total_non_subs_products += 1;}});if (total_subs_products > 0) {reportRevenue(total_subs_revenue, total_subs_products, SUBSCRIPTION_GOAL_ID, 'subscriptions', Currency);}if (total_non_subs_products > 0) {reportRevenue(total_non_subs_revenue, total_non_subs_products, ONE_TIME_GOAL_ID, 'non subscription', Currency);}
Then the catch, confirmed with Convert's support over the following week: the split needed Convert's manual integration, not the Shopify app the store was running. I wrote that into the task the day it was confirmed.
Goals that went quiet
In March 2025 a three-arm urgency test on a probiotics brand was showing no conversion events. Shopify had deprecated the post-purchase page scripts its goals depended on, and the repair came to me on 31 March. On 1 April I rebuilt the test's goals on Shopify's standard customer events, fifteen of them, and relaunched it that afternoon.
- Shopify events
Thirteen standard customer events, from page viewed to checkout completed, fired by Shopify itself.
- Convert goals
One goal per event. Checkout completed feeds the revenue goal; subscription and one-time orders get their own.
- Experiment report
Conversions and revenue per arm, counted from events Shopify still supports.
- The strategist
Reads the result. The build is done when that number can be trusted.
The same setup could also give the supplements brand its subscription and one-time split, through Shopify's own events rather than a checkout script.
What the tests I built read
Of the 18 tests I built variants for, seven have a result on record. The others were never read into the tracking sheets before the program ended, which is the same gap case study 03 is about.
| Test | Outcome | As the tracking sheet recorded it |
|---|---|---|
| Returns note in the shipping dropdown, product page, 2024 | Lost | Original ahead by 13.8% at 98.7% confidence. Not deployed. |
| Larger red sale price, collections, 2024 | Won | Variant ahead at 95.7% confidence, on a sample too small to lean on. Deployed. |
| Checkout button copy, cart, 2024 | Not significant | Original ahead by 6.5% at 94.1%, short of the 95% bar. |
| Price moved below the title, product page, 2024 | Inconclusive | No difference after 53 days. Stopped. |
| Site-wide button colour, 2024 | Inconclusive | No difference. Stopped for Black Friday. |
| Recommendation widget logic, product page, 2024 | Inconclusive | No difference after a week. Retested. |
| Mobile product page redesign, 2022 | Inconclusive | Both versions finished at the same conversion rate. |
One clear loss, one win I would not lean on, and five that did not clear the bar. The loss was the useful one: the test kept a change off the site that it said would cost conversions.
<style>.price-item.price-item--sale.price-item--last {font-size: 18px;font-weight: bold;color: #F87666;}/* Small devices (phones, less than 768px) */@media (max-width: 767.98px) {.price-item.price-item--sale.price-item--last {font-size: 16px;}}
What I would do differently
Two of the failures above were caught by someone noticing an empty report, five and thirteen days after launch. What I would do differently is check that every goal fires on a preview before a test goes live, and again on its first day.
The variant can be perfect and the test still worthless, if nothing is counting.
Client names, revenue and traffic figures are withheld under NDA. Every claim traces to a dated primary record: the agency's task records and their comment threads, Convert experiment and variation IDs, a Slack thread from 1 April 2025, and the client tracking sheets, all preserved in a private archive built on 27 August 2026. Available for verification in a private conversation.