Lesson 8.2 · 60 minImproving conversion

What you will have

The student has one written hypothesis and a running A/B test on a hero section: each visitor keeps one variant, the variant travels with the analytics events, and every new lead in the CRM records the variant it saw.

Video

The video for this lesson is not recorded yet.

Prompts used in this lesson

Part 4 — Implementing the test with Claude

Purpose: run one A/B test on the hero section so I can learn whether a new
headline produces more leads. The hypothesis is in docs/experiments.md.

Context: the page to test is the landing page I name below. Analytics events
(form_start, form_submit, generate_lead, cta_click) are pushed to the data layer
as described in docs/measurement-plan.md. Consent handling is described in the
code from lesson 6.5. The leads table should have an ab_variant column; check
supabase/migrations/ and add a migration only if it is missing.

Build:
1. Two hero variants on [page path]: A is the current hero, unchanged. B differs
   only in the headline, using the text marked "Variant B" in docs/experiments.md.
2. Assignment: random, 50/50, decided on the server on the first visit so the
   visitor never sees the headline switch. It must stay the same for that visitor
   on later visits for the life of the test.
3. Store only the variant label ("hero1-a" or "hero1-b") in a first-party cookie.
   No visitor ID. Add one setting, AB_ASSIGN_BEFORE_CONSENT (true/false). When
   false, do not set the cookie until the visitor has accepted the analytics
   category; until then show A and treat the visitor as not in the test.
4. Push ab_variant to the data layer on page load and include it on the four
   existing events. Do not create new event names.
5. Send ab_variant with the form and save it in leads.ab_variant. Leads from
   visitors not in the test store nothing in that column.
6. In the CRM: show the variant on the lead card, and add a small table to the
   dashboard with leads and qualified leads per variant.
7. A preview link for me (for example a query parameter) that forces a variant
   without changing a real visitor's assignment logic.
8. Add the cookie to the cookie list on /privacy.

Constraints: no A/B testing library or third-party service. Do not change the
form, the other sections or any other page. Tell me whether this changes how the
page is rendered or cached, and what that does to speed.

Done when: you have run the build, loaded the page in a browser with each forced
variant and with a fresh visitor several times, shown me the data layer contents
for each, submitted one test lead per variant locally, and shown me both rows
with ab_variant filled. Report anything you could not verify.

Do along

Work on your own project and pause the video where a step says so.

1. Pause after Part 2. From your 8.1 report, pick the page with the most visitors and the weakest step in its funnel.
2. Write your hypothesis in docs/experiments.md: change, expected effect and reason, measure. Add the text for "Variant B".
3. Pause after Part 3. Decide your consent mode, note it in the same file, and set AB_ASSIGN_BEFORE_CONSENT in .env.local and Vercel. If unsure, use false.
4. Pause after Part 4. Run the prompt from Part 4 in plan mode with your page path. Review the plan, then let Claude build.
5. Check both variants in two private windows, the data layer, and two test leads locally.
6. Pause after Part 5. Add the variable and parameter in GTM, test in preview mode, publish. Register the custom dimension in GA4.
7. Pause after Part 6. Use the calculator with your own baseline and effect; write the required sample and the end date into docs/experiments.md.
8. Deploy, then repeat the variant checks on the live site. Delete or mark your test leads.
9. Commit and tag with the commands under "Recap and next".

Check your work

1. Open the tested page on the live site in several fresh private windows. Expected: you see both headlines across the tries, and each window keeps its headline after a reload.
2. In GTM preview, load the page and submit the form. Expected: ab_variant appears on generate_lead with the same label as the headline shown.
3. In the CRM, open the two newest test leads. Expected: each lead card shows its variant; the dashboard table counts one lead under each.
4. With AB_ASSIGN_BEFORE_CONSENT set to false, open the page in a fresh window and reject cookies. Expected: headline A, and no A/B cookie in the browser's storage panel.

Common problems

• The headline flashes from A to B on load → the variant is chosen in the browser after the page renders → tell Claude: "Assignment must happen on the server before the page is sent; fix it and show me a slow-network reload."
• Every visitor gets the same variant → the page is cached as one copy → ask Claude how the page is rendered now and to make the tested page vary by the cookie; confirm with five fresh windows.

Homework

About 30 minutes, on your own, after the lesson. Lesson 8.3 does not depend on it, and none of it touches the running test.

1. Start an experiment backlog. Under a heading "Backlog" in docs/experiments.md, write three more hypotheses in the same three-part form, each naming the finding or recording behind it. Do not build any of them. Done when: three entries each have a change, an expected effect with its reason, and a measure, and one is marked "next".
2. Write your stopping rule. Run the calculator twice more with your own baseline: once with half your minimum effect, once with double. Add to the running test's entry the three durations in weeks, and one sentence saying what you will do if the end date arrives with no clear difference. Done when: the entry shows three durations and that sentence.
3. Ask an outsider. Show both headlines (your preview link, or two screenshots) to one person who does not know your business, and ask what each one promises, in their own words. Write both answers in the log as a note, not as a test result, and leave the variants as they are. Done when: the note with both answers is in the log.

Commit without a tag: git add -A, git commit -m "Homework 8.2: experiment backlog", git push.

Save your work

git add -A
git commit -m "Lesson 8.2: hero A/B test with variant in data layer and CRM"
git tag lesson-8.2
git push
git push --tags