A/B testing with PostHog

Use A/B Test in Storyblok to compare versions of a section, such as a headline or a call to action. The website chooses which version to display, and PostHog records exposures and interactions.

Start with one change and one success measure, for example: “Does a benefits-focused headline lead to more demo bookings?”

Set up the content in Storyblok

1. Open the page and add an A/B Test component where the section should appear.
2. Enter a unique run name in PostHog feature flag key, for example hero-message-v1. Keep this name unchanged during the test.
3. Under Variants, add two A/B Test Variant components.
4. In the first variant, set PostHog variant key to control. Add the existing section to its Body.
5. In the second variant, set PostHog variant key to benefits. Add the alternative section to its Body.
6. Review both versions in Storyblok preview. With multiple variants, preview shows them together with labels so you can edit each version.
7. Leave Enabled switched on when you are ready to run the test, publish the page, and wait for the website build and deployment to finish.

Publishing alone does not update the test configuration already deployed on the website. Starting, changing, and stopping a test all require a new website build and deployment.

Settings reference

Component	Field in Storyblok	What to enter or check
A/B Test	PostHog feature flag key	A unique name for this test run. Use the same name in PostHog reporting.
A/B Test	Variants	The versions to compare, with the existing version first. At least two are needed for a test.
A/B Test	Enabled	On by default. Turning it off makes the first variant the public content after deployment.
A/B Test Variant	PostHog variant key	A unique name within the test, such as control or benefits.
A/B Test Variant	Body	The content displayed for this version.

The first variant is always reported as control, even if you give it another name. Naming it control keeps the editor and reports easy to compare. Alternative names must match PostHog exactly, including capitalization.

Variant names can contain letters, numbers, underscores, and hyphens, up to 80 characters. Do not reuse a name within a test. Reserve control for the first variant and never use mixed; that name is used for visitors who encounter multiple versions.

Use a new run name, such as hero-message-v2, for a materially different test. Reusing an old name mixes the results.

What visitors see

Two variants receive an equal random split; three receive approximately one third each. Small samples will not have exactly equal counts.

The website selects the content before displaying the page. A fresh page load or navigation to another page can select a different version. There is no permanent assignment that keeps a person in the same version across visits. Accepting cookies does not make that assignment permanent.

Storyblok controls the content and whether the test runs. Changing traffic percentages, targeting rules, or the flag state in PostHog does not change the website’s selection. To stop the website test, use Enabled in Storyblok and deploy the change.

A disabled test, a missing run name, or a component with only one variant displays the first available variant on the public website. Preview still shows multiple authored variants for editing; it is not a way to test production exposure reporting.

Compare results in PostHog

The website sends exposure events and separate attribution events for interactions. Use the following Insights reports to inspect the current implementation. Set the same date range for each report, starting when the test deployment became available.

Which versions were displayed?

Create an event-count report for ab_test_exposure, filter flag_key to your run name, and break down by variant_key.

An exposure means the variant was rendered on the page. It does not prove the visitor scrolled to that section. Each rendered test reports once per page view; repeated placements of the same test share that exposure. Reloads can generate another exposure, so this count is not a count of unique people.

Which versions led to bookings or clicks?

Create an event-count report for experiment_attribution with these filters:

Property in PostHog	Example value	Purpose
experiment	hero-message-v1	Select this test run.
scope	journey	Include the test’s recorded history leading to the action.
action	demo-booked	Count confirmed bookings from the embedded scheduler.

Break down by variant. For a report on tracked CTA clicks, replace demo-booked with cta-click. Form submissions use form-submitted; custom CTA events can have their own action names.

Choose exactly one scope per report:

Scope	Question it answers
journey	Which versions did the visitor encounter during the attribution window?
page	Which tests were rendered on the page where the action happened?
element	Which tests contained the clicked element or conversion component?
source	Which tests contained the last tracked CTA leading to the action?

One action can produce several attribution records, covering different tests and scopes. Do not add scopes together or add different experiments’ booking counts to calculate total bookings.

For overall conversion totals, use the original hubspot_form_submitted event. Filter category to demo-booked for scheduler bookings. The attribution event uses action; it does not copy the original event’s category property.

Native PostHog experiment results

If you create an experiment in PostHog, match its feature flag key to PostHog feature flag key, keep control as the baseline, and match the alternative variant names. Configure ab_test_exposure as its custom exposure event. The website does not emit the default $feature_flag_called event for these tests.

Native experiment conversion results need an integration follow-up before you rely on them. PostHog requires the experiment’s $feature/<run-name> property on both exposure and metric events. The current website supplies it on exposures, but its conversion attribution uses the separate records described above. Creating an experiment in PostHog alone does not complete that connection. See PostHog’s requirements for experiments with external assignment.

The Insights reports above show recorded activity. Raw booking counts alone do not establish a winning variant: account for exposure volume, repeat visits, and visitors who see multiple versions before making a decision.

Bookings on a later page

Experiment context can travel through supported links to other pages on the same website, including a page with the embedded booking scheduler. The attribution window expires after 30 minutes of inactivity.

When someone sees two versions of the same test during that window, their journey attribution becomes mixed. Review this group separately. Page and element attribution still describe the version displayed at the time of the action.

This handoff does not automatically connect externally hosted booking pages, HubSpot CRM records, other devices, or later-day visits. Source attribution describes the preceding tracked CTA context; copied or shared links mean it is not proof of a unique person’s click-to-booking path.

Consent and measurement

PostHog can send events before marketing consent, using memory-only storage without a PostHog cookie or localStorage entry. Accepting marketing consent allows persistent PostHog storage; withdrawing it resets the identity and returns to memory-only storage.

This is different from the consent requirement for ChatGPT Ads conversions. The Track ChatGPT Ads conversion and Track ChatGPT Ads appointment settings do not switch PostHog A/B measurement on or off.

Without persistent storage, reporting cannot reliably recognize the same visitor across fresh page loads. Blocked analytics or an unavailable PostHog connection can also leave gaps in measurement while the content test continues to run.

Multiple tests on one page

A page supports up to five independent tests and 32 possible combinations. Two tests with two variants each create four combinations. Three tests with three, three, and four variants create 36 combinations and exceed the limit. Nested tests count towards this limit too.

Use different run names for independent tests. Repeating a run name on the same page makes those placements share a version; their variant names and order must match. Invalid or oversized configurations can prevent the website build from completing.

Check a launch

1. Check each variant’s content, links, and mobile layout in Storyblok preview.
2. Publish the page and confirm the website deployment succeeds.
3. On the deployed page, inspect repeated fresh visits to confirm the variants can appear. A handful of visits is not enough to check the split statistically.
4. In PostHog, check that ab_test_exposure contains the correct run name and variant names.
5. Use an agreed test environment to verify a CTA and a confirmed booking appear in the corresponding attribution reports. Check both immediate conversion and navigation to another page before conversion.
6. Confirm each report filters one experiment and one scope, and review any mixed activity separately.

End a test

To return to the baseline, turn off Enabled, publish, and deploy. The first variant becomes the displayed content.

To keep an alternative, put the desired content in the first variant or replace the test component with that content, then publish and deploy. Ending an experiment or choosing a winner in PostHog does not publish website content automatically.