Meta A/B test eligibility: what the check actually checks
The connector answers the eligibility question fast. What it verifies is narrower than the name suggests.
Meta's ads connector will tell you whether an A/B test is possible before you build one, but the check is narrower than its name suggests. Handed only an ad account it skips the A/B question altogether and answers about a different study type. Handed two ad sets it runs seven named checks — and not one of them asks whether the ad sets are delivering. On 30 August 2026 it cleared two fully paused ad sets, in two separate accounts, as ready to test.
The appeal of handing a split test to a model is that the fiddly part is not the idea, it is the wiring: two ad sets differing in exactly one way, same budget, same window, nothing else touching either while it runs. That is bookkeeping, and bookkeeping is worth delegating.
So the first move is to ask whether a test can run at all. The connector has a tool for that, and what it verifies is not what an operator assumes.
One call, two unrelated questions
The eligibility tool covers two things with almost nothing in common: an A/B test, where two variants split traffic and a cost metric picks a winner, and a Conversion Lift study, a randomised holdout estimating how many conversions the advertising actually caused. Only the second is evaluated when you pass an ad account by itself. The A/B half comes back untouched, flagged as skipped:
"split_test": {"eligible": null, "skipped": true,
"reason": "A/B test eligibility requires at least 2 ad_entity_ids
(campaign, ad set, or ad). Re-run with ad_entity_ids populated
to evaluate split test eligibility."}What surrounds it is the problem. The lift half does return a full verdict, and on every account tried that verdict was a failure with identical wording, naming three requirements: spend, conversion volume, and signal quality. Identical on accounts with real delivery history and on one with no payment method attached. It never says which of the three you missed.
Ask "can I run a test?" with just an account, then, and you get a confident-looking no — to a question you did not ask.
The seven checks, and the one that is missing
Pass two entity IDs and the A/B half wakes up, reporting named checks each with a pass or fail: entity resolution, consistent entity types, minimum entity count, same ad account, no active studies, no iOS 14 campaigns, no Reach & Frequency campaigns.
Read that list for what is absent. Nothing asks whether the entities are running, or about budget, audience size, or whether either ad set could gather enough events to separate from the other. The checks are structural: they confirm the objects are of a kind that can be wired into a study.
Two ad sets whose parent campaigns were paused returned this:
"eligible": true, "test_level": "ad_set",
"recommendations": ["Ready to create a ad_set-level A/B test with 2 entities."]Both reported a paused effective status through the same connector, minutes apart, and it reproduced in a second account with a different objective. "Ready" is a statement about shape, not about readiness.
A real finished test pulled off one account returned a winner_result block carrying
has_winner, an empty list of cells, and a confidence_threshold value. Meta
publishes what that bar is and means under
confidence in your
tests and experiments — confidence being the likelihood the same variant would win if the
test ran again. Read that page before accepting a winner: the threshold is not a parameter the
connector exposes, and it is not the bar most people picture when they hear a test reached
significance. That same record reported its start and end date as the same day — the dates a
study carries are the ones it was configured with, not evidence it ran long enough to learn
anything.
Two more things the output does not say plainly
A wrong entity ID does not surface as a failed entity-resolution check, though that check exists and passes when the IDs are good. It returns an internal error asking you to try again later, flagged as retryable — and repeating the call returned the identical error, so nothing was transient. A typo reads as a Meta outage, the same misdirection as the error tool's own blind spots.
And the lift section labels everything it echoes back as a campaign. Ad set IDs went in; the breakdown returned them under campaign keys, with campaign names, and signed off counting how many campaigns were compatible. Nothing there was a campaign, and a model summarising that reply in prose repeats the wrong noun.
| Question | Does the eligibility check answer it? | Where the answer lives |
|---|---|---|
| Are these objects the same type? | Yes — mixed types fail, level comes back unknown | The check output |
| Are they currently delivering? | No — paused entities are cleared as ready | Entity status, fetched separately |
| Is there enough volume to learn anything? | No — nothing in the check looks at size | Your judgement, before you build |
| What bar decides the winner? | Not at eligibility time | The finished study record, and Meta's docs |
The habit that makes this safe is reading the reply, not the summary of it. See how the weekly review is structured →
How to use the check without being misled by it
-
Fetch the entity IDs first, then ask START HERE
The A/B half is dead weight without two IDs of the same type. Pull the ad sets you care about, then run the check with both, so you see which structural check would have stopped you. The trade: two calls instead of one, and you have to know which level you are testing at before you ask.
-
Confirm delivery yourself, in the same sitting
Ask for the effective status of both entities in a separate call and read it back. A paused parent campaign is the failure the eligibility check will not catch, and it is a common state for the ad sets you reach for when hunting two comparable things. The trade: status only rules out the obvious case. It says nothing about whether either ad set has the budget to produce a separable result.
-
List existing studies before you change anything
An account with nothing running returns an empty set of studies and an explicit flag saying no study is active. Cheap, and it is what stops you editing budgets underneath a measurement window that is still open. The trade: the listing includes finished studies by default, so a non-empty result does not mean something is live.
Eligible means the pieces fit together. It does not mean the test is worth running.
- Asking about testing with only an account ID. The A/B half is skipped and the verdict belongs to a different study type.
- Reading "Ready to create" as an endorsement. It confirms two objects can be wired together. Delivery, budget and volume sit outside the checks.
- Treating the internal error as Meta being down. A bad entity ID produces a retryable-looking error that repeats exactly on retry.
- Repeating the nouns in the lift breakdown. Ad sets come back described as campaigns, and a prose summary carries that forward.
Try this tonight Ask for the studies on your ad account. If the list is empty, pick the two ad sets you would genuinely want to compare, ask for their IDs and effective status in one go, then run the eligibility check with both IDs. If consistent entity types fails, you picked one campaign and one ad set — fix that and you are done for tonight. If everything passes, look at the status you just fetched: a paused entity means the check would have cleared a test that could never produce a result, and tonight's job is turning delivery back on, not building the test.
The pattern runs right through this connector: the call succeeds, the wording is confident, and the important part is what was never in scope. The counted manifest lists everything it exposes, choosing winners on the wrong column covers what to do with a result, and the connection walkthrough covers setup.
Frequently asked questions
Can Claude set up an A/B test on my Meta ads?
The official connector carries tools to check eligibility, create a split test, read one back, and cancel or end one. Checked on 30 August 2026, the eligibility step works but verifies only that the objects are structurally testable. Whether the test is worth running is not something it evaluates.
Why does the eligibility check say my account cannot run a test?
Most likely it answered about a Conversion Lift study rather than an A/B test. With only an ad account supplied, the A/B half is skipped and flagged as such, while the lift half returns a verdict naming spend, conversion volume and signal quality requirements. Re-run with at least two entity IDs of the same type to get an A/B answer.
Does the check know whether my ads are running?
No. On 30 August 2026 it cleared paused ad sets in two different accounts as ready to test, with no warning about delivery. Entity status is available through the connector, but you have to ask for it separately and read it yourself.
What decides the winner of a Meta split test?
A primary cost metric, defaulting to cost per result, judged against a confidence threshold Meta sets and returns in the finished study record. Meta documents what confidence means and the level it uses in its help centre. The threshold is not something the connector lets you change.
Read the reply, not the summary
The weekly routine for running a Meta account through Claude: the calls to make, and the fields to read back before you believe the summary.
See what's inside — $67 founding pricePre-sale: the modules are still being recorded and founding members get the checkout link first. No charge today. Public price $97.