How to A/B test a lead source change with small volumes (estate agents and brokerages)
Oshy Labs Ltd. Published 19 Oct 2026.
You can test one change to how you handle seller leads with the volumes a single office gets, as long as you accept two things: nothing below 20 leads per cohort is a result, and anything above it is a direction, not a verdict. Tag each lead control or variant in your CRM, run for six weeks, read the gap with its sample size warning, and record a decision to continue, revise or stop.
This guide is for owners, directors and lead managers at independent estate agencies in the UK and residential brokerages in the US whose offices receive tens of seller leads a month, not thousands, and who still want to know whether a change in reply, speed, follow up or routing did anything.
Apply for a 30 day pilotThe pilot includes two tests run on your own export.
Why a conversion rate on 12 leads means nothing
Take an office that routed 12 portal leads through a new lead manager last month and signed 3 of them. That is a 25 per cent signed rate, which sounds like a number you could put in a meeting.
Now do the arithmetic (example figures, invented to show the sums; not a benchmark, not from any customer):
- Each lead is worth 1 ÷ 12 = 8.3 points of signed rate. One vendor changing their mind takes you from 25 per cent to 16.7 per cent. One extra signing takes you to 33.3 per cent. The whole range a reasonable person would argue about fits inside two leads.
- The standard error of a proportion is √(p × (1 − p) ÷ n). At 25 per cent on 12 leads that is √(0.25 × 0.75 ÷ 12) = 12.5 points. A rough 95 per cent interval is the rate plus or minus about two standard errors, so roughly 0 to 50 per cent. "Somewhere between none of them and half of them" is not a finding.
- At 20 leads, each lead is worth 5 points and the standard error at 25 per cent is 9.7 points. At 100 leads, each lead is 1 point and the standard error is 4.3 points.
Now put a control cohort next to it. Suppose the 12 leads that went straight to the branch signed 2, which is 16.7 per cent. The lead manager route is "ahead by 8.3 points". It is ahead by one lead. Swap one outcome between the two piles and the result reverses.
That is why a rate on 12 leads means nothing on its own, and why "the new thing is converting better" is usually a sentence about one or two vendors rather than about the new thing.
The 20 per cohort floor, and why a directional result is still worth having
Broker Lead Lab will not show a test result until each cohort, control and variant, has at least 20 leads. Below that the screen says "Not enough data" and nothing else; not a rate, not a provisional read. The number 20 is a product rule chosen to stop small piles being read as results. It is not a claim that 20 makes anything certain, and the interval arithmetic above shows it does not.
Above the floor you get a directional statement with a warning attached. Two cohorts of 20, control signing 4 (20 per cent) and variant signing 6 (30 per cent), reads, in the product's own words:
Variant is ahead of control by 10.0 points on signed rate.
With 20 leads in the smaller cohort, a gap this size could be chance. Let both cohorts grow before deciding.
The method behind the warning is a two proportion z test; the warning flips to "would appear by chance in fewer than 1 in 20 tests" only when the gap is large relative to the sample. For the 4 of 20 against 6 of 20 above, z is about 0.73, so it stays as "could be chance". For 9 of 60 against 21 of 60 (15 per cent against 35 per cent, again example figures) z is about 2.5 and the warning reads:
A gap this size would appear by chance in fewer than 1 in 20 tests with 60 leads in the smaller cohort. It is still one test in one period. Keep the cohorts running or repeat the test before changing how every office works.
No p value is shown and no winner is declared. A person closes the test and writes the decision.
Why bother with a directional result at all, if it can still be chance? Because the alternative in a small office is not a bigger sample, it is no test. A directional read with its sample size printed next to it does three useful things: it tells you whether the change is pointing the way you hoped; it tells you how far you are from a gap that would be hard to get by luck; and it stops the office from adopting a change on the strength of one good fortnight. A read labelled "could be chance" that you keep running is a better position than an untested change you have already rolled out.
A six week plan to test one change
One change, one metric, one tag. If you want to test two things, run them one after the other.
Before week 1: choose the template and write the hypothesis
Broker Lead Lab has four test templates. Each is a single operational change a manager can actually make, with a suggested primary metric:
| Template | What changes | Example | Suggested metric |
|---|---|---|---|
| First reply | Change what the first message or call says. | A short, specific first reply that offers two valuation slots. | Appointment rate |
| Response time | Change how fast a lead gets its first contact. | Call within 10 minutes during office hours instead of the next working day. | Contact rate |
| Follow up sequence | Change the number, timing or channel of follow ups. | Three follow ups over ten days instead of one. | Signed rate |
| Source or campaign | Compare two sources, campaigns or handling routes. | Portal leads routed to a lead manager versus straight to the branch. | Signed rate |
Pick the metric that can mature inside your window. Contact rate settles within days. Appointment rate settles within a couple of weeks. Signed rate depends on your lag from lead to instruction, which for many agencies is longer than six weeks; if you choose it, plan to read the result later than week 6, or test an earlier stage first. Write the hypothesis as one sentence of the form "X raises Y compared with Z", for example "Calling within 10 minutes raises the contact rate compared with our usual response time."
Before week 1: decide the allocation rule and write it down
The rule that puts a lead into control or variant must be something nobody on the team can bend towards the leads they like. Alternate leads as they arrive, or split by odd and even days, or by the last digit of the CRM reference. Do not split by agent, by source or by price band unless that is the thing you are testing, because then you are testing the agent, the source or the price band.
Weeks 1 to 4: run it and tag as you go
Add one field to the lead record in your CRM, or one column to the sheet you export, and fill it at the moment the lead is allocated: fast-reply:control or fast-reply:variant (the tag format is in the next section). Do not read interim numbers as results. If you must look, look at the count of leads per cohort, not the rate, because that tells you how far you are from the floor.
How many weeks you need depends only on volume. An office that receives about ten seller leads a week and splits them evenly reaches 20 per cohort in about four weeks (example figures). If your office receives five a week, it takes eight. Extend the running period rather than lower the floor.
Week 5: let the outcomes catch up
Stop allocating new leads into the test at the end of week 4 (or whenever both cohorts have passed 20) and give the last leads time to reach the stage you are measuring. For contact rate a few days is enough. For appointment rate, give it the fortnight.
Week 6: import, read, decide
Export the leads created during the test window with the cohort column included, import the file, open the test, and read the gap and its warning. Then use the decision table below, close the test, and write the decision in the box. Closing records the decision in the audit log with your name and the time; nothing in the product closes a test for you.
Apply for a 30 day pilotIf you want the two tests in the pilot to be the first two in this plan.
How to tag cohorts in a CSV export
The importer looks for one optional column, mapped to the field called Test cohort. Its header can be anything; the automatic mapping recognises cohort, test cohort, test, variant, experiment, cohort tag, test group and group, and you can map any other header by hand.
Each value is a tag, a colon, and the cohort:
test-name:control test-name:variant
The tag is lowercase letters, digits and single hyphens between words. The cohort must be exactly control or variant. fast-reply:variant is valid. Fast Reply: Variant, fast-reply:test, fast_reply:variant and fast-reply:a are not; a row with a malformed tag is rejected with the reason "Test cohort must look like test-name:control or test-name:variant", and you fix the cell and import again.
The tag is the one shown when you create the test in the app; the test name you type is turned into it (lowercase, words joined by hyphens, up to 40 characters). Leads with a matching tag join the test when the test is created, and leads imported before the test existed join it automatically when the tag matches, so you can tag in your CRM from day one and create the test in the app later.
Leave the cell empty for leads that are not in any test. Do not reuse a tag for a second run of the same idea; call the second one fast-reply-2 so the two runs stay separate.
The same column format is used in the two sample files on the docs page, which carry a running cohort tag through 180 leads each, so you can see the result screen before you tag anything of your own. The export you build for this is the same one described in the cost per signed instruction guide: one row per lead, created date, source, a status or stage date, and no names, emails or phone numbers.
Decision table: continue, revise or stop
Read the gap and the warning together, then pick a row. The gap is "variant ahead of (or behind) control by so many points on" the metric you chose; the warning is either "could be chance" or "would appear by chance in fewer than 1 in 20 tests".
| What the screen shows | Decision | What you do |
|---|---|---|
| “Not enough data” in either cohort | Neither | Keep allocating leads. Do not read a rate from your CRM instead; that is the number the floor exists to stop. |
| Variant ahead in the direction you predicted, warning says could be chance | Continue | Keep the variant running for the leads already in it and let both cohorts grow, or repeat the tag in the next period. Do not roll it out to other offices yet. |
| Variant ahead in the direction you predicted, warning says fewer than 1 in 20 | Continue | Repeat it in a second period or a second office before changing how every office works. One test in one period is one test in one period. |
| Variant level with control, or ahead by less than one lead’s worth | Revise | The change as written is not doing enough to see. Change the variant (a sharper first reply, a faster target, a different channel), give it a new tag, start again. |
| Variant ahead on the primary metric but you can see a cost elsewhere (more appointments, fewer signings; faster calls, more complaints) | Revise | Keep the part that moved the metric, change the part that caused the cost, new tag. Record the side effect in the decision. |
| Variant behind control, warning says could be chance | Revise or stop | If the change is cheap to run, let it grow one more period before deciding. If it costs staff time or money every week, stop. |
| Variant behind control, warning says fewer than 1 in 20 | Stop | Close the test, write down what the variant was and that it ran behind, and do not run the same variant again without a reason. |
| The team could not apply the change consistently (the 10 minute target was hit half the time) | Stop | You tested the intention, not the change. Fix the operation first, then test it with a new tag. |
Whatever row you land on, write the decision as a sentence that someone reading the audit log next year will understand: what changed, what the gap was, how many leads were in the smaller cohort, and what you are doing about it.
What Broker Lead Lab does and does not do
Taken from what is built in version 0.8.0.
It does. Import the CSV your CRM already exports, propose a column mapping and let you correct it, drop columns that look like a person's name, email address, phone number or home address before mapping (by header and by value, listed on the import screen), validate each row with a reason, flag duplicates, and roll an import back. Report leads, contact rate, appointment rate, signed rate, estimated commission and cost per signed by source, campaign, agent, office and month, with the definition shown on every metric label. Run tests on the four templates above from cohort tags in your file, refuse to show a result below 20 leads per cohort, give a directional read with a sample size warning above it, and record your decision when you close a test. Keep an audit log, export CSV, print a report, and delete an organisation with a ledger. Hold data on Supabase in London (eu-west-2); the raw file is kept for 30 days by default and then removed by the retention job.
It does not. Connect to your CRM or to portals; imports are CSV only. Allocate leads to cohorts for you; the tag comes from your CRM or your sheet. Declare a winner, show a p value, or pick a decision; a person closes the test. Attribute anything the export does not carry; if the outcome is not in the file, the lead counts as not having reached it. Promise more instructions, more listings, or any lift. The import completeness and source classification assistant (Jev) is off by default.
Pricing, as shown on the live pricing page, excluding tax: 30 day Pilot, $299 one off or £239 one off (one office, approved CSV import, baseline report, two tests, 30 days). Solo Office, $79 per month or £59 per month (up to 3 users, up to 3 sources, monthly imports, tests and exports). Growth Office, $149 per month or £119 per month (up to 10 users, up to 8 sources, up to 10 offices, tests, exports and audit log). The pricing page states that pricing is a test and may change with 30 days notice to existing customers.
Frequently asked questions
Can I run two tests at the same time?
On different leads, yes; each lead carries one tag, so a lead is in at most one test. Running two tests on the same leads is not supported, and splitting a small office's leads across two tests doubles the time each takes to reach 20 per cohort. Run them one after the other.
What if control reaches 20 leads and variant does not?
The result stays at "Not enough data" until both cohorts have 20. Check your allocation rule: an uneven split usually means someone is choosing which leads get the change. If the rule is sound and the split is just uneven by chance, keep allocating.
Does “ahead by 10 points” mean the variant won?
No. It means that, among the leads in this test so far, the variant's rate is 10 points higher than the control's. The warning underneath says whether a gap that size would usually appear by luck at that sample size. Either way it is one test in one period, which is why the decision table says continue or repeat rather than roll out.
Can I tag leads from a past month and test them retrospectively?
You can tag them and the importer will accept the tags, but you are then describing which leads happened to get the change, not testing it. If the "variant" leads were chosen after the fact, or the change was applied to the leads an agent preferred, the gap tells you about the choosing. Use past leads to estimate your volume and lag, then run the test forward with an allocation rule.
Next step
The pilot includes an approved import, a baseline report and two tests. Six weeks does not fit inside 30 days, so the way to use the pilot for this plan is to start tagging leads in your CRM before the pilot begins; the import then already carries weeks of tagged leads, and the pilot's two tests are the read and the decision rather than the waiting.