Marketing and measurement
Test one generator marketing change without fooling yourself
Write a small test with one clear change and one main result. Keep honest notes when the answer is still unclear.
Test one generator marketing change by naming the idea, the lead group and the main result before you start. Keep the old version and the date of the change.
For a service-area test, count out-of-area leads among reached leads with checked job addresses. Keep unknown and unreached leads visible too.
On this page
Write the idea before the test
Use a plain sentence. “We think a clearer service-area line will reduce out-of-area enquiries.” This gives the test a task.
Use the share of reached leads whose checked job address is outside your area. Divide that count by all reached leads with a checked address. Keep four counts: in-area, out-of-area, reached but area unknown, and not reached. The last two stay out of that share. If no addresses are checked, the share is unknown.
Keep the old version and change date. Note who worked the leads. Check costs and held visits too. They are extra checks, not the main test.
Set a fair comparison
Keep other parts as steady as practical. If you compare two versions at the same time, use a setup that splits them fairly. If that is not possible, state the limits.
A before-and-after check can be affected by weather, timing or staff cover. Do not call it a clean experiment when those facts changed too.
Let real outcomes catch up
Watch for broken forms or poor-fit traffic right away. Check sales results only when those leads have had time to move.
A pair of good leads does not prove the new version wins. Small groups can swing by chance. There is no single magic lead count that settles every test.
If the result is unclear, say so. Keep the cost of learning in view before extending the test.
Check the missing rows before calling a win
Here is made-up data. Before: 16 in-area and four out-of-area leads had checked addresses. Three reached leads had no checked area, and seven were not reached. The outside share is four of 20, or 20%.
After: nine were in-area and one was out-of-area. Eight had no checked area, and 12 were not reached. The outside share is one of ten, or 10%.
Both groups have 30 leads. Yet address coverage fell from 20 to ten. The lower outside share alone is not a clear win. Check the missing rows and staff cover before crediting the change.
Write the decision and its limit
At review time, choose keep, change, stop or learn more. Explain the reason in a few lines. Include any new gap that weakens the comparison.
Save the losing version too. It prevents the same idea from being tested again without context. The useful output is a better decision, not a graph made to look successful.
Sources and further reading
Sources checked .
