Randomize / Randomly assign people to different versions.
Random assignment between groups is the gold standard of A/B testing. Without randomizing you may not be able to attribute any improved results to your program. Download our randomization tool below to sort program participants quickly and easily into two or more groups. Once you’ve randomly assigned people to receive different versions, you’re ready to run the different versions and conduct your test!
Why Randomize
Randomization is a powerful tool for understanding the impact of a policy or program. Randomization allows you to compare accurately between groups by:
- Balancing observable factors, like English-language skills, age, or income, between groups that get Version A vs. Version B
- Balancing important factors that you can’t observe, like motivation or engagement, between groups that get Version A vs. Version B.
As long as you have a large enough sample size, randomizing allows you to assume that these factors will be balanced between groups on average. This can allow you to attribute the difference between groups to the policy or program that you are testing, and not to any other factors.
For example, let’s say we offer the program to a group that only has people ages 18-28 and withhold the program from a group with only people ages 29-38.
When we compare differences in outcome between the two age groups, we won’t know if any observed difference in outcomes is due to the program or because of differences in age between groups.
Random selection helps us achieve statistically identical comparison groups that have an equal amount of men, women, people of certain ages, socio-economic status, etc.
Many ways of assigning people to groups that might seem random at first actually aren’t.
Geography: Not Random! People live in different neighborhoods for many reasons. If you assign some neighborhoods to Version A and others to Version B, then any results you observe might be due to differences in neighborhoods instead of the designs you are testing.
Name: Not Random! It can be tempting to sort groups into Version A and B by alphabetical order, but this isn’t random. Some names are more prevalent among certain ethnic groups or age ranges.
Sign-up order: Not Random! People who sign up early might be different than people who sign up later.
ID Number: Maybe Random. Sometimes program ID numbers are assigned randomly, but usually they aren’t. Avoid randomizing this way unless you know exactly how the ID numbers were assigned.
Computerized number: Random! The best way to random is to use a computerized random-number generator to assign individuals to different versions. Don’t worry, it’s easy! We’ve made a simple excel sheet that can help you leverage true randomization at the touch of a button. Download it by clicking the button above.
PREVIOUS STEP:
Previous: Prepare Your StudyNEXT STEP:
Next: See ResultsCASE STUDY: RANDOMIZE
Paying parking tickets is a hassle. To help people remember to pay on time and avoid extra penalties and possible legal consequences, the City of Chicago partnered with ideas42 to find a solution. Together we designed a series of behaviorally informed postcard reminders encouraging residents to pay before their fine doubles 68 days after the ticket was issued.
The A/B test randomized people into two groups. One group – Group A – received the normal letters and a postcard with a behaviorally designed message two weeks before their fine doubles. Another group – the control group or Group B – received only the normal City letters. Researchers decided to randomize the groups by the notice number associated with the ticket. The notice numbers served as objective identification numbers free of statistical bias. With a large sample size, randomization created balanced demographics for treatment and control groups that allowed researchers to isolate the effects of the postcards. Randomization was also fair. Each notice had an equal chance to be placed in one of the two groups and potentially benefit from the intervention.
In the end, the behavioral postcard increased the payments made by 35% and saved both the City and its residents time and money by cutting mail payments in half, shifting them to online payments instead.