Test, Learn, Adapt - Developing Public Policy with Randomised Controlled Trials Laura Haynes

Brilliant behavioral economics paper Test, Learn, Adapt released by Behavioral Insights Team, a tiny branch of government established by David Cameron (Britain's conservative prime minister).

Randomised controlled trials (RCTs) are the best way of determining whether a policy is working. They are now used extensively in international development, medicine, and business to identify which policy, drug or sales method is most effective. They are also at the heart of the Behavioural Insights Teamʼs methodology. However, RCTs are not routinely used to test the effectiveness of public policy interventions in the UK. We think that they should be.

The ʻtest, learn, adaptʼ philosophy set out in this paper is at the heart of the way that the Behavioural Insights Team works. We believe that a ʻtest,learn, adaptʼ approach has the potential to be used in almost all aspects of public policy:

• Testing an intervention means ensuring that you have put in place robust measures that enable you to evaluate the effectiveness or otherwise of the intervention.
• Learning is about analysing the outcome of the intervention, so that you can identify ʻwhat worksʼ and whether or not the effect size is great enough to offer good value for money.
• Adapting means using this learning to modify the intervention (if necessary), so that we are continually refining the way in which the policy is designed and implemented.

We have identified nine separate steps that are required to set up any RCT. Many of these steps will be familiar to anyone putting in place a well-designed policy evaluation – for example, the need to be clear, from the outset, about what the policy is seeking to achieve. Some – in particular the need to randomly allocate individuals or institutions to different groups which receive different treatment – are what lend RCTs their power. The nine steps are at the heart of the Behavioural Insights Teamʼs ʻtest, learn, adaptʼ methodology, which focuses on understanding better what works and continually improving policy interventions to reflect what we have learnt. They are described in the box adjacent.

TEST

1. Identify two or more policy interventions to compare (e.g. old vs new policy; different variations of a

policy).

2. Determine the outcome that the policy is intended to influence and how it will be measured in the trial.

3. Decide on the randomisation unit: whether to randomise to intervention and control groups at the level of individuals, institutions (e.g. schools), or geographical areas (e.g. local authorities).

4. Determine how many units (people, institutions, or areas) are required for robust results.

5. Assign each unit to one of the policy interventions, using a robust randomisation method.

6. Introduce the policy interventions to the assigned groups.

LEARN

7. Measure the results and determine the impact of the policy interventions.

ADAPT

8. Adapt your policy intervention to reflect your findings.

9. Return to Step 1 to continually improve your understanding of what works.

Also, check out this brilliant TED talk by Tim Harford on importance of Trail and Error.