ABC Brisbane Mornings with Ellen Fanning
SUBJECTS: Randomised policy trials and the Australian Centre for Evaluation.
ELLEN FANNING, HOST: Andrew Leigh is the Assistant Treasurer, a former professor of economics, no less, and earlier this week, he gave a stirring speech calling for a bit of evidence before we, well, let’s say, threaten parents with the suspension of welfare payments if they don’t get their kids to school. Or we make our surly teenagers sit through healthy living lectures. And why? Because although these programs seem like, well, they would work, Andrew Leigh reckons they probably don’t. He joins me now. Dr Leigh, thank you so much for joining us.
ASSISTANT MINISTER ANDREW LEIGH: Real pleasure, Ellen, thanks for having me on.
FANNING: Yes, I have enjoyed watching your passions over the years for research and they are undimmed in office. I love the start of this speech where you sort of list four things that would seem to be, as the Americans would say, self-evident. Cut the welfare if they won’t send the kids to school. Literacy tests always are a fair representation of student ability. And let’s just stick to one more if you lecture the adolescents about risky behaviour, they’ll undertake less risky behaviour. That’s all obviously true, isn’t it, Andrew?
LEIGH: You would have thought so, wouldn’t you? And then you go and run the randomised trials and it turns out that none of these things are true.
FANNING: None of them.
LEIGH: Randomised trials of taking away welfare if kids don’t go to school showed that it didn’t have any impact on attendance. A test which looked at the fairness of literacy tests found that if you had examples that were closer to home for the students, then they did better.
So, in other words, literacy tests weren’t objective. And a carefully co-designed program to improve healthy living for teenagers and schools, including in Brisbane, in a randomised trial, was found to have no impact. So, it really is an illustration that gut feel doesn’t take us all the way. We need to take a page from the medical book and rigorously evaluate programs.
That really wasn’t done under the former government. We’ve had a study from CEDA that looked at a number of programs, with total expenditure over $200 billion, and found that 95 per cent of the time, the programs weren’t properly evaluated. So, we set up the Australian Centre for Evaluation in Treasury to try and change that.
FANNING: Ok, I just want to dig into that Healthy for Life program. A lot of listeners will know that it’s rolled out across Brisbane, Perth, Sydney and around the country, is it?
LEIGH: The study was in Brisbane, Perth and Sydney. The program was carefully co-designed with students and educators, but it turned out that the schools that had it didn’t see any improvement in outcomes such as alcohol misuse, tobacco smoking, recreational screen time, poor diet or poor sleep. So, it seemed to be a carefully designed study. But the impact on the ground wasn’t what the researchers had hoped.
FANNING: And, you know, we could ask that, well ask this question about a range of other programs. You wonder about the education around consent to sexual activity that we’ve been rolling out for years and years and years, but we still see troubling attitudes among young men and women who have undertaken these programs. So, how is it that you know how government works? You’re in it. How is this possible?
LEIGH: Well, the infamous milkshake ads really did come from an absence of good evidence. When we brought down our new consent ads, that have been released in the last week, they’re grounded in a much stronger evidence base. We need to do more randomised trials, better evaluate programs, and then do a better job of disseminating those findings. Getting evidence implemented in a way that lines up with what we know works. In the UK, they’ve got ‘What Works Centres’. In Australia, we’ve set up the Australian Centre for Evaluation. And all of it is about being more scientific and critical as we seek to solve some of these major challenges.
FANNING: And I’m interested as to why this isn’t routine. You make an interesting point about medicine. I mean, people keel over if you get it, if you don’t establish what works and you go on the vibe. But education seems to be particularly blighted around the vibe. You’re saying here that observational study, just having a look and seeing what you reckon is how we decide how to educate our children, not randomised, proper, rigorous trials?
LEIGH: Not always. I mean, sometimes we do have good evidence. And indeed, the study that I talked about from Dubbo is a good example of a great educational randomised trial that helped improve the quality of how we deliver education and how we run student tests.
FANNING: And that’s probably worth speaking to, right? So you’re running a trial to work out the literacy standards of kids and they’re in Dubbo. And they said, okay, well, we’re asking kids to read a passage about lighthouses. But they’re in Dubbo, and if we ask them to read a passage about the big dish in nearby parks, they might be more engaged with that content. And what it actually found, according to your speech, is a substantial improvement in their literacy levels, amounting to halving the gap between Indigenous and non-indigenous students. Give them something interesting to read.
LEIGH: Perfectly encapsulated. And it halves the gap between Indigenous and non-Indigenous students, in part because the impact is so much bigger for Indigenous students. So, having literacy examples that are close to home means that the test is a better reflection of the literacy skills of the student.
This is a really important insight in terms of how we test. And the Education Endowment Foundation in the UK, and the Australian Education Research Organisation here, are both organisations which are aimed at doing more rigorous evaluations. Not everything is going to be a randomised trial, but we do know from medicine the value of randomised trials in saving lives.
You know, you look at something like hydroxychloroquine for COVID. It looked good in the observational studies. The randomised trials showed that it was actually doing harm and ultimately hydroxychloroquine for COVID cost over 16,000 lives. So there really is a cost to making mistakes in medicine, as there is in social policy.
FANNING: I’m speaking to Andrew Leigh, the Assistant Minister for Competition Charities and Treasury, the Assistant Minister for Employment as well, and a former economics professor.
I wonder whether or not people listening to this, maybe educators, early childhood educators, say, yeah, yeah, yeah, you’re an academic and you want to go to all this bother and do all these randomised trials when it’s self-evident. I know what works. I know what works here, just get out of the way and let me do it.
LEIGH: Well, the programs that we started off with were all programs that had a good feel about them. They’re all programs which had strong advocates who thought that they would work, and yet the randomised evidence showed that they didn’t.
The Health for Life Program, carefully co-designed with students and educators, turned out not to have the impact they expected. Another one example I saw recently, the UK had a Social Workers in Schools program: supported by social workers, loved by educators, loved by students. Just before they did a multi-billion pound rollout, they did a randomised trial. It turned out that social workers in schools had no measurable impact on student outcomes. So, just because a program feels good doesn’t mean it’s actually doing good on the ground. And that’s where our evidence-based approach, through the Australian Centre for Evaluation, really is a game changer. More efficiently using public dollars and more effectively closing the gaps.
FANNING: Okay, so a randomised trial is you have a control group of X number of people and you have a group where you’re applying the healthy living program or indeed the medicine or whatever, and you compare the two. That’s what rigorous research is, right?
LEIGH: That’s right. And that’s almost a requirement of getting a new drug on the pharmaceutical benefits register these days, because you’ve got to put it through a randomised trial. And yet, in the area of policy, we’re doing very few. CEDA estimates that only less than 2% of all evaluations nationwide are randomised. So, there’s opportunities not to do everything through a randomised trial, but to use more randomised trials than we are right now.
FANNING: Yeah. Okay. One final comment very quickly, if you can. What if politicians prefer the vibe? What if public servants don’t speak up? And what if ideas are so entrenched that the public don’t care that they don’t work? They just like the sound of them? Quickly.
LEIGH: I don’t know anybody who got into public life who doesn’t say that the main reason was to make a difference. You can better make a difference if you better evaluate your policies that’ll help the community. It’ll help public servants have pride in what they do and to help politicians know that they really are making a difference.
FANNING: Yeah. Good on you. Lovely to talk to you, Andrew Leigh.
LEIGH: Thanks so much.