Random trials of policy are just what doctor ordered
One of the Australian Government’s announcements this year has been the creation of the Australian Centre for Evaluation. The Centre will conduct rigorous evaluations, including randomised trials, to figure out whether policies are effective.
But why do we need randomised trials? To answer this, it helps to turn to medicine, a field that has been transformed by randomised trials.
A major reason why medicine saves more lives today than it did in the late-19th century is that theories are put to the test using trials.
The real story of medicine today is one of good evidence driving out bad theories.
To see this, it’s worth taking a moment to tell the story of the radical mastectomy.
In the 1880s, US surgeon William Halsted formed the view that breast cancer was most effectively treated by excising large portions of the patient’s tissue. Previous operations, he argued, had been too timid. Observing that patients often relapsed after surgery that removed only the tumour, Halsted advocated removing considerable amounts of surrounding tissue.
Halsted’s surgery removed the pectoralis major, the muscle that moves the shoulder and hand. He called it the ‘radical mastectomy’, drawing on the Latin meaning of radical to mean ‘root’.
In The Emperor of All Maladies, Siddhartha Mukherjee describes how Halsted and his students took the procedure further and further. They began to cut into the chest. Through the collarbone. Into the neck. Some removed ribs. They sought out the lymph nodes and claimed they had ‘cleaned out’ the cancer.
Women who endured these operations were left permanently disfigured, often with gaping holes in their chests. Recovery could take years. But Halsted was unrepentant, referring to less aggressive surgery as “mistaken kindness”.
Yet whether a patient survived breast cancer depended not on how much tissue was removed but whether the cancer had metastasised and spread through her body. If it had not metastasised, a more precise operation to remove the cancer would have been just as effective.
If it had metastasized, then a radical mastectomy would still fail to remove it.
In 1967, researchers at the University of Pittsburgh who were sceptical of the radical mastectomy decided to put these claims to the test. They began recruiting for patients to take place in a clinical trial that would test the impact of the radical mastectomy by comparing the surgery against a more moderate alternative, the lumpectomy, that involved removing only the cancerous tissue.
The eventual randomised trial covered 1765 patients. When the results were finally published in 1981, they showed that there were no differences in mortality between the groups. The women who had undergone radical mastectomies had suffered considerably from the surgery – yet they had not benefited in terms of survival. The randomised trial changed how surgeons treat breast cancer but it took a century. Between the 1880s and the 1980s, around half a million women underwent radical mastectomies, an unnecessary surgical treatment.
Randomised trials are valuable in instances where experts have strong views. In the case of breast cancer treatment, it took data to overturn decades of ideology.
An advantage of randomised trials is that they identify a clear counterfactual – what would have happened without the intervention.
At the Australian Centre for Evaluation, we won’t only conduct randomised trials. But they will be an important component of the work of the centre.
Its work will be conducted within a careful ethical framework, ensuring that we are as rigorous about issues of ethics as about issues of causality. It will cooperate across government, looking to implement low-cost randomised trials where possible.
Getting evaluation right will save taxpayers money and produce better policies. Our government is keen to avoid public policy making the same mistake that medicine made with the radical mastectomy.
And we want to provide an opportunity for testing innovative new policies.
Originally published in the Daily Telegraph on Friday the 3rd of November 2023.