When faced with uncertainty, we often look for predictions by experts: from election result forecasts, to the likely outcomes of medical treatment. In nature conservation, we turn to expert opinion to assess extinction risk, or predict the long-term responses of plant species to fire management.
But how reliable are these predictions? There’s a well known saying, “Prediction is difficult, particularly when it involves the future”.
In our new research, we put this to the test. We asked eight experienced ornithologists to predict how bird species respond when farmland is revegetated – a common conservation practice.
The result? There was a surprising amount of variation among experts. And there were consistent biases, such as favouring birds commonly seen on farms while underestimating small woodland species. However, when we combined their responses, we got better outcomes.
Does this mean we shouldn’t use such expertise? No. Expert knowledge has a vital role in conservation decisions.
But like anything, it has limitations we should recognise. We should treat expert knowledge as a guide, rather than a source of truth.
David Cook/Flickr, CC BY
How do you put experts to the test?
Expert knowledge is commonly used for making decisions in conservation, yet it is seldom tested.
We asked our expert ornithologists to predict which bird species would be found at 20 revegetation sites on farms in western Victoria, which were spread across an area of more than 1,400 square kilometres.
We gave each expert detailed information about the sites, including a map, the size of the revegetation plot and when it was planted, the number of tree species planted, and management at the site.
Our experts then had to make judgements about how likely specific bird species were to be detected there. This was based on a list of more than 100 species for each site, which had been recorded in the surrounding district.
Then we compared their predictions with data from bird surveys of the sites – undertaken by different experienced ornithologists – as well as a random selection of bird species.
New research shows planting trees and shrubs brings woodland birds back to farms, from superb fairy wrens to spotted pardalotes
What did we find? A lot of variation between our experts.
Across the eight ornithologists, the average number of species they considered likely to be detected at sites ranged from 15 to 45. The average recorded from bird surveys was 19.
The predicted composition of the bird community at each site also varied between experts. Some were closer in their predictions than others, but all differed significantly from the community of bird species actually observed.
Ed Dunens/Flickr, CC BY
We all have biases – and experts are not immune
You might wonder if there were similarities in what the experts got wrong. There were.
By and large, our experts overestimated how likely common farmland species – like the galah, eastern rosella, willie wagtail and magpie-lark – would be. They also overestimated the likelihood of larger species that can occur in open country with scattered trees, such as the laughing kookaburra and black-faced cuckoo-shrike.
Why might this be? These species are very visible and common in farm landscapes, but they also range widely and are hence less likely to occur at a particular site while it was being surveyed.
By contrast, our experts tended to underestimate the presence of small woodland birds, such as the brown thornbill, superb fairy-wren, silvereye and grey fantail.
When we combined the expertise of our eight ornithologists, we saw less variability. When grouped, our experts performed much better than a random selection of bird species. Even so, their predictions still differed strongly from those actually observed.
Why does this matter? The loss of woodland birds in farmland areas of southern Australia is a major conservation concern. Revegetation helps restore wooded habitats for such species. These biases could lead to conservation managers discounting the benefits of revegetation for conservation.
The task we gave our experts was not easy. To make reliable predictions is complex. For each species, they had to make multiple judgements. How common is the species in this region? Was the revegetated area likely to provide suitable habitat? How might this species be influenced by the age of the planting and the diversity of tree species? Would it be a resident species, or a visitor? Regular or irregular?
Some of the variation we found is likely due to differing levels of familiarity with the birds of western Victoria. Our experts also had different levels of experience in carrying out surveys and studying birds in revegetated habitats.
Shutterstock
What does this mean for our reliance on expertise?
Expert knowledge may be the best – or only – source of information for decision making when we need to rely on predictions, where knowledge has to be applied in novel circumstances, or where management involves complex interacting factors.
So how can we improve the accuracy and reliability of expert predictions, given we all have biases and gaps in our knowledge?
Carefully selecting experts based on relevant experience, using a structured protocol to draw out information, and combining knowledge from multiple experts can help.
Where possible, we should seize the opportunity to test expert predictions and identify potential biases. At present, this kind of testing is rare.
Good conservation management benefits from the wealth of knowledge held by experts, but it also depends on evidence from well-designed empirical studies.
Expert predictions are only as good as the data – and the experience – on which their judgements are based.
The limits of expert judgment: Lessons from social science forecasting during the pandemic
