Massey University scientist Mike Joy has been in the media recently claiming that the Ministry for the Environment has been misleading the public over water quality data. Is he right?
We looked at this issue as part of the MyRiver project. What piqued our interest were the differences between the picture of water quality put forward by the Ministry for the Environment (MFE) and the National Institute for Water and Atmospheric Research (NIWA).
NIWA data demonstrates a decline in water quality over the past twenty years, particularly in areas in pasture. More recently MFE painted a far rosier picture with the claim “Four of our parameters show stable or improving trends in 90% of sites.” Why the difference? There are a few reasons which we will explore below. But first, why should we care?
Why this is important
The MFE findings have been faithfully parrotted by farming advocacy organisations such as Federated Farmers to show that water quality is on the mend, so we don’t have any reason to worry or regulate further. To make an informed decision about what should happen on water quality, the public needs to be fully informed, which is the job of official data. So this may seem like a statistical quibble but is actually really important.
What is ‘Stable’?
There is a technical definition for ‘stable’ in statistics, but this wasn’t what was used in the MFE data. Instead they used the term stable where there was no statistical trend in the data.
To those with a statistics degree this is obviously wrong, but everyone else probably just feels their eyes glazing over. MFE argue that they used term ‘stable’ in the non technical summary, as they felt a layperson would understand a lack of evidence as being the same as stable. Mike Joy thinks people think of ‘stable’ as we use it in the emergency ward sense – their condition isn’t getting better or worse. Let’s try using an analogy so you can decide that for yourself.
Imagine you are trying to measure whether the sea level is rising. For some reason you decide to do your test by going out on a boat in the Southern Ocean where the average wave height is 5 metres. After throwing up a few times, you collect your data and sail home. Obviously the data you collected was hugely variable, so you wouldn’t be able to tell if the sea level was rising or not. Would you come back saying the sea level was ‘stable’? Or would you say ‘I have no idea, please pass me a towel’?
Another important part of this is the time period used. The NIWA analysis is over a longer time period, which gives them more data, which means they have a better idea of the underlying trends. In short, this makes their results more robust. The MFE data was over a shorter time period, so it is natural that there would be more rivers without evidence of a trend.
Did MFE use a shorter time period in order to define more rivers as ‘stable’ and make water quality data look better? Who knows. MFE would probably counter that they feel the tide truly has turned on water quality, and therefore they wanted to use more recent data to show that point. Back to our analogy:
While going out on the Southern Ocean to measure sea level rise is a dumb idea, if you stay out there long enough you would eventually either drown or gather enough data to pick up a trend. Time periods can make a difference.
So what is happening to water quality?
NIWA analysis shows pretty clearly that over the long term water quality has been deteriorating. They also show that water quality declines in an area as more land is put into pasture, particularly more intensive agriculture like dairy.
MFE analyses indicate that these trends may have recently reversed or stabilised for particular indicators of water quality (phosphorus, bacteria and ammonia). For rivers where they do have a trend (which are the minority), more are improving than are declining, particularly phosphorus. This may be a result of improved fencing and riparian planting, as well as better effluent management. Or, as Mike Joy argues, it may just be because we have had fewer floods over that time period, which washes soil and the phosphorus it contains into rivers.
So really the best that MFE or Federated Farmers should be claiming is that some indicators of water quality may be showing early signs of improvement. That does not mean that water quality is improving, far from it. Even according to MFE results the best indicator we have of life in a river – the Macroinvertebrate Index, is still deteriorating.
Lies, damn lies, or statistics? You decide.