Employment Policy Built on Quality Evidence or Reckons?

Jess Berentson-ShawTax and Welfare4 Comments

Recently the Prime Minister Bill English followed in the tradition of the previous Prime Minster John Key and suggested the reason we had a skills shortage was because young people could not pass drugs tests .

He said this was based on “what employers tell him”, or as we call it in the evidence trade – anecdote. The problem with anecdote is that it cannot be trusted to provide reliable information about an issue, which is especially problematic if one is going to spend taxpayer dollars on the basis of it.

Later, one work based drug-testing agency reported that the rate of their positive tests had remained steady at 5% per year (pretty low to be claiming this is a key driver of skills shortage). However, this is not the full picture. The 5% figure is only the rate of positive tests (about 7,200) in a selective sample (those 144,000 test referrals that were made). What we don’t know is the rate of positive tests in the whole of the eligible population – everyone in New Zealand who is seeking work. Given, also that the drug testing company only reported the number of tests referrals made and coming back as positive, not the number of individuals referred and testing as positive, the rate of individual drug use among jobseekers is unknown – it could be a small group of people being repeatedly tested and failing.

The point is none of the information discussed is what could be described as a full and complete understanding of the evidence about drug use in the labour market.

Governance based on anecdote and bad science is likely to fail

Here is something we prepared earlier about how decision-making should ideally work if we want cost effective solutions that lead to positive outcomes for New Zealand and New Zealanders.

Anecdote does not have a place in making good decisions, experience has a role once the science and evidence has been established, but mixing anecdote and science up is very problematic. The environment of decision-making also has a place – in other words politics is always the art of the possible, but again at the heart of this needs to be good science. You can read more on good science over on The Spinoff.

Our Prime Minster Loves Data so why the Anecdotes?

Prime Minister English has talked very consistently about using good quality, comprehensive, easy to access data to determine where money in Government should be targeted for best return (note this is quite different from using evidence to figure out what actually works). Here he is talking about the integrated data infrastructure – or as he calls it the “data supermarket” to improve Government policy. Using anecdote does not seem in line with the value the Prime Minister appears to place on good data.

And which do taxpayers, and those who create policy, believe good policy should be based on? Good data or reckons? One has a very high chance of wasting money on ineffective approaches – something the Prime Minister has rightly expressed concern about. 

What is the real story about drug use and unemployment here?

What do we all value when we are talking about young people and work? As a society we want to see young people gainfully employed in satisfying well-paid jobs. The Government wants them to be plugged in to the economy and adding value – and hopefully it wants them to be doing well. Employers want enough good workers they can rely on, and employees want good conditions and to be treated well. It is all pretty straightforward stuff. So how do we get that? First we have to figure out why these things are not happening for all people.

What the anecdote signals is that the problem is the young people. This is not unfamiliar territory – moralising about those on lower incomes has had a good flogging for the last few decades. The problem is it does not get us anywhere near the solutions that we all think are important. If we want young people to be gainfully employed, and a country filled with people with the right skills, then we need to go deeper than just implying young people are a bit useless; it does the Prime Minster and New Zealanders a disservice.

There is no evidence those on jobseekers can’t get work due to drug use

When we look at the reliable numbers we do have, the unemployed are not failing drugs tests in any numbers of note. For example between 2014 and 2015, there were 29, 049 referrals by WINZ to jobs that required a drug test (this is not individuals but number of referrals) and there were 159 failures or refusal to be tested (which is not confirmation of drug use). That means of those referrals to a job with a drugs test there was a fail (or fail to take the test) rate of 0.5%. Again this is not individuals.

We can also look at ALL jobseekers, in December 2015 there were 67, 670 people who were “work ready” and receiving jobseekers support. The number varies a bit based on season, and if we assume (a big assumption here) that each test referral represented a unique individual then 0.6% of all jobseekers fail or refuse a drug test. We still need better data, but this has the distinct whiff of a moral panic about it.

Perhaps the Prime Minster spoke to all the employers involved in these tests and got the wrong impression it was a tsunami of people too drugged up to work out there rather than a small drip? That is the thing about anecdote it skews your view of the situation. It is the reason we don’t let medical practitioners decide what medical interventions work best based on their experience with their patients: sampling errors and only seeing the evidence for what you already think happens.

There is also no good evidence those on low incomes use harmful substances more than anyone else

There is some evidence that wealthy families and individuals engage in some harmful behaviors at a greater rate than those on lower incomes. Consumption and health data indicates that alcohol use is higher in better off communities. Ministry of Health alcohol-use surveys have also found there are more non-drinkers in communities where poverty is higher. Tobacco use is however, higher in poorer communities.

There is no strong evidence to support the idea that drug abuse or dependence is more prevalent in communities that have more unemployment or poverty. Men living in poorer communities are more likely to report recreational drug use than those in wealthier communities; women however are not. Notably, those in the wealthiest neighbourhoods are more likely to report receiving help for drug use. (This could indicate that problematic drug use is higher in the wealthier communities, or that they have better access to help for it or both.)

Despite the era of ‘alternative facts’, it is important that our politicians are held up to a certain standard of evidence, even if that evidence jars with what we might believe. Because ultimately it is this standard that can give us confidence that policies will get us what we value – in this case positive employment options for our youth. Bad data leads to poor policy. The Prime Minster himself has highlighted before how inappropriate it is to force ineffective interventions on ‘high-risk’ individuals and communities.

There is evidence a failure to invest in children when they are young has limiting impact on skill acquisition

Why do youth not engage in employment? Well business people have suggested some reasons, including a mismatch between expectations and the reality of the job market.

Other analysis can tell us about the changing nature of work, the role of casualised labour, insufficient training, lack of investment in productive sectors as we invest in housing instead, and poor planning with regard to the demand cycle of the building industry. But lets talk about the young people themselves, if we want to focus on employee issues.

What does the research tell us about those young people?

It tells us that if young people are not invested in as babies (yes it goes back as far as this when our brains are in the critical phases of development) then they are less prepared by the time they attend school, it is harder for them to do well at school, to acquire the skills they might need, and go on to engage in the job market in the way that is ideal. You can read all about the positive impact of investing in all families and babies in our research “Pennies from heaven”.

So before we use anecdote or poor data to blame these young people for a skills shortage in a complex employment environment, we should consider what we want for these young people and the country and whether using bad data will really get us to that outcome.

Employment Policy Built on Quality Evidence or Reckons? was last modified: March 1st, 2017 by Jess Berentson-Shaw
About the Author
Jess Berentson-Shaw

Jess Berentson-Shaw

Dr Jess Berentson-Shaw is a science researcher working for the Morgan Foundation. Jess holds a PhD in Health Psychology from Victoria University.Jess has over 10 years’ experience working on applying science and evidence to public policy. She worked on improving the use of science in public health practice in NZ, before working as a Research Fellow at University College in London, where she researched how doctors and clinicians translate scientific evidence into their clinical practice. While in the UK she also developed a national data collection system, which was used to determine what factors contribute to poor outcomes for women and babies during pregnancy and birth. On her return to New Zealand she directed a research group that specialised in the independent evaluation and application of research and science to health policy and practice. Jess loves science and what it can do to make the world a fairer place.

  • kiwithinking

    Finally a sensible article about the issue, RNZ did a good job pointing out the benefit testing results but your information is even better.

  • Pietrad

    When the same drug testing requirements are aimed at ALL politicians and members of local bodies, then justice may be served. Until then it will be used as a ‘stick’ to control the poor and further condition them to accept intrusions in their lives by ‘Authorities’.

  • John Pearce

    A great review of the situation. Sadly, politicians seek evidence, or anecdote, to support their policies, not evidence to use to evolve the best policies. Their policies are evolved on “public opinion surveys” not public benefit evidence. FInd me a party capable of achieving power which will advocate policies beneficial to the nation, and inimical to getting itself elected. This is the conflict of interest at the heart of party politics. MP’s pray “Almighty God, humbly …., and laying aside all private and personal interests, we beseech Thee to grant that we may conduct the affairs of this House and of our country to…, the maintenance of …. and justice,… and the public welfare, peace, and tranquillity of New Zealand, …. Amen” But the choosing of policy is not really focussed on public welfare, but on party welfare and personal interest in winning a seat. Sp please encourage more positive analysis of evidence denying policy positions.

  • Brenda Pilott

    Interesting article. You might also want to keep an eye on the debate about MSD seeking individual client data from NGOs. It shows some of the real-life difficulties about collecting and sharing data and about how the role administrative data plays (or doesn’t play) in building up an evidence picture.