Wage Subsidy Scheme Evaluation

15 January 2024

MartinJenkins was awarded the competitive contract to evaluate the development and implementation of the Wage Subsidy Scheme (the Scheme), an economically significant intervention of an uncommon scale developed during the Covid-19 response.

The evaluation of the Wage Subsidy was commissioned in late 2021 and was coordinated by MSD in partnership with IR, MBIE, and the Treasury through cross-agency working and steering groups.

The full Wage Subsidy evaluation has three parts:

a process evaluation by MartinJenkins (with support from Te Paetawhiti, ConnectEd, and OliverShaw),
an outcomes evaluation by Motu Research, and
a synthesis report based on findings from the first and second parts above.

A full copy of our comprehensive process evaluation is available on the MSD website. The other components of the evaluation are also fully publicly available, including detailed appendices that set out the method and key data.

The detailed process evaluation examined how the Scheme was developed and implemented using a methodology approved by a cross-agency working group and feedback from a separate ethics panel.

The process evaluation scope was limited. Topics covered in the Wage Subsidy management audit by the OAG were out of scope for the process evaluation, as were consideration of alternative responses to the Wage Subsidy and comparisons of findings with job retention schemes implemented internationally. Outcomes and the macroeconomic impact of the scheme were also out of scope.

In addition, the process evaluation was expected to incorporate Māori interests and the concepts of Te Pae Tata (MSD’s Māori strategy), and pay particular attention to accessing perspectives of Māori and Pacific stakeholders, stakeholders who can articulate the voice of workers (including all employment arrangements – permanent, contract, casual, full-time, and part-time) and vulnerable and low-income workers (for example, casual, part-time, women, Māori, Pacific peoples, migrants, and workers with a disability).

A multimethod approach was taken, drawing insights across five core data sources, including existing policy and operational documents, existing administrative data, 1-1 and small group interviews with employers, workers, and their representatives, online workshops, and online surveys.

2,550 stakeholders took part in the evaluation by completing online surveys, and a further 188 individuals took part in individual or small-group interviews.

Response of the Peer Review

An external peer review was also commissioned by the Ministry of Social Development (MSD) to comment on the preliminary draft report. This occurred early in the evaluation.

There have been some external media comments on the peer review following its release under the Official Information Act.

MartinJenkins agreed with MSD that the peer review commented on a number of elements related to the wage subsidy that were not relevant to the scope of our evaluation nor were usual for this type of early-stage peer review. MartinJenkins raised concerns with MSD about the process of commissioning and the quality of the peer review. MSD confirmed the points of the peer review that they considered were relevant to the evaluation, noting that large numbers of comments were deemed out of scope and irrelevant.

Our conclusion

Overall, we are proud of the work we undertook on the evaluation of the Wage Subsidy. It was a large and complex evaluation of economically significant Government interventions. We stand by our findings and consider them valuable to future policy makers.

Previous media comment to David Williams

Attributable to Sarah Baddeley, Partner

The detailed process evaluation examined how the Scheme was developed and implemented using a methodology approved by a cross-agency working group and feedback from a separate ethics panel. An external peer review was also engaged by the Ministry of Social Development (MSD) to comment on the preliminary draft report.

MartinJenkins was not involved in the scoping, commissioning, management, or oversight of the peer review. These questions should be directed to MSD.

What’s your view of the Chapple review which, objectively speaking, was extremely scathing? Did its contents come as a shock? Did Martin Jenkins believe the draft was of such low quality? Why was the draft report so much longer than expected?

Did MartinJenkins complain about the peer review? If so, on what basis? MSD also released to Newsroom a file note, dated August 4, headlined: Wage Subsidy Scheme Process Evaluation: Response to external peer review. Do you have any comment about that document?

Once we received the full peer review, MartinJenkins raised concerns about the process of commissioning and quality of the peer review. MSD confirmed the points of the peer review that they considered were relevant to the evaluation, noting that large numbers of comments were deemed out of scope and irrelevant.

Were significant changes made to the draft after the Chapple review?

The final report considered the feedback from the peer review that MSD considered relevant, alongside other feedback provided by the cross-agency working group. This was a straightforward process and was consistent with our usual methodology. The main changes we made were presentational, more clearly setting out our method, and did not go to the substantive findings.

Our intention was to prepare an independent evaluation that would be helpful to future policymakers; it was not an academic evaluation nor was it commissioned as such.

MartinJenkins met all major milestones provided to us and within the timeframes accepted by the cross-agency working group.

Has this quality assurance episode prompted Martin Jenkins to strengthen its internal quality assurance processes?

Review of quality assurance processes is good practice. Following receipt of this peer review, our assessment was that a rigorous quality process had been followed. We stand by this process and the relevance of the findings.

Throughout the assignment, we were very mindful of the significance and importance of this process evaluation, and we continue to stand by our process and findings.

“By welcoming robust review processes, we ensure agencies and the public can have confidence in the relevance and accuracy of the process evaluation findings,” said Sarah Baddeley 