Whole of government approaches to evaluation

 EeMun Chen, a Principal Consultant at MartinJenkins based in Tāmaki Makaurau, talks breakfast and evaluation, arguing that Aotearoa needs a more coordinated and centralised approach to evaluating the impact of different government initiatives.

About 20 years ago I heard someone describe New Zealand’s approach to public funding in this way:

• Picture Aotearoa New Zealand and its government agencies as a slice of toast. You could even think of Budget 2023, due to be delivered on 18 May, as that slice of toast.
• Spread Marmite — the funding — over that slice of toast. (I think you skip the butter in this metaphor.)
• Then chop up that slice of toast into lots of little bits — think of each one as an agency or, if we chop smaller, as a particular programme or policy.

The metaphor creator was concerned about a general lack of connectedness between the different bits in the funding picture in Aotearoa, as well as the funding being sparse.

And then there’s the tasting …
This toast-and-Marmite image has stuck in my mind ever since. I think though that we can add another stage to that metaphor:

• To assess how the Marmite spreading went, hand each one of those little bits to a different taster.

That’s how we usually do evaluation in Aotearoa: it’s at the little-bit-of-toast level — that is, usually of a single programme or policy — and only rarely cross-toast. This means no-one has a good overview of what the Marmite is doing as a whole across all those different little bits of toast.

So there’s no-one who can see that, say, the Marmite is spread well and tastes right in the centre of the toast but that out in the margins, at the crust, the Marmite tends to be light and patchy. There’s no-one who can report back that, perhaps, the Marmite tastes differently to different groups of people.

Evaluations help us understand whether policies and programmes are achieving their aims
To address this we need a joined-up, whole-of-government approach to evaluation, to understanding what works and what doesn’t. This is at the heart of having effective programmes that improve New Zealanders’ lives and also provide good value for money.

The UK Treasury gives this useful explanation of evaluation and its purpose:

“Evaluation is a systematic assessment of the design, implementation and outcomes of an intervention. It involves understanding how an intervention is being, or has been, implemented and what effects it has, for whom and why. It identifies what can be improved and estimates its overall impacts and cost-effectiveness.” ("Magenta book — Central government guidance on evaluation," 2020, page 5)

Well-said, but here in Aotearoa we’ve got two key problems — one’s an expertise problem, the other a planning and prioritising problem.

There’s not a single body of evaluation expertise in our government about what works and what doesn’t
In Aotearoa, most agencies have their own monitoring and evaluation functions, and they do evaluations through a combination of in-house and out-sourced work. But this means evaluation expertise and capacity is dispersed throughout many different government organisations (and their private-sector advisers, like MartinJenkins). Out-sourcing the task can also lead to a loss of important institutional knowledge and experience.

There’s no single body of knowledge about what works and what doesn’t when it comes to applying the Marmite/funding. For example, should you add avocado or cheese with the Marmite? And if you’re going to add cheese, what type would make the toast tastier? And should the cheese be melted? We need a central record of lessons from past interventions, lessons that can then be applied, with appropriate modifications, to other programmes and initiatives in the same or different sectors.

For example, a 2009 evaluation of a range of services and programmes delivered by New Zealand Trade and Enterprise led to the winding-up of free management capability training. The free training scheme was replaced with what is now the Regional Business Partner network, which subsidises up to 50% of the cost of management training. The move to subsidisation was based on the evaluation’s findings that:

• businesses tended to value the training more if they had to pay for it
• subsidisation would reduce the number of no-shows
• subsidisation would allow businesses to choose the type of courses they needed and to choose the providers, and
• it would also improve the supply and quality of private training providers.

I’m concerned that these lessons may have been lost in the intervening years. There needs to be a whole-of-government approach to amassing and preserving all that specialist expertise and knowledge.

There’s no clear oversight function that prioritises, plans, and resources evaluations
Because we have no central evaluation function overseeing evaluations of government funding, there’s no-one identifying opportunities to run concurrent or collaborative evaluations.

Evaluations are rarely done across policy portfolios and across agencies (cross-toast). And there are few opportunities for formal collaboration and cross-agency evaluative activity. A more joined-up approach would, among other things, help ensure that particular groups of people are not approached for the same or similar data repeatedly.

We see some attempts at joint-agency evaluations, but often with these each agency focuses on “their” bit or bits of toast, with no-one taking a wider view.

Making moves in Aotearoa
There have been murmurings across the New Zealand public service of the need for a more integrated approach to evaluation.

For example, the 2022 conference of the Aotearoa New Zealand Evaluation Association (ANZEA) included it as a sub-theme. David Stuart, from Manatū Taonga | Ministry for Culture and Heritage, presented on “evaluation as a public service”, and called attention to the absence of a centralised support or standard function. By contrast, other professions have sector leads in the Public Service Commission.

The Auditor-General and their Deputy have also lamented the lack of public accountability of central government, and the fact that a law change may be needed to get a better picture of how public money is being spent and what it’s achieving.

The New Zealand Productivity Commission’s interim report on economic inclusion and social mobility, “A fair chance for all”, calls for a systemic shift in our approach to monitoring and evaluation, noting that “there is currently no system leadership for what should be core public sector business”. It describes this shift as “essential to break the cycle of persistent disadvantage, by building evidence for what does and does not work.”

The Productivity Commission’s report recommends system-level leadership of evaluation, to counter short-term tendencies and to provide guidance so that evaluation approaches are both effective and cost-effective. This, the Commission says, would improve the quality and effectiveness of policies, programmes, and services.

Learning from Australia: A review at the federal level
In December 2019, in the remaining days before COVID-19 hit, the Australian Government released an independent review of the Australian Public Service (APS). The review proposed establishing a central enabling evaluation function to drive a common approach to evaluation across the public service and to uphold minimum standards of evaluation.

The review envisaged that the changes would strengthen the basis for government decisions, and also help provide explanations when activities stop or change and new strategies and policies are pursued.

Under the proposal, the main responsibility for evaluations would stay with individual agencies, but the central function would provide guidance and support for agencies on best practice. It’s an approach supported by the Australian Evaluation Society, which referred to it as a “hub and spoke model”, with a whole-of-government centralised function collaborating with evaluation functions in each agency.

The federal government seems to now be implementing that approach. In late April 2023, it announced that a centralised evaluation unit will be established within the Australian Treasury — to the tune of $10 million over four years. The new unit will apparently be dedicated to partnering with agencies to evaluate the effectiveness of programmes and to strengthening the evaluation capacity of the Australian Public Service.

Learning from the Australian states: What does a whole-of-government evaluation model look like?
There’s also much we can learn from the efforts of individual Australian states to take a whole-of-government approach to evaluation.

In 2021 I attended an Australian Evaluation Society panel discussion on whole-of-government approaches, with representatives from five of Australia’s states and territories. I came away from that session — well before the new federal unit was announced — with the overall sense that the Australians were far ahead of New Zealand in their approach to evaluation at the central-government level.

Things certainly weren’t perfect, but it was good to hear the conversation, over Zoom, on what a whole-of-government approach could achieve — and what that could mean for Aotearoa.

We look a lot like Victoria’s mostly decentralised approach
In New Zealand, we have no centralised evaluation function — only some general guidance, and an evaluation section to fill out in the Budget template. In this our approach is most similar to Victoria’s.

In Victoria, evaluation functions sit in all state government departments, with no central function, and only some limited guidance from the centre — for example, the Victorian Department of Treasury and Finances sets out its evaluation expectations as well as some evaluation requirements for when programmes lapse.

By contrast, all the other Australian states and territories represented at that 2021 Zoom panel discussion have gone well beyond Victoria and Aotearoa and are operating centralised hub-and-spoke models.

For example, New South Wales has a Centre for Evidence & Evaluation, which sits in the NSW Treasury. Its role is to develop an “evidence bank” that state agencies can draw on for decision-making, to assess the evaluations that agencies do, to develop standards and guides for state agencies to follow in their evaluations (that is, the NSW Evaluation Policy and Guidelines), and to work with each agency on how to apply those standards.

In the Northern Territory, a three-person Program Evaluation Unit is part of the Budget Development Team in the Department of Treasury and Finance. Responsibility for evaluation and commissioning evaluation sits within the different agencies, but the Program Evaluation Unit coordinates, supports, and encourages the use of evaluation across government. The Unit has developed a Program Evaluation Framework, which also includes a repository of completed evaluations.

Designing a central evaluation function
In research done for the review of the Australian public service, the Australia and New Zealand School of Government (ANZSOG) contrasts the Australian approach to evaluation with practices in the Netherlands, the US, Canada, and the UK. ANZSOG concluded that no one approach appears to work best.

A centralised function across departments provides momentum, focus, and commitment for evaluation activity. But there are unanswered questions as to what the role of the function should be, as well as where in the public service it should be placed and in what kind of organisation.

The next section of this article poses those questions and looks at more international examples.

What precisely should be the role of a central evaluation function?
Should the central function provide leadership, support, or oversight, or set formal requirements? Should it establish priorities for evaluation? And should its role include doing evaluations itself?

In Australia, at the federal level, the Department of Finance has been responsible for the Commonwealth Evaluation Policy. Taking effect from 1 December 2021, the policy provides a principles-based approach for evaluations across the Commonwealth.

The policy is intended to improve the way agencies assess implementation, measure the impact of government programmes and activities, and frame policy decisions about new or revised programmes. The policy is supported by a range of guidance and an Evaluation Toolkit. The policy does not create any new mandatory requirements in relation to evaluation.

The new Treasury evaluation unit announced by the Australian federal government last month will be charged with evaluating programmes in partnership with departments and agencies. It’s expected that this will guide more effective spending, and help build evaluation capabilities.

In Canada, the Results Division of the Treasury Board of Canada Secretariat is responsible for providing leadership for performance measurement and evaluation functions. The Canadian central function appears to take more of a “must do” approach than the Australian example, however, with the Policy on Results issued by the Treasury Board imposing a range of formal requirements on governmental agencies.

The Treasury Board of Canada Secretariat also appears to play a more hands-on role, carrying out and overseeing resource alignment reviews and centrally led evaluations of departmental spending. It’s also responsible for making reports of centrally led evaluations publicly available, for raising any compliance issues that arise, and for working with departments to ensure that the data collection and reporting process is efficient.

Where should the function sit, and in what kind of organisation?
What type of organisation or part-organisation should the central function be, given its role, and where should it sit?

Should it be placed within a central agency, and if so which one? Or should it be placed within an independent or statutory body? Should it be established as a new body?

If a new body, should it be a government department, an autonomous Crown entity, an independent Crown entity, a Crown agent, or something else?

In Australia, the Treasury, the Department of Finance, and the Department of Prime Minister and Cabinet were all identified as candidates, along with an independent “Evaluator General”. It appears they have now settled on the Treasury.

In the UK, the What Works Network provides independent, evidence-based, practical advice. There are currently 10 independent What Works Centres covering policy areas that account for more than £250 billion of public spending.

The What Works Centres ensure that robust evidence shapes decision-making by collating existing evidence on the effectiveness of programmes and practices, by producing synthesis reports and systematic reviews in areas where they do not currently exist, and by assessing the effectiveness of policies and practices against an agreed set of outcomes.

The What Works Centres are funded by a combination of government and non-government sources, including the National Lottery Community Fund. The Evaluation Task Force, a joint Cabinet Office–HM Treasury unit, is the secretariat of the What Works Network.

In the United States, the Foundations for Evidence-Based Policymaking Act 2018 — also just called “the Evidence Act” — established processes for the federal government to modernise its data-management practices, evidence-building functions, and statistical efficiency.

The Evidence Act requires agencies to submit annually to Congress and the Office of Management and Budget (OMB) a systematic plan for identifying and addressing policy questions. The OMB issues guidance for the evaluation of programmes and is responsible for identifying best practices for evaluation. Agencies are required to implement the OMB’s guidance.

Each agency’s strategic plan must include “an assessment of the coverage, quality, methods, effectiveness, and independence of the statistics, evaluation, research, and analysis efforts of the agency”. The US Government Accountability Office (GAO) must submit a report to Congress, summarising agencies’ findings and highlighting trends in their assessments, and recommending actions to further improve agencies’ capacity to use evaluation techniques and data to support evaluation efforts.

New Zealand needs to follow the Australian hub-and-spoke model
As I said at the beginning, I fear Aotearoa is getting left behind in this area — we’re all eating our little bits of toast, and don’t really know whether our Marmite could be spread and used in better ways.

With limited government resources and skills, it appears that New Zealand could benefit from a whole-of-government approach to evaluation. There needs to be a reaffirmed commitment to improving our evaluative evidence base, in the form of a centralised hub-and-spoke evaluation function. Then we might be able to understand things like how successful Budget 2023 is, or not.

With that evaluation infrastructure in place, we could then work on how best to achieve our evaluation aspirations, within the larger frame of working effectively to achieve government priorities for the people and communities of Aotearoa.

Disclaimer: No pieces of toast or jars of Marmite were harmed in writing this article. Personal toast preference is with butter, a thin and patchy spread of Vegemite, and melted cheese.