Effective Programme Evaluation

About

This seminar offers an introduction to programme evaluation techniques and how they can be used to generate quality information about programme performance and its enhancement.  Drawing on state-of-the-art methods for evaluation research, the seminar gives participants the knowledge and tools needed to effectively commission and conduct programme evaluations.

Strategies for programme evaluation can run from the simple and straight-forward to the more sophisticated, and the strategies used in any given instance will depend on time constraints and resource considerations.  As well as introducing participants to the fundamentals of programme evaluation, the seminar recognises the necessity for pragmatism.  Integral to the seminar is an understanding of the constraints that many public sector managers and analysts work under, and how those constraints limit the scale of the studies that can be undertaken.

Key issues for discussion in this seminar include:
By the end of this course you will be able to:
• Understand the role of evaluation in the contemporary public sector
• Understand the needs of the evaluation sponsor
• Establish the focus, scope, and purpose of an evaluation
• Develop a suitable and manageable research design
• Implement and manage an evaluation
• Organise interviews, surveys, and other data collection methods
• Manage and combine primary and secondary data
• Be familiar with a range of quantitative and qualitative methods for data analysis 
• Build organisational learning into the evaluation process
• Understand the uses of evaluation results

Who should attend?
All those involved in planning for, and producing programme evaluations at a local and central government level, including public sector managers involved in:

• Managing government programmes
• Monitoring, assessing, and reviewing programmes
• Implementing policies, projects, and programmes
• Policy development and analysis

Training methodology
This two day master class will utilise adult learning methods, small group work and case studies with interactive learning exercises. All attendees will be provided with a workbook and a certificate of completion.

Outline

Day One
Scoping and Planning Programme Evaluations

Programme evaluation: An overview
An introduction to modern programme evaluation, the basics of evaluation research, and how evaluation can support policy development and organisational learning.
• Definition of evaluation
• The place of evaluation in the policy process
• Outcome and process evaluation
• Evaluation and other research initiatives
• Linking evaluation to evidence-based policy development, best practice interventions, creating organisational report cards, and organisational learning

Understanding a programme
A review of strategies for building knowledge about a programme. Evaluators must balance the quest for objectivity with the need to gain intimate knowledge of why a programme was established, how stakeholders make sense of the programme, how the programme works, and what it delivers for the target population.
• Characterising the programme
• Getting to know the programme
• Identifying programme goals
• Programme theory and implementation theory
• Building a programme theory
• Comparing programme theory to actual developments

Identifying issues and formulating questions
Evaluators must be judicious in determining the issues and questions that will become a focus of their research.
• Assessing the need for a programme
• Good evaluation questions
• Devising specific questions for the evaluation
• Prioritising questions to be explored

Assessing and monitoring the programme process
Often, evaluators wish to determine how effectively a programme operates, by focusing attention on the processes associated with that programme. Issues in the assessment of programme process are reviewed.
• Setting criteria for judging programme process
• Common forms of programme process evaluation
• Assessing service utilisation
• Assessing organisational functions

Assessing and monitoring program outcomes
The question: ‘Is this programme working’ can be most readily answered by assessing and monitoring programme outcomes. Various approaches to identifying and assessing programme outcomes are reviewed.
• Identifying relevant outcomes
• Establishing base-line and current outcomes
• Considering unintended outcomes

Assessing program impact: The classic experimental design
The classic experimental design is often upheld as the ‘gold standard’ for evaluation techniques. Here, the classic experimental design is reviewed. Practical limitations on the use of such randomized experiments are also considered.
• Random assignment
• Planned variation
• The classic experimental design
• Analysing experimental results
• Coping with difficulties that arise
• Conditions that make randomised experiments difficult
• Criticisms

Alternative strategies for program evaluation
Acknowledgement of the practical limitations on the use of the classic experimental design has led researchers to consider alternative strategies for programme evaluation. Several key approaches are reviewed here. Consideration is also given to the virtues of building triangulation and replication into research designs.
• Quasi-experimental research designs
• Qualitative methods
• Cost benefit and cost effectiveness analysis
• Meta-analysis
• Triangulation and replication

Conclusion: Taking stock and looking ahead
An interactive discussion concerning the material covered to this point, what participants should do to prepare for what comes next, and an overview of the focus for Day Two.

Day Two
Managing Programme Evaluations

Working with programme stakeholders
Programme evaluation is never simply a technical exercise. It always takes place in a politicised environment. While those working on the evaluation need to be protected from political interference, there is merit in establishing open and effective lines of communication with stakeholders. The key issues around working with stakeholders are considered.
• Identifying key individuals and groups associated with the programme
• Developing a communication plan
• Making effective use of an advisory committee
• Gathering advice, acting upon it, and reporting back

Measurement issues
Programme evaluators seek to make strong claims about the nature and impacts of the programmes they study. Quantitative research methods hold the greatest potential for achieving analytical rigor and for justifying the claims being made. Approaches are discussed for transforming concepts into collectable, good quality data.
• Dependent and independent variables
• Desirable characteristics of variables
• Concepts and measures
• Measuring variables
• Using multiple measures
• Markers of progress
• Program inputs, resources, and environments

Techniques for data collection
After decisions have been made about the research design and the kind of measures required to support further analytical work, several techniques might be used to collect appropriate data. Those techniques are reviewed.
• Sources of data
• Sampling
• Interviewing
• Coding responses
• Utilising existing statistical data
• Merging primary and secondary data

Techniques for data analysis and interpretation
Evaluators must find ways to undertake appropriate data analysis that can yield insights into programme effectiveness. An overview is provided of regression techniques, designed to isolate programme effects while controlling for alternative explanations of the observed relationships.
• Regression analysis
• Working with limited dependent variables
• Working with censored data
• Time series analysis
• Choosing case studies based on analysis of quantitative evidence
• Working with qualitative data
• Good practice in data presentation and interpretation

Presenting and utilizing evaluation findings
Evaluation work is always costly. Programme evaluators can take a number of concrete steps to increase the odds that their work will have an impact on decision-making, that it will improve programme outcomes, and that it will inform broader policy conversations.
• Meeting expectations of the immediate client
• Supporting change processes
• Serving multiple audiences
• Exploring opportunities for broader dissemination

Aspects of ethical practice
Programme evaluators must act with integrity at all times, both because their actions can have major implications for the wellbeing of others and because integrity is a hallmark of professionalism.
• Understanding the programme and its place
• Maintaining high technical quality
• Using balance and judgment
• Adopting a utilization focus from the outset
• Behaving with integrity when working with others
• Surviving under difficult circumstances

Continuing to build your evaluation research capabilities
A review of strategies that reflective evaluators can use to improve their potential as professionals.
• Inviting feedback and constructive criticism
• Lesson drawing from other evaluation work
• Joining professional associations
• Improving your professional reading habits
• Integrating an evaluation mindset into your everyday work

Seminar review and evaluation
An interactive discussion focusing on topics covered in the seminar, issues that have arisen as the seminar has progressed, what participants have gained from the seminar, how participants will integrate what they have learned into their work practices, and how the seminar could be improved or extended in the future.

Facilitator

Michael Mintrom, Associate Professor Political Studies Department, University of Auckland

Michael Mintrom is an associate professor in the Political Studies Department at the University of Auckland, where, since 2002, he has offered courses on Policy Analysis, Evaluation, and Research Methods.  He coordinates the University of Auckland Master of Public Policy (MPP) degree and works with others to run the annual Auckland Public Policy Seminar Series.  He has extensive experience as a trainer of public sector professionals in New Zealand, Australia, Asia, and the United States.

Throughout his academic career, Michael has studied and written about aspects of policy analysis, policy change, and approaches to securing change in the public sector.  His books include Public Entrepreneurs: Agents for Change in American Government (Princeton University Press, 1995), Policy Entrepreneurs and School Choice (Georgetown University Press, 2000), People Skills for Policy Analysts (Georgetown University Press, 2003), and Political Leadership in New Zealand (Auckland University Press, 2006).  Most recently, Michael has been studying approaches used by universities to create stronger research cultures and promote greater engagement between academics and knowledge-based enterprises.

Michael received his Ph.D. in Political Science from the State University of New York at Stony Brook in 1994.  Prior to that, he received an MA in Economics from the University of Canterbury and worked from 1987-1990 as a policy analyst in The Treasury.  From 1994-2002, he was a faculty member at Michigan State University, where he attained the rank of associate professor with tenure in the Department of Political Science and the Institute for Public Policy and Social Research.  He has also been a visiting professor in the School of Policy, Planning, and Development at the University of Southern California.

Michael Mintrom is also facilitating:

• Effective Policy Analysis and Development
• Managing Policy Projects
• Strategic Change Management in the Public Sector

In-house Training

Prices and Registration

* Early bird price available when you register and pay before the dates listed.