In this interview, I speak with Amy Gullickson, acting Co-Director and Senior Lecturer at the Centre for Program Evaluation at Melbourne Graduate School of Education. She is also Chair of the International Society for Evaluation Education. We talk about what is the best definition of evaluation and why it is important to have a clear definition. Amy also gives us some of her specific resources for people just starting to learn evaluation.
What is Amy’s definition of evaluation?
Amy explained that it’s important for us to think about the implications of the definition. She does that in detail in her article titled, The Whole Elephant: Defining Evaluation.
She indicates that evaluation is the generation of a credible and systematic determination of merit, worth, and/or significance of an object through the application of defensible criteria and standards to demonstrably relevant empirical facts.
Amy states that it is the implications of the definitions that are important – it’s worth exploring what you (or your clients, or stakeholders) think evaluation is. That will shape what they expect you to deliver, and what may or may not be appropriate.
Amy believes a definition of evaluation must include valuation. This is our task as evaluators and has been overshadowed by social science research. We’ve got much work to do to become as informed (and have as much empirical evidence about what good looks like) in our valuation practice as we are in our research practice.
Why Amy thinks it’s important to have a clear definition of evaluation
People often think evaluation and research are the same things. Amy talks to me about why it is clear to understand the difference and have a clear definition. Amy gives an example, if you are trying to find the value of p (probability of a type I error), how big was the change? But evaluation asks, “so what?” Did it actually reach the people that are most important? Was it big enough to make a difference? Does that p value actually mean anything?
The task defines the knowledge, skills, and attributes that are necessary to accomplish it. If evaluation is just applied social science, then there’s no need to have skills and knowledge related to valuation.
Amy thinks this is a significant flaw in common evaluation practice. You might not get to summative judgment every time (and for good reasons- it might not be appropriate to do so), but if we take the valuation process out of the definition, then we are allowing the implicit values of the most powerful to determine what good is, what evidence is. Then we become complicit in upholding systems that oppress the global majority, in effect, giving our blessing to programs and systems that actually create harm. Amy explains this is exactly counter to what most people say they aspire to when they engage in evaluation.
How Amy believes evaluator competencies relate to how someone might define evaluation
Most competency sets have more than 60 competencies (Amy tells us the Australian Evaluation Society has 94). Canada has decided that anyone who can demonstrate an acceptable level of skill on a percentage in each domain can be credentialed as an evaluator. But are all competencies equally important? Are all competencies equally unique to evaluation? What we emphasize in terms of training or educating evaluators rests on the competencies that are essential to its practice. What makes an evaluation different than social science research? Valuation.
What makes evaluators different from researchers, or general organizational consultants? The ability to provide explicit, clear, evidenced reasoning for valuation claims.
Amy’s thoughts on evaluation being a profession that requires licensing or credentials to practice
Amy believes the profession of evaluation should require a license and credentials. But that will be a long time coming to the US. Voluntary organizations of Professional Evaluators (VOPEs) in smaller countries will do it first (e.g., Canada.) Evaluation is at the top end of the cognitive taxonomies (Bloom’s, SOLO) and it is deeply and intrinsically political. But because everyone does it to get dressed in the morning, we seem to think anyone can do it in their professional life. This seems fundamentally foolish and also increases the possibility that evaluation will do harm.
What students can bring to their future work of evaluation
Amy indicated that criteria are always present in the collection of evidence and in decision making. Be the person who asks questions about what makes the organization, program, project, or system you’re working on or in good – and how would you know that? (with what evidence?)
Learn how to recognize good evaluation and advocate for it in your workplace and community (i.e., know whether you need research and/or valuation, don’t conflate the two). Be an educated commissioner or consumer.
Lessons that inform your evaluation work
What we need to be looking at is what are the lessons from other disciplines that we need to improve our work? Evaluation is a trans-discipline. We have a contribution to make to all the other disciplines in their pursuit of good, and the skills and knowledge they’ve developed can contribute to ours. Look at the competency sets – they cover all kinds of skills and knowledge that other disciplines have been studying and training people to do for years. Why would we stick with only social science research methods?
Improving the world with evaluation – tackling fake news, climate change, and social inequities
Amy pointed to the importance of being explicit about what makes something – something, and what makes that something good – and asking questions about it. Genuine, naive curiosity in pursuit of answers to those questions can be powerful.
Let’s take the pro-life movement in the US. What is meant by a pro-life candidate? That they don’t support abortion. A good candidate is one that believes every fetus should live. This implies that life = fetus.
As many critics have discussed, if we assume a broader definition of “life”, it would probably encompass all human and non-human life. For humans, that would mean being “pro” things like health care (including mental health care), a living wage, decent housing, and protection from harm within or from the systems of government. For the planet, that would mean restoring habitat, natural systems like marshes, cutting fossil fuel dependency, and generally all the actions needed to reverse climate change. COVID-19 has kick-started us down that path by reducing pollution – can we maintain it?
Good evaluation asks us to check our definitions – 1. What are the boundaries we are setting around a thing (like “life” – what counts as life and what doesn’t?); and 2. What are the definitions of good for that thing, look for values and criteria – (e.g, – how would we know that “life” is good for everyone?)
Mindfulness and Evaluation
Amy has found that meditation is a good way to be able to take a research stance on her own functioning and get less caught up in the monkey mind and more able to keep a clear head. You’ve got to know your default functioning to be able to have any chance of avoiding it – and to use it to your advantage.
Amy’s recommendations for people just starting to learn evaluation
- Favorite evaluation authors – both of them are practical, plain-speaking writers who understand valuation, reasoning, and research methods:
- Jane Davidson https://realevaluation.com/jane-davidson/
- Wolfgang Beywl https://www.fhnw.ch/en/people/prof-phd-wolfgang-beywlgrab them (use DeepL to translate: https://www.deepl.com/translator).
- Read the New Zealand evaluation journal (Evaluation Matters) – it’s free and they’ve been working on the valuation task for longer than anyone else in English speaking countries (because Jane used to live there). https://www.nzcer.org.nz/nzcerpress/evaluation-matters
- Pick up Mathea Roorda’s handbook on generating defensible criteria. Download it here: https://melbourneuni.au1.qualtrics.com/jfe/form/SV_6QokokiqGHEvjNz.
- Check out Better Evaluation https://www.betterevaluation.org/
- Look for resources by EvaluATE: https://www.evalu-ate.org/
- Do your origin story (see ISEE video from Vidhya Shanker: https://vimeo.com/476085190 ). Understand where you come from, the values you grew up with, and that were instilled in you by your life experience and your studies. Unless you can see them, you will be implicitly governed by them. Along with that, you could use the Bowen Family Systems Theory to explore your family history: https://thebowencenter.org/. The multi-generational family you come from is part of your origin story (even if you think it isn’t!)
- Join your local evaluation association. And join AEA (https://eval.org) – the student rates are cheap and they have heaps of resources. Create a community of critical evaluation friends
- Volunteer to work on evaluation projects to get some experience.
Special issues of journals related to evaluation education and values in evaluation:
- Evaluation and Program Planning https://www.sciencedirect.com/journal/evaluation-and-program-planning/special-issue/10RJ5T2S1BK
- The Canadian Journal of Program Evaluation has a special issue on evaluation education in print at the moment.
- Evaluation Journal of Australasia did two special issues in the last 12 months
https://journals.sagepub.com/toc/evja/19/4; https://journals.sagepub.com/toc/evja/20/2
Connect with Amy
Twitter: @amyg4ce
