Staying ahead of technological change is no longer optional for evaluators. Exponential technologies, the innovations that drive accelerating gains in performance while costs fall, are reshaping how evaluation work gets done. Artificial intelligence, blockchain, quantum computing, and robotics all represent real opportunities to strengthen evaluation methods and deliver more useful results for clients.
The rapid advancement in computing power, exemplified by Moore’s Law and the exponential growth of transistors on computer chips, has created new possibilities for the practice of program evaluation. This post walks through what those possibilities look like in day-to-day work.
Why Evaluators Should Engage with AI Now
According to McKinsey & Company’s research on AI adoption, organizations that have expanded their use of AI have reported substantial operational benefits and a meaningful competitive edge. Evaluation professionals and the nonprofit organizations they serve typically lag behind for-profit sectors when it comes to adopting new technology. That gap is also an opportunity. Evaluators willing to experiment can differentiate their practice and bring more capacity to the nonprofits they support.
From Traditional to Tech-Enhanced Evaluation
My own path mirrors the technological shift. My father worked in computer programming during the early years of the computer revolution and saw firsthand how new tools could rewire organizational workflows. Today, doing program evaluation, I do not need to be a programmer to tap into a similar kind of transformative power.
In 2021, I began integrating GPT-3 (Generative Pre-trained Transformer 3) into my evaluation work. The applications I found have meaningfully changed how I support clients and conduct evaluations.
Understanding the Technology
GPT-3 was a breakthrough in accessible AI for evaluation professionals. Built on a neural network architecture called a Transformer, it offers a “text in, text out” interface that requires no coding expertise. The model was trained on large datasets, including Common Crawl, web texts, books, and Wikipedia, which makes it surprisingly versatile for evaluation-related tasks.
Practical Applications for Evaluators
Evaluation professionals can use GPT-3 (and the more capable models that have followed it) through OpenAI’s Playground or by building custom workflows with the API. Some of the most useful applications include:
- Qualitative data analysis: first-pass classification and coding of interview transcripts and open-ended survey responses.
- Report drafting: faster creation of evaluation reports with consistent formatting and tone.
- Stakeholder analysis: structured analysis of stakeholder conversations and feedback.
- Semantic search: intelligent searching across an archive of evaluation reports and notes.
- Client communications: drafting accurate, contextual responses to client questions.
- Instrument development: creating evaluation instruments and educational materials.
- Data transformation: converting data between formats and structures for analysis.
Addressing Common Concerns About AI in Evaluation
The concerns evaluators raise about AI are legitimate. Three come up most often.
Data privacy and confidentiality. The concern is how to protect sensitive client and participant data. The practical answer is to never input personally identifiable information into public AI tools. Use anonymized data, build synthetic examples, or work with aggregated information. Establish clear data handling protocols and choose platforms that offer enterprise-level security.
Maintaining evaluation rigor. The concern is whether AI will compromise the validity and reliability of evaluations. The answer is that AI enhances rather than replaces evaluator expertise. Use AI for initial analysis, then apply your professional judgment. Always validate AI-generated insights against established evaluation frameworks and methods.
Ethical considerations. The concern is how to ensure ethical use of AI in evaluation. The answer is to develop an AI ethics framework that includes transparency with clients, human oversight of all AI-generated content, acknowledgment of AI assistance in reports, and regular bias checks of AI outputs.
Case Study 1: Enhancing Reflective Practice
I have used GPT-3 to develop reflective questions that deepen stakeholder engagement. By feeding the model a small set of effective questions, I generated novel variations that maintain theoretical rigor while offering fresh angles. This approach has been especially useful for facilitating deeper stakeholder learning, encouraging critical thinking about program outcomes, and supporting organizational growth and adaptation.
For more on that experiment, see my earlier post on using GPT-3 to generate reflective questions and enhance evaluation practice, which includes a video demonstration.
Case Study 2: Streamlining Assessment Development
For a recent nutrition program evaluation, I used GPT-3 to draft comprehensive multiple-choice knowledge assessments. The result was a roughly 75% reduction in assessment development time, AI-assisted item generation that supported content validity, and more time freed up for strategic evaluation planning.
Resources for Tech-Forward Evaluators
Continuous learning is essential to keep up with this space. A few resources I return to:
- Exponential View by Azeem Azhar: weekly insights on exponential technologies and their societal impact. This newsletter is what first introduced me to GPT-3’s potential for evaluation work.
- Future Today Institute: Amy Webb’s team produces annual trend reports and regular updates on the technological, social, and economic shifts affecting professional services.
- McKinsey & Company Analytics Insights: research on AI applications in professional services, including evaluation and assessment.
- State of AI Report: Nathan Benaich and Ian Hogarth’s annual review of developments in AI that are relevant to evaluation work.
Action Steps
- Start small. Begin with one AI tool and master it before expanding.
- Document your process. Track efficiency gains and quality improvements.
- Share your journey. Connect with other tech-forward evaluation professionals.
- Invest in learning. Dedicate time each week to exploring new tools.
- Consider ethics. Develop guidelines for responsible AI use in your evaluation practice.
Looking Forward
Evaluators stand at the threshold of unusual opportunity. Integrating exponential technologies into evaluation practice is not just about efficiency. It is about expanding what is possible in understanding and improving the programs that serve communities.
As Timbuk 3 sang in the 1980s, “the future’s so bright, I gotta wear shades.” For evaluators willing to embrace technological change, that future is now.
EvalNetwork Podcast
Join me on the EvalNetwork podcast, where I interview thought leaders in program evaluation, data visualization, survey development, and emerging evaluation technologies.
Recent episodes have featured conversations with Dr. Melanie Hwalek on retrospective pretest surveys, Dr. Tiffany Berry on evaluation training and frameworks, and Dr. Michael Quinn Patton on utilization-focused evaluation. The show is available on Apple Podcasts and YouTube.
A Reflection
“Human freedom involves our capacity to pause between stimulus and response and, in that pause, to choose the one response toward which we wish to throw our weight.” (Rollo May)
As we integrate powerful technologies into evaluation practice, this quote is a useful reminder that human judgment, ethics, and values remain central to meaningful program evaluation.
EvalNetwork helps nonprofits in health, education, and human services design program evaluations that combine deep evaluation expertise with innovative approaches. Get in touch to discuss your project.
