Take Home Lessons from #EVAL13

Introduction

The American Evaluation Association (AEA) annual conference in Washington, D.C. just wrapped up last week. With about 3,000 evaluators in attendance, it is the largest annual gathering focused on evaluation. The American Evaluation Association hosts the event each year. For me, it was an enlightening experience. In fact, it left me wondering why I do not attend every year. Below, I share four take home lessons from the 2013 American Evaluation Association conference (#EVAL13) that continue to shape my practice.

Evaluators Are Friendly and Obsessed with Improvement

First, almost everyone I met was friendly and helpful. There was a clear focus on how to help each other do our work better. People shared tips on data visualizations, reports, presentations, and even blogs. That makes sense. After all, this is arguably our main task as evaluators: how can we help make programs, projects, and products better?

For example, I attended the Data Visualization TIG meeting. It was filled with comedy and insight. Many of the affiliated evaluators write blogs, which finally encouraged me to stop obsessing over my WordPress theme and start posting. In addition, it is refreshing to attend a conference and be immersed in new ideas. At the same time, you get reminded of older ideas that deserve to be revisited and reworked.

Importantly, I also realized that I can get my evaluation intellectual dose on a regular basis between conferences. Ongoing communication and contact with other evaluators through their Twitter feeds, blogs, and other posts make this easy.

Strive to Increase the Utility of Evaluation Reports

Next, let us talk about reports. It is a rare soul that reads the entire evaluation report after I submit it. So how can I put the information into a more digestible format? How can I provide more concise feedback that takes less time to understand?

One presenter suggested an executive summary style report with most of the results placed in an appendix. To top it off, she develops a one page handout with visuals that summarize the main points. In my view, that approach makes sense most of the time. However, the right format will largely depend on the stakeholders, their backgrounds, and their roles on the project. As a result, I plan to give it a try on a report I just developed.

In a similar effort to create a more accessible visual product, my team developed what we call a Visual Evaluation Report. Essentially, it is a video summary of a domestic violence and high conflict divorce program, Bridging Families and Communities, along with a review of the evaluation findings. Take a look at the YouTube video below. I would love to hear your feedback.

Improve the Visual Display of Quantitative Results

Another big theme at the American Evaluation Association conference was data visualization. There were tons of examples of “good” versus “bad” graphs in the conference talks. Most of the time, our products are not shared with researchers and evaluators. Instead, they go to stakeholders with limited quantitative experience. As a result, the design of a graph is key to its interpretability. The colors chosen, the layout, the labeling, everything matters.

While the APA style guide and similar resources offer suggestions, we should still strive to continually make our visualizations better. For instance, I liked the idea of examining palettes of colors that work well together. I also liked the call to consider how color blind individuals would process colored figures. So the design of my bar charts will not be a stagnant template. Instead, it will be constantly improving.

In addition, we should think about how evaluation reports that use color in figures will look when printed in black and white. We have all seen the result, and it can be a disaster. Sometimes, it is worth printing high quality color reports and sending them directly to stakeholders. After all, many people still prefer to read on paper.

Cost Effectiveness Is a Perennial Evaluation Issue

Another highlight of the American Evaluation Association conference was a three hour cost effectiveness workshop with Edward Broughton, Director of Research and Evaluation at USAID-ASSIST and the USAID-Health Care Improvement Project. We examined methods for determining the true costs of initiatives. After all, funders typically want to know if the programs they fund make an impact in a cost effective way compared to other approaches.

There are also innovative approaches that assess the relative value of programs aimed at social equality, environmental sustainability, and well being. For example, the Social Value International network (formerly the Social Return on Investment Network) offers practical methods for this kind of analysis. I am excited to incorporate these methods into my practice.

Connecting These Lessons to Everyday Evaluation Work

Taken together, these four take home lessons from the American Evaluation Association conference reinforce a single theme. Evaluation is most useful when it is shared in formats that real people can absorb and act on. In other words, the technical work matters, but the communication work matters just as much.

If you want to dig deeper into related ideas, check out our discussion of the CIPP evaluation framework with Dr. Guili Zhang. For a closer look at how a logic model and CIPP work together in a real study, see our post on evaluating a mindfulness program. Finally, for a sense of where evaluation practice is heading, see our piece on how AI and exponential technologies are reshaping evaluation work.

Looking Ahead

For me, there are many reasons to travel for AEA each year. Visiting friends from high school, college, and beyond who have ended up in different regions is a big draw. So is hiking, exploring new neighborhoods, and reconnecting with the wider evaluation community. Next October, #EVAL14 is firmly on my list.

Above all, the American Evaluation Association conference reminds me why I do this work. We get to help leaders make smarter decisions, improve services, and show real impact. That is a privilege, and it is worth investing in the relationships and skills that make it possible.

Frequently Asked Questions

What is the AEA conference? The American Evaluation Association annual conference is the largest yearly gathering of evaluation professionals in the United States. It draws around 3,000 attendees each year for sessions on methods, tools, and applied practice.

Why does report utility matter? Evaluation reports are only useful when stakeholders actually read and apply them. Concise formats such as executive summaries, one page handouts, and visual summaries make it more likely that findings will inform decisions.

What is a Visual Evaluation Report? It is a short video summary of a program and its evaluation findings. Visual reports can complement traditional written reports and reach audiences who learn better through video.