Using AI to Improve Mindfulness and Evaluation Practice

I’ve written and talked before about how the work of evaluators can be supported through mindfulness practice. Well, would you believe it if I said that mindfulness can be indirectly enhanced through the use of an artificial intelligence technology?

Let me explain how. Also, see the video below.

GPT-3

Generative Pre-trained Transformer – 3 (GPT-3) was introduced in mid-2020 by Open AI, a San Francisco-based artificial intelligence research company. GPT-3 is what is referred to as a Transformer, which is based on a deep learning model called a neural network, and was first introduced in 2017. Shout out to Vasili Shynkarenka who taught a fantastic course that accelerated my understanding of how to use GPT-3 and its possibilities.

GPT-3 can be used by individuals who are not proficient in coding as it has a “text in, text out” interface that can complete many types of English language tasks. It was trained on data from Common Crawl, which is a regularly obtained archive of the web, as well as other web text, books, and Wikipedia. It utilizes 175 billion parameters compared with 1.5 billion parameters for the previous version, GPT-2, resulting in a sizable step up in performance.

Individuals and organizations can use GPT-3 through the OpenAI Playground, a web-based interface, or create applications that use the GPT-3 API in the background. The types of use cases for GPT-3 include: classification, completion, conversation, semantic search, summarization, factual responses, generation, and transformation. Developers have been working on many creative applications of GPT-3; here are some great examples.

GPT-3 has the potential to be dangerous as it can be difficult to tell the difference between its output and what is written by actual people. There are a multitude of nefarious opportunities, for instance, generating SPAM, fraud, and plagiarism.

The developers highlight the potential danger of GPT-3 in their academic paper. Also, researchers, such as Timnit Gebru and colleagues, have identified challenges pertaining to these types of large scale language models including environmental costs (due to high energy consumption) that may impact marginalized communities disproportionately and the effect of racist, sexist, and abusive language that exists in the training data. Clearly, GPT-3 has to be used with care and supervision.

Regardless, in the coming years, Transformers such as GPT-3 and similar technologies will likely become ubiquitous because they can be very useful for many text-related tasks. Microsoft obtained an exclusive license for GPT-3 for a billion dollars so I imagine that they will integrate it into their products or offer it in other ways soon. Also, Google recently indicated that they trained a language model, called a Switch Transformer, with 1.6 trillion parameters, so there is competition and growth is likely in this space.

Currently, GPT-3 can provide substantial benefits for professional evaluators when working on program evaluation projects. I have also used GPT-3 for various purposes including brainstorming learning outcomes and developing multiple-choice questions to test knowledge related to text about Daniel Kahneman’s heuristics and biases research. It’s been great to have access to this tool.

Discipline of Evaluation

How does all of this relate to evaluation? First off, what is the discipline of evaluation? Various definitions of evaluation have been proposed through the years. In 2014, an American Evaluation Association task force proposed the following definition as a starting point for discussion and elaboration: “Evaluation is a systematic process to determine merit, worth, value or significance.”

I talked with Amy Gullickson about her 2020 article related to this topic. She proposed the following definition building on the work of other evaluators: “Evaluation is the generation of a credible and systematic determination of merit, worth, and/or significance of an object through the application of defensible criteria and standards to demonstrably relevant empirical facts.”

When most people ask me I say something like: “I help to determine how programs that are run by organizations that help people are doing and how to make them better.” If they ask me more questions I use a more proper definition.

But in order to accomplish this aim evaluators have to be able to think clearly. Specifically, as Ernest House (2015) argues, it is important for evaluators to become aware of our biases in thinking and take active steps toward mitigating them, learning from both the research methodology and social psychology research literature.

Mindfulness and the Role of Reflective Questions

But how does all of this relate to mindfulness? First off, what is mindfulness? Jon Kabat-Zinn (2017) described it as follows, “Mindfulness is the awareness that arises through paying attention, on purpose, in the present moment, non-judgmentally… in the service of self-understanding and wisdom.”

Some mindfulness-based programs encourage the use of reflective questions to promote mindfulness in participants. Ronald Epstein and his colleagues (2008) describe the habit of self-questioning as important to developing self-monitoring and mindfulness in physicians. They indicated, “Using reflective questions enhances the ability to see familiar situations with new eyes and to self-monitor one’s actions during actual practice.”

This metacognitive practice can improve one’s ability to become more self-aware of bias in thinking. They also state, “In this way, we can consciously sense our tendency to draw rapid conclusions to quell our anxiety and have a conscious choice to inhibit such automatic reactions from controlling our decision-making processes.”

Cognitive Dissonance as an Example of Biased Thinking

Cognitive dissonance, an example of a process that can lead to bias, occurs when we hold contradictory cognitions or engage in a behavior that is inconsistent with a cognition. To manage the stress or discomfort that is experienced with this dissonant state, we may rationalize or justify how it is possible to have what appear to be incompatible views or behaviors.

An example of cognitive dissonance used by Leon Festinger, the originator of cognitive dissonance theory, is the case of one who smokes cigarettes even though they are confronted with evidence indicating smoking is harmful. Such an individual may initially experience a dissonant state that is emotionally uncomfortable. This dissonance can be alleviated by changing beliefs, for instance, devaluing the evidence of harm provided by research studies or by rationalizing that smoking improves stress management leading to an overall positive impact on cardiovascular health. We all do this, often beyond awareness, sometimes leading to holding irrational beliefs and poor decision-making.

I will come back to how reflective questions can be used to mitigate the impact of this process.

Reflective Questions for Evaluators

In their paper, Epstein and colleagues list a sample of reflective questions for physicians they use in their program that I thought could be modified so they were applicable for professional evaluators. Below are three of their questions along with reworded versions I developed for program evaluators.

Physician: If there were data that I ignored, what might they be?

Evaluator: What are other data sources to which I have not paid attention?

Physician: Is there another way in which I can formulate this patient’s story and/or my response?

Evaluator: Is there another way I can understand what is happening in the evaluation and/or my response?

Physician: What would a trusted peer say about how I am managing or feeling about this situation?

Evaluator: What would a respected colleague say about how I am conducting, managing, and feeling about this evaluation?

Using GPT-3 to Generate Reflective Questions

Using these as samples for what’s called “few shot training” and iteratively adjusting the GPT-3 parameters and running the API about 30 times, it generated some useful physician and evaluator reflective questions. I am most interested in the evaluator reflective questions.

Here are 10 of the better questions that it generated after some editing on my part:

1. What aspects of this evaluation have caused me to feel uncomfortable?
2. What is the most difficult issue or emotion that I have encountered in this evaluation?
3. What is the part of this evaluation that I find most difficult to understand?
4. What would I tell a colleague to do if they were conducting this evaluation?
5. What am I most grateful for in this evaluation?
6. What can I do to reduce any biases I might have with respect to this evaluation?
7. What questions have I not asked that I should have asked?
8. What have I done that I would do differently if I could do it again?
9. What have I learned about myself as an evaluator?
10. What have I learned from this evaluation?

This video shows how I used GPT-3 to generate these questions starting at 8:46.