Jeffrey Kaster: Why I love program evaluation
I love evaluation. Yes, that’s right. I love evaluation.
I realize that many people in theological circles would equate program evaluation to St. Paul’s “enduring all things.” But I think we should instead consider it a “more excellent way.”
Paul might take exception to relating evaluation to the more excellent way of the spiritual gift of love. But perhaps one could consider that providing excellent ministerial programming is a way that pastoral leaders show Christ’s love.
For 35 years, I’ve heard colleagues complain that evaluation is basically hoop-jumping to appease others. And I understand this — many theological leaders don’t see the benefits of evaluation.
But I love evaluation, because I’ve seen how it can be an astonishingly effective tool in building excellent ministry. I want to share four steps to developing a “more excellent way” of ministerial leadership through evaluation.
Step 1: Think strategically
When beginning an evaluation process, two questions arise: “What should be evaluated?” and “How will this help?” Answered thoughtfully, with a clear understanding of the context and needs of those being served, these questions can lead to deeper, strategic thinking about program goals and outcomes and about how the evaluation data might be used to improve the program.
Excellence in ministry demands strategic thinking, especially as ministerial leaders take on more responsibility.
This might seem obvious, but I’ve found that many pastoral leaders resist strategic thinking. The Gallup Organization’s research provides some insight here; the book “Strengths Based Leadership,” drawn from their findings, suggests four domains of leadership strength: executing, influencing, relationship building and strategic thinking.
Different leaders have different strengths. Pastoral leaders, who are often quite good at fostering relationships with people, are often lacking in the strategic thinking domain. To achieve organizational excellence, these relational ministers must seek out team members with complementary strategic thinking strength.
Step 2: Articulate goals and outcomes
Successful evaluation always begins with goals. If a theological program does not have clearly articulated goals and outcomes, then there is nothing to evaluate. Stephen Covey got it right: “Begin with the end in mind.”
Last summer, I taught a practical theology course on evaluation. I asked the graduate students who primarily served as catechetical leaders in congregations whether their programs had written goals and outcomes. A few said yes, but many of the students stared blankly at me.
Then I asked whether they used published curricula. Most said yes. But when I asked whether they knew the learning outcomes for those published curricula, their clueless eyes suggested the quote from Yogi Berra, “If you don’t know where you are going, you might wind up someplace else.”
Theological leaders will absolutely end up somewhere else without clearly articulated goals and outcomes, so defining them is a crucial step in the evaluation process.
Step 3: Develop tools to assess results
How can a pastoral leader know whether a program’s outcomes are achieved? It’s important to ask this question before programming begins. Unfortunately, we often employ very limited measures of success.
When I was a youth minister, the only evaluation question I was ever asked was, “How many youths showed up last night?” Participation is important, but Christian formation is about much more than numbers. Yet that’s all I measured.
In my current role, I have the privilege of working with the directors of seven seminary- and college-based youth theology programs. They are part of the Youth Theology Network coordinated by the Forum for Theological Exploration (FTE) and initiated by Lilly Endowment Inc.
In planning the summer youth theology institutes, we met with an evaluation consultant to identify common outcomes. We then developed surveys to give to participants before and after the institutes.
One desired outcome was “The youth will experience Christian community.” To measure this, we would ask participants to agree or disagree with the statement “I have been part of a vibrant (lively, animated, vital) Christian community.”
Another desired outcome, “The youth will grow in their excitement for theological learning,” would be measured through their response to “I am considering theological study in college.”
These relatively easy-to-create surveys would help us measure whether and how the participants changed as a result of the programs.
Step 4: Ponder the data and make adjustments
When we looked at the pre- and post-institute survey results from approximately 200 youth in the summer of 2019, we got some good news: the number of youth who strongly agreed that they had been part of a vibrant Christian community increased 60%. The number who strongly agreed that they were considering studying theology increased more than 90%.
But evaluation data aren’t always positive. In 2019, the youth theology program that I direct, Youth in Theology and Ministry (YTM), had an alarming decline in participants’ assessment of their positive experience of community.
For nearly two decades, 90% of youth participants responding to the statement “Assess your overall experience of community at YTM” had said “excellent.” In 2019, only 61% said “excellent” — the lowest in 19 years.
Something was wrong. Reviewing and analyzing all the data, we discovered that they reflected our failure to understand the complexity of developing Christian community within a diverse population of youth.
In 2019, YTM had achieved for the first time a population of youth that was 50% persons of color. The evaluation question helped us realize that racial and ethnic barriers could affect the development of community in ways we did not understand.
As a result, the YTM staff committed to learning more about intercultural ministry and effective strategies to deal with intercultural conflict. For example, we recently attended a lecture and workshop on intercultural ministry by Arturo Chávez, the president and chief executive officer of the Mexican American Catholic College in San Antonio, Texas, and we’ll be applying this new knowledge to next year’s institute.
Without that survey, we might not have realized that something had shifted and that we needed to make changes to this long-standing program.
That’s why I love evaluation. It provides insight into what is working and what isn’t — and challenges a team to address the issues that arise.
Most of all, I love evaluation because it is a window into a “more excellent way.”
Why do religious organizations find evaluation work so mysterious? What do they need to help them appreciate the importance of evaluation and to do evaluation well?
I asked these questions when I first began coaching new grantees of the religion division of Lilly Endowment Inc. As I sat alongside grantees, I often noticed a hesitation and reluctance about embarking on evaluation. It wasn’t as if evaluation was a new idea. Rather, I realized that the mere word “evaluation” raised less-than-pleasant memories that limited their openness to this work. Grantees wondered: Is my work being judged? Will someone form an opinion about the merit or quality of our program?
Indeed, how many of us hold a memory about an evaluation experience gone bad or a moment of being judged or of having someone else’s opinion supersede our own?
At the heart of the word “evaluation” is “value” — to find the value of — and this notion guides my evaluation coaching work to this day. At its most effective, evaluation is an activity of valuing and learning from one’s work, not a judgment. Principles such as collaboration, inquiry, learning, curiosity and mutuality guide such an approach to evaluation.
This evaluative lens recognizes that projects are conceived with the aim of creating good in the world and ultimately making a tangible difference in people’s faith lives. It also changes the tone and character of the questions one asks:
- How is our program/project making a difference in the lives of young people, pastoral leaders, teachers, students or other target audience? How do we know?
- What conditions will change as a result of our program? What new knowledge is being gained? What new actions are being taken as a result of our program, and why? What new attitudes are being formed?
Such purposeful questions engage stakeholders’ curiosity and allow them to be open to developments they did not anticipate in the original program design. From my studies of Appreciative Inquiry, I know how thoughtful, positive questions can change the trajectory of an evaluation. For example, reframing, “What is not working in our program?” to, “What gives life to our program when it is functioning at its best?” can open up constructive, nondefensive conversation and new methods for discovering answers.
If projects are to improve and grow, they will do so as a function of what we choose to ask and study. In “Appreciative Inquiry Handbook: For Leaders of Change,” the authors write:
“Inquiry and change are not truly separate moments. … Inquiry is intervention. The seeds of change are the things people think and talk about, the things people discover and learn, and the things that inform dialogue and inspire images of the future. They are implicit in the very first questions asked. … The questions set the stage for what is ‘found’ and what is ‘discovered’ (the data). These data become the stories out of which the future is conceived, discussed, and constructed.”
If projects are to be sustainable and potentially imbedded in the very institutions that house them, then reflective conversations must be woven into the life of a project from the beginning. Moreover, a project director cannot engage in this kind of thoughtful conversation alone. Creating communities of learning and practice is key if grantees are to become their own active agents of evaluation and to gain confidence in this work.
“What is at stake is the efficacy of love and care and service” writes Craig Dykstra, former vice president for religion at Lilly Endowment, in Kathleen Cahalan’s book “Projects That Matter: Successful Planning and Evaluation for Religious Organizations.”
“What is at stake is how human beings engage with and relate to one another. What is at stake is how minds are illuminated, hearts are moved, burdens are lifted, wounds are healed. And because so much is ultimately at stake, it matters a lot that the projects and programs be done really well.”
A community of learning and practice embraces the responsibility to inquire and to learn from its work, to improve its project as a result of that learning and, in turn, to contribute to strengthening the larger landscape of Christian life, ministry and institutions. A community of learning and practice is made up of thoughtful stakeholders who care deeply about the work at hand and can ask the kinds of hard questions about a program that are essential to program improvement and sustainability.
So what is the result when evaluation is focused on learning and not judging?
This approach provides a lens through which and a structure out of which grantees can assess the impact of their work, learn from that work, and revisit original questions and assumptions to strengthen, sustain and bring the program to full maturity within the fabric and mission of the institution. Their organizations can develop renewed energy to risk and experiment with innovative strategies. Their team members can become more enthusiastic for and appreciative of evaluation and can experience it as joyful work.
Reframing evaluation can also broaden grantees’ awareness, to a wider community. The body of knowledge, information and insight gained through evaluation has great value not only to the institution running the program but to a larger community that cares deeply about the future of religious institutions entrusted with caring, teaching and serving others.
Finally, this approach to evaluation can offer a pathway to theological reflection. Evaluative findings can provide meaningful moments for grantees to pause and reflect on how a project is advancing the mission and ministry of the institution and to ask deeper questions about how they understand the work in relationship to the call and presence of God.
As inspiring stories from projects begin to emerge — how a program has changed the vocational choices of a young adult, taken a pastor out of isolation into a trusted circle of peers, reshaped the culture and priorities of an institution — they can open our eyes to a larger canvas of vital institutions carrying out the mission of the church and flourishing as a result.
The question, “What is your mission?” sends waves of panic through many Christian congregations and other institutions. Why? Do we not know what to do? Are we so confused by the rapid pace of change that even our basic purpose feels unsettled?
Many congregations are caught between finding meaning in the worship, study, ministry and missions that have sustained faith through several generations and addressing the concern that young people are not coming to church. We read about the rising number of people with no religious affiliation (the “nones”), and many of us realize that our grandchildren and their friends are not in church.
This situation leaves many with doubts about what our congregations are doing. We wonder whether the church down the road has things figured out.
The vocabulary also trips us up, with people using “mission,” “purpose,” “vision” and “strategy” to mean different things. The simplest approach defines “mission” as the completion of the sentence, “We exist to …” Mission is action-oriented; purpose is more about being. The two are closely connected.
Another challenge is that a clear mission does not always result in a specific strategy. The mission of a congregation might be to “make disciples,” using the phrase from Mathew 28’s Great Commission. The statement is short, memorable and rooted in the Christian tradition. Unfortunately, it does not indicate to a congregation how or where to do that work.
Seminaries also have this challenge. At one level, the mission of a seminary could be simple: “We exist to train pastors.” A denominationally owned and operated seminary trains pastors for that denomination’s congregations. The denomination wants the education to be accredited so that students can get federal loans. The accrediting agency determines the objectives of the curriculum through its standards. The seminary hires a faculty that meets the standards of both the denomination and the accrediting agency.
But what happens if congregations don’t need as many pastors? What happens if congregations decide to train their own leaders? What happens if the cost of education is beyond the ability of the students and denomination to pay? In light of changing conditions, seminaries are revisiting their missions and strategies. Everything is getting reshaped.
Effective strategies are focused on activities with specific people and places. But determining and following those strategies can be a challenge. The ways that we have lived out the mission don’t seem to be having the same impact as before. We feel a need to do something different, yet many of our current strategies have meaning for us and our communities.
In some ways, congregations have the most complicated challenge. Congregations exist to bear witness to God’s love shown in creation, the gift of the Son for salvation and the Holy Spirit. Communities need that witness, but in the current moment, the witness is not systematically welcomed.
What do we do when the ways that we witness bring us comfort but no one outside the church is paying attention? How does it affect strategy when the mission is clear to us but those outside don’t understand it?
Along with this concern, the fact is that changing a strategy in a congregation is very difficult. Every single element of the strategy includes choices made at some point in the congregation’s history. Every strategy includes activities that are meaningful to some of the members. All the choices took time to become habits. Members feel loyalty to one another, to the place, to the activity and to their experience of the activity over a lifetime. It is challenging to help them learn to see how an activity affects those outside the congregation.
Rather than wringing our hands about the clarity of a mission statement, we might make more progress by discussing the impact of our strategies. How do we know that we are bearing witness to the Triune God? What is the evidence? If it takes a long time for the evidence to show up, what are the signs that are early indicators of the evidence?
These questions require conversation among the leaders and members, along with people outside the church. Do principals, teachers, judges, public health professionals, police officers and social workers see the evidence of your congregation’s impact? If not, why not? Who outside the church should be noticing the impact you have named?
With the mission in mind and the impact articulated, the most difficult challenges are ahead — imagining new ways to increase the impact, giving less effort to activities that are contributing less and staying consistent on activities that are making a difference.
The art of leadership lies in discerning when to press for the new, when to maintain speed and direction for the established, and when to let go of the no longer effective — and then bringing others along in executing these moves.
Circling around and around a mission statement is a way to delay all the other important conversations and decisions. For Christians, mission statements matter, but from a human standpoint, faithfulness is measured by our deeds, our impact in the world.