SOCIAL PROGRAM: Re-evaluating a Good Idea

vachurchBackground
One of the country’s oldest, largest, and most respected community foundations had long worked beside churches and other faith-based groups to address community issues. The foundation had ample funds and a track record of successful programs, but its funding guidelines precluded direct support of religious groups and ad-hoc projects.

Faith-based organizations – known as FBOs – always needed more funding, always maintained credibility with the communities and hurting people they served, and always stayed skeptical of restrictions that any funding source might put on their work.

Of special interest to the foundation was creating permanent change in the future of urban young people. Foundation leaders decided to accelerate that change by experimenting with a program that funded faith-based groups. Consequently, they launched an initiative called the Youth Preparation Program with grants ranging from $10-75,000.

Despite considerable research and careful planning, after nine months, the foundation staff wanted to evaluate the effectiveness of the program itself, not necessarily any results FBOs were producing. Basically, they wanted to know:

  • Was the concept translating into effective implementation?
  • Was the bold idea workable on the streets?
  • Was the implementation sustainable?
  • Could benchmarks be set and eventually used to measure the program?

To find answers, the foundation retained me and a community development expert to undertake a “formative evaluation.”

Challenges

  • Stay focused on assessing the impact of the foundation’s program on funded FBOs and not the FBOs impact as a result of the funding.
  • Define “faith-based” broadly enough to include non-traditional groups and yet narrow enough to prevent losing the program’s distinctiveness.
  • Get honest feedback from funded groups who may be reluctant to criticize the source of their funding.

Strategy/tactics
We set a strategy that would explore the trail from the original intent of the youth program, through the perceptions of implementers, and finally to how the program was translated and used by youth.

We would listen closely for both information and implications, probe areas of inconsistency, identify expectations, and unravel frustrations– all focused on whether or not the program was actually serving the program’s target population.

We adopted five tactics to reach a.) foundation representatives who created and administered the program b.) leaders of faith-based organizations who interfaced with the foundation and c.) youth who participated in the FBOs’ offshoot programs.

Tactics included:

  • content analysis of foundation documents – memos, reports, independent research, etc. – and program communication to look for patterns that would tell us if the initiative was built on a well-reasoned, team-endorsed, and mission-driven purpose, or was it simply one person’s crusade or just another good-idea-at-the-time.
  • in-depth interviews with:

foundation staff — board members, leaders and staff, and program advisors – to:

    • determine how data, disparate ideas, and opinions gelled to form the youth program.
    • compare how the program was conceived and designed against the reality of how it was being administered.

leadership of funded groups to:

    • ask them what they thought they would get from the program, how it was working for them, and what more they might need.
    • ferret out any ongoing giver-recipient problems that could jeopardize the program and/or organizations’ chances for renewal of funding.
  • site visits to group centers around the city to learn first-hand how the organizations turned the broad foundation program into specific activities.
  • surveys of the first year’s funded groups to set up and supplement our findings from focus groups.
  • focus groups — two with funded organizations and one with organizations that were not funded – to uncover how they understood the program initially, how they thought it fit with their ongoing programs, and how they would improve it.

Results

  1. Our study showed that the foundation’s courage in launching the youth preparation program was not only applauded by faith-based community leaders but viewed, albeit with some wait-and-see, as a significant start of a long-term change in communities.
  2. A number of issues surfaced regarding how communication of the program could have been handled better, though, overall, community groups understood the initiative and were willing to participate.
  3. My colleague’s community expertise and personal style, combined with my communication skills, enabled us to extract honest feedback that produced an in-depth assessment and achieve the foundation’s objectives.
  4. The foundation Board and staff implemented our long list of recommendations and set a course to use the formative evaluation as the benchmark for the eventual summative evaluation.