Editorial note: Critics of philanthropy’s obsession with quantifying and measuring grant outcomes are few, but they have lately come from various quarters in the sector. Among other articles on the topic, we have recently published Jeff Polet’s The Problem With “Measurable Impact” and Jerry Muller’s Against metrics: how measuring performance by numbers backfires.
This week we are bringing back from the archives a speech delivered on July 2008 by William Schambra on the difficulty of measuring outcomes in the field of public policy grantmaking. This talk is nearly a decade old and yet it is still extremely relevant to our current discussions on the subject.
As will soon become apparent, I am not a trained evaluator. And while I worked for a decade at the Lynde and Harry Bradley Foundation – described by some as an effective policy advocate – I confess we never generated a single logic model or theory of change to guide our work.
Rapidly displacing such slovenly amateurism is a new discipline called “advocacy evaluation.” Its models and schools of thought, its clearinghouses, websites, and journals, all funded handsomely by our largest foundations, are only beginning to develop and multiply across the land.
It is probably ill-disposed, therefore, to hear some grumpy old man, particularly a grumpy conservative old man, harrumph about the good old days when we didn’t need any new-fangled theory of change to tell us how to get up in the morning. Nonetheless, I hope a few cautionary notes about the new discipline might not be out of place.
I hardly need warn you that advocacy evaluation is extremely complex and difficult to do. Seemingly countless complicated and unpredictable developments impinge on advocacy efforts, not the least of which are the equal and opposite reactions generated by directly contrary points of view.
Particularly perplexing are those moments when the arrows of causality otherwise neatly pointed away from you toward your milestones and objectives suddenly come whizzing back at you, dangerously close to your ear.
I witnessed just such a moment in my time at Bradley, on August 26, 1995. The front page of that day’s Milwaukee Journal Sentinel featured a picture of a dejected Maria Colunga and her two daughters, standing in front of St. Joan Antida High School.
Ms. Colunga had hoped to register her daughters at the all-girls Catholic school with the vouchers she was to receive from the Milwaukee school choice program, which had recently been expanded to include religious schools.
But just days before the new school year was to begin, she and some 2300 other parents aiming to enroll their children at 80 religious schools found someone standing in the schoolhouse door. The Wisconsin Supreme Court had issued an injunction suspending the school choice program.
Somewhere, probably deep within the headquarters of the voucher-phobic Milwaukee Teachers Association, an advocacy evaluator proudly put a large red check next to the chief objective the union had set for itself that year: “persuade court to close voucher program.” Mission accomplished. Surely there could be no clearer example of an advocacy victory than this.
Or was it? Almost immediately, Milwaukee’s private tuition assistance program, Partners Advancing Values in Education, launched an emergency campaign to raise the $1.6 million that would enable effected parents to go ahead and register anyway in the religious schools.
The local newspaper and TV stations, until then critics of school choice, began to report the totals raised each day, cheering the effort on in spite of themselves. One station filmed Ms. Colunga taking a phone call from a donor, pledging personally to finance her children’s tuitions.
Even the New York Times, likewise no friend of school choice, took note of the hubbub. They reported the remarks of Sedgwick Daniels, pastor of the largest African American Church in Milwaukee: “I say this to the civil liberties union and the teachers union: ‘You’re picking on a giant that’s bigger than you are.’”
And this quote from a parent: “When I heard about the ruling last week, I thought, ‘I’m not sending [my daughter] to public school; I’d rather go to jail.’”
Many court rulings, laws, academic studies, and other developments were to follow in the school choice story. But I would have to say that the moral and political momentum shifted suddenly and decisively toward the choice program the day the Supreme Court issued an injunction against it, followed by that vivid, front-page depiction of its effect on Maria Colunga’s family.
What would have to be considered under any known evaluation model as a major loss for school choice proponents was in truth a major win, and vice versa.
Now, one could multiply endlessly the examples of hard, counterintuitive cases hoping to shake the confidence of evaluators, but it would be to no avail. They already know full well how difficult it is to do, and they are not daunted.
They skillfully accommodate complexity by spinning out vast webs of qualifications, codicils and variants within their models, like so many epicycles in a desperate attempt to save the Ptolemaic system of astronomy.
Why such insistence on theorizing, modeling, and measuring? This bespeaks a familial tie to the larger social science project for America, launched a century ago with the full backing of the most powerful foundations of that day. Those foundations – Rockefeller, Carnegie, and Russell Sage – believed that the newly developing social sciences offered a way to redesign public policy on a genuinely rational, coherent, and objective basis.
Professionals trained in those sciences and housed in research universities and think tanks would take policy making away from amateurish politicians and everyday citizens, who tended haphazardly to cobble together laws on the basis of outdated and parochial allegiances to tradition, religion, ethnicity, and local community.
The primary challenge facing the social science project is that it seeks to impose rationality and order upon a political system reflecting the view that human beings are irredeemably irrational and disorderly, at least by professional standards. So believed the nation’s founders, especially James Madison, who constructed for us a large, commercial republic which would encompass so many different and conflicting opinions, passions, and interests that oppressive factions could be avoided and liberty preserved.
The divided and counterpoised systems of government established by the Founders at all levels likewise gave full play to, and thereby nurtured, a vast multiplicity of economic interests, political ambitions, and contending religious sects.
Advocacy evaluation as a discipline is situated precisely where these two conceptions of public policy – the social service state and the Madisonian system – meet, or rather, collide.
Evaluation is expected to understand both systems, and to be able to interpret one to the other. But it is by no means neutral as between the systems, though it seldom reflects on its own implicit political inclinations.
It is in fact a pathfinder or trailblazer for the social service state, aiming to push out the boundaries of professional service provision, subduing and colonizing unruly regions of contentious and irrational Madisonian politics.
If this seems an unfair description, reflect for a moment on the numerous examples of successful advocacy presented in the literature. One bright thread runs through them all: they invariably involve efforts to launch new social service programs, to expand existing ones, or at least to block their reduction.
One searches in vain for an example involving, say, dismantling a service program, cutting spending, or reducing a tax. The unquestioned assumption is that public policy advocacy means expanding the reach of enlightened professionals, bringing ever larger regions under the sway of rational and orderly policy delivery, redeeming them from the amateurish bumbling of unenlightened politicians and ordinary citizens.
Some advocacy evaluators, of course, insist to the contrary that they are by no means hostile to democratic engagement. Indeed, their models focus directly on the need to draw into the evaluation design the people most affected by it. But as Michael Quinn Patton notes in his fine article “Evaluation for the Way We Work,” one reason for that is to teach people “to think and act evaluatively.”
Citizens should be brought into evaluation because this will have a “lasting impact on how they think, on their openness to reality-testing, on how they view the things they do, and on their capacity to engage in innovative processes.”
In other words, evaluators are also evangelists, aiming to reform the thinking and behavior of citizens, making them more like the dutiful rationalists of the social service state, and less like the parochial, erratic partisans of the Madisonian system. And so the social service frontier moves ever inward, as well as outward.
There is, of course, something deeply elitist about the social service project. After all, it suggests to citizens that their interests and ambitions, their religious and ethnic affiliations, their allegiances to family and community, and other outmoded ways of thinking to which they cling, can only obstruct the advance of sound public policy.
Good evaluators should warn clients that difficult moments ensue when those elitist assumptions come to light. History is full of instances when the sound and sober advance of rationalism is abruptly halted by a messy populist uprising, fueled by resentment of such condescension.
The Great Society, for instance, with its echelons of social science theorists, planners, and evaluators, was busy gobbling up vast stretches of public policy, when suddenly the reaction came.
And it came from both the left and the right. The New Left savaged the cold, rationalistic bureaucracies of corporate liberalism in the name of populist, participatory democracy.
And on the right, populist presidential candidate George Wallace attracted millions of Democratic voters with his vow to “throw the briefcases” of those “pointy-headed bureaucrats” into the Potomac, among them no doubt some briefcases belonging to evaluators.
Shrewd advocacy evaluation, we are told, seeks ways to preempt such reaction, by marking out various hidden backroads and byways over which social service expansion can travel, without attracting undue attention from the hoi-poloi.
As the literature suggests, quiet judicial rulings and administrative mandates will do nicely, if the main legislative road is blocked by an unsympathetic democratic electorate.
Mr. Patton reports one instance in which a subtle, foundation-backed advocacy campaign led directly to a favorable Supreme Court decision. But, he notes, we cannot be told which case it is, because the ruling was achieved, as he puts it, by a “stealth campaign, so designed as to avoid arousing strong opposition.”
That is, the campaign’s success hinged precisely on its ability to steer clear of, rather than engage forthrightly in, the open, contentious give-and-take of democratic politics.
The problem with stealth campaigns is that, while they may slip along unnoticed for a while, sooner or later citizens do take notice. And then they’re doubly angry, not only for being treated as too foolish to handle public policy, but also as too stupid to notice they’re being so treated.
Something like this surely explains why the stealth campaign for a state Supreme Court injunction against the Milwaukee school choice program in August of 1995 aroused such widespread anger.
In a region of public policy aggressively colonized for decades by social service providers, school choice stalls the relentless advance of the professionals by returning authority for educational decisions to parents, hopelessly mired though they may be in backward traditions and religions.
When the injunction sought quietly to relieve parents of that authority, as Pastor Daniels noted, it suddenly became apparent that the courts, the teachers union, the civil liberties union, and all their assorted experts had inadvertently awakened a giant.
The stealth campaign to strangle the choice program through judicial mandate had instead provoked the same sort of populist backlash that had doomed the Great Society.
What appeared to be a “win” gained by proven tactics for advancing the social service state had instead proven to be a resounding “loss” in the larger world of democratic, Madisonian politics.
We are now entering a period in American politics, the public opinion experts tell us, when the social service state is likely once again to pick up steam, aiming to subdue more of those obstreperous and rebellious outlying regions of public policy. If so, advocacy evaluation will soon be blazing new trails.
But I do hope at least some evaluators reflect on the strengths and virtues of the Madisonian system, even as they seek to corral it with their orderly theories of change. I suspect they might come to appreciate its indispensability for the preservation of American democratic freedom.