After the first 12 years and $31-million investment in developing the nonexperimental research methodology, the contractor reported blandly, the project could not "guarantee that a reliable nonexperimental evaluation will be possible." So, at that point, all of the effort so far might have been fruitless.
When I heard the contractor present this proposal at a USDA meeting this month, I expected the officials at FNS to sputter and guffaw at the implausible time line and budget (several times the total annual food assistance research budget at USDA). Instead, the FNS officials smiled and nodded and struck a pose of reasonable and deliberative skepticism.
There had to be a backstory.
Under the new banner "Expect More", the federal government's Office of Management and Budget (OMB) publishes controversial evaluations of federal government programs. These evaluations, which are sometimes ridiculed even by leading experts within OMB, are widely perceived as a political exercise to justify imminent White House proposals for program cuts.
In the world of food assistance programs, the National School Lunch Program received the dreaded classification, "results not demonstrated." The larger Food Stamp Program fared much better, earning a classification from OMB of "moderately effective" -- this rating despite the fact that the program's goal is to reduce food insecurity and hunger, and federal statistics show the prevalence of these social ills have been increasing during every year of the current administration.
When I first read the favorable rating of the Food Stamp Program on OMB's predecessor to the "Expect More" website a couple years ago, it occurred to me that the political folks in OMB may have been too cowardly to state the obvious: the current food assistance programs are not sufficient to halt the increase of hunger in America. Far from justifying program cuts to the popular Food Stamp Program, the confession of failure implied by an OMB rating of "results not demonstrated" might have convinced the American political establishment to strengthen these programs.
Or worse (from the administration's perspective), if food stamps were declared officially ineffective, somebody might eventually have asked whether any program that merely distributes food benefits would suffice to stop the rise of hunger in an economic and political environment where increasing wealth for the few and deteriorating welfare for the many are accepted with complacency. Maybe hunger simply follows hopelessness as naturally in real life as it does in the dictionary.
But back to my story.... I had missed reading the important small print in OMB's assessment of the Food Stamp Program. Cognizant of the statistical ironies in its evaluation, OMB instructed USDA's Food and Nutriton Service (FNS), which oversees federal food assistance programs, to develop new studies "to demonstrate the impact of program participation on hunger and dietary status."
One imagines that OMB staff expected FNS to propose a scientific evaluation, perhaps with random assignment of program participants to several variants of the program, so that one could be sure the program parameters -- rather than confounding variables -- were responsible for the observed outcomes. However, it is almost a matter of religion at FNS that random assignment research designs are illegal or unethical (I doubt both points, while admitting that some such designs are illegal or unethical, and will continue to post on this topic in the future. I know that the same contractor has in the past articulated the logic in favor of serious random assignment research designs for such programs).
FNS gave the task of developing a research agenda in response to a contractor, Abt Associates, with a long history of competent research on food assistance programs. FNS and a committee of outside experts both gave the contractor the self-contradictory instructions that it could not rely on random assignment research design, and yet it had to take the problem of confounding variables in nonexperimental research very seriously. Only in this light can you see how the contractor could see its resulting proposal as reasonable. If random assignment is off the table, and yet the nonexperimental research design must solve the tough challenge that random assignment solves, one would have to first spend an immense effort -- 12 years and $31 million, as it turns out -- just to make sure the nonexperimental approach works properly. See the figure below for an illustration of the research plan's many individual studies.
So, that explains the contractor's astonishing proposal, but what explains the cheerful response from the FNS officials who requested the report? None of the FNS officials said it explicitly, but I doubt a single one of them believes this research plan will be implemented.
There may have been two reasons they liked the report. First, everybody praised the report for laying out the research issues clearly and honestly.
Second, the report may have served a rhetorical purpose in FNS' dealings with OMB. By publishing the report on the FNS website, they are saying to OMB: "You want a study of the impact of food stamps? Here you go. Let us spend $41 million chasing this one down, and we'll give you the answer in 17 years. Then, if the results are unfavorable, we can start discussing program cuts at that time."
No comments:
Post a Comment