 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
|
|
||
|
||
|
||
|
||
|
||
|
||
|
|
|
| ||||||
Can We Get Beyond Subjectivity and Judgment? Published in various forms by The Institute of Public Policy, George Mason University and the International Food Policy Research Institute as a working paper. "Public policy is made in a political environment. Politics affects, to a greater or lesser degree, what problems are analyzed, who does the analyzing, when and even how it is done, what decisions are made as a consequence, and how those decisions are implemented. Policy analysis must thus cope with politics (E. S. Quade, 373)." What are the "returns" to policy-oriented research in the social sciences? The expected outcome of social science research is typically a public policy program, legislation, rulemaking, or judicial decision. Some level of government in response to this research finding implements a public policy. It results in a new program, the elimination of an old one, a significant change in an existing program, or a significant change in a law or interpretation of law. One presumes that positive net benefits to society or at least a certain segment of society would be "returns," but how does one determine what these benefits are? Clearly there must be benefits of some sort to some social science research because society continues to fund it, albeit at different levels in different places and times. What is it one seeks to measure when one measures social science research? Of course, we wish to measure the application and use of the research findings, but what is it beyond this that interests society? Economic benefits? Quality of life? Knowledge for its own sake? Longevity and health? Our measurement tools may in part be dependent upon what it is we seek to measure. This paper offers a discussion of the issues involved in evaluating social science research and an overview of methodologies that can be used for either or both ex post and ex ante evaluation of the benefits of social science research. Its primary focus will be on the assessment of research that eventually supports a particular program. It should be noted that research must be continuously evaluated from a period prior to the time the actual research is undertaken to a period following the event that a program to disseminate the benefits is put into place. However, this paper cannot fully answer the question of what it is we seek to measure in any empirical sense. The returns (or often the "expected returns") in the marketplace for social science research are those which exist in the eye of the customer who bears the cost of the research. How does the researcher determine the returns to policy-oriented research in the social sciences? First, there should be both ex ante and an ex post evaluations of research. There must be an ex ante evaluation of potential benefits in order to determine whether research should be funded initially. Then there must be ex post evaluations to determine if additional research should be funded (one might also call these ex ante evaluations occurring prior to the next round of funding) and then further ex post evaluations to determine whether the research has paid off or not and whether it should be given additional funds. These further ex post evaluations may be based upon evaluating programs that are put into effect as a result of the research. Since benefits of social science research may accrue quite slowly (as Julian W. Alston, et. al express it in a treatise on agricultural research, "...there are long, variable, and uncertain lags in the interval between commencing a research activity and generating useful knowledge..." (1995, p. 29)), methods of determining whether to go forward with additional research at an early stage (i.e., an initial assessment of benefits from completed research) may be somewhat different than the methods employed when one is able to more fully evaluate the results. However, if one waits until all costs and benefits are fully measured before proceeding with additional funding for promising research, start-up costs must be re-incurred and institutional knowledge may be lost. Program Evaluation Program evaluation, defined by Joseph S. Wholey, et al (1994, p. xiii) as a systematic assessment of the performance, and particularly the outcomes and impacts, of public and nonprofit programs and policies, is a difficult task as it is. As Caroline Wagner and Ann Flanagan (1995, p. v) note in a report on a workshop to assess the means of evaluating federal investments in science, linkages between specific activities and subsequent economic or social outcomes remain very difficult to measure. Furthermore, program evaluation often seems to be much like the weather - everybody talks about the weather, but nobody does anything about it. This systematic analysis is not undertaken nearly as often as it should be, even for the limited purpose of determining whether the program is actually being carried out as envisioned. As Lana D. Muraskin (1993, p. 9) points out, evaluations are not always welcomed, carry risks, use scarce resources, and are often a low priority for programs. Otis Graham (1992) found that, as of 1984, an astounding eighty five percent of the 153 state industrial policy programs established between 1979 and 1982 had not been evaluated by the governments which paid for them. More recent literature in this field suggests that the situation has not changed. Hence, it is obvious that program evaluation has a long way to go before it becomes a routine practice. The evaluation of the research that goes into developing the programs that eventually are implemented is necessarily a step removed from the program evaluation process. It is a rare event that a program is adopted exactly as the original researchers intended. What is more likely is that a program will bear only a resemblance to the concepts that were originally envisioned. A program must pass through the political process before being implemented, while the research may proceed with somewhat less regard as to what is politically feasible. Benefits may be slow to accrue and the "law of unexpected consequences" may produce commensurate costs to go with these benefits. Hence, measurement must always be uncertain. Furthermore, there are often different levels of government involved in policy implementation based upon social science research, not to mention that the research is most often country specific. Interest groups, political leadership, resource availability and institutional capabilities play a role in adoption. The adoption of the research into a program does not mean that the program will be effective even if the original research is sound (Robert H. Wilson, 1993, p. 266-67). The fact that uncertainties abound shall surely not distract from the goal of evaluating social science research. Policymakers, particularly in an era of concern over government spending exhibited in the post-industrial societies and, reflecting a charge from their constituents, demand accountability for the use of scarce funds by those who carry out programs. With this goal in mind, this essay will draw upon various literatures that may be of use in assessing the benefits of social science research. It will propose a limited theoretical framework with which to use to select a method of evaluating social science research. It will explain the political influences, biases, and difficulties in measuring the benefits of social science research. Finally, the paper will propose methods that are used in other evaluation programs, and analyze them as to their efficacy in evaluating the benefits of social science research. Policy-oriented social science research is a "public good. A pure public good has the following attributes: 1) It does not cost anything for another individual to enjoy its benefits (the marginal cost, or the cost of producing one additional unit of a public good is zero), and 2) It is difficult or impossible to exclude individuals from the enjoyment of a public good. The owners of social science research cannot exclude others from its use once the research is disseminated. The marginal cost of producing an additional unit is essentially equal to zero, particularly since research is normally a fixed-cost item. Another way of looking at this is that social science research is said to give off positive externalities. The Literature on Measuring the Benefits of Scientific Research The literature on measuring the benefits of policy-oriented social science research is so sparse as to be practically non-existent. There is no coherent body of theory to draw upon that does not draw heavily from theory established by another field of endeavor. However, the literature on program evaluation (including policy evaluation), research and development (R&D) and basic science research evaluation, as well as a policy analysis literature (public policy process), are available. As Wagner (1995) contends, there is no theoretical framework within which to place quantitative tools for measuring the output of scientific inquiry, although there is a very comprehensive literature which examines the issue of calculating returns to scientific research as well as R&D. These literatures include empirical work as well as instruction as to the methodologies underpinning the work. Some generalized theory is available in the program evaluation, scientific research and policy analysis literature. These literatures will form a large portion of the basis for the recommendations for social science research evaluation theory and practice that are considered in this paper. The Political Setting of Social Science Research Social science research, by its nature, has the goal of influencing policy decisions made by government. Often the very government it is attempting to influence funds it, but this is not necessarily the case. However, due to the public good and positive externality aspects of social science research (which will be discussed in greater detail below), it is rarely funded by purely private, profit-seeking interests, unless these private interests have a fairly good idea as to the results that the research will reach. This is because they ordinarily expect to use the results to lobby politicians. Hence, research financed by private interests (other than foundations and other non-profit organizations) will not be a great subject of concern here, particularly since those who purchase it at least intuitively have some idea as to whether they are getting their money s worth. Let us trace a social science research project from start to implementation. First, the researcher (or research team) is likely to take an interest because existing literature in the researcher's field or public information diffused through the popular media acknowledges that an unsatisfactory situation either presently exists or is projected to develop if not addressed with due diligence. Funding is made available through a government program or private grant and the researcher competes for it. Often there are a number of projects funded, but the paper will focus on one particular project here. The researcher completes the project according to a schedule, and presents the evidence in the form of a report or study to the agency or foundation. The research will recommend a specific project that should be implemented, or at least it is assumed that this is the case here. The study is generally followed by its publication in a journal as well as other announcement to the researcher s professional peers, such as presentation at professional conferences. The popular press may pick it up and diffuse it to the public at large. The next step is that a legislator or government executive, viewing it as something that is attractive to his or her constituents, takes an interest, or a government agency or outside group recommends legislation to carry out the program. The actual details of the legislative process will not be discussed here, but to the extent that the program creates any winners and losers or draws significantly upon the resources of the public, there will be a battle of sorts to modify the program. Legislators will pack the legislation with their own amendments, which are generally designed to create advantages for their constituents or to mitigate damages to their constituents that the proposed program will create. The legislation that finally becomes law may not implement the program in its entirety, or may change the program significantly from that which the researcher proposed. Details of implementation are usually left up to the agency in charge. More changes to the program may come from within the bureaucracy, or as the result of a public rulemaking process whereby interested parties are allowed further input. There are a few evaluation methodologies that focus directly upon research without regard to its results on society at large. However, no matter how vexatious it may prove to be, a results-oriented program to measure the benefits of research must at some point contend with the final product - the tangible results of a program. To sum up, we may chart the usual course of events in social science research as follows: DISCOVERY OF ISSUE ® RESEARCH CONTRACT ® RESEARCH CONDUCTED AND REPORTED ® LEGISLATION PROPOSED ® MODIFICATION OF PROPOSED LEGISLATION ® PROGRAM IMPLEMENTATION ® EVALUATION OF PROGRAM Note that there will be continuous feedback to other parts of the process. In the interest of simplicity, this is not modeled here. We seek to measure the benefits of the research in a quantifiable manner. However, as the chart illustrates, the benefits do not often begin to flow with the performance of the research. More often than not, a specific focus, which takes the form of a program that may be heavily modified prior to implementation, is necessary to produce benefits from the research. The Uncertain Relationship of Research to Policy While policy-oriented social science research rarely, if ever, gets translated directly into policy in its entirety, there are other things evaluators must be concerned about as a prelude to the determination of how to measure its benefits. Assuming that the particular research in question is in fact propitious, evaluators are still faced with a number of problems beyond the control of the researchers. Good research may not become policy because of laws, institutions, prevailing culture and opposition by interest groups or the public at large to the proposed program. As Quade (1989, p. 7) accurately states, "We may be able to design an efficient system on paper, but as of yet we have no algorithms for finding ways to overcome the resistance offered by tradition, legal restrictions, and a host of privileged interests that inhibit radical or even morphological change." Competent research may be (significantly) translated into policy but the underlying conditions that the research is based upon may change. Finally, good research may be translated into policy, but unanticipated consequences may negate any positive policy resulting from the original research. As noted earlier, even assuming the research is of the highest quality and the program captures the essence of the research, there is still the problem of the time path of benefits accrual. Does the research produce immediate benefits which are steady, increase or dissipate over time? Does it take considerable time for the benefits to start accruing? Are there any spillover effects of the research which should also be measured? What About Research That Does Not Get Translated Into Policy? It has so far been assumed in this paper that the purpose of social science research is to create policy. However, it is highly questionable that all or even most social science research finds its way there, at least by a direct route. The National Academy of Sciences devoted a volume to this issue in the late 1970s (Laurence E. Lynn, Jr., p. 1978). Casual observation suggests that it is highly unlikely that a significantly greater proportion of social science research is implemented into policy two decades later. Should research that has been put on the shelf and not used for any practical purpose be evaluated for its benefits to society? Unless its benefits to society can be readily identified by observers (even if it cannot specifically be quantified without further evaluation), or it can be evaluated in a wholesale fashion with similar research (perhaps to gain an understanding of why this research does not appear to have further value), as a general rule, it probably should not be. Resources are scarce. The fact that research does not produce or change current policy should normally be prima facie evidence of the lack of efficacy of the research, even if this evaluation is sometimes unfair. As Harvey Averch (1994) contends, the ultimate social test of information commissioned by investors in research is its third party utility, or the difference the research makes in the behavior or decisions of users. |
 Sign Guestbook |
|
. |
| Visitors: 01616 |