 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
|
|
||
|
||
|
||
|
||
|
||
|
||
|
|
|
| ||||||
| More Direct Methods: User Evaluations User evaluations in the context of social science research evaluation entail surveying the actual users of the research that is implemented by a policy program to elicit information on the value of the research. This is a rather costly form of analysis that may involve conducting interviews, as well as mail or telephone surveys and the development of a survey questionnaire. The metric is ordinal rather than cardinal. As Cozzens (1995, p. 35) points out, there may be an inherent conflict of interest in surveying users, particularly if they receive free or low-cost benefits. They have a vested interest in expressing satisfaction in order to keep the benefits flowing. A survey is highly dependent upon the biases of those surveyed as well as those biases inherent in the questions posed to the sample of respondents (or even the order in which the questions are asked). None of our methods is perfect. However, given its costs, survey methods do not appear to be among the best means of conducting social science research evaluation other than to determine the satisfaction levels of the beneficiaries, particularly if the benefits to a broader society are of greater interest than benefits to a select group. Focus groups, or informal, small group discussions conducted by a moderator with the users of the research, are another means of user evaluation. This is somewhat less costly than survey methods, but selection of participants is not necessarily random and the output is often difficult to present in a systematic fashion (other than an account of the reactions of the group). This technique is probably better reserved for the realms of program evaluation and customer service, when the provider wishes to improve delivery of its product or service rather than in the evaluation of social science research. Benefit/Cost Analysis or Measurement of Social Rates of Return Benefit/cost analysis, which reached its peak of popularity in environmental studies, seeks to measure the benefits and the costs of a given project in a systematic, rational manner. The logic underpinning benefit/cost analysis is quite simple. One calculates the benefits of a project or program in monetary terms. Then one calculates the total costs and compares the two. Alternatively, one may calculate and compare marginal costs and benefits. Financial methods such as the internal rate of return (the discount rate that makes the present value of estimated benefits equal the present value of estimated costs) may be used in conjunction with this methodology in order to calculate the benefits (assuming benefits outweigh the costs) of a project. While this is rudimentary enough in principle, it poses great challenges in practice. Quantifying costs and benefits may prove elusive or incomplete at best. Particularly thorny is question of determining the value of human life and health (Weiss, 1978, 84-88). There is often someone who asks the inflammatory question, "how can we put a value on a human life?" in order to put those performing the analysis on the defensive (even though society makes these judgments all the time). The analyst makes judgments and critical assumptions are buried in the calculations (Quade, 1989, p. 225). James Edwin Kee (1994, p. 487) labels the practice of evaluators' attempts to hide the messiness of the process from the decision makers the "black box syndrome." Furthermore, the process is prone to manipulation by those who are determined to reach a particular outcome. The discount rate is never known with certainty and may be subject to manipulation (See Leonard Merewitz and Stephen H. Sosnick (1971) for an account of how discount rates may be manipulated to reflect the views of those doing the evaluation). Finally, externalities are difficult to estimate with simple benefit/cost approaches (Roger R. Stough and Piet Rietveld (1996, p. 12)). While formal benefit/cost analysis has lost much of the glamour it had two decades ago, it is still a useful tool in policy-oriented research evaluation as long as the premises upon which the evaluator makes his or her judgements and calculations are clearly spelled out. It should be pointed out that it might prove difficult to separate the results of the research from the results of the implementation program, which usually modifies the research in practice. Furthermore, while formal benefit/cost analysis may have lost some of its luster, all methods of evaluation in the final analysis require the analyst to make a determination of benefits versus costs. Statistical Methods: Regression Analysis Statistical regression analysis takes the form of the standard regression equation Y = a+ bC + m where Y is the dependent variable, a is a constant, C represents the independent variables with coefficient, b , and m is the error term. The method fits a line or curve by the statistical means of least squares. It is a well known, but intricate, statistical technique. This paper will not attempt to serve as a training manual for its general use, but will instead discuss its possibilities for the purposes of evaluating social science research. Popper (1995) refers to analyses conducted upon R&D using regression methods as production function analyses, since they involve the use of production functions in a regression equation. This method is as follows. The engineering and economic concept of a production function is simply the change in output due to a change in an index of inputs. The empirical production function is estimated by the specification of various models. Further technical details will not be presented here. Popper (1995) contains an adequate discussion of the techniques for those who are interested. This methodology, generally without the inclusion of a human capital variable, has been used with reasonable success in R&D research by employing aggregate data. Assessing the benefits of specific research by use of production functions may prove to be highly elusive, particularly in social science research. There is an advantage in using it to assess the benefits of R&D in private markets that does not exist in public markets - the existence of measurable input to firms and measurable output in the form of goods and services produced. However, it is possible to roughly model the results of social science research in the following manner: use a time series of program budgets minus personnel costs as proxies for capital costs, personnel costs as proxies for labor, and the variable that the research is designed to act upon as the output variable. In particular, one may be quite interested in determining whether certain social science research leads to increasing returns to scale, or the situation whereby a doubling of inputs leads to a more than doubling of outputs. Regression analysis does not need to take the form of a production function in order to be useful in evaluation practices. Furthermore, while it is not guaranteed that this will be possible in every case, one may be able to partially isolate the original research from the program by proper selection of variables. The pitfalls of regression are that it is a modeling technique that must relinquish rich details in order to create a workable model, the model may be mis-specified (through faulty judgement and subjectivity), and it may create a false sense of quantitative certainty. As in any analysis, the evaluator must be clear as to the variables selected and the reasons for their selection. However, this is likely to be a workable means of evaluating the benefits of social science research in certain cases. Operations Research Modeling Methods Operations research modeling techniques, which were developed in the 1940s, have also been used in assessing programs for a number of years. Linear programming is a quantitative methodology that makes use of the mathematical modeling of constrained optimization. An optimal solution is sought, given certain constraints imposed by the model. A number of methods are available beyond simple linear programming. These include goal programming and data envelopment analysis (DEA). Goal programming may be traced to a paper by A. Charnes, W. W. Cooper, and R. Ferguson (1955). It represents an attempt to put the concept of satisficing into operations research modeling. The technique puts goals and aspirations (which may not necessarily be achieved) on one side of the equation and constraints that must be satisfied on the other (Carlos Romero, 1990). Priorities of individuals and groups may be taken into account by means of goal programming (this cannot be done in traditional linear programming models). While this method does not appear to receive much discussion in the evaluation literature, perhaps because it is associated with applied science and engineering, it is likely to hold some promise as a quantitative methodology for the purposes specified in this paper. Selecting the researcher s goals, using a goal programming methodology rather than selecting the goals of the program which may offer divergent goals as discussed earlier, may be a means of isolating the effects of the research from the program. This method should receive further study. DEA contains the concept of the production function that was mentioned in the section on regression analysis. Rather than fitting a line in a least squares fashion, it fits a boundary that contains all points within it (the production possibilities curve in economics jargon). This method is likely to be successful in those cases where production function methodology is deemed appropriate. The use of both methods will increase the rigor of the study, but, of course, this has a cost. Simulation: A More Recent Development in Operations Research Simulation is a heuristic, probability-based methodology that conducts experiments by proxy using a model and runs that consist of selecting values of uncontrolled variables. This is accomplished by using Monte Carlo analysis, or repeated simulation of a stochastic model to investigate the properties of statistical techniques applied to it (John G. Cragg, 1990, p. 171). An abstract model, but one that contains enough stylized facts to approach authenticity, is necessary for simulation. This technique has a long history in applied sciences, particularly in defense work. Gary D. Eppen, et al (1988, pp. 674-75) list a number of advantages and limitations of simulation for modeling in general. Some of its advantages are that simulation can deal with uncertainty, it is versatile because it can be run many times, it can reproduce variability that actually occurs in the system being simulated, multiple objectives can be analyzed, and it is a rational tool. The limitations are that there may be a temptation to build a model that is too big and too expensive, and one could proceed with the modeling process only to find out that no one understands the interactions of the variables well enough to build a workable model. Judgement and elements of subjectivity enter into specifying the model as they do in other methodologies. Finally, it must be said that simulation is based upon limited real world data rather than the analysis of historical processes using only knowledge of actual events. Simulation has been most heavily used in the realm of the applied sciences. It may prove to be useful in social science research evaluation when data are hard to come by or one suspects processes are at work that are extraordinarily difficult to model by conventional means. Simulation is currently being used in actual social science research. Two examples are Richard R. Nelson and Sidney G. Winter's (1982) modeling of evolutionary change and the sugarscape model of Joshua M. Epstein and Robert L. Axtell (1996). The latter attempts to model social science systems from the ground up. Additional simulation models involving the use of genetic algorithms, nonlinear regression, and other modeling techniques are currently being developed in the growing field of computational economics (see Manfred Gilli, 1996). Simulation is quite useful to determine system sensitivity and the range of intervention options - perhaps this could be termed scenario analysis. Of course, the major problem with this method is that it is not based upon actual observed data, although the models may take actual data into consideration when generating scenarios. Hence, our output is more along the lines of potential reality rather than measurement of actual circumstances. Conclusion This paper has discussed the evaluation of policy oriented social science research by reference to literature in other fields. It has outlined a rudimentary theory to be used in the application of evaluation methodologies. A number of methods to accomplish a benefits analysis of social science research have been offered and discussed above. Most of these methods are adopted from other literatures such as program, science and R&D evaluation and policy studies. Many are well-known research methodologies that are used in the completion of social science research as well as evaluation in other fields. This paper advises that replication should be conducted as part of the evaluation process prior to reaching the question of whether the research is beneficial (although evaluators and their clients will have some prior belief as to whether the research has produced benefits before they conduct any analysis of any kind). None of the methods presented as potential tools for carrying out evaluations is completely satisfactory insofar as answering our theoretical needs. A mix of methods may prove satisfactory in many cases. However, the link between research and application of results is likely to remain tenuous. This means that an investigation into the benefits of social science research will likely remain an exercise in muddling through. Decision-makers will find it necessary to select among the evaluation methodologies, or have the evaluators propose methodologies that they believe offer the best answers to evaluating social science research in particular cases. In completing the loop, perhaps one could proclaim that more research is needed to determine how best to measure the benefits of social science research. However, research efforts into improving research benefits analysis should at most focus upon improving existing methods, particularly given that the cost of processing large quantities of data have dropped significantly and will continue to drop. In particular, operations research methodologies appear ripe for adaptation to social science research evaluation. It is known from the literatures of program evaluation, policy analysis, and scientific and R&D research evaluation, that there is unlikely to be a single quantitative solution, that stands head and shoulders above the rest, waiting to be discovered if only society pours enough resources into the effort of unearthing it. Greater results are likely to be obtained if programs to evaluate research are implemented rather than if we consume time and resources looking for better means of evaluation. This does not mean that we should not seek to improve our evaluation methods. The striving for better and better tools is innate - humans are tool-building animals. However, we should not let the perfect be the enemy of the good by striving to perfect our methods while passing up opportunities to put the tools we have into practice. ENDNOTES The program may be anything from a low cost educational program announcing the benefits of certain actions, such as eating more fiber, to a social welfare program costing billions of dollars per year. Of course, not all social science research results in governmental action, but much of it that interests us does. A related paper that emphasizes the evaluation of tools used in the evaluation of social science research has been submitted to the International Food Policy Research Institute for inclusion in a monograph. Of course, this may be partially due to the less legitimate reasons for evaluation that Carol H. Weiss (1972, pp. 11-12) establishes. They are 1) postponement, 2) ducking responsibility, 3) public relations, and 4) fulfilling grant requirements. It is essential that the evaluation of policy-oriented social science research also avoid these reasons when the decision is made to evaluate the benefits of research in order to provide credible evaluations. See Robert H. Wilson (1993). The program evaluation literature suggests the same is true in general. The U.S. General Accounting Office (GAO), an arm of Congress, does program evaluations as Congress calls for them. Given its institutional placement, typically it is reactive rather than proactive, and examines problems after there has been an embarrassment or scandal. Privately funded organizations that evaluate programs also exist, but they quite often have a particular point of view to represent. This is not to say that the political process does not influence the research though. Grants to perform the research may come from budgets that have been shaped by the political process. Furthermore, requests for proposals also may reflect the influence of the political order. See Caroline S. Wagner (1995, p. 1) for a listing of new U.S. legislation requiring increased accountability. In addition, The Gore Report on Reinventing Government (1993) gives additional impetus to a greater accountability from those who expend resources on behalf of the public at the federal level in the U.S. Joseph Stiglitz (1986, pp. 87-89) has an excellent discussion of public goods. Assuming the research has positive net benefits associated with it, which one anticipates to be the case in any research that others might wish to use. See Julian W. Alston, et al (1995) and Paula E. Stephan (1996). Sometimes research is funded indirectly by government through budgets of state-supported colleges and universities. This is not always the case. Private interests may donate resources for research directly out of a sense of civic duty, but generally this money is funneled through established charities and foundations. Furthermore, this should not be taken to mean that the interests of governments, foundations and non-profits are pure, while private interests are not. Any organization may have a particular outcome in mind when it funds research. Often research is undertaken that points to a need that a government agency is authorized to carry out without further legislation, but a budget is needed or resources must be shifted. We will assume that legislation is necessary in this case. "Program" is used in a very broad sense here. A program may consist of de-funding existing programs or a different interpretation or application of legal principle. As is true in the field of physical and mathematical sciences, there is some social science research that is undertaken as fundamental research, whereby the goal is to acquire basic knowledge. This research is among the riskiest in the portfolio, with uncertain payoffs that tend to accumulate in the long term if at all. The benefits of this research are extraordinarily difficult to calculate, but the end result of the research should result in programs. Of course, there is the question of how many frogs must be kissed before one of them turns into a prince, since more failure is expected at the riskier end of a portfolio. However, given that government bureaucracies hire social scientists as employees and these social scientists are familiar with the literature in their respective fields, it appears that much of the research that is deemed respectable is likely find its way to the policy arena eventually. Thus, social science research is often used as ammu |
 Sign Guestbook |
|
. |
| Visitors: 01522 |