 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
|
|
||
|
||
|
||
|
||
|
||
|
||
|
|
|
| ||||||
| Biases in Social Science Research Social science is highly value-laden. While there do exist questions in the arena of the social sciences that are relatively value free, they are of far less research interest than those that are value-laden. Basic research assumptions are often highly subject to whatever school of thought the researcher subscribes to, which, in turn, is dependent upon a particular political belief or value system. Furthermore, research, being a human endeavor, is subject to bias, both of the avoidable and unavoidable nature. The legal phrase, conflict of interest, is most likely to come to the lay person s mind when she or he encounters information on research bias. That is, the researcher may be overly concerned with the likelihood of the research discovering exactly what the financier of the research wishes to find rather than determining the correct solution to the research questions (if one correct solution indeed exists. This is not necessarily the case in social science research). Another variation of conflict of interest is that in-house analyses may favor solutions that confer more resources upon the analysts. Both Paula E. Stephen (1996) and Susan Feigenbaum and David M. Levy (1996) cite pursuit of fame as another source of research bias. Publication bias, or the favoritism shown by editors to research that demonstrates a large positive effect of a given experimental approach, often affects research by encouraging scientists to emphasize efforts on new approaches rather than the perfection of old methods (Feigenbaum and Levy (1996), David Card and Alan B. Krueger (1995, pp. 238-39), Colin B. Begg and Jesse A. Berlin (1988)). Data mining, or the act of fitting as many alternative equations as there are alternative subsets of potentially explanatory variables, and then picking the best data according to the researcher's criteria invalidates conventional statistical tests (Frank Denton, 1985). However, those who review or evaluate the research do not necessarily know if data mining has taken place. A more serious form of data mining is the reporting only of results that the researcher wishes to report, while earlier data runs that may refute the hypothesis being tested are left on the cutting room floor. There is simple careless, haphazard, and hurried research to contend with. There is even fraudulent research and the manipulation of numbers, which one hopes is a rare exception. Furthermore, a particular research project will require judgments to be made about the direction the research will take at critical paths when options are available. Why does bias matter in determining how social science research should be evaluated? It has been noted that our evaluation tools are inexact, benefits may accrue slowly, and social science is value-laden. In early stages of evaluation, an individual makes inferences as to whether the research will be productive without programs in place to produce tangible results. Even when there are programs in place, the subjective nature of much of social science research carries over to our measurement tools and requires that we be alert to bias in the research. Designing Social Science Research Evaluation A design is necessary to measure the benefits and otherwise evaluate social science research. Furthermore, while it may be intuitively obvious to many readers, it should be recognized that the analysis involved in determining the benefits of social science research might also be characterized as research of sorts. In carrying out social science research evaluation, one must first determine the purpose of the analysis. Will it be conducted to determine if the research is profitable for society, or at least for the financier? Is it to decide whether further research in the field should be funded? Next, a clear, precise and manageable process designed to elicit information about the research that is to be analyzed needs to be created (Quade, 1989, p. 137). Following this, the analysis must clearly define exactly what is to be measured. It has already been pointed out that research is but one step in a process. Hence, the actual research must be isolated from the program that implements it. The subject research must also be separated from related research, as well as the rest of the surrounding policies and processes in order to analyze it effectively. Furthermore, the question of whether the evaluation analysis is to cover one particular item of research, or will it analyze a group of projects in tandem, must be answered. If the target research is a group of projects, it must be determined whether one evaluation methodology or one mix of methodologies will serve to measure the entire range of research. After isolating the research, a metric must be designed in order to measure or rank it against a scale of sorts. The users of the research must be clearly defined. While this may seem obvious, there may be multiple users and spillover effects. Furthermore, as David M. Levy and Nestor E. Terleckyj (1996, p. 7) point out, new knowledge may enhance the value of old, ignored, used up or simply unrecognized research by lending new life to previous research results. This effect, if it is present, should be captured in any measure of benefits. How rigorous must the analysis be? As Kathryn E. Newcomer, et al note (1994, p. 1) rigor in program evaluation can be very costly to achieve. The same is true in measuring the benefits of social science research. Hence, the rigor of the specific analysis should be defined prior to undertaking the analysis. An empirical method of analysis that will accomplish the above program must be chosen. The analysis must be carried out according to procedure without bias (it is well known that this is possible only in theory), and results of the analysis must be reported clearly and precisely. This report must also specify the limitations of the evaluation methodology that is used. With this framework in mind, let us proceed to the nuts and bolts of the evaluation process. It should be noted that the available methodologies to be discussed generally fall short of the theoretical mark in actual operation. The ability of an evaluator to isolate a particular body of research is questionable. Rigor is a matter of judgement. In fact, the evaluator's judgement, and sometimes the judgements of those the evaluator relies upon, is a crucial element of every methodology that is available. The First Step in the Process: Replication of Original Studies While replication is used extensively in science and technology research, its use in social science is paltry at best. For one thing, replication is much more difficult in the realm of social science where controlled experiments are generally not available as they are in the purer sciences. "Reality" within the realm of the social sciences is a social construct, it is unstable, and it is subject to continuous change. It is more dependent upon culture than is pure scientific research. The fact that the social sciences are value-laden makes the task of evaluation even harder. One might fairly ask why replication (as well as other indirect tools such as bibliometrics) should be used in a program to determine the benefits of social science research. In fact, one might argue that replication is further research rather than research evaluation. As already noted, benefits may accrue slowly, one needs means of determining whether to fund additional research, bias may exist and social science is value-laden. Methods are needed to assure that social science research that at least initially appears to be beneficial is correctly reporting results prior to program development and prior to further evaluation of the research itself. Replication is often left to graduate students in the social science arena, if it is even done at all. Social science journals apparently do not wish to use scarce space for replication studies (Feigenbaum and Levy, 1993, 1996). The system of professional honors and rewards in the social science system does not place a high value on replication. Weiss (1978) maintains that repetition of results is the basis of scientific generalization, yet at the time she wrote this, replication was a relative rarity in evaluation research. It continues to be. As further argued by Feigenbaum and Levy (1990, 1993), the ability to reproduce a researcher's methods in order to validate results is a clearly recognized requirement of scientific inquiry and is de rigueur in the sciences. Hence, it should become a part of social science research evaluation as a first step in establishing the benefits of a particular work or group of studies. The simplest replication is to repeat the experiment exactly using the same data set. This is often possible in the realm of social science research where one performs statistical analyses on a particular data set consisting of historical events. One may also consider multiple cases as one would consider multiple experiments (see Sharon L. Caudle in Wholey, et. al, (1994, p. 91)). In this case, literal replication is defined as the selection of a case so that it predicts similar results. Theoretical replication produces contrary results but for predictable reasons. The purposes of conducting replication studies are several. First, they provide useful checks on the work of the original researchers, particularly to provide evidence as to whether the procedures were carried out as described in the replicated paper. Second, a number of different procedures are used in addition to the ones originally used. This provides evidence as to whether different methods produce different results. If they do not, the study is replicated successfully. If they do produce different results, judgements are necessary as to why this is true. There is another reason for replication. Evaluating the initial research is a futile exercise if there are major problems with the original inquiry. Hence, it is useful to carefully examine the work for accuracy before proceeding with further analysis. As Feigenbaum and Levy also point out, replication is sensitive to the provision of the data to the replicator by the researcher either formally or by publication of the sources with the original study. Hence, it is good research management practice for those who contract for and publish research to require that data be published or made available for replication. Most Evaluation Methods Are Highly Judgmental and Difficult to Quantify Having identified replication as the first necessary step in the evaluation process, a number of potential methods of evaluation, mostly taken from the program and R&D evaluation literature, will be discussed below. These methods are not equal; some are better than others for particular tasks. Some of these methods will be rejected for most uses in evaluating social science research. Why should the paper present methods that, in the opinion of the author, should be rejected? They are in the literature and should be discussed lest it be thought that they were ignored or forgotten. Most of the methods available are highly judgmental, subjective and offer little in the way of quantifiable results that will satisfy those who wish to have a simple metric by which to accept or reject. This is unavoidable with the tools that are available to us. Furthermore, it is unavoidable in the original research we seek to evaluate. Bibliometrics Bibliometrics, or the measurement of published materials such as citations and publication counts, is a tool that is often used as a proxy to measure returns to R&D spending and evaluation of scientific programs. It has been used since the turn of the century (Julia Melkers, 1993, p. 45). E. Garfield (1963) and de Sola Price (1961) pioneered the modern use of bibliometrics. Susan Cozzens (1995, p. 26-27) writes that this method is used extensively in the British university system as well as the U.S. National Institutes of Health and National Science Foundation. This methodology is useful in showing which research others believe to be important since citations are an indication of a researcher's familiarity with a particular literature and the relevance of a particular article or study to a researcher s own work. There are a number of methods used to perform an analysis of citations. These range from simple counts, to linking citations to each other. However, bibliometrics suffers from a number of flaws. The esteem in which a particular author holds a paper in which he or she cites cannot be garnered from references of citations. Bibliometrics cannot measure quality. As Popper (1995, p. 10) points out, the discredited cold fusion paper of Pons and Fleischmann was among the top 10 of the most heavily cited papers in the scientific literature for several years. The quality of the article in which the citation appears cannot easily be measured. Some disciplines are more publication oriented than others (Cozzens; 1995, p. 27). Finally, this analysis cannot measure benefits to society at large from the research undertaken. One of the suitable characteristics of bibliometrics, from the standpoint of our evaluation methodology, is that it imposes measurement upon the original research rather than the programs that may result from it. However, given its flaws, it may be useful as a supplement to other methods, but cannot stand alone as a method for evaluating social science research. Similar Peer Review Measures Measures of rating research that are similar to bibliometrics include convening panels of experts for evaluation based upon peer review (the most widely used methodology in the world for research evaluation, according to Cozzens (1995, p. 29). It is the earliest method used - it was first used in 1665 - according to Barry Bozeman (1994, p. 80). Peer review relies upon the expert judgement of colleagues within the same field. Many R&D laboratories as well as government agencies use it. In fact, social science research contracts are likely to be awarded as a result of peer review in the competition process. There are a number of different forms of peer review. The reader is referred to Bozeman for a review of the different methods as well as guidelines for enhancing this evaluation. The results of these measures are highly dependent upon the choice of reviewers. Furthermore, this methodology does not satisfy the yearning for a less judgement-oriented, more direct and more empirically based means of measurement. Peer-based appraisals such as enumerating awards and other recognition from professional societies, and patent counts (which have very limited, if any, use for social science research) provide a quantitative means of measuring output, but not a scalable one. Peer review methods may be used along with other methods, but cannot stand alone as satisfactory means of measuring social science output. Case Study Methodology Case study methodology makes use of historical comparative data to tell a story that is much richer in detail than statistical analysis, which by its nature must be somewhat abstract. Yin (1989, p. 23) defines this methodology as follows: A case study is an empirical inquiry that (emphasis in original): Investigates a contemporary phenomenon within its real-life context; when the boundaries between phenomenon and context are not clearly evident; and in which multiple sources of evidence are used. Case study methodology is a viable tool for the evaluation of social science research if it is carefully applied. For one thing, the evaluator can tell a story of how the research is translated into policy and how the policy then carries out the goals of the research. While modeling and advanced statistical methods are rarely used in the context of a case study, statistical data in the form of charts and tables have long been deemed necessary in order to present a successful case study (Robert T. Davis, 1955). Case study methodology can be qualitative, quantitative or both. Yin (1989, pp. 21-22) notes that the primary weaknesses of the case study are that there may be a lack of rigor in the study, they may provide little basis for scientific generalization, and they can often take too long and result in massive, unreadable documents. Gordon Kingsley (1993, p. 36) lays out means of overcoming these weaknesses. As modified for use in social science research evaluation, his recommendations include being specific about the research question that is asked, and developing a rationale for the methods used and how these methods address the specific evaluation needs. He also suggests examining both the successful and unsuccessful projects in case studies. As noted before, this may use up resources that could be put to better use in evaluating research that at least gives the first impression of being successful. Delphi Method The Delphi method, first used in an attempt to improve betting scores at horse races in 1948, was developed at the Rand Corporation in the 1950s and 1960s as a method to increase the accuracy of forecasts. It involves the picking of an expert panel, which is surveyed through a questionnaire. The questionnaire is evaluated then returned to the experts with the findings presented statistically. They are allowed to change their opinions to reflect new information and the judgements of their colleagues (however, the experts are not allowed to have contact with each other). This process goes through a number of iterations until opinions converge. One problem with this methodology is that convergence can be dependent upon peer pressure towards conformity rather than rational processing of information. Fred Woudenberg (1991) presents a rather negative evaluation of the method. It appears that the Delphi method presently offers little for social science research analysis that cannot be gained through use of another methodology. |
 Sign Guestbook |
|
. |
| Visitors: 01663 |