How Widely is Blind Assessment Used in Scientific Research?

RESULTS AND DISCUSSION
The widespread neglect of possible experimenter effects
The use of blind procedures in different branches of science gives a measure of the importance researchers in that field attach to experimenter effects. In Table 1, I summarize the results of a survey of papers published recently in a range of scientific journals. In the physical sciences, no blind experiments were found among the 237 papers reviewed. In the biological sciences, there were 7 blind experiments out of 914 (0.8%); in psychology and animal behavior, 7 out of 143 (4.9%); and in the medical sciences, 55 out of 227 (24.2%); By far the highest proportion, 23 out of 27 papers (85.2%), was in parapsychology.

In the medical journals, out of the 55 reports involving blind methods, only 25 (11.0% of the total) represented double-blind trials. The other 30 employed single-blind methods, with one or more of the investigators carrying out blind evaluations or analyses. The majority of the papers did not involve blind methods.

Confirming the findings of the literature survey, the survey of science departments at 11 British Universities confirmed that blind procedures are rare in most branches of the physical and biological sciences. They were neither used nor taught in 22 out of 23 physics and chemistry departments, or in 14 out of 16 biochemistry and molecular biology departments (Table 2). By contrast, blind methodologies were practiced and taught in 4 out of 8 genetics departments, and in 6 out of 8 physiology departments. In most of these departments they are used occasionally rather than routinely, and are mentioned only briefly in lectures.

When academic scientists were interviewed for this survey, some did not know what was meant by the phrase "blind methodology". Most were aware of blind techniques, but thought that they were necessary only in clinical research or psychology. They believed that their principal purpose was to avoid biases introduced by human subjects, rather than by experimenters. The commonest view expressed by physical and biological scientists was that blind methodologies are unnecessary outside psychology and medicine because "nature itself is blind", as one professor put it. Some admitted the theoretical possibility of bias by experimenters, but thought it of little importance in practice. And one chemist added, "Science is difficult enough as it is without making it even harder by not knowing what you are working on."

Only in exceptional cases are blind techniques used routinely. This survey revealed 3 examples. All 3 involved industrial contracts, according to which the university scientists were required to analyze or evaluate coded samples without knowing their identity.

Limitations on the use of blind methodologies
In the biological and physical sciences, the fact that almost no published research involves blind techniques reflects the fact that researchers, reviewers and journal editors in these fields assume they are unnecessary. They are not part of their scientific culture. There may be many situations in which they would be desirable and informative, but practically no attention has yet been paid to this possibility.

By contrast, blind methods are part of the culture of medical research, both conventional and unconventional. In this context, the publication of so many papers in mainstream medical journals that do not involve blind techniques indicates that these methods are not always appropriate or applicable. I have not carried out an analysis of the situations where blind methods are not used in mainstream medical research, because my primary focus was on the biological and physical sciences. But an analysis of this kind would probably be very illuminating.

One kind of medical research not involving blind methods is the study of case histories. In a recent paper in this journal, Lukoff and colleagues (note 6) observed that: "The highly regarded randomized controlled clinical trial, though often powerful and useful, is neither feasible nor ideal for understanding the effects of many unconventional treatment approaches". My survey suggests that blinded, randomized, controlled trials are also neither feasible nor ideal for studying many conventional treatment approaches.

A SIMPLE EXPERIMENT TO TEST FOR EXPERIMENTER EFFECTS
Although most "hard" scientists take it for granted that blind techniques are unnecessary in their own field of study, this assumption is so fundamental that it deserves to be tested empirically.(note 7)
In all branches of experimental science we can ask: do the expectations of researchers introduce a bias, conscious or unconscious, into the way they carry out the experimental procedures, make their observations or select data?

I propose the following procedure to test this matter. Take a typical experiment that involves a test sample and a control - for example the comparison of an inhibited enzyme with an uninhibited control in a biochemical experiment. Then carry out the experiment both under open conditions, and also under blind conditions, in which the samples are labeled A and B. In student practical classes, for instance, half the class would do the experiment blind. The other half would know which sample is which, as usual.

If no significant experimenter effects are found in such tests, then for the first time there will be evidence to support the belief that blind techniques are unnecessary. On the other hand, significant differences between the results under open and blind conditions would reveal the existence of experimenter effects. Further research would then be needed to find out whether the experimenters' expectations were influencing experimental systems themselves, or merely the way that the data were recorded or selected.

The more independent investigations, the better. It cannot be healthy for the supposed objectivity of regular science to rest on untested assumptions.(note 8) This is an inquiry in which the critical skills of skeptics could play a major role. The use of blind methodologies, pioneered by skeptics in the field of unconventional medicine, has now been internalized within medicine and psychology, resulting in improved rigor and a more sophisticated awareness of the effects of experimenter bias. The so-called hard sciences have largely escaped skeptical inquiry, but there seems no good reason why they should continue to be granted this immunity.(note 9)

Perhaps it will turn out, after all, that "hard" scientists need not bother with blind techniques. They may indeed be exceptions to the principle that "our beliefs, desires and expectations can influence, often subconsciously, how we observe and interpret things." On the other hand, they may be like everybody else, including researchers in psychology and medicine.


Acknowledgments
I am grateful to Dr Amanda Jacks for her help with the literature review and to Jane Turney for carrying out the university survey. This work was supported by the Institute of Noetic Sciences, Sausalito, CA and the Lifebridge Foundation of New York.

References
1. Mussachia, M. Objectivity and repeatability in science. Skeptical Inquirer 1995; 19 (6): 33-35, 56.
2. Rosenthal, R. Experimenter Effects in Behavioral Research. New York: John Wiley; 1976.
3. Roberts, A.H., Kewman, D.G., Mercier, L. & Hovell, H. (1993) The power of nonspecific effects in healing: implications for psychosocial and biological treatments. Clinical Psychology Review 13, 375.
4. White, L., Tursky, B. & Schwartz, G. (eds) Placebo: Theory, Research and Mechanisms. New York: Guilford Press; 1985.
5. Kaptchuck, T.J. Intentional ignorance: a history of blind assessment in medicine. Bulletin of the History of Medicine 1998; in the press.
6. Lukoff, D., Edwards, D. and Miller, M. The case study as a scientific method for researching alternative therapies. Alternative Therapies in Health and Medicine 1998; 4: 44-52.
7. Sheldrake, R. Seven Experiments that Could Change the World. London: Fourth Estate;1994.
8. Sheldrake R. Experimenter effects in scientific research: how widely are they neglected? J Sci Exploration 1998;12(1):73-78
9. Sheldrake R. Could experimenter effects occur in the physical and biological sciences? Skeptical Inquirer 1998;22(3):57-58