Reproducibility

Reproducibility is one of the main principles of the scientific method, and refers to the ability of a test or experiment to be accurately reproduced, or replicated. The term is closely related to the concept of testability. A related concept is the need for measured parameters to have an operational definition, that is, that the parameter or component of a theory have a defined procedure for its measurement.

Reproducibility of experimental data is not an absolute scientific requirement in all of sciences; some fields, such as history and astronomy, rely in part upon the observation of singular phenomena. It is also the case that some phenomena are so unusual that an experiment is practically not reproducible, for example we can not take probes frequently from Comet Halley. However, in most of natural sciences we expect experiments to be designed in a way that makes them potentially independently reproducable.

Whether a given experiment is in fact replicated independently largely depends on whether the outcome of the experiment was predictable or unexpected, and on whether the results have potentially important implications. Direct exact replication of an experiment is something quite unusual, and indeed it is generally difficult to publish results that are mere confirmations of previous work. In general, scientists will prefer to repeat the test but by independent scientific means, on the grounds that if two different, complementary lines of evidence both come to the same, unexpected conclusion, an artefactual explanation for the unexpected result is less likely. Thus direct replication of an original experiment is something that usually takes place only when the validity of the original result is explicitly challenged.

Reproducibility is also the variation in measurements taken by different persons or instruments on the same item and under the same conditions.

Famous problems

In the late 1980's there was a rush to publish on the subject of cold fusion, a technology that offered promise of low-cost energy. In March 1989, University of Utah chemists Stanley Pons and Martin Fleischmann reported the production of excess heat that could only be explained by a nuclear process. The report was astounding given the simplicity of the equipment: it was essentially an electrolysis cell containing heavy water and a palladium cathode which rapidly absorbed the deuterium produced during electrolysis. The newsmedia reported on the experiments widely, and it was a front-page item on many newspapers around the world. Over the next several months others tried to replicate the experiment. Many were unsuccessful, especially experiments performed by plasma fusion researchers who had no expert knowledge of electrochemistry. At the end of May the US Energy Research Advisory Board (ERAB) formed a special panel to investigate cold fusion. The scientists in the panel found the evidence to be unconvincing.

Electrochemists pointed out that the materials preparation and initial loading usually takes several months, so it was unlikely that anyone would replicate at the time the ERAB panel surveyed results (May 1989). M. Miles wrote:

In retrospect, it would be impossible for any research group to adequately investigate the multitude of variables involved with this field in only a few months. These variables range from the palladium metallurgy to the D₂O purity, the type of electrolyte and concentration, the electrochemical cell, the electrode arrangement, the type of calorimeter, proper scaling of the experiments, the handling of materials, the current densities used, the duration of the experiments, the loading of deuterium into the palladium, the use of additives, and so on.

In late 1989, Miles and others informed the ERAB panel members that their experiments were now producing excess heat, tritium and other effects, but the ERAB panel did not revise their conclusions. By September 1990, 92 groups from 10 countries reported successful replications, and subsequently more than 200 groups reported thousands of replications in the peer-reviewed literature.

The science of cold fusion was severely damaged by the affair, although research continues around the world. By 2000, excess heat could be reproduced roughly 80% of the time, and surface transmutations 100% of the time.

Although the experiment appears simple, most electrochemists feel that it is extraordinarily difficult. Prof. R. Oriani said that in his 50-year career this was the most difficult experiment he ever performed.

The history of cold fusion illustrates the importance of expert knowledge in establishing the reproducibility of a claim. It is very unlikely that a scientist from an outside field, such as plasma fusion, will be able to replicate a difficult experiment in electrochemistry. Fields such as cold fusion require a multidisciplinary approach.

References

• Turner, William (1903), History of Philosophy, Ginn and Company, Boston, MA, Etext. See especially: "Aristotle".
• Definition (PDF)