(selected messages from http://groups.google.com/group/sci.philosophy.tech/browse_frm/thread/cd3a4f031757c68b/eb82663b3e8ca4d1) Could be a Philosophy of Science research project when I retire. Philosophy of Reproducibility Newsgroups: sci.philosophy.tech Date: 1999/05/21 Subject: Reproducibility I want to do some reproducibility research. This is not a science issue. I'd like to study it from the philosophical perspective. Suppose one scientist says "I redid the experiment three more times right after the initial experiment and the result was exactly the same". The second scientist says "I did the initial experiment. Many days later John repeated it. Then Mary did it in her country (far away from ours). All three experiments gave exactly the same result." Question: why do most scientists think the second scientist's experiment has higher probability to be reproduced than the first scientist's, even though it was done only 3 times, not 4? Common Sense Answer: Because it was reproduced by different people, in different places and at different times. I'm searching for a more rigorous answer possibly based on logic. Anybody knows someone already did this study? Yong Huang Email:yong321@yahoo.com -------------------------------------------------------------------------------- From: Bjoern Brembs Date: 1999/05/22 yong...@yahoo.com wrote: > Common Sense Answer: Because it was reproduced by different people, in > different places and at different times. > > I'm searching for a more rigorous answer possibly based on logic. The idea of chosing reproducibility as a definition for "truth" in the natural sciences stems from the idea that we are erroneous humans. We make mistakes during the course of the experiments: construction of the experiment, process of the experiment, perception of the experiment and so on. By reproducing the same experiment by different people in different places you try to reduce the amount of error the experimenter introduces. That's the definition of subjective and objective: everything that varies between observers is subjective, everything that is constant between observers is objective. > Anybody knows someone already did this study? I don't know of anybody. Bjoern -------------------------------------------------------------------------------- From: Hans Blom Date: 1999/05/26 yong...@yahoo.com wrote: > I want to do some reproducibility research. This is not a science > issue. I'd like to study it from the philosophical perspective. > > Suppose one scientist says "I redid the experiment three more times > right after the initial experiment and the result was exactly the > same". The second scientist says "I did the initial experiment. Many > days later John repeated it. Then Mary did it in her country (far away > from ours). All three experiments gave exactly the same result." My question: what do you mean when you say "exactly the same" result? I know of no two experiments that can turn out to be "exactly the same" in all dimensions. It appears to me that the notion "exactly the same" can only be maintained if one focusses on just a few dimensions and disregards all others, and if one discretizes the meaurements or observations coarsely enough. Many people prefer yes/no answers for just these reasons. But if you look closely, reality is always far more complex. So, your first question ought to be, I believe, what "reproducibility" is, and when you think one thing is "exactly the same" as another. Mathematics has a well-defined answer. Regrettably, mathematics cannot talk about everyday life. Mathematics, for instance, posits that 1 = 1, even though, if you look closely enough, the 1 to the left is slightly different from the one (1) on the right. You cannot step into the same river twice, is an old saying. You cannot do the same experiment twice, either. So you cannot have identical outcomes from identical tests, either. Sorry to make your question even more complex. Hans -------------------------------------------------------------------------------- From: yong321@yahoo.com Date: 1999/05/26 In article <374BE0C2.C34BD...@ele.tue.nl>, Hans Blom wrote: > You cannot step into the same river twice, is an old saying. You > cannot do the same experiment twice, either. So you cannot have > identical outcomes from identical tests, either. > > Sorry to make your question even more complex. > > Hans You're very correct. But my interest for now is not "how reproducible" may be called reproducible. Instead I want to focus on the effect of the factors in performing an experiement on a qualitatively-defined reproducibility. So for the sake of my study, I assume the scientists in question are not arguing about a measured result of 1.2 m versus 1.1 m; they agree "the object has moved in both experiments", for instance. I think Bjoern Brembs's first response to my question is quite useful. Varying factors affecting reproducibility should be given different weights in calculating the final credibilty of the experiment being redone. A weight is given based on how effective the factor can reduce subjectivity. But then the problem is, measuring this effectiveness becomes a subjective process on the part of the philosopher. Yong -------------------------------------------------------------------------------- From: Hans Blom Date: 1999/05/27 yong...@yahoo.com wrote: ... my interest for now is not "how reproducible" > may be called reproducible. Instead I want to focus on the effect of > the factors in performing an experiement on a qualitatively-defined > reproducibility. So for the sake of my study, I assume the scientists > in question are not arguing about a measured result of 1.2 m versus 1.1 > m; they agree "the object has moved in both experiments", for instance. That's the easy part. > I think Bjoern Brembs's first response to my question is quite useful. > Varying factors affecting reproducibility should be given different > weights in calculating the final credibilty of the experiment being > redone. A weight is given based on how effective the factor can reduce > subjectivity. But then the problem is, measuring this effectiveness > becomes a subjective process on the part of the philosopher. That all belongs to the easy part, I'm afraid. The difficult part is agreeing on the _dimensions_ for which to assign weights. Are you going to measure, for instance, how far a test animal moves or the x, y, z dimensions separately, or how long the move took, or how much its heart rate changed while moving, or how much it learned from this test/experience, or what? What is of interest to you? There's an infinity of dimensions to choose from, I guess, not counting the ways in which these can be combined into a multi-dimensional measurement. My point of view is that subjectivity arises mainly from each person's idiosyncratic way of establishing/using perceptual dimensions rather than from the measurement problem once these dimensions have been agreed on. Hans -------------------------------------------------------------------------------- From: "M...@rtin" Date: 1999/05/29 May I throw in another word? What about "repeatability" (same person, same experimental set, another try) as opposed to "reproducibibilty" (another person, another experimental set/other lab or country)? The topic also is a two way problem. You have your result limited to the precision and accuracy of your measurement, but on the other side you need a representative sample of what you want to measure. In practice, you very often deal with confidence levels creeping around 70% due the nature of sample taking (For example sample taking of bulk powders to do, let's say, sieve residue testing. If you look for more than 65% condfidence level to have 10 ppm on a 625 mesh, for example, you need to take a 13 kg sample size, which is far too much to realize). Regards from France, Martin