> No, if 10 predictions each have a 10% chance of a false positive, you will always have on average 1 false positive. Whether you test them on the same data or not doesn't matter
That's true.
I should have said it differently. In fact, I'm not even sure that my understanding is correct.
The problem with not using new data to test each new prediction is, that if a scientist wants to show A on data X, but data X doesn't confirm A, the scientist modifies A slightly and now tests A' on X, which is again rejected, and then modifies it again, testing A'' on X, and so on... until the data X actually confirms hypothesis A'''''''!
That's the real problem - using data without a predefined plan for how you will use this data. In the above example, the data that you collected affected your decision-making process, so your results are not independent of the data (and thus not replicable!).
That's true.
I should have said it differently. In fact, I'm not even sure that my understanding is correct.
The problem with not using new data to test each new prediction is, that if a scientist wants to show A on data X, but data X doesn't confirm A, the scientist modifies A slightly and now tests A' on X, which is again rejected, and then modifies it again, testing A'' on X, and so on... until the data X actually confirms hypothesis A'''''''!
That's the real problem - using data without a predefined plan for how you will use this data. In the above example, the data that you collected affected your decision-making process, so your results are not independent of the data (and thus not replicable!).