Hacker News

If the question is the same, the method is the same, and the data is the same, then the conclusion should be the same.

If the question or the data are different: then there's no problem. Apples and oranges. But then the results shouldn't be compared on first place.

If the question and data are the same, but the theory is different: why? Are those alternative theories actually handling exceptions, or are they there just so someone can bend the conclusions to the ones that one wants?

If question, data, theory are the same, but the method is different: why? You'll usually have a better method, and the others will yield less accurate results. It's still acceptable to use those other methods, mind you, but you need to be aware of the error that they incur. And the usage of the less optimal method should be justified if the justification isn't obvious.

If the question, data, theory and method are the same, but the conclusion is different: then sorry, at least someone is assuming / vomiting certainty / rushing conclusions / making shit up.

The approach of "the sick man of the sciences" (Psychology) boils down to "We got lots of clocks. They don't agree on the time. So let's assoooome that most clocks are fine, only a few are broken clocks, and keep assuming."

This affects Psychology specially bad because its object of study is a blackbox concept, called "mind", so there are lots and lots of room for assumers to make shit up. And last time that someone tried to fix this issue, the assumers rose up in arms against it. This is only being fixed now, due to development in other fields being actually able to contradict or back up claims about the "mind" (whatever it means).