The next time you see an experiment in psychology – or any controversial claim that purports to prove something, doesn’t matter if you support the conclusion or not – look at the data, the procedures and possible motivations of the scientists doing the study. Assume it is wrong, there has been an error, and try to find it. If you don’t find an error, you can ascribe a degree of confidence but still be open to other possibilities.
I’d do this to any kind of unusual “hard science” results as well. Use these principles for anything. Vaccinations for example. Or the shape of the Earth. Or whether or not a gay person can be “converted”. Or if eating large amounts of colloidal silver is a good thing for you or if planet Nibiru is about to destroy us. (But getting a look at the data set of some more “speculative” assertions may be difficult. For me, no hard data set means no faith in the hypothesis.)
Anecdotes don’t count. A good scientist does not a base a conclusion on such a small data set. Nor does the number of awards someone may have gotten.

Now you are a scientist!

mad scientist
Hi there! I am a scientist.
Any time a physicist performs an experiment and discovers something important, it has a grueling road ahead of it. Before being published it will be analyzed in fine detail by people looking to poke holes in it. Then the experiment will be duplicated by other physicists to determine its accuracy and correctness. If you claim to find the Pikachu particle, it isn’t accepted as the truth until it has been hammered on every possible way.
Should someone come up with a peer-reviewed claim to the contrary, you are dead in the water unless that study, in turn, can be verified or refuted by the same process.
You see, in real science, a hypothesis can never be proven. It can be sustained by experiments that fail to disprove it. Confidence can be greatly enhanced by accurate predictions it can make but that doesn’t make it fact. Euclidian geometry was considered fact and sustained over and over daily for millennia until we became able to measure with sufficient accuracy. And then Einstein upset the apple cart and blew away a lot of Newton at the same time.
Close enough to be a useful approximation in everyday life.

That kind of certainty can only exist with a completely self-contained realm such as math. Euclidian geometry still works with a flat space assumption but we now believe that space is curved. So in the real world, Euclid is wrong. Very very very close in ordinary life but still fundamentally wrong. Einstein is probably correct. But he might not be – so we keep theorizing and probing ever deeper.

The “soft” sciences have a tradition of not working that way.
But if you really want to understand the universe – or even modern electronics – Einstein is the winner. So far.
Things like psychology, sociology, economics, even moderately hard sciences like climatology, don’t work that way. The reason is that they don’t focus on something as incremental as a Pikachu particle. They work on general concepts. Very often these concepts are also deeply intertwined with public policy. Public policy is subject to the political factions involved as well as what kind of money is available from the ruling faction to research it.

Nobody in high office really cares if the Pikachu particle exists. If it did, they would fund lots of research that would amazingly prove that it did. Their opponents would try to do the opposite. That is what passes for science when politics get involved. We might never learn anything reliable about poor Pikachu..

What happened to my particle?
Science that lacks a control set just isn’t as good as science that has one. I can understand that it is not always possible to have a control set but the scientists working on such problems need to accept that their science is just not as reliable as science that does. It requires more repetitions and more confirmations and needs to be attacked from many different directions.
When the government attempted to reproduce the results of experimental medicines, many times it couldn’t. Science that has a direct profit motive behind it tends to be a bit on the optimistic side.
(Of course, this itself is a study that needs to be critiqued for methodology and reproducibility.  😉 )
Science that deals in areas of political controversy is automatically suspect even if you agree with the outcomes. It happens in many areas of the various psychological sciences because how to control or change human behavior is a fundamental aspect of any political doctrine. It is far more vulnerable to observer bias. It is vulnerable to bias from what the faction in control of funding is willing to fund. It is vulnerable to the preexisting beliefs of those who enter a field.
…studies rated most contextually sensitive were also least likely to have been successfully replicated.” So the more sensitive the the context of the question, the less relaible that body of study is. Very sensitive issues like race and gender tend to be less replicable. That’s bad science.
Do not doubt that people of a particular persuasion are more likely to enter a profession that is already a comfortable fit to their belief system. (People who don’t agree go elsewhere.) This creates an echo chamber. It profoundly affects the conclusions that “experts” in the field draw.
One tests a hypothesis by trying to prove it wrong. One tries to prove the null hypothesis, the hypothesis, that what you see isn’t really caused by what your hypothesis suggests.
If you love a hypothesis, it is not easy to do so. If the powers that be won’t fund it, it may be impossible to do. Simple as that. Even a hard science can be crippled or abused under the right conditions. Genetics was used to justify Hitler and most of the western eugenics movement. It’s polar opposite, Lysenkoism, may well have killed just as many in the Soviet Union by famine.
Now think about how politics has messed up the science of climatology and the debate over climate change. Hardly a respectful and objective discussion of a difference in opinion. Truth – or as close as we can get to it – cannot triumph in an intensely political environment.
Very few people go into psychology if they don’t believe psychology can provide an answer to problems of human behavior. If a particular behavior does not have a psychological solution, it will be a very long time before anyone in the field notices. In the meantime, many untested hypotheses will be advanced to doctrine status and various conclusions reached and those conclusions will be implemented on real human beings.
So we have the behaviorists and the cognitive behaviorists and Freud and Jung and Adler, and a host of other schools some of which may offer insights but none of which even bother with statistics, let alone testing against a null hypothesis. Geneticists are finally coming on line with genetics markers for certain psychological conditions – and where there are a few there could be many more. Psychopharmacology seems to be driven by the notion that pills are the way to fix our unhappy brain waves.
800px-Blind_monks_examining_an_elephantAll these different psychological principles remind me of the parable about the five blind men and an elephant. They’d never encountered an elephant before but each now was feeling a different piece. Each had an idea what a small peice of elephant was like but once separated into different doctrines they could not agree since they all thought there own experince was the totality.
Hopefully psychology can become useful in a scientific way. It has never been so in the past, being much closer to religion.