Scientists can draw very different meanings from the same data, study shows

Scientists can draw very different meanings from the same data, study shows


Giving the same information to multiple scientific teams can lead to very different conclusions, a report published today in Nature shows.

And that’s exactly why two researchers think scientists should share their data with others — well before they publish.

In this experiment, 29 scientific teams were given the same information about soccer games. They were asked to answer the question “Are dark-skinned players more likely to be given red cards than light-skinned ones?” Some scientists found that there was no significant difference between light-skinned and dark-skinned players, whereas others found a very strong trend toward giving more red cards to dark-skinned players. So, even though a pooled result showed that dark-skinned players were 30 percent more likely than light-skinned players to receive red cards, the final conclusion drawn from this exercise — that a bias exists — was a lot more nuanced than it likely would have been if only one team had conducted the analysis.

http://www.thinkgasms.com/2015/10/09/scientists-can-draw-very-different-meanings-from-the-same-data-study-shows/


Science: a "truth" that is ever-changing and never seen the same by everyone.
Some truth!

That's fairly meaningless without the actual data and the averages or medians. If some teams said that there was no significant difference, there needed to be teams which said there was a large difference. If one team had been asked, they may well have said there was an even larger difference than the pooled result showed.

One would also need to define what "significant" means in this context, and also one would need to know the definition of "dark" versus "light" re skin colour, and whether all the teams applied the same definition or not because it's open to interpretation otherwise.

In medical trials they do share data, which results in meta-analyses - pooled results. However, this is not really about pooled data, it's about how a number of people interpret the same data.

There's nothing magical about the term 'science'. You just have to pay attention to what is being said. People often say 'statistics are another way to lie', but it's all in the framing of the statement.

Let's say in a tournament, overall 10 cards are pulled on blacks, and 8 cards are pulled on non-blacks. I can say:

Blacks are carded more;

or

Blacks need to be carded more.

The evidence will show the same thing...more cards are pulled on black players. Which is the truth? That there is discrimination in officiating, or that blacks are unruly and need more watching?

We have this going on with the Oakland Raiders in the past 15-years. The Raiders consistently get more flags than any other team in any year. The Raiders also successfully sued the NFL twice in that span of time, and the lawyer for the League became the Commissioner for the League.

Which is the case? Are the Raiders breaking the rules more than other teams? Or, is the League retaliating against the Raiders for having twice successfully embarrassed the League in a court of law? Same facts make both cases.

Original Quill wrote:There's nothing magical about the term 'science'.  You just have to pay attention to what is being said.  People often say 'statistics are another way to lie', but it's all in the framing of the statement.  

Let's say in a tournament, overall 10 cards are pulled on blacks, and 8 cards are pulled on non-blacks.  I can say:

Blacks are carded more;

or

Blacks need to be carded more.

The evidence will show the same thing...more cards are pulled on black players.  Which is the truth?  That there is discrimination in officiating, or that blacks are unruly and need more watching?

We have this going on with the Oakland Raiders in the past 15-years.  The Raiders consistently get more flags than any other team in any year.  The Raiders also successfully sued the NFL twice in that span of time, and the lawyer for the League became the Commissioner for the League.  

Which is the case?  Are the Raiders breaking the rules more than other teams?  Or, is the League retaliating against the Raiders for having twice successfully embarrassed the League in a court of law?  Same facts make both cases.

It would also depend on how many black players they have as opposed to non-black players. I assume that the data in the experiment was adjusted for that.

Then you'd have to know if there was an unjustified bias, or if the black players simply broke the rules more often. The data wouldn't necessarily lead to a conclusion of "bias".

Raggamuffin wrote:

Original Quill wrote:There's nothing magical about the term 'science'.  You just have to pay attention to what is being said.  People often say 'statistics are another way to lie', but it's all in the framing of the statement.  

Let's say in a tournament, overall 10 cards are pulled on blacks, and 8 cards are pulled on non-blacks.  I can say:

Blacks are carded more;

or

Blacks need to be carded more.

The evidence will show the same thing...more cards are pulled on black players.  Which is the truth?  That there is discrimination in officiating, or that blacks are unruly and need more watching?

We have this going on with the Oakland Raiders in the past 15-years.  The Raiders consistently get more flags than any other team in any year.  The Raiders also successfully sued the NFL twice in that span of time, and the lawyer for the League became the Commissioner for the League.  

Which is the case?  Are the Raiders breaking the rules more than other teams?  Or, is the League retaliating against the Raiders for having twice successfully embarrassed the League in a court of law?  Same facts make both cases.

It would also depend on how many black players they have as opposed to non-black players. I assume that the data in the experiment was adjusted for that.

Then you'd have to know if there was an unjustified bias, or if the black players simply broke the rules more often. The data wouldn't necessarily lead to a conclusion of "bias".

Correct. The way they show bias in American law is by use of a bi-nomial analysis. The Supreme Court arbitrarily says if the two populations vary by more than 3 SD's, there is an extraneous reason for it.

Of course the experiment didn't specify black people versus non-black people, it merely says "dark-skinned" and "light-skinned". As I said, that would need to be defined because people have different perceptions of what is dark skinned or not.

It would also depend on how they looked at the data. If they put several photos of players side by side, they might define "dark-skinned" differently to someone who did not do that. For example, if I stood next to someone with blonde hair, someone might describe my hair as "dark", but if I stood next to someone with black hair, my hair might be described as "light".

umm that is the point of science, it is called peer review.

But in addition that is a Social Science, like all psychology it is mocked by the real sciences because it fails to meet the requirements to be a real science. where one must be able to define and measure.

As raggs points out none of the variables in the above are definitively defined.