6 thoughts on “Diversity and team performance: What the research says”

1. Pingback: Rational Feed – deluks917

2. habryka 2017, Oct 26 at 3:21 pm

   (crossposted from the EA Forum)

   For whatever it’s worth, I looked into the 4 sources you cite in your article as improving the effectiveness of diverse teams and found the following:

   1 didn’t replicate, and the replication found the opposite effect with a much larger sample size (which you link to in your article)
   One is a Forbes article that cites a variety of articles, two of which I looked into and didn’t say at all what the Forbes article said they say, with the articles usually saying “we found no significant effects”

   One study you cited directly found the opposite result of what you seemed to imply it does, with its results table looking like this:

   View post on imgur.com

   And the results section of the study explicitly saying:

   “whereas background diversity displayed a small negative, yet nonsignificant, relationship with innovation (.133).” (the thing that did have a positive relation was “job-related diversity” which is very much not the kind of diversity the top-level article is talking about)

   The only study that you cited that did seem to cite some positive effects was one with the following results table:

   View post on imgur.com

   Which found some effects on innovation, though overall it found very mixed effects of diversity, with its conclusion stating:

   “Based on the results of a series of meta-analyses, we conclude that cultural diversity in teams can be both an asset and a liability. Whether the process losses associated with cultural diversity can be minimized and the process gains be realized will ultimately depend on the team’s ability to manage the process in an effective manner, as well as on the context within which the team operates.”

   LikeLiked by 1 person

   Reply ↓
   1. Georgia Post author2017, Nov 09 at 10:08 am

      Thanks for the in-depth response!

      Regarding the replication, I think its results make a decent case that the original finding might have been a statistical fluke. The replication found that individual IQ was strongly linked with group IQ, and that gender diversity wasn’t linked with performance.

      That said, the replication seems to have been slightly smaller than the original, not much larger – the original used 40 groups of 3 and 152 groups of 2-5, and the replication used 26 groups of 3, and two sets of 40 groups of 3 for replicating 3 parts of the hypothesis, so the original should have been slightly larger. Am I reading this right? I’m tempted to call it ambiguous for now.

      (Here’s a discussion in which the authors of both papers show up in the comment thread – the statistics quickly got over my head, but the interested and more literate may learn from it.)

      The review piece you mentioned was this one, right? (I didn’t refer to anything from Forbes.) I looked into three random sources (e.g. scrolling down the page and clicking on a link without reading) and saw:
      * http://www.tandfonline.com/doi/abs/10.5172/impp.2013.15.2.149 – This article was correctly described as finding a fairly strong link between gender diversity in companies and increased radical innovation. The paper also found no correlation with incremental innovation, though the article didn’t mention this. (It didn’t investigate other kinds of diversity.)
      * https://www.credit-suisse.com/corporate/en/media/news/articles/media-releases/2012/07/en/42035.html – I was unable to find the full version of this study, but according to the press release, it did find effects that supported the claims from it.
      * http://onlinelibrary.wiley.com/doi/10.1111/ecge.12016/full – This one found “small but significant effects” from cultural diversity.
      I didn’t read these studies in detail, and they’re probably not the same ones you cross-checked. I should probably be more careful about review articles, but the bits I found seemed supportive of what the final article said. Which ones didn’t, if you have that easily available?

      This article did in fact have an altogether different conclusion than what I said it did. I must have somehow misread it – this is embarrassing. I also cite it as saying something about decision-making, but maybe I was thinking of a different one? In any case, I unintentionally misrepresented the study and am going to strike it. Much obliged for catching that.

      I agree that the last study doesn’t find overall positive or negative effects for diversity – I even cite it as saying such in the first section.

      —

      Now I’m wondering about the decision-making and creativity claims. If we lump the reported positive aspects into two clusters – “creativity/innovation” and “problem-solving/decision-making” – the whole section does seem to be much more ambiguous than before. (Mixed evidence for each.)

      I spent a few hours looking around (with the explicit intent of not just considering things that reported what I already thought) and was hard-pressed finding other recent review articles finding results between demographic diversity and creativity/decision-making/problem solving. The Work Group Diversity study I cited describes some of the confusing results (for diversity and positive effects on performance) from the literature.

      Based on that, I’m now considering that the positive aspect of diversity on performance might be mostly mediated through informational diversity, which demographic diversity sometimes relates to. (e.g. different lived experiences, women tending to be more empathetic, etc.) As summarized in this review, for instance:

      “Thus, overall, most of the support for the value-in-diversity hypothesis comes from studies concentrating on functional differences (serving as a proxy for diversity in skills, information, and expertise). These differences have typically been shown to improve performance through vigorous debate that leads to creativity and improved problem solving (see also Bunderson & Sutcliffe, 2002; Carpenter, 2002; Pitcher & Smith, 2000).”

      In any case, I do somewhat expect to find some upside, because if diversity’s effects on performance are overall neutral, and we have a mechanism for them to be negative, I’d also expect a mechanism for them to be positive.

      I wouldn’t be at all surprised if the answer is out there, and this is important, but I’m at the limit of how much time I can spend on this right now. I’d like to do some more reading on this later (and encourage others to do the same).

      LikeLike

      Reply ↓
      1. habryka 2017, Dec 27 at 7:59 pm

         Thanks for double-checking all of my double-checking!

         “Based on that, I’m now considering that the positive aspect of diversity on performance might be mostly mediated through informational diversity, which demographic diversity sometimes relates to. (e.g. different lived experiences, women tending to be more empathetic, etc.)”

         This is roughly my current model of the benefits of diversity, though I do actually mostly expect the net benefits of not explicitly informational diversity to be net-negative because of obvious reporting biases. Though I haven’t done a rigorous analysis (i.e. funnel plot or similar stuff) to be confident about this.

         LikeLike

3. Pingback: EAG 2017 SF: Diversity in EA, and lightning talks – EARadio

4. Jay 2026, Jan 03 at 2:26 pm

   I have a somewhat speculative, partial critique of the following:

   “Based on that, I’m now considering that the positive aspect of diversity on performance might be mostly mediated through informational diversity, which demographic diversity sometimes relates to.”

   This is a little off the fly but, I have two main concerns with this:

   1. Methodological: Are the studies “over controlling” in their definition of diversity?
   2. Mechanism based: Is “informational diversity” really different?

   To start, the diversity between real world demographic communities and the definitions used in a lot of business contexts/studies seem to be significantly different. The definitions used in business contexts/studies often turn diversity into tightly defined, fairly isolated variable have to do with demographic group. There are statistics methodological reasons for this of course, and I think its not terrible practice, you’ve got to use something after all. But the average members of real world demographic communities often differ along multiple dimensions in a way that makes me think the “demographic box ticking alone” definitions are problematic.

   For example, the average member of the Black or Native American communities are also significantly poorer, less educated, and have very different experiences of institutions than the average member of the white community. One can isolate those different aspects for each other (and in many contexts one should), and treat race as one factor along side class and “information giving experiences”. But my read of the evidence (and my anecdotal experience) is that that’s not how the members of these groups think about themselves being diverse from dominant society, nor how these factors actually work in the real world. Class is racialized in America. There is a strong link between being a person of color and being poor having to do with well documented mechanisms specific to racial groups, both historical and contemporary. These racial communities are in large part shaped by their experience of racialized poverty, and are acutely aware of this fact. So separating these factors of race and class too much, or uniting a certain factor (like class), risks creating a typology that is distorting.

   • Someone might think I’m making claims about intersectionality here. I think that’s somewhat true, but its actually somewhat adjacent to that. Rather I’m critiquing the way typologies are constructed in a more deep sense, not just not combining demographic categories, but rather thinking we can abstract out general categories in the way we have in been doing. It’s not clear to me that can happen without distortion, and at the very least I don’t see much research thinking about it.

   Thus, I wonder how much these definitions flattens “diversity” into something that the members of disadvantaged communities wouldn’t even recognize as the inclusion of people like them. If one draws on only a fairly narrow, unrepresentative sample of the disadvantaged communities, for example disproportionately affluent, higher education backgrounds, that’s already flattening the “diversity” a lot. And my impression is that is what happens in many of the businesses being studied. Put differently, it’s plausible that what is happening is that the business community expanding their “diversity” is mostly limited to the already existing overlap between the disadvantaged group and the advantaged groups, which raises significant methodological concerns about what the “demographic group” variables being used in analysis are really measures of.

   So, if a study finds that racial or ethnic diversity alone as a isolated factor isn’t associated with better performance, I would not find that surprising, but its also strikes me as a really thin conclusion with limited real-world usefulness. I would certainly be concerned about people letting ANY of this research influence their thinking too much, whether it suggests group diversity increases or decreases group performance.

   • Another concern I have with the wider debates about this is the seemingly disproportionate focus on businesses. There are more studies from other areas of social science and political science that indicate group performance improvements that seem to be left out of some discussions.
   • I’d also like more natural experimental or quasi-experimental evidence than I have seen in most discussions to date. Those are better for drawing conclusions than observational studies, even if those conclusions are more context specific.
   • Also I want to acknowledge that the studies being discussed look at more demographic diversity than just racial and ethnic, but covering examples from all of them would take a lot of time, and I am most familiar with the evidence of racial and ethnic diversity.

   My second concern has to do with the mechanisms being discussed with informational diversity. Much hay has been made recently from findings that various forms of group resource diversity (information, cognitive, skills) is more important that demographic diversity. But putting the two in tension is weird to me. When you ask members of disadvantaged groups why diversity would be better for group performance, they basically say some version of the group resource mechanism (or another commonly proposed mechanism, that the diversity creates a structural check on weaknesses of individual members). So, what’s the proposed mechanism supposed to even be if not that? It seems obtuse to strongly separate resource diversity from demographic diversity when the most commonly provided mechanisms for why diversity would help is resource diversity.

   • Now the reply to my critique here would probably be something along the lines of, “You’re misrepresenting what we’re saying. The important finding is that demographic diversity tracks resource diversity less than previously asserted.” That has some merit, but also is itself problematic. As per my point above, it’s not clear to me that the samples of people from disadvantaged groups being studied are representative of the diversity between average members of the relevant communities. And if the diversity is unrepresentatively low, or only happening along some dimensions, that would imply that these samples would have disproportionately lower potential for increases in resource diversity relative to their increases in demographic diversity. We need better evidence on this in many ways.

   • The discussions of mediating mechanisms, context specific outcomes, heterogeneous effects, etc. are also strange to me. My main concern is that the commonly proposed mechanisms ALREADY intuitively imply limitations and tradeoffs that seem unobjectionable. You correctly report about some of these above. But these conclusions are often treated with great weight, as if they reveal something incredibly insightful, and I struggle to see why. The findings often seem to me to be perfectly consistent with the positions they are critiquing.
     • Anecdotally, when I have asked members of disadvantaged groups I know personally, who are themselves strong supporters of increasing diversity, they are already aware of and are reasonable about the intuitive limitations and tradeoffs.
   • Also, I want to acknowledge that folk-wisdom is not a reliable guide to many things. But the members of disadvantaged groups do have a lot of experience with the realities around diversity, and at least some of that is both representative and not personally accessible to many researchers.

   Now I want to acknowledge that the studies making hay of resource diversity are often critiquing older studies that had serious methodological flaws around demographic diversity. That is true, and an important step forward. But my critique here is less about any specific camp in this debate, and more about the entire research community’s mode of operating. In a weird way these debates seem very poorly thought through to me. The people engaging work hard, are smart, and have dedicated significant intellectual work into designing their research. But one significant outcome is a set of weakness common to a lot of research, that seem to me like they could have been corrected decades ago if the research community had just taken a random sample of people from disadvantaged groups and had them explain why they thought diversity would help. But regardless: There’s something going wrong in the research process, and it seems like researchers could do better.

   In conclusion: I would strongly recommend to anyone doing research in this area to step back a try to have a really thorough debate about what it is they are trying to measure.

   As for what all this implies about the social usage of this research, I would suggest a few practical takeaways:

   1. Earlier research often had poorly thought through ideas of diversity, and sometimes was used to justify claims about diversity’s benefits that were overly broad or overstated. Use this research cautiously.
   2. More recent research often has poorly thought through ideas of diversity, and sometimes is used to overstate the case against diversity’s benefits. Use this research cautiously.
   3. Regardless of the benefits or lack thereof, current diversity efforts likely poorly perform in terms of the full inclusion of disadvantaged communities, disproportionately including those with the strongest ties/overlap with advantaged communities. Defending current diversity efforts too much risks entrenching systems that might not be good of diversity.
   4. Critics of current diversity efforts should not use the weaknesses of previous research to lull them into a laziness that results in overreach in the conclusions they draw, or as a permission slip to make promiscuous claims about the ills of diversity. (I’m sure everyone here knows this, but it should be said anyways)
   5. Discussing diversity in the abstract has a place, but always keep one eye on the specifics, and think through what you are claiming concretely means with any given statement.

   Anyways I’ve got more thoughts, but this comment is already long enough. For anyone you actually reads this, thank you!

   LikeLike

   Reply ↓