Open Codex HISTORICAL entry

2009 Sep 25 | Gender Bias in Wikipedia Coverage?

The recent controversy about gender imbalance and sexism in open content communities has been remarkable this summer, and this week's news about Shuttleworth's comments might mean it will extend into the autumn. While I think these events merit a historical and cultural analysis -- and prompts the questions if sexism increased, is it being noticed more, or both? -- I want to postpone that undertaking for the moment. Instead, I wonder if the recent demographic data that shows women are about 13% of Wikipedians affects its topical coverage?

As you can see in this Comparison of Biographies , Wikipedia does very well in its coverage of National Women's History Project (NWHP) biographies relative to Encyclopaedia Britannica .

64 entries are missing from EB; 23 from WP. 23 entries are in neither. For articles existing in both, WP articles are 6.29 times larger on average (median of 4.00).

That is, of 174 women, Wikipedia is missing articles on 23 of them. That's almost a third of those missing from Britannica , which doesn't have any articles not at Wikipedia. When both do have an article, Wikipedia articles have much more content. Of course, those are just the quantitative numbers. Even so, when I browse the actual articles, I am partial to the extra content and images of Wikipedia.

Yet, a difficulty in this work is finding a useful corpus of biographical persons. To say that there are more articles about men than women in any reference work, isn't saying much given world history. So, for this analysis I use those women recognized by the NWHP for Women's History Month. The NWHP is a nice collection in that it has both well known women and lesser-known women who are thought to be notable nonetheless. However, this only tells us that Wikipedia has greater coverage of women than a traditional encyclopedia. (And while this is one of the first large and topical -- rather than quality -- comparisons it should not be all that surprising given Wikipedia's size.) And, Wikipedians are aware of their own systemic bias and make attempts to counter it. For example, those recognized by Black History Month were the focus of a WikiProject that documented every person recognized. (Ironically, this list was taken from Britannica. And perhaps the NWHP list will prompt a similar project at Wikipedia, which is why I use permanent links to the specific versions I analyzed.)

What would really be nice is a source corpus of notable persons, both male and female. I could then compare this against Wikipedia and Britannica to see how they fare relative to the source corpus. That is, a source corpus of 100 people might recognize 75 men and 25 women (25% female), and if one of the references had a 60/15 split, it'd be less "feminine" (20%) than the source. How, then, does each reference work compare to each other, relative to their source? If you have a suggestion for corpus, please leave a comment.

Finally, while speaking with Nora about this, she also raised the question of if the gender ratio of disruptive editors differs from that of the larger community? Our hypothesis is disruptive editors might be disproportionately male. But who can say? Unfortunately, I expect it's difficult to get survey responses from such editors.

this entry posted to social/wikipedia ;
comments (17)

Posted by Sage Ross at Sat Sep 26 09:17:17 2009
Nice post, Joseph.  Did you see this? http://www.washingtoncitypaper.com/blogs/sexist/2009/09/25/sexist-beatdown-consensual-incest-and-john-phillips-fanboys-edition/

Of course, that's not necessarily as reflective of Wikipedia culture as of the broader culture (I hope), but low-grade sexism is pretty common to observe at the level of individual articles.

Posted by Joseph Reagle at Sat Sep 26 10:11:44 2009
Thanks for the comment Sage. I didn't see the Phillips' thing. And I'm not sure if using the term "consensual" to differentiate the subsequent relationship is horribly sexist, but it is interesting that it was added. In any case, the character of that relationship appears to be a complex issue and I expect will continue to be debated at large -- and on WP.

Posted by Circeus at Sat Sep 26 11:24:55 2009
How about the national biographic encyclopedias? we have a Category (Biographical dictionaries by country) documenting them.

Posted by Siebrand at Sat Sep 26 14:08:33 2009
Interesting. Analysis may contain errors.

The table shows that there would be no article on Judith Francisca Baca, but [[Judith Baca]] has existed since 2007-11-20. The redirect [[Judith Francisca Baca]] has existed since mid 2006.

I checked the next 10 reported as missing from Wikipedia also, and found no additional errors.

Posted by Kaldari at Sat Sep 26 16:13:30 2009
There is one more error: Mary Louise Defender Wilson is listed incorrectly as Mary Louise Wilson (who is a different person).

Posted by Kaldari at Sat Sep 26 16:16:48 2009
And also I believe Frances Elizabeth Caroline Willard is listed on Wikipedia as "Frances Willard (suffragist)", so that cancels out the above error for the totals :)

Posted by Joseph Reagle at Sat Sep 26 17:10:17 2009
@ Circeus: Yes, I will check those out, or I was thinking "100 Most ..." type listings.

@ Siebrand, Kaldari: Thanks for pointing these out. I posted this hoping for some bug spotting! :) Coming up with the correct names and seeing if there were articles was the most time consuming (and error prone) task (e.g., nicknames, popular v. indigenous, maiden v. married, spelling and accents). I want an approach that is consistent/fair across both works. Yet, the source has spelling variations and typos. Then, some subjects have similar names, some subjects are listed by different names (e.g., Bonnin in the source and at EB, but Zitkala-Sa at WP), and there are other complexities (e.g., The Grimke sisters have a single page at EB, but separate pages at WP).

I will address such bugs and hope to describe my method shortly.

Posted by Kaldari at Sat Sep 26 17:51:34 2009
Before you get too far on expanding this investigation, I would like to constructively question one of its underlying assumptions: Why do you think that women's biographies would be the most apt category to indicate whether or not the dominance of male editors affects Wikipedia's topical coverage? As an active member of both WikiProject Biographies and Wikipedia's Feminism Task Force I can tell you that most of those articles weren't written by women. Just because someone is a women doesn't mean they are necessarily going to be more interested in writing women's biographies. Such an assumption seems overly simplistic to me. A couple of areas that I know have a fair number of active female editors include textile arts and children's literature. But even for those, there is no guarantee that the gender ratio of the editors is going to be reflected in the quantity or quality of the coverage. Personally, I don't think there is a reliable way to test what you are wanting to test simply because male and female Wikipedia editors cannot be put into simple boxes concerning what they are most likely to write about. Such an assumption seems sexist in and of itself, if you don't mind me saying so.

Posted by Joseph Reagle at Sun Sep 27 02:08:40 2009
On the bugs, the challenge is I want to run this over large collections and so I take advantage of probabilistic heuristics by making use of Google search API to return results it thinks are appropriate and I then compare those results in the title using another probabilistic textual comparison of the source name with the title (0.0 > 1.0). I can set the threshold on the latter and have been playing with that. But whatever I do, I want it to be even handed (i.e., making a false positive or negative) across the reference works. (Given an accurate/clean source corpus, which is harder than I anticipated.) However, I improved my use of the difflib heuristics and I just realized how to do something I'd been scratching my head about (site specific searches) through the Google AJAX API. The results are improved though it makes very little differences in the big picture conclusions one might draw. On that....

On the assumption front, it's a ("null") hypothesis given that (1) you can find both male dominance and corresponding lack of female biographic coverage in historical reference works which historians (non-controversially) imply is causal and (2) people have previously inferred (some informally, some in studies) that WP contributor representation affects topical coverage (i.e., "systemic bias".) That is, does gender imbalance in contributors affect biographical coverage relative to Britannica? This preliminary analysis appears to disprove the null hypothesis and implies support for your sense of how things work at WP.

Of course, doing the ratio comparison I discuss above would be better.

Posted by Joseph Reagle at Sun Sep 27 02:17:59 2009
Oh, and then to the more generic claim. One could do a comparison within WP. For example, if one found an authoritative listing of topics typically associated with male interests and another authoritative source on female interests (perhaps as identified from some of the surveys Wikipedians) then could see the percent of coverage on each. Of course, those two sources may differ for any number of reasons... And that said, given the biographical experiment, and Wikipedia's size, I would now be much more skeptical of claims of such bias.

Posted by Kaldari at Sun Sep 27 12:50:06 2009
I definitely agree that there is systemic bias on Wikipedia. Gender bias, however, would be the hardest of all to judge, IMO. Geographic and language bias are relatively straightforward. Race is trickier, and gender (and age) even more so, as any generalizations you could make about people's interests according to gender are going to be dubious at best and stereotypical at worst. I think in the case of gender, other variables are going to strongly compete for significance, and it will be difficult to eliminate them all. Also, there is an inherent flaw with doing comparisons against Britannica: Wikipedia is aware of and responds to what is covered in Britannica. This is probably one reason why there are no articles in your study present in Britannica but absent from Wikipedia.

With all that said, I appreciate your efforts to address this topic, as it is certainly an important one. I hope my constructive criticism hasn't come across too negatively. BTW, you'll be happy to learn that your blog post has inspired an effort to complete Wikipedia's coverage of National Women's History Project biogrpahies:
http://en.wikipedia.org/wiki/Wikipedia_talk:WikiProject_Gender_Studies/Feminism_Task_Force#National_Women.27s_History_Project_biographies_yet_to_be_written

Posted by Joseph Reagle at Mon Sep 28 07:58:54 2009
@Kaldari: I very much appreciate the comments,and thought this might prompt an effort to then document those women. Not so quickly though! :)

Posted by Axel Boldt at Sat Oct 3 11:54:58 2009
Nice analysis, thanks. One tiny point: the female/male ratio being 0.28 doesn't mean that 28% of subjects are females. 23/105 = 22% of the subjects are females. 0.28 to 1 are the odds for a subject to be female.

Posted by Joseph Reagle at Sat Oct 3 12:33:25 2009
Right you are Axel. I mispoke. I use the ratio instead of percent of total in case there are gender "unknowns". (I could also do percentage of a gender relative to all less the unknowns.)

Posted by Sue Gardner at Mon Oct 12 20:11:43 2009
Really interesting, Joseph, thanks. I've been looking for this kind of analysis.

Posted by Joseph Reagle at Tue Oct 13 15:33:47 2009
Thanks Sue. I did find a nice corpus that I'm prepping for analyzing, but don't expect to find anything contrary to what we see here.

Posted by Danielle Maedchenspiele at Wed Oct 14 09:16:55 2009
I can't believe that Britannica missed 65 articles! However, congrats for the analysis, I found it really interesting because I am studying this topic as well!

Open Communities, Media, Source, and Standards

by Joseph Reagle

reagle.org
Open Codex by Joseph Reagle

Open Codex HISTORICAL entry

2009 Sep 25 | Gender Bias in Wikipedia Coverage?

The recent controversy about gender imbalance and sexism in open content communities has been remarkable this summer, and this week's news about Shuttleworth's comments might mean it will extend into the autumn. While I think these events merit a historical and cultural analysis -- and prompts the questions if sexism increased, is it being noticed more, or both? -- I want to postpone that undertaking for the moment. Instead, I wonder if the recent demographic data that shows women are about 13% of Wikipedians affects its topical coverage?

As you can see in this Comparison of Biographies , Wikipedia does very well in its coverage of National Women's History Project (NWHP) biographies relative to Encyclopaedia Britannica .

64 entries are missing from EB; 23 from WP. 23 entries are in neither. For articles existing in both, WP articles are 6.29 times larger on average (median of 4.00).

That is, of 174 women, Wikipedia is missing articles on 23 of them. That's almost a third of those missing from Britannica , which doesn't have any articles not at Wikipedia. When both do have an article, Wikipedia articles have much more content. Of course, those are just the quantitative numbers. Even so, when I browse the actual articles, I am partial to the extra content and images of Wikipedia.

Yet, a difficulty in this work is finding a useful corpus of biographical persons. To say that there are more articles about men than women in any reference work, isn't saying much given world history. So, for this analysis I use those women recognized by the NWHP for Women's History Month. The NWHP is a nice collection in that it has both well known women and lesser-known women who are thought to be notable nonetheless. However, this only tells us that Wikipedia has greater coverage of women than a traditional encyclopedia. (And while this is one of the first large and topical -- rather than quality -- comparisons it should not be all that surprising given Wikipedia's size.) And, Wikipedians are aware of their own systemic bias and make attempts to counter it. For example, those recognized by Black History Month were the focus of a WikiProject that documented every person recognized. (Ironically, this list was taken from Britannica. And perhaps the NWHP list will prompt a similar project at Wikipedia, which is why I use permanent links to the specific versions I analyzed.)

What would really be nice is a source corpus of notable persons, both male and female. I could then compare this against Wikipedia and Britannica to see how they fare relative to the source corpus. That is, a source corpus of 100 people might recognize 75 men and 25 women (25% female), and if one of the references had a 60/15 split, it'd be less "feminine" (20%) than the source. How, then, does each reference work compare to each other, relative to their source? If you have a suggestion for corpus, please leave a comment.

Finally, while speaking with Nora about this, she also raised the question of if the gender ratio of disruptive editors differs from that of the larger community? Our hypothesis is disruptive editors might be disproportionately male. But who can say? Unfortunately, I expect it's difficult to get survey responses from such editors.

this entry posted to social/wikipedia ;
comments (17)

Posted by Sage Ross at Sat Sep 26 09:17:17 2009
Nice post, Joseph.  Did you see this? http://www.washingtoncitypaper.com/blogs/sexist/2009/09/25/sexist-beatdown-consensual-incest-and-john-phillips-fanboys-edition/

Of course, that's not necessarily as reflective of Wikipedia culture as of the broader culture (I hope), but low-grade sexism is pretty common to observe at the level of individual articles.

Posted by Joseph Reagle at Sat Sep 26 10:11:44 2009
Thanks for the comment Sage. I didn't see the Phillips' thing. And I'm not sure if using the term "consensual" to differentiate the subsequent relationship is horribly sexist, but it is interesting that it was added. In any case, the character of that relationship appears to be a complex issue and I expect will continue to be debated at large -- and on WP.

Posted by Circeus at Sat Sep 26 11:24:55 2009
How about the national biographic encyclopedias? we have a Category (Biographical dictionaries by country) documenting them.

Posted by Siebrand at Sat Sep 26 14:08:33 2009
Interesting. Analysis may contain errors.

The table shows that there would be no article on Judith Francisca Baca, but [[Judith Baca]] has existed since 2007-11-20. The redirect [[Judith Francisca Baca]] has existed since mid 2006.

I checked the next 10 reported as missing from Wikipedia also, and found no additional errors.

Posted by Kaldari at Sat Sep 26 16:13:30 2009
There is one more error: Mary Louise Defender Wilson is listed incorrectly as Mary Louise Wilson (who is a different person).

Posted by Kaldari at Sat Sep 26 16:16:48 2009
And also I believe Frances Elizabeth Caroline Willard is listed on Wikipedia as "Frances Willard (suffragist)", so that cancels out the above error for the totals :)

Posted by Joseph Reagle at Sat Sep 26 17:10:17 2009
@ Circeus: Yes, I will check those out, or I was thinking "100 Most ..." type listings.

@ Siebrand, Kaldari: Thanks for pointing these out. I posted this hoping for some bug spotting! :) Coming up with the correct names and seeing if there were articles was the most time consuming (and error prone) task (e.g., nicknames, popular v. indigenous, maiden v. married, spelling and accents). I want an approach that is consistent/fair across both works. Yet, the source has spelling variations and typos. Then, some subjects have similar names, some subjects are listed by different names (e.g., Bonnin in the source and at EB, but Zitkala-Sa at WP), and there are other complexities (e.g., The Grimke sisters have a single page at EB, but separate pages at WP).

I will address such bugs and hope to describe my method shortly.

Posted by Kaldari at Sat Sep 26 17:51:34 2009
Before you get too far on expanding this investigation, I would like to constructively question one of its underlying assumptions: Why do you think that women's biographies would be the most apt category to indicate whether or not the dominance of male editors affects Wikipedia's topical coverage? As an active member of both WikiProject Biographies and Wikipedia's Feminism Task Force I can tell you that most of those articles weren't written by women. Just because someone is a women doesn't mean they are necessarily going to be more interested in writing women's biographies. Such an assumption seems overly simplistic to me. A couple of areas that I know have a fair number of active female editors include textile arts and children's literature. But even for those, there is no guarantee that the gender ratio of the editors is going to be reflected in the quantity or quality of the coverage. Personally, I don't think there is a reliable way to test what you are wanting to test simply because male and female Wikipedia editors cannot be put into simple boxes concerning what they are most likely to write about. Such an assumption seems sexist in and of itself, if you don't mind me saying so.

Posted by Joseph Reagle at Sun Sep 27 02:08:40 2009
On the bugs, the challenge is I want to run this over large collections and so I take advantage of probabilistic heuristics by making use of Google search API to return results it thinks are appropriate and I then compare those results in the title using another probabilistic textual comparison of the source name with the title (0.0 > 1.0). I can set the threshold on the latter and have been playing with that. But whatever I do, I want it to be even handed (i.e., making a false positive or negative) across the reference works. (Given an accurate/clean source corpus, which is harder than I anticipated.) However, I improved my use of the difflib heuristics and I just realized how to do something I'd been scratching my head about (site specific searches) through the Google AJAX API. The results are improved though it makes very little differences in the big picture conclusions one might draw. On that....

On the assumption front, it's a ("null") hypothesis given that (1) you can find both male dominance and corresponding lack of female biographic coverage in historical reference works which historians (non-controversially) imply is causal and (2) people have previously inferred (some informally, some in studies) that WP contributor representation affects topical coverage (i.e., "systemic bias".) That is, does gender imbalance in contributors affect biographical coverage relative to Britannica? This preliminary analysis appears to disprove the null hypothesis and implies support for your sense of how things work at WP.

Of course, doing the ratio comparison I discuss above would be better.

Posted by Joseph Reagle at Sun Sep 27 02:17:59 2009
Oh, and then to the more generic claim. One could do a comparison within WP. For example, if one found an authoritative listing of topics typically associated with male interests and another authoritative source on female interests (perhaps as identified from some of the surveys Wikipedians) then could see the percent of coverage on each. Of course, those two sources may differ for any number of reasons... And that said, given the biographical experiment, and Wikipedia's size, I would now be much more skeptical of claims of such bias.

Posted by Kaldari at Sun Sep 27 12:50:06 2009
I definitely agree that there is systemic bias on Wikipedia. Gender bias, however, would be the hardest of all to judge, IMO. Geographic and language bias are relatively straightforward. Race is trickier, and gender (and age) even more so, as any generalizations you could make about people's interests according to gender are going to be dubious at best and stereotypical at worst. I think in the case of gender, other variables are going to strongly compete for significance, and it will be difficult to eliminate them all. Also, there is an inherent flaw with doing comparisons against Britannica: Wikipedia is aware of and responds to what is covered in Britannica. This is probably one reason why there are no articles in your study present in Britannica but absent from Wikipedia.

With all that said, I appreciate your efforts to address this topic, as it is certainly an important one. I hope my constructive criticism hasn't come across too negatively. BTW, you'll be happy to learn that your blog post has inspired an effort to complete Wikipedia's coverage of National Women's History Project biogrpahies:
http://en.wikipedia.org/wiki/Wikipedia_talk:WikiProject_Gender_Studies/Feminism_Task_Force#National_Women.27s_History_Project_biographies_yet_to_be_written

Posted by Joseph Reagle at Mon Sep 28 07:58:54 2009
@Kaldari: I very much appreciate the comments,and thought this might prompt an effort to then document those women. Not so quickly though! :)

Posted by Axel Boldt at Sat Oct 3 11:54:58 2009
Nice analysis, thanks. One tiny point: the female/male ratio being 0.28 doesn't mean that 28% of subjects are females. 23/105 = 22% of the subjects are females. 0.28 to 1 are the odds for a subject to be female.

Posted by Joseph Reagle at Sat Oct 3 12:33:25 2009
Right you are Axel. I mispoke. I use the ratio instead of percent of total in case there are gender "unknowns". (I could also do percentage of a gender relative to all less the unknowns.)

Posted by Sue Gardner at Mon Oct 12 20:11:43 2009
Really interesting, Joseph, thanks. I've been looking for this kind of analysis.

Posted by Joseph Reagle at Tue Oct 13 15:33:47 2009
Thanks Sue. I did find a nice corpus that I'm prepping for analyzing, but don't expect to find anything contrary to what we see here.

Posted by Danielle Maedchenspiele at Wed Oct 14 09:16:55 2009
I can't believe that Britannica missed 65 articles! However, congrats for the analysis, I found it really interesting because I am studying this topic as well!

Open Communities, Media, Source, and Standards

by Joseph Reagle

reagle.org
Open Codex by Joseph Reagle

Open Codex HISTORICAL entry

2009 Sep 25 | Gender Bias in Wikipedia Coverage?

The recent controversy about gender imbalance and sexism in open content communities has been remarkable this summer, and this week's news about Shuttleworth's comments might mean it will extend into the autumn. While I think these events merit a historical and cultural analysis -- and prompts the questions if sexism increased, is it being noticed more, or both? -- I want to postpone that undertaking for the moment. Instead, I wonder if the recent demographic data that shows women are about 13% of Wikipedians affects its topical coverage?

As you can see in this Comparison of Biographies , Wikipedia does very well in its coverage of National Women's History Project (NWHP) biographies relative to Encyclopaedia Britannica .

64 entries are missing from EB; 23 from WP. 23 entries are in neither. For articles existing in both, WP articles are 6.29 times larger on average (median of 4.00).

That is, of 174 women, Wikipedia is missing articles on 23 of them. That's almost a third of those missing from Britannica , which doesn't have any articles not at Wikipedia. When both do have an article, Wikipedia articles have much more content. Of course, those are just the quantitative numbers. Even so, when I browse the actual articles, I am partial to the extra content and images of Wikipedia.

Yet, a difficulty in this work is finding a useful corpus of biographical persons. To say that there are more articles about men than women in any reference work, isn't saying much given world history. So, for this analysis I use those women recognized by the NWHP for Women's History Month. The NWHP is a nice collection in that it has both well known women and lesser-known women who are thought to be notable nonetheless. However, this only tells us that Wikipedia has greater coverage of women than a traditional encyclopedia. (And while this is one of the first large and topical -- rather than quality -- comparisons it should not be all that surprising given Wikipedia's size.) And, Wikipedians are aware of their own systemic bias and make attempts to counter it. For example, those recognized by Black History Month were the focus of a WikiProject that documented every person recognized. (Ironically, this list was taken from Britannica. And perhaps the NWHP list will prompt a similar project at Wikipedia, which is why I use permanent links to the specific versions I analyzed.)

What would really be nice is a source corpus of notable persons, both male and female. I could then compare this against Wikipedia and Britannica to see how they fare relative to the source corpus. That is, a source corpus of 100 people might recognize 75 men and 25 women (25% female), and if one of the references had a 60/15 split, it'd be less "feminine" (20%) than the source. How, then, does each reference work compare to each other, relative to their source? If you have a suggestion for corpus, please leave a comment.

Finally, while speaking with Nora about this, she also raised the question of if the gender ratio of disruptive editors differs from that of the larger community? Our hypothesis is disruptive editors might be disproportionately male. But who can say? Unfortunately, I expect it's difficult to get survey responses from such editors.

this entry posted to social/wikipedia ;
comments (17)

Posted by Sage Ross at Sat Sep 26 09:17:17 2009
Nice post, Joseph.  Did you see this? http://www.washingtoncitypaper.com/blogs/sexist/2009/09/25/sexist-beatdown-consensual-incest-and-john-phillips-fanboys-edition/

Of course, that's not necessarily as reflective of Wikipedia culture as of the broader culture (I hope), but low-grade sexism is pretty common to observe at the level of individual articles.

Posted by Joseph Reagle at Sat Sep 26 10:11:44 2009
Thanks for the comment Sage. I didn't see the Phillips' thing. And I'm not sure if using the term "consensual" to differentiate the subsequent relationship is horribly sexist, but it is interesting that it was added. In any case, the character of that relationship appears to be a complex issue and I expect will continue to be debated at large -- and on WP.

Posted by Circeus at Sat Sep 26 11:24:55 2009
How about the national biographic encyclopedias? we have a Category (Biographical dictionaries by country) documenting them.

Posted by Siebrand at Sat Sep 26 14:08:33 2009
Interesting. Analysis may contain errors.

The table shows that there would be no article on Judith Francisca Baca, but [[Judith Baca]] has existed since 2007-11-20. The redirect [[Judith Francisca Baca]] has existed since mid 2006.

I checked the next 10 reported as missing from Wikipedia also, and found no additional errors.

Posted by Kaldari at Sat Sep 26 16:13:30 2009
There is one more error: Mary Louise Defender Wilson is listed incorrectly as Mary Louise Wilson (who is a different person).

Posted by Kaldari at Sat Sep 26 16:16:48 2009
And also I believe Frances Elizabeth Caroline Willard is listed on Wikipedia as "Frances Willard (suffragist)", so that cancels out the above error for the totals :)

Posted by Joseph Reagle at Sat Sep 26 17:10:17 2009
@ Circeus: Yes, I will check those out, or I was thinking "100 Most ..." type listings.

@ Siebrand, Kaldari: Thanks for pointing these out. I posted this hoping for some bug spotting! :) Coming up with the correct names and seeing if there were articles was the most time consuming (and error prone) task (e.g., nicknames, popular v. indigenous, maiden v. married, spelling and accents). I want an approach that is consistent/fair across both works. Yet, the source has spelling variations and typos. Then, some subjects have similar names, some subjects are listed by different names (e.g., Bonnin in the source and at EB, but Zitkala-Sa at WP), and there are other complexities (e.g., The Grimke sisters have a single page at EB, but separate pages at WP).

I will address such bugs and hope to describe my method shortly.

Posted by Kaldari at Sat Sep 26 17:51:34 2009
Before you get too far on expanding this investigation, I would like to constructively question one of its underlying assumptions: Why do you think that women's biographies would be the most apt category to indicate whether or not the dominance of male editors affects Wikipedia's topical coverage? As an active member of both WikiProject Biographies and Wikipedia's Feminism Task Force I can tell you that most of those articles weren't written by women. Just because someone is a women doesn't mean they are necessarily going to be more interested in writing women's biographies. Such an assumption seems overly simplistic to me. A couple of areas that I know have a fair number of active female editors include textile arts and children's literature. But even for those, there is no guarantee that the gender ratio of the editors is going to be reflected in the quantity or quality of the coverage. Personally, I don't think there is a reliable way to test what you are wanting to test simply because male and female Wikipedia editors cannot be put into simple boxes concerning what they are most likely to write about. Such an assumption seems sexist in and of itself, if you don't mind me saying so.

Posted by Joseph Reagle at Sun Sep 27 02:08:40 2009
On the bugs, the challenge is I want to run this over large collections and so I take advantage of probabilistic heuristics by making use of Google search API to return results it thinks are appropriate and I then compare those results in the title using another probabilistic textual comparison of the source name with the title (0.0 > 1.0). I can set the threshold on the latter and have been playing with that. But whatever I do, I want it to be even handed (i.e., making a false positive or negative) across the reference works. (Given an accurate/clean source corpus, which is harder than I anticipated.) However, I improved my use of the difflib heuristics and I just realized how to do something I'd been scratching my head about (site specific searches) through the Google AJAX API. The results are improved though it makes very little differences in the big picture conclusions one might draw. On that....

On the assumption front, it's a ("null") hypothesis given that (1) you can find both male dominance and corresponding lack of female biographic coverage in historical reference works which historians (non-controversially) imply is causal and (2) people have previously inferred (some informally, some in studies) that WP contributor representation affects topical coverage (i.e., "systemic bias".) That is, does gender imbalance in contributors affect biographical coverage relative to Britannica? This preliminary analysis appears to disprove the null hypothesis and implies support for your sense of how things work at WP.

Of course, doing the ratio comparison I discuss above would be better.

Posted by Joseph Reagle at Sun Sep 27 02:17:59 2009
Oh, and then to the more generic claim. One could do a comparison within WP. For example, if one found an authoritative listing of topics typically associated with male interests and another authoritative source on female interests (perhaps as identified from some of the surveys Wikipedians) then could see the percent of coverage on each. Of course, those two sources may differ for any number of reasons... And that said, given the biographical experiment, and Wikipedia's size, I would now be much more skeptical of claims of such bias.

Posted by Kaldari at Sun Sep 27 12:50:06 2009
I definitely agree that there is systemic bias on Wikipedia. Gender bias, however, would be the hardest of all to judge, IMO. Geographic and language bias are relatively straightforward. Race is trickier, and gender (and age) even more so, as any generalizations you could make about people's interests according to gender are going to be dubious at best and stereotypical at worst. I think in the case of gender, other variables are going to strongly compete for significance, and it will be difficult to eliminate them all. Also, there is an inherent flaw with doing comparisons against Britannica: Wikipedia is aware of and responds to what is covered in Britannica. This is probably one reason why there are no articles in your study present in Britannica but absent from Wikipedia.

With all that said, I appreciate your efforts to address this topic, as it is certainly an important one. I hope my constructive criticism hasn't come across too negatively. BTW, you'll be happy to learn that your blog post has inspired an effort to complete Wikipedia's coverage of National Women's History Project biogrpahies:
http://en.wikipedia.org/wiki/Wikipedia_talk:WikiProject_Gender_Studies/Feminism_Task_Force#National_Women.27s_History_Project_biographies_yet_to_be_written

Posted by Joseph Reagle at Mon Sep 28 07:58:54 2009
@Kaldari: I very much appreciate the comments,and thought this might prompt an effort to then document those women. Not so quickly though! :)

Posted by Axel Boldt at Sat Oct 3 11:54:58 2009
Nice analysis, thanks. One tiny point: the female/male ratio being 0.28 doesn't mean that 28% of subjects are females. 23/105 = 22% of the subjects are females. 0.28 to 1 are the odds for a subject to be female.

Posted by Joseph Reagle at Sat Oct 3 12:33:25 2009
Right you are Axel. I mispoke. I use the ratio instead of percent of total in case there are gender "unknowns". (I could also do percentage of a gender relative to all less the unknowns.)

Posted by Sue Gardner at Mon Oct 12 20:11:43 2009
Really interesting, Joseph, thanks. I've been looking for this kind of analysis.

Posted by Joseph Reagle at Tue Oct 13 15:33:47 2009
Thanks Sue. I did find a nice corpus that I'm prepping for analyzing, but don't expect to find anything contrary to what we see here.

Posted by Danielle Maedchenspiele at Wed Oct 14 09:16:55 2009
I can't believe that Britannica missed 65 articles! However, congrats for the analysis, I found it really interesting because I am studying this topic as well!

Open Communities, Media, Source, and Standards

by Joseph Reagle

reagle.org
Open Codex by Joseph Reagle

Open Codex HISTORICAL entry

2009 Sep 25 | Gender Bias in Wikipedia Coverage?

The recent controversy about gender imbalance and sexism in open content communities has been remarkable this summer, and this week's news about Shuttleworth's comments might mean it will extend into the autumn. While I think these events merit a historical and cultural analysis -- and prompts the questions if sexism increased, is it being noticed more, or both? -- I want to postpone that undertaking for the moment. Instead, I wonder if the recent demographic data that shows women are about 13% of Wikipedians affects its topical coverage?

As you can see in this Comparison of Biographies , Wikipedia does very well in its coverage of National Women's History Project (NWHP) biographies relative to Encyclopaedia Britannica .

64 entries are missing from EB; 23 from WP. 23 entries are in neither. For articles existing in both, WP articles are 6.29 times larger on average (median of 4.00).

That is, of 174 women, Wikipedia is missing articles on 23 of them. That's almost a third of those missing from Britannica , which doesn't have any articles not at Wikipedia. When both do have an article, Wikipedia articles have much more content. Of course, those are just the quantitative numbers. Even so, when I browse the actual articles, I am partial to the extra content and images of Wikipedia.

Yet, a difficulty in this work is finding a useful corpus of biographical persons. To say that there are more articles about men than women in any reference work, isn't saying much given world history. So, for this analysis I use those women recognized by the NWHP for Women's History Month. The NWHP is a nice collection in that it has both well known women and lesser-known women who are thought to be notable nonetheless. However, this only tells us that Wikipedia has greater coverage of women than a traditional encyclopedia. (And while this is one of the first large and topical -- rather than quality -- comparisons it should not be all that surprising given Wikipedia's size.) And, Wikipedians are aware of their own systemic bias and make attempts to counter it. For example, those recognized by Black History Month were the focus of a WikiProject that documented every person recognized. (Ironically, this list was taken from Britannica. And perhaps the NWHP list will prompt a similar project at Wikipedia, which is why I use permanent links to the specific versions I analyzed.)

What would really be nice is a source corpus of notable persons, both male and female. I could then compare this against Wikipedia and Britannica to see how they fare relative to the source corpus. That is, a source corpus of 100 people might recognize 75 men and 25 women (25% female), and if one of the references had a 60/15 split, it'd be less "feminine" (20%) than the source. How, then, does each reference work compare to each other, relative to their source? If you have a suggestion for corpus, please leave a comment.

Finally, while speaking with Nora about this, she also raised the question of if the gender ratio of disruptive editors differs from that of the larger community? Our hypothesis is disruptive editors might be disproportionately male. But who can say? Unfortunately, I expect it's difficult to get survey responses from such editors.

this entry posted to social/wikipedia ;
comments (17)

Posted by Sage Ross at Sat Sep 26 09:17:17 2009
Nice post, Joseph.  Did you see this? http://www.washingtoncitypaper.com/blogs/sexist/2009/09/25/sexist-beatdown-consensual-incest-and-john-phillips-fanboys-edition/

Of course, that's not necessarily as reflective of Wikipedia culture as of the broader culture (I hope), but low-grade sexism is pretty common to observe at the level of individual articles.

Posted by Joseph Reagle at Sat Sep 26 10:11:44 2009
Thanks for the comment Sage. I didn't see the Phillips' thing. And I'm not sure if using the term "consensual" to differentiate the subsequent relationship is horribly sexist, but it is interesting that it was added. In any case, the character of that relationship appears to be a complex issue and I expect will continue to be debated at large -- and on WP.

Posted by Circeus at Sat Sep 26 11:24:55 2009
How about the national biographic encyclopedias? we have a Category (Biographical dictionaries by country) documenting them.

Posted by Siebrand at Sat Sep 26 14:08:33 2009
Interesting. Analysis may contain errors.

The table shows that there would be no article on Judith Francisca Baca, but [[Judith Baca]] has existed since 2007-11-20. The redirect [[Judith Francisca Baca]] has existed since mid 2006.

I checked the next 10 reported as missing from Wikipedia also, and found no additional errors.

Posted by Kaldari at Sat Sep 26 16:13:30 2009
There is one more error: Mary Louise Defender Wilson is listed incorrectly as Mary Louise Wilson (who is a different person).

Posted by Kaldari at Sat Sep 26 16:16:48 2009
And also I believe Frances Elizabeth Caroline Willard is listed on Wikipedia as "Frances Willard (suffragist)", so that cancels out the above error for the totals :)

Posted by Joseph Reagle at Sat Sep 26 17:10:17 2009
@ Circeus: Yes, I will check those out, or I was thinking "100 Most ..." type listings.

@ Siebrand, Kaldari: Thanks for pointing these out. I posted this hoping for some bug spotting! :) Coming up with the correct names and seeing if there were articles was the most time consuming (and error prone) task (e.g., nicknames, popular v. indigenous, maiden v. married, spelling and accents). I want an approach that is consistent/fair across both works. Yet, the source has spelling variations and typos. Then, some subjects have similar names, some subjects are listed by different names (e.g., Bonnin in the source and at EB, but Zitkala-Sa at WP), and there are other complexities (e.g., The Grimke sisters have a single page at EB, but separate pages at WP).

I will address such bugs and hope to describe my method shortly.

Posted by Kaldari at Sat Sep 26 17:51:34 2009
Before you get too far on expanding this investigation, I would like to constructively question one of its underlying assumptions: Why do you think that women's biographies would be the most apt category to indicate whether or not the dominance of male editors affects Wikipedia's topical coverage? As an active member of both WikiProject Biographies and Wikipedia's Feminism Task Force I can tell you that most of those articles weren't written by women. Just because someone is a women doesn't mean they are necessarily going to be more interested in writing women's biographies. Such an assumption seems overly simplistic to me. A couple of areas that I know have a fair number of active female editors include textile arts and children's literature. But even for those, there is no guarantee that the gender ratio of the editors is going to be reflected in the quantity or quality of the coverage. Personally, I don't think there is a reliable way to test what you are wanting to test simply because male and female Wikipedia editors cannot be put into simple boxes concerning what they are most likely to write about. Such an assumption seems sexist in and of itself, if you don't mind me saying so.

Posted by Joseph Reagle at Sun Sep 27 02:08:40 2009
On the bugs, the challenge is I want to run this over large collections and so I take advantage of probabilistic heuristics by making use of Google search API to return results it thinks are appropriate and I then compare those results in the title using another probabilistic textual comparison of the source name with the title (0.0 > 1.0). I can set the threshold on the latter and have been playing with that. But whatever I do, I want it to be even handed (i.e., making a false positive or negative) across the reference works. (Given an accurate/clean source corpus, which is harder than I anticipated.) However, I improved my use of the difflib heuristics and I just realized how to do something I'd been scratching my head about (site specific searches) through the Google AJAX API. The results are improved though it makes very little differences in the big picture conclusions one might draw. On that....

On the assumption front, it's a ("null") hypothesis given that (1) you can find both male dominance and corresponding lack of female biographic coverage in historical reference works which historians (non-controversially) imply is causal and (2) people have previously inferred (some informally, some in studies) that WP contributor representation affects topical coverage (i.e., "systemic bias".) That is, does gender imbalance in contributors affect biographical coverage relative to Britannica? This preliminary analysis appears to disprove the null hypothesis and implies support for your sense of how things work at WP.

Of course, doing the ratio comparison I discuss above would be better.

Posted by Joseph Reagle at Sun Sep 27 02:17:59 2009
Oh, and then to the more generic claim. One could do a comparison within WP. For example, if one found an authoritative listing of topics typically associated with male interests and another authoritative source on female interests (perhaps as identified from some of the surveys Wikipedians) then could see the percent of coverage on each. Of course, those two sources may differ for any number of reasons... And that said, given the biographical experiment, and Wikipedia's size, I would now be much more skeptical of claims of such bias.

Posted by Kaldari at Sun Sep 27 12:50:06 2009
I definitely agree that there is systemic bias on Wikipedia. Gender bias, however, would be the hardest of all to judge, IMO. Geographic and language bias are relatively straightforward. Race is trickier, and gender (and age) even more so, as any generalizations you could make about people's interests according to gender are going to be dubious at best and stereotypical at worst. I think in the case of gender, other variables are going to strongly compete for significance, and it will be difficult to eliminate them all. Also, there is an inherent flaw with doing comparisons against Britannica: Wikipedia is aware of and responds to what is covered in Britannica. This is probably one reason why there are no articles in your study present in Britannica but absent from Wikipedia.

With all that said, I appreciate your efforts to address this topic, as it is certainly an important one. I hope my constructive criticism hasn't come across too negatively. BTW, you'll be happy to learn that your blog post has inspired an effort to complete Wikipedia's coverage of National Women's History Project biogrpahies:
http://en.wikipedia.org/wiki/Wikipedia_talk:WikiProject_Gender_Studies/Feminism_Task_Force#National_Women.27s_History_Project_biographies_yet_to_be_written

Posted by Joseph Reagle at Mon Sep 28 07:58:54 2009
@Kaldari: I very much appreciate the comments,and thought this might prompt an effort to then document those women. Not so quickly though! :)

Posted by Axel Boldt at Sat Oct 3 11:54:58 2009
Nice analysis, thanks. One tiny point: the female/male ratio being 0.28 doesn't mean that 28% of subjects are females. 23/105 = 22% of the subjects are females. 0.28 to 1 are the odds for a subject to be female.

Posted by Joseph Reagle at Sat Oct 3 12:33:25 2009
Right you are Axel. I mispoke. I use the ratio instead of percent of total in case there are gender "unknowns". (I could also do percentage of a gender relative to all less the unknowns.)

Posted by Sue Gardner at Mon Oct 12 20:11:43 2009
Really interesting, Joseph, thanks. I've been looking for this kind of analysis.

Posted by Joseph Reagle at Tue Oct 13 15:33:47 2009
Thanks Sue. I did find a nice corpus that I'm prepping for analyzing, but don't expect to find anything contrary to what we see here.

Posted by Danielle Maedchenspiele at Wed Oct 14 09:16:55 2009
I can't believe that Britannica missed 65 articles! However, congrats for the analysis, I found it really interesting because I am studying this topic as well!

Open Communities, Media, Source, and Standards

by Joseph Reagle

reagle.org