Sometimes things happen in the world that researchers could never make happen on there own, but when it does happen, much can be learned. For example, in Group Size And Incentives To Contribute: A Natural Experiment At Chinese Wikipedia Zhang and Zhu analyzed the effect of China blocking access to the Chinese Wikipedia on non-mainland (non-blocked) contributors (e.g., those in Taiwan and Hong Kong). They found that after a block, non-blocked contributors decreased their contributions by 42.8% on average; decreases in contributions were further associated with those who had user/talk pages and for those whose collaborators were blocked. So losing one’s collaborators apparently lessens motivation.
The SOPA/PIPA protests of January 18 were a natural experiment as well. Some things I’m wondering (feel free to add any of your own):
- What is an cumulative total amount of ad revenue lost by participating sites? In lobbying terms, was this a good value?
- Did Digg (or others) get more traffic given Reddit’s participation in the “blackout”? (Hence, non-participating sites were “free-riders” benefiting from participating sites?)
- Was overall Internet traffic higher or lower that day?
- Did Google queries increase since Wikipedia was largely unavailable?
- Was Wikipedia busier the following day than typical?
Joseph Reagle on 2012-01-26
Good point. Even so, people's strategies for dealing with the blackout would be interesting as well.
Benjamin Mako Hill on 2012-01-25
So, it probably qualifies as a natural experiment for some of these questions but not for others.
In the context of a natural experiment using this kind of design, what you are effectively saying is that along some variable you are interested in, we can treat the people/events/whatever on either side of the disruption as being otherwise equal in expectations. We can sort of think of this as saying that on side is the treatment and on the other is the control.
The problem, of course, is that these blackouts weren't surprising. Wikipedia told everyone for a day in advance that they would black out. The result is that folks had the opportunity to change their behavior. They could get in a little bit of editing or research before the deadline or download some pages they knew they might need. In other words, they could essentially jump out of the treatment into the control, or something like that.
For many questions, that might not be a concern. I agree it is an interesting question.