Human Computation
Monday: volunteer crowdsourcing: motivation, scale, and quality.
Wednesday: human coordination: design patterns and programming models.
Discussion Questions:
Monday
- When is input agreement useful? when is output agreement useful? consider examples of problems that would be better solved with one vs the other
- In ClickWorkers, how could the researchers verify that the work was any good?
- What are the key design take-aways from how eBird evolved over time?
- What are the various forms of motivations/incentives used in these human computation systems? What makes them good fits for their application’s data collection goals?
Wednesday
- What are the kinds of tasks that have been solved with human computers that can’t be solved with digital computers?
- What are the unique challenges of crowd computing over digital computing? How do the programming models suggested in Miller et. al.’s article address these challenges?
- What is crash and re-run? Why is it useful?
- Identify the design patterns used for programming humans on Mechanical Turk to produce good quality work. Can you see how these patterns can be useful for other problems? What are their shortcomings?
Readings and Media:
Monday
- [PLAY] The Sheep Market, Aaron Koblin.
- [PLAY] The Johnny Cash Project, Chris Milk.
- [READ] Games with a Purpose, Luis von Ahn, Computer, June 2006.
- [WATCH] Duolingo: The next chapter in human computation, Luis von Ahn, Ted Talk, April 2011.
- [READ] Can Distributed Volunteers Accomplish Massive Data Analysis Tasks?, Kanefsky et al., Lunar and Planetary Science, 2001.
- [SKIM] eBird: A citizen-based bird observation network in the biological sciences, Sillivan et al., Biological Conservation, October 2009.
Wednesday
- [READ] Crowd Control, Leah Hoffmann, CACM, March 2009.
- [READ] Heads in the Cloud, Miller et al., XRDS, Winter 2010.
XKCD: Suspicion
