CS 497: Peer Grading

Current Term: Fall 2017

Synopsis:
This interdisciplinary graduate seminar explores the science of peer grading systems.  A peer grading system is an online tool that collects student submissions, and assigns review tasks to the students and graders, and aggregates reviews to produce assessments of both the submissions and the peer reviews.  Peer grading systems can improve learning outcomes and lessen the time and effort necessary to give students high quality feedback.  Students in this seminar will read and present research papers on topics that include peer prediction, rubric design, scoring rules, auction design, human computation, machine learning, and measurement of learning outcomes.  These papers are from fields of algorithms, game theory, machine learning, human computer interaction, and learning science.  Students will complete a research project that is either a theoretical or empirical study related to peer grading; empirical studies can be based on data collected in a peer grading system that is being developed and used in Northwestern CS classes.

Prerequisites: This interdisciplinary graduate seminar is targeted to Ph.D. students with knowledge in areas relevant to peer grading: algorithms, game theory, machine learning, learning science, or human computer interaction. Advanced undergraduates and masters students are recommended to consult the instructor before enrolling.

Schedule:

Week 0 (Sept. 19): Introductory lecture on peer grading [slides]

    (no readings)

Week 1 (Sept. 26): Peer grading systems:

Week 2 (Oct. 3): Peer prediction:

Week 3 (Oct. 10): Eliciting peer feedback:

Week 4 (Oct. 17): Incentivizing effort and accuracy:

Week 5 (Oct. 24): Assigning Reviews:

Week 6 (Oct. 31): Cardinal grade aggregation:

Week 7 (Nov. 7): Accuracy of peer reviews:

Week 8 (Nov. 14): Ordinal grade aggregation:

Week 9 (Nov. 21): Evaluating learning outcomes:

Week 10 (Nov. 27): Student presentations

    (no readings)