A few words of advice

It is important to present CrowdGrader to students in a class, and explain what are its benefits, and what are the rules of the game.

For the students, the main benefits are two:

• • They usually get far more feedback on their work, than they would get from over-worked teaching assistants.

  • They get to see what other students are doing, and they can learn from the work of others. 

In exchange for this, they need to put in some amount of work in reviewing the work of others. 

The instructor should explain to the students that their final grade is determined both by the quality of their work, and by the precision of the grades they give, and the helpfulness of the reviews they write, as explained in detail here.

To ensure that the grades assigned by the students are somewhat consistent, the instructor should provide as clear and unambiguous grading rubric as possible. 

It is also important to reassure students that CrowdGrader has sophisticated algorithms to remove outliers and compute high-quality consensus grades, so that they do not need to worry because of a single inappropriate grade or review. 

All the same, the instructor, and any teaching assistants, need to be involved in the assignment, read many reviews, and intervene in cases of obvious unfairness.  Instructors can easily monitor the review process, and single out the submissions where the review grades have most discrepancy for re-grading.  The creator of CrowdGrader generally asks the TAs to regrade all submissions where the review grade delta is over a threshold. 

CrowdGrader can also be used in auto-pilot mode, with very little instructor engagement, but it is not the way that leads to the best learning experience for the students.