ML Reproducibility Challenge 2021

We are happy to announce the official launch of the ML Reproducibility Challenge 2021 [1] (RC2021)!

Over the last four years we have hosted a series of Machine Learning reproducibility challenges, in partnership with PapersWithCode and OpenReview (see V1 [2], V2 [3], V3 [4], V4 [1]). The challenges have led to several reproducibility reports, some of which were published in three volumes of the journal ReScience (see J1 [5], J2 [6], J3 [7]), and have helped emphasize the importance of reproducibility, sound scientific methodology, and robust results. Course instructors have incorporated the ML reproducibility challenge as a component of their course, typically as the final course project.

For more details, our blog post [8] describes how educators can incorporate the ML reproducibility challenge into their course curriculum. You may also read the experience of course instructors at University of Amsterdam on using the Reproducibility Challenge as a course requirement in their blog post [9]. If you are teaching a course where this may be appropriate, we strongly encourage you to participate! If you are a student in an ML course and want to participate in it, you are more than welcome, and we encourage you to share this information to your instructor/TAs. Submissions are also most welcome from the industry as well.

We will be accepting reproducibility reports of papers published in:
* ICML 2021 Accepted Papers
* ICLR 2021 Accepted Papers
* NeurIPS 2021 Accepted Papers
* ACL-IJCNLP 2021 Accepted Papers
* EMNLP 2021 Accepted Papers
* CVPR 2021 Accepted Papers
* ICCV 2021 Accepted Papers
* AAAI 2021 Accepted Papers
* IJCAI 2021 Accepted Papers

Key Dates:
* Notification of the challenge: August 31st, 2021
* Deadline to submit reports: February 4th, 2022
* Notification of acceptance: May 15th, 2022

More information regarding the task and related resources are available on our website [1]. Reports will undergo a comprehensive review process, and selected reports will be featured in ReScience journal [10], as well as linked to their original papers in PapersWithCode [11]. If you are a course instructor and choose to include this in your course, please fill out this form [12] to feature your course on our website! All information will be treated confidentially and used only to send important updates related to this challenge.

We hope you will participate in this challenge, include this in your course, and encourage your colleagues and students to participate!

Koustuv Sinha, Jesse Dodge, Sasha Luccioni, Jessica Forde, Robert Stojnic and Joelle Pineau 2021 ML Reproducibility Challenge [1] Organizing Team

Links:
——
[1] https://paperswithcode.com/rc2021
[2] https://www.cs.mcgill.ca/~jpineau/ICLR2018-ReproducibilityChallenge.html
[3] https://www.cs.mcgill.ca/~jpineau/ICLR2019-ReproducibilityChallenge.html
[4] https://reproducibility-challenge.github.io/neurips2019/
[5] https://rescience.github.io/read/#volume-5-2019
[6] https://rescience.github.io/read/#volume-6-2020
[7] https://rescience.github.io/read/#volume-7-2021
[8] https://medium.com/paperswithcode/the-reproducibility-challenge-as-an-educational-tool-cd1596e3716c
[9] https://medium.com/paperswithcode/case-study-how-your-course-can-incorporate-the-reproducibility-challenge-76e260a2b59
[10] https://rescience.github.io/
[11] https://paperswithcode.com/
[12] https://forms.gle/4ogw4QrTQacjj4h37
[13] https://paperswithcode.com/rc2020
[14] https://docs.google.com/forms/d/e/1FAIpQLSfNDVkj4slprEYXSuR2Vb8in8dGQnwPU-ShkVIW93cyroZOrA/viewform