Evaluating Performance Consistency in Competitive Programming: Educational Implications and Contest Design Insights
Article Sidebar
Main Article Content
Abstract
Competitive programming (CP) contests are often treated as interchangeable proxies for algorithmic skill, yet the extent to which results at lower contest tiers anticipate performance at higher tiers, and how closely any tier resembles the ubiquitous online-contest circuit, remains unclear.
We analyze ten years (2015--2024) of International Collegiate Programming Contest (ICPC) standings, comprising five long-running superregional championships (Africa \& Arab, Asia East, Asia West, North America, and Northern Eurasia), associated local regionals of North America and Northern Eurasia, and the World Finals. For 366 World Finalist teams (2021--2024) we augment the dataset with pre-contest Codeforces ratings. Pairwise rank alignment is measured with Kendall's $\tau$.
Overall, superregional ranks predict World Final ranks only moderately (weighted $\tau=0.407$), but regional-to-superregional consistency varies widely: Northern Eurasia exhibits the strongest alignment ($\tau=0.521$) while Asia West exhibits the weakest ($\tau=0.188$). Internal consistency within a region can exceed its predictive value for Worlds --- e.g., Northern Eurasia and North America regionals vs. superregionals ($\tau=0.666$ and $\tau=0.577$, respectively). Codeforces ratings correlate more strongly with World Final results ($\tau=0.596$) than any single ICPC tier, suggesting that high-frequency online contests capture decisive skill factors that many superregional sets miss.
We argue that contest organizers can improve both fairness and pedagogical value by aligning problem style and selection rules with the formats that demonstrably differentiate teams, in particular the Northern-Eurasian model and well-curated online rounds. All data, scripts, and additional analyses are publicly released to facilitate replication and further study.
Article Details
Issue
Section
This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Publisher agrees to provide to Author within 14 days of first publication and at no charge an electronic copy of the published Article in a format, such as the Portable Document Format (.pdf), that preserves final page layout, formatting, and content. No technical restriction, such as security settings, will be imposed to prevent copying or printing of the document.
In addition, where applicable and without limiting the retention of rights above, Publisher acknowledges that Author’s assignment of copyright or Author’s grant of exclusive rights in the Publication Agreement is subject to Author’s prior grant of a non-exclusive copyright license to Author’s employing institution and/or to a funding entity that financially supported the research reflected in the Article as part of an agreement between Author or Author’s employing institution and such funding entity, such as an agency of the United States government.
How to Cite
Share
References
@Misc{icpc-regional-results2024,
author = "icpc.foundation",
title = "The {ICPC} Regional Results",
howpublished = {url{https://icpc.global/regionals/results}},
note = {Accessed: December 3, 2024}
}
@Misc{codeforces-rating2024,
author = "Federico Nahuel Quijada",
title = "Teams going to {ICPC} {WF} 2024 ({Kazakhstan} 2024) --- {WIP} List",
howpublished = {url{https://codeforces.com/blog/entry/129887}},
note = {Accessed: December 3, 2024}
}
@Misc{codeforces-rating2023,
author = "Federico Nahuel Quijada",
title = "Teams going to {ICPC} {WF} 2023 ({Egypt} 2023, 2nd final) --- {WIP} List",
howpublished = {url{https://codeforces.com/blog/entry/117183}},
note = {Accessed: December 3, 2024}
}
@Misc{codeforces-rating2022,
author = "Federico Nahuel Quijada",
title = "Teams going to {ICPC} {WF} 2021 ({Dhaka} 2022) --- {WIP} List",
howpublished = {url{https://codeforces.com/blog/entry/102593}},
note = {Accessed: December 3, 2024}
}
@misc{luo2024codeforces,
title={The Reason You are Bad at Codeforces — You are Not Russian Enough},
url={https://zhtluo.com/cp/the-reason-you-are-bad-at-codeforces-you-are-not-russian-enough.html},
author={Luo, Zhongtang},
year={2024},
month={Feb}
}
@INPROCEEDINGS{mascio2018framework,
author={Di Mascio, Tania and Laura, Luigi and Temperini, Marco},
booktitle={2018 17th International Conference on Information Technology Based Higher Education and Training (ITHET)},
title={A Framework for Personalized Competitive Programming Training},
year={2018},
volume={},
number={},
pages={1-8},
keywords={Programming profession;Training;Task analysis;Data structures;Google;Heuristic algorithms},
doi={10.1109/ITHET.2018.8424620}
}
@INPROCEEDINGS{lertbanjongngam2022empirical,
author={Lertbanjongngam, Sila and Chinthanet, Bodin and Ishio, Takashi and Kula, Raula Gaikovina and Leelaprute, Pattara and Manaskasemsak, Bundit and Rungsawang, Arnon and Matsumoto, Kenichi},
booktitle={2022 IEEE 16th International Workshop on Software Clones (IWSC)},
title={An Empirical Evaluation of Competitive Programming AI: A Case Study of AlphaCode},
year={2022},
volume={},
number={},
pages={10-15},
keywords={Codes;Conferences;Natural languages;Cloning;Manuals;Programming;Licenses;code generation;code similarity;code performance},
doi={10.1109/IWSC55060.2022.00010}
}
@INPROCEEDINGS{bandeira2019teaching,
author={Bandeira, Ian Nery and Machado, Thiago Veras and Dullens, Vitor F. and Canedo, Edna Dias},
booktitle={2019 IEEE Frontiers in Education Conference (FIE)},
title={Competitive programming: A teaching methodology analysis applied to first-year programming classes},
year={2019},
volume={},
number={},
pages={1-8},
keywords={Education;Programming profession;Sociology;Statistics;Computer languages;Competitive programming;teaching methodology;programming logic and algorithms subjects;empirical studies in software engineering},
doi={10.1109/FIE43999.2019.9028518}
}
@article{yuen2023computationalThinking,
author = {Yuen, Kevin K. F. and Liu, Dennis Y. W. and Leong, Hong Va},
title = {Competitive programming in computational thinking and problem solving education},
journal = {Computer Applications in Engineering Education},
volume = {31},
number = {4},
pages = {850-866},
keywords = {competitive programming, computational thinking education, contest education, learning technologies},
doi = {https://doi.org/10.1002/cae.22610},
url = {https://onlinelibrary.wiley.com/doi/abs/10.1002/cae.22610},
eprint = {https://onlinelibrary.wiley.com/doi/pdf/10.1002/cae.22610},
abstract = {Abstract Building practical programming competency requires a long-lasting journey of discovery, trial and error, learning and improvement. This article presents essential findings of a case study of a Python programming contest with an automatic judgement system for Competitive Programming training extending the learning experiences for students in an introductory course, computational thinking and problem-solving. The benefits and challenges are discussed. Due to the coronavirus disease 2019 (COVID-19) epidemic, a hybrid model of the contest was adopted, that is, some students participated in the contest on-site, while others participated remotely. To alleviate human effort in judging the submissions, the DOMjudge platform, a web-based automatic judgement system, has been deployed as an online automatic judgement system and contest management in competitive programming. The implementation roadmap and framework were provided. The contest problems and contestants' performances were discussed. Not many junior contestants could solve at least one problem(s), and competitive computing training should be offered if the students are keen on open competitions. An empirical study was conducted to evaluate the student feedback after the contest. Preliminary results revealed that the contest offering the chance to stimulate student learning interests could enhance their independent learning, innovative thinking and problem-solving skills, and could thus lead to the overall benefits of the learning experiences, which further encourage them to participate in future contests to improve their learning and therefore enhance their employability. Employers often treasure student experiences in competitive programming events, like association for computing machinery programming contests, Google Code Jam or Microsoft Imagine Cup. Sharp vision requiring skills to tackle unseen problems within a short period is also instrumental to students planning for graduate school.},
year = {2023}
}
@InProceedings{luo2021innovative,
author="Luo, Yongjun and Zheng, Hong",
editor="Sugumaran, Vijayan and Xu, Zheng and Zhou, Huiyu",
title="An Innovative Teaching Mode Based on Programming Contest",
booktitle="Application of Intelligent Systems in Multi-modal Information Analytics",
year="2021",
publisher="Springer International Publishing",
address="Cham",
pages="469--477",
abstract="Starting from the needs and orientation of the cultivation of innovative talents in computer science, combined with the current years of programming competitions in universities, the current situation of the teaching curriculum and the problems that need to be solved are analyzed, and the role of programming competition in the cultivation of innovative entrepreneurs is discussed. Combining the programming competition with the practical teaching of the course, the teaching reform of software design practice based on programming competition is proposed. Through the follow-up observation of the students who participated in the programming competition and won the award, it proves that the programming competition plays an important role in cultivating innovative and entrepreneurial talents. In terms of technical ability, self-management, team building, industry awareness and other aspects, the advantages are obvious, and a large proportion of self-employment and success.",
isbn="978-3-030-51556-0"
}
@article{raman2018students,
title={Students motivation for adopting programming contests: Innovation-diffusion perspective},
author={Raman, Raghu and Vachharajani, Hardik and Achuthan, Krishnashree},
journal={Education and Information Technologies},
volume={23},
number={5},
pages={1919--1932},
year={2018},
publisher={Springer}
}
@inproceedings{bloomfield2016programming,
title={A programming contest strategy guide},
author={Bloomfield, Aaron and Sotomayor, Borja},
booktitle={Proceedings of the 47th ACM technical symposium on computing science education},
pages={609--614},
year={2016}
}
@article{blum2023competitive,
title={Competitive programming participation rates: an examination of trends in US ICPC regional contests},
author={Blum, Jeremy J},
journal={Discover Education},
volume={2},
number={1},
pages={11},
year={2023},
publisher={Springer}
}
@article{madeira2009methodology,
title={A methodology and a framework proposal for competitive learning in programming courses},
author={Madeira, R Neves and Silva, Bruno and Grueau, C},
journal={Research, Reflections and Innovations in Integrating ICT in Education},
volume={1},
pages={315--319},
year={2009},
publisher={Citeseer}
}
@article{ribeiro2008early,
title={Early introduction of competitive programming},
author={Ribeiro, Pedro and Guerreiro, Pedro},
journal={Olympiads in Informatics},
volume={2},
pages={149--162},
year={2008}
}
@inproceedings{guerra2024enhancing,
title={Enhancing Problem-Solving Skills: The Synergy of Competitive Programming and Gamification Strategy},
author={Guerra, Luis H Gonz{'a}lez and Ju{'a}rez, Gilberto Huesca and P{'e}rez-Lezama, Claudia V and Gonz{'a}lez, Elda G Quiroga and Campos-Garc{'i}a, Gabriela A and Cacho, M{'o}nica Larre Bola{~n}os},
booktitle={2024 IEEE Global Engineering Education Conference (EDUCON)},
pages={1--7},
year={2024},
organization={IEEE}
}
@inproceedings{gonzalez2019toward,
title={Toward improving collaborative behaviour during competitive programming assignments},
author={Gonzalez-Escribano, Arturo and Lara-Mongil, Victor and Rodriguez-Gutiez, Eduardo and Torres, Yuri},
booktitle={2019 IEEE/ACM Workshop on Education for High-Performance Computing (EduHPC)},
pages={68--74},
year={2019},
organization={IEEE}
}
@article{moreno2018competitive,
title={Competitive programming and gamification as strategy to engage students in computer science courses},
author={Moreno, Julian and Pineda, Andres F},
journal={Revista Espacios},
volume={39},
number={35},
year={2018}
}
@inproceedings{garcia2014learning,
title={The learning curves of competitive programming},
author={Garcia, Jose R and Aguirre, Vanessa E},
booktitle={AIP Conference Proceedings},
volume={1618},
number={1},
pages={934--937},
year={2014},
organization={American Institute of Physics}
}
@misc{Kurtz_2013,
title={The World’s Smartest Programmers Compete: ACM ICPC},
url={https://cacm.acm.org/blogcacm/the-worlds-smartest-programmers-compete-acm-icpc/},
journal={Communications of the ACM},
author={Kurtz, Chas},
year={2013},
month={Jul}
}
@book{pelech2010comprehensive,
title={The comprehensive handbook of constructivist teaching: From theory to practice},
author={Pelech, James},
year={2010},
publisher={IAP}
}
@INPROCEEDINGS{picanco2018didactic,
author={de Picanço, Wollace Souza and de Lucena, Juliana M.V. Martinez and de Lira, Antônio Fonseca and de Lucena, Vicente Ferreira},
booktitle={2018 IEEE Frontiers in Education Conference (FIE)},
title={Didactic Framework for Teaching C Programming Language: A Proposal Based on Cooperative and Competitive Learning Techniques},
year={2018},
volume={},
number={},
pages={1-9},
keywords={Computer languages;Tools;Robots;Programming profession;Engineering students;Learning with Robots;C Programming Language Learning;Collaborative Learning;Competitive Learning},
doi={10.1109/FIE.2018.8658825}
}
@article{jara2012synchronous,
title={Synchronous collaboration of virtual and remote laboratories},
author={Jara, Carlos A and Candelas, Francisco A and Torres, Fernando and Dormido, Sebasti{'e}n and Esquembre, Francisco},
journal={Computer Applications in Engineering Education},
volume={20},
number={1},
pages={124--136},
year={2012},
publisher={Wiley Online Library}
}
@article{moreno2013trivialcv,
title={TrivialCV: Competitive activities for the classroom integrated in a moodle virtual campus},
author={Moreno-Ger, Pablo and Mart{'i}nez-Ortiz, Iv{'a}n and Gilmartin, Virginia Francisco and Ballesteros, Raquel Herv{'a}s},
journal={IEEE Revista Iberoamericana de Tecnologias del Aprendizaje},
volume={8},
number={1},
pages={31--38},
year={2013},
publisher={IEEE}
}
@Inbook{reeve2012SDT,
author="Reeve, Johnmarshall",
editor="Christenson, Sandra L.
and Reschly, Amy L.
and Wylie, Cathy",
title="A Self-determination Theory Perspective on Student Engagement",
bookTitle="Handbook of Research on Student Engagement",
year="2012",
publisher="Springer US",
address="Boston, MA",
pages="149--172",
abstract="This chapter pursues three goals. First, it overviews self-determination theory (SDT). SDT is a macrotheory of motivation comprised of five interrelated minitheories---basic needs theory, organismic integration theory, goal contents theory, cognitive evaluation theory, and causality orientations theory. Each minitheory was created to explain specific motivational phenomena and to address specific research questions. Second, the chapter uses the student-teacher dialectical framework within SDT to explain how classroom conditions sometimes support but other times neglect and frustrate students' motivation, engagement, and positive classroom functioning. Third, the chapter highlights student engagement. In doing so, it overviews recent classroom-based, longitudinally designed research to reveal three new and important functions of student engagement---namely, that student engagement fully mediates and explains the motivation-to-achievement relation, that changes in engagement produce changes in the learning environment, and that changes in engagement produce changes in motivation, as students' behavioral, emotional, cognitive, and agentic engagements represent actions taken not only to learn but also to meet psychological needs. The chapter concludes with implications for teachers and with suggestions for future research.",
isbn="978-1-4614-2018-7",
doi="10.1007/978-1-4614-2018-7_7",
url="https://doi.org/10.1007/978-1-4614-2018-7_7"
}
@inproceedings{wulf2005constructivist,
author = {Wulf, Tom},
title = {Constructivist approaches for teaching computer programming},
year = {2005},
isbn = {1595932526},
publisher = {Association for Computing Machinery},
address = {New York, NY, USA},
url = {https://doi.org/10.1145/1095714.1095771},
doi = {10.1145/1095714.1095771},
abstract = {This paper discusses the application of constructivist pedagogical approaches to teaching computer programming in high school and undergraduate courses. It provides an overview of constructivist pedagogy and contrasts it with traditional objectivist pedagogy. The paper then describes how a variety of sources on constructivist educational practice from computer science educational literature can be combined to transform under-graduate courses in computer programming and information technology. The resulting courses are accessible to a wider range of students and incorporate active learning, collaborative group-based instruction, and cognitive apprenticeships. This pedagogical style focuses on the higher cognitive levels of Bloom's taxonomy and addresses multiple student learning styles and intelligences. Course design, classroom activities, and assessment methods using this pedagogical approach for computer programming instruction will be covered in detail. The presentation relies heavily on the experiences of the presenter in adopting this style of instruction within undergraduate computer programming courses and will include a discussion of the some of the attitudinal issues they encountered in this process. The presentation will include coverage of resources for constructivist teaching of computer science and technology and some recommendations for instructors who may wish to adopt this style of instruction.},
booktitle = {Proceedings of the 6th Conference on Information Technology Education},
pages = {245–248},
numpages = {4},
keywords = {programming instruction, constructivist pedagogy, cognitive apprenticeships, active learning},
location = {Newark, NJ, USA},
series = {SIGITE '05}
}
@article{robins2003learning,
title={Learning and teaching programming: A review and discussion},
author={Robins, Anthony and Rountree, Janet and Rountree, Nathan},
journal={Computer science education},
volume={13},
number={2},
pages={137--172},
year={2003},
publisher={Taylor & Francis}
}
@article{doi:10.1177/1536867X0200200103,
author = {Roger Newson},
title ={Parameters behind “Nonparametric” Statistics: {Kendall}'s tau, {Somers}’ {D} and Median Differences},
journal = {The Stata Journal},
volume = {2},
number = {1},
pages = {45-64},
year = {2002},
doi = {10.1177/1536867X0200200103},
URL = {https://doi.org/10.1177/1536867X0200200103},
eprint = {https://doi.org/10.1177/1536867X0200200103},
abstract = { So-called “nonparametric” statistical methods are often in fact based on population parameters, which can be estimated (with confidence limits) using the corresponding sample statistics. This article reviews the uses of three such parameters, namely Kendall's τa, Somers’ D and the Hodges–Lehmann median difference. Confidence intervals for these are demonstrated using the somersd package. It is argued that confidence limits for these parameters, and their differences, are more informative than the traditional practice of reporting only p-values. These three parameters are also important in defining other tests and parameters, such as the Wilcoxon test, the area under the receiver operating characteristic (ROC) curve, Harrell's C, and the Theil median slope. }
}
@article{di2016oii,
title={oii-web: An interactive online programming contest training system},
author={Di Luigi, William and Farina, Gabriele and Laura, Luigi and Nanni, Umberto and Temperini, Marco and Versari, Luca and others},
journal={Olympiads in Informatics},
volume={10},
number={1},
pages={207--222},
year={2016},
publisher={Vilnius University}
}
@article{luo2025curriculum,
title={Curriculum Design of Competitive Programming: a Contest-based Approach},
author={Luo, Zhongtang},
journal={arXiv preprint arXiv:2504.00533},
year={2025}
}
@article{manzoor2008common,
title={Common mistakes in online and real-time contests},
author={Manzoor, Shahriar},
journal={XRDS: Crossroads, The ACM Magazine for Students},
volume={14},
number={4},
pages={10--16},
year={2008},
publisher={ACM New York, NY, USA}
}
@article{dickey2025removing,
title={Removing the Competition from Competitive Programming: A Novel Curriculum Design},
author={Dickey, Ethan},
journal={arXiv preprint arXiv:TBA},
year={2025}
}
@article{trotman2008programming,
title={Programming contest strategy},
author={Trotman, Andrew and Handley, Chris},
journal={Computers & Education},
volume={50},
number={3},
pages={821--837},
year={2008},
publisher={Elsevier},
doi = {https://doi.org/10.1016/j.compedu.2006.08.008},
url = {https://www.sciencedirect.com/science/article/pii/S0360131506001357},
keywords = {Programming contest},
abstract = {Each year the ACM hosts a truly international programming contest – the International Collegiate Programming Contest (ICPC). Dating back to a contest held by Texas A&M University in 1970, this annual event, along with the associated regional contests, has grown to 5606 teams from 1733 universities in 84 countries (in the year 2006). Despite the maturity of the event, and the number of competitors, there has never been a systematic examination of contest strategy. Herein several strategies are proposed and examined to determine whether a team can gain an advantage by choosing a good strategy; and, if so, then what that strategy should be. We show that a team can gain an advantage by choosing a good strategy, but that there is no one best strategy. A team must choose between winning by number of solved problems and winning by points. Finding the optimal strategy to win by problems is shown to be NP-complete, while to win by points a team must solve problems in order from easiest to hardest.}
}
@misc{wicklin2023weakorstrong,
title={Weak or strong? How to interpret a Spearman or Kendall correlation},
url={https://blogs.sas.com/content/iml/2023/04/05/interpret-spearman-kendall-corr.html},
journal={SAS Blogs},
author={Wicklin, Rick},
year={2023},
month={Apr}
}
@Misc{icpc-beijing-headquarter,
author = "{ICPC Beijing Headquarter}",
title = "{ICPC} Beijing Headquarter",
howpublished = {url{https://icpc.pku.edu.cn}},
note = {Accessed: May 5, 2025}
}
@Misc{sua,
author = "{SUA Competitive Programming Judges}",
title = "{SUA Competitive Programming Judges}",
howpublished = {url{https://sua.ac/about.html}},
note = {Accessed: May 5, 2025}
}
@misc{meza2025email,
journal={ICPC Regional Structure – Research Inquiry},
author={Meza, Federico},
year={2025},
month={Apr},
note={Private communication.}
}