We use cookies to ensure that we give you the best experience on our website. By continuing to browse this repository, you give consent for essential cookies to be used. You can read more about our Privacy and Cookie Policy.

Durham Research Online
You are in:

Towards a Human-AI hybrid system for categorising programming problems

Pereira, Filipe Dwan and Piris, Francisco and Cristo da Fonseca, Samuel and Cristea, Alexandra and Oliveira, Elaine H. T. and Carvalho, Leandro and Fernandes, David (2021) 'Towards a Human-AI hybrid system for categorising programming problems.', in SIGCSE '21: Proceedings of the 52nd ACM Technical Symposium on Computer Science Education. , pp. 94-100.


As programming skills are increasingly required world-wide and across disciplines, many students use online platforms that provide automatic feedback through a Programming Online Judge (POJ) mechanism. POJs are very popular e-learning tools, boasting large collections of programming problems. Despite their many benefits, students often struggle when solving problems not compatible with their prior knowledge. One important cause of this is that usually statements of problems are not classified according to programming topics (paradigms, data structures, etc.) and, hence, students waste time and effort in trying to solve exercises that are not tailored to their level and needs. Thus, to support students, we propose a new, "front-heavy" pipeline method to predict topics of POJ problems, using Bidirectional Encoder Representations from Transformers (BERT) for contextual text augmentation over the problem statements and further allowing for (lighter-weight) classical machine learning for classification. Our model outperformed all current state-of-the art, with an F1-score of 86% using stratified 10 fold cross-validation in a classically challenging multi-classification problem with seven categories. As a proof of concept, we conducted an experiment to show how our predictive model can be used as a human-AI hybrid complement for POJ, where learners would use AI-based recommendations to find the most appropriate problems.

Item Type:Book chapter
Full text:(VoR) Version of Record
Download PDF
Publisher Web site:
Date accepted:No date available
Date deposited:01 November 2021
Date of first online publication:03 March 2021
Date first made open access:01 November 2021

Save or Share this output

Look up in GoogleScholar