We use cookies to ensure that we give you the best experience on our website. By continuing to browse this repository, you give consent for essential cookies to be used. You can read more about our Privacy and Cookie Policy.

Durham Research Online
You are in:

Distillation of human–object interaction contexts for action recognition

Almushyti, Muna and Li, Frederick W. B. (2022) 'Distillation of human–object interaction contexts for action recognition.', Computer animation and virtual worlds., 33 (5). e2107.


Modeling spatial-temporal relations is imperative for recognizing human actions, especially when a human is interacting with objects, while multiple objects appear around the human differently over time. Most existing action recognition models focus on learning overall visual cues of a scene but disregard a holistic view of human–object relationships and interactions, that is, how a human interacts with respect to short-term task for completion and long-term goal. We therefore argue to improve human action recognition by exploiting both the local and global contexts of human–object interactions (HOIs). In this paper, we propose the Global-Local Interaction Distillation Network (GLIDN), learning human and object interactions through space and time via knowledge distillation for holistic HOI understanding. GLIDN encodes humans and objects into graph nodes and learns local and global relations via graph attention network. The local context graphs learn the relation between humans and objects at a frame level by capturing their co-occurrence at a specific time step. The global relation graph is constructed based on the video-level of human and object interactions, identifying their long-term relations throughout a video sequence. We also investigate how knowledge from these graphs can be distilled to their counterparts for improving HOI recognition. Finally, we evaluate our model by conducting comprehensive experiments on two datasets including Charades and CAD-120. Our method outperforms the baselines and counterpart approaches.

Item Type:Article
Full text:(VoR) Version of Record
Available under License - Creative Commons Attribution Non-commercial No Derivatives 4.0.
Download PDF
Publisher Web site:
Publisher statement:© 2022 The Authors. Computer Animation and Virtual Worlds published by John Wiley & Sons Ltd. This is an open access article under the terms of the Creative Commons Attribution-NonCommercial-NoDerivs License, which permits use and distribution in any medium, provided the original work is properly cited, the use is non-commercial and no modifications or adaptations are made.
Date accepted:03 July 2022
Date deposited:12 October 2022
Date of first online publication:10 August 2022
Date first made open access:12 October 2022

Save or Share this output

Look up in GoogleScholar