We use cookies to ensure that we give you the best experience on our website. By continuing to browse this repository, you give consent for essential cookies to be used. You can read more about our Privacy and Cookie Policy.

Durham Research Online
You are in:

On modality bias in the TVQA dataset.

Winterbottom, T. and Xiao, S. and McLean, A. and Al Moubayed, N . (2020) 'On modality bias in the TVQA dataset.', The British Machine Vision Conference (BMVC) Manchester, England, 7-10 September 2020.


TVQA is a large scale video question answering (video-QA) dataset based on popular TV shows. The questions were specifically designed to require “both vision and language understanding to answer”. In this work, we demonstrate an inherent bias in the dataset towards the textual subtitle modality. We infer said bias both directly and indirectly, notably finding that models trained with subtitles learn, on-average, to suppress video feature contribution. Our results demonstrate that models trained on only the visual information can answer ∼45% of the questions, while using only the subtitles achieves ∼68%. We find that a bilinear pooling based joint representation of modalities damages model performance by 9% implying a reliance on modality specific information. We also show that TVQA fails to benefit from the RUBi modality bias reduction technique popularised in VQA. By simply improving text processing using BERT embeddings with the simple model first proposed for TVQA, we achieve state-of-the-art results (72.13%) compared to the highly complex STAGE model (70.50%). We recommend a multimodal evaluation framework that can highlight biases in models and isolate visual and textual reliant subsets of data. Using this framework we propose subsets of TVQA that respond exclusively to either or both modalities in order to facilitate multimodal modelling as TVQA originally intended.

Item Type:Conference item (Paper)
Full text:(AM) Accepted Manuscript
Download PDF
Publisher Web site:
Date accepted:03 August 2020
Date deposited:26 August 2020
Date of first online publication:25 August 2020
Date first made open access:01 December 2020

Save or Share this output

Look up in GoogleScholar