Winterbottom, T. and Xiao, S. and McLean, A. and Al Moubayed, N . (2020) 'On modality bias in the TVQA dataset.', The British Machine Vision Conference (BMVC) Manchester, England, 7-10 September 2020.
TVQA is a large scale video question answering (video-QA) dataset based on popular TV shows. The questions were specifically designed to require “both vision and language understanding to answer”. In this work, we demonstrate an inherent bias in the dataset towards the textual subtitle modality. We infer said bias both directly and indirectly, notably finding that models trained with subtitles learn, on-average, to suppress video feature contribution. Our results demonstrate that models trained on only the visual information can answer ∼45% of the questions, while using only the subtitles achieves ∼68%. We find that a bilinear pooling based joint representation of modalities damages model performance by 9% implying a reliance on modality specific information. We also show that TVQA fails to benefit from the RUBi modality bias reduction technique popularised in VQA. By simply improving text processing using BERT embeddings with the simple model first proposed for TVQA, we achieve state-of-the-art results (72.13%) compared to the highly complex STAGE model (70.50%). We recommend a multimodal evaluation framework that can highlight biases in models and isolate visual and textual reliant subsets of data. Using this framework we propose subsets of TVQA that respond exclusively to either or both modalities in order to facilitate multimodal modelling as TVQA originally intended.
|Item Type:||Conference item (Paper)|
|Full text:||(AM) Accepted Manuscript|
Download PDF (754Kb)
|Publisher Web site:||https://www.bmvc2020-conference.com/programme/accepted-papers/|
|Date accepted:||03 August 2020|
|Date deposited:||26 August 2020|
|Date of first online publication:||25 August 2020|
|Date first made open access:||01 December 2020|
Save or Share this output
|Look up in GoogleScholar|