We use cookies to ensure that we give you the best experience on our website. By continuing to browse this repository, you give consent for essential cookies to be used. You can read more about our Privacy and Cookie Policy.

Durham Research Online
You are in:

An exploration of dropout with RNNs for natural language inference.

Gajbhiye, Amit and Jaf, Sardar and Al-Moubayed, Noura and McGough, A. Stephen and Bradley, Steven (2018) 'An exploration of dropout with RNNs for natural language inference.', in Artificial neural networks and machine learning - ICANN 2018 : 27th international Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, proceedings. Part III. Cham: Springer, pp. 157-167. Lecture notes in computer science. (11141).


Dropout is a crucial regularization technique for the Recurrent Neural Network (RNN) models of Natural Language Inference (NLI). However, dropout has not been evaluated for the effectiveness at different layers and dropout rates in NLI models. In this paper, we propose a novel RNN model for NLI and empirically evaluate the effect of applying dropout at different layers in the model. We also investigate the impact of varying dropout rates at these layers. Our empirical evaluation on a large (Stanford Natural Language Inference (SNLI)) and a small (SciTail) dataset suggest that dropout at each feed-forward connection severely affects the model accuracy at increasing dropout rates. We also show that regularizing the embedding layer is efficient for SNLI whereas regularizing the recurrent layer improves the accuracy for SciTail. Our model achieved an accuracy 86.14% on the SNLI dataset and 77.05% on SciTail.

Item Type:Book chapter
Full text:(AM) Accepted Manuscript
Download PDF
Publisher Web site:
Publisher statement:The final publication is available at Springer via
Date accepted:10 July 2018
Date deposited:03 August 2018
Date of first online publication:01 October 2018
Date first made open access:No date available

Save or Share this output

Look up in GoogleScholar