Skip to main content

Research Repository

Advanced Search

An Exploration of Dropout with RNNs for Natural Language Inference

Gajbhiye, Amit; Jaf, Sardar; Al-Moubayed, Noura; McGough, A. Stephen; Bradley, Steven

An Exploration of Dropout with RNNs for Natural Language Inference Thumbnail


Authors

Amit Gajbhiye

Sardar Jaf

A. Stephen McGough



Contributors

V. Kurková
Editor

Yannis Manolopoulos
Editor

Barbara Hammer
Editor

Lazaros S. Iliadis
Editor

Ilias G. Maglogiannis
Editor

Abstract

Dropout is a crucial regularization technique for the Recurrent Neural Network (RNN) models of Natural Language Inference (NLI). However, dropout has not been evaluated for the effectiveness at different layers and dropout rates in NLI models. In this paper, we propose a novel RNN model for NLI and empirically evaluate the effect of applying dropout at different layers in the model. We also investigate the impact of varying dropout rates at these layers. Our empirical evaluation on a large (Stanford Natural Language Inference (SNLI)) and a small (SciTail) dataset suggest that dropout at each feed-forward connection severely affects the model accuracy at increasing dropout rates. We also show that regularizing the embedding layer is efficient for SNLI whereas regularizing the recurrent layer improves the accuracy for SciTail. Our model achieved an accuracy 86.14% on the SNLI dataset and 77.05% on SciTail.

Citation

Gajbhiye, A., Jaf, S., Al-Moubayed, N., McGough, A. S., & Bradley, S. (2018). An Exploration of Dropout with RNNs for Natural Language Inference. In V. Kurková, Y. Manolopoulos, B. Hammer, L. S. Iliadis, & I. G. Maglogiannis (Eds.), Artificial neural networks and machine learning - ICANN 2018 : 27th international Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, proceedings. Part III (157-167). https://doi.org/10.1007/978-3-030-01424-7_16

Conference Name ICANN 2018: 27th International Conference on Artificial Neural Networks
Conference Location Rhodes
Acceptance Date Jul 10, 2018
Online Publication Date Oct 1, 2018
Publication Date Oct 1, 2018
Deposit Date Aug 2, 2018
Publicly Available Date Aug 3, 2018
Publisher Springer Verlag
Pages 157-167
Series Title Lecture notes in computer science
Series Number 11141
Book Title Artificial neural networks and machine learning - ICANN 2018 : 27th international Conference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, proceedings. Part III.
ISBN 9783030014230
DOI https://doi.org/10.1007/978-3-030-01424-7_16
Public URL https://durham-repository.worktribe.com/output/1146132

Files





You might also like



Downloadable Citations