Incremental Reading Language Modelling

This repository contains the data and code needed to compute word surprisal values for the L1 and L2 stimuli used in the experiment. Ultimately these surprisal values will be used to predict reading times of L1 and L2 speakers of English.

Data

The language models are trained on preprocessed versions of the WikiText-2 dataset introduced by Merity et al.. The dataset can be downloaded from https://huggingface.co/datasets/wikitext.

Language Models

N-gram

KenLM

PCFG

Roark incremental parser

RNNG

Recurrent neural network grammars

LSTM

Transformer