StorytimeLM

In this PhD project, we focus on improving the effectiveness and efficiency of LMs with narrative data. The majority of contemporary models is immense in terms of parameters and training data size, which is in line with the evidence that the capabilities of LMs scale with size. However, the research community have raised some concerns regarding these models; they have a huge energy consumption, and it is impossible to check the ever-growing datasets. To overcome these concerns, in the past years, the NLP community has worked towards smaller models without a loss of quality. The current project approaches this goal by focusing on pre-training LMs on narrative training data. We believe that narrative data has the potential to improve the efficiency of LMs due to its ‘self-grounded’ nature.