New Deep Contextualized Word Vectors Advance Natural Language Processing
Researchers introduce deep contextualized word representations that capture complex syntax and semantics using a bidirectional language model. These new vectors significantly improve performance across six major NLP tasks.
Researchers introduce a new type of deep contextualized word representation that models complex characteristics of word use, such as syntax and semantics. Unlike traditional static word vectors, these representations capture how the meaning of a word varies across different linguistic contexts to effectively handle polysemy.
The system learns these dynamic word vectors from the internal states of a deep bidirectional language model that is pre-trained on a massive text corpus. Developers can easily integrate these pre-trained representations into existing neural networks without requiring complex architectural changes.
Testing shows that adding these contextualized vectors significantly improves the state of the art across six challenging natural language processing problems, including question answering, textual entailment, and sentiment analysis. The researchers also find that exposing the deep internal layers of the pre-trained network is crucial, as it allows downstream models to mix different types of semi-supervision signals for better accuracy.