Parse2Vec: Transformer Embeddings Deeper than Surface Forms

<jats:p>Objective: The minimum transformer performance can move forward by incorporating the progress of the last decade. Background: Current efforts begin with surface forms and build meanings for tagging, parsing, identifying semantic relationships, etc. But a parser and outside resources can also provide these morphological, syntactic, and semantic features. Method: Instead of inputting tokens of surface forms into a transformer, we could start with feature vectors whose summed embeddings represent the deep features of the words. This is an empirical study on how inputs with feature vectors perform on masked word prediction.</jats:p>