BERT with context information encoding from knowledge graphs

Abstract

We introduce a method to extend the vocabulary encoding of BERT with context encoding containing rich information of an input token, in a given sequence of text. The context encoding is output by another BERT model, named CTX-BERT, dedicated to infer relations to entities of the specified token in its context. To simplify the model, we combine three objectives: entity detection, entity encoding, and relation recovery into one by requiring CTX-BERT to recover the relation triples as a textual sequence when given a context sequence with the target token masked. Experimental results demonstrate that CTX-BERT could enhance the performance of the second BERT on question answering tasks.