LinkBERT: Improving Language Model Training with Document Link

Bridging Knowledge Gaps: LinkBERT Enhances Language Models with Inter-Document Understanding
New pretraining method leverages document links to improve reasoning and knowledge acqu

Bridging Knowledge Gaps: LinkBERT Enhances Language Models with Inter-Document Understanding

New pretraining method leverages document links to improve reasoning and knowledge acquisition in AI.

Modern artificial intelligence systems rely heavily on large language models (LMs) like BERT and GPT. These models, trained on vast amounts of text data, form the backbone of many everyday technologies, from search engines to virtual assistants. Their power stems from a self-supervised pretraining process that allows them to learn language patterns and world knowledge without requiring explicit human labeling. However, a significant limitation in current pretraining strategies is their tendency to process documents in isolation, potentially missing crucial knowledge that spans across related texts.

The Challenge of Isolated Document Training

Traditional language model pretraining often involves splitting a large text corpus into individual documents and drawing training examples from each document independently. This approach, while effective for learning from within a single text, can overlook the rich dependencies that exist between documents. The internet and academic literature, common sources for LM training data, are replete with connections like hyperlinks and citation links. These links are not merely navigational aids; they often serve to connect related concepts and provide layered information that is not present in any single document alone.

Consider, for example, a Wikipedia article about Washington D.C.’s Tidal Basin that mentions the National Cherry Blossom Festival. If a hyperlink leads to another article detailing the festival’s celebration of Japanese cherry trees, a user can connect these pieces of information to understand that the Tidal Basin hosts Japanese cherry trees. This multi-hop knowledge, acquired by traversing links, is vital for sophisticated tasks such as answering complex questions or discovering new information. Models trained solely on individual documents may fail to capture these cross-document relationships, limiting their ability to perform knowledge-intensive applications.

Introducing LinkBERT: A Graph-Based Approach

To address this limitation, researchers have developed LinkBERT, a novel pretraining method designed to incorporate document link information. This approach treats a text corpus not as a simple collection of documents, but as a graph where documents are nodes and links are edges. By explicitly modeling these connections, LinkBERT aims to train language models that can better understand and utilize knowledge distributed across multiple documents.

The LinkBERT methodology involves three core steps:

• Document Graph Construction: The first step is to identify and map the links between documents. Researchers primarily utilize hyperlinks and citation links, which are generally considered high-quality indicators of relevance and are widely available. Each document is represented as a node in a graph, with a directed edge connecting document A to document B if document A contains a hyperlink to document B.
• Link-Aware Input Creation: To enable the language model to learn dependencies between linked documents, segments from these documents are combined into single training instances. The model’s input sequences are constructed by concatenating pairs of text segments. Three strategies are employed for creating these pairs:
  • Contiguous Segments: Segments are taken from the same document, mimicking standard LM pretraining.
  • Random Segments: Segments are sampled from two entirely different, randomly chosen documents.
  • Linked Segments: One segment is sampled from a document, and the second segment is sampled from a document directly linked to the first.

  This variety of segment pairing is designed to provide the model with a training signal that helps it distinguish between different types of relationships between text segments.

• Link-Aware Pretraining Tasks: The combined text segments are then used to train the language model using two self-supervised tasks:
  • Masked Language Modeling (MLM): Similar to BERT, this involves masking certain tokens in the input and training the model to predict them using their surrounding context. When linked segments are present, MLM encourages the model to learn multi-hop knowledge by predicting masked words based on information from related documents. For instance, if segments about the Tidal Basin, its Cherry Blossom Festival, and Japanese cherry trees are presented together, the model can learn the association between these concepts.
  • Document Relation Prediction (DRP): This task trains the model to classify the relationship between two segments (e.g., contiguous, random, or linked). DRP explicitly teaches the model to recognize relevance and dependencies between documents, helping it to bridge conceptual gaps.

These pretraining tasks can be viewed as performing graph-based self-supervised learning, analogous to predicting node features using neighbors (MLM) and predicting the existence or type of edges between nodes (DRP).

Evaluating LinkBERT’s Performance

LinkBERT has been evaluated in both general and biomedical domains. In the general domain, it was pretrained on Wikipedia, incorporating hyperlinks. For the biomedical domain, a variant called BioLinkBERT was trained on PubMed, utilizing citation links. These LinkBERT models were then tested on various downstream natural language processing tasks, including general question answering (MRQA), benchmarks like GLUE, and biomedical NLP tasks such as BLURB, MedQA, and MMLU.

The results indicate that LinkBERT consistently outperforms baseline models like BERT and PubMedBERT, which were trained without explicit document link information. The improvements were particularly significant in the biomedical domain, likely due to the dense network of citation links in scientific literature, which BioLinkBERT effectively leverages. BioLinkBERT achieved new state-of-the-art results on several key biomedical benchmarks.

Key Strengths of LinkBERT

LinkBERT demonstrates several notable advantages:

• Enhanced Multi-Hop Reasoning: LinkBERT shows substantial improvements on tasks requiring multi-hop reasoning, such as those found in the HotpotQA and TriviaQA benchmarks. By training with linked documents, the model becomes better equipped to connect information across different texts to arrive at correct answers, overcoming the tendency of standard models to rely solely on information within a single document.
• Improved Document Relation Understanding: The Document Relation Prediction task in LinkBERT’s pretraining helps the model to better discern relationships between documents. This is particularly beneficial in open-domain question answering scenarios where models must sift through multiple retrieved documents, some of which may be irrelevant. LinkBERT proves more robust to distracting documents, maintaining accuracy where standard BERT might falter.
• Data Efficiency in Few-Shot Learning: When fine-tuned with limited training data (1% or 10% of available data), LinkBERT significantly outperforms BERT. This suggests that LinkBERT has acquired a richer and more generalized understanding of world knowledge during pretraining, making it more effective in low-resource settings.

Utilizing LinkBERT in Practice

LinkBERT is designed for easy integration into existing NLP pipelines. The pretrained models, including the general LinkBERT and the biomedical BioLinkBERT, are available on the HuggingFace platform. Developers can load these models as a direct replacement for models like BERT, enabling their applications to benefit from enhanced cross-document understanding and reasoning capabilities.

The development of LinkBERT represents a significant step forward in language model pretraining, highlighting the importance of inter-document connections for building more knowledgeable and capable AI systems. By incorporating the rich relational information present in linked documents, LinkBERT offers a promising path towards more sophisticated natural language understanding and knowledge discovery.

Key Takeaways

• Bridging Knowledge Gaps: LinkBERT Enhances Language Models with Inter-Document Understanding
• New pretraining method leverages document links to improve reasoning and knowledge acquisition in AI.
• Modern artificial intelligence systems rely heavily on large language models (LMs) like BERT and GPT. These models, trained on vast amounts of text data, form the backbone of many everyday technologies, from search engines to virtual assistants. Their power stems from a self-supervised pretraining process that allows them to learn language patterns and world knowledge without requiring explicit human labeling. However, a significant limitation in current pretraining strategies is their tendency to process documents in isolation, potentially missing crucial knowledge that spans across related texts.
• The Challenge of Isolated Document Training
• Traditional language model pretraining often involves splitting a large text corpus into individual documents and drawing training examples from each document independently. This approach, while effective for learning from within a single text, can overlook the rich dependencies that exist between documents. The internet and academic literature, common sources for LM training data, are replete with connections like hyperlinks and citation links. These links are not merely navigational aids; they often serve to connect related concepts and provide layered information that is not present in any single document alone.