LinkBERT: Improving Language Model Training with Document Link

Unlocking Deeper Knowledge: LinkBERT Enhances Language Models with Document Connections
New pretraining method leverages hyperlinks and citations to improve AI understandin

html

Unlocking Deeper Knowledge: LinkBERT Enhances Language Models with Document Connections

New pretraining method leverages hyperlinks and citations to improve AI understanding and reasoning.

Modern artificial intelligence systems, particularly in the realm of natural language processing (NLP), rely heavily on sophisticated language models (LMs). Models like BERT and the GPT series have become foundational, powering everyday tools such as search engines and virtual assistants. Their effectiveness stems from a pretraining phase where they learn from vast amounts of unlabelled text data, allowing them to be adapted to various tasks with minimal fine-tuning.

These models typically learn by predicting missing words within a document or anticipating the next word in a sequence. This process allows them to encode a significant amount of world knowledge, understanding associations between concepts that frequently appear together. For instance, an LM can learn the relationship between “dog,” “fetch,” and “ball” by encountering these terms in proximity within its training data.

The Limitations of Isolated Documents

However, a common limitation in current LM pretraining strategies is their tendency to process documents in isolation. While effective for capturing knowledge within a single text, this approach can overlook the rich interdependencies that exist between documents, especially in resources like the web or academic literature. These connections, often manifested as hyperlinks or citation links, serve as crucial pathways for knowledge that spans multiple sources.

Consider the example of Wikipedia articles: one article on the Tidal Basin in Washington D.C. might mention the “National Cherry Blossom Festival,” while a linked article details that this festival celebrates “Japanese cherry trees.” Individually, these documents offer limited information. However, the hyperlink facilitates a connection, enabling the acquisition of multi-hop knowledge—such as “Tidal Basin has Japanese cherry trees”—which is not explicitly stated within either document alone. Models trained without considering these links may struggle to grasp such interconnected knowledge, impacting their performance on tasks requiring complex reasoning or knowledge discovery, like answering questions that necessitate combining information from multiple sources.

Introducing LinkBERT: A Graph-Based Approach

Recognizing this gap, researchers have developed LinkBERT, a novel pretraining method designed to incorporate document link information. This approach treats a text corpus not as a mere collection of independent documents, but as a graph where documents are nodes and links are edges. By explicitly leveraging these connections during pretraining, LinkBERT aims to equip language models with a more comprehensive understanding of world knowledge.

The LinkBERT methodology involves three key stages:

• Document Graph Construction: The process begins by identifying and establishing links between related documents. Hyperlinks and citation links are prioritized due to their high relevance and widespread availability. Each document is represented as a node, and a directed edge is created from document A to document B if document A contains a link to document B.
• Link-Aware Input Creation: To enable the model to learn dependencies across documents, linked documents are strategically placed together within training instances. Each document is segmented into smaller units, and pairs of segments are concatenated. The method explores three strategies for pairing segments: contiguous segments from the same document (similar to existing models), random segments from different documents, and crucially, linked segments from documents connected by a link. This last option is key to exposing the model to cross-document relationships.
• Link-Aware Pretraining Tasks: The concatenated segments are then used to train the language model using specialized self-supervised tasks. These include:
  • Masked Language Modeling (MLM): Similar to standard BERT, some tokens are masked, and the model learns to predict them using surrounding context. When linked segments are present, this encourages learning multi-hop knowledge by utilizing information from connected documents.
  • Document Relation Prediction (DRP): This task trains the model to classify the relationship between two segments (e.g., contiguous, random, or linked). This explicitly teaches the model to recognize relevance and dependencies between documents, identifying bridging concepts.

These tasks can be conceptually understood as performing node feature prediction (MLM) and link prediction (DRP) on the document graph.

Evaluating LinkBERT’s Performance

LinkBERT has been evaluated in both general and biomedical domains. In the general domain, it was pretrained on Wikipedia, incorporating hyperlinks. In the biomedical domain, a version named BioLinkBERT was pretrained on PubMed, utilizing citation links.

The results show that LinkBERT consistently outperforms baseline models that do not utilize document links across a variety of downstream tasks, including general question answering (MRQA), broader NLP benchmarks (GLUE), and specialized biomedical NLP tasks (BLURB, MedQA, MMLU). The gains in the biomedical domain were particularly significant, suggesting that the dense network of citation links in scientific literature provides substantial benefits when incorporated into the pretraining process. BioLinkBERT, in particular, achieved state-of-the-art results on several key biomedical benchmarks.

Key Strengths of LinkBERT

LinkBERT demonstrates several notable advantages:

• Enhanced Multi-Hop Reasoning: The model shows marked improvement on tasks requiring multi-hop reasoning, such as those found in question-answering datasets like HotpotQA and TriviaQA. By training with linked documents, LinkBERT can better connect information across multiple sources to arrive at correct answers, unlike models that might default to information within a single document.
• Improved Document Relation Understanding: LinkBERT exhibits a stronger ability to discern relationships between documents. In open-domain question answering scenarios, where multiple retrieved documents may contain irrelevant information, LinkBERT proves more robust. It maintains accuracy even when presented with distracting documents, a capability attributed to the Document Relation Prediction task during pretraining.
• Data Efficiency and Few-Shot Learning: When fine-tuned with limited training data (1% or 10% of available data), LinkBERT significantly outperforms standard BERT. This suggests that LinkBERT has acquired a richer and more generalized understanding of knowledge during pretraining, making it more effective in low-resource settings.

Utilizing LinkBERT

LinkBERT is designed for straightforward integration into existing NLP pipelines. Pretrained LinkBERT and BioLinkBERT models are publicly available on the HuggingFace platform, allowing developers to use them as direct replacements for models like BERT in their applications. This accessibility enables researchers and practitioners to leverage its enhanced capabilities for a wide range of language understanding and generation tasks.

References

• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• Language Models are Unsupervised Multitask Learners (GPT-2)
• Language Understanding with BERT
• How Google Search Works
• Amazon Alexa
• What Makes BERT Good for Question Answering?
• HTML Standard
• Elsevier
• LinkBERT: Improving Language Model Training with Document Links (ACL 2022)
• Few-Shot Learning for Natural Language Processing
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Knowledge Graph Enhanced Pre-trained Language Model
• PubMed
• PubMedBERT: a biomedical language representation model for biomedical text mining
• Open-Domain Question Answering with DocRED

Key Takeaways

• Unlocking Deeper Knowledge: LinkBERT Enhances Language Models with Document Connections
• New pretraining method leverages hyperlinks and citations to improve AI understanding and reasoning.
• Modern artificial intelligence systems, particularly in the realm of natural language processing (NLP), rely heavily on sophisticated language models (LMs). Models like BERT and the GPT series have become foundational, powering everyday tools such as search engines and virtual assistants. Their effectiveness stems from a pretraining phase where they learn from vast amounts of unlabelled text data, allowing them to be adapted to various tasks with minimal fine-tuning.
• These models typically learn by predicting missing words within a document or anticipating the next word in a sequence. This process allows them to encode a significant amount of world knowledge, understanding associations between concepts that frequently appear together. For instance, an LM can learn the relationship between "dog," "fetch," and "ball" by encountering these terms in proximity within its training data.
• However, a common limitation in current LM pretraining strategies is their tendency to process documents in isolation. While effective for capturing knowledge within a single text, this approach can overlook the rich interdependencies that exist between documents, especially in resources like the web or academic literature. These connections, often manifested as hyperlinks or citation links, serve as crucial pathways for knowledge that spans multiple sources.