LinkBERT: Enhancing Language Models with Document Link Information

Language models (LMs) such as BERT and the GPT series have revolutionized natural language processing (NLP), forming the backbone of many contemporary NLP applications. Their power stems from a pretraining phase that utilizes self-supervised learning on vast amounts of text data from the web, eliminating the need for labeled datasets. Following this pretraining, these models can be efficiently adapted to a wide array of new tasks with minimal task-specific fine-tuning.

The Foundation of Modern NLP: Pretrained Language Models

The pretraining process for LMs typically involves tasks like masked language modeling (MLM), where the model learns to predict masked words within a sentence, or causal language modeling (CLM), where it predicts the next word in a sequence. Through these methods, LMs acquire a rich understanding of language, encoding world knowledge and associations between concepts that are crucial for downstream applications like question answering and text generation.

A Key Limitation: Independent Document Processing

A significant challenge in conventional LM pretraining is the common practice of processing each document in isolation. This approach, while effective for learning within a single document, overlooks the inherent dependencies and connections that exist between documents, particularly in data sourced from the web or scientific literature. These connections, often manifested as hyperlinks and citation links, are vital because knowledge frequently spans across multiple documents.

For instance, a Wikipedia article about the “Tidal Basin, Washington D.C.” might mention the “National Cherry Blossom Festival.” By following a hyperlink to the festival’s article, one can learn that it celebrates “Japanese cherry trees.” This interconnectedness allows for the acquisition of multi-hop knowledge, such as “Tidal Basin has Japanese cherry trees,” which is not discernible from either document in isolation. Models trained without considering these links may fail to capture such distributed knowledge, impacting their performance on knowledge-intensive tasks.

Introducing LinkBERT: Leveraging Document Links

To address this limitation, researchers have developed LinkBERT, a novel pretraining method designed to incorporate document link information and enhance LMs with richer world knowledge. LinkBERT operates through a three-step process:

1. Document Graph Construction: The first step involves identifying and utilizing links between documents to construct a document graph. Hyperlinks and citation links are particularly valuable due to their high relevance and ubiquity. Each document is treated as a node, and a directed edge is added between document A and document B if there is a link from A to B.
2. Link-Aware Input Creation: With the document graph established, LinkBERT creates input sequences that are sensitive to these links. To facilitate learning of multi-document dependencies, linked documents are placed together within the same input instance. Each document is segmented into chunks of approximately 256 tokens. Then, pairs of segments are concatenated as input sequences for the LM. Three options for segment pairing are employed:
• Contiguous Segments: Two adjacent segments from the same document, similar to existing LMs.
• Random Segments: One segment from a random document and another from a different random document.
• Linked Segments: One segment from a random document and another from a document directly linked to it in the graph.

These varied pairing strategies help train LinkBERT to recognize relationships between text segments.

3. Link-Aware LM Pretraining: The final step involves pretraining the LM using link-aware self-supervised tasks. Two primary tasks are utilized:
   • Masked Language Modeling (MLM): This task involves masking tokens in the input text and training the LM to predict them using surrounding tokens, including those from linked documents. This encourages the model to learn multi-hop knowledge by bringing related concepts from different documents into the same context.
   • Document Relation Prediction (DRP): This task trains the model to classify the relationship between two segments (contiguous, random, or linked). DRP helps the LM learn the relevance and dependencies between documents, as well as bridging concepts.

   These two tasks are trained jointly. Conceptually, these tasks align with graph-based self-supervised learning: node feature prediction (MLM) and link prediction (DRP).

Performance and Strengths of LinkBERT

LinkBERT has been evaluated on various downstream NLP tasks in both general and biomedical domains, demonstrating significant improvements over baseline models that do not leverage document links. Notably, it achieves state-of-the-art performance on several biomedical benchmarks when fine-tuned on tasks like BLURB, MedQA, and MMLU.

Key Strengths Identified:

• Effective Multi-Hop Reasoning: LinkBERT excels at tasks requiring multi-hop reasoning, such as answering questions that necessitate information from multiple interconnected documents. For example, it can correctly infer answers by connecting information across documents, unlike standard BERT which might default to information within a single document.
• Improved Document Relation Understanding: The DRP task in LinkBERT’s pretraining enhances its ability to understand relationships between documents. This makes it more robust to irrelevant or distracting documents in open-domain question answering scenarios, maintaining accuracy where BERT might falter.
• Data-Efficient and Few-Shot Learning: LinkBERT shows superior performance in low-resource settings, achieving substantial gains when fine-tuned with only 1% or 10% of the training data. This indicates that it internalizes more world knowledge during pretraining, making it more effective with limited data.

Utilizing LinkBERT

LinkBERT can be readily integrated into existing NLP pipelines as a direct replacement for BERT. Pretrained LinkBERT models are available on platforms like HuggingFace, allowing easy loading and use. Researchers and developers can leverage LinkBERT for various applications, including question answering and text classification, by either using provided fine-tuning scripts or adapting their existing BERT-based workflows.

Summary and Future Directions

LinkBERT represents a significant advancement in language model pretraining by incorporating document link information. By jointly training with MLM and DRP on linked documents, LinkBERT models acquire a deeper understanding of world knowledge and inter-document relationships. This leads to improved performance on tasks requiring multi-hop reasoning, document relation understanding, and efficient learning in data-scarce environments.

The availability of LinkBERT models opens doors for more knowledgeable and capable NLP systems. Future research directions include extending this approach to other LM architectures like GPT and sequence-to-sequence models for document link-aware text generation, and exploring the incorporation of other forms of relational links, such as source code dependencies, for specialized language models.

Pros and Cons

Pros:

• Significantly improves performance on tasks requiring multi-hop reasoning.
• Enhances understanding of relationships between multiple documents.
• More effective in data-efficient and few-shot learning scenarios.
• Robust to irrelevant documents in open-domain QA.
• Leverages readily available document link structures (hyperlinks, citations).
• Can be easily integrated as a drop-in replacement for BERT.

Cons:

• Requires a corpus with identifiable document links for effective pretraining.
• The process of constructing the document graph can be computationally intensive for very large corpora.
• Performance gains might vary depending on the density and quality of links within the pretraining corpus.

Key Takeaways

• Language models can benefit from pretraining that considers relationships between documents, not just content within single documents.
• Document links (like hyperlinks and citations) provide a valuable signal for acquiring multi-hop knowledge.
• LinkBERT, through its novel pretraining tasks (MLM and DRP) and input construction methods, effectively leverages these links.
• LinkBERT demonstrates superior capabilities in multi-hop reasoning, document relation understanding, and few-shot learning compared to models trained without link information.
• The LinkBERT models are publicly available, facilitating their adoption in various NLP applications.