LinkBERT: Enhancing Language Models with Document Link Information

Language Models (LMs) like BERT and the GPT series have become foundational to modern Natural Language Processing (NLP). Their remarkable performance stems from a pretraining phase using self-supervised learning on vast amounts of text data. This pretraining allows them to learn a rich understanding of language and world knowledge, which can then be adapted to various downstream tasks with minimal task-specific fine-tuning. Common pretraining strategies include masked language modeling (MLM), where the model predicts masked words, and causal language modeling, where it predicts the next word in a sequence.

The Challenge of Isolated Documents

A significant limitation of many existing LM pretraining methods is their tendency to process documents in isolation. This means that training instances are drawn from individual documents independently, disregarding the inherent relationships that often exist between them. In contexts like the web or scientific literature, documents are frequently interconnected through hyperlinks and citation links. These links are not merely navigational aids; they represent crucial pathways to knowledge that often spans across multiple documents. For example, an article about the “Tidal Basin, Washington D.C.” might mention the “National Cherry Blossom Festival,” and a hyperlinked article on the festival could reveal that it celebrates “Japanese cherry trees.” Together, these linked documents provide multi-hop knowledge, such as “Tidal Basin has Japanese cherry trees,” which is not fully present within either document alone.

Models that fail to leverage these inter-document dependencies may miss out on capturing crucial knowledge or facts distributed across a corpus. Learning this multi-hop knowledge during pretraining is vital for applications requiring deeper understanding, such as question answering and knowledge discovery. Recognizing that a text corpus can be viewed as a graph of interconnected documents, researchers have developed LinkBERT, a novel pretraining method designed to incorporate document link information for more knowledgeable language model training.

LinkBERT: An Approach to Link-Aware Pretraining

LinkBERT’s approach involves three core steps to imbue language models with a more comprehensive understanding of interconnected knowledge:

1. Document Graph Construction: The first step involves identifying and utilizing links between documents to construct a document graph. This graph represents documents as nodes and the links between them as directed edges. Hyperlinks and citation links are particularly valuable due to their high relevance and widespread availability.
2. Link-Aware Input Creation: To enable the LM to learn dependencies across linked documents, training instances are created by placing segments from linked documents together within the same input sequence. This is achieved by chunking documents into segments of approximately 256 tokens. Three options are used for creating these input pairs:
   • Contiguous Segments: Two adjacent segments from the same document.
   • Random Segments: Segments sampled from two entirely random documents.
   • Linked Segments: One segment sampled from a document, and the second from a document directly linked to it in the graph.

   These options create a diverse training signal, allowing the model to learn not only inter-document relationships but also to distinguish between linked, contiguous, and random document pairings.

3. Link-Aware LM Pretraining: The constructed input sequences are then used to train the language model using two self-supervised tasks:
   • Masked Language Modeling (MLM): This task involves masking tokens within the input and training the model to predict them using the surrounding context, including segments from linked documents. This encourages the model to learn multi-hop knowledge by associating concepts that appear together due to document links.
   • Document Relation Prediction (DRP): This task requires the model to classify the relationship between two input segments (contiguous, random, or linked). DRP trains the LM to understand document relevance and dependencies, facilitating the learning of bridging concepts.

   These two tasks are trained jointly. The pretraining objectives can be analogized to graph-based learning tasks: MLM corresponds to node feature prediction (predicting masked features using neighbors), and DRP corresponds to link prediction (predicting the existence or type of an edge between nodes).

Performance and Strengths of LinkBERT

LinkBERT has been evaluated on various downstream NLP tasks in both general and biomedical domains, demonstrating significant improvements over baseline models trained without document links. Notably, LinkBERT trained on Wikipedia (including hyperlinks) and PubMed (including citation links) showed consistent gains across tasks like question answering and general NLP benchmarks (GLUE, MRQA) and biomedical NLP benchmarks (BLURB, MedQA, MMLU). The gains in the biomedical domain were particularly substantial, highlighting the importance of citation links in scientific literature.

Key Strengths Identified:

• Effective Multi-Hop Reasoning: LinkBERT exhibits a notable ability to perform multi-hop reasoning. In tasks like HotpotQA, where answering a question requires synthesizing information from multiple documents, LinkBERT proved superior to BERT in correctly connecting disparate pieces of information, leading to more accurate answers. This is attributed to its pretraining on linked document segments, which helps the model learn to reason across multiple concepts and documents.
• Improved Document Relation Understanding: The inclusion of the Document Relation Prediction (DRP) task in LinkBERT’s pretraining enhances its ability to model relationships between documents. This proved beneficial in open-domain question answering scenarios where models need to sift through multiple retrieved documents, some of which may be irrelevant. LinkBERT demonstrated robustness to distracting documents, maintaining accuracy where BERT experienced performance degradation.
• Data-Efficient and Few-Shot Learning: LinkBERT showed significant improvements when fine-tuned with limited training data (1% or 10% of available data). This suggests that LinkBERT internalizes more world knowledge during pretraining, making it more effective in low-resource settings and for knowledge-intensive tasks.

Conclusion and Future Directions

LinkBERT represents a significant advancement in language model pretraining by effectively leveraging document link information. By placing linked documents together in training instances and employing joint MLM and DRP tasks, LinkBERT models learn to capture multi-hop knowledge and understand inter-document relationships. As a drop-in replacement for existing models like BERT, LinkBERT offers improved performance across a range of NLP tasks, particularly those requiring complex reasoning and knowledge integration. The availability of pretrained LinkBERT and BioLinkBERT models on HuggingFace makes them accessible for researchers and developers. Future work could explore extending this approach to other model architectures like GPT or sequence-to-sequence models for enhanced text generation and incorporating different types of links, such as source code dependencies, for specialized domains.

Pros and Cons of LinkBERT:

Pros:

• Improved performance on knowledge-intensive and reasoning-heavy tasks.
• Enhanced ability to capture multi-hop knowledge across documents.
• Greater robustness to irrelevant information in downstream tasks.
• Superior performance in few-shot and data-efficient learning scenarios.
• Leverages existing, high-quality document link structures (hyperlinks, citations).

Cons:

• Requires a corpus with identifiable and usable document links for effective pretraining.
• The process of constructing a comprehensive document graph can be computationally intensive.
• Performance gains might be domain-dependent, relying on the density and relevance of links within the corpus.

Key Takeaways:

• Document links (hyperlinks, citations) are valuable sources of knowledge that can be leveraged for language model pretraining.
• LinkBERT’s approach of creating link-aware input instances and using joint MLM and DRP tasks enables learning of multi-hop reasoning and document relationships.
• LinkBERT demonstrates significant improvements over traditional LMs, especially in tasks requiring cross-document understanding and knowledge synthesis.
• The method is particularly effective in domains with rich inter-document connectivity, such as scientific literature.
• LinkBERT offers a practical path to more knowledgeable and capable language models.