LinkBERT: Improving Language Model Training with Document Link

Bridging Information Gaps: LinkBERT's Novel Approach to Language Model Training
Leveraging Document Links to Enhance Knowledge Acquisition in AI

Language models (LMs) have become

Bridging Information Gaps: LinkBERT’s Novel Approach to Language Model Training

Leveraging Document Links to Enhance Knowledge Acquisition in AI

Language models (LMs) have become the backbone of modern natural language processing (NLP), powering tools from search engines to personal assistants. These AI systems, like Google’s search capabilities and Amazon’s Alexa, achieve their impressive performance through a process called pretraining. This self-supervised learning method allows models to absorb vast amounts of information from text data available online, enabling them to adapt to various tasks with minimal task-specific adjustments.

The Foundation of Modern AI Language Understanding

At their core, LMs like BERT and the GPT series learn by predicting missing words or the next word in a sequence. For instance, BERT might learn to fill in a blank: “My ____ is fetching the ball,” correctly predicting “dog.” Similarly, GPT models excel at predicting the next word: “My dog is fetching the ____,” anticipating “ball.” Through this process, LMs internalize world knowledge, understanding relationships between concepts like “dog,” “fetch,” and “ball” as they appear together in training texts. This encoded knowledge is crucial for applications requiring a deep understanding of language, such as answering complex questions.

The Challenge of Isolated Documents

A significant limitation in many current LM pretraining strategies is their tendency to process documents in isolation. Text corpora, whether from the general web or scientific literature, are often broken down into individual documents, with training instances drawn independently from each. This approach can overlook the rich interdependencies that frequently exist between documents. For example, a Wikipedia article about the Tidal Basin in Washington D.C. might mention the National Cherry Blossom Festival. A linked article could then elaborate on the festival’s connection to Japanese cherry trees. This hyperlink provides a crucial piece of multi-hop knowledge – that the Tidal Basin hosts Japanese cherry trees – which is not fully present within either document alone.

Models that fail to account for these cross-document connections may miss vital knowledge spanning multiple sources. Capturing this multi-hop knowledge is essential for tasks like question answering and knowledge discovery, enabling AI to answer questions such as “What trees can you see at the Tidal Basin?” Humans naturally navigate these connections using hyperlinks and citations to learn and discover. Recognizing this, a research team has developed LinkBERT, a new pretraining method designed to incorporate document link information, thereby enhancing the world knowledge LMs can acquire.

Introducing LinkBERT: A Graph-Based Pretraining Approach

LinkBERT’s methodology involves three core steps to build more knowledgeable language models:

1. Document Graph Construction: The first step is to identify and map the links between documents, creating a structured graph. This graph represents documents as nodes and links (such as hyperlinks and citation links) as edges. Hyperlinks and citation links are often chosen due to their high relevance and widespread availability.
2. Link-Aware Input Creation: To enable LMs to learn dependencies across documents, linked documents are strategically placed together within training instances. Each document is segmented into smaller pieces, and pairs of segments are concatenated. These pairs can be from the same document (contiguous), from different random documents (random), or from documents explicitly linked in the graph (linked). This variety in pairing helps the model learn to recognize different types of relationships between text segments.
3. Link-Aware Pretraining: The final step involves training the LM using these link-aware inputs with self-supervised tasks. Two key tasks are employed:
   • Masked Language Modeling (MLM): This task involves masking certain words in the input text and training the model to predict them using the surrounding context, including information from linked documents. This encourages the model to learn multi-hop knowledge by connecting concepts across documents that are presented together in the training data.
   • Document Relation Prediction (DRP): This task trains the model to classify the relationship between two text segments (e.g., contiguous, random, or linked). DRP helps the LM understand relevance and dependencies between documents, facilitating the bridging of concepts across different sources.

These pretraining tasks can be viewed as performing self-supervised learning on the document graph itself. MLM aligns with predicting node features using neighboring nodes, while DRP mirrors link prediction tasks, determining the type of connection between nodes.

Evaluating LinkBERT’s Performance Across Domains

LinkBERT has been pretrained and evaluated in two distinct domains: a general domain using Wikipedia and a biomedical domain using PubMed. In the general domain, LinkBERT leverages hyperlinks between Wikipedia articles, similar to how BERT uses its training corpus but with the added advantage of link information.

For the biomedical field, a specialized version called BioLinkBERT utilizes citation links between PubMed articles, enhancing models like PubMedBERT. The researchers evaluated these LinkBERT models on various downstream tasks, including general question answering (MRQA) and NLP benchmarks (GLUE), as well as biomedical NLP tasks (BLURB) and question answering (MedQA, MMLU).

Key Strengths and Improvements Demonstrated by LinkBERT

The evaluations revealed consistent improvements of LinkBERT over baseline models that did not incorporate document links. The gains were particularly pronounced in the biomedical domain, suggesting that the interdependencies captured by citation links are highly valuable. BioLinkBERT achieved new state-of-the-art results on the BLURB, MedQA, and MMLU benchmarks.

LinkBERT exhibits several notable strengths:

• Enhanced Multi-Hop Reasoning: In tasks requiring reasoning across multiple pieces of information, such as HotpotQA and TriviaQA, LinkBERT demonstrated significant improvements. For instance, when answering a question that required identifying an organization and its headquarters from two different documents, BERT sometimes defaulted to information within a single document. LinkBERT, however, was better able to connect information across the documents to arrive at the correct answer, such as identifying “Montreal” as the headquarters. This capability stems from its pretraining process, which brings related concepts and documents together.
• Improved Document Relation Understanding: LinkBERT shows a superior ability to model relationships between documents. In open-domain question answering, where models must sift through multiple retrieved documents, LinkBERT proved more robust to irrelevant or distracting documents. While BERT’s performance declined in the presence of such noise, LinkBERT maintained its accuracy, likely due to the Document Relation Prediction task equipping it to better recognize document relevance.
• Data Efficiency in Few-Shot Learning: When fine-tuned with limited training data (1% or 10% of available data), LinkBERT significantly outperformed BERT. This suggests that LinkBERT has acquired a richer and more generalized understanding of knowledge during pretraining, making it more effective in low-resource scenarios.

Integrating LinkBERT into Your Applications

LinkBERT is designed to be a straightforward replacement for existing BERT models. The pretrained LinkBERT and BioLinkBERT models are accessible via HuggingFace, a popular platform for NLP resources. Developers can easily load these models using standard libraries, allowing them to leverage LinkBERT’s enhanced capabilities in their own AI applications.

For example, to use the general LinkBERT model:

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained('michiyasunaga/LinkBERT-large')
model = AutoModel.from_pretrained('michiyasunaga/LinkBERT-large')
inputs = tokenizer("Hello, my dog is cute", return_tensors="pt")
outputs = model(**inputs)

And for the biomedical BioLinkBERT model:

from transformers import AutoTokenizer, AutoModel
tokenizer = AutoTokenizer.from_pretrained('michiyasunaga/BioLinkBERT-large')
model = AutoModel.from_pretrained('michiyasunaga/BioLinkBERT-large')
inputs = tokenizer("Sunitinib is a tyrosine kinase inhibitor", return_tensors="pt")
outputs = model(**inputs)

By incorporating document link information into the pretraining process, LinkBERT represents a significant step forward in developing AI systems that can more effectively understand and utilize knowledge distributed across multiple sources.

References

• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
• GPT-3: Language Models are Few-Shot Learners
• Language Models are Unsupervised Multitask Learners
• Search On Google: Introducing Naturally
• Alexa Language Understanding
• LinkBERT: Improving Language Model Training with Document Links
• HTML Living Standard
• Scopus: Scientific Literature Database
• LinkBERT: Improving Language Model Training with Document Links
• PubMed: A biomedical literature database
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• Reformer: The Efficient Transformer
• Longformer: The Long-Document Transformer
• PubMedBERT: a biomedical language representation model
• Open-Domain Question Answering with Knowledge Graphs

Key Takeaways

• Bridging Information Gaps: LinkBERT's Novel Approach to Language Model Training
• Leveraging Document Links to Enhance Knowledge Acquisition in AI
• Language models (LMs) have become the backbone of modern natural language processing (NLP), powering tools from search engines to personal assistants. These AI systems, like Google’s search capabilities and Amazon’s Alexa, achieve their impressive performance through a process called pretraining. This self-supervised learning method allows models to absorb vast amounts of information from text data available online, enabling them to adapt to various tasks with minimal task-specific adjustments.
• The Foundation of Modern AI Language Understanding
• At their core, LMs like BERT and the GPT series learn by predicting missing words or the next word in a sequence. For instance, BERT might learn to fill in a blank: "My ____ is fetching the ball," correctly predicting "dog." Similarly, GPT models excel at predicting the next word: "My dog is fetching the ____," anticipating "ball." Through this process, LMs internalize world knowledge, understanding relationships between concepts like "dog," "fetch," and "ball" as they appear together in training texts. This encoded knowledge is crucial for applications requiring a deep understanding of language, such as answering complex questions.