NLP SUMMARIZATION: Use BERT and BART to summarize your favourite newspaper articles

NLP summarization has many applications, such as in news aggregation, document summarization, and social media analysis. It can help people quickly understand the main points of a long document or article or assist in the automation of tasks such as summarizing customer feedback or summarizing legal documents for legal practitioners.

Photo by Nathan Dumlao on Unsplash

What is NLP summarization?

NLP summarization (Text-Extraction) refers to the process of generating a concise and coherent summary of a longer piece of text, using natural language processing techniques. The goal of NLP summarization is to extract the most important information from the original text and present it in a condensed form that is easy to understand and digest.

There are two main approaches to NLP summarization: extractive summarization and abstractive summarization.

• Extractive summarization involves identifying the most important sentences or phrases in the original text and extracting them to create a summary. This approach relies on statistical and machine learning techniques to identify the most relevant information in the text.
• Abstractive summarization, on the other hand, involves generating new sentences that capture the essence of the original text. This approach requires a deep understanding of the language and context of the original text, and often involves the use of advanced NLP techniques such as natural language generation.

When to use what?

In general, extractive summarization is better suited for tasks where accuracy and legal implications are important, while abstractive summarization is better suited for tasks where generating a concise and coherent summary is more important than retaining the original wording of the text. However, both approaches have their own strengths and weaknesses, and the choice between them depends on the specific use case and requirements.

Code Example

Let’s summarize the first section of the following article:

An Indian startup could revolutionize ocean farming with its 'sea combine harvester' | CNN Business

# Input - first section of article
text = '''Often used to wrap sushi and flavor soups, seaweed has much greater potential – both as a food and for use in a wide range of products from cosmetics and textiles to biodegradable packaging and even biofuel.
Usually, seaweed is grown on ropes or nets suspended in the ocean, but current techniques make large-scale cultivation near impossible. Ocean farming is in the “stone ages,” according to Shrikumar Suryanarayan, co-founder and CEO of Bangalore-based Sea6 Energy and former head of research and development at Biocon, an Indian pharmaceutical company specializing in biologically-sourced medicines. “It’s like using a trowel and a pick to farm land.”
Founded in 2010, Sea6 Energy wants to mechanize ocean farming, just as tractors did for agriculture, with its “Sea Combine,” an automated catamaran that simultaneously harvests and replants seaweed in the ocean.
The machine travels back and forth between lines of seaweed, harvesting the fully-grown plants and replacing them with freshly-seeded lines.
A prototype is currently deployed at the company’s seaweed farm off the coast of Indonesia. The Southeast Asian nation has a tradition of seaweed farming that involves villagers tying pieces of seaweed to ropes and hauling them out to sea, before manually harvesting the lines, and there is a strong appetite for the crop there, according to Suryanarayan. As the technology develops and the market widens, the company intends to deploy more Sea Combines, including in its home country, India.'''

Extractive summarization with BERT

This code uses the SBert library to encode the input sentences into embeddings, and then calculates the cosine similarity between each pair of embeddings to create a similarity matrix. The sentence scores are then calculated by summing the cosine similarity scores for each sentence, and the top 3 sentences with the highest scores are selected to create the summary. Finally, the summary sentences are concatenated to create the summary text.

import numpy as np
from sentence_transformers import SentenceTransformer

# Load the pre-trained SBERT model
model = SentenceTransformer('all-MiniLM-L6-v2')

# Split the text into sentences
sentences = text.split('. ')

# Encode the sentences using the SBERT model
sentence_embeddings = model.encode(sentences)

# Calculate the cosine similarity between each sentence embedding
similarity_matrix = np.zeros([len(sentences), len(sentences)])
for i in range(len(sentences)):
    for j in range(len(sentences)):
        if i != j:
            similarity_matrix[i][j] = util.cos_sim(sentence_embeddings[i], sentence_embeddings[j])
print(similarity_matrix)
sentence_scores = np.sum(similarity_matrix, axis=1)

# Select the top 3 sentences with the highest scores
summary_sentences = []
for i in range(3):
    index = np.argmax(sentence_scores)
    summary_sentences.append(sentences[index].strip())
    sentence_scores[index] = -1

# Concatenate the summary sentences to create a summary
summary = '. '.join(summary_sentences)
print(summary)

#Output:
# "It's like using a trowel and a pick to farm land."
# Founded in 2010, Sea6 Energy wants to mechanize ocean farming, just as tractors did for agriculture, with its "Sea Combine," an automated catamaran that simultaneously harvests and replants seaweed in the ocean.
# The machine travels back and forth between lines of seaweed, harvesting the fully-grown plants and replacing them with freshly-seeded lines.
# A prototype is currently deployed at the company's seaweed farm off the coast of Indonesia. The Southeast Asian nation has a tradition of seaweed farming that involves villagers tying pieces of seaweed to ropes and hauling them out to sea, before manually harvesting the lines, and there is a strong appetite for the crop there, according to Suryanarayan. Often used to wrap sushi and flavor soups, seaweed has much greater potential – both as a food and for use in a wide range of products from cosmetics and textiles to biodegradable packaging and even biofuel.
# Usually, seaweed is grown on ropes or nets suspended in the ocean, but current techniques make large-scale cultivation near impossible

Abstractive summarization with Huggingface

In this example, we use the Hugging Face Transformers library to load the summarization pipeline. This pipeline uses a pre-trained transformer model to generate abstractive summaries from input text. To do that, we can use the BART (large-sized model), fine-tuned on CNN Daily Mail powered by Meta (Facebook). BART is a denoising autoencoder for pretraining sequence-to-sequence models and can be seen as generalizing of BERT and GPT.

Advice: Overall, using a pretrained language model for abstractive summarization can be a practical and effective approach for many use cases. However, it is important to choose a model that has been trained on a similar type of text and to fine-tune it on your specific task to ensure the best possible performance.

# install Transformers: pip install transformers
from transformers import pipeline

summarizer = pipeline("summarization", model="facebook/bart-large-cnn")

ARTICLE = """ """
print(summarizer(text, max_length=130, min_length=30, do_sample=False))

#Output:
#Seaweed has great potential as a food and for use in a wide range of products. 
#Usually, seaweed is grown on ropes or nets suspended in the ocean. Sea6 Energy
#wants to mechanize ocean farming, just as tractors did for agriculture.

If you intressted in reading more details about BART I recommend to read the paper: BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension

What you have learned?

You learned the differences between extractive and abstractive summarization. You learned when do use what. You have seen a code example for both types that helps you quickly to start on your own project.

Don’t Forget to clap — if you find this helpful:

Leave a comment if you have any questions, recommendations or something is not clear and I’ll try to answer soon as possible.

Like — Share — Commet — Follow

Recommendation for you: If you like this, read my other article on NLP Similarity, where I explain word embeddings and their importance for semantic similarity search.

NLP SIMILARITY: Use pretrained word embeddings for semantic similarity search with BERT