Text Summarization – Part 2

In part 1 we saw how extractive summarization was performed. In part 2 we will now look at the more esoteric solution, abstractive summarization. While extractive summarization can be thought of as a classification task(does this sentence belong to summary or not), Abstractive summarization is much harder. It involves natural language generation. Lets see what approaches are there and what’s the state of the art.

Abstractive:

 In an abstractive approach, the process understands semantics and suggests words accordingly. Unlike the extractive approach, these words could be novel as well. It restructures the whole sentence with new words while keeping the original information intact.  In this approach, new words can come into the summary, enabling the system to produce more fluent summaries.

Figure 2: Abstractive Approach

Abstractive summarization is an efficient form of summarization compared to the extractive summarization as Abstractive Summarization retrieves information from multiple documents to create a precise summary of information. This approach has gained its popularity due to the ability of developing new sentences to tell important information from text documents. This summarizer displays the summarized information in a sequential form that is easily readable and grammatically correct. Reading ability or linguistic quality is an important catalyst for improving the quality of a summary. 

We can divide abstractive summarization into 2 types

  1. Structure-based Approach (using prior knowledge): The base of structure-based techniques is to utilize previous data and psychological feature schemas, like templates, extraction rules moreover as versatile different structures like trees, ontologies, lead and body, graphs, to inscribe the foremost very important knowledge.
  • Tree-based method: In this approach, multiple solutions are taken into consideration. This caters to the problem of incorporating the different issues into a single algorithm. For example, in this technique document representing text and theme intersection are handled at a time. A language generator or a degree algorithm-based system is used to generate the required output.
  • Ontology-based method: Creating a knowledge-based solution and summarizing based on past knowledge, comes under the Ontology-based method. Most of the documents online have a common domain which helps to retrieve its information, which is an Ontology-based solution.
  • Lead and body phrase method: As the name suggests this method based on insertion and substitution. These methods have identical syntactic head chunk under the lead and body sentences, to rewrite the lead sentence.
  • Rule-based method: In this technique, based on classes, documents are summarized. Further, the choice module, based on data extraction selects the most suited candidate and uses generation patterns for outline sentences.
  • Graph-based method: Specially Opinosis-Graph is used by several developers to represent language text. In this system, every node represents a word and a unit represents the structure of sentences for directed edges.    

2. Semantic-based Approach (using NLP Generation): In the semantic-based technique, the natural language generation (NLG) system is feed by linguistics demonstration of the document. By processing linguistic data, this technique identifies noun phrases and verb phrases by processing:

  • Multimodal semantic model: In this system, contents like text and images that are used for multimodal documents which are rated using some measures.
  • Information item-based method: In this system, the summary generation is done by supply documents and the abstract illustration is considered as information which is the smallest part of coherent information in a text.
  • Semantic Graph Model: This technique aims to develop creating a linguistics graph like a rich semantic graph (RSG). The initial document generates a linguistics graph for the final abstractive outline.
  • Semantic Text Representation Model: This technique aims to analyze input text using the semantics of words rather than syntax/Structure of text.

   Benefits

    i) Generates more Human-like Summary.

    ii) High accuracy on large and variant data

   Challenges 

  1. The major issue of abstractive summarization is there is no generalized framework, parsing and alignment of parse trees are difficult.
  2. Extracting the important sentences and sentence ordering as it has to be in the order as in the source document for producing an efficient summary is an open issue.
  3. Compressions involving lexical substitutions, paraphrase and reformulation are difficult with abstractive summarization.
  4. The capability of the system is constrained by the richness of their representation and their way to produce such structure is the greatest challenge for the abstractive summary. The capability of the system is constrained by the richness of their representation and their way to produce such structure is the greatest challenge for the abstractive summary.

Proposed Text Summarization tool 

To achieve the Text summarization of given inputs, we have used Google’s Bidirectional Encoder Representations from Transformers (BERT) model [1] to build our product. It is a powerful open-source model that constitutes pre-train Bidirectional representations from an unlabeled text by jointly conditioning on both left and right context in all layers. We have fine-tuned the pre-trained BERT model with an additional output layer to perform text summarization. BERT model, which makes use of a ‘transformer’ for managing contextual relations between words. Furthermore, an encoder is used to read the input and after transformation a predictive phrase comes out. We have developed a layer on top of the BERT core model, which generates meaningful information from large documents as shown below

Figure 3: Text Summarization Tool

Conclusion and Future work

The contribution of this project is to apply Google’s BERT model to generate short summaries. Further, we are working to utilize BERT in following applications: 

a)   Sentimental Analysis: Classification tasks in sentimental analysis.

b)   Question Answer Task: The tool would receive a text order in the form of a question and results in answering the question.

References:

[1]  https://arxiv.org/abs/1810.04805v2

[2] Y. Sankarasubramaniam, K. Ramanathan, and S. Ghosh, “Text summarization using Wikipedia,” Information Processing & Management, vol. 50, no. 3, pp. 443-461, 2014.

[3] J. M. Kleinberg, “Authoritative sources in a hyperlinked environment,” Journal of the ACM (JACM), vol. 46, no. 5, pp. 604–632, 1999.

[4] G. Erkan and D. R. Radev, “Lexrank: Graph-based lexical centrality as salience in text summarization,” Journal of Artificial Intelligence Research, pp. 457-479, 2004.

[5] S. M. R .. W. T. L., Brin, “The pagerank citation ranking: Bringing order to the web,” Technical report, Stanford University, Stanford, CA., Tech. Rep., (1998)

[6] G. Erkan and D. R. Radev, “Lexrank: Graph-based lexical centrality as salience in text summarization,” Journal of Artificial Intelligence Research, pp. 457-479, 2004.

[7] S. M. R .. W. T. L., Brin, “The pagerank citation ranking: Bringing order to the web,” Technical report, Stanford University, Stanford, CA., Tech. Rep., (1998).

[8]  X. W. Meng Wang and  C. Xu, “An approach  to concept oriented text 

summarization,”  in in Proceedings of ISClTS05,  IEEE international conference, China,1290-1293″ 2005.

[9] K. M. Svore, L. Vanderwende, and C. J. Burges, “Enhancing singledocument summarization by combining ranknet and third-party sources.” in EMNLP-CoNLL, 2007, pp. 448-457.

[10] K. Kaikhah, “Automatic text summarization with neural networks,” 2004.

Comments

No comments yet. Why don’t you start the discussion?

Leave a Reply

Your email address will not be published. Required fields are marked *