Useful Algorithms for Keyword extraction.

[19bsr16054]

Various Algorithms related to NLP will be discussed in brief.
Current algorithms and their assignment

Sunag - Rake
Priti - yake
Dhananjay - gensim
Kavin - a custom variation for Bert
Karthik - TD - IDF
Gayathri - keyBERT

[sunagparasu]

• Collaborator: Parasu Nagesh Sunag

Rake: Rapid Automatic Keyword Extraction

• Rapid Automatic Keyword Extraction (RAKE) is a Domain-Independent keyword extraction algorithm in Natural Language Processing.

Installation and Use

• RAKE can be downloaded from the terminal using pip install rake-nltk

• To use RAKE, The python code needs to be imported into the environment. This can be achieved by import rake and import operator

• After the code has been imported, enter the path of your results file and apply the algorithm on the text.

filepath = "keyword_extraction.txt" rake_object = rake.Rake(filepath) text = "Feature extraction is not that complex. There are many algorithms available that can help you with feature extraction. Rapid Automatic Keyword Extraction is one of those" sample_file = open(“data/docs/fao_test/w2167e.txt”, ‘r’) text = sample_file.read() keywords = rake_object.run(text) print “Keywords:”, keywords

• A simpler method would be to use the keyword extraction section of the google colaboratory to obtain the keywords.
• Change the chapter paths to your requirement and enter the method as rake and run the cell to obtain the result.

Features

• The concept of RAKE is built on three matrices Word Degree, Word Frequency, and ratio of the degree to frequency.

Evaluation

• A good feature of RAKE would be the use of matrices and ratios while a disadvantage would be the lack of an argument to customize the extraction of n_grams.

Result

• The following .csv file was obtained by running the keywords extraction section of the google colab.
  Rake_keywords.csv

[19bsr16054]

• Collaborator - Karthikeyan V

Method TF - IDF

• stands for Term Frequency & Inverse Document Frequency algorithm

Installation

• It is a part of the scikit-learn python module and can be installed with

  pip install scikit-learn  
  

• It can be imported using

  import TfidfTransformer from sklearn.feature_extraction.text
  

Features

• It compares the Term Frequency of a word that is the number of times a word appears in one document against the total word count of the document.
• It compares the Term Frequency to the Document Frequency which is the number of times the same word appears in a set of documents (also known as corpus), inversely.
• TF gives us information on how often a term appears in a document and IDF gives us information about the relative rarity of a term in the collection of documents. By multiplying these values together we can get our final TF-IDF value.

Pros and Cons

• Works perfectly fine for monograms/ single words.
• Its also super cheap on resources used to compute, hence used as a base algorithm with custom weighted options
• It cannot carry or analyse the semantic meaning and relation between the words

[sunagparasu]

• Collaborator: Dhananjay Sharma

GenSim: Generate Similar

• Gensim is billed as a Natural Language Processing package that extracts the underlying topics from large volumes of text.

Installation and Use

• This software depends on NumPy and Scipy, two Python packages for scientific computing. You must have them installed prior to installing gensim.

• Install the latest version of gensim using pip install --upgrade gensim

• After the code has been imported, enter the path of your results file and apply the algorithm on the text.

import gensim from gensim import corpora from pprint import pprint How to create a dictionary from a list of sentences? documents = ["The Saudis are preparing a report that will acknowledge that", "Saudi journalist Jamal Khashoggi's death was the result of an", "interrogation that went wrong, one that was intended to lead", "to his abduction from Turkey, according to two sources."]

documents_2 = ["One source says the report will likely conclude that", "the operation was carried out without clearance and", "transparency and that those involved will be held", "responsible. One of the sources acknowledged that the", "report is still being prepared and cautioned that", "things could change."] Tokenize(split) the sentences into words texts = [[text for text in doc.split()] for doc in documents] Create dictionary dictionary = corpora.Dictionary(texts) Get information about the dictionary print(dictionary)

• A simpler method would be to use the keyword extraction section of the google colaboratory to obtain the keywords.
• Change the chapter paths to your requirement and enter the method as gensim and run the cell to obtain the result.

Features

• Gensim provides algorithms like LDA (Latent Dirichlet Allocation), LSI (Latent Semantic Analysis) or word2vec and the necessary sophistication to build high-quality dictionaries.

Evaluation

• A good feature of gensim would be the use of algorithms like word2vec which converts words to vectors. The vectors try to capture various characteristics of that word with regard to the overall text. These characteristics can include the semantic relationship of the word, definitions, context, etc. While a disadvantage would be that word2vec cannot handle ambiguities.

Result

• The following .csv file was obtained by running the keywords extraction section of the google colab.
  gensim_keywords.csv

[priti-chahal]

Collaborator: Priti

YAKE

Yet Another Keyword Extractor (Yake) library selects the most important keywords using the text statistical features method from the article.

Installation and Usage

Installing Yake using pip:

pip install yake

pip install git+https://github.com/LIAAD/yake

Usage (Command line)

Usage: yake [OPTIONS]

Options:
	-ti, --text_input TEXT                            Input text, SURROUNDED by single quotes(\')
	-i, --input_file TEXT                              Input file
	-l, --language TEXT                              Language
	-n, --ngram-size INTEGER                     Max size of the ngram.
	-df, --dedup-func [leve|jaro|seqm]       Deduplication function.
	-dl, --dedup-lim FLOAT                        Deduplication limiar.
	-ws, --window-size INTEGER                Window size.
	-t, --top INTEGER                                  Number of keyphrases to extract
	-v, --verbose			                      Gets detailed information (such as the score)
	--help                                                    Show this message and exit.

Chapter08 Keyword extraction (Yake) google colab: Click here