omterms

.. role:: math(raw) :format: html latex ..

Discovery and Representation of Open Making Related Terms

+-------------+ | Bulent | | Ozel, UZH | +-------------+ | bulent.oz | | [e-mail] | | m | +-------------+

Support for this work is partly covered by the OpenMaker Project: http://openmaker.eu/

Collaborator(s): * Hamza Zeytinoglu

The first objective of this module is to provide a customizable and standardized text preprocessing prior to further analyses where more advanced machine learning and or statistical techniques can be applied and compared with each other. In that sense, it provides a pipelined set of functionalities (i) to be able to inspect, organize, prune and merge texts around one or very few specific theme(s) or topic(s), (ii) remove unwanted terms or literals from the texts, (iii) tokenize the texts, (iv) count the terms in texts, and (v) when desired stem the tokenized terms.

The second objective of this module is to be able compare or score a foreground corpus or a specific corpus against a background corpus or reference corpus. Example use cases could be, for instance, exploring the language of a sub-culture, a community, or a movement looking at to what extend the specific use of the language of the group differentiates itself from the common language.

In cases when there are more than a few number of themes or topics, and where each topic is represented with a large set of documents that validates the employment of standardized matrix decomposition based methodologies, then the scoring option of this module can be skipped entirely. More specifically, in use cases where the objective is being able to classify and differentiate a number of topics or issues from each other and where there are sufficient data that fulfills the underlining assumptions of NMF, LDA or LSI based approaches, then tools from, for instance, Python’s sklearn.decomposition <http://scikit-learn.org/stable/modules/decomposition.html#non-negative-matrix-factorization-nmf-or-nnmf>__ package are suggested.

Nevertheless, the outputs of this module, such as its normalized term frequencies or the specificity scores it associates to them regarding to a reference background corpus, can be used as input to other matrix decomposition techniques.

Install

A. Via Python's standard distribution channel PyPI

.. code:: bash

    pip install omterms

B. Via from its GitHub source
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

.. code:: bash

    git clone https://github.com/bulentozel/omterms.git

.. code:: bash

    cd omterms

.. code:: bash

    pip install .

A quick use
-----------

.. code:: python

    >>> from omterms.interface import *
    >>> extract_terms("Some input X text to process less then 3 seconds.").head()
    Configuring the text cleaner ...
    A single text is provided.
    Extracting the terms ...
    Tokenizing the input text ..
    Done. Number of terms: 10
    Cleaning process: Initial size of tokens = 10
    Reduction due to punctuations and stopwords = 3.
    Reduction due to all numeral terms = 1
    Reduction due to short terms = 1
    Reduction due to rare terms = 0
    Reduction due to partially numeral terms = 0
    Reduction due to terms with not allowed symbols = 0
    The total term count reduction during this cleaning process = 5
    Percentage = 50%
    COMPLETED.
       TF     Term  wTF
    0   1    input  0.2
    1   1     text  0.2
    2   1  process  0.2
    3   1     less  0.2
    4   1  seconds  0.2
    >>> 

More on usage
-------------

`Please see the
tutorial. <https://github.com/bulentozel/omterms/blob/master/tutorial.ipynb>`__

--------------

--------------

Roadmap on Keyword and Keyphrase Extraction
===========================================

The method outlined here aims to set-up a base line for future
improvements.

-  It uses a statistical approach combined with standardized procedures
   that are widely applied in standard NLP workflows.
-  In this base line, it aims to present a work flow that can be ablied
   to

   -  different languages
   -  differrent problem domains
   -  analysis on a single theme with limited training set

1. Overall work flow
--------------------

In short, the workflow presented on this notebook is the second stage on
a workflow objective of which is being able to measure relevance of a
given external input to a specific theme, issue or topic. The steps of
the work flow is as follows.

1. Forming a specific corpus where the corpus consists of set of
   documents around a topic. The corpus could be

   -  a set of blog articles around an issue let say green finance
   -  or a set of Wikipedia articles around the same subject
   -  or collection of news articles around the green finance
   -  or collection of tweets around the same issue.

   At the moment we have another module that given a set of seed
   Wikipedia articles around an issue the crawler scrapes textual data
   from articles. For the details of the module please `see the scraper
   module. <https://github.com/bulentozel/OpenMaker/tree/master/Scraping>`__.
   The output of that module is a set of input texts stored in a
   collection in JSON format.

2. Given an input set of texts on a theme a concept or a topic identify
   set of terms that more likely or less likely can occur within a
   discussion on the topic. This module hereby presents one of the
   simple methods for this purpose.

3. Given a list of weighted terms which are more likely to occur or
   reprsent a theme, concept or topic and input query text measure the
   relevance of the input text to the topic/theme/concept. `The notebook
   in this
   link <https://github.com/bulentozel/OpenMaker/blob/master/Semantics/Score%20Text.ipynb>`__
   demonstrates one way doing such scoring of a given text against the
   curated set of terms of this particular module.

2. Suggested future work
------------------------

-  Comparing and combining this comparison based scoring with matrix
   decompostion based topic modelling approaches such as NMF, LDA, LSI.

-  Using language specicif term frequency counts of Wikipedia itself for
   comparisons. In NLP terminology, the *foreground* corpus around a
   topic needs to be compared and contrasted to a *background* corpus.

-  Improving the semantic crawler of the previous stage to be able to
   increase quality of the specific corpuses

Methodological Improvements
~~~~~~~~~~~~~~~~~~~~~~~~~~~

-  Instead of tokenizing all terms, examine possibilities of key-phrase
   extrcation combining with *tf-idf* and

   -  experiment with extracting noun phrases and words, for this use
      NLTK's regular expression module for POS (part of speeach)
      analysis.
   -  extract n-grams where n=1,2,3

3. Definitions and assumptions
------------------------------

Assumptions
~~~~~~~~~~~

-  At the comparison stage, it is assumed that a document's terms tend
   to be relatively frequent within the document as compared to an
   external reference corpus. However, it should be noted this
   assumption is contested in the field. See the paper by Chuang et el.

-  Condidering the fact that the crawler is used to aggregate
   semantically related set of documents into a single document, *tf x
   idf* is equivalent to *tf*. As can be seen below, we use a normalized
   version of *tf*: *ntS / NS*.

-  Fewer number of but relatively more relevant training (input corpus)
   is prefered in order to reduce term extraction problems due to length
   of documents. However, it should be noted that the crawling depth of
   an identiefied wiki article from stage 1 of this document can be used
   as an additional weight on relevance/reprsesntation of keywords.

-  We have limited ourselves to terms instead of n-grams and phrases or
   use of POS to be able to develop a base model that can work on
   different languages.

Term
~~~~

Given for instance a set of texts around open source software movement a
term that is identified can be a word such as *openness*, a person such
as *Stallman* a license type such as *GNU*, an acronym for an
organization such as *FSF* the Free Software Foundation, or a technology
such as *Emacs*.

Likelihood ratio
~~~~~~~~~~~~~~~~

It is a simple measure computed comparing frequency count of a term in a
specific corpus versus its frequency count in the reference reference
corpus. Here assumption is that the reference corpus is a large enough
sample of the language at observing the occurance of a term. Then having
a higher/lower observation frequency of a term in the specific corpus is
a proxy indicator for the term choice while having a debate on the
topic.

The likelihood ratio for a term :math:`P_t` is calculated as:

:math:`P_t = log ( (ntS/NS) / (ntR/NR) )`

where

-  *ntS* is the raw frequency count of the term in the entire specific
   corpus
-  *ntR* is the raw frequenccy count of the term in the reference corpus
-  *NS* is the total number of terms in the specific corpus
-  *NR* is the total number of terms in the reference corpus

It should be noted that frequency counts are calculated after having
applied the same tokenization and post processing such as excluding
stop-words, pancuations, rare terms, etc both on the reference corpus
and the specific corpus.

4. Some thoughts on a conceptual approach at using the extracted keywords or phrases to predict topical relevance of a new text
-------------------------------------------------------------------------------------------------------------------------------

Using the outcome of this technique to score arbitrary input texts
against a single issue such as financial sustainability or against a set
of issues such as the 10 basic human values requires a set of
normalization of the raw scores and their rescaling/transformation.

The factors that need to be considered are:

-  **Differing document lengths:** The likelihood of repetion of a key
   phrase increases as the size of the input text gets larger. In more
   concrete terms, when a scoring that simply sums up detection of
   weighted keyphrases or words within a given input text would be very
   sensitive to the document length. For isntance, the an executive
   summary of an article would very likely get quite lower score than
   the full article on any issue.

   *Among other methods, this can simply be resolved by computing per
   word scores, where the word set to be conidered is the tokenized and
   cleaned set of words that represent the input text.*

-  **Topical relevance:** This factor would be important when the
   subject matter of the inputs texts vary among each other. In other
   words, this factor would matter to a very high significance, let's
   say when one wants to compare perceptions of indivuduals on the role
   of privacy in democracies and when this question is not asked them in
   a uniform manner, that is under the same social, cultural,
   environmental and physical conditions.

   Let’s assume that issue under investigation is again pricacy in
   democracies. It is possible that the same individual as a blogger who
   has a strong pro-privacy opinion (i) may not touch the issue while
   talking on data science, (ii) he would slightly touch the issue while
   he talks about her preferences in mobile devices (iii) He dives into
   subject using all keywords and phrases when he talks about impact of
   privacy in a democratic life. In brief, it is necessary to offset the
   variability of the topical relavance of an input text to the issue
   under investigation when arbitrary text samples are used for scoring.

   *An offsetting scheme can be devised when opinion or perception of an
   actor is to be measured with respect to more than one factor that
   define the issue under investigation. For instance, when we want to
   measure the position of a political leader on individual liberties vs
   social security or when we want to profile discourse of the political
   leader as of a number of basic human values we could employ some
   simple statistical methods in order to offset the topical relevance
   of the discourses or the speeches of the political figure to what we
   would like to measure.*

   *A simple method could be rescaling the scores on each sub factor
   such as the scores of liberty and security that are measured from the
   same speech into a range of -1 to 1. This can simply be done by
   taking the mean of the two and then deducting the mean from each
   score and scaling them into a scala of -1 to 1. This way it may be
   possible to use multiple speeches of the same political figure on
   different topics to evalaute his or her postion on liberty vs
   security matter.*

In statistical terms this problem corresponds to adjusting or
normalizing ratings or scores measured on different scales to a
notionally common scale. Given the fact that in most cases a normal
distribution for underlying factors may not be assumed the
quantile-normalization technique is suggested. The quantile
normalization sorts and ranks the variables with a non-negative
amplitudes. Then these rankings can be scaled to for instance to a 0-1
interval.

-  **Level of subjectivity**. This is variability in terms of relevant
   importance attributed to each issue out of a given set of issues. For
   instance, it is possible that a great many individuals or political
   leaders would attach a higher importance to individual liberties than
   secuirty or otherway around. But the question might be rather to
   understand to what extend one attaches more importance to an issue
   more than the others. So when the objective of the scoring is not
   simply to make an order of importance, then a comparative importance
   with respect to overall observations needs to be tackled.

   *Observed variances in each query texts can be considered. That is, a
   simple statistical methods can be used for instance to be able to
   compare two or more query texts with respect to each other. A
   suggested method would be (1) estimate coefficient of variation for
   each input text using per-word scores (2) the rescale
   quantile-normalized scores that is suggested above using the
   estimated coefficient of variation in each case.*

   *When this rescaling is applied, for instance, liberty vs security
   the coeffcient of variation would act as a polarization measure.*

Scoring a group of variables
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

When one attempts to use scores generated by this package, using
specific vs reference corpus comparisons, on a group of variable then
both ranking of the scores as well as the relevant importance of each
score from a number of texts from the same source should be taken into
consideration.

5. State of the art
-------------------

-  Survey Paper: Kazi Saidul Hasan and Vincent Ng, 2014. “Automatic
   Keyphrase Extraction: A Survey of the State of the Art” Proceedings
   of the 52nd Annual Meeting of the Association for Computational
   Linguistics, pages 1262–1273.

-  Survey Paper: Sifatullah Siddiqi and Aditi Sharan. Article: Keyword
   and Keyphrase Extraction Techniques: A Literature Review.
   International Journal of Computer Applications 109(2):18-23, January
   2015

-  Survey Paper: Z. A. Merrouni, B. Frikh, and B. Ouhbi. Automatic
   keyphrase extraction: An overview of the state of the art. In 2016
   4th IEEE Colloquium on Information Science and Technology (CiSt),
   pages 306–313, Oct 2016

-  PageRank - Topical: Zhiyuan Liu, Wenyi Huang, Yabin Zheng and Maosong
   Sun, 2010. “Automatic Keyphrase Extraction via Topic Decomposition”.
   Proceeding EMNLP '10 Proceedings of the 2010 Conference on Empirical
   Methods in Natural Language Processing Pages 366-376

-  RAKE (Rapid Automatic Keyword Extraction ): Stuart Rose, Dave Engel,
   Nick Cramer, and Wendy Cowley. Automatic keyword extraction from
   individual documents. Text Mining, pages 1–20, 2010.

-  TextRank - Graph Based : Rada Mihalcea and Paul Tarau. Textrank:
   Bringing order into texts. Association for Computational Linguistics,
   2004.

-  STOPWORDS: S. Popova, L. Kovriguina, D. Mouromtsev, and I. Khodyrev.
   Stopwords in keyphrase extraction problem. In 14th Conference

-  Corpus Similarity - Keyword frequency based: Adam Kilgarriff. Using
   word frequency lists to measure corpus homogeneity and similarity
   between corpora. In Proceedings of ACLSIGDAT Workshop on very large
   corpora, pages 231–245, 1997.

-  Recommendation - Keyphrase Based: F. Ferrara, N. Pudota and C. Tasso.
   A keyphrase-based paper recommender system. In: Digital Libraries and
   Archives. Springer Berlin Heidelberg, 2011. p. 14-25.

-  Jason Chuang, Christopher D. Manning, Jeffrey Heer, 2012. "Without
   the Clutter of Unimportant Words": Descriptive Keyphrases for Text
   Visualization" ACM Trans. on Computer-Human Interaction, 19(3), 1–29.

+--------------------------------------------------------------+
| Learn more about the OpenMaker project: http://openmaker.eu/ |
+--------------------------------------------------------------+