short-text-analyzer

Short-text-analyzer

This ShortTextAnalyzer was created to help analyze the open-ended survey response which usually has less than three sentences. The analysis includes topic modeling, sentiment analysis, and visualization. This topic modeling was done using pre-trained representations of language, namely BERT, combine with the clustering algorithm.

Documentation Page: https://thisisphume.github.io/short-text-analyzer/

Install

pip install short-text-analyzer

Install all the required packages in requirement file.

pip install -r requirements.txt

How to use

from shorttextanalyzer.core import *

analyzer = shortTextAnalyzer(comments_series, 4)
output_result = analyzer.analyze_getResult()

Here we specify that we want 4 clusters/topic from this data.

Output: result

• sentimentScore: Polarity score ranges from [-1,1] where 1 means positive statement and -1 means a negative statement.
• Subjective: score ranges from [0,1] where 1 refer to personal opinion, emotion or judgment and 0 means it is factual information.
• clusterByKMeans: assigned cluster number for each comments using KMeans
• clusterByHDBSCAN: assigned cluster number for each comments using HDBSCAN

output_result.sample(2)

.dataframe tbody tr th:only-of-type { vertical-align: middle; } .dataframe tbody tr th { vertical-align: top; } .dataframe thead th { text-align: right; }

	comments	comment_lang	comments_clean	sentimentScore	subjectiveScore	clusterByKMeans	clusterByHDBSCAN
50	sondage parfait	fr	perfect poll	1.00	1.000000	2	1
875	it wasn't very clear what the purpose of the f...	en	it wasn't very clear what the purpose of the f...	0.19	0.415833	1	1

Visualization: how good is our clusters? HDBSCAN and KMeans

analyzer.plot_output()

Reference

• tBERT: Topic Models and BERT Joining Forces for Semantic Similarity Detection: https://www.aclweb.org/anthology/2020.acl-main.630.pdf
• https://github.com/dmmiller612/bert-extractive-summarizer
• https://github.com/MilaNLProc/contextualized-topic-models
• https://github.com/MaartenGr/BERTopic