github.com/vench/nlp

Natural Language Processing

An implementation of selected machine learning algorithms for basic natural language processing in golang. The initial focus for this project is Latent Semantic Analysis to allow retrieval/searching, clustering and classification of text documents based upon semantic content.

Built upon the Gonum library for linear algebra and scientific computing with some inspiration taken from Python's scikit-learn.

Check out the companion blog post or the go documentation page for full usage and examples.

Features

• Sparse matrix implementations for more effective memory usage
• Convert plain text strings into numerical feature vectors for analysis
• Stop word removal to remove frequently occuring English words e.g. "the", "and"
• Feature hashing('the hashing trick') implementation (using MurmurHash3) for reduced memory requirements and reduced reliance on training data
• TF-IDF weighting to account for frequently occuring words
• LSA (Latent Semantic Analysis aka Latent Semantic Indexing (LSI)) implementation using truncated SVD (Singular Value Decomposition) for dimensionality reduction.
• PCA (Principal Component Analysis)
• SimHash implementation of LSH (Locality Sensitive Hashing) using sign random projection to support approximate cosine similarity using significantly less memory and processing time.
• Random Indexing (RI) and Reflective Random Indexing (RRI) (which extends RI to support indirect inference) for scalable Latent Semantic Analysis (LSA) with semantic vector space models.
• Cosine, Angular and Hamming similarity/distance measures to calculate the similarity/distance between feature vectors.
• Persistence for trained models (persistence for Vectorisers coming soon)

Planned

• Ability to persist trained vectorisers
• LDA (Latent Dirichlet Allocation) implementation for topic extraction
• Stemming to treat words with common root as the same e.g. "go" and "going"
• Clustering algorithms e.g. Heirachical, K-means, etc.
• Classification algorithms e.g. SVM, random forest, etc.

References

1. Rosario, Barbara. Latent Semantic Indexing: An overview. INFOSYS 240 Spring 2000
2. Latent Semantic Analysis, a scholarpedia article on LSA written by Tom Landauer, one of the creators of LSA.
3. Thomo, Alex. Latent Semantic Analysis (Tutorial).
4. Latent Semantic Indexing. Standford NLP Course
5. Charikar, Moses S. Similarity Estimation Techniques from Rounding Algorithms
6. Kanerva, Pentti, Kristoferson, Jan and Holst, Anders (2000). Random Indexing of Text Samples for Latent Semantic Analysis
7. Rangan, Venkat. Discovery of Related Terms in a corpus using Reflective Random Indexing
8. Vasuki, Vidya and Cohen, Trevor. Reflective random indexing for semi-automatic indexing of the biomedical literature