Probabilistic data structures for processing and searching very large datasets
.. image:: https://static.pepy.tech/badge/datasketch/month :target: https://pepy.tech/project/datasketch
.. image:: https://zenodo.org/badge/DOI/10.5281/zenodo.598238.svg :target: https://zenodo.org/doi/10.5281/zenodo.598238
datasketch gives you probabilistic data structures that can process and search very large amount of data super fast, with little loss of accuracy.
This package contains the following data sketches:
+-------------------------+-----------------------------------------------+
| Data Sketch | Usage |
+=========================+===============================================+
| MinHash_ | estimate Jaccard similarity and cardinality |
+-------------------------+-----------------------------------------------+
| Weighted MinHash_ | estimate weighted Jaccard similarity |
+-------------------------+-----------------------------------------------+
| HyperLogLog_ | estimate cardinality |
+-------------------------+-----------------------------------------------+
| HyperLogLog++_ | estimate cardinality |
+-------------------------+-----------------------------------------------+
The following indexes for data sketches are provided to support sub-linear query time:
+---------------------------+-----------------------------+------------------------+
| Index | For Data Sketch | Supported Query Type |
+===========================+=============================+========================+
| MinHash LSH_ | MinHash, Weighted MinHash | Jaccard Threshold |
+---------------------------+-----------------------------+------------------------+
| MinHash LSH Forest_ | MinHash, Weighted MinHash | Jaccard Top-K |
+---------------------------+-----------------------------+------------------------+
| MinHash LSH Ensemble_ | MinHash | Containment Threshold |
+---------------------------+-----------------------------+------------------------+
| HNSW_ | Any | Custom Metric Top-K |
+---------------------------+-----------------------------+------------------------+
datasketch must be used with Python 3.7 or above, NumPy 1.11 or above, and Scipy.
Note that MinHash LSH_ and MinHash LSH Ensemble_ also support Redis and Cassandra
storage layer (see MinHash LSH at Scale_).
To install datasketch using pip:
::
pip install datasketch
This will also install NumPy as dependency.
To install with Redis dependency:
::
pip install datasketch[redis]
To install with Cassandra dependency:
::
pip install datasketch[cassandra]
.. _MinHash: https://ekzhu.github.io/datasketch/minhash.html
.. _Weighted MinHash: https://ekzhu.github.io/datasketch/weightedminhash.html
.. _HyperLogLog: https://ekzhu.github.io/datasketch/hyperloglog.html
.. _HyperLogLog++: https://ekzhu.github.io/datasketch/hyperloglog.html#hyperloglog-plusplus
.. _MinHash LSH: https://ekzhu.github.io/datasketch/lsh.html
.. _MinHash LSH Forest: https://ekzhu.github.io/datasketch/lshforest.html
.. _MinHash LSH Ensemble: https://ekzhu.github.io/datasketch/lshensemble.html
.. _Minhash LSH at Scale: http://ekzhu.github.io/datasketch/lsh.html#minhash-lsh-at-scale
.. _HNSW: https://ekzhu.github.io/datasketch/documentation.html#hnsw
FAQs
Probabilistic data structures for processing and searching very large datasets
We found that datasketch demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.
Security News
The Ultralytics' PyPI Package was compromised four times in one weekend through GitHub Actions cache poisoning and failure to rotate previously compromised API tokens.