pymilvus-distributed
Advanced tools
Python SDK for Milvus. To contribute code to this project, please read our contribution guidelines first.
For detailed SDK documentation, refer to API Documentation.
remove get_index_info, the index info can be obtained by get_collection_info
add get_collection_stats, more detailed collection stats.
pymilvus only supports Python 3.5 or higher.
You can install pymilvus via pip or pip3 for Python3:
$ pip3 install pymilvus
The following collection shows Milvus versions and recommended pymilvus versions:
| Milvus version | Recommended pymilvus version |
|---|---|
| 0.3.0 | 0.1.13 |
| 0.3.1 | 0.1.25 |
| 0.4.0 | 0.2.2 |
| 0.5.0 | 0.2.3 |
| 0.5.1 | 0.2.3 |
| 0.5.2 | 0.2.3 |
| 0.5.3 | 0.2.5 |
| 0.6.0 | 0.2.6, 0.2.7 |
| 0.7.0 | 0.2.8 |
| 0.7.1 | 0.2.9 |
| 0.8.0 | 0.2.10 |
| 0.9.0 | 0.2.11 |
| 0.9.1 | 0.2.12 |
| 0.10.0 | 0.2.13 |
| >=0.10.1, <0.11.0 | 0.2.14 |
| 0.11.0 | 0.3.0 |
You can install a specific version of pymilvus by:
$ pip install pymilvus==0.3.0
You can upgrade pymilvus to the latest version by:
$ pip install --upgrade pymilvus
Refer to examples for more example programs.
# Import pymilvus
>>> from milvus import Milvus, DataType
# Connect to Milvus server
>>> client = Milvus(host='localhost', port='19530')
Note: In the above code, default values are used for
hostandportparameters. Feel free to change them to the IP address and port you set for Milvus server.
>>> client = Milvus(uri='tcp://localhost:19530')
# create collection name
>>> collection_name = 'test01'
# create a collection of 4 fields, fields A, B and C are int type fields
# and Vec is a float vector field.
# segment_row_limit is default as 524288 if not specified
>>> collection_param = {
... "fields": [
... {"name": "A", "type": DataType.INT32},
... {"name": "B", "type": DataType.INT32},
... {"name": "C", "type": DataType.INT64},
... {"name": "Vec", "type": DataType.FLOAT_VECTOR, "params": {"dim": 128}}
... ],
... "segment_row_limit": 4096,
... "auto_id": True
... }
test01 with dimension of 128, size of the data file for Milvus to automatically
create indexes as 4096. If metric_type isn't offered, default metric type is Euclidean distance (L2).
For FLOAT_VECTOR field, dim is a must.# Create a collection
>>> client.create_collection(collection_name, collection_param)
get_collection_info>>> info = client.get_collection_info('test01')
>>> info
{'fields': [
{'name': 'A', 'type': <DataType.INT32: 4>, 'params': {}, 'indexes': [{}]},
{'name': 'C', 'type': <DataType.INT64: 5>, 'params': {}, 'indexes': [{}]},
{'name': 'B', 'type': <DataType.INT32: 4>, 'params': {}, 'indexes': [{}]},
{'name': 'Vec', 'type': <DataType.FLOAT_VECTOR: 101>, 'params': {'dim': 128},
'indexes': [{}]}
],
'auto_id': True,
'segment_row_limit': 4096
}
You can see from the info, there is an auto_id option in collection info, and its True by default.
So if you have your own ids and don't want auto generated ids, you may want to set auto_id to False
while creating collections.
# Drop collection
>>> status = client.drop_collection('test01')
>>> status
Status(code=0, message='OK')
You can split collections into partitions by partition tags for improved search performance.
# Create partition
>>> client.create_partition(collection_name='test01', partition_tag='tag01')
Use list_partitions() to verify whether the partition is created.
# Show partitions
>>> partitions = client.list_partitions(collection_name='test01')
>>> partitions
['_default', 'tag01']
# Drop partitions
>>> status = client.drop_partition(collection_name='test01', partition_tag='tag01')
Status(code=0, message='OK')
Note: In production, it is recommended to create indexes before inserting vectors into the collection. Index is automatically built when vectors are being imported. However, you need to create the same index again after the vector insertion process is completed because some data files may not meet the
index_file_sizeand index will not be automatically built for these data files.
IVF_FLAT with nlist = 100 for the collection.# Create index
>>> status = client.create_index('test01', "Vec", {"index_type": "IVF_FLAT", "metric_type": "L2", "params": {"nlist": 100}})
>>> status
Status(code=0, message='OK')
get_collection_info>>> info = client.get_collection_info('test01')
>>> info
{'fields': [
{'name': 'A', 'type': <DataType.INT32: 4>, 'params': {}, 'indexes': [{}]},
{'name': 'C', 'type': <DataType.INT64: 5>, 'params': {}, 'indexes': [{}]},
{'name': 'B', 'type': <DataType.INT32: 4>, 'params': {}, 'indexes': [{}]},
{'name': 'Vec',
'type': <DataType.FLOAT_VECTOR: 101>,
'params': {'dim': 128, 'metric_type': 'L2'},
'indexes': [{'index_type': 'IVF_FLAT', 'metric_type': 'L2', 'params': {'nlist': 100}}]}],
'auto_id': True,
'segment_row_limit': 4096
}
# Drop an index of a specific field "Vec"
>>> status = client.drop_index('test01', "Vec")
Status(code=0, message='OK')
>>> import random
>>> num = 5000
# Generate a list of integer.
>>> list_of_int = [random.randint(0, 255) for _ in range(num)]
# Generate 20 vectors of 128 dimension
>>> vectors = [[random.random() for _ in range(128)] for _ in range(num)]
>>> hybrid_entities = [
{"name": "A", "values": list_of_int, "type": DataType.INT32},
{"name": "B", "values": list_of_int, "type": DataType.INT32},
{"name": "C", "values": list_of_int, "type": DataType.INT64},
{"name": "Vec", "values": vectors, "type":DataType.FLOAT_VECTOR}
]
If you create a new collection with auto_id = True, Milvus automatically generates IDs for the vectors.
# Insert vectors
>>> ids = client.insert('test01', hybrid_entities)
If you create a new collection with auto_id = False, you have to provide user-defined vector ids:
# Generate fake custom ids
>>> vector_ids = [id for id in range(num)]
# Insert to the non-auto-id collection
>>> ids = client.insert('test01', hybrid_entities, ids=vector_ids)
The examples below are based on collection with auto_id = True.
>>> inserted_vector_ids = client.insert('test01', hybrid_entities, partition_tag="tag01")
To verify the entities you have inserted, use get_entity_by_id().
>>> entities = client.get_entity_by_id(collection_name='test01', ids=inserted_vector_ids[:10])
You can delete these entities by:
>>> status = client.delete_entity_by_id('test01', ids[:10])
>>> status
Status(code=0, message='OK')
When performing operations related to data changes, you can flush the data from memory to disk to avoid possible data loss. Milvus also supports automatic flushing, which runs at a fixed interval to flush the data in all collections to disk. You can use the Milvus server configuration file to set the interval.
>>> client.flush(['test01'])
A segment is a data file that Milvus automatically creates by merging inserted vector data. A collection can contain multiple segments. If some vectors are deleted from a segment, the space taken by the deleted vectors cannot be released automatically. You can compact segments in a collection to release space.
>>> status = client.compact('test01')
>>> status
Status(code=0, message='OK')
"term" and "range" is optional, "params" in "vector" stands for index params.# This dsl will search topk `entities` that are
# close to vectors[:1] searched by `IVF_FLAT` index with `nprobe = 10` and `metric_type = L2`,
# AND field "A" in [1, 2, 5],
# AND field "B" greater than 1 less than 100
>>> dsl = {
... "bool": {
... "must":[
... {
... "term": {"A": [1, 2, 5]}
... },
... {
... "range": {"B": {"GT": 1, "LT": 100}}
... },
... {
... "vector": {
... "Vec": {"topk": 10, "query": vectors[:1], "metric_type": "L2", "params": {"nprobe": 10}}
... }
... }
... ]
... }
... }
A search without hybrid conditions with IVF_FLAT index would be like:
>>> dsl = {
... "bool": {
... "must":[
... {
... "vector": {
... "Vec": {"topk": 10, "query": vectors[:1], "metric_type": "L2", "params": {"nprobe": 10}}
... }
... }
... ]
... }
... }
A FLAT search doesn't need index params, so the query would be like:
>>> dsl = {
... "bool": {
... "must":[
... {
... "vector": {
... "Vec": {"topk": 10, "query": vectors[0], "metric_type": "L2"}
... }
... }
... ]
... }
... }
With fields=["B"], not only can you get entity ids and distances, but also values of a spacific field B.
# search entities and get entity field B back
>>> results = client.search('test01', dsl, fields=["B"])
You can obtain ids, distances and fields by entities in results.
# Results consist of number-of-query entities
>>> entities = results[0]
# Entities consists of topk entity
>>> entity = entities[0]
# You can get all ids and distances by entities
>>> all_ids = entities.ids
>>> all_distances = entities.distances
# Or you can get them one by one by entity
>>> a_id = entity.id
>>> a_distance = entity.distance
>>> a_field = entity.entity.B # getattr(entity.entity, "B")
Note: If you don't provide fields in search, you will only get ids and distances.
# Search entities in a partition `tag01`
>>> client.search(collection_name='test01', dsl=dsl, partition_tags=['tag01'])
Note: If you do not specify
partition_tags, Milvus searches the whole collection.
>>> client.close()
I'm getting random "socket operation on non-socket" errors from gRPC when connecting to Milvus from an application served on Gunicorn
Make sure to set the environment variable GRPC_ENABLE_FORK_SUPPORT=1. For reference, see https://zhuanlan.zhihu.com/p/136619485
FAQs
Python Sdk for Milvus-Distributed
We found that pymilvus-distributed demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 2 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.
Security News
The Ultralytics' PyPI Package was compromised four times in one weekend through GitHub Actions cache poisoning and failure to rotate previously compromised API tokens.