redis-search-django

redis-search-django

About

A Django package that provides auto indexing and searching capabilities for Django model instances using RediSearch.

Features

• Management Command to create, update and populate the RediSearch Index.
• Auto Index on Model object Create, Update and Delete.
• Auto Index on Related Model object Add, Update, Remove and Delete.
• Easy to create Document classes (Uses Django Model Form Class like structure).
• Index nested models (e.g: OneToOneField, ForeignKey and ManyToManyField).
• Search documents using redis-om.
• Search Result Pagination.
• Search Result Sorting.
• RediSearch Result to Django QuerySet.
• Faceted Search.

Requirements

• Python: 3.7, 3.8, 3.9, 3.10
• Django: 3.2, 4.0, 4.1
• redis-om: >= 0.0.27

Redis

Downloading Redis

The latest version of Redis is available from Redis.io. You can also install Redis with your operating system's package manager.

RediSearch and RedisJSON

redis-search-django relies on the RediSearch and RedisJSON Redis modules to support rich queries and embedded models. You need these Redis modules to use redis-search-django.

The easiest way to run these Redis modules during local development is to use the redis-stack Docker image.

Docker Compose

There is a docker-compose.yaml file provided in the project's root directory. This file will run Redis with RedisJSON and RediSearch modules during development.

Run the following command to start the Redis container:

docker compose up -d

Example Project

There is an example project available at Example Project.

Documentation

Installation

pip install redis-search-django

Then add redis_search_django to your INSTALLED_APPS:

INSTALLED_APPS = [
    ...
    'redis_search_django',
]

Usage

Document Types

There are 3 types of documents class available:

• JsonDocument: This uses RedisJSON to store the document. If you want to use Embedded Documents (Required For OneToOneField, ForeignKey and ManyToManyField) then use JsonDocument.
• EmbeddedJsonDocument: If the document will be embedded inside another document class then use this. Embedded Json Documents are used for OneToOneField, ForeignKey and ManyToManyField or any types of nested documents.
• HashDocument: This uses RedisHash to store the documents. It can not be used for nested documents.

Creating Document Classes

You need to inherit from The Base Document Classes mentioned above to build a document class.

Simple Example

1. For Django Model:

# models.py

from django.db import models


class Category(models.Model):
    name = models.CharField(max_length=30)
    slug = models.SlugField(max_length=30)

    def __str__(self) -> str:
        return self.name

2. You can create a document class like this:

Note: Document classes must be stored in documents.py file.

# documents.py

from redis_search_django.documents import JsonDocument

from .models import Category


class CategoryDocument(JsonDocument):
    class Django:
        model = Category
        fields = ["name", "slug"]

3. Run Index Django Management Command to create the index on Redis:

python manage.py index

Note: This will also populate the index with existing data from the database

Now category objects will be indexed on create/update/delete.

More Complex Example

1. For Django Models:

# models.py

from django.db import models


class Tag(models.Model):
    name = models.CharField(max_length=30)

    def __str__(self) -> str:
        return self.name


class Vendor(models.Model):
    name = models.CharField(max_length=30)
    email = models.EmailField()
    establishment_date = models.DateField()

    def __str__(self) -> str:
        return self.name


class Product(models.Model):
    name = models.CharField(max_length=256)
    description = models.TextField(blank=True)
    vendor = models.OneToOneField(Vendor, on_delete=models.CASCADE)
    tags = models.ManyToManyField(Tag, blank=True)
    price = models.DecimalField(max_digits=6, decimal_places=2)

    def __str__(self) -> str:
        return self.name

2. You can create a document classes like this:

Note: Document classes must be stored in documents.py file.

# documents.py

from typing import List

from django.db import models
from redis_om import Field

from redis_search_django.documents import EmbeddedJsonDocument, JsonDocument

from .models import Product, Tag, Vendor


class TagDocument(EmbeddedJsonDocument):
    custom_field: str = Field(index=True, full_text_search=True)

    class Django:
        model = Tag
        # Model Fields
        fields = ["name"]

    @classmethod
    def prepare_custom_field(cls, obj):
        return "CUSTOM FIELD VALUE"


class VendorDocument(EmbeddedJsonDocument):
    class Django:
        model = Vendor
        # Model Fields
        fields = ["name", "establishment_date"]


class ProductDocument(JsonDocument):
    # OnetoOneField, with null=False
    vendor: VendorDocument
    # ManyToManyField
    tags: List[TagDocument]

    class Django:
        model = Product
        # Model Fields
        fields = ["name", "description", "price"]
        # Related Model Options
        related_models = {
            Vendor: {
                "related_name": "product",
                "many": False,
            },
            Tag: {
                "related_name": "product_set",
                "many": True,
            },
        }

    @classmethod
    def get_queryset(cls) -> models.QuerySet:
        """Override Queryset to filter out available products."""
        return super().get_queryset().filter(available=True)

    @classmethod
    def prepare_name(cls, obj):
        """Use this to update field value."""
        return obj.name.upper()

Note:

• You can not inherit from HashDocument for documents that include nested fields.
• You need to inherit from EmbeddedJsonDocument for document classes that will be embedded inside another document class.
• You need to explicitly add OneToOneField, ForeignKey or ManyToManyField (e.g: tags: List[TagDocument]) with an embedded document class if you want to index them. you can not add it in the Django.fields option.
• For related_models option, you need to specify the fields related_name and if it is a ManyToManyField or a ForeignKey Field then specify "many": True.
• related_models will be used when a related object is saved that contributes to the document.
• You can define prepare_{field_name} method to update the value of a field before indexing.
• If it is a custom field (not a model field) you must define a prepare_{field_name} method that returns the value of the field.
• You can override get_queryset method to provide more filtering. This will be used while indexing a queryset.
• Field names must match model field names or define a prepare_{field_name} method.

3. Run Index Django Management Command to create the index on Redis:

python manage.py index

Note: This will also populate the index with existing data from the database

Management Command

This package comes with index management command that can be used to index all the model instances to Redis index if it has a Document class defined.

Note: Make sure that Redis is running before running the command.

Run the following command to index all models that have Document classes defined:

python manage.py index

You can use --migrate-only option to only update the index schema.

python manage.py index --migrate-only

You can use --models to specify which models to index (models must have a Document class defined to be indexed).

python manage.py index --models app_name.ModelName app_name2.ModelName2

Views

You can use the redis_search_django.mixin.RediSearchListViewMixin with a Django Generic View to search for documents. RediSearchPaginator which helps paginate ReadiSearch results is also added to this mixin.

Example

# views.py

from django.utils.functional import cached_property
from django.views.generic import ListView
from redis.commands.search import reducers

from redis_search_django.mixins import RediSearchListViewMixin

from .documents import ProductDocument
from .models import Product


class SearchView(RediSearchListViewMixin, ListView):
    paginate_by = 20
    model = Product
    template_name = "core/search.html"
    document_class = ProductDocument

    @cached_property
    def search_query_expression(self):
        query = self.request.GET.get("query")
        query_expression = None

        if query:
            query_expression = (
                self.document_class.name % query
                | self.document_class.description % query
            )

        return query_expression

    @cached_property
    def sort_by(self):
        return self.request.GET.get("sort")

    def facets(self):
        if self.search_query_expression:
            request = self.document_class.build_aggregate_request(
                self.search_query_expression
            )
        else:
            request = self.document_class.build_aggregate_request()

        result = self.document_class.aggregate(
            request.group_by(
                ["@tags_name"],
                reducers.count().alias("count"),
            )
        )
        return result

Search

This package uses redis-om to search for documents.

Example

from .documents import ProductDocument


categories = ["category1", "category2"]
tags = ["tag1", "tag2"]

# Search For Products That Match The Search Query (name or description)
query_expression = (
    ProductDocument.name % "Some search query"
    | ProductDocument.description % "Some search query"
)

# Search For Products That Match The Price Range
query_expression = (
    ProductDocument.price >= float(10) & ProductDocument.price <= float(100)
)

# Search for Products that include following Categories
query_expression = ProductDocument.category.name << ["category1", "category2"]

# Search for Products that include following Tags
query_expression = ProductDocument.tags.name << ["tag1", "tag2"]

# Query expression can be passed on the `find` method
result = ProductDocument.find(query_expression).sort_by("-price").execute()

For more details checkout redis-om docs

RediSearch Aggregation / Faceted Search

redis-om does not support faceted search (RediSearch Aggregation). So this package uses redis-py to do faceted search.

Example

from redis.commands.search import reducers

from .documents import ProductDocument


query_expression = (
    ProductDocument.name % "Some search query"
    | ProductDocument.description % "Some search query"
)

# First we need to build the aggregation request
request1 = ProductDocument.build_aggregate_request(query_expression)
request2 = ProductDocument.build_aggregate_request(query_expression)

# Get the number of products for each category
ProductDocument.aggregate(
    request1.group_by(
        ["@category_name"],
        reducers.count().alias("count"),
    )
)
# >> [{"category_name": "Shoes", "count": "112"}, {"category_name": "Cloths", "count": "200"}]


# Get the number of products for each tag
ProductDocument.aggregate(
    request2.group_by(
        ["@tags_name"],
        reducers.count().alias("count"),
    )
)
# >> [{"tags_name": "Blue", "count": "14"}, {"tags_name": "Small", "count": "57"}]

For more details checkout redis-py docs and RediSearch Aggregation docs

Settings

Environment Variables

• REDIS_OM_URL (Default: redis://localhost:6379): This environment variable follows the redis-py URL format. If you are using external redis server You need to set this variable with the URL of the redis server following this pattern: redis://[[username]:[password]]@[host]:[post]/[database number]

Example: redis://redis_user:[e-mail]:6379/0

For more details checkout redis-om docs

Django Document Options

You can add these options on the Django class of each Document class:

# documents.py

from redis_search_django.documents import JsonDocument

from .models import Category, Product, Tag, Vendor


class ProductDocument(JsonDocument):
    class Django:
        model = Product
        fields = ["name", "description", "price", "created_at"]
        select_related_fields = ["vendor", "category"]
        prefetch_related_fields = ["tags"]
        auto_index = True
        related_models = {
            Vendor: {
                "related_name": "product",
                "many": False,
            },
            Category: {
                "related_name": "product_set",
                "many": True,
            },
            Tag: {
                "related_name": "product_set",
                "many": True,
            },
        }

• model (Required): Django Model class to index.
• auto_index (Default: True, Optional): If True, the model instances will be indexed on create/update/delete.
• fields (Default: [], Optional): List of model fields to index. (Do not add OneToOneField, ForeignKey or ManyToManyField here. These need to be explicitly added to the Document class using EmbeddedJsonDocument.)
• select_related_fields (Default: [], Optional): List of fields to use on queryset.select_related().
• prefetch_related_fields (Default: [], Optional): List of fields to use on queryset.prefetch_related().
• related_models (Default: {}, Optional): Dictionary of related models. You need to specify the fields related_name and if it is a ManyToManyField or a ForeignKey Field then specify "many": True. These are used to update the document data if any of the related model instances are updated. related_models will be used when a related object is saved/added/removed/deleted that contributes to the document.

For redis-om specific options checkout redis-om docs

Global Options

You can add these options to your Django settings.py File:

• REDIS_SEARCH_AUTO_INDEX (Default: True): Enable or Disable Auto Index when model instance is created/updated/deleted for all document classes.

Example Application Screenshot

License

The code in this project is released under the MIT License.