Product
Introducing License Enforcement in Socket
Ensure open-source compliance with Socket’s License Enforcement Beta. Set up your License Policy and secure your software!
django-mysql-compressed-fields
Advanced tools
A large text field, stored compressed in the database, for Django and MySQL.
This package provides CompressedTextField, a MySQL-specific Django model field similar to TextField or CharField that stores its value in compressed form via zlib.
In particular you can replace a TextField or CharField like:
from django.db import models
class ProjectTextFile(models.Model):
content = models.TextField(blank=True)
with:
from django.db import models
from mysql_compressed_fields import CompressedTextField
class ProjectTextFile(models.Model):
content = CompressedTextField(blank=True)
such that the text value of the field is actually compressed in the database.
String-based lookups are supported:
html_files = ProjectTextFile.objects.filter(content__contains='<html')
html_files = ProjectTextFile.objects.filter(content__startswith='<!DOCTYPE')
html_files = ProjectTextFile.objects.filter(content__endswith='</html>')
empty_html_files = ProjectTextFile.objects.filter(content__in=['', '<html></html>'])
Advanced manipulations with MySQL's COMPRESS(), UNCOMPRESS(), and UNCOMPRESSED_LENGTH() functions are also supported:
from django.db.models import F
from mysql_compressed_fields import UncompressedLength
files = ProjectTextFile.objects.only('id').annotate(
content_length=UncompressedLength(F('content'))
)
To migrate an existing TextField or CharField to be a CompressedTextField:
pip3 install django-mysql-compressed-fields
from django.db import models
class ProjectTextFile(models.Model):
content = models.TextField(blank=True)
*_compressed
sibling field that will be used to hold the compressed
version of the original field. Mark it as default=''
. Include an explicit
db_column=...
value:from django.db import models
from mysql_compressed_fields import CompressedTextField
class ProjectTextFile(models.Model):
content = models.TextField(blank=True)
content_compressed = CompressedTextField(
blank=True,
default='', # needed by Django when adding a field
db_column='content_compressed', # pin column name
)
python3 manage.py makemigrations
python3 manage.py makemigrations --empty __APP_NAME__
from django.db import migrations
class Migration(migrations.Migration):
dependencies = [
('ide', '0002_projecttextfile_content_compressed'),
]
operations = [
]
operations
field to use a RunPython step to populate
the compressed field from the uncompressed field:from django.db import migrations
from django.db.models import F
from mysql_compressed_fields import Compress
def _populate_content_compressed(apps, schema_editor):
ProjectTextFile = apps.get_model('ide', 'ProjectTextFile')
# NOTE: Assumes "content" field is already UTF-8 encoded,
# because CompressedTextField assumes UTF-8 encoding.
ProjectTextFile.objects.update(content_compressed=Compress(F('content')))
class Migration(migrations.Migration):
dependencies = [
('ide', '0002_projecttextfile_content_compressed'),
]
operations = [
migrations.RunPython(
code=_populate_content_compressed,
reverse_code=migrations.RunPython.noop,
atomic=False,
)
]
from django.db import models
from mysql_compressed_fields import CompressedTextField
class ProjectTextFile(models.Model):
content_compressed = CompressedTextField(
blank=True,
default='', # needed by Django when adding a field
db_column='content_compressed', # pin column name
)
python3 manage.py makemigrations
*_compressed
suffix
so that it now has the name of the original uncompressed field:from django.db import models
from mysql_compressed_fields import CompressedTextField
class ProjectTextFile(models.Model):
content = CompressedTextField(
blank=True,
default='', # needed by Django when adding a field
db_column='content_compressed', # pin column name
)
python3 manage.py makemigrations
y
(for yes).This project is brought to you by TechSmart, which seeks to inspire the next generation of K-12 teachers and students to learn coding and create amazing things with computers. We use Django heavily.
All classes and functions below should be imported directly from
mysql_compressed_fields
. For example:
from mysql_compressed_fields import CompressedTextField
CompressedTextField
class CompressedTextField(encode_errors='strict', decode_errors='strict', **options)
A large text field, stored compressed in the database.
Generally behaves like TextField. Stores values in the database using the same database column type as BinaryField. The value is compressed in the same format that MySQL's COMPRESS() function uses. Compression and decompression is performed by Django and not the database.
encode_errors
controls how encoding errors are handled when saving the field. decode_errors
controls how decoding errors are handled when loading the field. If 'strict'
(the default), a UnicodeError exception is raised. Other possible values are 'ignore'
, 'replace'
, and any other name registered via codecs.register_error(). See Error Handlers for details.
If you specify a max_length
attribute, it will be reflected in the
Textarea widget of the auto-generated form field. However it is not
enforced at the model or database level. The max_length
applies to the
length of the uncompressed text rather than the compressed text.
String-based lookups can be used with this field type. Such lookups will transparently decompress the field on the database server.
html_files = ProjectTextFile.objects.filter(content__contains='<html')
html_files = ProjectTextFile.objects.filter(content__startswith='<!DOCTYPE')
html_files = ProjectTextFile.objects.filter(content__endswith='</html>')
empty_html_files = ProjectTextFile.objects.filter(content__in=['', '<html></html>'])
Note that F-expressions that reference this field type will always refer to the compressed value rather than the uncompressed value. So you may need to use the Compress() and Uncompress() database functions manually when working with F-expressions.
# Copy a TextField value (in utf8 collation) to a CompressedTextField
ProjectTextFile.objects.filter(...).update(content=Compress(F('name')))
# Copy a CompressedTextField value to a TextField (in utf8 collation)
ProjectTextFile.objects.filter(...).update(name=Uncompress(F('content')))
# Copy a CompressedTextField value to a CompressedTextField
ProjectTextFile.objects.filter(...).update(content=F('content'))
The default form widget for this field is a
django.contrib.admin.widgets.AdminTextareaWidget
(a kind of TextInput).
Compress
The MySQL COMPRESS() function, usable in F() expressions.
Uncompress
The MySQL UNCOMPRESS() function, usable in F() expressions.
UncompressedLength
The MySQL UNCOMPRESSED_LENGTH() function, usable in F() expressions.
compress
def compress(uncompressed_bytes: bytes) -> bytes:
The MySQL COMPRESS() function.
uncompress
def uncompress(compressed_bytes: bytes) -> bytes:
The MySQL UNCOMPRESS() function.
uncompressed_length
def uncompressed_length(compressed_bytes: bytes) -> int:
The MySQL UNCOMPRESSED_LENGTH() function.
compressed_length
def compressed_length(
uncompressed_bytes: bytes,
*, chunk_size: int=64 * 1000,
stop_if_greater_than: Optional[int]=None) -> int:
Returns the length of COMPRESS(uncompressed_bytes).
If stop_if_greater_than
is specified and a result greater than
stop_if_greater_than
is returned then the compressed length is
no less than the returned result.
brew install mysql-client@8.0
export PATH="/usr/local/opt/mysql-client@8.0/bin:$PATH"
docker run --name ide_db_server -e MYSQL_DATABASE=ide_db -e MYSQL_ROOT_PASSWORD=root -p 127.0.0.1:8889:3306 -d mysql:8.0
cd tests/test_data/mysite
poetry install
poetry run python3 manage.py test
encode_errors
and decode_errors
options to CompressedTextField
.FAQs
A large text field, stored compressed in the database, for Django and MySQL.
We found that django-mysql-compressed-fields demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Product
Ensure open-source compliance with Socket’s License Enforcement Beta. Set up your License Policy and secure your software!
Product
We're launching a new set of license analysis and compliance features for analyzing, managing, and complying with licenses across a range of supported languages and ecosystems.
Product
We're excited to introduce Socket Optimize, a powerful CLI command to secure open source dependencies with tested, optimized package overrides.