django-pgcrypto-fields
Advanced tools
Encrypted fields for Django dealing with pgcrypto postgres extension.
django-pgcrypto-fields is a Django extension which relies upon pgcrypto to
encrypt and decrypt data for fields.
pgcryptoLast version of this library that supports Django 1.8.x, 1.9.x, 1.10.x
was django-pgcrypto-fields 2.2.0.
Last version of this library that supports Django 2.0.x and 2.1.x was
was django-pgcrypto-fields 2.5.2.
pip install django-pgcrypto-fields
Our library support different crypto keys for multiple databases by
defining the keys in your DATABASES settings.
In settings.py:
import os
BASEDIR = os.path.dirname(os.path.dirname(__file__))
PUBLIC_PGP_KEY_PATH = os.path.abspath(os.path.join(BASEDIR, 'public.key'))
PRIVATE_PGP_KEY_PATH = os.path.abspath(os.path.join(BASEDIR, 'private.key'))
# Used by PGPPublicKeyField used by default if not specified by the db
PUBLIC_PGP_KEY = open(PUBLIC_PGP_KEY_PATH).read()
PRIVATE_PGP_KEY = open(PRIVATE_PGP_KEY_PATH).read()
# Used by TextHMACField and PGPSymmetricKeyField if not specified by the db
PGCRYPTO_KEY='ultrasecret'
DIFF_PUBLIC_PGP_KEY_PATH = os.path.abspath(
os.path.join(BASEDIR, 'tests/keys/public_diff.key')
)
DIFF_PRIVATE_PGP_KEY_PATH = os.path.abspath(
os.path.join(BASEDIR, 'tests/keys/private_diff.key')
)
# And add 'pgcrypto' to `INSTALLED_APPS` to create the extension for
# pgcrypto (in a migration).
INSTALLED_APPS = (
'pgcrypto',
# Other installed apps
)
DATABASES = {
# This db will use the default keys above
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'pgcryto_fields',
'USER': 'pgcryto_fields',
'PASSWORD': 'xxxx',
'HOST': 'psql.test.com',
'PORT': 5432,
'OPTIONS': {
'sslmode': 'require',
}
},
'diff_keys': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'pgcryto_fields_diff',
'USER': 'pgcryto_fields_diff',
'PASSWORD': 'xxxx',
'HOST': 'psqldiff.test.com',
'PORT': 5432,
'OPTIONS': {
'sslmode': 'require',
},
'PGCRYPTO_KEY': 'djangorocks',
'PUBLIC_PGP_KEY': open(DIFF_PUBLIC_PGP_KEY_PATH, 'r').read(),
'PRIVATE_PGP_KEY': open(DIFF_PRIVATE_PGP_KEY_PATH, 'r').read(),
},
}
The public key is going to encrypt the message and the private key will be needed to decrypt the content. The following commands have been taken from the pgcrypto documentation (see Generating PGP Keys with GnuPG).
Generating a public and a private key (The preferred key type is "DSA and Elgamal".):
$ gpg --gen-key
$ gpg --list-secret-keys
/home/bob/.gnupg/secring.gpg
---------------------------
sec 2048R/21 2014-10-23
uid Test Key <example@example.com>
ssb 2048R/42 2014-10-23
$ gpg -a --export 42 > public.key
$ gpg -a --export-secret-keys 21 > private.key
This library currently does not support Public Key Encryption private keys that are password protected yet. See Issue #89 to help implement it.
The 2.4.0 version of this library received a large rewrite in order to support auto-decryption when getting encrypted field data as well as the ability to filter on encrypted fields without using the old PGPCrypto aggregate functions available in previous versions.
The following items in this library have been removed and therefore references in your application to these items need to be removed as well:
managers.PGPManageradmin.PGPAdminaggregates.*django-pgcrypto-fields has 3 kinds of fields:
Supported hash based fields are:
TextDigestFieldTextHMACFieldTextDigestField is hashed in the database using the digest pgcrypto function
using the sha512 algorithm.
TextHMACField is hashed in the database using the hmac pgcrypto function
using a key and the sha512 algorithm. This is similar to the digest version however
the hash can only be recalculated knowing the key. This prevents someone from altering
the data and also changing the hash to match.
Supported PGP public key fields are:
CharPGPPublicKeyFieldEmailPGPPublicKeyFieldTextPGPPublicKeyFieldDatePGPPublicKeyFieldDateTimePGPPublicKeyFieldTimePGPPublicKeyFieldIntegerPGPPublicKeyFieldBigIntegerPGPPublicKeyFieldDecimalPGPPublicKeyFieldFloatPGPPublicKeyFieldPublic key encryption creates a token generated with a public key to encrypt the data and a private key to decrypt it.
Public and private keys can be set in settings with PUBLIC_PGP_KEY and
PRIVATE_PGP_KEY.
Supported PGP symmetric key fields are:
CharPGPSymmetricKeyFieldEmailPGPSymmetricKeyFieldTextPGPSymmetricKeyFieldDatePGPSymmetricKeyFieldDateTimePGPSymmetricKeyFieldTimePGPSymmetricKeyFieldIntegerPGPSymmetricKeyFieldBigIntegerPGPSymerticKeyFieldDecimalPGPSymmetricKeyFieldFloatPGPSymmetricKeyFieldEncrypt and decrypt the data with settings.PGCRYPTO_KEY which acts like a password.
| Django Field | Public Key Field | Symmetric Key Field |
|---|---|---|
CharField | CharPGPPublicKeyField | CharPGPSymmetricKeyField |
EmailField | EmailPGPPublicKeyField | EmailPGPSymmetricKeyField |
TextField | TextPGPPublicKeyField | TextPGPSymmetricKeyField |
DateField | DatePGPPublicKeyField | DatePGPSymmetricKeyField |
DateTimeField | DateTimePGPPublicKeyField | DateTimePGPSymmetricKeyField |
TimeField | TimePGPPublicKeyField | TimePGPSymmetricKeyField |
IntegerField | IntegerPGPPublicKeyField | IntegerPGPSymmetricKeyField |
BigIntegerField | BigIntegerPGPPublicKeyField | BigIntegerPGPSymmetricKeyField |
DecimalField | DecimalPGPPublicKeyField | DecimalPGPSymmetricKeyField |
FloatField | FloatPGPPublicKeyField | FloatPGPSymmetricKeyField |
Other Django model fields are not currently supported. Pull requests are welcomed.
from django.db import models
from pgcrypto import fields
class MyModel(models.Model):
digest_field = fields.TextDigestField()
digest_with_original_field = fields.TextDigestField(original='pgp_sym_field')
hmac_field = fields.TextHMACField()
hmac_with_original_field = fields.TextHMACField(original='pgp_sym_field')
email_pgp_pub_field = fields.EmailPGPPublicKeyField()
integer_pgp_pub_field = fields.IntegerPGPPublicKeyField()
pgp_pub_field = fields.TextPGPPublicKeyField()
date_pgp_pub_field = fields.DatePGPPublicKeyField()
datetime_pgp_pub_field = fields.DateTimePGPPublicKeyField()
time_pgp_pub_field = fields.TimePGPPublicKeyField()
decimal_pgp_pub_field = fields.DecimalPGPPublicKeyField()
float_pgp_pub_field = fields.FloatPGPPublicKeyField()
email_pgp_sym_field = fields.EmailPGPSymmetricKeyField()
integer_pgp_sym_field = fields.IntegerPGPSymmetricKeyField()
pgp_sym_field = fields.TextPGPSymmetricKeyField()
date_pgp_sym_field = fields.DatePGPSymmetricKeyField()
datetime_pgp_sym_field = fields.DateTimePGPSymmetricKeyField()
time_pgp_sym_field = fields.TimePGPSymmetricKeyField()
decimal_pgp_sym_field = fields.DecimalPGPSymmetricKeyField()
float_pgp_sym_field = fields.FloatPGPSymmetricKeyField()
Data is automatically encrypted when inserted into the database.
Example:
>>> MyModel.objects.create(value='Value to be encrypted...')
Hash fields can have hashes auto updated if you use the original attribute. This
attribute allows you to indicate another field name to base the hash value on.
from django.db import models
from pgcrypto import fields
class User(models.Model):
first_name = fields.TextPGPSymmetricKeyField(max_length=20, verbose_name='First Name')
first_name_hashed = fields.TextHMACField(original='first_name')
In the above example, if you specify the optional original attribute it would take the unencrypted value from the first_name model field as the input value to create the hash. If you did not specify an original attribute, the field would work as it does now and would remain backwards compatible.
When accessing the field name attribute on a model instance we are getting the decrypted value.
Example:
>>> # When using a PGP public key based encryption
>>> my_model = MyModel.objects.get()
>>> my_model.value
'Value decrypted'
Filtering encrypted values is now handled automatically as of 2.4.0. And aggregate
methods are not longer supported and have been removed from the library.
Also, auto-decryption is support for select_related() models.
from django.db import models
from pgcrypto import fields
class EncryptedFKModel(models.Model):
fk_pgp_sym_field = fields.TextPGPSymmetricKeyField(blank=True, null=True)
class EncryptedModel(models.Model):
pgp_sym_field = fields.TextPGPSymmetricKeyField(blank=True, null=True)
fk_model = models.ForeignKey(
EncryptedFKModel, blank=True, null=True, on_delete=models.CASCADE
)
Example:
>>> import EncryptedModel
>>> my_model = EncryptedModel.objects.get().select_releated('fk_model')
>>> my_model.pgp_sym_field
'Value decrypted'
>>> my_model.fk_model.fk_pgp_sym_field
'Value decrypted'
To filter hash based values we need to compare hashes. This is achieved by using
a __hash_of lookup.
Example:
>>> my_model = MyModel.objects.filter(digest_field__hash_of='value')
[<MyModel: MyModel object>]
>>> my_model = MyModel.objects.filter(hmac_field__hash_of='value')
[<MyModel: MyModel object>]
.distinct('encrypted_field_name')Due to a missing feature in the Django ORM, using distinct() on an encrypted field
does not work for Django 2.0.x and lower.
The normal distinct works on Django 2.1.x and higher:
items = EncryptedFKModel.objects.filter(
pgp_sym_field__startswith='P'
).only(
'id', 'pgp_sym_field', 'fk_model__fk_pgp_sym_field'
).distinct(
'pgp_sym_field'
)
Workaround for Django 2.0.x and lower:
from django.db import models
items = EncryptedFKModel.objects.filter(
pgp_sym_field__startswith='P'
).annotate(
_distinct=models.F('pgp_sym_field')
).only(
'id', 'pgp_sym_field', 'fk_model__fk_pgp_sym_field'
).distinct(
'_distinct'
)
This works because the annotated field is auto-decrypted by Django as a F field and that
field is used in the distinct().
Migrating existing fields into PGCrypto Fields is not performed by this library. You will need to migrate the data in a forwards migration or other means. The only migration that is supported except to create/activate the pgcrypto extension in Postgres.
Migrating data is complicated as there might be a few things to consider such as:
The library has no way of doing all these guesses or to make all these decisions.
If you need to migrate data from unencrypted fields to encrypted fields, three ways to solve it:
Option 1: No data is in the db
Option 2: No data in the table
Option 3: Migrating in a non-destructive way
The goal here is to be able to use to legacy field if something goes wrong.
Part 1:
Part 2:
Taken direction from the PostgreSQL documentation:
https://www.postgresql.org/docs/9.6/static/pgcrypto.html#AEN187024
All pgcrypto functions run inside the database server. That means that all the data and passwords move between pgcrypto and client applications in clear text. Thus you must:
If you cannot, then better do crypto inside client application.
The implementation does not resist side-channel attacks. For example, the time required for a pgcrypto decryption function to complete varies among ciphertexts of a given size.
aggregates, PGPManager and PGPAdmin as they are no longer neededget_or_create() and update_or_create() (#27)get_by_natural_key() (#23)only() and defer() as they were not supported with PGPManagerdistinct() (Django 2.1+ with workaround available for 2.0 and lower)__range lookup for Date / DateTime fields (#59)Django 1.8, 1.9, and 1.10 (#62)setup.py:
make file for release to use twineREADME.coveragerc into setup.cfg.gitignore fileFlake8 and pycodestyle
are incompatible with each other)README with better explanations of the fieldsThanks to @peterfarrell:
DatePGPSymmetricKeyField and DateTimePGPSymmetricKeyField
including support for serializing / deserializing django form fields.Django 1.7.Django 1.10.Django 1.9 to the travis matrix.pgcrypto_fields to pgcrypto.get_placeholder accepts a new argument compilerAggregateNote: these changes have been done for django > 1.8.0.
MaxLengthValidator from email fields.max_length on PGP fields.NULL values for:
TextDigestField;
TextHMACField;
EmailPGPPublicKeyField;
IntegerPGPPublicKeyField;
TextPGPPublicKeyField;
EmailPGPSymmetricKeyField.
IntegerPGPSymmetricKeyField.
TextPGPSymmetricKeyField.casting bug when sending negative values to integer fields.EmailPGPPublicKeyField and EmailPGPSymmetricKeyField.PGPPublicKeyField to TextPGPPublicKeyField;
PGPSymmetricKeyField to TextPGPSymmetricKeyField;
DigestField to TextDigestField;
HMACField to TextHMACField.IntegerPGPPublicKeyField;
IntegerPGPSymmetricKeyField.memoryview for PGP and keyed hash fields.EncryptedProxyField to select the correct item.PGPPublicKeyField and PGPSymmetricKeySQL decrypted values with
field's proxy _decrypted.DigestField and HMACField.DigestField, HMACField, PGPPublicKeyAggregate, PGPSymmetricKeyAggregate.FAQs
Encrypted fields for Django dealing with pgcrypto postgres extension.
We found that django-pgcrypto-fields demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 2 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
In this segment of the Risky Business podcast, Feross Aboukhadijeh and Patrick Gray discuss the challenges of tracking malware discovered in open source softare.
Research
Security News
A threat actor's playbook for exploiting the npm ecosystem was exposed on the dark web, detailing how to build a blockchain-powered botnet.
Security News
NVD’s backlog surpasses 20,000 CVEs as analysis slows and NIST announces new system updates to address ongoing delays.