pgserver

pgserver: pip-installable, embedded postgres server + pgvector extension for your python app

pgserver lets you build Postgres-backed python apps with the same convenience afforded by an embedded database (ie, alternatives such as sqlite). If you build your app with pgserver, your app remains wholly pip-installable, saving your users from needing to understand how to setup a postgres server (they simply pip install your app, and postgres is brought in through dependencies), and letting you get started developing quickly: just pip install pgserver and pgserver.get_server(...), as shown in this notebook:

To achieve this, you need two things which pgserver provides

• python binary wheels for multiple-plaforms with postgres binaries
• convenience python methods that handle db initialization and server process management, that deals with things that would normally prevent you from running your python app seamlessly on environments like docker containers, a machine you have no root access in, machines with other running postgres servers, google colab, etc. One main goal of the project is robustness around this.

Additionally, this package includes the pgvector postgres extension, useful for storing associated vector data and for vector similarity queries.

Basic summary:

• Pip installable binaries: built and tested on Manylinux, MacOS and Windows.
• No sudo or admin rights needed: Does not require root privileges or sudo.
• but... can handle root: in some environments your python app runs as root, eg docker, google colab, pgserver handles this case.
• Simpler initialization: pgserver.get_server(MY_DATA_DIR) method to initialize data and server if needed, so you don't need to understand initdb, pg_ctl, port conflicts.
• Convenient cleanup: server process cleanup is done for you: when the process using pgserver ends, the server is shutdown, including when multiple independent processes call pgserver.get_server(MY_DATA_DIR) on the same dir (wait for last one). You can blow away your PGDATA dir and start again.
• For lower-level control, wrappers to all binaries, such as initdb, pg_ctl, psql, pg_config. Includes header files in case you wish to build some other extension and use it against these binaries.

# Example 1: postgres backed application
import pgserver

db = pgserver.get_server(MYPGDATA)
# server ready for connection.

print(db.psql('create extension vector'))
db_uri = db.get_uri()
# use uri with sqlalchemy / psycopg, etc, see colab.

# if no other process is using this server, it will be shutdown at exit,
# if other process use same pgadata, server process will be shutdown when all stop.

# Example 2: Testing
import tempfile
import pytest
@pytest.fixture
def tmp_postgres():
    tmp_pg_data = tempfile.mkdtemp()
    pg = pgserver.get_server(tmp_pg_data, cleanup_mode='stop')
    yield pg
    pg.cleanup()

Postgres binaries in the package can be found in the directory pointed to by the pgserver.POSTGRES_BIN_PATH to be used directly.

This project was originally based on , which provides a linux wheel. But adds the following differences:

1. binary wheels for multiple platforms (ubuntu x86, MacOS apple silicon, MacOS x86, Windows)
2. postgres python management: cross-platfurm startup and cleanup including many edge cases, runs on colab etc.
3. includes pgvector extension but currently excludes postGIS