Research
Security News
Malicious npm Package Targets Solana Developers and Hijacks Funds
A malicious npm package targets Solana developers, rerouting funds in 2% of transactions to a hardcoded address.
Thanks to freesvg.org for the logo assets
Upsert with pandas DataFrames (ON CONFLICT DO NOTHING
or ON CONFLICT DO UPDATE
) for PostgreSQL, MySQL, SQlite and potentially other databases behaving like SQlite (untested) with some additional optional features (see features). Upserting can be done with primary keys or unique keys.
Pangres also handles the creation of non-existing SQL tables and schemas.
aiosqlite
for SQlite, asyncpg
for PostgreSQL and aiomysql
for MySQLFor using pangres
together with sqlalchemy>=2.0
(sqlalchemy is one of pangres dependencies
listed in requirements.txt) - you will need the following base requirements:
alembic>=1.7.2
pandas>=1.4.0
pandas>=1.4.0
only supports Python >=3.8)For using asynchronous engines (such as aiosqlite
, asyncpg
or aiomysql
) you will need Python >= 3.8.
This is a library I was using in production in private with very good results and decided to publish.
Ideally such features will be integrated into pandas since there is already a PR on the way and I would like to give the option to add columns via another PR.
There is also pandabase which does almost the same thing (plus lots of extra features) but my implementation is different. Btw big thanks to pandabase and the sql part of pandas which helped a lot.
pip install pangres
Additionally depending on which database you want to work with you will need to install the corresponding library (note that SQlite is included in the standard library):
pip install psycopg2
pip install pymysql
pip install asyncpg
pip install aiomysql
pip install aiosqlite
Head over to pangres' wiki! Note that the wiki is also available locally under the wiki folder.
Note:
The wiki is generated with a command which uses my library npdoc_to_md.
It must be installed with pip install npdoc_to_md
and you will also need the extra dependency fire
which you
can install with pip install fire
. Replace $DESTINATION_FOLDER
with the folder of you choice in the command below:
npdoc-to-md render-folder ./wiki_templates $DESTINATION_FOLDER
Pull requests/issues are welcome.
I develop the library inside of Jupyter Lab using the jupytext extension.
I recommend using this extension for the best experience. It will split code blocks within modules in notebook cells and will allow interactive development.
If you wish you can also use the provided conda environment (see environment.yml
file) inside of Jupyter Lab/Notebook
thanks to nb_conda_kernels.
You can test one or multiple of the following SQL flavors (you will of course need a live database for this): PostgreSQL, SQlite or MySQL.
NOTE: in one of the tests of pangres
we will try to drop and then create a PostgreSQL schema called pangres_create_schema_test
. If the schema existed and was not empty an error will be raised.
Clone pangres then set your curent working directory to the root of the cloned repository folder. Then use the commands below. You will have to replace the following variables in those commands:
SQLITE_CONNECTION_STRING
: replace with a SQlite sqlalchemy connection string (e.g. "sqlite:///test.db")ASYNC_SQLITE_CONNECTION_STRING
: replace with an asynchronous SQlite sqlalchemy connection string (e.g. "sqlite+aiosqlite:///test.db")POSTGRES_CONNECTION_STRING
: replace with a Postgres sqlalchemy connection string (e.g. "postgres:///user:password@localhost:5432/database"). Specifying schema is optional for postgres (will default to public)ASYNC_POSTGRES_CONNECTION_STRING
: replace with an asynchronous Postgres sqlalchemy connection string (e.g. "postgres+asyncpg:///user:password@localhost:5432/database"). Specifying schema is optional for postgres (will default to public)MYSQL_CONNECTION_STRING
: replace with a MySQL sqlalchemy connection string (e.g. "mysql+pymysql:///user:password@localhost:3306/database")ASYNC_MYSQL_CONNECTION_STRING
: replace with an asynchronous MySQL sqlalchemy connection string (e.g. "mysql+aiomysql:///user:password@localhost:3306/database")PG_SCHEMA
(optional): schema for postgres (defaults to public)# 1. Create and activate the build environment
conda env create -f environment.yml
conda activate pangres-dev
# 2. Install pangres in editable mode (changes are reflected upon reimporting)
pip install -e .
# 3. Run pytest
# -s prints stdout
# -v prints test parameters
# --cov=./pangres shows coverage only for pangres
# --doctest-modules tests with doctest in all modules
# --benchmark-XXX : these are options for benchmarks tests (see https://pytest-benchmark.readthedocs.io/en/latest/usage.html)
pytest -s -v pangres --cov=pangres --doctest-modules --async_sqlite_conn=$ASYNC_SQLITE_CONNECTION_STRING --sqlite_conn=$SQLITE_CONNECTION_STRING --async_pg_conn=$ASYNC_POSTGRES_CONNECTION_STRING --pg_conn=$POSTGRES_CONNECTION_STRING --async_mysql_conn=$ASYNC_MYSQL_CONNECTION_STRING --mysql_conn=$MYSQL_CONNECTION_STRING --pg_schema=tests --benchmark-group-by=func,param:engine,param:nb_rows --benchmark-columns=min,max,mean,rounds --benchmark-sort=name --benchmark-name=short
Additionally, the following flags could be of interest for you:
-x
for stopping at the first failure--benchmark-only
for only testing benchmarks--benchmark-skip
for skipping benchmarksflake8 must run without errors for pipelines to succeed.
If you are not using the conda environment, you can install flake8 with: pip install flake8
.
To test flake8 locally you can simply execute this command:
flake8 .
FAQs
Postgres insert update with pandas DataFrames.
We found that pangres demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
A malicious npm package targets Solana developers, rerouting funds in 2% of transactions to a hardcoded address.
Security News
Research
Socket researchers have discovered malicious npm packages targeting crypto developers, stealing credentials and wallet data using spyware delivered through typosquats of popular cryptographic libraries.
Security News
Socket's package search now displays weekly downloads for npm packages, helping developers quickly assess popularity and make more informed decisions.