python-package-sync-tool
Advanced tools
PythonPackageSyncTool is a Python utility to "fix" requirements.txt.
It is used to make manipulation on requirements.txt that is produces by pip freeze
First of all it is better to install alex_ber_utils (this dependency is used in setup.py).
pip3 install -U alex_ber_utils
Than:
pip3 install -U python-package-sync-tool
In some directory create config.yml. Here is example
(https://github.com/alex-ber/alpine-anaconda3/blob/master/python_package_sync_tool_config.yml)
treeroot:
source: installed.txt
destination: rm.txt
mutual_exclusion: True
remove:
- conda
- conda-package-handling
- icc-rt
- intel-openmp
- mkl
- mkl-fft
- mkl-random
- mkl-service
- pycosat
- sip
- tbb
add:
# - graphviz==2.38.0
# - entrypoints==0.2.3
# - numpy==1.16.2
# - PyYAML==5.1
# - sasl==0.2.1
# - scipy==1.2.1
# - tables==3.4.4
# - terminado==0.8.2
# - tornado==5.1.1
# - Twisted==19.2.0
Create installed.txt with installed packages (in pip freeze format).
Now, you can run
python3 -m alexber.reqsync.data
You can also override parameters in the yaml file:
python3 -m alexber.reqsync.data --treeroot.add=some_new_package:1.0.0
shorthand
python3 -m alexber.reqsync.data --add=some_new_package:1.0.0
is also supported.
Of course, you can provide all parameters via CLI and not to use YAML file.
You can also supply --general.log (no shorthand here) parameter (you can do it also via config.yml).
There is some default log configuration (to write everything to stdout).
Note: config.yml now also supports format that alex-ber-utils
https://github.com/alex-ber/AlexBerUtils support for a long time.
Please, consult https://medium.com/analytics-vidhya/my-major-init-app-conf-module-1a5d9fb3998c for
full supported list of general parameters.
See documentation here
My parser module https://medium.com/analytics-vidhya/my-parser-module-429ed1457718
My ymlparsers module https://medium.com/analytics-vidhya/my-ymlparsers-module-88221edf16a6
My major init_app_conf module https://medium.com/analytics-vidhya/my-major-init-app-conf-module-1a5d9fb3998c
Note:
Semicolomn and not equal sign is used here due to Python limitaion of usage of equal sign in the value in the command line.
You can specified multiple packages using comma delimiter.
You can specify path to your config file using --config_file (or --general.config_file).
It can be absolute or relative.
Alternatively,
cd /opt/anaconda3/lib/python3.8/site-packages/alexber/reqsync/data/
Note: This is Path where you're actually install my utility, it can be different in your machine.
If you use venv it will look something like:
cd /opt/MyProject/venv/Lib/site-packages/alexber/reqsync
Alternatively you can create script file for yourself, named, say, driver.py:
#!/usr/bin/python3
import alexber.reqsync.app as app
if __name__ == "__main__":
app.main()
Than create file config.yml near your script (see data/config.yml) or provide all parameter using command line argruments. Use ':' in places where you should naturally write '==' (see explanation below).
Parameters 'source' and 'destination' are required. You should also provide (requirements) file for 'source'.
mutual_exclusion has default value True.
Now, type
python3 -m alexber.reqsync.data --add=some_new_package:1.0.0
or if you're using script (driver.py) go the directory with the script and type
./driver.py --add=some_new_package:1.0.0
or if you install my tool to Anaconda/Python/venv that has it's bin folder is in the Path you can run
python_package_sync_tool --add=some_new_package:1.0.0
or alternativley
you can run
reqsync --add=some_new_package:1.0.0
This will run quick check whether package is not in remove list. If it is, the utility will fail. You can override this
beahivor by supplying --mutual_exclusion=False.
Then, this will add some_new_package with version 1.0.0 to the requirements-dest.txt
Note:
Semicolomn and not equal sign is used here due to Python limitaion of usage of equal sign in the value in the command line.
You can specified multiple packages using comma delimiter.
You can specifiy path to your config file using --config_file (or --general.config_file).
It can be absolute or relative. If you're running using script (driver.py), that it can be relative to the directory
whether you put your script. If you're running as the module (python3 -m), it can be relative to
/opt/anaconda3/lib/python3.8/site-packages/alexber/reqsync/data/ (exact path can be different, see above).
You can supplied multiply packages by using comma:
python3 -m alexber.reqsync.data --add=some_new_package:1.0.0,another_new_package:2.0.0
or if you're using script (driver.py) go the directory with the script and type
./driver.py --add=some_new_package:1.0.0,another_new_package:2.0.0
python3 -m pip install -U https://github.com/alex-ber/PythonPackageSyncTool/archive/master.zip
Optionally installing tests requirements.
python3 -m pip install -U https://github.com/alex-ber/PythonPackageSyncTool/archive/master.zip#egg=python-package-sync-tool[tests]
Or explicitly:
wget https://github.com/alex-ber/PythonPackageSyncTool/archive/master.zip -O master.zip; unzip master.zip; rm master.zip
And then installing from source (see below).
python3 -m pip install . # only installs "required"
python3 -m pip install .[test] # installs dependencies for tests
alexberkovich/python_package_sync_tool:latest contains all python_package_sync_tool dependencies.
This Dockerfile is very simple, you can take relevant part for you and put them into your Dockerfile.
Alternatively, you can use it as base Docker image for your project and add/upgrade another dependencies as you need.
For example:
FROM alexberkovich/python_package_sync_tool:latest
COPY requirements.txt etc/requirements.txt
RUN set -ex && \
#latest pip,setuptools,wheel
pip install --upgrade pip setuptools wheel && \
pip install python_package_sync_tool
pip install -r etc/requirements.txt
CMD ["/bin/sh"]
#CMD tail -f /dev/null
where requirements.txt is requirements for your project.
From the directory with setup.py
python3 setup.py test #run all tests
pytest
PythonPackageSyncTool requires the following modules.
Python 3.7+
alex-ber-utils==0.6.2
alex-ber-utils[yml]==0.6.2
See https://github.com/alex-ber/AlexBerUtils
All notable changes to this project will be documented in this file.
#https://pypi.org/manage/project/python-package-sync-tool/releases/
REAMDE.md, requiremetes section.alexber.reqsync.utils.app_conf removed.app.py to config.yml file.config.yml now also supports format that alex-ber-utils
https://github.com/alex-ber/AlexBerUtils support for a long time.Please, consult https://medium.com/analytics-vidhya/my-major-init-app-conf-module-1a5d9fb3998c for
full supported list of general parameters.
See documentation here
My parser module https://medium.com/analytics-vidhya/my-parser-module-429ed1457718
My ymlparsers module https://medium.com/analytics-vidhya/my-ymlparsers-module-88221edf16a6
My major init_app_conf module https://medium.com/analytics-vidhya/my-major-init-app-conf-module-1a5d9fb3998c
Integrating Python’s logging and warnings packages [https://medium.com/analytics-vidhya/integrating-pythons-logging-and-warnings-packages-7ffd6f65e02d]
Making relative path to file to work [https://alex-ber.medium.com/making-relative-path-to-file-to-work-d5d0f1da67bf]
config.xml - to remove package was added mkl*-related packagages and anaconda-specific packages.
REAMDE.md was updated to reflect all changes above.
__init__.py file added to alexber.reqsync.data.FAQs
Small tool to sync package from different machines
We found that python-package-sync-tool demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Meet Socket at Black Hat & DEF CON 2025 for 1:1s, insider security talks at Allegiant Stadium, and a private dinner with top minds in software supply chain security.
Security News
CAI is a new open source AI framework that automates penetration testing tasks like scanning and exploitation up to 3,600× faster than humans.
Security News
Deno 2.4 brings back bundling, improves dependency updates and telemetry, and makes the runtime more practical for real-world JavaScript projects.