Enable to work with Python projects containing lots of packages, of which you only want to develop some.
mxdev [mɪks dɛv] is a utility that makes it easy to work with Python projects containing lots of packages, of which you only want to develop some.
It builds on top of the idea to have stable version constraints and then develop from a VCS on top of it.
As part of the above use-case sometimes versions of the stable constraints need an override with a different (i.e. newer) version. Other software follow the same idea are mr.developer for Python's zc.buildout or mrs-developer for NPM packages.
mxdev procedure is:
mxdev will not run pip for you!
Given a requirements.txt (or similar named) file which itself references a constraints.txt file inside.
Create an INI file, like mx.ini in configparser.ExtendedInterpolation syntax.
[settings]The main section must be called [settings], even if kept empty.
In the main sections the input and output files are defined.
requirements-inMain requirements file to start with. This can be an URL too.
If given an empty value mxdev will only generate output from the information given in INI file itself.
Default: requirements.txt
requirements-outOutput of the combined requirements including development sources to be used later with pip install.
Default: requirements-mxdev.txt
constraints-outOutput of the combined constraints.
Default: constraints-mxdev.txt
includeInclude one or more other INI files.
The included file is read before the main file, so the main file overrides included settings. Included files may include other files. Innermost inclusions are read first.
If an included file is an HTTP-URL, it is loaded from there.
If the included file is a relative path, it is loaded relative to the parents directory or URL.
The feature utilizes the ConfigParser feature to read multiple files at once.
Default: empty
default-targetTarget directory for sources from VCS. Default: ./sources
default-install-modeDefault for install-mode on section, read there for details
Allowed values: direct or skip
Default: direct
default-updateDefault for update on section, read there for details
Allowed values: yes or no
Default: yes
threadsNumber of threads to fetch sources in parallel with. Speeds up fetching from VCS.
Default: 4
offlineDo not fetch any sources. Handy if working offline.
Default: False
version-overridesOverride package versions which are already defined in a dependent constraints file.
I.e. an upstream constraints.txt contains already somefancypackage==2.0.3.
Given that, for some reason (like with my further developed sources), we need version 3.0.0 of the above package.
Then in this section, this can be defined as:
[settings]
version-overrides =
somefancypackage==3.0.0
otherpackage==33.12.1
It is possible to add as many overrides as needed. When writing the constraints-out, the new version will be taken into account. If there is a source section defined for the same package, the source will be used and entries here are ignored.
Note: When using uv pip install the version overrides here are not needed, since it supports overrides nativly.
With uv it is recommended to create an overrides.txt file with the version overrides and use uv pip install --overrides overrides.txt [..] to install the packages.
ignoresIgnore packages that are already defined in a dependent constraints file.
No new version will be provided.
This is specifically handy if a package is going to be installed editable from local file system (like -e .), but was already pinned in an upstream constraints file.
This can be defined as:
[settings]
ignores =
somefancypackage
otherpackage
default-useTrue by default. When false, the source is not checked out,
and the version for this package is not overridden.
Additionally, custom variables can be defined as key = value pair.
Those can be referenced in other values as ${settings:key} and will be expanded there.
main-packagemxdev can handle one Python package as main package directly via ini config. If defined, it will be added as last entry in the resulting requirements out file.
This can be defined as:
[settings]
main-package = -e .[test]
If the main package is defined in a dependent constraint file, it's name must be added to ignores.
mxdev provides default settings which can be used inside package or custom sections.
directoryContains the current working directory and can be used like this
[sectionname]
param = ${settings:directory}/some/path
All other sections are defining the sources to be used.
[PACKAGENAME]The section name is the package name.
url = URLThe checkout URL of the repository.
The URL is required.
pushurl = URLOptional a writable URL for pushes can be specified.
If the pushurl is set after initial checkout it is not applied.
To apply it remove the repository and checkout again.
vcs = VCSThe version control system to use.
Supported are:
git (stable, tested)fs (stable, tested) - in fact no vcs, but points to a local directory.
This can be achieved without mxdev by using -e PATH in the requirements input file.svn (unstable, test needs rewrite)gitsvn (unstable, test needs rewrite)hg (unstable, test needs rewrite)bzr (unstable, test needs rewrite)darcs (unstable, test needs rewrite)Defaults to git.
branch = BRANCHNAME_OR_TAGThe branch name or tag to checkout.
Defaults to main.
extras = EXTRA1,EXTRA2Package extras to install. Default empty.
subdirectory = SUBPATHFor specifying the path to the Python package, when it is not in the root of the VCS directory.
Default empty.
targetThe target directory for source from this section.
Default to default target directory configured in the main section [settings] default-target = value.
install-modeThere are different modes of pip installation
skipDo not install with pip, just clone/update the repository
directInstall the package using pip -e PACKAGEPATH.
Dependencies are resolved immediately
Defaults to default mode configured in main section [settings] default-install-mode = value.
useTrue by default, unless default-use in the general settings is false.
When false, the source is not checked out,
and the version for this package is not overridden.
submodulesThere are 3 different options:
always(default) git submodules will always be checked out, they will be updated if already presen
checkoutsubmodules get only fetched during checkout, existing submodules stay untouche
recursiveFetches submodules recursively, results in git clone --recurse-submodules on checkout and submodule update --init --recursive on update.
Run mxdev (for more options run mxdev --help).
Mxdev will
mx.ini,Now, use the generated requirements and constraints files with i.e. pip install -r requirements-mxdev.txt.
mxdev >=4.0 needs pip version 23 at minimum to work properly
mx.iniThis looks like so:
[settings]
requirements-in = requirements.txt
requirements-out = requirements-mxdev.txt
contraints-out = constraints-mxdev.txt
version-overrides =
baz.baaz = 1.9.32
ignores =
my.ignoredpackage
# custom variables
github = git+ssh://git@github.com/
mygit = git+ssh://git@git.kup.tirol/
[foo.bar]
url = ${settings:github}orga/foo.bar.git
branch = fix99
extras = test,baz
[kup.fancyproject]
url = ${settings:mygit}customers/fancycorp/kup.fancyproject.git
branch = fix99
extras = test,baz
The functionality of mxdev can be extended by hooks. This is useful to generate additional scripts or files or automate any other setup steps related to mxdev's domain.
Extension configuration settings end up in the mx.ini file.
They can be added globally to the settings section, as dedicated config sections or package specific.
To avoid naming conflicts, all hook-related settings and config sections must be prefixed with a namespace.
It is recommended to use the package name containing the hook as a namespace.
This looks like so:
[settings]
myextension-global_setting = 1
[myextension-section]
setting = value
[foo.bar]
myextension-package_setting = 1
The extension is implemented as a subclass of mxdev.Hook:
from mxdev import Hook
from mxdev import State
class MyExtension(Hook):
namespace = None
"""The namespace for this hook."""
def read(self, state: State) -> None:
"""Gets executed after mxdev read operation."""
def write(self, state: State) -> None:
"""Gets executed after mxdev write operation."""
The default settings section from the INI file is available at state.configuration.settings.
The package configuration is available at state.configuration.packages.
Hook-related config sections are available at state.configuration.hooks.
The hook must be registered as an entry point in the pyproject.toml of your package:
[project.entry-points.mxdev]
hook = "myextension:MyExtension"
There is a constraint file like -c constraints.txt with a package foo.bar with a version pin.
Then it is not possible to install this package in a requirements file editable like -r requirements.txt with -e git+ssh://git@github.com/orga/foo.bar.git@fix-99.
Neither it is possible to override inherited version constraints with custom ones.
A pre-processor fetches (as this can be an URL) and expands all -c SOMEOTHER_FILE_OR_URL and -r SOMEOTHER_FILE_OR_URL files into one, filtering out all packages given in a configuration file.
For each of those packages, a -e ... entry is generated instead and written to a new TARGET.txt.
Same is true for version overrides: a new entry is written to the resulting constraints file while the original version is disabled.
The configuration is read from a file mx.ini in ExtendedInterpolation INI syntax (YAML would be nice, but the package must have as less dependencies as possible to other packages).
Mx (generally pronounced like mix [mɪks], or [məks] in the UK) is meant to be a gender-neutral alternative to the titles Mr. and Ms. but also associates with the word "mix".
The VCS-related code is taken from mr.developer.
Thanks to Florian Schulze and Contributors.
If you want to help with the development (improvement, update, bug-fixing, ...) of mxdev this is a great idea!
The code is located in the GitHub MXStack Organization / mxdev.
You can fork it, work on the project and create a pull request.
Maintainers are Jens Klein and the BlueDynamics Alliance developer team.
We appreciate any contribution! If you have an idea, found a bug, want to drop us a question, or a release is needed, please just file an issue at the mxdev issue tracker.
pkg_resoures to load entry points and parse requirements.
This enables mxdev to work on Python 3.12, where pkg_resources is no longer installed by default in virtual_envs.
[jensens]Breaking: Remove --pre on sources from generated requirements-mxdev.txt.
Usually it is not needed any longer, at least withy pip 23.x.
This is a breaking change if you rely on the --pre option being present in the generated file.
Now the --pre option should be added to pip install when the generated file is used.
This change enables the use of the generated file with the alternative pip replacement uv.
[jensens]
Breaking: Drop official support for Python 3.7 (it is end of life). [jensens]
Document mx.ini sections vcs setting.
[jensens]
directory default setting [rnix]Fix usage of --install-option='pre' and use --pre option in requirements files instead.
The install options are deprecated in pip 23 which Plone switched to recently.
More info:
https://github.com/pypa/pip/issues/11358
https://discuss.python.org/t/passing-command-line-arguments-to-pip-install-after-install-options-deprecation/22981/6
[thet, fredvd]
Fix reading sections from the config parser without defaults if the section contains a setting that also exists as default. [rnix]
Do not write constraints out to the file if no constraints are defined. [rnix]
Add the main-package option to the settings.
[rnix]
In this package, use pyproject.toml and markdown for README et al.
[jensens]
Add use option to sources, and default-use to the settings.
default-use is true by default.
When false, the source is not checked out, and the version for this package is not overridden.
[maurits]
Do not use libvcs, but recycled and updated (type hints, tests) mr.developer VCS code.
Code for GIT is tested well, code for SVN, Mercurial, Bazaar and DARCS needs contributors with knowledge in this area.
Additional options, like pushurl, ... (see README) were added.
pip style VCS URLs are not supported any longer.
[jensens, rnix, zworkb]
Config parser options are now considered case-sensitive. [rnix]
Do not fail mxdev run if requirements.txt is missing.
[rnix]
Add flag to only fetch repositories and skip generating files. [rnix]
Add flag to skip fetching of repositories. [rnix]
Add support for custom hooks. [rnix]
Rename sources.ini to mx.ini in the documentation.
[rnix]
Introduce state object and pass it to read/fetch/write. State object contains all required runtime data. [rnix]
Depend on pip 22, where interdependency mode is no longer needed. Remove all interdependency-related code. [jensens]
Better error message if the requirements-in file does not exist. [jensens]
Better last message with the full pip command. [jensens]
Allow empty requirements-in configuration.
[jensens]
ignores with a list of packages (one per line) to ignore constraints without providing a version.
[jensens]--pre option in order to allow the other non-final/in-development release.
[jensens]*-mxdev.txt now.
[jensens]Add -s or --silent option.
[jensens]
Beautified output. [jensens]
Fixed missing CR if *.txt does not end with a newline.
[jensens]
sources.ini.
[zworkb]Added interdependency handling to avoid manual dependency order resolution. [jensens, gogobd]
Added skip mode to exclude packages from installation (clone/update only). [jensens, gogobd]
Removed position feature. [jensens, gogobd]
Fix/simplify packaging. [jensens]
Implement subdirectory editable install [jensens]
Implement package extras [jensens]
Copyright (c) 2022-2023, mxstack Contributors
All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
FAQs
Enable to work with Python projects containing lots of packages, of which you only want to develop some.
We found that mxdev demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 2 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Research
The Socket Research Team breaks down a malicious wrapper package that uses obfuscation to harvest credentials and exfiltrate sensitive data.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.
Security News
The Ultralytics' PyPI Package was compromised four times in one weekend through GitHub Actions cache poisoning and failure to rotate previously compromised API tokens.