poetry-multiproject-plugin

Poetry Multiproject Plugin

This is a Python Poetry plugin, adding the build-project and check-project commands.

The build-project command will make it possible to use relative package includes. This feature is very useful for monorepos and when sharing code between projects.

The check-project command is useful to check that dependencies are added properly in a project. It uses the MyPy tool under the hood, and will output any errors from the static type checker.

• Use cases
• Usage
• Installation
• What does it do?
• How is it different from the "poetry build" command?
• Organizing code

Use cases

Microservices and apps

The main use case is to support having one or more microservices or apps in a Monorepo, and share code between the services with namespaced packages. The build-project command will collect the project-specific packages and build an installable artifact from it (i.e. a wheel or an sdist).

A basic building block for the Polylith Architecture

The Multiproject plugin makes it possible to organize Python projects according to the Polylith Architecture. The plugin is the foundation for the Python tools for the Polylith Architecture - also implemented as a Poetry plugin.

For more about Polylith, have a look at the Python-specific Polylith documentation.

Libraries?

Building libraries is also supported, but you will need to consider that the code will likely share the same top namespace with other libraries built from the same monorepo. It depends on your monorepo structure. This will likely be a problem when more than one of your libraries are installed into the same virtual environment.

Since Python libraries by default are installed in a "flat" folder structure, two libraries with the same top namespace will collide.

There is a way to solve this issue, by using the --with-top-namespace flag of the build-project command. See usage for libraries.

Usage

Navigate to the project folder (where the pyproject.toml file is).

Build a project:

poetry build-project

Check the code used in a project:

poetry check-project

Check the code, with a custom MyPy configuration to override the defaults:

poetry check-project --config-file <PATH-TO-MYPY.INI-CONFIG-FILE>

Available command options

Custom temp path

A custom temporary path to use for reading, writing and deleting project content during the project build. This option is useful for environments with restrictions on where scripts are allowed to store content.

NOTE: The default temp path will be created as a sibling to the project to build, and is the recommended way in most cases.

poetry build-project --custom-temp-path /tmp
poetry check-project --custom-temp-path /tmp

Usage for libraries

The build-project has a solution to the problem with top namespaces in libraries for Python 3.9 and more. You can choose a custom namespace to be used in the build process, by using the --with-top-namespace flag.

The command will organize the namespaced packages according to the custom top namespace, and more importantly, re-write the imports made in the actual source code. The re-organizing and re-writing is performed on the relative includes.

The build-project command, with a custom top namespace:

poetry build-project --with-top-namespace my_namespace

The build output

Default(no flag)

/my_package
   __init__.py
   my_module.py

Namespace(--with-top-namespace=my_namespace)

my_namespace/
    /my_package
       __init__.py
       my_module.py

Namespace with path(--with-top-namespace=my_namespace/subdir)

my_namespace/
    /subdir
        /my_package
           __init__.py
           my_module.py

And will re-write the relevant module(s):

Default(no flag)

from my_package import my_function

Namespace(--with-top-namespace=my_namespace)

from my_namespace.my_package import my_function

Namespace with path(--with-top-namespace=my_namespace/subdir)

from my_namespace.subdir.my_package import my_function

How is this done?

The code in this repo uses AST (Abstract Syntax Tree) parsing to modify source code. The Python built-in ast module is used to parse and un-parse Python code.

Installation

This plugin can be installed according to the official Poetry docs.

poetry self add poetry-multiproject-plugin

What does it do?

the poetry build-project command will:

1. copy the actual project into a temporary folder.
2. collect relative includes - such as include = "foo/bar", from = "../../shared" - and copy them into the temprary folder.
3. generate a new pyproject.toml.
4. run the poetry build command in the temporary folder.
5. copy the built dist folder (containing the wheel and sdist) into the actual project folder.
6. remove the temporary folder.

the poetry check-project command will:

1. copy the actual project into a temporary folder.
2. collect relative includes - such as include = "foo/bar", from = "../../shared" - and copy them into the temprary folder.
3. generate a new pyproject.toml.
4. run poetry install in the temporary folder.
5. run poetry run mypy in the temporary folder.
6. remove the temporary folder.

The default setting for the underlying MyPy configuration is:

--explicit-package-bases --namespace-packages --no-error-summary --no-color-output

How is it different from the "poetry build" command?

Poetry does not allow package includes outside of the project root.

# Note the structure of the shared folder: namespace/package

packages = [
    { include = "my_namespace/my_package", from = "../../shared" }
    { include = "my_namespace/my_other_package", from = "../../shared" }
]

This plugin will allow relative package includes. You will now be able to share code between projects.

Organizing code

An example Monorepo structure, having the shared code extracted into a separate folder structure:

projects/
  my_app/
    pyproject.toml (including selected shared packages)

  my_service/
    pyproject.toml (including selected shared packages)

shared/
  my_namespace/
    my_package/
      __init__.py
      code.py

    my_other_package/
      __init__.py
      code.py

A suggested structure, using Polylith:

workspace/
  bases/
  components/
  development/
  projects/

  poetry.lock

  pyproject.toml
  workspace.toml

  README.md