rolling-pin

Introduction

Rolling-pin is a library of generic tools for ETL work and visualization of JSON blobs and python repositories

See documentation for details.

See blob notebook for a demo of the BlobETL class.

See conform notebook for a demo of the ConformETL class.

See cli notebook for a demo of the rolling-pin CLI.

On the documentation main page, under the Architecture section, is a dynamically generated dependency graph of rolling-pin's current architecture. It is generated using the RepoETL class.

If you look under the Metrics section you will find Radon code metric plots and data of the rolling-pin source code, generated by the RadonETL class.

Installation for Developers

Docker

• Install docker-desktop
• Ensure docker-desktop has at least 4 GB of memory allocated to it.
• git clone [e-mail]:theNewFlesh/rolling-pin.git
• cd rolling-pin
• chmod +x bin/rolling-pin
• bin/rolling-pin docker-start
  • If building on a M1 Mac run export DOCKER_DEFAULT_PLATFORM=linux/amd64 first.

The service should take a few minutes to start up.

Run bin/rolling-pin --help for more help on the command line tool.

ZSH Setup

• bin/rolling-pin must be run from this repository's top level directory.

• Therefore, if using zsh, it is recommended that you paste the following line in your ~/.zshrc file:

  • alias rolling-pin="cd [parent dir]/rolling-pin; bin/rolling-pin"
  • Replace [parent dir] with the parent directory of this repository

• Consider adding the following line to your ~/.zshrc if you are using a M1 Mac:

  • export DOCKER_DEFAULT_PLATFORM=linux/amd64

• Running the zsh-complete command will enable tab completions of the cli commands, in the next shell session.

  For example:

  • rolling-pin [tab] will show you all the cli options, which you can press tab to cycle through
  • rolling-pin docker-[tab] will show you only the cli options that begin with "docker-"

Installation for Production

Python

pip install rolling-pin

Please see the prod.dockerfile for an official example of how to build a docker image with rolling-pin.

Docker

• Install docker-desktop
• docker pull theNewFlesh/rolling-pin:[mode]-[version]

Quickstart Guide

This repository contains a suite commands for the whole development process. This includes everything from testing, to documentation generation and publishing pip packages.

These commands can be accessed through:

• The VSCode task runner
• The VSCode task runner side bar
• A terminal running on the host OS
• A terminal within this repositories docker container

Running the zsh-complete command will enable tab completions of the CLI. See the zsh setup section for more information.

Command Groups

Development commands are grouped by one of 10 prefixes:

Command	Description
build	Commands for building packages for testing and pip publishing
docker	Common docker commands such as build, start and stop
docs	Commands for generating documentation and code metrics
library	Commands for managing python package dependencies
session	Commands for starting interactive sessions such as jupyter lab and python
state	Command to display the current state of the repo and container
test	Commands for running tests, linter and type annotations
version	Commands for bumping project versions
quickstart	Display this quickstart guide
zsh	Commands for running a zsh session in the container and generating zsh completions

Common Commands

Here are some frequently used commands to get you started:

Command	Description
docker-restart	Restart container
docker-start	Start container
docker-stop	Stop container
docs-full	Generate documentation, coverage report, diagram and code
library-add	Add a given package to a given dependency group
library-graph-dev	Graph dependencies in dev environment
library-remove	Remove a given package from a given dependency group
library-search	Search for pip packages
library-update	Update dev dependencies
session-lab	Run jupyter lab server
state	State of
test-dev	Run all tests
test-lint	Run linting and type checking
zsh	Run ZSH session inside container
zsh-complete	Generate ZSH completion script

Development CLI

bin/rolling-pin is a command line interface (defined in cli.py) that works with any version of python 2.7 and above, as it has no dependencies. Commands generally do not expect any arguments or flags.

Its usage pattern is: bin/rolling-pin COMMAND [-a --args]=ARGS [-h --help] [--dryrun]

Commands

The following is a complete list of all available development commands:

Command	Description
build-edit-prod-dockerfile	Edit prod.dockefile to use local package
build-local-package	Generate local pip package in docker/dist
build-package	Build production version of repo for publishing
build-prod	Publish pip package of repo to PyPi
build-publish	Run production tests first then publish pip package of repo to PyPi
build-test	Build test version of repo for prod testing
docker-build	Build development image
docker-build-from-cache	Build development image from registry cache
docker-build-no-cache	Build development image without cache
docker-build-prod	Build production image
docker-build-prod-no-cache	Build production image without cache
docker-container	Display the Docker container id
docker-destroy	Shutdown container and destroy its image
docker-destroy-prod	Shutdown production container and destroy its image
docker-image	Display the Docker image id
docker-prod	Start production container
docker-pull-dev	Pull development image from Docker registry
docker-pull-prod	Pull production image from Docker registry
docker-push-dev	Push development image to Docker registry
docker-push-dev-latest	Push development image to Docker registry with dev-latest tag
docker-push-prod	Push production image to Docker registry
docker-push-prod-latest	Push production image to Docker registry with prod-latest tag
docker-remove	Remove Docker image
docker-restart	Restart container
docker-start	Start container
docker-stop	Stop container
docs	Generate sphinx documentation
docs-architecture	Generate architecture.svg diagram from all import statements
docs-full	Generate documentation, coverage report, diagram and code
docs-metrics	Generate code metrics report, plots and tables
library-add	Add a given package to a given dependency group
library-graph-dev	Graph dependencies in dev environment
library-graph-prod	Graph dependencies in prod environment
library-install-dev	Install all dependencies into dev environment
library-install-prod	Install all dependencies into prod environment
library-list-dev	List packages in dev environment
library-list-prod	List packages in prod environment
library-lock-dev	Resolve dev.lock file
library-lock-prod	Resolve prod.lock file
library-remove	Remove a given package from a given dependency group
library-search	Search for pip packages
library-sync-dev	Sync dev environment with packages listed in dev.lock
library-sync-prod	Sync prod environment with packages listed in prod.lock
library-update	Update dev dependencies
library-update-pdm	Update PDM
quickstart	Display quickstart guide
session-lab	Run jupyter lab server
session-python	Run python session with dev dependencies
state	State of repository and Docker container
test-coverage	Generate test coverage report
test-dev	Run all tests
test-fast	Test all code excepts tests marked with SKIP_SLOWS_TESTS decorator
test-format	Format all python files
test-lint	Run linting and type checking
test-prod	Run tests across all support python versions
version	Full resolution of repo: dependencies, linting, tests, docs, etc
version-bump-major	Bump pyproject major version
version-bump-minor	Bump pyproject minor version
version-bump-patch	Bump pyproject patch version
version-commit	Tag with version and commit changes to master
zsh	Run ZSH session inside Docker container
zsh-complete	Generate oh-my-zsh completions
zsh-root	Run ZSH session as root inside Docker container

Flags

Short	Long	Description
-a	--args	Additional arguments, this can generally be ignored
-h	--help	Prints command help message to stdout
	--dryrun	Prints command that would otherwise be run to stdout

Production CLI

Rolling-pin comes with a command line interface defined in command.py.

Its usage pattern is: rolling-pin COMMAND [ARGS] [FLAGS] [-h --help]

Commands

bash-completion

Prints BASH completion code to be written to a _rolling-pin completion file

Usage: rolling-pin bash-completion

zsh-completion

Prints ZSH completion code to be written to a _rolling-pin completion file

Usage: rolling-pin zsh-completion

conform

Copies source files to target filepaths according to given conform file.

Usage: rolling-pin conform [OPTIONS] SOURCE

Argument	Description
source	conform YAML filepath

Flag	Argument	Description	Default
--groups	text	comma separated list of groups to be conformed	all
--dryrun		Print out conform table instead of run conform
--help		print help message

graph

Generate a dependency graph of a source repository and write it to a given filepath

Usage: rolling-pin graph [OPTIONS] SOURCE TARGET

Argument	Description
source	repository path
target	target filepath

Flag	Argument	Description	Default
--include	text	include files that match this regex pattern	.*.py$'
--exclude	text	exclude files that match this regex pattern	test
--orient	text	graph orientation	lr
--help		print help message

plot

Write radon metrics plots of given repository to given filepath.

Usage: rolling-pin plot [OPTIONS] SOURCE TARGET

Argument	Description
source	repository path
target	plot filepath

Flag	Description
--help	print help message

table

Write radon metrics tables of given repository to given directory.

Usage: rolling-pin table [OPTIONS] SOURCE TARGET

Argument	Description
source	repository path
target	table directory

Flag	Description
--help	print help message

toml

Generate a copy of a given TOML file with given edits indicated by flags. Flags are evalauted in the following order: edit, delete, search. Flags may be arbitrarily combined. Edit and delete flags may appear multiple times.

Usage: rolling-pin toml [OPTIONS] SOURCE

Argument	Description
source	TOML filepath

Flag	Argument	Description
--edit	text	replace key's value with given value, text is "=" separated key value pair in TOML format
--delete	text	delete keys that match this regular expression
--search	text	search for keys that match this regular expression
--target	text	target filepath to write to
--help		print help message

Example Usage

example file

>>>cat example.toml
[root]
a = 1
b = 2

[root.foo.bar]
x = "y"

[world]
hello = true

edit flag

>>>rolling-pin toml foobar.toml --edit 'root.a=999'
[root]
a = 999
b = 2...

>>>rolling-pin toml foobar.toml \
        --edit 'root.a=[1, 2]'   \
        --edit 'root.b="xxx"'
[root]
a = [
    1,
    2,
]
b = "xxx"...

>>>rolling-pin toml foobar.toml --edit 'root.foo.bar="baz"'
...
hello = true

[root.foo]
bar = "baz"...

delete flag

>>>rolling-pin toml foobar.toml \
        --delete 'root.foo.bar'  \
        --delete 'root.a'
[root]
b = 2

[world]
hello = true

search flag

>>>rolling-pin toml foobar.toml --search 'root.foo|world'
[world]
hello = true

[root.foo.bar]
x = "y"