tailucas-pylib

About The Project

Overview

This package was created by factoring out many reusable code artifacts from my various projects over a number of years. Since this work was not a part of a group effort, the test coverage is predictably abysmal :raised_eyebrow: and Python documentation notably absent :expressionless:. For each of my projects, which derive from a common Docker application found here, this package provides some specific and relatively powerful features to enable rapid offshoots for new ideas. Here is what functionality this package provides:

• aws.init.py: If AWS environment variables are set, global boto and boto3 session objects are instantiated.
• aws.metrics: CloudWatch metric helper.
• aws.swf: Client-side interface to botoflow with interfaces specific to a number of prior projects. Examples of this use are in respective swf branches of some projects. This mechanism has been since replaced with message passing via RabbitMQ.
• init.py: A lot of module bootstrap with a pitiful attempt to support unit testing. At application startup, a choice is made between logging to a configured logger or to the terminal if one exists, to support a scenario where the application is run interactively. A standard application configuration file is loaded with a Python ConfigParser and then a credentials client is loaded based on 1Password. The choice of 1Password is motivated by their secrets automation features and because I happened to use their service already. I have been tempted to try Bitwarden also but there has not been a need yet to create this abstraction. Next, the Sentry client is loaded with whatever extras the given application might specify. Some useful globals are stored within config.py.
• app: A few of my applications use ZeroMQ for "lockless" IPCs between Python threads and so it became apparent that some helpers were needed to support common patterns. ZmqRelay is one such example where an application thread can be used to receive some message on which processing is done, and then forward it to another ZeroMQ channel. The main thread logic is contained with a context manager which handles failures and shutdown gracefully.
• app.bluetooth: Bluetooth helper functions that make use of hcitool l2ping.
• app.creds Credential helper that provides a consistent interface between the 1Password Connect Server or 1Password Service Account modes, depending on your preference.
• app.data: Simple utility to make a common IPC or network payload using MessagePack.
• app.datetime: Date and timestamp manipulation with time zone normalization.
• app.handler: A useful Python context manager to handle a variety of failure conditions.
• app.process: Simple signal handler with interaction with application shutdown tracking.
• app.rabbit: A fairly good example of a RabbitMQ application framework with internal IPC handoff to ZeroMQ channels for inter-thread communication.
• app.threads: A useful thread nanny with shutdown debugging and metric support.
• app.zmq: Useful helper functions for a ZeroMQ enabled application.

Handy stand-alone tools:

• config_interpol: By making creative abuse of Python's ConfigParser, this tool designed for the command-line will take a configuration file with variables that are automatically substituted with with either an overlay configuration or environment variables by the same name and output the interpolated configuration. A good example of this tool being used is here.
• cred_tool: Useful to fetch an item from 1Password.
• yaml_interpol: A script useful to generate docker-compose YAML output from templates. A crude example is here which fetches application configuration from a 1Password vault. Note that application runtime secrets are only in memory and loaded in init.py. I happen to use 1Password to also store application key-value pairs for use in docker-compose templates.

Package Structure

The package is now organized under src/tailucas_pylib/ with the following key modules:

• Core Modules:

  • __init__.py: Bootstrap module with logging, configuration, and credential setup
  • config.py: Global configuration management
  • creds.py: 1Password credential management with both Connect and Service Account support

• Application Framework:

  • app.py: Thread management with ZMQ relay and worker patterns
  • threads.py: Thread monitoring and lifecycle management
  • handler.py: Exception handling context manager
  • process.py: Signal handling and process utilities

• Communication:

  • rabbit.py: RabbitMQ integration with ZMQ bridging
  • zmq.py: ZeroMQ socket management and utilities

• Utilities:

  • data.py: MessagePack payload creation
  • datetime.py: Timezone-aware datetime utilities
  • device.py: Pydantic device model
  • bluetooth.py: Bluetooth device detection via hcitool

• AWS Integration:

  • aws/__init__.py: Boto3 session management
  • aws/metrics.py: CloudWatch metrics publishing
  • aws/swf.py: Simple Workflow Service integration (deprecated)

• Command-line Tools:

  • tools/config_interpol.py: Configuration interpolation
  • tools/cred_tool.py: Credential retrieval utility
  • tools/yaml_interpol.py: YAML template processing

(back to top)

Built With

Technologies that help make this package useful:

Also:

• Cronitor
• MessagePack

• Botoflow

Core Technologies:

• Python 3.8+ - Primary runtime
• 1Password Connect/Service Account - Credential management
• ZeroMQ - Inter-process communication
• RabbitMQ - Message queuing
• MessagePack - Binary serialization
• Pydantic - Data validation and modeling
• Sentry - Error tracking and monitoring

AWS Integration:

• Boto3 - AWS SDK
• CloudWatch - Metrics and monitoring
• Simple Workflow Service - Workflow orchestration (deprecated)

Development Tools:

• uv - Dependency management
• Cronitor - Cron job monitoring

(back to top)

Getting Started

Here is some detail about the intended use of this package.

Prerequisites

A Python project or runtime environment. Since this project is already initialized with [uv][vu-url] dependency management, I recommend that you continue to use it. Beyond the Python dependencies defined in the configuration, the package init carries hardcoded dependencies on Sentry and 1Password in order to function. Unless you want these and are effectively extending my base project, you're likely better off forking this package and cutting out what you do not need.

Installation

This package is published to the Python Package index from time to time.

(back to top)

Usage

I have various projects that use this tool chain. For example, my Base Project which can be run stand-alone but also serves as my Docker base image from which other projects are derived.

(back to top)

License

Distributed under the MIT License. See LICENSE for more information.

(back to top)

Acknowledgments

• Template on which this README is based

(back to top)