####################### LSST the Docs: Conveyor #######################
.. image:: https://img.shields.io/pypi/v/ltd-conveyor.svg :target: https://pypi.python.org/pypi/ltd-conveyor :alt: Python Package Index .. image:: https://github.com/lsst-sqre/ltd-conveyor/workflows/Python%20CI/badge.svg :target: https://github.com/lsst-sqre/ltd-conveyor/actions?query=workflow%3A%22Python+CI%22 :alt: GitHub Actions .. image:: https://img.shields.io/badge/ltd--conveyor-lsst.io-brightgreen.svg :target: https://ltd-conveyor.lsst.io :alt: Documentation
Conveyor is a common Python library and command-line app for LSST the Docs (LTD) services and clients.
Key features:
ltd command-line app.
The ltd upload subcommand makes it easy to integrate documentation uploads with continuous integration jobs, particularly Travis CI.
The ltdconveyor Python package provides APIs for working with LSST the Docs.
ltdconveyor.s3 subpackage provides S3 object management services.
All documentation hosted by LTD is served through the Fastly CDN <https://www.fastly.com>_, which uses S3 object metadata to purge caches.
Conveyor ensures that this metadata is properly set by all clients and microservices that upload, copy, or delete objects from S3.
ltdconveyor.fastly provides Fastly cache management APIs.
ltdconveyor.keeper provides APIs for working with LTD Keeper's REST API.
You can learn more about LTD in our SQR-006 <https://sqr-006.lsst.io>_ technote.
Read the docs: https://ltd-conveyor.lsst.io
########## Change log ##########
pkg_resources for better compatibility with Python 3.12 environments.Fix parsing of the GITHUB_HEAD_REF environment variable in GitHub Actions.
The ltd upload command now works in pull request workflows on GitHub Actions.
This release focuses on infrastructure improvements, and should not have a significant impact on users.
This tool is officially tested and supported on Python 3.7 and 3.8. Python 3.6 is no longer officially supported.
Removed pins on third-party dependencies, including boto3, requests, uritemplate, and click. In doing so, LTD Conveyor is easier to integrate into your existing Python environment.
We have migrated from Travis CI to GitHub Actions for testing, linting, documentation builds and deployment, and deployments to PyPI. In addition to on-demand testing, GitHub Actions tests the package on a daily basis to ensure compatibility with third-party dependencies.
Internally, we use tox for running tests, linters, and to build documentation. tox makes it possible for local test runs to use the same set up as CI.
The codebase is now automatically formatted with black and isort. By using pre-commit, we ensure that formatting is always applied.
The codebase now has type annotations, which are tested with mypy. Type annotations help ensure that APIs are explicitly defined and correctly used.
--gh option for ltd upload now supports tag and PR events, in addition to branches.--gh option to the ltd upload command to support usage in GitHub Actions <https://help.github.com/en/actions>__.LTD Conveyor can now upload new builds to S3 using Amazon's S3's presigned POST URL feature.
This means that clients no longer need credentials for S3 — the LTD Keeper API server generates presigned POST URLs as part of the POST /products/<product>/builds/ build registration step.
The ltd command now uses the presigned POST-based URLs.
The --aws-id and --aws-secret options have been removed, but the --user and --password options remain.
If you use environment variables, there should be no change in how you use ltd upload (aside from not needing the $LTD_AWS_ID and $LTD_AWS_SECRET environment variables).
setup.cfg file and a pyproject.toml file (PEP 518).ltdconveyor.s3.delete_dir, since 0.4.1, would raise a TypeError while deleting an empty directory (no objects in the S3 prefix).ltdconveyor.s3.delete_dir would fail if there are more than 1000 objects under a path prefix that is being deleted.Documenteer_\ -based Sphinx set up.Added a new command-line app, ltd, that provides subcommands for clients to work with the LSST the Docs API.
This app is implemented with Click_, and its documentation is automatically generated from the command-line help with sphinx-click_.
The first subcommand is ltd upload, which lets a client upload a site to LTD as a new build.
This command includes special features for using the client from Travis CI (ltd upload --travis option) to populate the version information from the Travis environment.
Other flags allow ltd upload to become a no-op under certain circumstances (for example, skip uploads on pull requests) or arbitrarily (set ltd upload --skip or export LTD_SKIP_UPLOAD=true).
.. note::
With this feature, LTD Conveyor effectively deprecates LTD Mason.
LTD Mason was the original uploader for LSST the Docs, but it was also designed around the idea of building the LSST Science Pipelines documentation as well (hence the term "mason").
Over time, we realized it is better to have a general purpose client and uploader for LSST the Docs (LTD Conveyor) and a dedicated tool for assembling the multi-package Sphinx documentation site (Documenteer).
Added the ltdconveyor.keeper subpackage that widens the scope of LTD Conveyor to be a full-service library for building LTD clients, not just an S3 upload client.
The ltdconveyor.keeper.login.get_keeper_token function lets you obtain a temporary auth token for the LTD Keeper API.
The ltdconveyor.keeper.build module includes functions for performing the build upload handshake with the LTD Keeper API.
Removed Python 2.7 and 3.4 compatibility. Now the codebase is entirely Python 3 (3.5, 3.6)-oriented.
Completely reorganized Python namespace.
Now all S3 APIs are in ltdconveyor.s3.
Fastly APIs are available from ltdconveyor.fastly.
Switched to using setuptools_scm to generate version strings for releases.
Switched to using extras_require for development dependencies (pip install -e ".[dev]").
This lets us exclusively coordinate dependencies in setup.py.
Enable testing via python setup.py test.
Also upgrade the testing stack to pytest 3.5 and pytest-flake8 1.0.
Compatibility updates to the Sphinx documentation infrastructure.
Default to acl=None to support more AWS IAM users.
It turns out that not all IAM users with PutObject permissions also have permissions to set the ACL for an object.
We want to make it possible for many lightweight IAM users to upload to restricted sub-directories of the lsst-the-docs bucket, but it seems hard to make these users ACL grantees too.
We now seek an alternative:
lsst-the-docs bucket now has a PublicRead bucket policyRuntimeErrors.
Assertions shouldn't be used to raise exceptions in production code.open_bucket function.
This provides a convenient API for getting a boto3 bucket resource, particularly for clients that use the upload_object and upload_file APIs directly that take a bucket only.content_type parameter to upload_object so that a user can specify text/html for an HTML upload.
The upload_file/upload_dir functions avoid this problem by using Python's mimetypes library to guess the encoding, but upload_object is lower-level and warrants having the user explicitly provide the content type.create_dir_redirect_object.
This code used to be inside upload_dir, but in LTD Dasher_ it seems necessary to upload files one at a time, and thus it's necessary to directly create these directory redirect objects.'..' shows up in directory names (seen when doing upload_dir with LTD Dasher_ asset directories).purge_key function from LTD Keeper_.
Any LTD application that uploads objects to S3 might also need to purge Fastly CDN caches.
This purge capability is presented as a simple function.LTD Mason.
The purpose of LTD Conveyor is to provide a set of common S3 APIs that can be used from both client (uploading docs from CI) and server (LTD Keeper) settings... _LTD Keeper: https://ltd-keeper.lsst.io .. _LTD Mason: https://ltd-mason.lsst.io .. _LTD Dasher: https://github.com/lsst-sqre/ltd-dasher .. _Documenteer: https://documenteer.lsst.io .. _Click: http://click.pocoo.org/ .. _sphinx-click: https://sphinx-click.readthedocs.io/en/latest/ .. _LSST Science Pipelines: https://pipelines.lsst.io
MIT License
Copyright 2016-2020 Association of Universities for Research in Astronomy, Inc. (AURA)
Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
FAQs
LSST the Docs admin client.
We found that ltd-conveyor demonstrated a healthy version release cadence and project activity because the last version was released less than a year ago. It has 3 open source maintainers collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Research
Security News
Attackers used a malicious npm package typosquatting a popular ESLint plugin to steal sensitive data, execute commands, and exploit developer systems.
Security News
The Ultralytics' PyPI Package was compromised four times in one weekend through GitHub Actions cache poisoning and failure to rotate previously compromised API tokens.
Security News
Research
Socket researchers found a malicious Maven package impersonating the legitimate ‘XZ for Java’ library, introducing a backdoor for remote code execution.