goblet-gcp

GOBLET

Goblet is a framework for writing serverless rest apis in python in google cloud. It allows you to quickly create and deploy python apis backed by Cloud Functions and Cloud Run as well as other GCP serverless services.

It provides:

• A command line tool for creating, deploying, and managing your api
• A decorator based API for integrating with GCP API Gateway, Storage, Cloudfunctions, PubSub, Scheduler, Cloudrun Jobs, BQ remote functions, Redis, Monitoring alerts and other GCP services.
• Local environment for testing and running your api endpoints
• Dynamically generated openapispec
• Support for multiple stages

You can create Rest APIs:

from goblet import Goblet, jsonify, goblet_entrypoint

app = Goblet(function_name="goblet_example")
goblet_entrypoint(app)

@app.route('/home')
def home():
    return {"hello": "world"}

@app.route('/home/{id}', methods=["POST"])
def post_example(id: int) -> List[int]:
    return jsonify([id])

You can also create other GCP resources that are related to your REST api:

from goblet import Goblet, jsonify, goblet_entrypoint

app = Goblet(function_name="goblet_example")
goblet_entrypoint(app)

# Scheduled job
@app.schedule("5 * * * *")
def scheduled_job():
    return jsonify("success")

# Pubsub subscription
@app.pubsub_subscription("test")
def pubsub_subscription(data):
    app.log.info(data)
    return

# Example Redis Instance
app.redis("redis-test")

# Example Metric Alert for the cloudfunction metric execution_count with a threshold of 10
app.alert("metric",conditions=[MetricCondition("test", metric="cloudfunctions.googleapis.com/function/execution_count", value=10)])

Once you've written your code, you just run goblet deploy and Goblet takes care of deploying your app.

$ goblet deploy -l us-central1
...
https://api.uc.gateway.dev

$ curl https://api.uc.gateway.dev/home
{"hello": "world"}

Note: Due to breaking changes in Cloudfunctions you will need to wrap your goblet class in a function. See issue #88. In the latest goblet version (0.5.0) there is a helper function goblet_entrypoint that can be used as well.

goblet_entrypoint(app)

Resources Supported

Infrastructure

• vpc connector
• redis
• api gateway
• cloudtaskqueue
• pubsub topics
• bq spark stored procedures

Backends

• cloudfunction
• cloudfunction V2
• cloudrun

Routing

• api gateway
• http

Handlers

• pubsub
• scheduler
• storage
• eventarc
• cloudrun jobs
• bq remote functions
• cloudtask target
• uptime checks

Alerts

• Backend Alerts
• Uptime Alerts
• PubSub DLQ Alerts

Data Typing Frameworks Supported

• pydantic
• marshmallow

Installation

To install goblet, open an interactive shell and run:

pip install goblet-gcp

Make sure to have the correct services enabled in your gcp project depending on what you want to deploy

api-gateway, cloudfunctions, storage, pubsub, scheduler

You will also need to install gcloud cli for authentication

QuickStart

In this tutorial, you'll use the goblet command line utility to create and deploy a basic REST API. This quickstart uses Python 3.10. You can find the latest versions of python on the Python download page.

To install Goblet, we'll first create and activate a virtual environment in python3.10:

$ python3 --version
Python 3.10.10
$ python3 -m venv venv310
$ . venv37/bin/activate

Next we'll install Goblet using pip:

python3 -m pip install goblet-gcp

You can verify you have goblet installed by running:

$ goblet --help
Usage: goblet [OPTIONS] COMMAND [ARGS]...
...

Credentials

Before you can deploy an application, be sure you have credentials configured. You should run gcloud auth application-default login and sign in to the desired project.

Creating Your Project

create your project directory, which should include an main.py and a requirements.txt. Make sure requirements.txt includes goblet-gcp

$ ls -la
drwxr-xr-x   .goblet
-rw-r--r--   main.py
-rw-r--r--   requirements.txt

You can ignore the .goblet directory for now, the two main files we'll focus on is app.py and requirements.txt.

Let's take a look at the main.py file:

from goblet import Goblet, goblet_entrypoint

app = Goblet(function_name="goblet_example")
goblet_entrypoint(app)

@app.route('/home')
def home():
    return {"hello": "world"}

This app will deploy an api with endpoint /home.

Running Locally

Running your functions locally for testing and debugging is easy to do with goblet.

from goblet import Goblet

app = Goblet(function_name="goblet_example")
goblet_entrypoint(app)

@app.route('/home')
def home():
    return {"hello": "world"}

Then run goblet local Now you can hit your functions endpoint at localhost:8080 with your routes. For example localhost:8080/home

Building and Running locally using Docker

Make sure Docker Desktop and Docker CLI is installed, more information located here: https://docs.docker.com/desktop/

Refresh local credentials by running: gcloud auth application-default login

Set the GOOGLE_APPLICATION_CREDENTIALS variable by running: export GOOGLE_APPLICATION_CREDENTIALS=~/.config/gcloud/application_default_credentials.json

To build container run: docker build . -t <tag>

To start container run:

    docker run -p 8080:8080 \
        -v ~/.config/gcloud/application_default_credentials.json:/tmp/application_default_credentials.json:ro \
        -e GOOGLE_APPLICATION_CREDENTIALS=/tmp/application_default_credentials.json \
        -e GCLOUD_PROJECT=<gcp-project> <tag>:latest

Installing private packages during Docker Build

To install a private package located with GCP Artifact Registry, credentials will need to be mounted during the build process. Add this line to Dockerfile before requirements install:

RUN --mount=type=secret,id=gcloud_creds,target=/app/google_adc.json export GOOGLE_APPLICATION_CREDENTIALS=/app/google_adc.json \  
    && pip install -r requirements.txt

To build container run: docker build . --secret id=gcloud_creds,src="$GOOGLE_APPLICATION_CREDENTIALS" -t <tag>

Deploying

Let's deploy this app. Make sure you're in the app directory and run goblet deploy making sure to specify the desired location:

$ goblet deploy -l us-central1
INFO:goblet.deployer:zipping function......
INFO:goblet.deployer:uploading function zip to gs......
INFO:goblet.deployer:function code uploaded
INFO:goblet.deployer:creating cloudfunction......
INFO:goblet.deployer:deploying api......
INFO:goblet.deployer:api successfully deployed...
INFO:goblet.deployer:api endpoint is goblet-example-yol8sbt.uc.gateway.dev

You now have an API up and running using API Gateway and cloudfunctions:

$ curl https://goblet-example-yol8sbt.uc.gateway.dev/home
{"hello": "world"}

Try making a change to the returned dictionary from the home() function. You can then redeploy your changes by running golet deploy.

Next Steps

You've now created your first app using goblet. You can make modifications to your main.py file and rerun goblet deploy to redeploy your changes.

At this point, there are several next steps you can take.

Docs - Goblet Documentation

If you're done experimenting with Goblet and you'd like to cleanup, you can use the goblet destroy command making sure to specify the desired location, and Goblet will delete all the resources it created when running the goblet deploy command.

$ goblet destroy -l us-central1
INFO:goblet.deployer:destroying api gateway......
INFO:goblet.deployer:api configs destroying....
INFO:goblet.deployer:apis successfully destroyed......
INFO:goblet.deployer:deleting google cloudfunction......
INFO:goblet.deployer:deleting storage bucket......

Docs

Goblet Documentation

Blog Posts

Building Python Serverless Applications on GCP

Serverless APIs made simple on GCP with Goblet backed by Cloud Functions and Cloud Run

Tutorial: Publishing GitHub Findings to Security Command Center

Tutorial: Cost Spike Alerting

Tutorial: Setting Up Approval Processes with Slack Apps

Tutorial: API Deployments with Traffic Revisions and Centralized Artifact Registries in Google Cloud Run

Tutorial: Deploying Cloud Run Jobs

Tutorial: Connecting Cloudrun and Cloudfunctions to Redis and other Private Services using Goblet

Tutorial: Deploying BigQuery Remote Functions

GCP Alerts the Easy Way: Alerting for Cloudfunctions and Cloudrun using Goblet

Tutorial: Deploy CloudTaskQueues, enqueue CloudTasks and handle CloudTasks

Tutorial: Low Usage Alerting On Slack for Google Cloud Platform

Easily Manage IAM Policies for Serverless REST Applications in GCP with Goblet

Serverless Data Pipelines in GCP using Dataform and BigQuery Remote Functions

Examples

Goblet Examples

Issues

Please file any issues, bugs or feature requests as an issue on our GitHub page.

Github Action

Goblet Github Action

Roadmap

☑ Integration Tests
☑ Api Gateway Auth
☑ Configuration Options (function names, ...)
☑ Use checksum for updates
☑ Cloudrun Backend
☑ Scheduler trigger
☑ Pub Sub trigger
☑ Cloud Storage trigger
☑ Cloudrun Jobs trigger
☐ Firestore trigger
☐ Firebase trigger
☑ CloudTask and CloudTask Queues
☐ Cloud Endpoints trigger
☑ EventArc trigger
☑ Redis infrastructure
☑ BQ Remote Functions
☑ Deploy API Gateway from existing openapi spec
☑ Deploy arbitrary Dockerfile to Cloudrun
☑ Multi Container Deployments
☑ Create Deployment Service Accounts
☑ Automatically add IAM invoker bindings on the backend based on deployed handlers
☑ Uptime Checks

Want to Contribute

If you would like to contribute to the library (e.g. by improving the documentation, solving a bug or adding a cool new feature) please follow the contribution guide and submit a pull request.

Want to Support

---

Based on chalice