github.com/joelpm/gae_filepublisher
An AppEngine service that copies files from one CloudStorage location to another and publishes the names of the files (in their new location) to a PubSub topic. This could be used as the first step of a data pipline that brings log files into GCP. An external (to GCP) service would copy the logs into a well known location on CloudStorage. An AppEngine cron job would make a call to this service with the right parameters, which would then move the files to a staging area and publish them to a queue to be consumed by other workers. The handler will listen at: http://PROJECT_ID.appspot.com/tasks/filepublisher The paramters required by the handler defined in this service are: An example call could look like: To run the service locally for development the project ID must be specified in the environment and 'go run' can be used:
Readme
An AppEngine handler that moves files from one CloudStorage location to another and publishers the names of those files to PubSub.
Consider the situation where a cron job is copying files to a bucket on Google Cloud Storage. You want to process those files, and you need some way of knowing which ones are new. You also want to make sure that they only get processed once. One way to achieve that is to move the files from the incoming directory to a processing directory. When a file is moved to the processing directory, an event is published to a Google PubSub Topic with the name of the file (in its new location). This way you can have mutiple subscribers to the queue and each one of them can process a file.
The handler takes the following query-string parameters:
| Paramter | Description |
|---|---|
| topic | The name of the PubSub topic that events should be published to |
| dst_bucket | The bucket the files should be moved to |
| dst_path | The path in the destination bucket where files should be copied to |
| src_bucket | The bucket where files will be looked for (can be same as destination bucket) |
| src_prefix | A prefix used to find the files |
This URL would copy all files matching gs://my-log-files/inbound* to gs://my-log-files/processing/ and publish a message on the PubSub topic named "freshfiles" for each one.
The handler can be run locally using "goapp serve" by updating the app.yaml file and adding project information.
FAQs
An AppEngine service that copies files from one CloudStorage location to another and publishes the names of the files (in their new location) to a PubSub topic. This could be used as the first step of a data pipline that brings log files into GCP. An external (to GCP) service would copy the logs into a well known location on CloudStorage. An AppEngine cron job would make a call to this service with the right parameters, which would then move the files to a staging area and publish them to a queue to be consumed by other workers. The handler will listen at: http://PROJECT_ID.appspot.com/tasks/filepublisher The paramters required by the handler defined in this service are: An example call could look like: To run the service locally for development the project ID must be specified in the environment and 'go run' can be used:
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
A hospital in Mobile, Alabama, agreed to a settlement in a landmark ransomware death lawsuit, but is now reportedly reconsidering the agreement and refusing to pay.
Security News
A new report explores how advancements in LLMs are enhancing cyber threats, including polymorphic malware, personalized spearphishing, and the risk of hijacking customer service bots.
Security News
ESLint has approved an RFC that adds support for TypeScript configuration files, which is aimed at improving the developer experience and recognizing changes in the evolving JavaScript ecosystem.
