th2-grpc-crawler-data-processor

th2 gRPC crawler data processor library (0.3.2)

This project contains the gRPC interface to implement if you want to create your own crawler data processor.

The crawler data processor work with the crawler

How to transform template

• Create a directory with the same name as project name (use underscores instead of dashes) under src/main/proto directory (remove other files and directories if they exist).
• Place your custom .proto files in created directory. Pay attention to package specifier and import statements.
• Edit release_version and vcs_url properties in gradle.properties file.
• Edit rootProject.name variable in settings.gradle file. This will be the name of Java package.
• Edit package_info.json file in order to specify name and version for Python package (create file if it's absent).
• Edit parameters of setup.py in setup function invocation such as: author, author_email, url. Do not edit the others.
• Edit README.md file according to the new project.

Note that the name of created directory under src/main/proto directory is used in Python (it's a package name).

How to maintain project

• Make your changes.
• Up version of Java package in gradle.properties file.
• Up version of Python package in package_info.json file.
• Commit everything.

How to run project

Java

If you wish to manually create and publish package for Java, run these command:

gradle --no-daemon clean build publish artifactoryPublish \
       -Pbintray_user=${BINTRAY_USER} \
       -Pbintray_key=${BINTRAY_KEY}

BINTRAY_USER and BINTRAY_KEY are parameters for publishing.

Python

If you wish to manually create and publish package for Python:

• Generate services by gradle:

     gradle --no-daemon clean generateProto
  

  You can find the generated files by following path: src/gen/main/services/python
• Generate code from .proto files and publish everything:

  pip install -r requirements.txt
  python setup.py generate
  python setup.py sdist
  twine upload --repository-url ${PYPI_REPOSITORY_URL} --username ${PYPI_USER} --password ${PYPI_PASSWORD} dist/*
  

  PYPI_REPOSITORY_URL, PYPI_USER and PYPI_PASSWORD are parameters for publishing.

Changes:

0.3.1

• Update th2-grpc-data-provider Python dependency from 0.1.4 to 0.1.6

0.3.0

• Update th2-bom from 3.0.0 to 3.1.0
• Update th2-grpc-common from 3.1.2 to 3.8.0
• Add stubs for Python

0.2.0 (Breaking changes)

Breaking:

• Use list of MessageID instead of mapping between session and MessageID. User now will have to specify MessageID for both directions in the response if he or she needs to set a checkpoint. The list should contain a single MessageID for each pair alias + direction. If more than one is found the last one (according to their sequences) will be taken.
• The rpc methods was renamed according to the Protobuf naming convention (uses PascalCase).
• The event and message IDs from the response to connect method are removed because this functionality requires additional improvements on Crawler's side.

Added:

• New method that will be invoked by the crawler each time the new interval is started.