gitlab-secret_detection

Secret Detection Service

Secret Detection service is primarily responsible for detecting secrets in the given input payloads with RPC methods as the communication interface served via gRPC. This service will initially be invoked by Rails monolith when performing access checks for Git Push event, and eventually extended for the other usecases too.

Reference Issue: https://gitlab.com/groups/gitlab-org/-/epics/13792

Tools and Framework

• Ruby 3.3.X
• gRPC framework for serving RPC requests

Feature Distribution

In addition to offering the feature as an gRPC service, this project also includes the provision for distributing the same feature into a Ruby Gem. This provision was added to fulfil certain limitations. Here's the illustration representing the approach:

Project Layout

├── .runway
│   ├── runway.yml                       # Runway configuration file for Production environment
│   ├── runway-staging.yml               # Runway configuration file for Staging environment
│   └── env-*.yml                        # Environment vars for the respective environments. Uses vault for secrets
├── bin
│   └── start_server                     # gRPC server initiator that loads the server configuration file
├── config
│   └── log.rb                           # Logger configuration
├── ci
│   └── scripts/..                       # CI scripts used in the CI templates
│   └── templates
│       ├── build.yml                    # CI jobs for building container image and ruby gems
│       ├── test.yml                     # CI jobs for running tests
│       └── release.yml                  # CI jobs for releases related to Ruby gem, GitLab releases
├── lib
│   └── gitlab
│       └── secret_detection
│           ├── version.rb               # Secret Detection Gem release version
│           ├── core/..                  # Secret detection logic (most of it pulled from existing gem)
│           └── grpc
│               ├── generated/..         # gRPC generated files and secret detection gRPC service
│               ├── client/..            # gRPC client to invoke secret detection service's RPC endpoints
│               └── scanner_service.rb   # Secret detection gRPC service implementation
├── examples
│   └── sample-client/..                 # Sample Ruby RPC client that connects with gRPC server and calls RPC scan
├── proto
│   └── secret_detection.proto           # Service Definition file containing gRPC request and response interface
├── server
│   ├── interceptors/..                  # gRPC server-side interceptors like Auth, Log etc.
│   └── server.rb                        # gRPC server file with configuration
├── spec/..                              # Rspec tests and related test helpers
├── gitlab-secret_detection.gemspec      # Gemspec file for Ruby Gem
├── Dockerfile                           # Dockerfile for running gRPC server
├── Makefile                             # All the CLI commands placed here
└── RULES_VERSION                        # Current version of secret-detection-rules

Makefile commands

Usage make <command>

Command	Description
install_secret_detection_rules	Downloads secret-detection-rules based on package version defined in RULES_VERSION
install	Installs ruby gems in the project using Ruby bundler
lint_fix	Fixes all the fixable Rubocop lint offenses
gem_clean	Cleans existing gem file(if any) generated through gem build process
gem_build	Builds Ruby gem file wrapping secret detection logic (lib directory)
generate_proto	Generates ruby(.rb) files for the Protobud Service Definition files(.proto)
grpc_docker_build	Builds a docker container image for gRPC server
grpc_docker_serve	Runs gRPC server via docker container listening on port 8080. Run grpc_docker_build make command before running this command.
grpc_serve	Runs gRPC server on the CLI listening on port 50001. Run install make command before running this command.
run_core_tests	Runs RSpec tests for Secret Detection core logic
run_grpc_tests	Runs RSpec tests for Secret Detection gRPC endpoints
run_utils_tests	Runs RSpec tests for Secret Detection utilities
run_all_tests	Runs all the RSpec tests in the project

Secret Detection Rules

The make target install_secret_detection_rules fetches the version of the secret detection rules package specified in the RULES_VERSION file and extracts the secret_push_protection_rules.toml rule file. The command is called when running tests as well as when building the Dockerfile. The secret_push_protection_rules.toml file is added to .gitignore to avoid checking in changes.

Generating a ruby gem

In the project directory, run make gem command in the terminal that builds a ruby gem(ex: secret_detection-0.1.0.gem) in the root of the project directory.

Server

This project currently runs on gRPC server.

Service Definitions

• Scanner Service: ProtoBuf Service Definition

• Health Check Service: ProtoBuf Service Definition

Running the server locally

Pre-requisite: gRPC installed on your system (brew install grpc)

server	mode	command	Listening port
gRPC	CLI	make grpc_serve	50001
gRPC	Docker	make grpc_docker_build && make grpc_docker_serve	8080

gRPC server port can be configured via RPC_SERVER_PORT environment variable

Calling gRPC endpoints from terminal

Pre-requisite:

• gRPC installed on your system (via brew install grpc)
• gRPC client like Postman or you can install grpcurl (via brew install grpcurl)

Authentication:

The RPC service uses a basic form of token-based authentication. When invoking an RPC request on the client side, we need to append an x-sd-auth RPC Header whose value is dependent on the environment where the server is running.

The auth token value when the server is running in:

• Localhost: 12345
• Staging: env/staging/service/secret-detection/API_AUTH_TOKEN path in Vault
• Production: env/production/service/secret-detection/API_AUTH_TOKEN path in Vault

Note that only Secret Detection RPC requests are guarded with authentication.

More details can be found in this issue.

Health Check

RPC Method: grpc.health.v1.Health/Check

Example

$ grpcurl -plaintext -d '{"service":"gitlab.secret_detection.Scanner"}' localhost:50001 grpc.health.v1.Health/Check

You should see the following response as a result:

{
  "status": "SERVING"
}

Secret Detection Scan (Unary RPC Call)

RPC Method: gitlab.secret_detection.Scanner/Scan Default Timeout(configurable): per-request: 180 seconds, per-payload: 30 seconds

Example: Basic

grpcurl -plaintext -d @ \
  -rpc-header 'x-sd-auth:12345' \
  localhost:50001 \
  gitlab.secret_detection.Scanner/Scan <<EOM
{
  "payloads": [
    {
      "id": "94283",
      "data": "glpat-12345123451234512345"
    }
  ]
}
EOM

You should see the following response as a result:

{
  "results": [
    {
      "payload_id": "94283",
      "status": "FOUND",
      "type": "gitlab_personal_access_token",
      "description": "GitLab Personal Access Token",
      "line_number": 1
    }
  ],
  "status": "FOUND"
}

Example: Using Exclusions and Tags

$ grpcurl -d @ \
  localhost:50001 \
  -rpc-header 'x-sd-auth:12345' \
  gitlab.secret_detection.Scanner/Scan <<EOM
{
    "payloads": [
        {
            "id": "94283",
            "data": "glpat-12345123451234512345"
        },
        {
            "id": "37405",
            "data": "glrt-12345123451234512345"
        },
        {
            "id": "14954",
            "data": "GR1348941__DUMMY_RUNNER_TOKEN"
        }
    ],
    "exclusions": [
        {
            "exclusion_type": "EXCLUSION_TYPE_RULE",
            "value": "gitlab_personal_access_token"
        },
        {
            "exclusion_type": "EXCLUSION_TYPE_RAW_VALUE",
            "value": "glrt-12345123451234512345"
        }
    ],
    "tags": [
        "gitlab_blocking"
    ]
}
EOM

You should see the following response as a result:

{
  "results": [
    {
      "payload_id": "14954",
      "status": "STATUS_FOUND",
      "type": "gitlab_pipeline_trigger_token",
      "description": "GitLab Pipeline Trigger Token",
      "line_number": 1
    }
  ],
  "status": "STATUS_FOUND"
}

Secret Detection Scan (Bi-directional RPC Streaming)

RPC Method: gitlab.secret_detection.Scanner/ScanStream Default Timeout(configurable): per-request: 180 seconds, per-payload: 30 seconds

Bi-directional RPC streaming allows the server to receive a continuous stream of requests and respond with a corresponding stream of responses as each request is processed. Unlike unary RPC calls, where requests are sent once and the channel closes after a single response, bi-directional streaming keeps the RPC channel open to handle ongoing requests and responses, enabling continuous communication between the client and server.

Example

To try this out, we will accept the request input from STDIN. As on when you provide request input json in the terminal, you should see a corresponding server response. The connection will continue to remain open for accepting requests unless you explicitly close the connection with Ctrl+C.

Here's the command to start an RPC streaming channel on the terminal:

$ grpcurl -d @ localhost:50001 -rpc-header 'x-sd-auth:12345' gitlab.secret_detection.Scanner/ScanStream

Once the channel is open, enter the following sample request input in the terminal:

{
  "payloads": [
    {
      "id": "94283",
      "data": "glpat-12345123451234512345"
    }
  ]
}

you should immediately see the Server's response like below:

{
  "results": [
    {
      "payload_id": "94283",
      "status": "STATUS_FOUND",
      "type": "gitlab_personal_access_token",
      "description": "GitLab Personal Access Token",
      "line_number": 1
    }
  ],
  "status": "STATUS_FOUND"
}

You may continue to provide more request inputs to the opened RPC channel to receive corresponding server responses. Features like Exclusions, Tag Filtering works same as outlined under Unary call section.

NOTE:

• In case you're running the local server via Docker, replace localhost:50001 with localhost:8080 in the example grpcurl commands.
• In case you want to access staging server, replace localhost:50001 with secret-detection.staging.runway.gitlab.net:443

How to invoke the server from the code (as an RPC Client)

There is a sample Ruby RPC Client in the project that contains following code for reference:

• Setting up the connection with the server (both local and remote server with SSL)
• Invoke Secret Detection Unary call
• Invoke Secret Detection bi-directional stream call

Run ruby examples/sample-client/sample_client.rb on your terminal to run the sample RPC client calling both unary and streaming RPC Scan methods.

Benchmark

RPC service is benchmarked using ghz, a powerful CLI-based tool for load testing and benchmarking gRPC services. More details added here.

Release Process

We do three primary actions for every merge to main branch:

• Build and Publish SD ruby gem to RubyGems.org:

  • The latest version for releasing Secret Detection gem is pulled from Gitlab::SecretDetection::Gem::VERSION (located atlib/gitlab/secret_detection/version.rb).
  • We build a ruby gem for the code snapshot and tag it to the extract release version.
  • The script for publising the gem to RubyGems.org is available here.

• Deploy SD gRPC server to GCP using Runway:

  • We build a docker container for the current code snapshot and tag it under $CI_REGISTRY_IMAGE/image:$CI_COMMIT_SHORT_SHA container registry path.
  • The same container registry path is given as input to the Runway CI downstream which takes it forward for deploying in Staging and Production environments.

• Make a GitLab Release:

  • We use a modified version of upsert git tag job where instead of fetching the version from the first changelog entry, we fetch it from Gitlab::SecretDetection::Gem::VERSION. The rest of the behaviour is retained i.e., creating a tag from the version and then creating a new GitLab release against that tag.
  • The job pulls the description of the latest version entry from the CHANGELOG.md and uses it for the Release description.
  • The script for creating a git tag and making GitLab release is available here.

NOTE: There is no logical requirement for the versions defined in Gitlab::SecretDetection::Gem::VERSION and latest entry of CHANGELOG.md to be the same. However, we expect them to be the same to keep it consistent. We've added a CI job(validate version sync) that ensures the version sync between them.