@data-leakage-protection/signatures

signatures (@data-leakage-protection/signatures)

Identify confidential and sensitive info in source code repositories by data-loss "signatures".

@data-leakage-protection/signatures is a Node.js module for storing and accessing to data-leakage detection definitions. We call the data structure that represents a data-leakage detection defintion a "signature." We store a community-tested list of signatures in a file called signatures.json.

Table of Contents

• 1. Security
• 2. Install
• 3. Usage
• 4. API
  • 4.1. @data-leakage-protection/signatures.Signature
  • 4.2. @data-leakage-protection/signatures.Signature.prototype.match
• 5. Accessing signatures with other tools and programming languages
• 6. Maintainers
• 7. Contributions
  • 7.1. Adding a Signature
  • 7.2. Editing a Signature
  • 7.3. Removing a Signature
• 8. License
• 9. References and Attributions

1. Security

Data leakage is the unauthorized transmission of data from within an organization to an external destination or recipient.¹

One of the most common forms of data-loss (aka, "data leakage") happens when developers (inadvertently) commit and push passwords, access-tokens, and sensitive data to a source-control management system (like Git). Consequently, confidential information "leaks" into search results and commit history.

The signatures.json contains a growing list of definitions to help you detect secrets in your source code repositories.

	Signature	Detected in
1	.asc file extension Potential cryptographic key bundle	extension
2	.p12 file extension PKCS#12 (.p12): potential cryptographic key bundle	extension
3	.pem file extension Potential cryptographic private key	extension
4	.pfx file extension PKCS#12 (.pfx): Potential cryptographic key bundle	extension
5	.pkcs12 file extension PKCS#12 (.pkcs12): Potential cryptographic key bundle	extension
6	1Password password manager database file Feed it to Hashcat and see if you're lucky	extension
7	AWS API Key __	contents
8	AWS CLI credentials file __	path
9	Apache htpasswd file __	filename
10	Apple Keychain database file __	extension
11	Azure service configuration schema file __	extension
12	Carrierwave configuration file Can contain credentials for cloud storage systems such as Amazon S3 and Google Storage	filename
13	Chef Knife configuration file Can contain references to Chef servers	filename
14	Chef private key Can be used to authenticate against Chef servers	path
15	Configuration file for auto-login process Can contain username and password	filename
16	Contains word: credential __	path
17	Contains word: password __	path
18	DBeaver SQL database manager configuration file __	filename
19	Day One journal file Now it's getting creepy...	extension
20	DigitalOcean doctl command-line client configuration file Contains DigitalOcean API key and other information	path
21	Django configuration file Can contain database credentials, cloud storage system credentials, and other secrets	filename
22	Docker configuration file Can contain credentials for public or private Docker registries	filename
23	Environment configuration file __	filename
24	Facebook Oauth __	contents
25	FileZilla FTP configuration file Can contain credentials for FTP servers	filename
26	FileZilla FTP recent servers file Can contain credentials for FTP servers	filename
27	GNOME Keyring database file __	extension
28	Generic API Key __	contents
29	Generic Secret __	contents
30	Git configuration file __	filename
31	GitHub __	contents
32	GitHub Hub command-line client configuration file Can contain GitHub API access token	path
33	GnuCash database file __	extension
34	Google (GCP) Service-account __	contents
35	Google Oauth __	contents
36	Heroku API Key __	contents
37	Hexchat/XChat IRC client server list configuration file __	path
38	Irssi IRC client configuration file __	path
39	Java keystore file __	extension
40	Jenkins publish over SSH plugin file __	filename
41	KDE Wallet Manager database file __	extension
42	KeePass password manager database file Feed it to Hashcat and see if you're lucky	extension
43	Little Snitch firewall configuration file Contains traffic rules for applications	filename
44	Log file Log files can contain secret HTTP endpoints, session IDs, API keys and other goodies	extension
45	Microsoft BitLocker Trusted Platform Module password file __	extension
46	Microsoft BitLocker recovery key file __	extension
47	Microsoft SQL database file __	extension
48	Microsoft SQL server compact database file __	extension
49	Mutt e-mail client configuration file __	filename
50	MySQL client command history file __	filename
51	NPM configuration file Can contain credentials for NPM registries	filename
52	Network traffic capture file __	extension
53	OmniAuth configuration file The OmniAuth configuration file can contain client application secrets	filename
54	OpenVPN client configuration file __	extension
55	PGP private key block __	contents
56	PHP configuration file __	filename
57	Password Safe database file __	extension
58	Password in URL __	contents
59	Pidgin OTR private key __	filename
60	Pidgin chat client account configuration file __	path
61	PostgreSQL client command history file __	filename
62	PostgreSQL password file __	filename
63	Potential Jenkins credentials file __	filename
64	Potential Linux passwd file Contains system user information	path
65	Potential Linux shadow file Contains hashed passwords for system users	path
66	Potential MediaWiki configuration file __	filename
67	Potential Ruby On Rails database configuration file Can contain database credentials	filename
68	Potential cryptographic private key __	extension
69	Potential jrnl journal file Now it's getting creepy...	filename
70	Private SSH key _rsa	filename
71	Private SSH key _dsa	filename
72	Private SSH key _ed25519	filename
73	Private SSH key _ecdsa	filename
74	RSA private key __	contents
75	Recon-ng web reconnaissance framework API key database __	path
76	Remote Desktop connection file __	extension
77	Robomongo MongoDB manager configuration file Can contain credentials for MongoDB databases	filename
78	Ruby IRB console history file __	filename
79	Ruby On Rails secret token configuration file If the Rails secret token is known, it can allow for remote code execution (http://www.exploit-db.com/exploits/27527/)	filename
80	Rubygems credentials file Can contain API key for a rubygems.org account	path
81	S3cmd configuration file __	filename
82	SFTP connection configuration file __	filename
83	SQL dump file __	extension
84	SQLite database file __	extension
85	SSH (DSA) private key __	contents
86	SSH (EC) private key __	contents
87	SSH (OPENSSH) private key __	contents
88	SSH configuration file __	path
89	Sequel Pro MySQL database manager bookmark file __	filename
90	Shell command alias configuration file Shell configuration files can contain passwords, API keys, hostnames and other goodies	filename
91	Shell command history file __	filename
92	Shell configuration file (.exports): Shell configuration files can contain passwords, API keys, hostnames and other goodies	filename
93	Shell configuration file (.functions): Shell configuration files can contain passwords, API keys, hostnames and other goodies	filename
94	Shell configuration file (.extra): Shell configuration files can contain passwords, API keys, hostnames and other goodies	filename
95	Shell configuration file (bash, zsh, csh): Shell configuration files can contain passwords, API keys, hostnames and other goodies	filename
96	Shell profile configuration file (profile): Shell configuration files can contain passwords, API keys, hostnames and other goodies	filename
97	Slack Token __	contents
98	Slack Webhook __	contents
99	T command-line Twitter client configuration file __	filename
100	Terraform variable config file Can contain credentials for terraform providers	filename
101	Tugboat DigitalOcean management tool configuration __	filename
102	Tunnelblick VPN configuration file __	extension
103	Twilio API Key __	contents
104	Twitter Oauth __	contents
105	Ventrilo server configuration file Can contain passwords	filename
106	Windows BitLocker full volume encrypted data file __	extension
107	cPanel backup ProFTPd credentials file Contains usernames and password hashes for FTP accounts	filename
108	git-credential-store helper credentials file __	filename
109	gitrob configuration file __	filename

2. Install

Before you begin, you'll need to have these

• Programming languages:

  • Node.js >= 10.x

  • NPM >= 4.0.0

• Skills:

  You'll need to know how to access the command line (aka, "Terminal") on your machine.

Open a Terminal and enter the following command:

# As a dependency in your Node.js app
npm i @data-leakage-protection/signatures --save-prod

3. Usage

Use @data-leakage-protection/signatures.signatures to find file extensions, names, and paths that commonly leak secrets.

const { signatures } = require('@data-leakage-protection/signatures')
// ⚠️ Note: the 'recursive-readdir' module is not bundled with
//    @data-leakage-protection/signatures. 'recursive-readdir' is referenced
//    only as an example.
const recursiveReaddir = require('recursive-readdir')

const potentialLeaks = recursiveReaddir('/path/to/local/repo')
  .then(files => files
    .map(file => signatures
    .map(signature => signature.match(file)))
  )
  .catch(err => err)

4. API

The @data-leakage-protection/signatures module provides a Signatures class, which validates @data-leakage-protection/signatures and converts regular expression strings to RE2 (whenever possible).

The @data-leakage-protection/signatures module's public API provides:

1. factory method: a convenience function that creates a signature object.
2. nullSignature: implements a default object literal with all signatures properties set to null.
3. Signature: a class that constructs a signature object.
4. signatures: an array of Signature instances.
5. toArray(data: {String|Array.<Object>}): generates an Array.<Signature> from a JSON string or object literal array.
6. validParts: a constants enum of valid Signature.prototype.part values.
7. validTypes: a constants enum of valid Signature.prototype.type values.

4.1. @data-leakage-protection/signatures.Signature

A class that constructs Signature objects.

const { Signature, validParts, validTypes } = require('@data-leakage-protection/signatures')

const signature = new Signature({
  caption: 'Potential cryptographic private key',
  description: '',
  part: validParts.EXTENSION,
  pattern: '.pem',
  type: validTypes.MATCH
})

4.2. @data-leakage-protection/signatures.Signature.prototype.match

Discover possible data leaks by matching a Signature pattern against file extensions, names, and paths.

const rsaTokenSignature = new Signature({
  'caption': 'Private SSH key',
  'description': '',
  'part': 'filename',
  'pattern': '^.*_rsa$',
  'type': 'regex'
})

const suspiciousFilePath = '/hmm/what/might/this/be/id_rsa'
rsaTokenSignature.match(suspiciousFilePath)
// => ['/hmm/what/might/this/be/id_rsa']

const fileThatIsJustBeingCoolBruh = 'file/that/is/just/being/cool/bruh'
rsaTokenSignature.match(suspiciousFilePath)
// => null

Review the source code for signature.

5. Accessing signatures with other tools and programming languages

You can access signatures.json without the @data-leakage-protection/signatures Node module. Select a tool or programming language below to view examples.

cURL

You can access data-loss rules using HTTPS. You can GET all signatures directly from Gitlab with cURL.

curl -X GET \
  'https://gitlab.com/data-leakage-protection/signatures/raw/master/signatures.json'

---

Golang

package main

import (
	"fmt"
	"net/http"
	"io/ioutil"
)

func main() {

	url := "https://gitlab.com/data-leakage-protection/signatures/raw/master/signatures.json"

	req, _ := http.NewRequest("GET", url, nil)

	req.Header.Add("Private-Token", "<your-personal-token>")
	req.Header.Add("cache-control", "no-cache")

	res, _ := http.DefaultClient.Do(req)

	defer res.Body.Close()
	body, _ := ioutil.ReadAll(res.Body)

	fmt.Println(res)
	fmt.Println(string(body))

}

---

Java (OK HTTP)

OkHttpClient client = new OkHttpClient();

String signaturesJson = "https://gitlab.com/data-leakage-protection/signatures/raw/master/signatures.json";

Request request = new Request.Builder()
  .url(signaturesJson)
  .get()
  .addHeader("Accept", "*/*")
  .addHeader("Cache-Control", "no-cache")
  .addHeader("Host", "gitlab.com")
  .addHeader("accept-encoding", "gzip, deflate")
  .addHeader("Connection", "keep-alive")
  .addHeader("cache-control", "no-cache")
  .build();

Response response = client.newCall(request).execute();

---

Node (native)

const http = require('https')

const options = {
  method: 'GET',
  hostname: ['gitlab', 'com'],
  path: ['api', 'v4', 'projects'],
  headers: {
    'Private-Token': '<your-access-token>',
    'cache-control': 'no-cache'
  }
}

const req = http.request(options, res => {
  const chunks = []

  res.on('data', chunk => {
    chunks.push(chunk)
  })

  res.on('end', () => {
    var body = Buffer.concat(chunks)
    console.log(body.toString())
  })
})

req.end()

---

Python (versions 2 and 3)

Python3

import http.client

conn = http.client.HTTPConnection("gitlab,com")

payload = ""

headers = {
  'Accept': "application/json",
  'cache-control': "no-cache"
}

conn.request("GET", "data-leakage-protection/signatures,raw,master,signatures.json", payload, headers)

res = conn.getresponse()
data = res.read()

print(data.decode("utf-8"))

Python2

import requests

url = "https://gitlab.com/data-leakage-protection/signatures/raw/master/signatures.json"

payload = ""
headers = {
  'Accept': "application/json",
  'cache-control': "no-cache"
}

response = requests.request("GET", url, data=payload, headers=headers)

print(response.text)

---

Ruby (NET::Http)

require 'uri'
require 'net/http'

url = URI("'https://gitlab.com/data-leakage-protection/signatures/raw/master/signatures.json")

http = Net::HTTP.new(url.host, url.port)

request = Net::HTTP::Get.new(url)
request["Private-Token"] = '<your-personal-token>'
request["cache-control"] = 'no-cache'

response = http.request(request)
puts response.read_body

6. Maintainers

@gregswindle

The Maintainer Guide has useful information for Maintainers and Trusted Committers.

7. Contributions

We gratefully accept Merge Requests! Here's what you need to know to get started.

Before submitting a Merge Request, please read our:

• Code of Conduct
• Contributing Aggreement
• Developer Guide
• Maintainer/Trusted Committer Guide

Thanks goes to our awesome contributors (emoji key):

Semantic Release Bot 🚧

gregswindle 💻 ⚠️ 📖 🐛 🚧

Christina Valdes 👀

sairam pooraj 👀

This project follows the all-contributors specification. Contributions of any kind welcome!

7.1. Adding a Signature

Before adding a new Signature, please review all current definitions: the Signature might already exist.

If the Signature does not exist, please be sure to add your Signature with the following properties:

1. caption: A succinct summary for the Signature. Think of caption as a well-written email subject.

2. description: Provide more details about the Signature if necessary. description is especially useful for differentiating similar Signatures.

3. hash: A hexidecimal SHA256 representation of a Signature (with ordered properties).

4. name: The Signature's caption, converted to kebab-case.

5. part: An enumeration that defines what the Signature is evaluating. Valid values are:

   • contents: The string(s) within a file.
   • extension: A file extension (which defines the Content-Type or mime-type).
   • filename: The unique name of the file.
   • path: The directory path relative to the repo and without the filename.

6. pattern: The string or regular expression to look for.

7. type: An enumeration that defines how to evaluate for secrets. Valid values are:

   • match: A strict string equivalency evaluation.
   • regex: A regular expression "search" or "test".

7.2. Editing a Signature

Edits are welcome! Just be sure to unit test.

7.3. Removing a Signature

Please provide a testable justification for any Signature removal.

8. License

© 2019 Greg Swindle.

View detailed legal NOTICEs .

9. References and Attributions

Footnotes

1. What is Data Leakage? Defined, Explained, and Explored | Forcepoint. (2019) Retrieved January 27, 2019, from https://www.forcepoint.com/cyber-edu/data-leakage ↩