Huge News!Announcing our $40M Series B led by Abstract Ventures.Learn More →
Socket
Sign inDemoInstall
Socket
Sign inDemoInstall

fpark

Package Overview
Dependencies
Maintainers
2
Versions
9
Alerts
File Explorer

Advanced tools

License
Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

fpark - npm Package Compare versions

Comparing version 0.5.0 to 0.6.0

doc/api.md
doc/cluster.md
doc/configuration.md
doc/file-encryption.md
doc/README.md
doc/token.md
doc/update-space.md
jobs/distribute-space.js
lib/whereis.js
src/config-verifier.js
src/stats.js

6

CHANGELOG.md
# Changelog
## v0.6.0
**2020-11-03**
- Add route `GET /node/stats` to retrieve useful usage statistics (ie README).
- **Breaking changes**:
- File repartition has changed. Previously, repartition started from the filename modulo the number of replicas. Now, repartition starts from the hash of the filename modulo the number of nodes.
## v0.5.0

@@ -4,0 +10,0 @@ **2020-09-30**

9

package.json
{
"name": "fpark",
"version": "0.5.0",
"version": "0.6.0",
"main": "index.js",

@@ -31,3 +31,8 @@ "scripts": {

},
"description": ""
"description": "",
"pkg": {
"scripts": [
"jobs/*.js"
]
}
}

167

README.md

@@ -15,28 +15,2 @@ # Fpark

## Table of content
<!-- TOC -->
- [Fpark](#fpark)
- [Table of content](#table-of-content)
- [Installation](#installation)
- [Node](#node)
- [Systemd](#systemd)
- [Usage](#usage)
- [Configuration](#configuration)
- [API](#api)
- [GET /file/:filename/container/:container](#get-filefilenamecontainercontainer)
- [PUT /file/:filename/container/:container](#put-filefilenamecontainercontainer)
- [DELETE /file/:filename/container/:container](#delete-filefilenamecontainercontainer)
- [POST /node/register](#post-noderegister)
- [Token](#token)
- [Multi-instances](#multi-instances)
- [Region](#region)
- [Write & Read](#write--read)
- [Cluster & File replication](#cluster--file-replication)
- [File encryption](#file-encryption)
- [License](#license)
<!-- /TOC -->
## Installation

@@ -73,145 +47,8 @@

## Usage
### Documentation
### Configuration
Find answers in [documentation](./doc/README.md)
```js
{
"ID" : null, // ID for the current instance
"SERVER_PORT" : 6000, // HTTP Server port
"SERVER_CLUSTERS" : 4, // Number of workers to start, by default number of CPU cores
"NODES" : [], // Instances of Fpark, { ID : INT, host : String }; ex: [{ ID : 100, host : 'http://localhost:3000' }]
"REPLICATION_NB_REPLICAS" : 3, // Default number of replcias for a new file
"LOGS_DIRECTORY" : "logs", // Relative path to logs
"FILES_DIRECTORY" : "data", // Relative path to data (ie files)
"KEYS_DIRECTORY" : "keys", // Relative path to where public keys are stored for PUT/DEL authorizations
"IS_REGISTRATION_ENABLED" : false, // Activate or Desactive container registration
"ENCRYPTION_IV" : "srp9zyldyxdzmddx", // Secret for encryption
"ENCRYPTION_IV_LENGTH" : 16, // String for encryption key length
"ENCRYPTION_ALGORITHM" : "aes-128-ctr", // Encryption algorithm used
"HASH_SECRET" : "2VVqHZ2x2qr54GUa", // Secret for hash
"HASH_ALGORITHM" : "sha256", // Algorithm for hash
"CACHE_CONTROL_MAX_AGE": 7776000, // Default max age for header cache control
"IMAGE_COMPRESSION_LIMIT" : 80, // Default comrpression for an image file
"IMAGE_COMPRESSION_LIMIT_JPEG" : 80, // Default comrpression for jpeg file
"IMAGE_COMPRESSION_LIMIT_WEBP" : 80, // Default compression for webp file
"IMAGE_SIZE_DEFAULT_WIDTH" : 1280, // Default max width of an image
"IMAGE_SIZES" : {}, // Sizes for omage resizing { 'S' : { width : 200, height : 100 }, 'M' : ... }
"MAX_FILE_SIZE" : 15000000 // Size limit for a file, default 15 Mo
}
```
### API
#### GET /c/:container/f/:filename
The url is **public**.
Get a file identified by `filename` from a container `container`.
`filename` is the complete name of the file : `fileId.extension`
Query options for the url are:
- `access_key` : access key to get a file for a container. It is mandatory. The key is given at the creation of the container (see Container creation).
- `size` : a valid size in `config.SIZES` to resize on the fly a file of type image.
#### PUT /c/:container/f/:filename
Put a file with id `filename` to a container `container`.
A JsonWebToken token issued by `container` is required to perform the action. See Token section.
#### DELETE /c/:container/f/:filename
Delete a file given by `filename` from a container `container`.
A JsonWebToken token issued by `container` is required to perform the action. See Token section.
#### POST /node/register
Create a container.
The body must be a valid JSON object with :
```json
{
"container" : "a unique key",
"key" : "public key",
"accessKey" : "a key to access GET /file/:filename"
}
```
The url can be disabled with `IS_REGISTRATION_ENABLED`.
### Token
Only the owner of a container can PUT and DELETE files. Make sure to always define the token as follows:
1. Register a container by calling the API `POST /node/register` **or** put the public key of the container in the keys directory as `container.pub` where `container` is the name of the container to create and set an access key for the container in a file as `container.access_key`.
1. Create a JsonWebToken token with the field `aud` equals to the registered `container`.
1. Add the token in the header `authorization` as `Authorization: Bearer <token>`.
To disable container registration, set `IS_REGISTRATION_ENABLED` to `false` in the configuration.
## Multi-instances
To enable multi-instances & replication, you must define nodes in `config.NODES`. A node is a running Fpark instance.
A node is defined as:
```js
{
"id" : Number, // example: 100
"host" : String // example: "http://region1.fpark.fr"
}
```
Then, you are able to define the number of replicas for a file with `REPLICATION_NB_REPLICAS`.
Each instance of Fpark must share the same configuration in `NODES` configuration parameter.
### Region
As a standard, Fpark allows you to define regions. As a result, Fpark will try to replicate a file between different regions (according to `REPLICATION_NB_REPLICAS`)
A region is defined by `node.id`. By convention, a region is represented by a hundred (1XX, 2XX, 3XX, etc.). For instance, if a node has `id = 201`, the region is `2`, `id = 300` -> `3` and so on.
### Write & Read
Fpark serves files from its data storage (`config.FILES_DIRECTORY`) or from another Fpark instance if multiple instances are defined (`config.NODES`).
If only one Fpark instance is running (no `config.NODES` defined), files are saved in the current Fpark instance.
In multiple instances configuration, an uploaded file is saved on a certain amount of instances as defined by `config.REPLICATION_NB_REPLICAS`.
### Cluster & File replication
Fpark replicates files among a number of Fpark instances (`REPLICATION_NB_REPLICAS`). When a file is uploaded, Fpark:
1. determine the nodes to save the file.
1. makes a hash of the filename (`HASH_ALGORITHM`, `HASH_SECRET`).
1. encrypts the content of the file with the filename (`ENCRYPTION_IV`, `ENCRYPTION_IV_LENGTH`, `ENCRYPTION_ALGORITHM`).
1. saves the file to the determined nodes.
When reading, Fpark:
1. determines where the file is stored.
1. decrypts the file
1. serves the file
### File encryption
All the files are encrypted by design. When posting a file to Fpark with `PUT /file/container/:containerId/:filename`, the parameter `filemname` is used to encrypt the content of the file.
The following config parameters allow you to customize encryption settings `"ENCRYPTION_IV", "ENCRYPTION_IV_LENGTH", "ENCRYPTION_ALGORITHM"`.
Internally, Fpark encrypts files with [`crypto.createCipheriv(algorithm, key, iv[, options])`](https://nodejs.org/api/crypto.html#crypto_crypto_createcipheriv_algorithm_key_iv_options).
The only way to decrypt a file is to know the filename **and** the `ENCRYPTION_IV`.
## License
Apache 2.0

12

src/api/del.js

@@ -21,2 +21,3 @@ const fs = require('fs');

const logger = kittenLogger.createPersistentLogger('del_file');
const stats = require('../stats');

@@ -31,4 +32,11 @@ /**

exports.delApi = function delApi (req, res, params, store) {
req.counters = [
stats.COUNTER_NAMESPACES.REQUEST_DURATION_AVG_DEL,
stats.COUNTER_NAMESPACES.REQUEST_DURATION_DEL,
stats.COUNTER_NAMESPACES.REQUEST_NUMBER_DEL
];
verify(req, res, params, () => {
let nodes = getNodesToPersistTo(params.id, store.CONFIG.NODES, store.CONFIG.REPLICATION_NB_REPLICAS);
let fileHash = file.getFileHash(store.CONFIG, params.id);
let nodes = getNodesToPersistTo(fileHash, store.CONFIG.NODES, store.CONFIG.REPLICATION_NB_REPLICAS);
let isAllowedToWrite = isCurrentNodeInPersistentNodes(nodes, store.CONFIG.ID);

@@ -67,3 +75,3 @@

let keyNodes = flattenNodes(nodes);
let keyNodes = flattenNodes(nodes, store.CONFIG.ID);
let filePath = file.getFilePath(store.CONFIG, keyNodes, params);

@@ -70,0 +78,0 @@ fs.unlink(filePath.path, err => {

16

src/api/get.js

@@ -29,2 +29,3 @@

const logger = kittenLogger.createPersistentLogger('get_file');
const stats = require('../stats');

@@ -44,3 +45,3 @@ /**

function handlerError (err) {
if (getHeaderNthNode(req.headers) === 3 || getHeaderFromNode(req.headers)) {
if (getHeaderNthNode(req.headers) === 3 || (getHeaderFromNode(req.headers) && !req.headers['x-forwarded-for'])) {
logger.warn({ msg : 'Depth reached', from : getHeaderFromNode(req.headers) }, { idKittenLogger : req.log_id });

@@ -80,4 +81,11 @@ return respond(res, 404);

exports.getApi = function getApi (req, res, params, store) {
req.counters = [
stats.COUNTER_NAMESPACES.REQUEST_DURATION_AVG_GET,
stats.COUNTER_NAMESPACES.REQUEST_DURATION_GET,
stats.COUNTER_NAMESPACES.REQUEST_NUMBER_GET
];
auth.verifyAccessKey(req, res, params, queryParams => {
let nodes = repartition.getNodesToPersistTo(params.id, store.CONFIG.NODES, store.CONFIG.REPLICATION_NB_REPLICAS);
let fileHash = file.getFileHash(store.CONFIG, params.id);
let nodes = repartition.getNodesToPersistTo(fileHash, store.CONFIG.NODES, store.CONFIG.REPLICATION_NB_REPLICAS);
let isAllowedToWrite = repartition.isCurrentNodeInPersistentNodes(nodes, store.CONFIG.ID);

@@ -115,4 +123,6 @@

let keyNodes = repartition.flattenNodes(nodes);
req.counters.push(stats.COUNTER_NAMESPACES.FILES_COUNT);
let keyNodes = repartition.flattenNodes(nodes, store.CONFIG.ID);
res.setHeader('Cache-Control', 'max-age=' + store.CONFIG.CACHE_CONTROL_MAX_AGE + ',immutable');

@@ -119,0 +129,0 @@ res.setHeader('Content-Encoding', 'gzip');

5

src/api/index.js

@@ -17,6 +17,9 @@ const { putApi } = require('./put');

if (store.CONFIG.IS_REGISTRATION_ENABLED) {
router.on('POST', '/node/register', nodeApi, store);
router.on('POST', '/node/register', nodeApi.nodeRegister, store);
}
if (store.CONFIG.IS_STATS_ENABLED) {
router.on('GET', '/node/stats', nodeApi.nodeStats, store);
}
}
module.exports = load;

74

src/api/node.js
const fs = require('fs');
const path = require('path');
const url = require('url');
const fetch = require('node-fetch');
const stats = require('../stats');
const { respond, queue } = require('../commons/utils');

@@ -8,3 +10,3 @@ const { setKey } = require('../commons/auth');

module.exports = function nodRegister (req, res, params, store) {
function nodeRegister (req, res, params, store) {
if (req.headers['content-type'] !== 'application/json') {

@@ -30,3 +32,9 @@ return respond(res, 400);

fs.writeFile(path.join(store.CONFIG.KEYS_DIRECTORY, _body.container + '.pub'), _body.key, { flag : 'wx' }, (err) => {
let _filename = _body.container;
if (typeof _body.container === 'string' && _body.container.includes('/')) {
console.log('Container cannot contain char "/"');
return respond(res, 500);
}
fs.writeFile(path.join(store.CONFIG.KEYS_DIRECTORY, _filename + '.pub'), _body.key, { flag : 'wx' }, (err) => {
if (err) {

@@ -40,3 +48,3 @@ if (err.code === 'EEXIST') {

fs.writeFile(path.join(store.CONFIG.KEYS_DIRECTORY, _body.container + '.access_key'), _body.accessKey, { flag : 'wx' }, (err) => {
fs.writeFile(path.join(store.CONFIG.KEYS_DIRECTORY, _filename + '.access_key'), _body.accessKey, { flag : 'wx' }, (err) => {
if (err) {

@@ -46,3 +54,3 @@ return respond(res, 500);

setKey(_body.container, _body.key);
setKey(_filename, _body.key);

@@ -90,1 +98,59 @@ if (getHeaderFromNode(req.headers)) {

}
function nodeStats (req, res, params, store) {
stats.getAll(statistics => {
if (!statistics) {
return _formatStatsForJSON(res, statistics);
}
let query = url.parse(req.url).search;
let queryParams = new URLSearchParams(query);
let format = queryParams.get('format');
if (format === 'prometheus') {
return _formatStatsForPrometheus(res, statistics);
}
_formatStatsForJSON(res, statistics);
});
}
/**
* Format statistics for prometheus
* @param {Object} res
* @param {Array} statistics [{ label : String, description : Object, value : * }]
*/
function _formatStatsForPrometheus (res, statistics) {
let result = '';
for (let i = 0; i < statistics.length; i++) {
let description = [];
for (let key in statistics[i].description) {
description.push(key + '="' + statistics[i].description[key] + '"');
}
result += statistics[i].label + '{' + description.join(',') + '} ' + statistics[i].value + '\n';
}
res.setHeader('Content-type', 'text/plain');
res.write(result);
respond(res, 200);
}
/**
* Format statistics for JSON
* @param {Object} res
* @param {Array} statistics [{ label : String, description : Object, value : * }]
*/
function _formatStatsForJSON (res, statistics) {
res.setHeader('Content-type', 'application/json');
res.write(JSON.stringify(statistics));
respond(res, 200);
}
module.exports = {
nodeRegister,
nodeStats
};

19

src/api/put.js

@@ -34,2 +34,3 @@ const fs = require('fs');

const logger = kittenLogger.createPersistentLogger('put_file');
const stats = require('../stats');

@@ -109,4 +110,11 @@ /**

exports.putApi = function put (req, res, params, store) {
req.counters = [
stats.COUNTER_NAMESPACES.REQUEST_DURATION_AVG_PUT,
stats.COUNTER_NAMESPACES.REQUEST_DURATION_PUT,
stats.COUNTER_NAMESPACES.REQUEST_NUMBER_PUT,
];
auth.verify(req, res, params, () => {
let nodes = repartition.getNodesToPersistTo(params.id, store.CONFIG.NODES, store.CONFIG.REPLICATION_NB_REPLICAS);
let fileHash = file.getFileHash(store.CONFIG, params.id);
let nodes = repartition.getNodesToPersistTo(fileHash, store.CONFIG.NODES, store.CONFIG.REPLICATION_NB_REPLICAS);
let isAllowedToWrite = repartition.isCurrentNodeInPersistentNodes(nodes, store.CONFIG.ID);

@@ -145,4 +153,6 @@

let keyNodes = repartition.flattenNodes(nodes);
req.counters.push(stats.COUNTER_NAMESPACES.FILES_COUNT);
let keyNodes = repartition.flattenNodes(nodes, store.CONFIG.ID);
let busboy = null;

@@ -218,2 +228,3 @@ try {

let nbNodes = 0;
queue(nodes, (node, next) => {

@@ -224,2 +235,4 @@ if (node.id === store.CONFIG.ID) {

nbNodes++;
let form = new FormData();

@@ -248,3 +261,3 @@ let streams = file.getFilePath(store.CONFIG, keyNodes, params).path;

isWritePending = false;
done(res, nodes.length ? 500 : 200);
done(res, nbNodes ? 500 : 200);
});

@@ -251,0 +264,0 @@ });

12

src/commons/file.js

@@ -42,3 +42,3 @@ const fs = require('fs');

let filename = this.getFileName(params.id, CONFIG.ENCRYPTION_IV_LENGTH);
let filenameDisk = encryption.hash(params.id, CONFIG.HASH_SECRET, CONFIG.HASH_ALGORITHM);
let filenameDisk = this.getFileHash(CONFIG, params.id);
let filePath = path.join(pathDisk, params.containerId, filenameDisk + '.enc');

@@ -50,2 +50,12 @@

/**
* Get file hash
* @param {Object} CONFIG
* @param {Object} filename
* @returns {String}
*/
getFileHash (CONFIG, filename) {
return encryption.hash(filename, CONFIG.HASH_SECRET, CONFIG.HASH_ALGORITHM);
},
/**
* Prepare streams to read file from disk

@@ -52,0 +62,0 @@ * @param {Object} CONFIG

21

src/commons/repartition.js

@@ -122,10 +122,10 @@

if (nbReplicas > nodes.length) {
nbReplicas = nodes.length;
}
setNodesByRegion(nodes);
let nodesToPersist = [];
let hashIndex = hash(str) % nbReplicas;
let hashIndex = hash(str) % nodes.length;
if (nbReplicas > nodes.length) {
throw new Error('Number of replica is superior to number of nodes');
}
_sortNodes(nodes);

@@ -183,14 +183,13 @@

* @param {Array} nodes
* @param {String} currentNode @optional
* @returns {String} "node1Id-node2Id-node3Id"
*/
function flattenNodes (nodes) {
function flattenNodes (nodes, currentNode) {
_sortNodes(nodes);
let res = '';
for (let i = 0; i< nodes.length; i++) {
res += '-' + nodes[i].id;
if (currentNode && nodes.length === 1 && currentNode === nodes[0].id) {
return '';
}
return res.slice(1);
return nodes.map(n => n.id).join('-');
}

@@ -197,0 +196,0 @@

1

src/config.json

@@ -9,2 +9,3 @@ {

"IS_REGISTRATION_ENABLED" : true,
"IS_STATS_ENABLED" : true,

@@ -11,0 +12,0 @@ "ENCRYPTION_IV" : "srp9zyldyxdzmddx",

27

src/logger.js
const crypto = require('crypto');
const kittenLogger = require('kitten-logger');
const stats = require('./stats');

@@ -18,5 +19,15 @@ kittenLogger.addFormatter('http:start', kittenLogger.formattersCollection.http_start);

function getMessageForReq (req) {
let url = req.url.split('/f/');
if (url.length > 1) {
url = url[0] + '/f/*';
}
else {
url = url[0];
}
return {
url,
method : req.method,
url : req.url,
ip : req.headers['x-forwarded-for'] || req.connection.remoteAddress || req.ip

@@ -27,6 +38,18 @@ };

function getMessageForRes (req, res) {
return {
let result = {
time : process.hrtime(req.log_start)[1] / 1000000, //ms
status : res.statusCode
};
if (req.counters) {
req.counters.forEach(counter => {
stats.update({
counterId : counter,
subCounterId : result.status,
value : result.time
});
});
}
return result;
}

@@ -33,0 +56,0 @@ 

fpark

Sorry, the diff of this file is not supported yet

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap
  • Changelog

About

Packages

npm

Go

Maven

PyPI

Rubygems

Stay in touch

Get open source security insights delivered straight into your inbox.

  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc