Huge News!Announcing our $40M Series B led by Abstract Ventures.Learn More →
Socket
Sign inDemoInstall
Socket
Sign inDemoInstall

skipper

Package Overview
Dependencies
Maintainers
1
Versions
52
Alerts
File Explorer

Advanced tools

License
Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

skipper - npm Package Compare versions

Comparing version 0.2.1 to 0.4.0

EXAMPLE-UPSTREAM-RECEIVER.md
standalone/logger.js
standalone/Upstream/build-or-normalize-receiver.js
standalone/Upstream/build-renamer-stream.js
standalone/Upstream/index.js
standalone/Upstream/prototype._read.js
standalone/Upstream/prototype.fatalIncomingError.js
standalone/Upstream/prototype.noMoreFiles.js
standalone/Upstream/prototype.pipe.js
standalone/Upstream/prototype.serializeFiles.js
standalone/Upstream/prototype.upload.js
standalone/Upstream/prototype.writeFile.js
standalone/Upstream/Upstream.js
test/tests.test.js

4

lib/multipart.js

@@ -6,4 +6,4 @@ /**

var _ = require('lodash');
var log = require('../standalone/logger');
var Parser = require('./Parser');
var log = require('./logger');

@@ -48,3 +48,3 @@

} catch (e) {}
} else log(msg.yellow.bold);
} else log.color('yellow').write(msg);
});

@@ -51,0 +51,0 @@ }

5

lib/Parser/prototype.acquireUpstream.js

@@ -6,4 +6,4 @@ /**

var _ = require('lodash');
var log = require('../logger');
var Upstream = require('../Upstream');
var log = require('../../standalone/logger');
var Upstream = require('../../standalone/Upstream');

@@ -37,2 +37,3 @@

});
noopUpstream.fieldName = 'NOOP_'+fieldName;
return noopUpstream;

@@ -39,0 +40,0 @@ }

6

lib/Parser/prototype.onTextParam.js

@@ -5,5 +5,5 @@ /**

var StringDecoder = require('string_decoder').StringDecoder,
log = require('../logger'),
STRINGFILE = require('../../stringfile.js');
var StringDecoder = require('string_decoder').StringDecoder;
var log = require('../../standalone/logger');
var STRINGFILE = require('../../stringfile.js');

@@ -10,0 +10,0 @@

2

lib/Parser/prototype.parseReq.js

@@ -8,4 +8,4 @@ /**

var Form = require('multiparty').Form;
var log = require('../../standalone/logger');
var STRINGFILE = require('../../stringfile');
var log = require('../logger');

@@ -12,0 +12,0 @@

5

package.json
{
"name": "skipper",
"version": "0.2.1",
"version": "0.4.0",
"description": "Bodyparser for Express/Sails. Exposes simple API for streaming multiple files to disk, S3, etc. without buffering to a .tmp directory.",

@@ -43,3 +43,4 @@ "main": "index.js",

"skipper-disk": "~0.3.1",
"guid": "0.0.12"
"guid": "0.0.12",
"node-uuid": "^1.4.1"
},

@@ -46,0 +47,0 @@ "devDependencies": {

504

README.md

@@ -6,13 +6,25 @@ # [![Skipper](http://i.imgur.com/P6gptnI.png)](https://github.com/balderdashy/skipper)

### Streaming Multipart File Upload Parsing
Skipper makes it easy to implement streaming file uploads to disk, S3, or any of its supported file adapters.
Skipper is an opinionated variant of Connect's body parser designed to support the streaming upload of monolithic files to a compatible blob receiver, while still allowing application code to run in a timely manner. It allows you to access textual metadata in the request body of multipart HTTP requests (ie. JSON/XML/urlencoded request body parameters) but still get streaming access to incoming files without writing .tmp files to disk.
============================================
> This module ~~may~~ **will** be included as a part of the stable release of Sails v0.10. However we need help with documentation, examples, and writing additional receivers (currently receivers for S3 and local disk exist.)
> The decision to include skipper in v0.10 was tough-- it has stalled our release. However, it was a consequence of rewriting this module to use streams2, as well as the spotty/fragmented/confusing state of file uploads around the community. We hope this module helps clear things up for everybody.
## Quick Start
The following example assumes skipper is already installed as the body parser in your Express or Sails app. It receives one or more files from a **file parameter** named `avatar` using the default, built-in file adapter (skipper-disk). This streams the file(s) to the default upload directory `.tmp/uploads/` on the server's local disk.
```javascript
req.file('avatar').upload(function (err, uploadedFiles){
if (err) return res.send(500, err);
return res.send(200, uploadedFiles);
});
```
============================================
### Installation
## Installation
Skipper is installed in [Sails](http://beta.sailsjs.org) automatically (see https://github.com/sails101/file-uploads for a sample Sails app that handles file uploads).
To install Skipper in a vanilla [Express](http://expressjs.org) app:
```sh

@@ -22,122 +34,378 @@ npm install skipper --save

> Skipper is installed in Sails automatically. To install it into another Connect-based app (i.e. Express):
>
> ```js
> app.use(require('skipper')());
> ```
```js
app.use(require('skipper')());
```
============================================
### Quick Start
> ##### Important
>
> Skipper is able to transparently stream your app's file uploads to any of its compatible receivers thanks to a crucial, > simplifying assumption: that all **text parameters** will be sent _before_ any **file parameters**.
## Using req.file()
As is true with most methods on `req` once installed, usage is identical between Sails (in a controller) and Express (in a route).
#### Stream file(s) to disk
```javascript
req.file('avatar').upload(function (err, uploadedFiles) {
if (err) return res.send(500, err);
return res.json({
message: uploadedFiles.length + ' file(s) uploaded successfully!',
files: uploadedFiles
});
});
```
The following example receives a file from a **file parameter** named `avatar`, then streams it `.tmp/test.jpg` on the server's local disk:
#### Options
Option | Type | Description
----------- | -------------------------------- | --------------
dirname | ((string)) | todo
saveAs | ((string)) -or- ((function)) | todo
maxBytes | ((integer)) | todo
============================================
## Use Cases
#### Uploading files to disk
[skipper-disk](https://github.com/balderdashy/skipper-disk) is a file adapter that uploads files to the local hard drive on the server. It is bundled with Skipper, so if an `adapter` option is not specified (as in the [Quick Start]() example above) it is used by default.
```js
// ...
return req.file('foobar').upload('./.tmp/test.jpg', function onUploadComplete (err, uploadedFiles) {
// ...
});
req.file('avatar').upload({
// ...any other options here...
}, ...);
```
To use dynamic filenames (i.e. whatever the name of the original file was), and still contain files within a particular directory (defaults to `.tmp/uploads/` relative to the current working directory):
It exposes the following adapter-specific options:
Option | Type | Description
---------- | -------------------------------- | --------------
onProgress | ((function)) | todo: document
#### Uploading files to S3
```shell
$ npm install skipper-s3 --save
```
[skipper-s3](https://github.com/balderdashy/skipper-s3) is a filesystem adapter which enables Skipper to stream file uploads directly to Amazon S3.
```js
// ...
return req.file('foobar').upload(function onUploadComplete (err, uploadedFiles) {
// ...
});
req.file('avatar').upload({
// ...any other options here...
adapter: require('skipper-s3'),
key: 'YOUR_S3_API_KEY',
secret: 'YOUR_S3_API_SECRET',
bucket: 'YOUR_S3_BUCKET'
}, ...);
```
It exposes the following adapter-specific options:
#### Stream file(s) somewhere else
Option | Type | Description
---------- | -------------------------------- | --------------
key | ((string)) | todo: document
secret | ((string)) | todo: document
bucket | ((string)) | todo: document
onProgress | ((function)) | todo: document
Alternatively, to upload the file with any receiver other than the default [`skipper-disk`](http://github.com/balderdashy/skipper-disk):
#### Uploading files to gridfs
```shell
$ npm install skipper-gridfs --save
```
[skipper-gridfs](https://github.com/willhuang85/skipper-gridfs) is a filesystem adapter which enables Skipper to stream file uploads directly to MongoDB's GridFS.
```js
// ...
var SkipperS3 = require('skipper-s3')({ key: '...', secret: '...', bucket: '...' });
var receiving = SkipperS3.receive();
return req.file('foobar').upload(receiving, function onUploadComplete (err, uploadedFiles) {
// ...
});
req.file('avatar').upload({
// ...any other options here...
adapter: require('skipper-gridfs'),
uri: 'mongodb://[username:password@]host1[:port1][/[database[.bucket]]'
}, ...);
```
It exposes the following adapter-specific options:
Option | Type | Description
--------- | -------------------------------- | --------------
uri | ((string)) | the MongoDB database where uploaded files should be stored (using [mongo client URI syntax](http://api.mongodb.org/java/current/com/mongodb/MongoClientURI.html)) <br/> e.g. `mongodb://jimmy@j1mtr0n1xx@mongo.jimmy.com:27017/coolapp.avatar_uploads`
#### With Sails (v0.10.0)
As of v0.10.0-rc6, skipper is installed as the default request body parser in Sails- you don't need to install it again.
#### Customizing at-rest filenames for uploads
> TODO: document `saveAs`
#### With Sails (v0.9.x)
#### Restricting file size
To use skipper with an existing v0.9.x Sails app, you'll need to install skipper, then modify `config/express.js`:
> TODO: document `maxBytes`
```javascript
module.exports.express = {
bodyParser: require('skipper')
};
<!--
#### Preventing/allowing uploads of a certain file type
> TODO
#### Compressing uploaded files
> TODO
#### Encrypting uploaded files
> TODO
#### Creating thumbnails for uploaded images
> TODO
-->
============================================
## Background
Skipper extends the [HTTP body parser in Express/Sails](https://github.com/expressjs/body-parser).
It parses multipart requests (file uploads) using [andrewrk/node-multiparty](https://github.com/andrewrk/node-multiparty), but instead of buffering to a `tmp` directory on disk, it exposes uploaded files as streams. This allows Skipper to run your application code _before the file uploads are saved_ affording a number of benefits. Additionally, Skipper implements a file adapter specification that can be used to connect "upstreams" with various APIs (Box.net, imgur, whatever) with minimal integration effort.
#### File Adapters
The upstreams you get from `req.file()` can be hooked up to any compatible skipper adapter (e.g. local filesystem, S3, grid-fs). File upload adapters maintain a consistent specification for certain options/features; things like specifying the at-rest filename for a file upload or indicating the maximum upload size for a particular request. However some file adapters may provide additional functionality/options which are not part of the specification-- that's ok too! When possible, new features/options introduced by individual adapters will be standardized and pulled into the core spec, where they can be uniformly tested (in an adapter-agnostic way) using the skipper-adapter-tests module (inspired by the approach taken in [waterline-adapter-tests](https://github.com/balderdashy/waterline-adapter-tests))
#### Stream Processing
Working with upstreams means you can process file uploads in flight-- _before_ they land on your server. This makes features like terrabyte file uploads (or ∞-byte, for that matter) a possibility by allowing you to restrict/limit uploads without receiving the entire thing. You can even tap into the raw file streams to peform streaming encryption, compression, or even image thumbnailing (although currently this requires implementing your own stream processing logic. Simple, one-line configuration to opt into those features is on the roadmap.)
#### Lazy Stream Initialization
Skipper only examines/processes any given file upload for as long as it takes to determine whether it is _actually used_ by your app. For instance, if you don't code `req.file('fake')` in your app code, the stream will be ignored, whether it contains a 25KB image or a 30TB binary payload. This conserves valuable system resources and reducing the effectiveness of would-be DDoS attacks.
#### Text Parameters
Not only do you get access to incoming file uploads as raw streams, Skipper allows you to access the other _non-file_ metadata parameters (e.g "photoCaption" or "commentId") in the conventional way. That includes url/JSON-encoded HTTP body parameters (`req.body`), querystring parameters (`req.query`), or "route" parameters (`req.params`); in other words, all the standard stuff sent in standard AJAX uploads or HTML form submissions. And helper methods like `req.param()` and `req.allParams()` work too.
> It is important to realize that the benefit above **relies on a crucial, simplifying assumption**: that user agents send any **text parameters** _before_ the first **file parameter** in the multipart HTTP request body. For instance, in an HTML form that means putting all of your `<input type="file"/>` tags **after** the other inputs. If you don't want to lay your form out that way, you'll want to use AJAX to submit it instead (see [jQuery File Upload](https://github.com/blueimp/jQuery-File-Upload) / [Angular File Upload](https://github.com/danialfarid/angular-file-upload)) or listen for the form's "submit" event to reorder the inputs before submitting the form to the server.
#### The Big Assumption: Field Order
After a lot of research, @zolmeister, @sgress454 and I came to understand a critical fact about multipart form data: that fields are sent _in order_ by browsers. This is true even as far back as IE6!
+ http://stackoverflow.com/a/7450170/486547
+ http://www.w3.org/TR/html4/interact/forms.html#h-17.13.4
+ https://github.com/jnicklas/capybara/issues/670#issue-3711585
#### Scenarios
I realize there's a lot going on in here, so for sanity/confidence, let's look at some edge cases and explain how Skipper addresses them:
#### EMAXBUFFER
> TODO: document
#### ETIMEOUT
> TODO: document
....others...
#### History
This module ~~may~~ **will** be included as a part of the stable release of Sails v0.10. However we need help with documentation, examples, and writing additional receivers (currently receivers for S3 and local disk exist.)
> The decision to include skipper in Sails v0.10 was tough-- it has stalled our release. However, it was a consequence of rewriting this module to use streams2, as well as the spotty/fragmented/confusing state of file uploads around the community. We hope this module helps clear things up for everybody.
## FAQ
#### How Does the Multipart Body Parser Work?
_When a multipart request is received..._
1. Skipper waits _just long enough_ to identify the first file being uploaded in a request, then gathers up all the text parameters it's seen _so far_ and **runs** your application code (i.e. calls `next()`).
2. In your application code, you call `req.file('foo')` to listen for file uploads sent as the "foo" parameter. This returns an "upstream"-- a stream of binary file streams; one for each file that was uploaded to the specified parameter ("foo") via an HTTP multipart file upload. At that point, if Skipper has already seen files come in as "foo", it will start uploading to the file adapter. This involves replaying the first buffered bytes its received of "foo" file uploads so far and "pumping" those streams so they start sending more bytes; i.e. relieving TCP backpressure. If Skipper has NOT already received any files as "foo", it will start listening for them.
3. When the request ends, or a significant (configurable) period of time has passed since a new file upload has shown up on "foo", the upstream is considered to be complete. If you're using `req.file('foo').upload(function(err,uploadedFiles){ ... })`, the callback will be triggered with `uploadedFiles` as an array of metadata objects describing the files that were uploaded (including the at-rest file descriptor.) If an error occured, the callback will be triggered with `err` instead. (Note that, if you're using the raw upstream directly, you can listen for the "finish" and "error" events to achieve the same effect.)
4. In general, when possible, you should put `req.file()` outside of asynchronous callbacks in your code. However this isn't always possible-- Skipper tolerates this case "holding on" to unrecognized file streams (e.g. "bar") for a configurable period of time. If you don't use `req.file("bar")` right away in your code, Skipper will "hold on" to any files received on that parameter ("bar"). However it __won't pump bytes from those files__ until you use `req.file('bar')` in your code. It does this without buffering to disk/memory by properly implementing Node's streams2 interface, which applies TCP backpressure and _actually slows down_, or even pauses, the upload. If you __never__ use `req.file('bar')` in your code, any unused pending file streams received on "bar" will be discarded when the request ends.
#### What are Upstreams?
An upstream is an [object-mode Readable stream]() of [Readable streams](). It's purpose is to pump out incoming files.
When you call `req.file('foo')`, the upstream for the "foo" parameter in the current request (`req`) is lazy-initialized and returned (subsequent calls to `req.file('foo')` will return the same upstream, calling `req.file('bar')` will return the upstream for "bar", etc.).
```
•---------------------•
| req.file('foo') | <------- an upstream
•----====-------------•
| |
| |
|f2| <------- an incoming file upload stream
| |
| |
|f1| <------- an incoming file upload stream
| |
| |
|f0| <------- an incoming file upload stream
_| |_
\ /
\ /
\/
```
#### With Express/Connect
This module is a drop-in replacement for the default Connect bodyParser, so if you're already using that bodyParser (`app.use(express.bodyParser)`), you'll need to replace it and hook up skipper instead.
e.g. in the module where you set up your middleware:
#### What are Upstream Receivers?
```javascript
// ...
app.use(require('skipper')());
// ...
An **upstream receiver** is an [object-mode Writable stream](http://nodejs.org/api/stream.html#stream_object_mode) of [Writable streams](http://nodejs.org/api/stream.html#stream_class_stream_writable).
It's purpose is to receive incoming files that get pumped out of an upstream. Whenever a new file shows up, the receiver's `_write()` method gets called, and the receiver writes it to the remote filesystem.
```
•-----------------•
| some upstream |
•----====---------•
| |
| |
|f2| <------- an incoming file upload stream
| |
| |
|f1| <------- an incoming file upload stream
| |
| |
|f0| <------- an incoming file upload stream
_| |_
\ /
o----ª==ª----o
| receiver |
o------------o
```
### Usage
#### What are Filesystem Adapters?
`req.file(foo)` returns a stream of binary streams- one for each file that was uploaded to the specified parameter (`foo`) via an HTTP multipart file upload. As is true with most middleware, the usage is identical between Sails and Express.
A **filesystem adapter** is a node module which exports an object with the following functions:
#### With Sails
Method | Description
----------- | ------------------
`ls()` | Get the contents of the specified directory on the remote filesystem (returns an array of string file/folder names)
`read()` | Get a file from the remote filesystem
`rm()` | Remove a file or directory from the remote filesystem
`receive()` | Return an "upstream receiver" which will receive files pumped from an upstream and write them to the remote filesystem
In one of your controller actions:
At present, the first three methods are actually only used by skipper-adapter-tests to test your module. That said, it is unlikely that any of one them involves writing more than ~5-20 lines of code, so it's worth it to create them so you can use the generic test suite. See [skipper-disk](https://github.com/balderdashy/skipper-disk/blob/master/index.js#L27) or [skipper-s3](https://github.com/balderdashy/skipper-s3/blob/master/index.js#L54) for example implementations.
```javascript
// ...
upload: function (req, res) {
req.file('avatar').upload(function (err, files) {
if (err) return res.serverError(err);
return res.json({
message: files.length + ' file(s) uploaded successfully!',
files: files
The most important method is `receive()` -- it builds the upstream receiver which is responsible for writing incoming files to the remote filesystem.
<!--
TODO: simplify the API for building receivers- we could make it so you really only need to define the _write() method
-->
#### Implementing `receive()`
The `receive()` method in a filesystem adapter must build and return a new upstream receiver.
This is the hardest part-- if you implement this, everything else in your adapter is a piece of cake. Here's a quick walk-through of how you can build and return a upstream receiver instance:
> TODO: make sure this fd stuff is working-- may need to roll back docs to match the version on npm and pull this on a branch so as not to confuse anybody while this is in flux
```js
function receive() {
// Build an instance of a writable stream in object mode.
var receiver__ = require('stream').Writable({ objectMode: true });
// This `_write` method is invoked each time a new file is pumped in
// from the upstream. `__newFile` is a readable binary stream.
receiver__._write = function onFile(__newFile, _unused, done) {
// If you are piping to this receiver using Skipper, adapter options like "bucket"
// will be available as `__newFile.options` (otherwise it will be an empty object).
var options = __newFile.options;
// To determine location where file should be written on remote filesystem,
// calculate the output path as follows:
var outputPath = require('path').join(__newFile.dirname, __newFile.fd);
// Then ensure necessary parent directories exist
// ...
// Create/acquire a writable stream to the remote filesystem
// (this may involve using options like "bucket", "secret", etc. to build an appropriate request)
// ...
var outs__ = getWriteStreamSomehow(outputPath, encoding);
// Pump bytes from the incoming file (`__newFile`)
// to the outgoing stream towards the remote filesystem (`outs__`)
__newFile.pipe(outs__);
// Bind `finish` event for when the file has been completely written.
outs.once('finish', function () {
done();
});
// Handle potential error from the outgoing stream
outs__.on('error', function (err) {
// If this is not a fatal error (e.g. certain types of "ECONNRESET"s w/ S3)
// don't do anything.
if ( ! findOutIfIsFatalSomehow(err) ) return;
// The handling of this might vary adapter-to-adapter, but at some point, the
// `done` callback should be triggered w/ a properly formatted error object.
// Since calling `done(err)` will cause an `error` event to be emitted on the receiver,
// it is important to format the error in this way (esp. with code===E_WRITE) so it can
// be detected and discriminated from other types of errors that might be emitted from a
// receiver.
return done({
incoming: __newFile,
outgoing: outs__,
code: 'E_WRITE',
stack: typeof err === 'object' ? err.stack : new Error(err),
name: typeof err === 'object' ? err.name : err,
message: typeof err === 'object' ? err.message : err
});
// If this receiver is being used through Skipper, this event will be intercepted and, if possible
// (if the adapter implements the `rm` method) any bytes that were already written for this file
// will be garbage-collected.
});
}
// ...
```
#### With Express
};
```javascript
app.post('/upload', function uploadAction (req, res) {
req.file('avatar').upload( function (err, files) {
if (err) return res.send(500, err);
return res.json({
message: files.length + ' file(s) uploaded successfully!',
files: files
});
});
});
return receiver__;
}
```
### Status
#### Advanced Usage
You can `.pipe()` an upstream directly to a custom upstream receiver, just bear in mind that you must first intercept the upstream and attach an `fd` (file descriptor) property to each incoming file stream.
============================================
## Status
This module is published on npm. Development takes place on the `master` branch.
============================================
### More Resources
## More Resources

@@ -152,4 +420,5 @@ - [Stackoverflow](http://stackoverflow.com/questions/tagged/sails.js)

============================================
### License
## License

@@ -163,5 +432,80 @@ **[MIT](./LICENSE)**

![bkgd_seaScene.png](http://i.imgur.com/JpaJ8qw.png)
![bkgd_seaScene.png](http://i.imgur.com/JpaJ8qw.png)
[![githalytics.com alpha](https://cruel-carlota.pagodabox.com/a22d3919de208c90c898986619efaa85 "githalytics.com")](http://githalytics.com/balderdashy/file-parser)
[![githalytics.com alpha](https://cruel-carlota.pagodabox.com/a22d3919de208c90c898986619efaa85 "githalytics.com")](http://githalytics.com/balderdashy/skipper)
<!--
#### Using an upstream receiver
Whether you get it from calling `.receive()` on an adapter, or build a quick custom receiver, the usage is the same:
```js
req.file('foobar').upload(myReceiver, function onUploadComplete (err, uploadedFiles) {
// ...
});
```
-or-
```js
req.file('foobar').pipe(myReceiver)
.once('error', function (err){
// ...
})
.once('finish', function (uploadedFiles){
// ...
});
```
-->
<!--
#### With Sails (v0.10.0)
As of v0.10.0-rc6, skipper is installed as the default request body parser in Sails- you don't need to install it again.
#### With Sails (v0.9.x)
To use skipper with an existing v0.9.x Sails app, you'll need to install skipper, then modify `config/express.js`:
```javascript
module.exports.express = {
bodyParser: require('skipper')
};
```
#### With Express/Connect
This module is a drop-in replacement for the default Connect bodyParser, so if you're already using that bodyParser (`app.use(express.bodyParser)`), you'll need to replace it and hook up skipper instead.
e.g. in the module where you set up your middleware:
```javascript
// ...
app.use(require('skipper')());
// ...
```
```js
// ...
return req.file('foobar').upload('./.tmp/test.jpg', function onUploadComplete (err, uploadedFiles) {
// ...
});
```
To use dynamic filenames (i.e. whatever the name of the original file was), and still contain files within a particular directory (defaults to `.tmp/uploads/` relative to the current working directory):
```js
// ...
return req.file('foobar').upload(function onUploadComplete (err, uploadedFiles) {
// ...
});
```
-->

2

test/default.adapter.test.js

@@ -73,3 +73,3 @@ /**

var filesUploaded = fsx.readdirSync(suite.outputDir.path);
assert.equal(filesUploaded.length,1, 'Expected `filesUploaded.length` === 1, but it was '+filesUploaded.length+'\nHere is the complete list: \n'+util.inspect(filesUploaded, false, null));
assert.equal(filesUploaded.length,1, 'Expected `filesUploaded.length` === 1, but after listing contents of "'+suite.outputDir.path+'", noticed there were '+filesUploaded.length+'\nHere is the complete list: \n'+util.inspect(filesUploaded, false, null));

@@ -76,0 +76,0 @@ // Check that its contents are correct

2

test/form.test.js

@@ -10,3 +10,3 @@ /**

var crypto = require('crypto');
var log = require('../lib/logger');
var log = require('../standalone/logger');
var async = require('async');

@@ -13,0 +13,0 @@ var path = require('path');

143

test/helpers/lifecycle.js

@@ -5,14 +5,13 @@ /**

var fsx = require('fs-extra')
, _ = require('lodash')
, tmp = require('temporary')
, crypto = require('crypto')
, path = require('path')
, Express = require('express')
, http = require('http');
var path = require('path');
var http = require('http');
var crypto = require('crypto');
var _ = require('lodash');
var fsx = require('fs-extra');
var Express = require('express');
var tmp = require('temporary');
var Skipper = require('../../');
var FileParser = require('../../');
/**

@@ -27,84 +26,84 @@ * [exports description]

*/
module.exports = function () {
// Create an array of file fixtures.
var fileFixtures = [];
module.exports = function() {
// Making a seperate group of larger files.
var bigFileFixtures = [];
// Create a tmp directory for our uploads to live in.
var outputDir = new tmp.Dir();
// Create an array of file fixtures.
var fileFixtures = [];
// Making a seperate group of larger files.
var bigFileFixtures = [];
// Expose some things
var public = {
// Create a tmp directory for our uploads to live in.
var outputDir = new tmp.Dir();
srcFiles: fileFixtures,
bigSrcFiles: bigFileFixtures,
// Expose some things
var public = {
outputDir: outputDir,
srcFiles: fileFixtures,
setup: function(done) {
bigSrcFiles: bigFileFixtures,
// Write nonsense bytes to our file fixtures.
for (var bytes=10; bytes < 10000000; bytes*=10) {
var f = new tmp.File();
var EOF = '\x04';
f.writeFileSync(crypto.pseudoRandomBytes(bytes)+EOF);
f.size = bytes;
fileFixtures.push(f);
}
outputDir: outputDir,
//5 MB?
var bigFileSize = 7 * 1000 * 1000;
setup: function(done) {
for (var i = 0; i < 1; i++){
var f = new tmp.File();
var EOF = '\x04';
f.writeFileSync(crypto.pseudoRandomBytes(bigFileSize)+EOF);
f.size = bigFileSize;
bigFileFixtures.push(f);
}
// Write nonsense bytes to our file fixtures.
for (var bytes = 10; bytes < 10000000; bytes *= 10) {
var f = new tmp.File();
var EOF = '\x04';
f.writeFileSync(crypto.pseudoRandomBytes(bytes) + EOF);
f.size = bytes;
fileFixtures.push(f);
}
// Bootstrap a little express app that uses file-parser
// to upload files to our outputDir
public.app = Express();
//5 MB?
var bigFileSize = 7 * 1000 * 1000;
// Use file-parser middleware
public.app.use( FileParser() );
for (var i = 0; i < 1; i++) {
var f = new tmp.File();
var EOF = '\x04';
f.writeFileSync(crypto.pseudoRandomBytes(bigFileSize) + EOF);
f.size = bigFileSize;
bigFileFixtures.push(f);
}
// Provide a default outputPath for testing purposes
// (gets used by the test receiver to write the test file to disk in each suite)
public.app.use(function (req, res, next) {
req.__FILE_PARSER_TESTS__OUTPUT_PATH__AVATAR = path.join(outputDir.path, 'avatar.jpg');
req.__FILE_PARSER_TESTS__OUTPUT_PATH = outputDir.path;
next();
});
// Bootstrap a little express app that uses file-parser
// to upload files to our outputDir
public.app = Express();
// Lift Express server on 3000
public.server =
http.createServer(public.app)
.listen(3000)
.on('listening', done);
},
// Use file-parser middleware
public.app.use(Skipper());
teardown: function () {
// Provide a default outputPath for testing purposes
// (gets used by the test receiver to write the test file to disk in each suite)
public.app.use(function(req, res, next) {
req.__FILE_PARSER_TESTS__OUTPUT_PATH__AVATAR = path.join(outputDir.path, 'avatar.jpg');
req.__FILE_PARSER_TESTS__OUTPUT_PATH = outputDir.path;
next();
});
// Clean up fixtures.
_.each(fileFixtures, function (f) {
f.unlinkSync();
});
// Clean up directory w/ test output.
fsx.removeSync(outputDir.path);
// Lift Express server on 3000
public.server =
http.createServer(public.app)
.listen(3000)
.on('listening', done);
},
// Lower Express server
public.server.close();
}
};
teardown: function() {
return public;
// Clean up fixtures.
_.each(fileFixtures, function(f) {
f.unlinkSync();
});
// Clean up directory w/ test output.
fsx.removeSync(outputDir.path);
// Lower Express server
public.server.close();
}
};
return public;
};

26

test/helpers/receiver.js

@@ -1,17 +0,11 @@

var log = require('../../lib/logger')
, fsx = require('fs-extra');
/**
* Module dependencies
*/
module.exports = {
var fsx = require('fs-extra');
var log = require('../../standalone/logger');
/**
* Build a mock readable stream that emits incoming files.
* (used for file downloads)
*
* @return {Stream.Readable}
*/
newEmitterStream: function newEmitterStream () {
// TODO:
},
module.exports = {

@@ -22,6 +16,6 @@

* (used for file uploads)
*
*
* @return {Stream.Writable}
*/
newReceiverStream: function newReceiverStream (options) {

@@ -43,3 +37,3 @@

log(('Receiver: Error writing `'+__newFile.filename+'`:: '+ require('util').inspect(err)+' :: Cancelling upload and cleaning up already-written bytes...').red);
// Garbage-collects the already-written bytes for this file.

@@ -55,3 +49,3 @@ blobAdapter.rm({id: options.id}, function (rmErr) {

};
return receiver__;

@@ -58,0 +52,0 @@ }

8

test/helpers/uploader.js

@@ -5,4 +5,4 @@ /**

var Request = require('request')
, _ = require('lodash');
var Request = require('request');
var _ = require('lodash');

@@ -12,3 +12,3 @@

/**
*
*
* @param {Object} optionalOptions

@@ -19,3 +19,3 @@ * @param {Function} optionalCallback

module.exports = function getUploader ( optionalOptions, optionalCallback ) {
var opts = optionalOptions || {};

@@ -22,0 +22,0 @@ _.defaults(opts, {

examples/diskReceiver/index.js
examples/express/app.js
examples/README.md
examples/s3Receiver/index.js
examples/sails/api/controllers/FileController.js
examples/sails/app.js
examples/sails/README.md
lib/logger.js
lib/Upstream/build-or-normalize-receiver.js
lib/Upstream/index.js
lib/Upstream/prototype._read.js
lib/Upstream/prototype.fatalIncomingError.js
lib/Upstream/prototype.noMoreFiles.js
lib/Upstream/prototype.pipe.js
lib/Upstream/prototype.serializeFiles.js
lib/Upstream/prototype.upload.js
lib/Upstream/prototype.writeFile.js
lib/Upstream/Upstream.js
test/meta.test.js
SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap
  • Changelog

About

Packages

npm

Go

Maven

PyPI

Rubygems

Stay in touch

Get open source security insights delivered straight into your inbox.

  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc