Configurable S3 or file system image storage and processing HTTP API server. It is using HTTP Thumbnailer as image processing backend.
validate_uri_hmac and validate_header_hmac supportoutput_store_url support additional arguments: scheme, port and hostoutput_store_uri support addedoutput_store_url URL formatting for file_storeoutput_data_uri_image supportsource_failover support400 Bad Request for non UTF-8 characters encoded in URLsoutput_store_path and output_store_url path argument supportoutput_ok and output_text supportidentify statement support (requires HTTP Thumbnailer v1.1.0 or higher)imagename to image_name, mimeextension to image_mime_extension, digest to input_digest (old names can still be used)uuid, input_sha256, image_digest, image_sha256, image_width, image_height, input_image_mime_extension, input_image_width, input_image_heightHTTP Image Store is released as gem and can be installed from RubyGems with:
gem install httpimagestore
To start HTTP Image Store server you need to prepare API configuration first. Configuration is written in SDL format.
Configuration consists of:
Configures HTTP Thumbnailer client. It will be used to perform image processing operations with thumbnail and identify statements.
Options:
url - URL of HTTP Thumbnailer service; default: http://localhost:3100Example:
thumbnailer url="http://10.2.2.2:3100"
Configures Amazon S3 client for S3 object storage and retrieval.
Options:
key - API key to usesecret - API secret for given keyssl - if true SSL protected connection will be used; source and storage URL will begin with http://; default: trueExample:
s3 key="AIAITCKMELYWQZPJP7HQ" secret="V37lCu0F48Tv9s7QVqIT/sLf/wwqhNSB4B0Em7Ei" ssl=false
This directive is used to define storage and retrieval paths that will be used when storing and sourcing file on file system or S3 service. They may also be used for generating output paths or URIs.
You can declare one path per statement or use {} brackets syntax to define more than one with single statement - they are semantically equal.
Arguments:
#{}Variables:
input_digest - input image digest based on it's content only; this is first 16 hex characters from SHA256 digest; ex. 9f86d081884c7d65digest - same as input_digest; deprecated, please use input_digest insteadinput_sha256 - input image digest based on it's content only; this is whole hex encoded SHA256 digest; ex. 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08input_image_mime_extension - extension of input image determined from image mime typeinput_image_width - width in pixels of the input imageinput_image_height - height in pixels of the input imageimage_digest - digest of image being stored based on it's content only; this is first 16 hex characters from SHA256 digest; ex. 9f86d081884c7d65image_sha256 - digest of image being stored based on it's content only; this is whole hex encoded SHA256 digest; ex. 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08image_name - name of the image that is being stored or sourcedimagename - same as image_name; deprecated, please use image_name insteadimage_mime_extension - extension of image being stored determined from image mime typemimeextension - same as image_mime_extension; deprecated, please use image_mime_extension insteadimage_width - width in pixels of the image that is being storedimage_height - height in pixels of the image that is being storedpath - remaining (unmatched) request URL pathbasename - name of the file without it's extension determined from pathdirename - name of the directory determined form pathextension - file extension determined from pathuuid - unique ID for request; ex. 1b68a767-8163-425a-a32c-db6ed7200343Notes:
identify statement.Example:
path "uri" "#{path}"
path "hash" "#{input_digest}.#{extension}"
path {
"hash-name" "#{input_digest}/#{image_name}.#{extension}"
"structured" "#{dirname}/#{input_digest}/#{basename}.#{extension}"
"structured-name" "#{dirname}/#{input_digest}/#{basename}-#{image_name}.#{extension}"
}
This group of statements allows to configure single API endpoint. Each endpoint can have one or more operation defined that will be performed on HTTP request matching that endpoint.
Endpoints will be evaluated in order of definition until one is matched or 404 Not Found is returned.
Statement should start with one of the following HTTP verbs in lowercase: get, post, put, delete, followed by list of URI segment matchers:
<string> - <string> literally match against URI segment in that position for the endpoint to be evaluated:<symbol> - match any URI segment in that position and store matched segment value in variable named <symbol>:<symbol>?[defalut] - optionally match URI segment in that position and store matched segment value in variable named <symbol>; request URI may not contain segment in that position (usually at the end of URI) to be matched for this endpoint to be evaluated; if [default] value is specified it will be used when no value was found in the URI, otherwise empty string will be used:<symbol>/<regexp>/ - match remaining URI using / surrounded regular expression and store matched segments in variable named <symbol>/<regexp>/ - match remaining URI using / surrounded regular expression and store zero or more named captures in variable named as corresponding capture name; e.g. /(?<my_number>[0-9]+)/ - match a number and store it in my_number variable&<key>=<value> - match this endpoint when query string contains key <key> with value of <value> and store the <value> in variable named <key>&:<key> - match query string parameter of key <key> and store it's value in variable named <key>&:<key>?[default] - optionally match query string parameter of key <key>; when [default] is specified it will be used as variable value, otherwise empty string will be usedNote that any remaining unmatched URI part is stored in path variable.
Variable query_string_options is build from all query string parameters and can be used to specify options to HTTP Thumbnailer.
Note that variables can get overwritten in order of evaluation:
path variable valuequery_string_options variable valueNote that URI components are URI decoded after they are matched. Query string parameter values are URI decoded before they are matched.
Example:
get "thumbnail" "v1" ":path" ":operation" ":width" ":height" ":options?" {
}
put "thumbnail" "v1" ":thumbnail_class/small|large/" {
}
post {
}
In this example first endpoint will be matched for GET request with URI like /thumbnail/v1/abc.jpg/fit/100/100 or /thumbnail/v1/abc.jpg/pad/100/100/background-color:green.
Second endpoint will be matched only for PUT requests with URIs beginning with /thumbnail/v1/small and /thumbnail/v1/large.
Third endpoint will match any POST request.
First any defined validators are executed against request to see if we should accept the request or reject it.
This statement sets up HMAC based URI authentication. If HMAC provided with the request does not have the expected value the server will return 403 Forbidden response.
Arguments:
Options:
secret - shared secret used to generate HMACdigest - digest used for HMAC; see OpenSSL digests for valid values; default: sha1 (for HMAC-SHA1 scheme)exclude - comma separated list of query string parameter keys that will be excluded form HMAC computationremove - comma separated list of query string parameter keys that will be removed from request for further processingexclude is useful when your URI contains extra query string parameters not related to this API that would otherwise brake HMAC computation.
remove can be used to remove all extra (apart form the HMAC parameter itself) HMAC related query string parameters like date or nonce so they don't interact with further statements.
Adding additional nonce or date query string parameter can help with HMAC security by 'randomizing' it's value for same URIs - this won't play well with caching though.
Example:
get "small" {
validate_uri_hmac "hmac" secret="key"
}
In this example request URI /small?hmac=a49182838f1b0b306614cea1d0fbcdab980717f7 will be valid.
The actual HMAC is computed from /small URI and secret key using HMAC-SHA1 scheme.
Note that hmac query string parameter will be removed from the request and won't be accessible by further statements.
This statement is identical in function as validate_uri_hmac but takes value for URI from given request header instead.
This is useful when HTTP Image Store is behind proxy that rewrites URIs. This would normally brake HMAC computation but if it can store the original URI in a header this can be used instead for HMAC computation.
Arguments:
Options:
secret - shared secret used to generate HMACdigest - digest used for HMAC; see OpenSSL digests for valid values; default: sha1 (for HMAC-SHA1 scheme)exclude - comma separated list of query string parameter keys that will be excluded form HMAC computationremove - comma separated list of query string parameter keys that will be removed from request for further processingExample:
get "small" {
validate_header_hmac "X-ORIGINAL-URI" "hmac" secret="key"
}
In this example request URI /my-rewritten-uri?hmac=a49182838f1b0b306614cea1d0fbcdab980717f7 will be valid if request comes with X-ORIGINAL-URI header of value /small.
Note that the actual request URI needs only to contain valid hamc query string parameter while the value of header needs only the URI necessary for validation - it is OK if it contains the hmac query string parameter since it will be removed before computation.
remove will operate on request as normal. exclude will operate on the value of the header for the computation to be valid.
Sourcing will load images into memory for further processing. Each image is stored under predefined or user defined name.
If endpoint HTTP verb is post or put then image data will be sourced from request body. It can be referenced using input name.
With this statements image can be sourced from file system.
Arguments:
Options:
root - file system path under which the images are looked uppath - name of predefined path that will be used to locate the image fileExample:
path "myimage" "myimage.jpg"
get "small" {
source_file "original" root="/srv/images" path="myimage"
}
Requesting /small URI will result with file /srv/images/myimage.jpg loaded into memory under original name.
This statement can be used to load images from S3 bucket.
To use this bucket global s3 statement needs to be used in top level to configure S3 client.
Arguments:
Options:
bucket - name of bucket to source image frompath - name of predefined path that will be used to generate key to object to sourceprefix - prefix object key with given prefix value; this does not affect format of output URL; prefix will not be included in source path output; default: emptycache-root - path to directory where S3 objects and meta-data will be cached when sourced from S3; read-through mode; if required information was found in cache object no S3 requests will be made; same directory can be used with different buckets since cache key consists of S3 bucket name and object keyExample:
s3 key="AIAITCKMELYWQZPJP7HQ" secret="V37lCu0F48Tv9s7QVqIT/sLf/wwqhNSB4B0Em7Ei" ssl=false
path "myimage" "myimage.jpg"
get "small" {
source_s3 "original" bucket="mybucket" path="myimage"
}
Requesting /small URI will result with image fetched from S3 bucket mybucket and key myimage.jpg and named original.
This statement can be used to wrap around other sources. HTTP Image Store will try wrapped sources one by one until one will succeed. If all sources fail status code for error from the first source will be returned.
This source has no options.
Example:
s3 key="AIAITCKMELYWQZPJP7HQ" secret="V37lCu0F48Tv9s7QVqIT/sLf/wwqhNSB4B0Em7Ei" ssl=false
path "myimage" "myimage.jpg"
get "small" {
source_failover {
source_s3 "original" bucket="mybucket" path="myimage"
source_s3 "original" bucket="mybucket-backup" path="myimage"
}
}
Requesting /small URI will result with image fetched from S3 bucket mybucket.
If mybucket does not contain key myimage.jpg than mybucket-backup will be tried.
If mybucket-backup does not contain the key than 404 status code will be returned.
This source will provide new images based on already sourced images by processing them with HTTP Thumbnailer backend. This statement can be used to do single thumbnail operation or use multipart output API of the HTTP Thumbnailer when multiple operation are defined. For more informations of meaning of options see HTTP Thumbnailer documentation.
Arguments:
Options:
operation - backend supported thumbnailing operation like pad, fit, limit, cropwidth - requested thumbnail width in pixelsheight - requested thumbnail height in pixelsformat - format in which the resulting image will be encoded; all backend supported formats can be specified like jpeg or png; default: jpegoptions - list of options in format key:value[,key:value]* to be passed to thumbnailer; this can be a list of any backend supported options like background-color or qualityedits - list of edits in format supported by HTTP Thumbnailer ([!<edit name>[,<edit arg>]*[,<edit option>:<edit option value>]*]*)Note that you can use #{variable} expansion within all of this options.
Example:
put ":operation" ":width" ":height" ":options" {
thumbnail "input" {
"original" operation="#{operation}" width="#{width}" height="#{height}" options="#{options}" quality=84 format="jpeg"
"small" operation="crop" width=128 height=128 format="jpeg"
"padded" operation="pad" width=128 height=128 format="png" background-color="gray"
}
}
Putting image under /pad/128/256/backgroud-color=green URI will result with three thumbnails generated with single HTTP Thumbnailer request from input image into original, small and padded references.
original image will have width of 128 pixels and height of 256 pixels; the image will be centered and padded with color green background to match this dimensions; it will be a JPEG image, saved with compression quality of 84.
You can also use shorter form that will perform only one thumbnailing operation per statement using HTTP Thumbnailer's single thumbnail API:
put ":operation" ":width" ":height" ":options" {
thumbnail "input" "original" operation="#{operation}" width="#{width}" height="#{height}" options="#{options}" quality=84 format="jpeg"
}
thumbnail statement supports sub statement edit that will apply additional edits to the thumbnail image.
This edits are applied before edits option specified edits.
Arguments:
Options
Example:
post "example" "&:rotate?0" "&:crop_x?0.0" "&:crop_y?0.0" "&:crop_w?1.0" "&:crop_h?1.0" "&:edits?" {
thumbnail "input" "thumbnail" operation="pad" width=100 height=100 format="jpeg" edits="#{edits}" {
edit "rotate" "#{rotate}"
edit "crop" "#{crop_x}" "#{crop_y}" "#{crop_w}" "#{crop_h}"
}
output_image "thumbnail"
}
In this example each thumbnail will be first rotated by rotate query string parameter value and cropped by query string provided values crop_x, crop_y, crop_w, crop_h.
If query string parameter edits is provided it will be used to do additional edits after rotation and cropping is applied.
Example URI can look like this: /example?rotate=90&crop_x=0.1&crop_y=0.1&crop_w=0.8&crop_h=0.8&edits=pixelate,0.2,0.1,0.6,0.6,size:0.03!rectangle,0.1,0.8,0.8,0.1,color:blue.
Posted image will be rotated by 90 degree. Than the result will be cropped by 10% from each sied. Next middle region of the image will be pixelated and then blue rectangle will be drawn at the bottom of it. Finally thumbnail of dimensions 100 by 100 will be generated and provided as JPEG encoded image in response body.
This statement allows for image mime type, width and height identification based on image content with use of HTTP Thumbnailer. This is usefull when image is not taking part in thumbnailing process but variables that are based on meta data needs to be used for storage.
Example:
path "size" "#{image_width}x#{image_height}.#{image_mime_extension}"
put "images" {
identify "input"
store_s3 "input" bucket="mybucket" path="size"
output_store_path "input"
}
In this example input image is indentified by HTTP Thumbnailer. Received information is used to populate image_width, image_height and image_mime_extension variables used to generate S3 storage key.
This statements are executed after all source statements are finished. They allow storing of any sourced images by specifying their references.
This statement can store image in file system.
Arguments:
Options:
root - file system path under which the images are storedpath - name of predefined path that will be used to store the image file under; relative to rootExample:
path "imagename" "#{image_name}"
path "hash" "#{input_digest}"
put "store" ":name" {
store_file "input" root="/srv/images" path="imagename"
store_file "input" root="/srv/images" path="hash"
}
Putting image data to /store/hello.jpg will store two copies of the image: one under /srv/images/hello.jpg and second under its digest like /srv/images/2cf24dba5fb0a30e.
S3 bucket can also be used for image storage.
To use this bucket top level s3 statement needs to be used to configure S3 client.
Arguments:
Options:
bucket - name of bucket to store image inpath - name of predefined path that will be used to generate key to store object underpublic - if set to true the image will be readable by everybody; this affects fromat of output URL; default: falseprefix - prefix storeage key with given prefix value; this does not affect fromat of output URL; prefix will not be included in storage path output; default: emptycache-root - path to directory where stored S3 objects and meta-data will be cached for future sourcing with source_s3; write-through mode; note that same directory needs to be configured with corresponding source_s3 statementExample:
s3 key="AIAITCKMELYWQZPJP7HQ" secret="V37lCu0F48Tv9s7QVqIT/sLf/wwqhNSB4B0Em7Ei" ssl=false
path "imagename" "#{image_name}"
path "hash" "#{input_digest}"
put ":name" {
store_s3 "input" bucket="mybucket" path="imagename"
store_s3 "input" bucket="mybucket" path="hash"
}
Putting image data to /store/hello.jpg will store two copies of the image under mybucket: one under key hello.jpg and second under its digest like 2cf24dba5fb0a30e.
After all images are stored the output statement is processed.
It is responsible for generating HTTP response for the API endpoint.
If no output statement is specified the server will respond with 200 OK and OK in response body.
This statement will produce custom text/plain response.
Arguments:
#{variable} expansionOptions:
status - numerical value for HTTP response status; default: 200cache-control - value for the Cache-Control response headerThis is the default output if none is specified.
It will always produce 200 status HTTP response with OK in it's body.
This statement will produce 200 OK response containing referenced image data.
It will set Content-Type header to mime-type of the image that was determined by HTTP Thumbnailer based on image data if image was used as input in thumbnailing or it is resulting thumbnail. If the mime-type is unknown application/octet-stream will be used.
Arguments:
Options:
cache-control - value of response Cache-Control header can be specified with this optionExample:
put "test" {
output_image "input" cache-control="public, max-age=999, s-maxage=666"
}
The output will contain the posted image with Content-Type header set to application/octet-stream and Cache-Control to public, max-age=999, s-maxage=666.
This statement will produce 200 OK response containing base64 encoded image data within data URI in format:
data:<image mime type>;base64,<base64 encoded image data>
No tailing new line/line feed is added to the output.
This may be used to include images directly in HTML pages or CSS.
The Content-Type of the response will be text/uri-list.
Image mime type will be provided within data URI and must be known before output is constructed or 500 Internal Server Error will be served.
Arguments:
Options:
cache-control - value of response Cache-Control header can be specified with this optionExample:
put "test" {
identify "input"
output_data_uri_image "input" cache-control="public, max-age=31557600, s-maxage=0"
}
The output will contain base64 encoded posted image within data URI with Content-Type header set to text/uri-list and Cache-Control to public, max-age=999, s-maxage=666.
This statement will output actual storage path on the file system (without root) or S3 key under witch the image was stored.
The Content-Type of response will be text/plain.
You can specify multiple image names to output multiple paths of referenced images, each ended with \r\n.
Arguments:
Options:
path - name of predefined path that will be used to generate output path; #{path} variable and all derivative will be replaced with given image storage path; if not specified the actual storage path will be providedExample:
path "out" "test.out"
put "single" {
store_file "input" root="/srv/images" path="out"
output_store_path "input"
}
Putting image data to /single URI will result with image data stored under /srv/images/test.out. The response body will consist of \r\n ended line containing test.out.
s3 key="AIAITCKMELYWQZPJP7HQ" secret="V37lCu0F48Tv9s7QVqIT/sLf/wwqhNSB4B0Em7Ei" ssl=false
path {
"in" "test.in"
"out" "test.out"
"out2" "test.out2"
}
put "multi" {
source_file "original" root="/srv/images" path="in"
store_file "input" root="/srv/images" path="out"
store_file "original" root="/srv/images" path="out2"
output_store_path {
"input"
"original"
}
}
Putting image data to /multi URI will result with image original sourced from /srv/images/test.in and stored under /srv/images/out2. The input data will also be stored under /srv/images/test.out. The response body will contain \r\n ended lines: test.out and test.out2.
This is similar statement to output_store_file but it will output file: URL for file stored images and valid S3 access URL for S3 stored images.
For S3 stored image if ssl is set to true on the S3 client statement (s3 ssl="true") the URL will start with https://. If public is set to true when storing image in S3 (store_s3 public="true") then the URL will not contain query string options, otherwise authentication tokens and expiration token (set to expire in 20 years) will be include in the query string.
The Content-Type header of this response is text/uri-list.
Each output URL is \r\n ended.
Arguments:
Options:
scheme - rewrite URL scheme with provided one; variables can be used; #{path} variable and all derivative will be replaced with given image storage path; #{url} variable will contain full original URLhost - add/rewrite provided host to the URL; variables can be used; #{path} variable and all derivative will be replaced with given image storage path; #{url} variable will contain full original URLport - add provided port to the URL; variables can be used; #{path} variable and all derivative will be replaced with given image storage path; #{url} variable will contain full original URLpath - name of predefined path that will be used to generate output URL path part; #{path} variable and all derivative will be replaced with given image storage path; if not specified the original URL will be provided; #{url} variable will contain full original URLExample:
s3 key="AIAITCKMELYWQZPJP7HQ" secret="V37lCu0F48Tv9s7QVqIT/sLf/wwqhNSB4B0Em7Ei" ssl=false
path "hash" "#{input_digest}.#{image_mime_extension}"
path "hash-name" "#{input_digest}/#{image_name}.#{image_mime_extension}"
put "thumbnail" {
thumbnail "input" {
"small" operation="crop" width=128 height=128 format="jpeg"
"tiny_png" operation="crop" width=32 height=32 format="png"
}
store_s3 "input" bucket="mybucket" path="hash" public=true
store_s3 "small" bucket="mybucket" path="hash-name" public=true
store_s3 "tiny_png" bucket="mybucket" path="hash-name" public=true
output_store_url {
"input"
"small"
"tiny_png"
}
}
Putting image data will result in storage of that image and two thumbnails generated from it. The output will contain \r\n ended lines like:
http://mybucket.s3.amazonaws.com/4006450256177f4a.jpg
http://mybucket.s3.amazonaws.com/4006450256177f4a/small.jpg
http://mybucket.s3.amazonaws.com/4006450256177f4a/tiny_png.png
This is similar statement to output_store_url but it will output only path component of the URL for stored images.
The Content-Type header of this response is text/uri-list.
Each output URL is \r\n ended.
Arguments:
Options:
path - name of predefined path that will be used to generate output URL path part; #{path} variable and all derivative will be replaced with given image storage path; if not specified the original URL will be provided; #{url} variable will contain full original URLExample:
s3 key="AIAITCKMELYWQZPJP7HQ" secret="V37lCu0F48Tv9s7QVqIT/sLf/wwqhNSB4B0Em7Ei" ssl=false
path "hash" "#{input_digest}.#{image_mime_extension}"
path "hash-name" "#{input_digest}/#{image_name}.#{image_mime_extension}"
put "thumbnail" {
thumbnail "input" {
"small" operation="crop" width=128 height=128 format="jpeg"
"tiny_png" operation="crop" width=32 height=32 format="png"
}
store_s3 "input" bucket="mybucket" path="hash" public=true
store_s3 "small" bucket="mybucket" path="hash-name" public=true
store_s3 "tiny_png" bucket="mybucket" path="hash-name" public=true
output_store_uri {
"input"
"small"
"tiny_png"
}
}
Putting image data will result in storage of that image and two thumbnails generated from it. The output will contain \r\n ended lines like:
/4006450256177f4a.jpg
/4006450256177f4a/small.jpg
/4006450256177f4a/tiny_png.png
Additional meta options can be used with selected statements.
It can be used with all source, storage and file/url output statements.
The argument will expand #{variable}.
If specified on or withing given statement it will cause that statement or it's part to be ineffective unless the image name used within the statement is on the list of image names specified as value of the option.
The list is in format image name[,image name]*.
This option is useful when building API that works on predefined set of image operations and allows to select witch set of operations to perform with list included in the URL.
It can be used on most operation statements and sub statements (edit).
The argument can contain name of variable to or be in format <name of variable>:<expected value>.
In first from the statement will be executed if variable of given name exists and is non empty.
In second form the statement will be executed if variable value equals to expected value.
This option is useful for example to selectively execute statements based on query string parameters.
Features two storage approaches: with JPEG conversion and limiting in size - for user provided content - and storing literally. POST requests will end up with server side generated storage key based on input data digest. PUT requests can be used to store image under provided storage key. Thumbnail GET API is similar to described in Facebook APIs for thumbnailing. Stored object extension and content type is determined from image data. S3 objects are cached on storage and on read if not cached already (read-through/write-through cache).
s3 key="AIAITCKMELYWQZPJP7HQ" secret="V37lCu0F48Tv9s7QVqIT/sLf/wwqhNSB4B0Em7Ei" ssl=false
path "hash" "#{input_digest}.#{image_mime_extension}"
path "path" "#{path}"
## User uploaded content - always JPEG converted, not bigger than 2160x2160 and in hight quality compression
post "pictures" {
thumbnail "input" "original" operation="limit" width=2160 height=2160 format="jpeg" quality=95
store_s3 "original" bucket="mybucket" path="hash" cache-root="/var/cache/httpimagestore"
output_store_path "original"
}
put "pictures" {
thumbnail "input" "original" operation="limit" width=2160 height=2160 format="jpeg" quality=95
store_s3 "original" bucket="mybucket" path="path" cache-root="/var/cache/httpimagestore"
output_store_path "original"
}
## Uploaded by admin for use on the website for example - store whatever was send
post "images" {
identify "input"
store_s3 "input" bucket="mybucket" path="hash" cache-root="/var/cache/httpimagestore"
output_store_path "input"
}
put "images" {
identify "input"
store_s3 "input" bucket="mybucket" path="path" cache-root="/var/cache/httpimagestore"
output_store_path "input"
}
## Thumbailing - keep input format; default JPEG quality is 85
### Thumbnail specification from query string paramaters
get "pictures" "&:width" "&:height" "&:operation?crop" "&:background-color?white" {
source_s3 "original" bucket="mybucket" path="path" cache-root="/var/cache/httpimagestore"
thumbnail "original" "thumbnail" operation="#{operation}" width="#{width}" height="#{height}" options="background-color:#{background-color}"
output_image "thumbnail" cache-control="public, max-age=31557600, s-maxage=0"
}
get "pictures" "&:width" "&:height?1080" "&:operation?fit" "&:background-color?white" {
source_s3 "original" bucket="mybucket" path="path" cache-root="/var/cache/httpimagestore"
thumbnail "original" "thumbnail" operation="#{operation}" width="#{width}" height="#{height}" options="background-color:#{background-color}"
output_image "thumbnail" cache-control="public, max-age=31557600, s-maxage=0"
}
get "pictures" "&:height" "&:width?1080" "&:operation?fit" "&:background-color?white" {
source_s3 "original" bucket="mybucket" path="path" cache-root="/var/cache/httpimagestore"
thumbnail "original" "thumbnail" operation="#{operation}" width="#{width}" height="#{height}" options="background-color:#{background-color}"
output_image "thumbnail" cache-control="public, max-age=31557600, s-maxage=0"
}
### Predefined thumbnailing specification
get "pictures" "&type=square" {
source_s3 "original" bucket="mybucket" path="path" cache-root="/var/cache/httpimagestore"
thumbnail "original" "thumbnail" operation="crop" width="50" height="50"
output_image "thumbnail" cache-control="public, max-age=31557600, s-maxage=0"
}
get "pictures" "&type=small" {
source_s3 "original" bucket="mybucket" path="path" cache-root="/var/cache/httpimagestore"
thumbnail "original" "thumbnail" operation="fit" width="50" height="2000"
output_image "thumbnail" cache-control="public, max-age=31557600, s-maxage=0"
}
get "pictures" "&type=normall" {
source_s3 "original" bucket="mybucket" path="path" cache-root="/var/cache/httpimagestore"
thumbnail "original" "thumbnail" operation="fit" width="100" height="2000"
output_image "thumbnail" cache-control="public, max-age=31557600, s-maxage=0"
}
get "pictures" "&type=large" {
source_s3 "original" bucket="mybucket" path="path" cache-root="/var/cache/httpimagestore"
thumbnail "original" "thumbnail" operation="fit" width="200" height="2000"
output_image "thumbnail" cache-control="public, max-age=31557600, s-maxage=0"
}
## By default serve original image as is - JPEG for user content and what was send for admin uploaded images
get "pictures" {
source_s3 "original" bucket="mybucket" path="path" cache-root="/var/cache/httpimagestore"
output_image "original" cache-control="public, max-age=31557600, s-maxage=0"
}
For more information see flexi feature test.
This example can be used to create new API and migration between old and new approach. It presents API compatible with previous version of httpimagestore (v0.5.0) and one possible thumbnail on demand approach API.
s3 key="AIAITCKMELYWQZPJP7HQ" secret="V37lCu0F48Tv9s7QVqIT/sLf/wwqhNSB4B0Em7Ei" ssl=false
# Compatibility API
path "compat-hash" "#{input_digest}.#{image_mime_extension}"
path "compat-hash-name" "#{input_digest}/#{image_name}.#{image_mime_extension}"
path "compat-structured" "#{dirname}/#{input_digest}/#{basename}.#{image_mime_extension}"
path "compat-structured-name" "#{dirname}/#{input_digest}/#{basename}-#{image_name}.#{image_mime_extension}"
put "thumbnail" ":name_list" ":path/.+/" {
thumbnail "input" {
# Make limited source image for migration to on demand API
"migration" operation="limit" width=2160 height=2160 format="jpeg" quality=95
# Backend classes
"original" operation="crop" width="input" height="input" format="jpeg" options="background-color:0xF0F0F0" if-image-name-on="#{name_list}"
"search" operation="pad" width=162 height=162 format="jpeg" options="background-color:0xF0F0F0" if-image-name-on="#{name_list}"
"brochure" operation="pad" width=264 height=264 format="jpeg" options="background-color:0xF0F0F0" if-image-name-on="#{name_list}"
}
# Store migartion source image into on demand API bucket
store_s3 "migration" bucket="mybucket_v1" path="hash"
# Save input image for future reprocessing
store_s3 "input" bucket="mybucket" path="compat-structured" public=true
# Backend classes
store_s3 "original" bucket="mybucket" path="compat-structured-name" public=true cache-control="public, max-age=31557600, s-maxage=0" if-image-name-on="#{name_list}"
store_s3 "search" bucket="mybucket" path="compat-structured-name" public=true cache-control="public, max-age=31557600, s-maxage=0" if-image-name-on="#{name_list}"
store_s3 "brochure" bucket="mybucket" path="compat-structured-name" public=true cache-control="public, max-age=31557600, s-maxage=0" if-image-name-on="#{name_list}"
output_store_url {
"input"
"original" if-image-name-on="#{name_list}"
"search" if-image-name-on="#{name_list}"
"brochure" if-image-name-on="#{name_list}"
}
}
put "thumbnail" ":name_list" {
thumbnail "input" {
# Make limited source image for migration to on demand API
"migration" operation="limit" width=2160 height=2160 format="jpeg" quality=95
# Backend classes
"original" operation="crop" width="input" height="input" format="jpeg" options="background-color:0xF0F0F0" if-image-name-on="#{name_list}"
"search" operation="pad" width=162 height=162 format="jpeg" options="background-color:0xF0F0F0" if-image-name-on="#{name_list}"
"brochure" operation="pad" width=264 height=264 format="jpeg" options="background-color:0xF0F0F0" if-image-name-on="#{name_list}"
}
# Store migartion source image into on demand API bucket
store_s3 "migration" bucket="mybucket_v1" path="hash"
# Save input image for future reprocessing
store_s3 "input" bucket="mybucket" path="compat-hash" public=true
# Backend classe
store_s3 "original" bucket="mybucket" path="compat-hash-name" public=true cache-control="public, max-age=31557600, s-maxage=0" if-image-name-on="#{name_list}"
store_s3 "search" bucket="mybucket" path="compat-hash-name" public=true cache-control="public, max-age=31557600, s-maxage=0" if-image-name-on="#{name_list}"
store_s3 "brochure" bucket="mybucket" path="compat-hash-name" public=true cache-control="public, max-age=31557600, s-maxage=0" if-image-name-on="#{name_list}"
output_store_url {
"input"
"original" if-image-name-on="#{name_list}"
"search" if-image-name-on="#{name_list}"
"brochure" if-image-name-on="#{name_list}"
}
}
# Thumbnail on demand API
path "hash" "#{input_digest}.#{image_mime_extension}"
path "path" "#{path}"
put "v1" "original" {
thumbnail "input" "original" operation="limit" width=2160 height=2160 format="jpeg" quality=95
store_s3 "original" bucket="mybucket_v1" path="hash"
output_store_path "original"
}
get "v1" "thumbnail" ":path" ":operation" ":width" ":height" ":options?" {
source_s3 "original" bucket="mybucket_v1" path="path"
thumbnail "original" "thumbnail" operation="#{operation}" width="#{width}" height="#{height}" options="#{options}" quality=84 format="jpeg"
output_image "thumbnail" cache-control="public, max-age=31557600, s-maxage=0"
}
get "v2" "thumbnail" ":operation" ":width" ":height" {
source_s3 "original" bucket="mybucket_v1" path="path"
thumbnail "original" "thumbnail" operation="#{operation}" width="#{width}" height="#{height}" options="#{query_string_options}" quality=84 format="png"
output_image "thumbnail" cache-control="public, max-age=31557600, s-maxage=0"
}
Compatibility API works by storing input image and selected (via URI) classes of thumbnails generated during image upload. Once the image is uploaded thumbnails can be served directly from S3. There are two endpoints defined for that API to handle URIs that contain optional image storage name that results in usage of different storage key.
With thumbnail on demand API user uploads original image. It is converted to JPEG and if it is too large also scaled down. Than that processed version is stored in S3 under key composed from hash of input image data and final image extension. Client will receive storage key for further reference in the response body. To obtain thumbnail GET request with obtained key and thumbnail parameters encoded in the URI needs to be send to the sever. It will read parameters from the URI and source selected image from S3. That image is then thumbnailed in the backend and sent back to client with custom Cache-Control header.
Note that Compatibility API will also store "migration" image in bucket used by on demand API. This allows for migration from that API to on demand API.
Compatibility API example:
# Uploading image and thumbnailing to 'original' and 'brochure' classes
$ curl -X PUT 10.1.1.24:3000/thumbnail/original,brochure -q --data-binary @Pictures/compute.jpg
http://s3-eu-west-1.amazonaws.com/test.my.bucket/4006450256177f4a.jpg
http://s3-eu-west-1.amazonaws.com/test.my.bucket/4006450256177f4a/original.jpg
http://s3-eu-west-1.amazonaws.com/test.my.bucket/4006450256177f4a/brochure.jpg
# Obtaining 'brochure' class thumbnail
$ curl http://s3-eu-west-1.amazonaws.com/test.my.bucket/4006450256177f4a/brochure.jpg -v -s -o /tmp/test.jpg
* About to connect() to s3-eu-west-1.amazonaws.com port 80 (#0)
* Trying 178.236.7.32... connected
> GET /test.my.bucket/4006450256177f4a/brochure.jpg HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-apple-darwin10.8.0) libcurl/7.22.0 OpenSSL/1.0.1c zlib/1.2.7 libidn/1.25
> Host: s3-eu-west-1.amazonaws.com
> Accept: */*
>
< HTTP/1.1 200 OK
< x-amz-id-2: ZXJSWlUBthbIoUXztc9GkSu7mhpGK5HK+sVXWPdbCX9+a3nVkr4A6pclH1kdKjM9
< x-amz-request-id: 3DD4C96B6B55B4ED
< Date: Thu, 11 Jul 2013 11:33:31 GMT
< Cache-Control: public, max-age=31557600, s-maxage=0
< Last-Modified: Thu, 11 Jul 2013 11:31:36 GMT
< ETag: "cf060f47d557bcf9316554d34411dc51"
< Accept-Ranges: bytes
< Content-Type: image/jpeg
< Content-Length: 39458
< Server: AmazonS3
<
{ [data not shown]
* Connection #0 to host s3-eu-west-1.amazonaws.com left intact
* Closing connection #0
$ identify /tmp/test.jpg
/tmp/test.jpg JPEG 264x264 264x264+0+0 8-bit sRGB 11.9KB 0.000u 0:00.009
On demand API example:
# Uploading image
$ curl -X PUT 10.1.1.24:3000/v1/original -q --data-binary @Pictures/compute.jpg
4006450256177f4a.jpg
# Getting fit operation 100x1000 thumbnail to /tmp/test.jpg
$ curl 10.1.1.24:3000/v1/thumbnail/4006450256177f4a.jpg/fit/100/1000 -v -s -o /tmp/test.jpg
* About to connect() to 10.1.1.24 port 3000 (#0)
* Trying 10.1.1.24... connected
> GET /v1/thumbnail/4006450256177f4a.jpg/fit/100/1000 HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-apple-darwin10.8.0) libcurl/7.22.0 OpenSSL/1.0.1c zlib/1.2.7 libidn/1.25
> Host: 10.1.1.24:3000
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.2.9
< Date: Thu, 11 Jul 2013 11:26:15 GMT
< Content-Type: image/jpeg
< Content-Length: 4681
< Connection: keep-alive
< Status: 200 OK
< Cache-Control: public, max-age=31557600, s-maxage=0
<
{ [data not shown]
* Connection #0 to host 10.1.1.24 left intact
* Closing connection #0
$ identify /tmp/test.jpeg
/tmp/test.jpeg JPEG 100x141 100x141+0+0 8-bit sRGB 4.68KB 0.000u 0:00.000
# Also form with query string passed options can be used to retrieve thumbnails
$ curl 10.1.1.24:3000/v2/thumbnail/pad/100/100/4006450256177f4a.jpg?background-color=green -v -s -o /tmp/test.jpg
* About to connect() to 10.1.1.24 port 3000 (#0)
* Trying 10.1.1.24... connected
> GET /v2/thumbnail/pad/100/100/4006450256177f4a.jpg?background-color=green HTTP/1.1
> User-Agent: curl/7.22.0 (x86_64-apple-darwin10.8.0) libcurl/7.22.0 OpenSSL/1.0.1c zlib/1.2.7 libidn/1.25
> Host: 10.1.1.24:3000
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx/1.2.9
< Date: Wed, 24 Jul 2013 11:38:39 GMT
< Content-Type: image/jpeg
< Content-Length: 3310
< Connection: keep-alive
< Status: 200 OK
< Cache-Control: public, max-age=31557600, s-maxage=0
<
{ [data not shown]
* Connection #0 to host 10.1.1.24 left intact
* Closing connection #0
$ identify /tmp/test.jpg
/tmp/test.jpg JPEG 100x100 100x100+0+0 8-bit sRGB 3.31KB 0.000u 0:00.000
After installation of the gem the httpimagestore executable is installed in PATH.
This executable is used to start HTTP Image Service with given configuration file path as it's last argument.
In this mode httpimagestore daemon is listening on TCP port directly. This is the easiest way you can start the daemon but it is not recommended for production use.
It is recommended to use nginx server in front of this daemon in production to buffer requests and responses.
To start this daemon in foreground for testing purposes with prepared api.conf configuration file use:
httpimagestore --verbose --foreground api.conf
Hitting Ctrl-C will ask the server to shutdown.
If you start it without --foreground switch the daemon will fork into background and write it's PID in httpimagestore.pid by default.
Note that in order to perform thumbnailing HTTP Thumbnailer needs to be running.
You can run httpimagestore --help to display all available switches, options and arguments.
PID file location can be controlled with --pid-file options.
To change number of worker processes use --worker-processes.
You can also change time after witch worker process will be killed if it didn't provide response to request with --worker-timeout.
By default httpimagestore will not keep more than 128MiB of image data in memory - if this is exceeded the daemon will abort processing and send response with 413 status. The limit can be changed with --limit-memory option.
--listener can be used multiple times to define listening sockets; use --listener /var/run/httpimagestore.sock to listen on UNIX socket instead of default TCP port 3000.
If running as root you can use --user option to specify user with whose privileges the worker processes will be running.
httpimagestore logs to httpimagestore.log file in current directory by default. You can change log file location with --log-file option and verbosity with --verbose or --debug switch.
Additionally httpimagestore will log requests in common NCSA format to httpimagestore_access.log file. Use --access-log-file option to change location of access log.
Syslog logging can be enabled with --syslog-facility option followed by name of syslog facility to use. When enabled log files are not created and both application logs and access logs are sent to syslog.
Access logs will gain meta information that will include "type":"http-access" that can be used to filter access log entries out from application log entries.
With --xid-header option name of HTTP request header can be specified. Value of this header will be logged in meta information tag xid along side all request related log entries.
Named header will be passed down with requests to HTTP Thumbnailer for grater traceability.
Using --perf-stats switch will enable logging of performance statistics. This log entries will be logged with "type":"perf-stats".
nginx if configured properly will buffer incoming requests before sending them to the backend and server response before sending them to client.
Since httpimagestore is based on Unicorn HTTP server that is based on single threaded HTTP request processing worker processes the number of processing threads is very limited. Slow clients could keep precious process busy for long time slowly sending request or reading response effectively rendering service unavailable.
Starting httpimagestore daemon with UNIX socket listener and /etc/httpimagestore.conf configuration file:
httpimagestore --pid-file /var/run/httpimagestore/pidfile --log-file /var/log/httpimagestore/httpimagestore.log --access-log-file /var/log/httpimagestore/httpimagestore_access.log --listener /var/run/httpimagestore.sock --user httpimagestore /etc/httpimagestore.conf
Starting httpthumbnailer daemon:
httpthumbnailer --pid-file /var/run/httpthumbnailer/pidfile --log-file /var/log/httpthumbnailer/httpthumbnailer.log --access-log-file /var/log/httpthumbnailer/httpthumbnailer_access.log --listener 127.0.0.1:3100 --user httpthumbnailer /etc/httpthumbnailer.conf
To start nginx we need to configure it to run as reverse HTTP proxy for our UNIX socket based httpimagestore backend.
Also we set it up so that it does request and response buffering and on disk caching of GET requests.
You may want to disable caching if your GET URL resource is not immutable.
Here is the example /etc/nginx/nginx.conf file:
user nginx;
worker_processes 1;
error_log /var/log/nginx/error.log error;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/mime.types;
default_type application/octet-stream;
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for" $request_time';
access_log /var/log/nginx/access.log main;
sendfile on;
tcp_nopush on;
tcp_nodelay off;
keepalive_timeout 600s;
client_header_timeout 10s;
upstream httpimagestore {
server unix:/var/run/httpimagestore.sock fail_timeout=0;
}
# cache GET requests up to 256MiB in RAM and 130GiB on disk for up to 30 days of no access
proxy_cache_path /var/cache/nginx/httpimagestore levels=2:2 keys_zone=httpimagestore:256m max_size=130g inactive=30d;
server {
listen *:3000;
server_name localhost;
location / {
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
client_body_buffer_size 16m;
client_max_body_size 128m;
proxy_buffering on;
proxy_buffer_size 64k;
proxy_buffers 256 64k;
proxy_busy_buffers_size 256k;
proxy_temp_file_write_size 128m;
proxy_read_timeout 120s;
proxy_connect_timeout 10s;
proxy_cache httpimagestore;
proxy_cache_key "$request_uri";
proxy_pass http://httpimagestore;
}
}
}
Now it can be (re)started via usual init.d or systemd.
HTTP Image Store will respond with different status codes on different situations.
If all goes well 200 OK will be returned otherwise:
HTTP Image Store comes with statistics API that shows various runtime collected statistics.
It is set up under /stats URI. You can also request single stat with /stats/<stat name> request.
Example:
$ curl 127.0.0.1:3000/stats
workers: 10
total_requests: 27032164
total_errors: 3785333
calling: 1
writing: 0
total_write_multipart: 0
total_write: 23246830
total_write_part: 0
total_write_error: 3785333
total_write_error_part: 0
total_thumbnail_requests: 16415809
total_thumbnail_requests_bytes: 1624475247201
total_thumbnail_thumbnails: 16414688
total_thumbnail_thumbnails_bytes: 61806698184
total_identify_requests: 9
total_identify_requests_bytes: 73948
total_file_store: 0
total_file_store_bytes: 0
total_file_source: 0
total_file_source_bytes: 0
total_s3_store: 115337
total_s3_store_bytes: 11420368878
total_s3_source: 17150098
total_s3_source_bytes: 1694190265856
total_s3_cache_hits: 17149566
total_s3_cache_misses: 1416670
total_s3_cache_errors: 0
$ curl 127.0.0.1:3000/stats/total_thumbnail_thumbnails
16414688
Copyright (c) 2013 - 2015 Jakub Pastuszek. See LICENSE.txt for further details.
FAQs
Unknown package
We found that httpimagestore demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
/Research
Malicious npm package impersonates Nodemailer and drains wallets by hijacking crypto transactions across multiple blockchains.
Security News
This episode explores the hard problem of reachability analysis, from static analysis limits to handling dynamic languages and massive dependency trees.
Security News
/Research
Malicious Nx npm versions stole secrets and wallet info using AI CLI tools; Socket’s AI scanner detected the supply chain attack and flagged the malware.