Table of Contents generated with DocToc
Alpha One
About: This is a meta-project to collect scattered thoughts about and working solutions for recurring
tasks in Web- and Backend-Application building. Do not expect polished, working code, but rather
fragmentary, sketchy hints.
The project takes its name from the 1970s British TV series
Moonbase Alpha / Space: 1999.
Tagging
Limiting
Rationale: We often want to restrict access to the ressources a web application publishes; for example,
you want subscribers to have unlimited access, guest users to enjoy a realistic peek into your offerings
without handing over the full power of your application and the data therein, and at the same time deter
any automated downloads. One part of the equation that can make this happen is User Authentication; the other
is identifying anonymous visitors and selectively throttle or block their access, based on automatically
collected behavior details and / or manual black- and whitelisting.
-
A limiter should be completely agnostic as to the task it is limiting—it should help to generally put
limits to ressource usage, not, say, be built to specifically limit HTTP requests (a drop-in middleware
could do that–one that uses a generic limiter).
-
Because of its genericity, a limiter should work with arbitrary ID tokens that represent clients—these
could be any piece of data (an internal user ID, an IP address, a session ID) that is fit to identify
a client in the sense of the application.
-
Since a limiter should be able to base its behavior on the past behavior of a given client and on the
client's ranking (say, customer vs. guest vs. rogue), it must either entertain a suitable datastructure
to keep such group affiliation and usage data (or else be fed (some of) these details when called).
-
Whether or not a limiter should take care of data persistence is an open question; ideally, server restart
or redirection to another server process should not impact limiter behavior (this consideration would
appear to favor Redis-like data persistence plans).
-
A limiter should distinguish between and be configurable for at least three behavioral patterns: full
access, deferred (throttled) access, and denial of service (the last two with timeouts and permanent).
CoffeeNode Limit is an sttempt to bring all of
the above points to the backend. It currently works with a slightly patched version of
node-rate-limiter.
Browser Console / ANSI Colors
Options
Using path-extra
to get the canonical OS-dependent location for user-level application option files:
### https://github.com/jprichardson/node-path-extra ###
njs_path = require 'path-extra'
info njs_path.tempdir()
info njs_path.homedir()
info njs_path.datadir 'app-name-here'
Encryption
See discussion of initialization vector (iv):
General Data
Probably use openpgpjs or sjcl:
User Authentication & Management
The specific purpose of password encryption should probably be done using bcrypt, for which there are 100%
JS implementations available. It does have the advantage of adaptable algorithmic complexity. APIs typically
do not offer decryption, only comparison of data.
'Easy DB'
NPM & GitHub Publishing Helpers, Module Administration
Our erstwhile solution to preparing CoffeeScript source files for both development and production,
larq, is very much on the way out.
It still has one distinctive feature, though, and that is larq dev
, which compiles a JavaScript file for
each CoffeeScript file that does not contain the actual code, but rather dynamically loads (and compiles)
the CoffeeScript source; in this way, the CoffeeScript sources may change any time, while to outside
consumers, everything appears as though code was actually loaded from JavaScript files, from the locations
where those JavaScript loader files are located.
Thus, every effort is made to make the existence of CoffeeScript sources transparent; the result is
convenience during development and assurance that production versions of the same code (produced with larq pub
or coffee --compile
) are fully functional (even in the absence of the CS sources—the plan is to
publish those, too, though, so potential contributors get a chance to work on either JS or CS, whatever they
feel most comfotable with).
I started larq ('Language-Aware ReQuire') when
I
realized that the global require.extensions
hook in NodeJS has huge problems—long story short, it's a global hook, so if any of your dependencies have
different ideas on how to resolve a particular filename extensions, you're BFE. It would be fine if it was
some very local thing, but as it stands, its not, and care must be exercised.
There are some thoughts on
how to repair this,
but so far no overarching facility or best-practices solution has emerged, short of the advisory to compile
your 3rd-party-language sources down to the standard ones (*.js
, *.css
, *.html
) and only ever deal
with these compilation targets in production.
A possible, maybe makeshift solution is to use CoffeeNode Monitor
or similar to watch all sources and compile on change; this may prove a better alternative than trying to
do the same, but in more convoluted ways, using yet another tool (larq).
Persisting, Responsive Long Running Processes
Rationale: When we want to have a single HTTP serving process to run on a production server, we can
accomplish that by daemonizing the server process. Having more than one process and processes that are
resumed after abortion requires more work. On a development machine, we want to typically have a single HTTP
server to be run explicitly (not automatically on system startup), and, when any of its dependencies have
been modified, we want to re-run any build steps and then restart the serving process. In case of syntax or
runtime errors, we typically would like the monitoring instance to wait for any subsequent changes to the
codebase and then try and re-run the build steps. A server is a Long Running Process—as distinguished from,
say, a backup script that runs once a day for a minute or so. A Responsive Process is one that responds to
(rather, is managed so it can respond to) changes in source files. A Persisting Process is one that is
resumed on termination.
Jizura Data Queries
- coffeenode-mingkwai
- coffeenode-solr
- coffeenode-mojikura
Paper Publishing
Publication Helpers / Build Tools
Generate TOC for your README.md
with npm install --global doctoc
:
Tested:
Git Automation
Git Automation
Rejected
NodeGit
-1 for not updating their examples on their homepage. -1 for the Pyramids of Doom in their examples. -1 for
an API that makes the programmatical equivalent of git add --all; commit -m "foo"
look like rocket
science. -1 for embracing asynchronicity so damn hard you'll get a hundred callbacks when you ask for a
commit history. -1 for embracing Object-Oriented Programming so damn hard that you'll never get an answer to
your question but an object whose methods you must call—to asynchronously receive another object on which to
call more methods.
Jake
Cake is better.
NodeJS Version Management
Rationale: With the advent of breaking changes in NodeJS unstable 0.11.x, many modules relying on gyp
/ waf
do not compile any more. Some tools (such as browserify) compile with 0.11.7 but not with 0.11.8.
Consequently we must be able to switch between NodeJS versions in a swift and painless manner; n by
visionmedia is one way to do that.
After the painless install: npm install --global n
, you get a new executable n
that lets you install and
/ or use new versions with commands like n latest
, n stable
or n 0.11.7
. Simply executing n
yields a
menu in the terminal that lets you choose one of the installed versions. The global node
command is always
re-bound to the last chosen version with n
; in order to get commands for specific versions, cd
into a
directory on the path and do something along the lines of:
ln -s /usr/local/n/versions/0.11.7/bin/node node0117
ln -s /usr/local/n/versions/0.11.7/bin/node node-latest
ln -s /usr/local/n/versions/0.10.22/bin/node node-stable
Application Architecture & Design
Control Flow
And the Winner is: Yield!
NB CoffyScript is still (highly) experimental; its use in underlying libraries is especially strongly
discouraged. CyS is currently implemented as a random patch of (an earlier version of) CoffeeScript 1.6.3
which has been shown to posses some subtle bugs (especially faulty error reporting that does not point out
the source of the error). There was an attempt to upgrade to the latest bugfix editions of CoffeeScript, but
the code had already suffieciently changed to make simple copy-and-paste of the changes unviable. The CS
Redux project is, unfortunately, not yet advanced /
adopted enough for me switch over, so i decided to just wait it out: wait for yield
to become the
available in a stable Node release (i.e. 0.12 or 1.x) and wait for CS Redux to land in the master branch of
CoffeeScript.
With all the caveats, CoffyScript is already usable and has greatly helped to write some very readable,
working code that feels magnitudes better than the Promises/A+ ridden code it replaced.
The Journey Continues
Making Things Synchronous
Sometimes the solution to async woes lies in choosing an underlying functionality provider that is
synchronous—for example, when doing application startup stuff like compiling sources and so on, you might
choose fs.writeFileSync
over fs.writeFile
because (1) it lets you write linear code, (2) you can't run
your app before this step has completed anyhow. Using synchronous IO is an antipattern in online serving,
but its much more of a best practice for a startup sequence.
In the same vein, it may be desirable to execute shell commands in a synchronous fashion. Sadly, this piece
of functionality is not included in NodeJS proper;
fortunately, there are modules to do that,
like execSync.
It compiles with warnings under NodeJS 0.11.7 on my OSX box but appears to work, so now you can do
TRM = require 'coffeenode-trm'
, result = TRM.execute 'ls'
alongside with
TRM.execute 'ls', ( error, result ) -> ...
.
Rejected
Promises
Promises, with all the
neat /A+
marketing ribbons tacked unto them,
are horrible IMHO.
They come fraught with conceptual burden (eg. promises vs. deferreds), force boilerplate on you (never forget to
call .done()
when you're done), and they urge you to wrap all of your callback-based functions so as to make them play nice (which is not nice—callbacks were there first, and they are the
conceptually simpler mechanism. You just threw out the trusty hammer, now you got a beeping flashy gadget).
And: The API of the leading promises implementation is huge.
If you insist on doing promises, i'll ask you to memorize this 60+ items list:
Q.all
, Q.allSettled
, Q.async
, Q.defer
, Q.delay
, Q.denodeify
, Q.getUnhandledReasons
,
Q.isPromise
, Q.longStackSupport
, Q.makePromise
, Q.nbind
, Q.nextTick
, Q.nfapply
, Q.nfcall
,
Q.ninvoke
, Q.npost
, Q.onerror
, Q.promise
, Q.promised
, Q.reject
, Q.resetUnhandledRejections
,
Q.spawn
, Q.stopUnhandledRejectionTracking
, promise.all
, promise.allSettled
, promise.catch
,
promise.delay
, promise.delete
, promise.dispatch
, promise.done
, promise.fapply
, promise.fbind
,
promise.fcall
, promise.finally
, promise.get
, promise.inspect
, promise.invoke
,
promise.isFulfilled
, promise.isPending
, promise.isRejected
, promise.keys
, promise.nodeify
,
promise.post
, promise.progress
, promise.set
, promise.spread
, promise.then
, promise.thenReject
,
promise.thenResolve
, promise.timeout
, deferred.promise
, deferred.resolve
, deferred.reject
,
deferred.notify
, deferred.makeNodeResolver
.
You're of course not done yet—care to learn that (from the
API docs)
"Q.when( 5, onFulfilled )
is equivalent to Q( 5 ).then( onFulfilled )
"? I never wanted to know this.
One might argue that this particular API is just the result of some particular diligent programmer going
somewhat over the top, but that doesn't devalidate the argument that there is more than a fair number of
promises libraries out there, some of them broken, others with uncertain merits. Finding out how the
~30 libraries listed alongside the /A+ specs
interact and compare in terms of features, performance and memory consumption is not something
i want to spend my time with. And interaction between promises libraries is a real concern. To cite
a proponent of promises (emphases mine):
As authors of Promises/A-consuming libraries, we would like to assume this statement to be true: that
something that is “thenable” [meaning you can say x.then()
on it] actually behaves as a
Promises/A promise, with all the power that entails.
If you can make this assumption, you can write very extensive libraries that are entirely agnostic to the
implementation of the promises they accept! Whether they be from Q, when.js, or even WinJS, you can use the
simple composition rules of the Promises/A spec to build on promise behavior. For example, here's a
generalized retry function that works with any Promises/A implementation.
Unfortunately, libraries like jQuery break this. This necessitates ugly hacks to detect the presence of
objects masquerading as promises, and who call themselves in their API documentation promises, but aren't
really Promises/A promises. If the consumers of your API start trying to pass you jQuery promises, you have
two choices: fail in mysterious and hard-to-decipher ways when your compositional techniques fail, or fail
up-front and block them from using your library entirely. This sucks.
Interesting to hear that the jQuery team tried to implement promises and failed. Interesting, too, that
(some) promises library authors (according to the above statement) appearently try to live up to the
duck typing dream—i.e. they rationalize "this must be a
promises object!" when they get a return value x
where x.then
happens to be a function. That's about
as reliable as saying "this must be a cinema!" because "it has seats inside".
Frankly, as much as asynchronous calls and Pyramids of Doom tend to be a PITA, promises look and feel wrong.
Untested
Standalone Apps & Isomorphic Modules
Rationale. NodeJS makes it possible to use the same code on the server and in the browser; tools like
node-webkit
make it possible to run one-file downloads as no-installation / 'portable' desktop apps (with
the VM (runtime) either bundled or installed globally on the target machine). This spells good-bye to all
(or at least most) of the many competing GUI toolkits (if and where a 70MB minimum size for the VM is
acceptable, where a given platform is supported, and where performance is sufficient).
Tested:
Untested
(Web) Application (Backend) Design
The Stack
Ordered from the earliest, most general, down to the latest, most specific steps, the 'stack of actions' in
a web application may be sketched as follows:
- Preparation
- Limit (blocking)
- Routing & Query Parsing
- Favicon
- Static
- Authentication
- Limit (throttle)
- App
- Response Initiation
- Session Handling
- DB Access
- Template (View) Building
- Finalization
Modern Wire Communication Protocols
Chunked HTTP
- what it means for templating
WebSockets
Web Application Frontend Design
Icons &c
Web Application Deployment
Untested
HTLM5 & CSS3
Flexboxes
Probably better than using float
:
Custom HTML Elements
Rationale: getting away from <div class='foo'><div class='bar'>
, moving towards <foo><bar>
.
Probably need a polyfill for IE compatibility; without namespacing, custom tags may face problems with
future updates to HTML5 standard.