this is a build server made to be real-time, fast and dead simple to use. In other words, this was made to be cool :)
installation
install node.js
I recommend using the
node version manager so that you
can have as many node.js versions you want and switch among them
pretty easily.
if you don't have node.js and npm installed already, run the commands below:
git clone git://github.com/creationix/nvm.git ~/.nvm
echo 'source $HOME/.nvm/nvm.sh' >> $HOME/.bash_profile
. $HOME/.nvm/nvm.sh
the snippet above must be run only once, but the one below you can do as many times you want
nvm install v0.6.10
nvm use v0.6.10
nvm alias default v0.6.10
install redis
MacOSX
$ brew install redis
Ubuntu and other Debian-ish GNU/Linux distros
$ sudo apt-get install redis-server
Windows
Forget it, bro. This CI server is cool, so it will NOT run on
windows. Sorry about that :)
install emerald
$ npm install -g emerald
running it
$ emerald run
other options
you can pass the -s
or --settings
parameter to the emerald
command line parameter, this will cause emerald to import the settings from the file and merge them over the default settings
below is a full description of all the options, if you want to write
your own settings file, you can pretty much copy and paste the code
below and add your own fine tune:
please consider $EMERALD_ROOT$
as the path that npm installed emerald for you when you ran npm install -g emerald
module.exports = {
LOG_LEVEL: 3,
SHUT_UP: false,
GIT_POLL_INTERVAL: 3000,
EMERALD_PORT: 3000,
EMERALD_HOSTNAME: 'localhost',
EMERALD_DOMAIN: 'http://localhost:3000',
REDIS_KEYS: {
current_build: "emerald:current-build",
build_queue: "emerald:build-queue"
},
EMERALD_PATH: "~/.emerald",
SANDBOX_PATH: "~/.emerald/builds",
ASSETS_PATH: "$EMERALD_ROOT$/public"
LOCAL_FILE: function(...){},
VIEW_PATH: "$EMERALD_ROOT$/public/app/server/html",
CLIENT_PATH: "$EMERALD_ROOT$/app/client",
BACKBONE_VIEW_PATH: "$EMERALD_ROOT$/app/client/html",
SCRIPT_PATH: "$EMERALD_ROOT$/app/terminal/main.js",
PID_PATH: PID_PATH,
STDOUT_PATH: STDOUT_PATH,
STDERR_PATH: STDERR_PATH
}
"hands on" hack guide
install the dependencies
1. node v0.6.10
2. npm 1.1.0-3
3. redis... just make sure it's running
4. global dependencies
npm install -g vows jshint
5. local dependencies
just run
npm install
*and should be enough to install all the dependencies in package.json
set up the pre-commit hook:
cd path/to/emerald
ln -s .development/pre-commit-hook .git/hooks/pre-commit
chmod +x .git/hooks/pre-commit
this is the result of activating the pre-commit hook:
![your commits are gonna be pretty](http://f.cl.ly/items/1b162c0A1V0o2M0K1O1n/Screen%20Shot%202012-02-09%20at%205.58.43%20PM.png)
run the tests!
unit tests
make unit
functional tests
make functional
or... run them all together
npm test
run the server!
npm start
fixtures for testing locally
make data
Dependencies and its explanations
clay.js
It's our persistency layer on redis, this lib an active record by
Yipit.com you can see
its documentation at github
redis
Mostly used for pub/sub and queue management, and connect-redis
sessions, although we also explicitly declare it as the first
connection in our models
clay.js configuration.
express / socket.io / connect-redis
The web layer: express is the
webserver, socket.io, in turn, is for real-time pushes to the clients. Connect-redis is a redis-backed session storage for express.
jade / less
Jade is a beautiful and
shorthand template language, and less a
shorthand styling language that has powers beyond just CSS.
underscore.js
this lib is like a
polyfill for
manipulating javascript objects with shorthand functions.
backbone.js
This awesome library allows us to organize the web app a lot better by
providing MVC support on the client side. It doesn't mean all our
client-side models are persisted on the server, they just leverage how
the views should update themselves.
async
it's our flow-control library. Helps avoiding too much callback soup
in our code.
colors / mkdirp
colors helps us making the
terminal feedback less boring by logging with color-formated
output. mkdirp is just for
creating directories recursively.
testing: vows and should
if you want to hack on emerald's code, then you're gonna need vows in order to run the tests which are written under the DSL provided by should.js
sketch of how this should work:
![diagram](https://github.com/Yipit/emerald/raw/master/resources/design/emerald.png)
## a little more detailed description:
Server actors
As you can notice Emerald's compound has not only a HTTP server but
also a few workers that runs in parallel to it:
Orchestrator
This guy subscribes to a set of meaningful signals sent to the redis
pub/sub, deserializes it into JSON objects and broadcasts to all the
socket.io connected clients.
In other words it captures all the server-side events and send to the
connected clients so that the UI can be updated in real time.
Queue consumer
It's basically a structured callback passed to setInterval
with the
settings.GIT_POLL_INTERVAL
as the interval param. So it sits there
checking if there is a build already running, if not it gets the next
instruction id to be build from the queue (if any instructions there)
and spawns a Build Runner for that given instruction
The build queue is to a redis
SortedSet
in which the key is
emerald:build-queue
by default, but can be configured through
settings.js
The Build Runner also sets a lock in Redis so that it doesn't run
parallel builds, avoiding builds to violate each other's environment.
After a build is finished it unsets the lock (key emerald:current-build
)
If the queue is locked, then it just yields by doing nothing :D
Build Runner
Is an object that is instantiated with a models.Build() object as parameter, the it actually do build-related stuff:
- git clone/pull
- run the build script
- send the appropriate events to the redis pub/sub (i.e:
BuildInstruction enqueued
, BuildInstruction created
, Repository being fetched
, so on...)
At this point you are probably asking yourself how can the Build Runner possibly take a build as instance since the runner is supposed to create it itself. Just take a look here and you will see that server.models.BuildInstruction
instances have a run
method that gets called by the queue consumer
Server-side events are sent to the clients
Since emerald was built with "real-time UI" in mind, its architecture
is all event-based.
In other words the server actors publish events to the redis pub/sub,
those events have JSON metadata serialized into it, so that the
orchestrator sends those events to the socket.io clients which
translates the events into Backbone model instances events.
This is basically how emerald's user interface looks like it's alive :D
Those are the current events that are handled by socket.io:
BuildInstruction created
BuildInstruction edited
BuildInstruction enqueued
BuildInstruction deleted
Repository started fetching
Repository finished fetching
Repository being fetched
Build started
Build finished
Build aborted
Build stdout
Build stderr
Build output
Build running
General error
license
Emerald is released under GNU Affero General Public License version 3
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as
published by the Free Software Foundation, either version 3 of the
License, or (at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU Affero General Public License for more details.
You should have received a copy of the GNU Affero General Public License
along with this program. If not, see <http://www.gnu.org/licenses/>.