experiments - npm Package Compare versions

Comparing version 0.2.1 to 0.3.0

bower.json

 {
   "name": "experiments",
-  "version": "0.2.1",
-  "main": "dist/ab.min.js",
-  "dependencies"" {
+  "version": "0.3.0",
+  "main": "dist/experiments.min.js",
+  "dependencies": {
     "seedrandom": "2.3.4"
   }
 }

main.js

@@ -1,3 +0,3 @@
-var Ab = require('./src/ab')
+var Experiments = require('./src/experiments')
 
-module.exports = Ab;
+module.exports = Experiments;

package.json

 {
   "name": "experiments",
-  "version": "0.2.1",
+  "version": "0.3.0",
   "author": "Matt Chadburn <[e-mail]>",
   "description": "Simple split testing",
   "contributors": [
-    "Matt Chadburn"
+    "Matt Chadburn",
+    "Guardian News & Media Ltd"
   ],
+  "repository": {
+    "type": "git",
+    "url": "[e-mail]:commuterjoy/experiments.git"
+  },
   "engines": {
     "node": "*"
   },
+  "keywords": [
+    "ab testing",
+    "split testing"
+  ],
   "dependencies": {
@@ -41,3 +50,3 @@ "seedrandom": "2.3.4"
   },
-  "license": "MIT"
+  "license": "Apache-2.0"
 }

README.md

@@ -1,45 +0,192 @@
-A JavaScript AB testing framework, ported from [http://www.github.com/guardian/frontend](guardian/frontend).
+A JavaScript AB testing framework, ported from
+[http://www.github.com/guardian/frontend](guardian/frontend).
 
-[AB testing's](http://en.wikipedia.org/wiki/A/B_testing) goal is to identify changes to web pages that increase or maximize an outcome of interest.
+[AB testing's](http://en.wikipedia.org/wiki/A/B_testing) goal is to identify
+changes to web pages that increase or maximize an outcome of interest.
 
-### Goals
+## Goals
 
-- 100% client-side.
-- Basic segmentation of audience.
-- Deterministic segmentation - allowing allocation of users in to tests based on some external key (Eg, user name)
+- 100% client-side
+- Audience segmentation:-
+    - Basic - audience is segmented in to random buckets. 
+    - Deterministic - allows predictable allocation of users in to tests based
+      on some external key (Eg, username) 
 - Fixed duration tests - that automatically close and delete their footprint.
-- Isolation of each test audience - so a user can not accidently be in several tests at once.
-- Agnostic of where the data is logged - most companies have their own customer data repoisitories.
-- Minimal payload - ~1kb (minified + gzip) with no additional cookie overhead created. 
+- Isolation of each test audience - so a user can not accidentally be in several
+  tests at once.
+- Agnostic of where the data is logged - most companies have their own customer
+  data repositories.
+- Minimal payload - ~2kb (minified + gzip) with no additional cookie overhead
+  created. 
 
-### Experiment profiles 
+## Usage
 
-Each AB test is represented by a JavaScript object describing the profile of the test to be undertaken. 
+Let's say we want to test the effectiveness of a button that says 'help' versus
+one that says 'stuck?'. Our hypothesis is that, when a user sees an error
+message, 2% more people will click 'help' than 'stuck?'.
 
+To represent the test in code form we create a **profile** - a plain JavaScript
+object with a unique test identifier.
+
 ```
-var p = { 
-	id: 'background', // A unique name for the test.
-	audience: 0.1, // A percent of the users you want to run the test on, Eg. 0.1 = 10%.
-	audienceOffset: 0.8, // A segment of the users you want to target the test at. 
-	expiry: new Date(2015, 1, 1), // The end date of the test 
-	variants: [ // An array of two functions - the first representing the control group, the second the variant.
+var profile = {
+    id: 'help'
+} 
+```
+
+Next we need to create our two **variants** - 'help' and 'stuck'.
+
+Each variant is represented by a JavaScript function, so we add them to the
+profile. In this experiment we simply need to update the copy of two buttons in
+the page, but tests can largely do whatever is needed - change the layout, load
+in extra data, truncate bodies of text etc.
+
+```
+var profile = {
+    id: 'help',
+    variants: [ 
 		{ 
-			id: 'control',
+			id: 'help',
 			test: function () {
-				document.body.style.backgroundColor = '#ffffff';
+				$('.help-button').text('help'); 
 			}
 		},
 		{
-			id: 'pink',
+			id: 'stuck',
 			test: function () {
-				document.body.style.backgroundColor = '#c52720'; // this test turns the page background red
+				$('.help-button').text('stuck?'); 
 			}
 		}
-	],
-	canRun: function () { // Preconditions that all the test to run, or not
-		return true;
+	]
+}
+```
+
+Now we need an **audience** for the test. Again, we define a property in our
+profile. The profile defines the audience as a % of your total visitors, so if
+we want 10% of people to participate in the test we can state an audience of
+'0.1', or for 5%, an audience value of '0.05'.
+
+```
+var profile = {
+    id: 'help',
+    audience: 0.1,   // ie. 0.1 = 10%
+    variants: [ 
+	    ...
+    ]
+}
+```
+
+Often we want to run several tests simultaneously and be confident that
+variations in one test aren't influencing a second. To avoid this overlap we
+use the idea of a **audience offset**.
+
+Each user in a test is assigned a persistent integer, evenly distributed from 1
+to 1,000,000, and we allocate blocks of these users to individual tests.
+
+For example, a profile with an audience of 10% and an offset of 0.3 will
+allocate all the users with an identifier in the range 300,000 to 400,000 to
+the test and split the variants evenly between this group of people.
+
+```
+var profile = {
+    id: 'help',
+    audience: 0.1,
+    audienceOffset: 0.3, 
+    variants: [ 
+	    ...
+    ]
+}
+```
+
+In the profile described above we would expect, 
+
+- 5% of people to be allocated to the 'help' group  
+- 5% of people to be allocated to the 'stuck' group  
+- 90% of people to not be in the test.
+
+The later group can obviously be selected for other tests.
+
+A good AB test runs for a fixed period of time. For this reason each test
+profile must have an **expiry** date, represented as JavaScript Date object.
+
+If the date is the past the test framework will refuse to execute the test.
+
+```
+var profile = {
+    id: 'help',
+    expiry: new Date(2015, 1, 1)    // ie. runs until 1 Jan, 2015
+    ...
+}
+```
+
+Lastly we might want to scope the test to an particular context. For example,
+our help test should not be executed on pages that do not produce error
+messages.
+
+The **canRun** function is evaluated each time the test is loaded and decides
+if the test should be run.
+
+```
+var profile = {
+    id: 'help',
+	canRun: function () {
+        return (page.config.type === 'article');
 	}
+    ...
 }
 ```
 
+With our profile we can now instantiate a test, segment the audience and run it.
+
+```
+var help = new Experiments(profile).segment().run();
+```
+
+### Events 
+
+Each experiment will at least need to track the number of participants and
+number of successful conversions.
+
+The framework itself does not do this, but provides events for each experiment
+to listen out for and hook their tracking code to.
+
+The events are fired on `document.body` and contain the following `event.detail`.
+
+#### experiemnts.{profile.id}.start
+
+- variant - the segment the user has been assigned to.
+
+#### experiemnts.{profile.id}.complete
+
+_None_.
+
+### Seeding
+
+Often the goal of an AB test is to track a users behaviour over a period of
+time - typically this will mean collecting data over several sessions across
+several devices. 
+
+Purely client-side test frameworks make this hard to do as cookies and
+localStorage are somewhat volatile and, more crtically, local to a web browser
+on a single device.
+
+One approach to solving this problem is using [psuedorandom number
+generators](https://github.com/davidbau/seedrandom).
+
+These are deterministic algorithms that can, given the same input, generate an
+evenly distributed set of numbers, which we can fairly segment on. 
+
+For example, lets say we have user ID of 'UID=314146' stashed in a cookie. We
+can extract that and pass it to the framework to repeatedly bucket the user in
+to the same ID and same test variant, Eg. 
+
+```
+var k = getCookie('UID').split('=')[1];  // k = 314146
+new Experiments(profile, { seed: key }).segment().run();
+```
+
+This is very useful when testing people across different devices over long
+periods of time and, helpfully, independent of needing a persistance layer
+beyond any current sign-in system. 
+
 ## Demo
@@ -51,20 +198,23 @@
 
-With developer tools, we can feed the above profile in to the AB test framework, force our variant to '_pink_', then _run_ the test. 
+With developer tools, we can feed the above profile in to the AB test
+framework, force our variant to '_pink_', then _run_ the test. 
 
 ```
-var a = new Ab(p, { variant: 'pink' })
-a.run();
+var a = new Experiments(p, { variant: 'pink' }).run()
 ```
 
-You should see the page background turn pink, and running the test on every subsequent visit will turn the page pink until the test has expired.
+You should see the page background turn pink, and running the test on every
+subsequent visit will turn the page pink until the test has expired.
 
 
-Allocate yourself in to the control group and re-run the test and the background should turn white.
+Allocate yourself in to the control group and re-run the test and the
+background should turn white.
 
 ```
-var a = new Ab(p, { variant: 'control' })
-a.run();
+var a = new Experiments(p, { variant: 'control' }).run();
 ```
 
-For the duration of the test we can track the data of that user (say, pages per visit or scroll depth) and compare with the control group to see if that variant had the positive impact we thought it would have.
+For the duration of the test we can track the data of that user (say, pages per
+visit or scroll depth) and compare with the control group to see if that
+variant had the positive impact we thought it would have.
 
@@ -75,6 +225,7 @@ ### Segmentation
 
-Firstly, each test subject is allocated a persistant id (an integer) that is shared across tests.
+Firstly, each test subject is allocated a persistant id (an integer) that is
+shared across tests.
 
 ```
-localStorage.getItem('ab__uid'); // Eg, "3467"
+localStorage.getItem('ab__uid');
 ```
@@ -85,15 +236,14 @@
 ```
-localStorage.getItem('ab__background'); // Eg, '{"id":"background","variant":"pink"}'
+localStorage.getItem('ab__background'); 
 ```
 
-In the real world we want the test subjects allocated randomly in to a variants (or excluded from the test), so we don't specify the variant in the _Ab_ constructor and invoke `segment()` instead, before running the experiment,
+In the real world we want the test subjects allocated randomly in to a variants
+(or excluded from the test), so we don't specify the variant in the _Experiments_
+constructor and invoke `segment()` instead, before running the experiment,
 
 ```
-var a = new Ab(profile);
-a.segment();
-a.run();
+var a = new Experiments(profile).segment().run();
 ```
 
-The `segment()` function decides if a user should be in the test, and, if they are, splits the audience between a 'control' group and a number of 'variants'.
-
-Segmentation is fairly trivial at the moment, but later it can be used to target certain types of users (Eg, every international user who has visited more than 3 times a week, or persona x).
+The `segment()` function decides if a user should be in the test, and, if they
+are, splits the audience between a 'control' group and a number of 'variants'.

New alerts

License Policy Violation

License

This package is not allowed per your license policy. Review the package's license to ensure compliance.

Found 1 instance in 1 package

Major refactor

Supply chain risk

Package has recently undergone a major refactor. It may be unstable or indicate significant internal changes. Use caution when updating to versions that include significant changes.

Found 1 instance in 1 package

Minified code

Quality

This package contains minified code. This may be harmless in some cases where minified code is included in packaged libraries, however packages on npm should not minify code.

Found 1 instance in 1 package

Unidentified License

License

(Experimental) Something that seems like a license was found, but its contents could not be matched with a known license.

Found 1 instance in 1 package

Fixed alerts

License Policy Violation

License

This package is not allowed per your license policy. Review the package's license to ensure compliance.

Found 1 instance in 1 package

Minified code

Quality

This package contains minified code. This may be harmless in some cases where minified code is included in packaged libraries, however packages on npm should not minify code.

Found 1 instance in 1 package

Mixed license

License

(Experimental) Package contains multiple licenses.

Found 1 instance in 1 package

No repository

Supply chain risk

Package does not have a linked source code repository. Without this field, a package will have no reference to the location of the source code use to generate the package.

Found 1 instance in 1 package

Improved metrics

Number of lines in readme file

247

increased by154.64%

Number of low supply chain risk alerts

4

decreased by-20%

Worsened metrics

Total package byte prevSize

53661

decreased by-39.57%

Number of package files

16

decreased by-33.33%

License quality

80

decreased by-20%

Lines of code

1049

decreased by-29.83%

Number of medium supply chain risk alerts

1

increased byInfinity%

No dependency changes