js-tokens - npm Package Compare versions

Comparing version 3.0.2 to 4.0.0

CHANGELOG.md

@@ -0,1 +1,18 @@
+### Version 4.0.0 (2018-01-28) ###
+
+- Added: Support for ES2018. The only change needed was recognizing the `s`
+  regex flag.
+- Changed: _All_ tokens returned by the `matchToToken` function now have a
+  `closed` property. It is set to `undefined` for the tokens where “closed”
+  doesn’t make sense. This means that all tokens objects have the same shape,
+  which might improve performance.
+
+These are the breaking changes:
+
+- `'/a/s'.match(jsTokens)` no longer returns `['/', 'a', '/', 's']`, but
+  `['/a/s']`. (There are of course other variations of this.)
+- Code that rely on some token objects not having the `closed` property could
+  now behave differently.
+
+
 ### Version 3.0.2 (2017-06-28) ###
@@ -2,0 +19,0 @@

index.js

@@ -1,2 +0,2 @@
-// Copyright 2014, 2015, 2016, 2017 Simon Lydell
+// Copyright 2014, 2015, 2016, 2017, 2018 Simon Lydell
 // License: MIT. (See LICENSE.)
@@ -10,6 +10,6 @@
 // (run `npm run build`).
-exports.default = /((['"])(?:(?!\2|\\).|\\(?:\r\n|[\s\S]))*(\2)?|`(?:[^`\\$]|\\[\s\S]|\$(?!\{)|\$\{(?:[^{}]|\{[^}]*\}?)*\}?)*(`)?)|(\/\/.*)|(\/\*(?:[^*]|\*(?!\/))*(\*\/)?)|(\/(?!\*)(?:\[(?:(?![\]\\]).|\\.)*\]|(?![\/\]\\]).|\\.)+\/(?:(?!\s*(?:\b|[\u0080-\uFFFF$\\'"~({]|[+\-!](?!=)|\.?\d))|[gmiyu]{1,5}\b(?![\u0080-\uFFFF$\\]|\s*(?:[+\-*%&|^<>!=?({]|\/(?![\/*])))))|(0[xX][\da-fA-F]+|0[oO][0-7]+|0[bB][01]+|(?:\d*\.\d+|\d+\.?)(?:[eE][+-]?\d+)?)|((?!\d)(?:(?!\s)[$\w\u0080-\uFFFF]|\\u[\da-fA-F]{4}|\\u\{[\da-fA-F]+\})+)|(--|\+\+|&&|\|\||=>|\.{3}|(?:[+\-\/%&|^]|\*{1,2}|<{1,2}|>{1,3}|!=?|={1,2})=?|[?~.,:;[\](){}])|(\s+)|(^$|[\s\S])/g
+exports.default = /((['"])(?:(?!\2|\\).|\\(?:\r\n|[\s\S]))*(\2)?|`(?:[^`\\$]|\\[\s\S]|\$(?!\{)|\$\{(?:[^{}]|\{[^}]*\}?)*\}?)*(`)?)|(\/\/.*)|(\/\*(?:[^*]|\*(?!\/))*(\*\/)?)|(\/(?!\*)(?:\[(?:(?![\]\\]).|\\.)*\]|(?![\/\]\\]).|\\.)+\/(?:(?!\s*(?:\b|[\u0080-\uFFFF$\\'"~({]|[+\-!](?!=)|\.?\d))|[gmiyus]{1,6}\b(?![\u0080-\uFFFF$\\]|\s*(?:[+\-*%&|^<>!=?({]|\/(?![\/*])))))|(0[xX][\da-fA-F]+|0[oO][0-7]+|0[bB][01]+|(?:\d*\.\d+|\d+\.?)(?:[eE][+-]?\d+)?)|((?!\d)(?:(?!\s)[$\w\u0080-\uFFFF]|\\u[\da-fA-F]{4}|\\u\{[\da-fA-F]+\})+)|(--|\+\+|&&|\|\||=>|\.{3}|(?:[+\-\/%&|^]|\*{1,2}|<{1,2}|>{1,3}|!=?|={1,2})=?|[?~.,:;[\](){}])|(\s+)|(^$|[\s\S])/g
 
 exports.matchToToken = function(match) {
-  var token = {type: "invalid", value: match[0]}
+  var token = {type: "invalid", value: match[0], closed: undefined}
        if (match[ 1]) token.type = "string" , token.closed = !!(match[3] || match[4])
@@ -16,0 +16,0 @@ else if (match[ 5]) token.type = "comment"

package.json

 {
   "name": "js-tokens",
-  "version": "3.0.2",
+  "version": "4.0.0",
   "author": "Simon Lydell",
@@ -25,7 +25,7 @@ "license": "MIT",
   "devDependencies": {
-    "coffee-script": "~1.12.6",
-    "esprima": "^4.0.0",
-    "everything.js": "^1.0.3",
-    "mocha": "^3.4.2"
+    "coffeescript": "2.1.1",
+    "esprima": "4.0.0",
+    "everything.js": "1.0.3",
+    "mocha": "5.0.0"
   }
 }

README.md

@@ -76,3 +76,4 @@ Overview [![Build Status](https://travis-ci.org/lydell/js-tokens.svg?branch=master)](https://travis-ci.org/lydell/js-tokens)
 
-The intention is to always support the latest stable ECMAScript version.
+The intention is to always support the latest ECMAScript version whose feature
+set has been finalized.
 
@@ -82,7 +83,5 @@ If adding support for a newer version requires changes, a new version with a
 
-Currently, [ECMAScript 2017] is supported.
+Currently, ECMAScript 2018 is supported.
 
-[ECMAScript 2017]: https://www.ecma-international.org/ecma-262/8.0/index.html
 
-
 Invalid code handling
@@ -105,3 +104,4 @@ =====================
 Invalid non-ASCII characters are treated as names, to simplify the matching of
-names (except unicode spaces which are treated as whitespace).
+names (except unicode spaces which are treated as whitespace). Note: See also
+the [ES2018](#es2018) section.
 
@@ -150,3 +150,4 @@ Regex literals may contain invalid regex syntax. They are still matched as
 Because humans can look at the whole code to put the `/` characters in context.
-A JavaScript regex cannot. It only sees forwards.
+A JavaScript regex cannot. It only sees forwards. (Well, ES2018 regexes can also
+look backwards. See the [ES2018](#es2018) section).
 
@@ -199,3 +200,3 @@ When the `jsTokens` regex scans throught the above, it will see the following
 allowed. This means that the above example will be identified as division as
-long as you don’t rename the `e` variable to some permutation of `gmiyu` 1 to 5
+long as you don’t rename the `e` variable to some permutation of `gmiyus` 1 to 6
 characters long.
@@ -223,3 +224,20 @@
 
+### ES2018 ###
 
+ES2018 added some nice regex improvements to the language.
+
+- [Unicode property escapes] should allow telling names and invalid non-ASCII
+  characters apart without blowing up the regex size.
+- [Lookbehind assertions] should allow matching telling division and regex
+  literals apart in more cases.
+- [Named capture groups] might simplify some things.
+
+These things would be nice to do, but are not critical. They probably have to
+wait until the oldest maintained Node.js LTS release supports those features.
+
+[Unicode property escapes]: http://2ality.com/2017/07/regexp-unicode-property-escapes.html
+[Lookbehind assertions]: http://2ality.com/2017/05/regexp-lookbehind-assertions.html
+[Named capture groups]: http://2ality.com/2017/05/regexp-named-capture-groups.html
+
+
 License
@@ -226,0 +244,0 @@ =======

New alerts

License Policy Violation

License

This package is not allowed per your license policy. Review the package's license to ensure compliance.

Found 1 instance in 1 package

Fixed alerts

License Policy Violation

License

This package is not allowed per your license policy. Review the package's license to ensure compliance.

Found 1 instance in 1 package

Improved metrics

Total package byte prevSize

15069

increased by11.56%

Number of lines in readme file

241

increased by8.07%

No dependency changes