![Oracle Drags Its Feet in the JavaScript Trademark Dispute](https://cdn.sanity.io/images/cgdhsj6q/production/919c3b22c24f93884c548d60cbb338e819ff2435-1024x1024.webp?w=400&fit=max&auto=format)
Security News
Oracle Drags Its Feet in the JavaScript Trademark Dispute
Oracle seeks to dismiss fraud claims in the JavaScript trademark dispute, delaying the case and avoiding questions about its right to the name.
code-tokenizer
Advanced tools
A JS implementation of a tokenizer to split up source code based on tokens
A JS implementation of a tokenizer to split up source code based on tokens
npm install code-tokenizer --save
Tokenizer = require 'code-tokenizer'
tokenizer = new Tokenizer()
tokens = tokenize 'print "Josh"'
# ['print']
tokens = tokenize 'path/to/file.html', 'file'
# ['<div>', 'id', 'class', '</div>']
This is based on the Github Linguist project. This node.js implementation strips any data strings or comments from the data and returns an array of language symbols.
You can use this to do sanity checking of source code, parsing, detection, etc. This is a general purpose library and I'm always looking to hear how you're using it. Happy Coding!
All tests are in the test/tokenizer.coffee file. It tests a variety of cases from different language types. To run the test for yourself, just do npm test
.
You can contact me @shamoons or read up on my blog: http://shamoon.me
FAQs
A JS implementation of a tokenizer to split up source code based on tokens
The npm package code-tokenizer receives a total of 0 weekly downloads. As such, code-tokenizer popularity was classified as not popular.
We found that code-tokenizer demonstrated a not healthy version release cadence and project activity because the last version was released a year ago. It has 1 open source maintainer collaborating on the project.
Did you know?
Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.
Security News
Oracle seeks to dismiss fraud claims in the JavaScript trademark dispute, delaying the case and avoiding questions about its right to the name.
Security News
The Linux Foundation is warning open source developers that compliance with global sanctions is mandatory, highlighting legal risks and restrictions on contributions.
Security News
Maven Central now validates Sigstore signatures, making it easier for developers to verify the provenance of Java packages.