New Case Study:See how Anthropic automated 95% of dependency reviews with Socket.Learn More
Socket
Sign inDemoInstall
Socket

code-tokenizer

Package Overview
Dependencies
Maintainers
1
Versions
4
Alerts
File Explorer

Advanced tools

Socket logo

Install Socket

Detect and block malicious and high-risk dependencies

Install

code-tokenizer

A JS implementation of a tokenizer to split up source code based on tokens

  • 1.0.6
  • latest
  • Source
  • npm
  • Socket score

Version published
Weekly downloads
1
decreased by-88.89%
Maintainers
1
Weekly downloads
 
Created
Source

code-tokenizer

A JS implementation of a tokenizer to split up source code based on tokens

Install

npm install code-tokenizer --save

How to use

Tokenizer = require 'code-tokenizer'

tokenizer = new Tokenizer()

tokens = tokenize 'print "Josh"'
# ['print']


tokens = tokenize 'path/to/file.html', 'file'
# ['<div>', 'id', 'class', '</div>']

Uses

This is based on the Github Linguist project. This node.js implementation strips any data strings or comments from the data and returns an array of language symbols.

You can use this to do sanity checking of source code, parsing, detection, etc. This is a general purpose library and I'm always looking to hear how you're using it. Happy Coding!

Testing

All tests are in the test/tokenizer.coffee file. It tests a variety of cases from different language types. To run the test for yourself, just do npm test.

Contact

You can contact me @shamoons or read up on my blog: http://shamoon.me

Keywords

FAQs

Package last updated on 15 May 2014

Did you know?

Socket

Socket for GitHub automatically highlights issues in each pull request and monitors the health of all your open source dependencies. Discover the contents of your packages and block harmful activity before you install or update your dependencies.

Install

Related posts

SocketSocket SOC 2 Logo

Product

  • Package Alerts
  • Integrations
  • Docs
  • Pricing
  • FAQ
  • Roadmap
  • Changelog

Packages

npm

Stay in touch

Get open source security insights delivered straight into your inbox.


  • Terms
  • Privacy
  • Security

Made with ⚡️ by Socket Inc