glsl-tokenizer

glsl-tokenizer

Maps GLSL string data into GLSL tokens, either synchronously or using a streaming API.

var tokenString = require('glsl-tokenizer/string')
var tokenStream = require('glsl-tokenizer/stream')
var fs = require('fs')

// Synchronously:
var tokens = tokenString(fs.readFileSync('some.glsl'))

// Streaming API:
fs.createReadStream('some.glsl')
  .pipe(tokenStream())
  .on('data', function(token) {
    console.log(token.data, token.position, token.type)
  })

API

tokens = require('glsl-tokenizer/string')(src, [opt])

Returns an array of tokens given the GLSL source string src

You can specify opt.version string to use different keywords/builtins, such as '300 es' for WebGL2. Otherwise, will assume GLSL 100 (WebGL1).

var tokens = tokenizer(src, {
  version: '300 es'
})

stream = require('glsl-tokenizer/stream')([opt])

Emits 'data' events whenever a token is parsed with a token object as output.

As above, you can specify opt.version.

Tokens

{ 'type': TOKEN_TYPE
, 'data': "string of constituent data"
, 'position': integer position within the GLSL source
, 'line': line number within the GLSL source
, 'column': column number within the GLSL source }

The available token types are:

• block-comment: /* ... */
• line-comment: // ... \n
• preprocessor: # ... \n
• operator: Any operator. If it looks like punctuation, it's an operator.
• float: Optionally suffixed with f
• ident: User defined identifier.
• builtin: Builtin function.
• eof: Emitted on end; data will === '(eof)'.
• integer
• whitespace
• keyword

License

MIT, see LICENSE.md for further information.

Similar packages

Other packages similar to glsl-tokenizer

glsl-parser

The glsl-parser package is used for parsing GLSL code into an abstract syntax tree (AST). While glsl-tokenizer focuses on breaking down the code into tokens, glsl-parser goes a step further by providing a structured representation of the code. This can be useful for more complex analysis and transformations.

glsl-transpiler

The glsl-transpiler package is designed to transpile GLSL code to other shading languages or JavaScript. It includes tokenization and parsing as part of its process but extends the functionality to code transformation. This makes it more comprehensive compared to glsl-tokenizer, which is focused solely on tokenization.

glsl-optimizer

The glsl-optimizer package is used for optimizing GLSL code. It includes tokenization and parsing as part of its optimization pipeline. While glsl-tokenizer provides the basic building blocks for breaking down GLSL code, glsl-optimizer focuses on improving the performance and efficiency of the code.