codsen-tokenizer^5.6.0

HTML and CSS lexer aimed at code with fatal errors, accepts mixed coding languages

Install via npm

npm i codsen-tokenizer

Consume via a require():

const { tokenizer } = require("codsen-tokenizer");

or as an ES Module:

import { tokenizer } from "codsen-tokenizer";

Install as a script, via CDN

Put this in your HTML:

<script src="https://cdn.jsdelivr.net/npm/codsen-tokenizer/dist/codsen-tokenizer.umd.js"></script>

Then, you’ll get a global variable codsenTokenizer in your Global object and you consume it like this:

const { tokenizer } = codsenTokenizer;

Trivia

This program has 838 assertions in its unit tests. That's more than all unit test assertions' median (49) and more than geometric mean (73).

In monorepo, 2 other packages consume this package.

Quick Take

import { strict as assert } from "assert";
import { tokenizer } from "codsen-tokenizer";

const gathered = [];

// it operates from a callback, like Array.prototype.forEach()
tokenizer(`<td nowrap>`, {
  tagCb: (obj) => {
    gathered.push(obj);
  },
});

assert.deepEqual(gathered, [
  {
    type: "tag",
    start: 0,
    end: 11,
    value: "<td nowrap>",
    tagNameStartsAt: 1,
    tagNameEndsAt: 3,
    tagName: "td",
    recognised: true,
    closing: false,
    void: false,
    pureHTML: true,
    kind: null,
    attribs: [
      {
        attribName: "nowrap",
        attribNameRecognised: true,
        attribNameStartsAt: 4,
        attribNameEndsAt: 10,
        attribOpeningQuoteAt: null,
        attribClosingQuoteAt: null,
        attribValueRaw: null,
        attribValue: [],
        attribValueStartsAt: null,
        attribValueEndsAt: null,
        attribStarts: 4,
        attribEnds: 10,
        attribLeft: 2,
      },
    ],
  },
]);