extract-search-index^2.2.6

No 3rd party dependencies. All dependencies and devDependencies, checked recursively, are Codsen packages.

Permalink to InstallationInstallation

Permalink to Quick TakeQuick Take

Permalink to ExamplesExamples

• Keep the first occurrence of each case-insensitive word
• Parse attribute boundaries before decoding retained text
• Strip HTML tags and decode entities
• Normalise fancy punctuation and omit digits and astral characters
• Exclude code, preformatted text, and CSS from the index
• Remove URLs and stop words from search-index text
• Preserve BMP letters while separating words around surrogate code units

Purpose

Search programs such as fuse.js work on a set of data, for example, it’s this list argument below:

const options = {
  keys: ["title", "author.firstName"],
};

const fuse = new Fuse(list, options);

// Change the pattern
const pattern = "";

return fuse.search(pattern);

What if you want to implement a search function which would search within large strings, let’s say blog articles?

It would be inefficient to bundle up raw article string, send it to user’s browser, initiate fuse.js there and perform a search.

It’s better to pre-format the sources — search function does not care about words present more than once; it does not care about letter case or article words.

This program prepares optimised strings to be consumed by the likes of fuse.js.

PS. Also, you might want to pre-index the list, to speed up the search, but that’s a separate thing.

HTML and encoded markup

The extractor parses the original HTML before decoding character references or simplifying typography. Attribute values do not contribute keywords, including encoded quotes or greater-than signs inside those values:

extract('<p title="&quot;&gt;secret">visible</p>');
// → "visible"

The contents of script, style, xml, code, and pre elements are excluded. Retained text is decoded repeatedly, and HTML revealed at each decoding layer is parsed before decoding continues. This preserves the interpretation of encoded HTML as markup:

extract("before <code>hidden</code> after");
// → "before after"

URLs in text

HTTP and HTTPS addresses are removed before extracting keywords. This includes long top-level domains, localhost, ports, IPv6 addresses, Unicode hosts and paths, query strings, and fragments:

extract("before https://[::1]:8080/help https://例子.测试/路径?tags[]=one after");
// → "before after"

Balanced parentheses and backslash-escaped punctuation stay inside an address. An unmatched closing parenthesis ends it, preserving prose immediately after a Markdown link:

extract("[Guide](https://example.technology/topic_(details))next steps");
// → "guide next steps"

Whitespace, double quotes, backticks, and angle brackets delimit an address. Apostrophes inside an unquoted address remain part of it; a single quote immediately before the address instead establishes a quoted wrapper. Square brackets remain URL data, including in IPv6 hosts and query parameters, unless they close an immediately surrounding [https://…] wrapper.

These are text-extraction conventions, not a complete Markdown parser. Candidates must have a nonempty authority immediately after :// and be accepted by the runtime’s URL parser after removing a trailing run of sentence punctuation (. , ; : ! ?). Invalid candidates remain available to the ordinary keyword rules. Bare hostnames, email addresses, and other schemes do not use this HTTP(S) removal path; the parser does not fetch the address or check whether its host exists.

Character handling

Letters in the Unicode Basic Multilingual Plane (BMP), including fullwidth Latin and CJK ideographs, remain searchable. Lowercase, typography, and canonical Unicode normalization apply:

extract("ＦＯＯ ＦＯＯ foo豈更bar");
// → "ｆｏｏ foo豈更bar"

After HTML parsing and character-reference decoding finish, NFC normalization applies before typography conversion and again after lowercasing. Canonically equivalent spellings therefore produce one keyword, including Greek words whose decomposed spelling contains a combining mark:

extract("ἀρχή α\u0313ρχή");
// → "ἀρχή"

extract("café cafe\u0301");
// → "café"

Combining marks U+0312–U+0315 are preserved during typography conversion, including marks that do not compose with their preceding letter. NFC can change the spelling of some CJK compatibility ideographs, as in the first example. It does not fold fullwidth letters or ligatures through NFKC, strip accents, or transliterate words.

Typography is simplified completely in one replacement pass before punctuation is removed and words are deduplicated, including quotes after a length-changing ellipsis. Repeated typography in large article strings receives the same treatment:

extract("…“hello” “hello”".repeat(1000));
// → "hello"

After character references are decoded and complete URLs are removed, UTF-16 surrogate code units are replaced with spaces, so removing them does not join neighboring words. This includes lone surrogates, astral emoji, and supplementary-plane letters. Literal, decimal, hexadecimal, and nested references to valid characters receive the same treatment:

extract("before😊after");
extract("before😊after");
extract("before😊after");
// Each returns "before after".

URL recognition runs first so an astral character inside an address does not expose its remaining path as keywords. Invalid numeric references follow HTML recovery rules: for example, � becomes U+FFFD, which is retained, while a literal lone surrogate is removed.

This is not a complete emoji filter: BMP heart symbols, variation selectors, and joiners are retained.

API — extract()

The main function extract() is imported like this:

It’s a function which takes one input argument:

This function will return a string.

API — version

You can import version:

Permalink to changelogChangelog