Spliterator 🎀

Streams delimited byte content — CSV, TSV, JSONL, XLSX — without loading the file into memory.

Quick start · Chaining · Choosing a primitive · CLI · Examples

Spliterator scans for delimiters rather than materializing lines, so a parse costs one pass over the bytes and a queue of [start, end] ranges. An embedded WebAssembly SIMD scanner does the scanning — roughly 5–6 GB/s for multi-byte delimiters against ~600 MB/s for the JavaScript fallback — with no extra files, fetches, or configuration. Files small enough that setup dominates are read whole and parsed synchronously instead, automatically.

Every fromAsync returns an AsyncSequence: a lazy, chainable async iterator matching the async iterator helpers proposal, fused into a single pass so chain depth is nearly free. Early exit closes the file handle. The core is isomorphic, Node file I/O lives behind a subpath, and a CLI ships in the box. Every export carries TSDoc, so your editor is the reference.

yarn add spliterator
# or
npm install spliterator

Say you have a newline-delimited JSON file too large to fit into memory:

{"name": "Jessie", "age": 30}
{"name": "Kelly", "age": 40}
{"name": "Loren", "age": 50}
// Several hundred thousand more lines...

Spliterator reads it line by line, holding only the current record:

import { JSONSpliterator } from "spliterator"

interface Person {
	name: string
	age: number
}

const reader = JSONSpliterator.fromAsync<Person>("example.jsonl")

for await (const person of reader) {
	console.log(person) // { name: "Jessie", age: 30 }, etc.
}

While Spliterator supports any delimited byte stream, it's particularly useful for character-delimited content such as comma-separated values (CSV), tab-separated values (TSV) – or any other delimiter you can think of.

Full Name, Occupation, Age
Morgan, Developer, 30
Nataly, Designer, 40
Orlando, Manager, 50

import { CSVSpliterator } from "spliterator"

const reader = CSVSpliterator.fromAsync("people.csv")

for await (const row of reader) {
	console.log(row) // { full_name: "Morgan", occupation: "Developer", age: "30" }, etc.
}

CSV defaults to objects keyed by its header row, with normalized property names. Supply a row type when you want the type checker to know those keys:

import { CSVSpliterator } from "spliterator"

interface Person {
	full_name: string
	occupation: string
	age: number
}

const reader = CSVSpliterator.fromAsync<Person>("people.csv")

for await (const columns of reader) {
	console.log(columns) // { full_name: "Morgan", occupation: "Developer", age: 30 }, etc.
}

For tab-separated files, reach for TSVSpliterator. It accepts the same options as CSVSpliterator and defaults columnDelimiter to a tab, so you can omit it for the common case:

import { TSVSpliterator } from "spliterator"

const reader = TSVSpliterator.fromAsync("people.tsv")

for await (const columns of reader) {
	console.log(columns)
}

fromAsync returns an AsyncSequence — a lazy, chainable async iterator whose core methods (map, filter, take, drop, flatMap, reduce, toArray, forEach, some, every, find) match the async iterator helpers proposal in name and semantics. No polyfill required.

const cakes = await JSONSpliterator.fromAsync<Row>("menu.jsonl", { delimiter: "\n" })
	.filter((row) => row.category === "Ice Cream Cake")
	.map((row) => row.item_name)
	.take(10)
	.toArray()

Filtering happens while streaming, and take(10) closes the file handle instead of reading the rest. The operators fuse into a single pass rather than nesting one async generator per step, so chain depth is nearly free — doubling the operator count costs about 10%, where nesting would roughly double it. flatMap, chunks, parallelMap, and parallelFilter are the exceptions, since they need inner-iterator state.

The synchronous from returns a plain generator, which already has the same helpers natively on Node 24+.

All included Spliterators implement the Generator and AsyncGenerator interfaces, so you can use them in for...of and for await...of loops, as well the web-native ReadableStreams, so you can use them in for await...of loops, as well as piping them through transformations to avoid nested and partially materialized streams.

import { JSONSpliterator } from "spliterator"

const people = [
	{ name: "Alice", age: 30 },
	{ name: "Bob", age: 40 },
	{ name: "Charlie", age: 50 },
]

const generator = JSONSpliterator.from(people.map(JSON.stringify).join("\n"))
const stream = ReadableStream.from(generator)

for await (const line of stream) {
	console.log(line) // {"name": "Alice", "age": 30}, etc.
}

Spliterator can read and write .xlsx workbooks through XLSXSpliterator. Support is powered by two optional peer dependencies — install the one you need:

yarn add read-excel-file  # for XLSXSpliterator.fromAsync
yarn add write-excel-file # for XLSXSpliterator.write

Reading mirrors CSVSpliterator's options — mode, header, normalizeKeys, transformers, drop, and take — plus a sheet option to pick a sheet by 1-based number or name. Unlike CSV columns, XLSX cells arrive typed: numbers, booleans, and dates are real values rather than strings, and empty cells are null.

import { XLSXSpliterator } from "spliterator"

const reader = XLSXSpliterator.fromAsync("people.xlsx", { sheet: "Employees" })

for await (const row of reader) {
	console.log(row) // { full_name: "Morgan", hired: Date, age: 30 }, etc.
}

Transformers receive those typed cell values, which makes them a natural place to coerce loosely-exported data — many real-world workbooks store everything as text:

const reader = XLSXSpliterator.fromAsync("form499.xlsx", {
	transformers: {
		filer_499_id: (value) => Number(value),
		alabama: (value) => value === "TRUE",
	},
})

Writing accepts any iterable or async iterable of rows — arrays of cells, or records whose keys become the header row — so a Spliterator pipeline can terminate in a workbook:

const rows = CSVSpliterator.fromAsync("people.csv").map((person) => ({
	...person,
	age: Number(person.age),
}))

await XLSXSpliterator.write(rows, { sheet: "People" }).toFile("people.xlsx")

A few caveats worth knowing:

• The whole workbook is held in memory, in both directions. XLSX is a ZIP archive of XML documents — shared strings live in a separate archive entry and the ZIP's central directory sits at the end of the file — so it cannot be parsed or produced as a bounded-memory stream the way delimited text can. Reading yields no rows until the entire sheet is parsed, and writing drains your source completely before producing bytes. Expect memory usage far above the file's size on disk (a 9 MB workbook can inflate to hundreds of MB parsed). For sources large enough that this matters, prefer CSV or JSONL.
• There is no synchronous reader. Decompression and XML parsing are asynchronous in the underlying reader, so XLSXSpliterator.from() always throws, pointing you to fromAsync.
• One sheet at a time. Reading targets a single sheet per call, and writing produces a single-sheet workbook. Cell styling, formats, and formulas are out of scope.

See examples/xlsx-to-jsonl.ts for a complete conversion script with derived transformers.

Globerator is the Node-only filesystem discovery API. Import it from the spliterator/node/fs subpath; it returns an AsyncSequence, so glob results compose with the same map, filter, take, and toArray operations as parsed rows.

import { Globerator } from "spliterator/node/fs"

for await (const path of Globerator.files(["json", ".jsonl"], {
	cwd: "data",
})) {
	console.log(path) // path relative to data
}

files() accepts one or more extensions, with or without a leading dot, and searches descendants with paths relative to cwd by default. Pass recursive: false to limit discovery to cwd itself, or absolute: true when a consumer needs absolute paths.

For arbitrary patterns, use from(). It supports multiple patterns, exclusions, symlink traversal, cancellation, and Dirent output:

const features = Globerator.from("**/data/**/*.geojson", {
	cwd: "wof-repositories",
	exclude: ["**/*-alt-*.geojson"],
	followSymlinks: true,
	throwIfUnmatched: true,
})

for await (const path of features) {
	// JSONSpliterator.fromAsync(path), upload(path), etc.
}

Missing cwd directories throw by default, while an unmatched pattern yields an empty sequence. Use throwIfDirectoryMissing: false for an absence-tolerant walk, or throwIfUnmatched: true when an empty result is an error. These checks run when iteration begins, preserving lazy construction.

const entries = await Globerator.from("*.csv", {
	cwd: "imports",
	withFileTypes: true,
}).toArray()

for (const entry of entries) {
	console.log(entry.parentPath, entry.name)
}

Spliterator also includes a CLI tool that can be used to stream delimited content from the command line, transform it, filter it, and more.

spliterator csv people.csv people.jsonl

The CLI also supports reading from standard input:

cat people.csv | spliterator csv people.jsonl

Run one command per delimited record with bounded process concurrency. Arguments are passed directly to the child — there is no intermediate shell — and {} is replaced with the record (or appended when no placeholder is present):

find data -type f -print0 | spliterator parallel -0 -j 8 -- sha256sum {}

--pipe sends record-aligned blocks to each child instead. This is the efficient shape for commands that already read standard input; blocks preserve the source bytes exactly and never split a record:

cat large.jsonl | spliterator parallel --pipe --block 16MiB -j 4 -- process-block

Output is grouped per job by default and spills to temporary files, so a noisy child does not consume unbounded memory. Use -k to preserve input order, --line-buffer for live complete lines, --no-group for raw output, and --halt soon|now to stop after a failure.

For information on all available commands, run spliterator --help.

The question that predicts the answer is not "how big is my file?" — it's how much work happens per row.

The line worth internalizing: the scan is almost never your bottleneck unless you aren't parsing. Measure before adopting a parallel primitive.

The naming encodes one rule:

If you can pass a closure, it runs on your thread. If you must pass a module path, it runs on another one.

Closures can't cross a postMessage boundary, so parallelMap takes a function and parallelMapWorkers takes a path — and asMany/asManyWorkers divide the same way.

For one large file with a CPU-bound per-row transform, AsyncSpliterator.asManyWorkers splits the file into delimiter-aligned segments and runs a handler module across worker threads — each worker owns its own handle and reads only its segment. Results stream back to the main thread as a single async iterator, for a single-thread writer (a database, a JSONL file).

import { AsyncSpliterator } from "spliterator"

// transform.js (runs in each worker; top-level code is per-worker init):
//   const dec = new TextDecoder(), enc = new TextEncoder()
//   export function handleRecord(bytes) {
//     return enc.encode(JSON.stringify(parse(dec.decode(bytes))) + "\n") // Uint8Array → zero-copy
//   }

for await (const jsonLine of AsyncSpliterator.asManyWorkers<Uint8Array>("huge.csv", {
	worker: new URL("./transform.js", import.meta.url),
	delimiter: "\n",
	concurrency: 8,
})) {
	out.write(jsonLine) // single-thread writer on main
}

Need just the byte ranges to drive your own pool? AsyncSpliterator.segments(path, { delimiter, concurrency }) returns them.

Both threaded primitives spawn and terminate their workers per call. Spawning measured 17ms for one worker and 48ms for eight — half to two-thirds of a small call — and a handler that loads a model or opens a connection at import pays far more than that again, every call.

Pass a WorkerPool to keep them warm:

import { AsyncSpliterator, WorkerPool } from "spliterator"

await using pool = new WorkerPool({ size: 4 })

for (const path of manySmallFiles) {
	for await (const row of AsyncSpliterator.asManyWorkers(path, {
		worker: new URL("./transform.js", import.meta.url),
		delimiter: "\n",
		concurrency: 4,
		pool, // `parallelMapWorkers` takes the same option
	})) {
		out.write(row)
	}
}

Measured over a 200KB file: 3.3× across 5 calls and 5.6× across 20 — and 0.98× on a 52MB file, because startup only matters when it is a large share of the call. Reach for a pool when you make many small calls, not when you parse one big file.

Two things follow from workers being reused, both of them the point rather than surprises:

• The handler module is imported once per worker, not once per call, so its top-level state persists across calls. That is what makes loading a model worthwhile. Handlers that assume a clean slate per call need to reset it themselves.
• workerData belongs to the pool, fixed when it spawns a worker. Passing it per call alongside pool throws rather than being silently ignored.

A pool smaller than concurrency bounds the real parallelism — segments queue for a worker instead of running at once, and parallelMapWorkers clamps to the pool's size. Dispose it when you are done, or bind it with await using as above.

Spliterator ships a small WebAssembly SIMD scanner that accelerates delimiter and quote scanning (roughly 5–6× over the JavaScript scanner for multi-byte delimiters, more for column splitting). It is embedded in the package — no extra files, fetches, or configuration.

The module loads asynchronously. Asynchronous parsing (fromAsync, streams) picks it up automatically once loaded. Purely synchronous parsing that finishes in a single tick would otherwise complete before the module is ready and transparently use the JavaScript scanner — to opt in, await it first:

import { CharacterSequence, CSVSpliterator } from "spliterator"

await CharacterSequence.whenReady() // resolves to true once the SIMD scanner is active

for (const row of CSVSpliterator.from(largeCsvString)) {
	// ...now backed by the SIMD scanner
}

Correctness is identical either way; whenReady() only affects which scanner runs.

Opening a file handle and standing up a read stream costs about 100µs, which is most of the work for a small file. So fromAsync reads sources of 128 KiB or less into memory and parses them synchronously — measured ~1.85× faster at 635 B and ~1.4× at 125 KiB. Output is identical either way.

The threshold is deliberately small. Above ~256 KiB the advantage stops being measurable, while the memory cost keeps growing — a 1 GiB file costs ~105 MB resident streamed against ~1.1 GB read whole. Raising it buys nothing and spends memory linearly.

// Force streaming, whatever the size — when a bounded footprint is the point.
JSONSpliterator.fromAsync("data.jsonl", { delimiter: "\n", bulkThreshold: 0 })

Sources with no knowable length (a pipe, a ReadableStream) get an end-of-input test instead: if the first chunk read is also the last, the whole input is already in memory and is parsed directly. Otherwise it streams as normal.

While Spliterator includes premade exports for most use-cases, custom generators can be created via Spliterator and AsyncSpliterator. These are the low-level interfaces the rest of the library is built on, and they handle any kind of delimited content.

For more advanced usage, check out the examples, the tests in test/, or the fully-annotated source.

Spliterator is licensed under the MIT License.

For commercial usage licensing, please contact us at hello@sister.software.