Handling large JSON file in Deno

by mmyoji

2 min read

deno

Conclusion

Avoid using large JSON file, but use CSV or other easy-to-parse file format for stream API.

Please tell me if you have a better solution.

Motivation

When you have to handle large file (mainly in server side), you normally use Stream API in Node.js.

In Deno, you don't have it but use Web Streams API instead. (Actually you could use std/node libraries: see)

Stream handles data as chunks, but JSON file, an object rather than an array of JSON objects especially, is hard to be handled.

Read file as stream in Deno

This is a sample code to read JSON file as string in Deno.

// tmp/
//   test.ts
//   test.json

import { readableStreamFromReader } from "https://deno.land/std@0.140.0/streams/mod.ts";

const file = await Deno.open("./tmp/test.json", { read: true });

// Uint8Array
const byteStream = readableStreamFromReader(file);

// Uint8Array -> String
const decodedStream = byteStream.pipeThrough(new TextDecoderStream());

for await (const chunk of decodedStream) {
  console.log({ chunk: JSON.parse(chunk) });
}

// $ deno run --allow-read ./tmp/test.ts

If the file is not so large, the chunk is its entire content as string. But if the file is large enough, the chunk is not JSON-parsable one.

How to parse JSON chunk

There is (was) a library called JSONParse on npm. But it's not maintained yet and only supports Node.js environment.

So there are several options:

  1. Write your own parser: like as this stackoverflow post
  2. Use other file format (like CSV): fast-csv (found in esm.sh) is a good library
  3. Avoid using large JSON

I rarely have experienced like this but I've tried thinking about it for exercise.