String Decoder

Node.js string decoder module for converting Buffer to strings with multi-byte character support

The node:string_decoder module provides an API for decoding Buffer objects into strings while preserving encoded multi-byte UTF-8 and UTF-16 characters.

Installation

import { StringDecoder } from 'node:string_decoder';
// or
const { StringDecoder } = require('node:string_decoder');

Why Use StringDecoder?

When converting buffers to strings, multi-byte characters can be split across buffer boundaries. StringDecoder handles this gracefully by buffering incomplete characters.

Problem without StringDecoder:

import { Buffer } from 'node:buffer';

// Euro symbol (€) is 3 bytes: 0xE2 0x82 0xAC
const euro = Buffer.from([0xE2, 0x82, 0xAC]);

// Splitting the character incorrectly
const part1 = Buffer.from([0xE2]);
const part2 = Buffer.from([0x82, 0xAC]);

console.log(part1.toString()); // � (replacement character)
console.log(part2.toString()); // �� (garbage)

Solution with StringDecoder:

import { StringDecoder } from 'node:string_decoder';
import { Buffer } from 'node:buffer';

const decoder = new StringDecoder('utf8');

const part1 = Buffer.from([0xE2]);
const part2 = Buffer.from([0x82, 0xAC]);

console.log(decoder.write(part1)); // '' (buffered)
console.log(decoder.write(part2)); // '€' (complete)

Class: StringDecoder

new StringDecoder([encoding])

Creates a new StringDecoder instance.

Parameters:

• encoding - Character encoding (default: 'utf8')

Supported encodings:

• 'utf8' - UTF-8 encoding (default)
• 'utf16le' (or 'ucs2') - UTF-16 Little Endian
• 'base64' - Base64 encoding
• 'base64url' - Base64 URL-safe encoding
• 'hex' - Hexadecimal encoding
• 'ascii' - ASCII encoding
• 'binary' (or 'latin1') - Latin-1 encoding

Example:

import { StringDecoder } from 'node:string_decoder';

const decoder = new StringDecoder('utf8');

stringDecoder.write(buffer)

Returns a decoded string, buffering incomplete multi-byte characters.

Parameters:

• buffer - Bytes to decode

Returns: - Decoded string

Example:

import { StringDecoder } from 'node:string_decoder';
import { Buffer } from 'node:buffer';

const decoder = new StringDecoder('utf8');

// Cent symbol (¢) is 2 bytes: 0xC2 0xA2
const cent = Buffer.from([0xC2, 0xA2]);
console.log(decoder.write(cent)); // '¢'

// Euro symbol (€) is 3 bytes: 0xE2 0x82 0xAC
const euro = Buffer.from([0xE2, 0x82, 0xAC]);
console.log(decoder.write(euro)); // '€'

stringDecoder.end([buffer])

Returns any remaining buffered input as a string and resets the decoder.

Parameters:

• buffer - Optional final bytes to decode

Returns: - Remaining decoded string

Example:

import { StringDecoder } from 'node:string_decoder';
import { Buffer } from 'node:buffer';

const decoder = new StringDecoder('utf8');

// Euro symbol split across writes
decoder.write(Buffer.from([0xE2]));
decoder.write(Buffer.from([0x82]));
const result = decoder.end(Buffer.from([0xAC]));

console.log(result); // '€'

// Decoder is now reset and can be reused
const newResult = decoder.write(Buffer.from('hello'));
console.log(newResult); // 'hello'

Common Use Cases

Streaming Text Data

import { StringDecoder } from 'node:string_decoder';
import { createReadStream } from 'node:fs';

const decoder = new StringDecoder('utf8');
const stream = createReadStream('file.txt');

stream.on('data', (chunk) => {
  // Properly handles multi-byte characters across chunks
  const text = decoder.write(chunk);
  console.log(text);
});

stream.on('end', () => {
  // Output any remaining buffered data
  const remaining = decoder.end();
  if (remaining) {
    console.log(remaining);
  }
});

Processing Network Data

import { StringDecoder } from 'node:string_decoder';
import { createServer } from 'node:net';

const server = createServer((socket) => {
  const decoder = new StringDecoder('utf8');
  
  socket.on('data', (chunk) => {
    const text = decoder.write(chunk);
    console.log('Received:', text);
  });
  
  socket.on('end', () => {
    const remaining = decoder.end();
    if (remaining) {
      console.log('Final:', remaining);
    }
  });
});

server.listen(8000);

Parsing Chunked Responses

import { StringDecoder } from 'node:string_decoder';
import https from 'node:https';

https.get('https://api.example.com/data', (res) => {
  const decoder = new StringDecoder('utf8');
  let data = '';
  
  res.on('data', (chunk) => {
    data += decoder.write(chunk);
  });
  
  res.on('end', () => {
    data += decoder.end();
    console.log(JSON.parse(data));
  });
});

Handling Different Encodings

import { StringDecoder } from 'node:string_decoder';
import { Buffer } from 'node:buffer';

// UTF-8 (default)
const utf8Decoder = new StringDecoder('utf8');
console.log(utf8Decoder.write(Buffer.from('Hello')));
// 'Hello'

// UTF-16 Little Endian
const utf16Decoder = new StringDecoder('utf16le');
console.log(utf16Decoder.write(Buffer.from('Hello', 'utf16le')));
// 'Hello'

// Base64
const base64Decoder = new StringDecoder('base64');
console.log(base64Decoder.write(Buffer.from('SGVsbG8=')));
// 'Hello' (base64 decoded)

// Hexadecimal
const hexDecoder = new StringDecoder('hex');
console.log(hexDecoder.write(Buffer.from('48656c6c6f', 'hex')));
// 'Hello'

Multi-byte Character Examples

UTF-8 Characters

import { StringDecoder } from 'node:string_decoder';
import { Buffer } from 'node:buffer';

const decoder = new StringDecoder('utf8');

// 1-byte character (ASCII)
const a = Buffer.from([0x41]);
console.log(decoder.write(a)); // 'A'

// 2-byte character (¢ - cent)
const cent = Buffer.from([0xC2, 0xA2]);
console.log(decoder.write(cent)); // '¢'

// 3-byte character (€ - euro)
const euro = Buffer.from([0xE2, 0x82, 0xAC]);
console.log(decoder.write(euro)); // '€'

// 4-byte character (𝌆 - musical symbol)
const musical = Buffer.from([0xF0, 0x9D, 0x8C, 0x86]);
console.log(decoder.write(musical)); // '𝌆'

Incomplete Character Handling

import { StringDecoder } from 'node:string_decoder';
import { Buffer } from 'node:buffer';

const decoder = new StringDecoder('utf8');

// Euro symbol (€) split across three writes
const byte1 = Buffer.from([0xE2]);
const byte2 = Buffer.from([0x82]);
const byte3 = Buffer.from([0xAC]);

console.log(decoder.write(byte1)); // '' (buffered, incomplete)
console.log(decoder.write(byte2)); // '' (buffered, still incomplete)
console.log(decoder.write(byte3)); // '€' (complete!)

Mixed Content

import { StringDecoder } from 'node:string_decoder';
import { Buffer } from 'node:buffer';

const decoder = new StringDecoder('utf8');

// ASCII followed by incomplete multi-byte character
const mixed = Buffer.from([0x48, 0x69, 0xE2]); // 'Hi' + incomplete €
console.log(decoder.write(mixed)); // 'Hi' (€ buffered)

// Complete the multi-byte character
const completion = Buffer.from([0x82, 0xAC]);
console.log(decoder.write(completion)); // '€'

StringDecoder vs buffer.toString()

When to Use StringDecoder

// ✅ Use StringDecoder for streaming/chunked data
import { StringDecoder } from 'node:string_decoder';

const decoder = new StringDecoder('utf8');
stream.on('data', (chunk) => {
  const text = decoder.write(chunk); // Handles split characters
});

When to Use buffer.toString()

// ✅ Use toString() for complete buffers
import { Buffer } from 'node:buffer';

const completeBuffer = Buffer.from('Complete message', 'utf8');
const text = completeBuffer.toString(); // Simple and efficient

Best Practices

1. Use for streaming data - Essential when processing chunked data
2. Call end() when done - Always call end() to flush buffered data
3. Choose correct encoding - Match the encoding of your source data
4. Reuse instances - Create one decoder per stream, not per chunk
5. Handle errors - Validate encoding parameters

Performance Considerations

import { StringDecoder } from 'node:string_decoder';
import { Buffer } from 'node:buffer';

// ✅ Good: Reuse decoder instance
const decoder = new StringDecoder('utf8');
for (const chunk of chunks) {
  const text = decoder.write(chunk);
}
const final = decoder.end();

// ❌ Bad: Creating decoder per chunk
for (const chunk of chunks) {
  const decoder = new StringDecoder('utf8');
  const text = decoder.write(chunk);
}

Related Modules

• Buffer - Binary data handling
• util - Utility functions including TextDecoder
• stream - Stream processing