[bug] L4Z implementation is broken on nodejs18

[eau-de-la-seine]

Describe the bug

Hi, the L4Z implementation suggested in the documentation is broken on nodejs18

To Reproduce

npm install kafkajs-lz4

import { Message, CompressionCodecs, CompressionTypes } from 'kafkajs';
import LZ4Codec from 'kafkajs-lz4';

// Initialize L4Z implementation
CompressionCodecs[CompressionTypes.LZ4] = new LZ4Codec().codec;

// ...
await producer.send({
  topic,
  messages,
  compression: CompressionTypes.LZ4,
});

Observed behavior

The error is about the _lz4.wasm file:

RuntimeError: abort(RuntimeError: abort(Error: ENOENT: no such file or directory, open '/var/task/_lz4.wasm'). Build with -s ASSERTIONS=1 for more info.). Build with -s ASSERTIONS=1 for more info.

Environment:

• OS: AWS Lambda
• KafkaJS version 2.2.3
• NodeJS version: 18

Additional context

The issue has already been reported, and there's a hackaround that doesn't work for me, surely because I use esbuild.

The kafkajs-lz4 implementation seems veeeery old (+2y without a commit, and still in beta), do you have an alternative? Thanks

[bjornmolin]

We switched to lz4-kafkajs yesterday (also nodejs 18)

[douglasg14b]

Seeing this on Node16 as well for some reason.

Also on Node 18 too.

[ArthurGLACET]

Hello all,

I'm having the issue too, here what i've made:

on documentation it's written that you can easely create you own codec:

const lz4 = require('lz4'); // using https://www.npmjs.com/package/lz4

module.exports = {
    /**
     * @param {Encoder} encoder
     * @returns {Promise}
     */
    async compress(encoder) {
        return lz4.encode(encoder.buffer);
    },
  
    /**
     * @param {Buffer} buffer
     * @returns {Promise}
     */
    async decompress(buffer) {
        return lz4.decode(buffer);
    },
  }

and then :

const { Kafka, CompressionTypes, CompressionCodecs } = require('kafkajs');
const lz4Codec = require('lz4Codec');

CompressionCodecs[CompressionTypes.LZ4] = () => lz4Codec;

const kafka = new Kafka({
        brokers: [broker],
        clientId: 'test client',
    });

const producer = kafka.producer();
await producer.connect();
await producer.send({
    compression: CompressionTypes.LZ4,
    topic: topic,
    messages: [{value: message}],
});