Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Transform stream/.pipe are not working as expected #685

Open
mStirner opened this issue Mar 20, 2022 · 0 comments
Open

Transform stream/.pipe are not working as expected #685

mStirner opened this issue Mar 20, 2022 · 0 comments

Comments

@mStirner
Copy link

I have a very simple setup, where i extend the bunyan logger:

const { Writable, Transform, PassThrough } = require("stream");
const { createLogger } = require("bunyan");

const formatter = require("./formatter");


function Records(options) {

    options = Object.assign({
        // default options
    }, options);

    return new Transform({
        transform: (chunk, encoding, cb) => {

            let rec = JSON.parse(chunk);
            console.log("transform", rec);

            delete rec.v;
            //delete rec.fields;
            delete rec.hostname;
            delete rec.pid;

            rec.time = Date.now();
            rec.msg = formatter(rec);

            cb(null, JSON.stringify(rec));

        }
    });

}


const myWritable = new Writable({
    write(chunk, encoding, cb) {

        let rec = JSON.parse(chunk);

        //console.log("myWritable input", rec)

        delete rec.v;
        //delete rec.fields;
        delete rec.hostname;
        delete rec.pid;

        rec.time = Date.now();
        rec.msg = formatter(rec);

        //console.log("myWritable output", rec);

        console.log(rec.msg);


        if (rec.err) {
            console.log(rec.err.stack)
        }

        cb(null);

    }
});

module.exports = class Logger extends createLogger {

    constructor(options = {}) {

        options = Object.assign({
            streams: [{
                level: "trace",
                type: "stream",
                stream: Records()
            }],
        }, options);

        super(options);

        this.tracer = (desc, count) => {

            let counter = 0;
            let fired = false;

            return (...args) => {

                counter += 1;

                this.trace(`${desc}; ${counter}/${count};`, ...args);

                if (counter === count && !fired) {
                    this.trace(`${desc}; finishd!`);
                    fired = true;
                }

            };
        };

    }

    static create(name, options = {}) {
        return new Logger({
            name,
            ...options
        });
    }

};

I wanted a generic transform stream where i can delete some fields, like hostname & v, but when i pipe be transform stream, the transform stream does not work.

When i use the following stream setup:

 streams: [{
   level: "trace",
   type: "stream",
   stream: Records().pipe(process.stdout)
 }],

my console.log("transform", rec); does log nothing, like the transform stream does nothing.
When is use my myWritable instead, it work as expected.

const Logger = require("./class.logger.js");

const logger = Logger.create("system");

//console.log(logger, logger.fucker, logger.tracer)

const trace = logger.tracer("Count down to 3", 3);

logger.trace("Trace");
trace("called 1");
trace("called 2");
trace("called 3");



logger.debug("Debug");
logger.info("Info");
logger.warn("Warning");
logger.error("error");

logger.fatal("Fuck off, im done...")

The reason for the transform stream is, that i want to use multiple streams, like fs.createWriteStream & process.stdout or what ever, and dont want to code for very stream a filter that deleted the unwanted fields.

Any help/hint is appreciated.

I cant wrap my head around why this dosnt work, and the transform stream is ignored as soon it is somewhere piped.

BTW: I switched from winston to bunyan, because its has less dependencies & got faster results "hacking" bunyan than winston.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant