What's the best way to setup ExpressJS for GoAccess?

[breck7]

I was previously using GoAccess with NGinx for our static site PLDB.com. Our site is now a dynamic site served by Express.js. I'd still want to use GoAccess.

I'm Googling around now looking at various NPM logging packages, but think the easiest thing to do would just be to write a simple one or two liner that appends to disk and just match the NGinx format, and then I don't have to change anything at all with my Goaccess.

Does anyone have this setup--Express.js + GoAccess? How do you do it?

[breck7]

Here is what I have so far. It's almost working, I just am messing up something with the formatting.

// Log all requests to a log file in the log folder similar to how NGINX does it:
    app.use((req: any, res: any, next: any) => {
      const userAgent = req.headers["user-agent"] || ""
      const date = new Date().toUTCString()
      const { method, originalUrl, httpVersion, ip, statusCode } = req
      const contentLength = res.get("Content-Length") || 0
      const referer = req.headers.referer || ""
      const logLine = `${ip} - - [${date}] "${method} ${originalUrl} HTTP/${httpVersion}" ${statusCode} ${contentLength} "${referer}" "${userAgent}"\n`
      // Now write to the log file:
      fs.appendFile(this.requestLog, logLine, "utf8", function() {})
      next()
    })

[breck7]

Here is a log line:

::ffff:136.243.228.179 - - [Fri, 03 Feb 2023 17:52:26 GMT] "GET /languages/bag-format.html HTTP/1.1" null 0 "" "Mozilla/5.0 (compatible; DataForSeoBot/1.0; +https://dataforseo.com/dataforseo-bot)"

And here is the GoAccess error:

==5833== FILE: /root/pldb/ignore/access.log
==5833== Parsed 10 lines producing the following errors:
==5833==
==5833== Token 'Fri, 03 Feb 2023 16' doesn't match specifier '%d'
==5833== Token 'Fri, 03 Feb 2023 16' doesn't match specifier '%d'
==5833== Token 'Fri, 03 Feb 2023 16' doesn't match specifier '%d'
==5833== Token 'Fri, 03 Feb 2023 16' doesn't match specifier '%d'
==5833== Token 'Fri, 03 Feb 2023 16' doesn't match specifier '%d'
==5833== Token 'Fri, 03 Feb 2023 16' doesn't match specifier '%d'
==5833== Token 'Fri, 03 Feb 2023 16' doesn't match specifier '%d'
==5833== Token 'Fri, 03 Feb 2023 16' doesn't match specifier '%d'
==5833== Token 'Fri, 03 Feb 2023 16' doesn't match specifier '%d'
==5833== Token 'Fri, 03 Feb 2023 16' doesn't match specifier '%d'
==5833==
==5833== Format Errors - Verify your log/date/time format

[breck7]

Still struggling, but here is my latest:

    // Log all requests to a log file in the log folder similar to how NGINX does it:
    app.use((req: any, res: any, next: any) => {
      const userAgent = req.headers["user-agent"] || ""
      const { method, originalUrl, httpVersion, ip, statusCode } = req
      const contentLength = res.get("Content-Length") || 0
      const referer = req.headers.referer || ""
      // https://en.wikipedia.org/wiki/Common_Log_Format
      // Set the date to match the format %d/%b/%Y:%H:%M:%S %z
      const date = dayjs().format("DD/MMM/YYYY:HH:mm:ss ZZ")
      const commonLogFormat = `${ip} - - [${date}] "${method} ${originalUrl} HTTP/${httpVersion}" ${statusCode} ${contentLength}`
      const combinedLogFormat = `${commonLogFormat} "${referer}" "${userAgent}"\n`
      // Now write to the log file:
      fs.appendFile(this.requestLog, combinedLogFormat, "utf8", function() {})
      next()
    })

Produces:

::ffff:17.241.219.127 - - [03/Feb/2023:18:26:14 +0000] "GET //languages/newlisp.html HTTP/1.1" null 0 "" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_5) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1.1 Safari/605.1.15 (Applebot/0.1; +http://www.apple.com/go/applebot)"

Now my error messages are:

==6598== FILE: /root/pldb/ignore/access.log
==6598== 
Parsed 7 lines producing the following errors:
==6598==
==6598== Token 'null' doesn't match specifier '%s'
==6598== Token 'null' doesn't match specifier '%s'
==6598== Token 'null' doesn't match specifier '%s'
==6598== Token 'null' doesn't match specifier '%s'
==6598== Token 'null' doesn't match specifier '%s'
==6598== Token 'null' doesn't match specifier '%s'
==6598== Token 'null' doesn't match specifier '%s'
==6598==
==6598== Format Errors - Verify your log/date/time format

[breck7]

Nevermind! I found the solution: use morgan: https://github.com/expressjs/morgan

I deleted my custom function and replaced with:

app.use(morgan("combined", { stream: fs.createWriteStream(requestLog, { flags: "a" }) }))

morgan is perfect. The other logging middleware for node.js express were bloated.

[allinurl]

Thanks for sharing those findings. I wasn't sure how to answer this as I got no experience using ExpressJS.