help with virtualmin log format #2596

atomGit · 2023-12-19T19:46:01Z

atomGit
Dec 19, 2023

in goaccess i'm seeing fields with ~T~, for example, in the OS category:

Hits     h% Vis.     v%  Tx. Amount Data
 ---- ------ ---- ------ ----------- ----
 8844 49.23% 1067 44.55%  508.69 MiB Windows (bars removed)
 7618 42.41%  890 37.16%  487.41 MiB   ~T~\ ~T~@ Windows 10 (bars removed)

i wasn't able to find what ~T~ means, but i'm guessing the log format might be wrong??? here's a sample log line (IP and UA removed for brevity):
<IP> - - [17/Dec/2023:04:31:14 -0500] "GET /robots.txt HTTP/1.1" 302 214 "-" "<UA>"

the goaccess default format in the config for Virtualmin is:
log-format %h %^ %v %^[%d:%t %^] "%r" %s %b "%R" "%u"

is this correct?

allinurl · 2023-12-19T23:56:59Z

allinurl
Dec 19, 2023
Maintainer

Please share the version of GoAccess you have installed and provide a snippet of your access log. Thank you.

0 replies

atomGit · 2023-12-20T00:22:52Z

atomGit
Dec 20, 2023
Author

goaccess v1.8.1

i set the IPs to 0.0.0.0

access.log

0.0.0.0 - - [17/Dec/2023:04:31:42 -0500] "GET /articles/history/rescuing-the-tank-index/rescuing-the-tank-the-resources/ HTTP/2.0" 301 272 "-" "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/116.0.1938.76 Safari/537.36"
0.0.0.0 - - [17/Dec/2023:04:32:00 -0500] "GET /feed.xml HTTP/1.1" 304 - "-" "Feedly/1.0 (+http://www.feedly.com/fetcher.html; 9 subscribers; )"
0.0.0.0 - - [17/Dec/2023:04:32:05 -0500] "GET /feed.json HTTP/1.1" 304 - "-" "Feedbin feed-id:1966344 - 5 subscribers"
0.0.0.0 - - [17/Dec/2023:04:32:11 -0500] "GET /feed.xml HTTP/1.1" 304 - "-" "Feedly/1.0 (+http://www.feedly.com/fetcher.html; 25 subscribers; )"
0.0.0.0 - - [17/Dec/2023:04:32:20 -0500] "GET /category/tech/software/feed HTTP/2.0" 301 250 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:120.0) Gecko/20100101 Firefox/120.0"
0.0.0.0 - - [17/Dec/2023:04:32:20 -0500] "GET /tags/tech/software/feed HTTP/2.0" 404 5627 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:120.0) Gecko/20100101 Firefox/120.0"
0.0.0.0 - - [17/Dec/2023:04:32:40 -0500] "GET /wp-login.php HTTP/1.1" 302 216 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
0.0.0.0 - - [17/Dec/2023:04:32:41 -0500] "GET /wp-login.php HTTP/1.1" 403 199 "http://example.org/wp-login.php" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:62.0) Gecko/20100101 Firefox/62.0"
0.0.0.0 - - [17/Dec/2023:04:32:55 -0500] "GET /articles/history/ HTTP/1.1" 301 228 "-" "-"
0.0.0.0 - - [17/Dec/2023:04:32:58 -0500] "GET /feed HTTP/1.1" 302 208 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/116.0.0.0 Safari/537.36"
0.0.0.0 - - [17/Dec/2023:04:33:06 -0500] "GET /feed/ HTTP/1.1" 301 236 "-" "NextCloud-News/1.0"
0.0.0.0 - - [17/Dec/2023:04:33:06 -0500] "GET /feed.xml HTTP/1.1" 200 8434 "-" "NextCloud-News/1.0"
0.0.0.0 - - [17/Dec/2023:04:33:07 -0500] "GET / HTTP/1.1" 200 17715 "https://example.org/articles/history/" "-"
0.0.0.0 - - [17/Dec/2023:04:33:12 -0500] "GET /articles/health/ HTTP/1.1" 301 228 "-" "-"
0.0.0.0 - - [17/Dec/2023:04:33:12 -0500] "GET /the-privacy-guide/ HTTP/2.0" 200 51474 "-" "Mozilla/5.0 (Windows NT 10.0; rv:120.0) Gecko/20100101 Firefox/120.0"
0.0.0.0 - - [17/Dec/2023:04:33:13 -0500] "GET /assets/css/style.css HTTP/2.0" 200 49676 "https://example.org/the-privacy-guide/" "Mozilla/5.0 (Windows NT 10.0; rv:120.0) Gecko/20100101 Firefox/120.0"
0.0.0.0 - - [17/Dec/2023:04:33:13 -0500] "GET /assets/js/scripts.min.js HTTP/2.0" 200 11496 "https://example.org/the-privacy-guide/" "Mozilla/5.0 (Windows NT 10.0; rv:120.0) Gecko/20100101 Firefox/120.0"
0.0.0.0 - - [17/Dec/2023:04:33:13 -0500] "GET /assets/svg/svg-map.svg HTTP/2.0" 200 9114 "https://example.org/the-privacy-guide/" "Mozilla/5.0 (Windows NT 10.0; rv:120.0) Gecko/20100101 Firefox/120.0"
0.0.0.0 - - [17/Dec/2023:04:33:13 -0500] "GET /media/posts/70/facespook-meme.png HTTP/2.0" 200 529005 "https://example.org/the-privacy-guide/" "Mozilla/5.0 (Windows NT 10.0; rv:120.0) Gecko/20100101 Firefox/120.0"
0.0.0.0 - - [17/Dec/2023:04:33:13 -0500] "GET /media/website/example.png HTTP/2.0" 200 37110 "https://example.org/the-privacy-guide/" "Mozilla/5.0 (Windows NT 10.0; rv:120.0) Gecko/20100101 Firefox/120.0"
0.0.0.0 - - [17/Dec/2023:04:33:20 -0500] "GET / HTTP/1.1" 302 204 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36"
0.0.0.0 - - [17/Dec/2023:04:33:21 -0500] "GET / HTTP/1.1" 200 17715 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36"
0.0.0.0 - - [17/Dec/2023:04:33:22 -0500] "GET /media/files/about-profiles_set-default-800x1015.png HTTP/2.0" 200 564047 "https://example.org/the-privacy-guide/" "Mozilla/5.0 (Windows NT 10.0; rv:120.0) Gecko/20100101 Firefox/120.0"
0.0.0.0 - - [17/Dec/2023:04:33:26 -0500] "GET /tags/privacy/ HTTP/2.0" 200 18259 "https://example.org/the-privacy-guide/" "Mozilla/5.0 (Windows NT 10.0; rv:120.0) Gecko/20100101 Firefox/120.0"
0.0.0.0 - - [17/Dec/2023:04:33:26 -0500] "GET /media/posts/252/meme-toast-to-big-brother.jpg HTTP/2.0" 200 54305 "https://example.org/tags/privacy/" "Mozilla/5.0 (Windows NT 10.0; rv:120.0) Gecko/20100101 Firefox/120.0"
0.0.0.0 - - [17/Dec/2023:04:33:26 -0500] "GET /media/posts/207/azirevpn-servers.jpg HTTP/2.0" 200 238680 "https://example.org/tags/privacy/" "Mozilla/5.0 (Windows NT 10.0; rv:120.0) Gecko/20100101 Firefox/120.0"
0.0.0.0 - - [17/Dec/2023:04:33:26 -0500] "GET /media/posts/92/censorship.png HTTP/2.0" 200 16042 "https://example.org/tags/privacy/" "Mozilla/5.0 (Windows NT 10.0; rv:120.0) Gecko/20100101 Firefox/120.0"
0.0.0.0 - - [17/Dec/2023:04:33:26 -0500] "GET /media/posts/230/steam-malware.png HTTP/2.0" 200 429695 "https://example.org/tags/privacy/" "Mozilla/5.0 (Windows NT 10.0; rv:120.0) Gecko/20100101 Firefox/120.0"

0 replies

allinurl · 2023-12-21T01:48:07Z

allinurl
Dec 21, 2023
Maintainer

Give this a try; it's working well on my side with the log you shared:

goaccess access.log --log-format=COMBINED

OR

goaccess access.log --log-format='%h %^[%x] "%r" %s %b "%R" "%u"' --datetime-format='%d/%b/%Y:%H:%M:%S %z'

0 replies

atomGit · 2023-12-21T03:35:04Z

atomGit
Dec 21, 2023
Author

hello

--log-format=COMBINED results in the same ~T~ strings for Visitor Hostnames and IPs

and...
--log-format='%h %^[%x] "%r" %s %b "%R" "%u"' --datetime-format='%d/%b/%Y:%H:%M:%S %z'
...results in:

==231184== Parsed 10 lines producing the following errors:
==231184==
==231184== Token '17/Dec/2023:04:31:14 -0500' doesn't match specifier '%x'
==231184== Token '17/Dec/2023:04:31:14 -0500' doesn't match specifier '%x'
==231184== Token '17/Dec/2023:04:31:42 -0500' doesn't match specifier '%x'
==231184== Token '17/Dec/2023:04:32:00 -0500' doesn't match specifier '%x'
==231184== Token '17/Dec/2023:04:32:05 -0500' doesn't match specifier '%x'
==231184== Token '17/Dec/2023:04:32:11 -0500' doesn't match specifier '%x'
==231184== Token '17/Dec/2023:04:32:20 -0500' doesn't match specifier '%x'
==231184== Token '17/Dec/2023:04:32:20 -0500' doesn't match specifier '%x'
==231184== Token '17/Dec/2023:04:32:40 -0500' doesn't match specifier '%x'
==231184== Token '17/Dec/2023:04:32:41 -0500' doesn't match specifier '%x'
==231184==
==231184== Format Errors - Verify your log/date/time format

attached is a bigger slice of the log if it helps...
access.log

0 replies

allinurl · 2023-12-23T02:40:31Z

allinurl
Dec 23, 2023
Maintainer

Is your system's locale set to English? If not, it's likely that your logs use English dates with a different date locale on your machine. I suggest running it with lc_time.

LC_TIME="en_US.UTF-8" goaccess access.log --log-format='%h %^[%x] "%r" %s %b "%R" "%u"' --datetime-format='%d/%b/%Y:%H:%M:%S %z'

Here's what I get with your log:

1 reply

allinurl Dec 23, 2023
Maintainer

Also, please build with --enable-utf8 as shown here

atomGit · 2023-12-23T21:51:07Z

atomGit
Dec 23, 2023
Author

Is your system's locale set to English?

yes

your screen cap looks the same as what i'm seeing - to see the ~T~ strings however, you'd have to tab to one of the affected sections, such as Visitor Hostnames and IPs, and then expand it

Also, please build with --enable-utf8

i'm using the package as supplied by the Alma Linux repo and updates are handled automatically, so i'm hesitant to build manually

0 replies

allinurl · 2023-12-24T02:40:41Z

allinurl
Dec 24, 2023
Maintainer

It sounds like the issue lies with Alma Linux package, as it likely does not include ncurses wide character support, and goaccess may have been built with the --enable-utf8 option. Thus the ~T~ is showing instead of ├─.

Can you provide the output of goaccess -V? Below is the extended panel expected from a correctly built package.

0 replies

atomGit · 2023-12-24T02:48:29Z

atomGit
Dec 24, 2023
Author

# goaccess -V
GoAccess - 1.8.1.
For more details visit: https://goaccess.io/
Copyright (C) 2009-2023 by Gerardo Orellana

Build configure arguments:
  --enable-debug
  --enable-utf8
  --enable-geoip=legacy
  --with-getline
  --with-openssl

# dnf list ncurses
Last metadata expiration check: 1:39:17 ago on Sat Dec 23 20:07:58 2023.
Installed Packages
ncurses.x86_64  6.1-10.20180224.el8  @baseos

1 reply

allinurl Dec 24, 2023
Maintainer

I don't currently have access to any Alma Linux machines. Could you please verify if the ncurses package on your system includes wide character support? I'm unsure whether the commands below will produce the necessary information but give it a shot.

# rpm -qi ncurses
# rpm -ql ncurses
# dnf search ncursesw
# find / -name '*ncursesw*'
# find / -name 'libncursesw.so'

I'd look for files with w in their names, such as libncursesw.so.6.

atomGit · 2023-12-24T03:16:41Z

atomGit
Dec 24, 2023
Author

looks like ncursesw is not installed and dnf search ncursesw produces no results

i don't think this is of any help, but...

# rpm -qi ncurses
Name        : ncurses
Version     : 6.1
Release     : 10.20180224.el8
Architecture: x86_64
Install Date: Wed Nov 22 03:45:36 2023
Group       : System Environment/Base
Size        : 589460
License     : MIT
Signature   : RSA/SHA256, Mon Oct 16 04:22:59 2023, Key ID 51d6647ec21ad6ea
Source RPM  : ncurses-6.1-10.20180224.el8.src.rpm
Build Date  : Sat Oct 14 14:54:58 2023
Build Host  : x64-builder01.almalinux.org
Relocations : (not relocatable)
Packager    : AlmaLinux Packaging Team <[email protected]>
Vendor      : AlmaLinux
URL         : https://invisible-island.net/ncurses/ncurses.html
Summary     : Ncurses support utilities
Description :
The curses library routines are a terminal-independent method of
updating character screens with reasonable optimization.  The ncurses
(new curses) library is a freely distributable replacement for the
discontinued 4.4 BSD classic curses library.

lastly, i didn't know that the T thingy is just a missing tree character, so i'm not really concerned about it, so no need to bother yourself if you have better things to do :)

that said, i'll let you decide whether to close this

0 replies

allinurl · 2024-01-04T01:52:25Z

allinurl
Jan 4, 2024
Maintainer

Appreciate the update. There shouldn't be any significant issues, just some cosmetic differences. If you get the opportunity to build or utilize ncursesw, or better yet, suggest to the package maintainer to include ncursesw, I would recommend it.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

help with virtualmin log format #2596

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 10 comments 2 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

help with virtualmin log format #2596

atomGit Dec 19, 2023

Replies: 10 comments · 2 replies

allinurl Dec 19, 2023 Maintainer

atomGit Dec 20, 2023 Author

allinurl Dec 21, 2023 Maintainer

atomGit Dec 21, 2023 Author

allinurl Dec 23, 2023 Maintainer

allinurl Dec 23, 2023 Maintainer

atomGit Dec 23, 2023 Author

allinurl Dec 24, 2023 Maintainer

atomGit Dec 24, 2023 Author

allinurl Dec 24, 2023 Maintainer

atomGit Dec 24, 2023 Author

allinurl Jan 4, 2024 Maintainer

atomGit
Dec 19, 2023

Replies: 10 comments 2 replies

allinurl
Dec 19, 2023
Maintainer

atomGit
Dec 20, 2023
Author

allinurl
Dec 21, 2023
Maintainer

atomGit
Dec 21, 2023
Author

allinurl
Dec 23, 2023
Maintainer

allinurl Dec 23, 2023
Maintainer

atomGit
Dec 23, 2023
Author

allinurl
Dec 24, 2023
Maintainer

atomGit
Dec 24, 2023
Author

allinurl Dec 24, 2023
Maintainer

atomGit
Dec 24, 2023
Author

allinurl
Jan 4, 2024
Maintainer