Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[BUG] zeek-files-labeler checks headers on Zeek JSON files when it should not #24

Open
verovaleros opened this issue May 21, 2023 · 0 comments
Labels
bug Something isn't working

Comments

@verovaleros
Copy link
Member

Describe the bug
The JSON conn.log files have no headers. Each line is a JSON line:

zeek@zeek:~/zeek-test/json$ cat conn.log
{"ts":1591367999.305988,"uid":"CMdzit1AMNsmfAIiQc","id.orig_h":"192.168.4.76","id.orig_p":36844,"id.resp_h":"192.168.4.1","id.resp_p":53,"proto":"udp","service":"dns","duration":0.06685185432434082,"orig_bytes":62,"resp_bytes":141,"conn_state":"SF","missed_bytes":0,"history":"Dd","orig_pkts":2,"orig_ip_bytes":118,"resp_pkts":2,"resp_ip_bytes":197}
{"ts":1591367999.430166,"uid":"C5bLoe2Mvxqhawzqqd","id.orig_h":"192.168.4.76","id.orig_p":46378,"id.resp_h":"31.3.245.133","id.resp_p":80,"proto":"tcp","service":"http","duration":0.25411510467529297,"orig_bytes":77,"resp_bytes":295,"conn_state":"SF","missed_bytes":0,"history":"ShADadFf","orig_pkts":6,"orig_ip_bytes":397,"resp_pkts":4,"resp_ip_bytes":511}

When running the zeek-files-labeler.py tool on a Zeek JSON file, the following error is shown:

Zeek Files labeler from labeled conn.log.labeled file. Version 0.1
https://stratosphereips.org
[+] Labeled file to use: dataset/test.log
The labeled file has not headers. Please add them.

To Reproduce
Sample Zeek JSON conn.log to test:

{"ts":1684227625.431351,"uid":"CMWzya3GuKbV6hku2l","id.orig_h":"192.168.1.243","id.orig_p":35358,"id.resp_h":"172.31.0.2","id.resp_p":12654,"proto":"tcp","service":"http","duration":6.867906093597412,"orig_bytes":531,"resp_bytes":0,"conn_state":"SH","local_orig":false,"local_resp":true,"missed_bytes":0,"history":"ScADFa","orig_pkts":6,"orig_ip_bytes":855,"resp_pkts":1,"resp_ip_bytes":52}
{"ts":1684227665.237912,"uid":"CRm5Me2cKCMEbBrDJc","id.orig_h":"192.168.1.18","id.orig_p":50118,"id.resp_h":"172.31.0.2","id.resp_p":12654,"proto":"tcp","service":"http","duration":5.09847617149353,"orig_bytes":625,"resp_bytes":0,"conn_state":"SH","local_orig":false,"local_resp":true,"missed_bytes":0,"history":"ScADFa","orig_pkts":8,"orig_ip_bytes":957,"resp_pkts":1,"resp_ip_bytes":40}
{"ts":1684227665.237911,"uid":"CXzhcA3CmZhXikjFQk","id.orig_h":"192.168.1.18","id.orig_p":50117,"id.resp_h":"172.31.0.2","id.resp_p":12654,"proto":"tcp","service":"http","duration":5.098514080047607,"orig_bytes":626,"resp_bytes":0,"conn_state":"SH","local_orig":false,"local_resp":true,"missed_bytes":0,"history":"ScADFa","orig_pkts":8,"orig_ip_bytes":958,"resp_pkts":1,"resp_ip_bytes":40}
{"ts":1684227664.630489,"uid":"ChtLqc1NJMnOu1maA7","id.orig_h":"192.168.1.18","id.orig_p":50110,"id.resp_h":"172.31.0.2","id.resp_p":12654,"proto":"tcp","service":"http","duration":5.712158918380737,"orig_bytes":1252,"resp_bytes":0,"conn_state":"SH","local_orig":false,"local_resp":true,"missed_bytes":0,"history":"ScADcFa","orig_pkts":45,"orig_ip_bytes":3064,"resp_pkts":1,"resp_ip_bytes":40}
{"ts":1684227665.23784,"uid":"C5l2rP1L5skfzoAZaa","id.orig_h":"192.168.1.18","id.orig_p":50119,"id.resp_h":"172.31.0.2","id.resp_p":12654,"proto":"tcp","service":"http","duration":5.12164306640625,"orig_bytes":946,"resp_bytes":0,"conn_state":"SH","local_orig":false,"local_resp":true,"missed_bytes":0,"history":"ScADcFa","orig_pkts":14,"orig_ip_bytes":1518,"resp_pkts":1,"resp_ip_bytes":40}
{"ts":1684227665.236117,"uid":"CYp05kkOOJ3pCe8Vb","id.orig_h":"192.168.1.18","id.orig_p":50116,"id.resp_h":"172.31.0.2","id.resp_p":12654,"proto":"tcp","service":"http","duration":5.1245081424713135,"orig_bytes":939,"resp_bytes":0,"conn_state":"SH","local_orig":false,"local_resp":true,"missed_bytes":0,"history":"ScADcFa","orig_pkts":11,"orig_ip_bytes":1391,"resp_pkts":1,"resp_ip_bytes":40}
{"ts":1684227664.630591,"uid":"CBghhD3zflPexGTckj","id.orig_h":"192.168.1.18","id.orig_p":50111,"id.resp_h":"172.31.0.2","id.resp_p":12654,"proto":"tcp","service":"http","duration":5.730525016784668,"orig_bytes":1253,"resp_bytes":0,"conn_state":"SH","local_orig":false,"local_resp":true,"missed_bytes":0,"history":"ScADcF","orig_pkts":21,"orig_ip_bytes":2105,"resp_pkts":0,"resp_ip_bytes":0}
{"ts":1684227665.255085,"uid":"CtWt0LXsI5Wv8gp71","id.orig_h":"192.168.1.18","id.orig_p":50121,"id.resp_h":"172.31.0.2","id.resp_p":12654,"proto":"tcp","service":"http","duration":5.215744972229004,"orig_bytes":674,"resp_bytes":0,"conn_state":"SH","local_orig":false,"local_resp":true,"missed_bytes":0,"history":"ScADFa","orig_pkts":9,"orig_ip_bytes":1046,"resp_pkts":1,"resp_ip_bytes":40}
{"ts":1684227665.25312,"uid":"CaCebC3q1DDGFKUNch","id.orig_h":"192.168.1.18","id.orig_p":50120,"id.resp_h":"172.31.0.2","id.resp_p":12654,"proto":"tcp","service":"http","duration":5.284203052520752,"orig_bytes":1017,"resp_bytes":0,"conn_state":"SH","local_orig":false,"local_resp":true,"missed_bytes":0,"history":"ScADcFa","orig_pkts":14,"orig_ip_bytes":1589,"resp_pkts":1,"resp_ip_bytes":40}
{"ts":1684227665.421915,"uid":"CYQlIQoh90Anqd2ae","id.orig_h":"192.168.1.18","id.orig_p":50123,"id.resp_h":"172.31.0.2","id.resp_p":12654,"proto":"tcp","duration":5.478057861328125,"orig_bytes":0,"resp_bytes":0,"conn_state":"SH","local_orig":false,"local_resp":true,"missed_bytes":0,"history":"ScAF","orig_pkts":4,"orig_ip_bytes":172,"resp_pkts":0,"resp_ip_bytes":0}

Expected behaviour
Expect the tool to parse this type of logs without error or print an unsupported type message.

Resources
Check documentation here: https://docs.zeek.org/en/master/log-formats.html

@verovaleros verovaleros added the bug Something isn't working label May 21, 2023
@verovaleros verovaleros moved this to 📋 Backlog in Netflow Labeler May 21, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
Status: 📋 Backlog
Development

No branches or pull requests

1 participant