File data logging v2

[regit]

Update of #12042

Contribution style:

• I have read the contributing guide lines at
  https://docs.suricata.io/en/latest/devguide/contributing/contribution-process.html

Our Contribution agreements:

• I have signed the Open Information Security Foundation contribution agreement at
  https://suricata.io/about/contribution-agreement/ (note: this is only required once)

Changes (if applicable):

• I have updated the User Guide (in doc/userguide/) to reflect the changes made
• I have updated the JSON schema (in etc/schema.json) to reflect all logging changes
  (including schema descriptions)
• I have created a ticket at
  https://redmine.openinfosecfoundation.org/projects/suricata/issues

Link to ticket: https://redmine.openinfosecfoundation.org/issues/7347

Describe changes:

• Rebase on latest master
• Remove conditional logging of sha256 and add comment
• Add suricata-verify test

Provide values to any of the below to override the defaults.

• To use an LibHTP, Suricata-Verify or Suricata-Update pull request,
  link to the pull request in the respective _BRANCH variable.
• Leave unused overrides blank or remove.

SV_BRANCH=OISF/suricata-verify#2113

[codecov]

Codecov Report

Attention: Patch coverage is 37.50000% with 15 lines in your changes missing coverage. Please review.

Project coverage is 79.79%. Comparing base (b1e7917) to head (1e7720c).

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #12080      +/-   ##
==========================================
- Coverage   83.37%   79.79%   -3.59%     
==========================================
  Files         910      910              
  Lines      257556   257399     -157     
==========================================
- Hits       214748   205383    -9365     
- Misses      42808    52016    +9208     

Flag	Coverage Δ
fuzzcorpus	61.55% <37.50%> (+<0.01%)	⬆️
livemode	19.41% <8.33%> (-0.01%)	⬇️
pcap	44.54% <37.50%> (+0.03%)	⬆️
suricata-verify	?
unittests	59.35% <0.00%> (-0.01%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

[suricata-qa]

Information: QA ran without warnings.

Pipeline 23242

[catenacyber]

Wonder if these fields should be grouped together in a subobject (making the meaning of offset more clear)

[regit]

It looks like a good idea. I can try implementing that to see if it works ok.

[catenacyber]

Why this include added ?

[regit]

I need to fix my LSP which is adding these headers by itself...

[catenacyber]

Why is this better than using filestore ?

[regit]

Why is this better than using filestore ?

I see multiple advantages which depends on the use case:

• this provide a field in JSON with basic information about the file so user can make sure about what type of data it is without doing correlation. It can also be used to run some data analysis on the start of the data without having to have full file extraction.
• JSON is not kept of the probe on a multi probe system so it allows basic understanding of binary content without querying a probe for the file

Where it definitely not work well:

• reverse engineering use case
• sandbox usage

[regit]

Why is this better than using filestore ?

I see multiple advantages which depends on the use case:

• this provide a field in JSON with basic information about the file so user can make sure about what type of data it is without doing correlation. It can also be used to run some data analysis on the start of the data without having to have full file extraction.
• JSON is not kept of the probe on a multi probe system so it allows basic understanding of binary content without querying a probe for the file

Where it definitely not work well:

• reverse engineering use case
• sandbox usage

These examples apart, I would be happy to take some feedback from data analyst or SOC expert to see if they would be interested.