Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

transform: luaxform transform script. #11733

Closed
wants to merge 7 commits into from
Closed

Conversation

jlucovsky
Copy link
Contributor

@jlucovsky jlucovsky commented Sep 9, 2024

Add the luaxform transform that uses a Lua script to receive and transform a sticky buffer.

Lua transforms receive arguments

  • (always): length of input buffer and input buffer
  • (always): number of additional arguments (may be 0)
  • (when present): argument array (bounded by argument count)

Arguments are passed to the lua script as they appear in the rule:

 luaxform:./lua/lua-transform.lua, bytes 4, offset 13, hash sha256;

When the transform is invoked, 3 additional arguments are passed (arg count will be 3)

  • bytes 4
  • offset 13
  • hash sha256

The Lua script is responsible for parsing and using the arguments and returning a tuple: output-buffer, output-buffer-byte-count

I used this lua script:

function init (args)
    local needs = {}
    return needs
end

function transform(input_len, input, argc, args)
    SCLogNotice("Input length: " .. input_len .. " arg-count: " .. argc)
    for i = 1, argc do
        print("Argument " .. i .. ":", args[i])
    end
    return string.upper(input), input_len
end

return 0

Example rule using the lua script:

alert http any any -> any any (msg:"Lua Xform example"; flow:established;  file.data; luaxform:./lua/lua-transform.lua; content: "abc"; sid: 1;)

Link to ticket: https://redmine.openinfosecfoundation.org/issues/2290

Describe changes:

  • Luaxform transform changes

Provide values to any of the below to override the defaults.

  • To use an LibHTP, Suricata-Verify or Suricata-Update pull request,
    link to the pull request in the respective _BRANCH variable.
  • Leave unused overrides blank or remove.

SV_REPO=
SV_BRANCH=OISF/suricata-verify#2044
SU_REPO=
SU_BRANCH=
LIBHTP_REPO=
LIBHTP_BRANCH=

@jasonish
Copy link
Member

jasonish commented Sep 9, 2024

Given the rule:

alert http any any -> any any (msg:"Lua Xform example"; flow:established;  file.data; luaxform:./lua/lua-transform.lua; content: "abc"; sid: 1;)

I wonder if we should reconsider the new keyword here... Would it make more sense to have:

xform:lua,./lua/lua-transform.lua;

in case we add another method for transforms? In theory we could also have plugins that register transforms, but I guess those would look more like the current transforms as they'd add a new keyword completely. Just thinking out loud here.

Copy link

codecov bot commented Sep 9, 2024

Codecov Report

Attention: Patch coverage is 76.97595% with 67 lines in your changes missing coverage. Please review.

Please upload report for BASE (master@1420c83). Learn more about missing BASE report.
Report is 23 commits behind head on master.

Additional details and impacted files
@@            Coverage Diff            @@
##             master   #11733   +/-   ##
=========================================
  Coverage          ?   82.58%           
=========================================
  Files             ?      922           
  Lines             ?   249246           
  Branches          ?        0           
=========================================
  Hits              ?   205838           
  Misses            ?    43408           
  Partials          ?        0           
Flag Coverage Δ
fuzzcorpus 60.40% <28.17%> (?)
livemode 18.71% <15.46%> (?)
pcap 44.07% <18.55%> (?)
suricata-verify 61.90% <76.63%> (?)
unittests 58.95% <21.30%> (?)

Flags with carried forward coverage won't be shown. Click here to find out more.

src/detect-lua.h Outdated Show resolved Hide resolved
@suricata-qa
Copy link

ERROR:

ERROR: QA failed on build_asan.

Pipeline 22483

@victorjulien
Copy link
Member

Given the rule:

alert http any any -> any any (msg:"Lua Xform example"; flow:established;  file.data; luaxform:./lua/lua-transform.lua; content: "abc"; sid: 1;)

I wonder if we should reconsider the new keyword here... Would it make more sense to have:

xform:lua,./lua/lua-transform.lua;

in case we add another method for transforms? In theory we could also have plugins that register transforms, but I guess those would look more like the current transforms as they'd add a new keyword completely. Just thinking out loud here.

I would consider this to be out of scope for this work. Transforms are currently implemented as regular keywords. We can change that of course, but it's not related to the PR.

@victorjulien
Copy link
Member

What would you see happen in the script's init function? I think it's probably not something we need, as the transform is tied to the buffer it will transform.

@jasonish
Copy link
Member

What would you see happen in the script's init function? I think it's probably not something we need, as the transform is tied to the buffer it will transform.

I was wondering about the same thing earlier... And was wondering if perhaps a data-file might be loaded, that would fail now due to the sandbox. But imagine a Lua transform that loaded something a TLD database to provide a domain name transformation or something. Or some other heavy initialization of pre-computing some table of sorts?

I don't think it should be required if not needed though, but might be nice to allow it for some one-time initialization.

@victorjulien
Copy link
Member

What would you see happen in the script's init function? I think it's probably not something we need, as the transform is tied to the buffer it will transform.

I was wondering about the same thing earlier... And was wondering if perhaps a data-file might be loaded, that would fail now due to the sandbox. But imagine a Lua transform that loaded something a TLD database to provide a domain name transformation or something. Or some other heavy initialization of pre-computing some table of sorts?

I don't think it should be required if not needed though, but might be nice to allow it for some one-time initialization.

This did make me think of one possible case: if the transform would like to query a dataset, it would need some init/thread_init to get a handle.

@jlucovsky
Copy link
Contributor Author

What would you see happen in the script's init function? I think it's probably not something we need, as the transform is tied to the buffer it will transform.

I was wondering about the same thing earlier... And was wondering if perhaps a data-file might be loaded, that would fail now due to the sandbox. But imagine a Lua transform that loaded something a TLD database to provide a domain name transformation or something. Or some other heavy initialization of pre-computing some table of sorts?
I don't think it should be required if not needed though, but might be nice to allow it for some one-time initialization.

This did make me think of one possible case: if the transform would like to query a dataset, it would need some init/thread_init to get a handle.

Let's keep the init function then?

@jasonish
Copy link
Member

Let's keep the init function then?

Optional? If it exists use it, if not assume an empty return value?

@victorjulien
Copy link
Member

Let's keep the init function then?

Optional? If it exists use it, if not assume an empty return value?

Works for me. But need to think about init vs thread_init (that I added in my dataset binding work). In init you can't set anything up in the LuaState that will be used at match time. Thats what that new thread_init would offer.

@suricata-qa
Copy link

ERROR:

ERROR: QA failed on build_asan.

Pipeline 22622

@suricata-qa
Copy link

ERROR:

ERROR: QA failed on SURI_TLPR1_alerts_cmp.

Pipeline 22627

@suricata-qa
Copy link

Information: QA ran without warnings.

Pipeline 22743

This commit makes the detection engine thread context available for
transforms to use. The Lua transform requires this value.

Issue: 2290
Issue: 2290

This commit extends the hash table logic with an alternate free function
that provides the detection engine context.

Users that wish to use the next functionality must use the
HashListTableInitWithCtx function when initializing the hash table.
Using this interface will result in the hash table "free with context"
function (new) being used instead.
Issue: 2290

This commit causes the keyword_hash pointer to be cleared after it's
been freed.
Issue: 2290

This commit adds the source files for the new transform -- luaxform.
@suricata-qa
Copy link

Information: QA ran without warnings.

Pipeline 22751

Issue: 2290

Fixup the macro usage to eliminate compiler warnings.
@suricata-qa
Copy link

WARNING:

field baseline test %
SURI_TLPR1_stats_chk
.uptime 649 628 96.76%

Pipeline 22762

@jlucovsky jlucovsky marked this pull request as ready for review September 21, 2024 14:50
@jlucovsky jlucovsky changed the title Draft: Luaxform support transform: luaxform transform script. Sep 21, 2024
@jlucovsky
Copy link
Contributor Author

Continued in #11817

@jlucovsky jlucovsky closed this Sep 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

4 participants