Added in parameter enabled timeframe support

[novaksam]

These changes allow a user to specify a timeframe, in either seconds or milliseconds, to poll an API that requires timestamps as a parameter. Example: https://docs.umbrella.com/umbrella-api/docs/security-activity-report

Obviously I didn't change the asciidoc, nor the spec, but I approached this as solving a problem I had.

Here's an example use:

  http_poller {
    urls => {
      Umbrella => {
        method => get
        url => "https://reports.api.umbrella.com/v1/organizations/[REMOVED]/security-activity?start=time_back_buffer&stop=time_forward_buffer"
        headers => {
          Accept => "application/json"
          Authorization => "Basic [REMOVED]"
        }
     }
    }
    request_timeout => 60
    # Supports "cron", "every", "at" and "in" schedules by rufus scheduler 
    schedule => { cron => "* * * * * UTC"}
    codec => "json"
    time_back_buffer_string => "time_back_buffer"
    time_back_buffer => 900
    time_forward_buffer_string => "time_forward_buffer"
    # A hash of request metadata info (timing, response headers, etc.) will be sent here
    metadata_target => "http_poller_metadata"
  }

[novaksam]

It looks like I may have an issue consistently updating the value; I'll look into it next week.

[hiven]

Nice this looks great

[chrisribe]

Nice work, but why so specific (forward time and backward time).

Don't we only need a way to store last cycle values ? Not all api's are the same, in the case of the reddit api they use after and before keys which are unique ids and not dates. https://www.reddit.com/dev/api/

All I would like to see is a way to keep a last run value just like the JDBC input plugin (state / sql_last_value). https://www.elastic.co/guide/en/logstash/current/plugins-inputs-jdbc.html#_state

Found the ruby code here related to sql_last_value which they store in a last run state file:
https://github.com/logstash-plugins/logstash-input-jdbc/blob/36d22795e55843dff23f18b4ec2c32952988e665/lib/logstash/inputs/jdbc.rb

Would this not simplify the code ?
Chris

[hiven]

Does this work if you want to put the variable in the body as well?

[novaksam]

@hiven Not at the moment, but should be doable; I'd just need to find what part of the request array it's stored in (which might make sense to put the loop back in so it can look at everything).

@chrisribe I suppose it's specific because it fitted the parameters I needed. I originally wrote a ruby script to deal with DUO admin APIs, but when I had to look at pulling other, slightly simpler APIs (DUO has this whole transaction code thing in their APIs (https://github.com/duosecurity/duo_api_ruby/blob/master/lib/duo_api.rb) I found that the ones I was looking at were filterable by timeframes, which was my path of least resistance.

I'm not opposed to putting in a 'last ran' tracker (which looks like just writing data to a file in the jdbc) but it'd take me a little bit of time to do so. Based on the quick glance at the reddit API, if I can get a 'last ran' tracker up, it should be pretty simple to setup the text swapping.

[kadensungbincho]

Any plan on merging this? @colinsurprenant