Request: paging/limit results sets

[edperry]

When walking a Large TREE 1.3.6.1, sometimes the tree is very large and created Huge documents

[2018-09-08T12:44:26,747][DEBUG][logstash.outputs.kafka   ] KafkaProducer.send() failed: org.apache.kafka.common.errors.RecordTooLargeException: The message is 1101208 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration. {:exception=>java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.RecordTooLargeException: The message is 1101208 bytes when serialized which is larger than the maximum request size you have configured with the max.request.size configuration.}

so obviously I could Up the size of MAX messages sending to kafka, but I think it would be great if there was a feature to, split the document output by the input based on some criteria

Some ideas I have are:

• every entry gets its own event
• spit by MIB Tree
• Set a max value of fields and just issue multiple events based on this max

[colinsurprenant]

@edperry trying to think about this with you, wouldn't be an options to simply split the walk into multiple sub-walks which would produce individual documents pre sub-walks?

[edperry]

Sorry for the delay, was on vacation

That maybe a good way of doing it, I was sort of focused on the size of the payload going though Kafka, but let me give that a try, I totally missed the idea of handling it as a filter. 👍

[edperry]

Hmmm, just looked at the data structure and not sure how I would split this data, its not JSON array.
not sure how I would split root fields off to other events :( any suggestions?

"_source": {
    "1.org.dod.internet.mgmt.1.1.9.1.4.8": 8,
    "1.org.dod.internet.mgmt.1.1.9.1.4.4": 8,
    "1.org.dod.internet.mgmt.1.1.5.0": "hd1pbk01lx",
    "1.org.dod.internet.mgmt.1.1.9.1.2.3": "1.3.6.1.6.3.10.3.1.1",
    "1.org.dod.internet.mgmt.1.1.9.1.3.6": "The MIB module for managing IP and ICMP implementations",
    "@timestamp": "2018-10-23T15:06:28.650Z",
    "1.org.dod.internet.mgmt.1.1.9.1.2.6": "1.3.6.1.2.1.4",
    "1.org.dod.internet.mgmt.1.1.6.0": "Linux",
    "1.org.dod.internet.mgmt.1.1.9.1.4.6": 8,
    "host_ip": "192.168.7.29",
    "dst_index": "snmp",
    "1.org.dod.internet.mgmt.1.1.9.1.3.4": "The MIB module for SNMPv2 entities",
    "1.org.dod.internet.mgmt.1.1.9.1.4.2": 8,
    "1.org.dod.internet.mgmt.1.1.9.1.2.5": "1.3.6.1.2.1.49",
    "1.org.dod.internet.mgmt.1.1.9.1.4.3": 8,
    "1.org.dod.internet.mgmt.1.1.9.1.3.8": "View-based Access Control Model for SNMP.",
    "1.org.dod.internet.mgmt.1.1.9.1.2.1": "1.3.6.1.6.3.11.2.3.1.1",
    "type": "snmp-systeminfo",
    "1.org.dod.internet.mgmt.1.1.9.1.3.3": "The SNMP Management Architecture MIB.",
    "1.org.dod.internet.mgmt.1.1.9.1.3.1": "The MIB for Message Processing and Dispatching.",
    "1.org.dod.internet.mgmt.1.1.9.1.4.1": 8,

[edperry]

So thinking about what you wrote more.

I think walking the mib by groups would be nicer, but you still have to worry about size of the results. For example, If I polled the "Packages" or "Processes" running it still could easily exceed any value.

I think an option to just to have a flag to create and event for every MIB object. This would create a lot of little objects, but it also provides the ability to provide the META information/description for each MIB.

[eperry]

Issue logstash-plugins/logstash-integration-snmp#25

I was thinking about converting from

  {
     "OID": oidvalue
  }

to something like

  {
      oid_name: "OID"
      value: oid_value
  }

so this would make 3 options,

mode => batch
mode => single
mode => single_name_value

[eperry]

IDK if making the total fields Limit and depth to higher values is good enough but let me know what you think

  "settings": {
    "index": {
      "mapping": {
        "total_fields": {
          "limit": "600000"
        },
        "depth": {
          "limit": "100"
        }
      },
      "refresh_interval": "5s"
    }
  }

[colinsurprenant]

looking at this and #42 - will get back to you shortly 👀

[edperry]

I was just thinking about this

  {
      oid_name: "OID"
      value: oid_value
  }

oid_value could be either string, int or etc..... so I don't think this format will work or maybe we just create multiple fields.

  {
      "oid_name": "OID"
      "oid_int_val": oid_int_value
      "oid_str_val": "oid_int_value"
       ..... Other types???...........
  }

I am not sure if that is a great idea now to create a "mode => single_name_value: because it is not a straightforward string: value combination, when it comes to ELK