Threat Bus 2021.02.24

We’re happy to announce the monthly release 2021.02.24 of Threat Bus. We now use STIX-2 as the internal format for indicators and sightings. This is the first step forward on our brand new roadmap for making Threat Bus a format-agnostic security content dissemination layer. Along with smaller improvements, the MISP plugin now comes with optional dependencies and pyvast-threatbus supports simultaneous live- and retro-matching with VAST.

STIX-2 Support

Threat Bus now supports STIX-2 (version 2.1) as internal format for indicators of compromise and sightings. We removed the hand-rolled types Intel and Sightingfrom threatbus.data together with all custom serialization logic.

We migrated half of the existing plugins for compatibility with the STIX-2 format. The two backbone plugins threatbus-rabbitmq and threatbus-inmem now multiplex messages on the stix2/ topic prefix for Threat Bus subscribers. Likewise, the threatbus-zmq-app plugin processes messages in STIX-2 format via that topic. Expect the other app plugins and pyvast-threatbus to be migrated to support STIX-2 with the next releases.

The decision to support STIX-2 is the first step towards making Threat Bus a format-agnostic security content dissemination layer. It will radically simplify integrations with tools that already use open standards, like OpenCTI. But that’s not all. We don’t want the project to be limited to IoCs and sightings. Instead, we plan to support other open threat intelligence standards like Sigma or YARA rules to generalize from point indicators to more powerful descriptions of detections.

For Threat Bus, this means we will gradually add more topic prefixes like sigma/ and yara/. Transported messages will always have a type corresponding to the topic prefix. For example, the stix2/ prefix only transports SITX-2 content and STIX-2 indicators will have the precise topic stix2/indicator. Sigma rules will be sent via the sigma/rule topic and so forth. We are excited to move forward with our roadmap and invite you to tag along on this journey!

Optional MISP Plugin Install Targets

The MISP plugin now uses extra dependencies to relieve the need of installing unwanted dependencies. Users can chose the wanted dependencies during installation by running pip install threatbus-misp[zmq] to install the ZeroMQ dependency, or
pip install threatbus-misp[kafka] to install the Kafka dependency. The plugin throws a fatal error if none of the dependencies are installed.

Simultaneous Live- and Retro-Matching

pyvast-threatbus, the wrapper to connect VAST with Threat Bus, now supports both live- and retro-matching for a single IoC. These features were mutually exclusive prior to this release. Users can now enable both settings in the config.yaml. Under the hood, pyvast-threatbus parses the IoC to a retro-query and forwards the IoC to a VAST matcher. pyvast-threatbus executes the retro-query immediately, converts query results into sightings, and pushes the sightings back to Threat Bus. At the same time, pyvast-threatbus feeds the parsed IoC into the VAST matcher component. From that point forward, VAST’s matcher checks incoming telemetry automatically against the added IoC and immediately reports sightings back to Threat Bus.

Changelog Highlights

As always, you can find the full scoop in our changelogs for Threat Bus and pyvast-threatbus

⚡️ Breaking Changes

• ⚡️ Threat Bus now uses STIX-2 (version 2.1) as internal transport format for Indicators and Sightings. App-plugins now have to provide required mappings between the STIX-2 format and app-specific formats (e.g., the Zeek plugin needs to map STIX-2 to the Zeek intel format). The home-made types threatbus.data.Sighting and threatbus.data.Indicator are removed from the codebase. Plugins with versions earlier than 2021.02.24 are incompatible to the new Threat Bus version. #97

🎁 Features

• 🎁 Feature
  The MISP plugin now uses extra dependencies. Users can now chose the wanted dependencies during installation by running pip install threatbus-misp[zmq] to install the ZeroMQ dependency, or pip install threatbus-misp[kafka] to install the Kafka dependency. The plugin throws a fatal error if none of these dependencies is installed and exits immediately. #99

• 🎁 Feature
  The RabbitMQ backbone plugin, the In-memory backbone plugins, and the Zmq-app plugin now support the STIX-2 (version 2.1) standard for Indicators and Sightings. #97

⚠️ Changes

• ⚠️ Users now can use both, retro-matching and live-matching with VAST simultaneously for any given IoC with pyvast-threatbus. On the flip side, there is no longer a default mode of operation. To use live-matching, users now must specifically configure it via setting live_match: truein theirconfig.yaml` file #95.

• ⚠️ pyvast-threatbus drops support to unflatten JSON that it receives from vast export because VAST can now return unflattened JSON by default #92.