TI Providers, Sentinel/Kusto Drivers, Query Editor

Main Changes in this release

Two new TI Providers

Two cool new providers to add to the growing family in MSTICPy:

• CrowdSec is a commercial Malicious IP threat service
  with a free tier for limited threat lookups. (big thanks to @sbs2001 for submitting this)
• AbuseIPDB - is an open/free provider of threat intel
  on malicious IP addresses, providing a central abuse list to lookup IP addresses that have
  been associated with malicious activity. (big thanks to @rrevuelta for submitting this.)

As with other providers, these are automatically enabled for use if you include settings
for the API keys in your msticpyconfig.yaml

Updated Data providers for Sentinel/Azure Monitor/Log Analytics and Kusto/Azure Data Explorer

In v2.5.0 we introduced replacement drivers for the MS Sentinel/LogAnalytics/Azure Monitor
and Kusto/Azure Data Explorer providers.

The new drivers are based on the Azure SDKs for each data service. You can read the release notes
for them here.

The new drivers give several advantages, like being able to run queries across multiple workspaces
or Kusto clusters in parallel. Splitting large queries by time chunks (split_query_by parameter)
will also run multiple segments in parallel, dramatically speeding up the query. The default
parallelism is 4 simultaneous threads but you can change this (although be wary of the impact
on the data service for highly parallel queries - this may affect other users and services accessing
the data).

The new drivers are now the default drivers for these providers. They are used by default for
the "MSSentinel" and "Kusto" data environment identifiers. For backward compatibility, they will
also continue to support the "MSSentinel_New" and "Kusto_New" identifiers.

To invoke the previous Kqlmagic-based drivers use "MSSentinel_Legacy" or "Kusto_Legacy".

This change also brings a dependency change for MSTICPy. The following packages are now
part of the core installed dependencies:

• azure-kusto-data
• azure-monitor-query

Kqlmagic and its dependencies are no longer installed by default but can be installed with the "kql" extra:

python -m pip install msticpy[kql]

See these links to read more about the MSSentinel provider and Kusto providers.

Query Editor

We've added an ipywidgets based query template editor .

note: this is somewhat provisional so please be sure to test and report bugs.

The query editor allows you to edit existing query files or create new ones and helps manage
the various query properties (like parameter definitions) and query metadata.

Check out the documentation on how to use this in the Extending section of the MSTICPy documentation.

Updates to Authentication.

The improvements here mainly affect the AzureData and MicrosoftSentinel classes but'
also bring some improvements to the core authentication - such as being able to specify
the Azure cloud from the az_connect function and authenticate by providing an
AzureCredential.

• You can now authenticate by supplying an AzureCredential as a credential parameter
  for AzureData and MicrosoftSentinel connect methods.
• The connect methods for both these classes also support cloud parameter to specify different sovereign clouds
• The __init__ and connect methods are instrumented with logging to help debug issues:

import msticpy as mp
from msticpy.context.azure.sentinel_core import MicrosoftSentinel

mp.set_logging_level("INFO")
mssentinel = MicrosoftSentinel()
mssentinel.connect()

Other major items

• MS Sentinel delete watchlist API added by @mbabinski
• Splunk fixes added by @Tatsuya-hasegawa

Thanks

Our thanks to the following folks who contributed to this release.
@FlorianBracq
@sbs2001
@rrevuelta
@mbabinski
@Tatsuya-hasegawa

What's Changed

• Add CrowdSec TIProvider by @sbs2001 in #673
• Added delete_watchlist_item method by @mbabinski in #682
• Update pandas requirement from <2.0.0,>=1.4.0 to >=1.4.0,<3.0.0 by @dependabot in #653
• Bump sphinx from 6.1.3 to 7.1.0 by @dependabot in #686
• Add AbuseIPDB TIProvider by @rrevuelta in #687
• Typo corrections in queries by @ianhelle in #684
• Ianhelle/query editor 2023 04 21 by @ianhelle in #685
• Few fix splunk driver by @Tatsuya-hasegawa in #688
• Ianhelle/mssentinel auth 2023 08 01 by @ianhelle in #690
• Updating timeline docs to prioritize pd accessors by @ianhelle in #691
• Fix splunk uploader create index option by @Tatsuya-hasegawa in #692
• v2.7.0 - changing new kql/sentinel drivers to be defaults by @ianhelle in #696

New Contributors

• @sbs2001 made their first contribution in #673
• @mbabinski made their first contribution in #682

Full Changelog: v2.6.0...v2.7.0