Lazy loading of Config Editor tabs

Minor release to address some performance problems with the config editor on Azure ML notebooks platform.
The control uses lots of ipywidgets and the AML notebook front-end has some perf issues with lots of widgets.
The improvement is to not load all controls/widgets for the settings editor - only load each tab as it is selected.

The release also includes a minor change to the entities classes - to allow them to be JSON serializable by adding a
JSON Encoder class to entities.Entity.

json.dumps(my_entity_list, cls=entities.Entity.JSONEncoder)

Azure CLI session check

Minor release that implements check for existing Azure CLI credentials and warns user if these are not valid/expired.

Pull request #209 Azure CLI credentials check

Hotfix++ Fixing settings editor not saving values.

I introduced a bug in the previous hotfix that caused the settings editor to ignore changes made.
This should be fixed along with a couple of other related items.

Hotfix - Azure cloud settings fix

Minor fix to cope with incomplete configuration in Azure settings.

1.4.0 Hotfix

Release details

Some of the features scheduled to go into the 1.4.0 release did not get published correctly.
1.4.1 should fix this.
It also includes some updates to fix problems in the Process Tree visualization.

Fixes

b2b7fd8@Bug fixes for process_tree.py and related modules.
Fixed:

• not working with generic data input.
• not displaying command line correctly
• not display time in Hover tool correctly
• not truncating command line for display correctly
• missing params in doc string
• removed dependency on Linux auditd schema for converting numeric fields to strings

Azure cloud, Matrix plot and ProcessTree for MDE

Highlights

We've added comprehensive support for Azure sovereign clouds (China, Germany and US Government).
To set the appropriate cloud open MpConfigEdit and choose the Azure tab.

Read more about this here

There are also additions and improvements to the visualizations.

Matrix Plot is an interaction plot letting you visualize the interactions between two sets of entities.
A typical use would be seeing connections between source and destination IP Addresses.

Read more about this here

Another visualization change is a major overhaul of the ProcessTree visualization. This now accepts generic data
sets (not specifically Azure Sentinel) and has special support for Microsoft Defender for Endpoint logs.
The data format should be auto-detected.

Finally, we've also collected our visualization methods into a single pandas accessor - mp_plot.
Using this accessor you can plot any of the visualizations from a pandas dataframe:

df.mp_plot.timeline()
df.mp_plot.process_tree()

New Features

889112d@Matrix plot visualization (#197)* Matrix plot visualization
Includes

• vis/matrix_plot.py - main plotting code
• MatrixPlot.ipynb - notebook
• visualization/MatrixPlot.rst - documentation
• tests/vis/test_matrix_plot.py - unit test
• mp_pandas_plot - generic pandas extension "mp_plot" that includes all
  visualizations

d425c77@Process tree updates to handle more generic data (#178)*

• Tidying up some of the code in timeline.py
• Moving the pd accessors to timeline_pd_accessor.py
• Updating notebook EventTimeline.ipynb and documentation EventTimeline.rst
• Unit tests for timeline controls in test_timeline.py
• Removed line to slim down columns - need to keep original data columns because they may be
  referenced in source_columns, etc.
• Process tree updates for MDE process data
• Updating docs, adding pid_fmt parameter
• Adding explanation of schema to NB and doc
• Adding updated notebook
• Expanded the schema table to show all types.

a00741d@Adding support for multiple Azure Clouds (#193)
Co-authored-by: Pete Bryan [email protected]

• Adding cloud options to settings, kql_driver and keyvault
• Updated help text and combined CESimpleSettings class.
• Azure auth multicloud support
• Replacing hard-coded URIs with msrestazure enums
• Modularize multicloud support
• Refactoring code to use AzureCloudConfig instead of reading directly from config
  • azure_auth.py - uses AzureCloudConfig, added credential_scopes option SubscriptionClient init, spelling corrections
  • azure_auth_core.py - create AzureCloudConfig, and _AzCachedConnect classes to read cloud config values and cache tokens
  • cred_wrapper.py - minor formatting
  • keyvault_settings.py - use AzureCloudConfig
  • ce_common.py - uses AzureCloudConfig in get_def_tenant_id function
  • azure_data.py - uses AzureCloudConfig, added credential_scopes option to client class inits, spelling corrections
  • azure_sentinel.py - uses AzureCloudConfig. added functions to set default subscription and workspace, changed _build_paths to method
  • kql_driver.py - uses AzureCloudConfig, added load_mode == Silent, added mp_az_auth to control using Azure auth instead of native Kqlmagic auth,
  • resource_graph_driver.py - uses AzureCloudConfig, added credential_scopes option SubscriptionClient init
  • test_azure_auth_core - Added test for AzureCloudConfig class
  • test_resource_graph_driver.py, test_azure_data.py, test_azure_sentinel.py - added real credential classes to test
  • msticpyconfig-test.yaml, msticpyconfig.yaml - added Azure sections to test config.
• Update azure-mgmt-subscription and azure_mgmt-resource versions

Fixes

d81801d@Ianhelle/tests and tidy 2021 09 01 (#200)*

• Added separate unit test for nbwidgets to cover cases missed in notebook.
• Removing deprecated code from utility.py
• Removing unneeded function from mordor_driver.py
• Sourcery code refactor in entity.py
• Deprecating create_ip_record function in ip_utils.py
• Split wsconfig tests out of test_pkg_config.py
• Adding test_wsconfig.py. Added new test for couple of utility methods
• Removing redundant import from test_nbwidgets.py
• Moved lots of test files into sub-package folders corresponding to tested modules
• Merged a few tests into single files - test_entity.py and test_timeline.py
• Re-enabling IP_stack/geoip notebook test in test_geoip.py
• Updating use of tqdm to remove deprecated tqdm_notebook in utility.py
• Adding some debugging output to geoip GeoLite class.
• Updating test_geoip.py to use predictable msticpyconfig-test.yaml
• Updating test to restore env var values after test in test_pkg_config.py
• Fixing aliases element in kql query records
• Handle aliases as list or string in help

• Updates from PR review

f58e35c@Changing behavior so that vtlookupv3.py does not throw an exception when an item is not found. (#199)

• This is especially annoying when doing multiple look-ups and one item is not found.
• Minor changes to VTLookupV3.ipynb - to use get_object API from the vtlookupv3 library

d425c77@Process tree updates to handle more generic data (#178)

• Fixing bug in customjs and adding mde test files
• Update NOTICE.txt
• Fixing progress widget to use correct visibility settings.
• Changing user_config to be more defensive against bad settings.
• Error in query_store passing list instead of strings to MsticpyUserConfigError exception

• Adding a bit more text/warning to check the browser.

Co-authored-by: Pete Bryan [email protected]
ff87ebb@cred-scan suppression (#196)

Kql from Python Scripts, Mordor provider fixes

New Features

We've updated the Kql/Azure Sentinel data provider so that it no longer depends on
IPython/Jupyter functionality to work. This prevented using the provider from
Python scripts and applications. The new implementation uses Kqlmagic as
a standard import and no longer requires IPython magics to work.

08c7cf1@Ianhelle/kql as function 2021 08 04 (#190) Kqlmagic operations called as func vs. IPython magics
Now using Kqlmagic.kql() function in kql_driver, allowing it to be used from python code.

Fixes

The main fix here was to adapt to changes in the Mordor attack data repository
Mordor is now known as OTRF Security Datasets
(Although it will continue to be known as Mordor in MSTICPy for the time being).

• 17f9704@Ianhelle/modor url refix 2021 08 04 (#189)
• fabde53@fix img src for broken preview images (#191)
• 17b9227@Pebryan/7 16 21 sent api update (#187)* added incident collection
  • Updated pandas version to pandas>=1.1.5
  • Updated docstrings for Azure and Azure Sentinel APIs

Potentially breaking changes

• MSTICPY now requires pandas 1.1.5 as a minimum

Timeline duration, MDE Query upgrade

New Features

Timeline duration visualization #176

This new variant of the timeline control allows you to group by one or more columns and view timelines as start/end of activity related to those columns. The graphic below shows durations of communication between different source and destination IPs using a given protocol.

Like other timeline controls it's accessible as a pandas extension.

az_net_flows_df.mp_timeline.plot_duration(
    group_by=["SrcIP", "DestIP", "L7Protocol"]
)

You can also import and use the plot function directly:

from msticpy.nbtools.timeline_duration import display_timeline_duration

display_timeline_duration(
    host_logons,
    group_by="Account",
    ref_events=host_logons.sample(3),
    ref_col="TargetUserName",
);

Read about this in our Event Timeline documentation

Fixes

#184 MDEFix Update MDE connectors

The Microsoft Defender for Endpoint provider and queries have had a major update, correcting many schema issues (we were still using old column names in a number of cases

#185 from microsoft/ianhelle/init_notebook_changes-2021-07-14

• Removed nb_check.py update from azure_ml_tools.py
• Updated text about how to fix missing PyGObject
• Fixed MpConfigEdit widget to show checkbox for making backups prior to updating msticpyconfig
• Fix to remove casefold of OptionButtons value

8707868@Fixing Mordor and azure_ml_tools tests

• Fixed Mordor URL for mordor data provider

1ec1a00@bypassing GeoIP notebook test because of IPStack intermittent problems (#181)

Hotfix for notebooks #2 - Pivot exceptions and nbinit

Fixes

788b4ee@

• Added new context manager in MsticpyUserError to stop standard display of msticpy user exceptions.
  Note this doesn't suppress the exception, just stops exception being output to notebook even if exception is caught.
  
  Example:

from msticpy.common.exceptions import MsticpyUserError
with MsticpyUserError.no_display_exceptions():
   # do stuff that may cause exceptions to be display.

• Fixed typo in path for msticpyconfig.yaml was causing searches to always fail.
• Updated test_nbinit.py - fix one test. Using temp_path instead of tmpdir fixture.

Hotfix for Azure Sentinel notebooks

A few hotfixes for bugs

33e9fd4@Fixes to data_providers, data view, pivots and nbinit

• Replace list.pop with list.remove in data_providers.py - caused an error when using provider internal query_time control to set query start and end params.
• Add exception catching around Pivot instantiations in pivot_register_reader.py to prevent errors when loading Pivots with some
  providers missing configuration settings.
• Prevented push_notebook from being called before displaying control - this was causing an error when first displaying the DataViewer control
• Removed pandas option to return html schema in nbinit.py - Azure ML notebook dataviewer throws a react exception when trying to display DataFrame columns with complex types (e.g. dict or list vs. string/int/etc). Turning this off prevents the AML viewer being used by default.