So this video (https://www.youtube.com/watch?v=WDERBaYL05U) provides an overview of the DUNE (detection of unknown novel events) project that was being talked about on LinkedIn after being shown at the Boston area meetups. It was originally presented at RSA 2024. In summary, the project consists of these mark three versions of the notebooks used for applying machine learning to hunting detection resistant threats. The mark three notebooks have increased performance and additional integrity checks to ensure the scored data is matched precisely to the raw event data.
Jupyer
- Silhouettes is a stand-alone notebook for calculating the optimal value of k, prior to running k-means
- k-means-mark3- is a notebook for running k-means on a dataframe using the silhouette score as an input
- pyod-mark-3 is a notebook for running an ensemble model on a dataframe using pyod.
- viewer-mark-3 is a tool for aggregating, sifting and querying the output of the models in a series fo interactive widgets.
Cloudtrail
The dashboards folder contains anomaly detection dashboards for Elastic and Splunk that do not require Jupiter. The Cloudtrail, Flow Logs, and Kubernetes folders contain notebooks for working with those data types. There are some new notebooks in the Cloudtrail folder:
- the two s3-ingestor notebooks contain code to read CloudTrail logs directly from an S3 bucket into a dataframe, including unzipping the files. They assume the notebook is running on a Jupiter instance that can reach the buckets using the s3 client via either an IAM user or an AssumeRole.
- aws-ip-discovery is a notebook that will enumerate IP addressed associated with an account, for cases when the question of wether an IP address is associated with your account or not, in order to ascertain if the source is someone else’s account.
Opendr 