Skip to content
This repository was archived by the owner on Dec 11, 2023. It is now read-only.

Commit f12c102

Browse files
johnwunderjohnwunder
committed
added training notebooks
1 parent 0ee87a2 commit f12c102

6 files changed

Lines changed: 450 additions & 0 deletions

File tree

‎README.md‎

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -13,6 +13,9 @@ Note: this repository is a work in progress. In the coming months we will be add
1313
2. Activate the environment: `source env/bin/activate`
1414
3. Install requirements into the virtual environment: `pip3 install -r requirements.txt`
1515

16+
## Training
17+
This repository also contains Jupyter notebooks and other material for ATT&CK training. The `trainings` directory has that content, which can be launched via Binder. The `binder` directory has requirements for that notebook, per the Binder documentation.
18+
1619
## Related MITRE Work
1720
#### CTI
1821
[Cyber Threat Intelligence repository](https://github.com/mitre/cti) of the ATT&CK catalog expressed in STIX 2.0 JSON.

‎binder/requirements.txt‎

Lines changed: 56 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
altair==3.2.0
2+
antlr4-python3-runtime==4.7.2
3+
appnope==0.1.0
4+
attackcti==0.2.7
5+
attrs==19.2.0
6+
backcall==0.1.0
7+
bleach==3.1.0
8+
certifi==2019.9.11
9+
chardet==3.0.4
10+
decorator==4.4.0
11+
defusedxml==0.6.0
12+
entrypoints==0.3
13+
idna==2.8
14+
ipykernel==5.1.2
15+
ipython==7.8.0
16+
ipython-genutils==0.2.0
17+
jedi==0.15.1
18+
Jinja2==2.10.1
19+
jsonschema==3.0.2
20+
jupyter-client==5.3.3
21+
jupyter-core==4.5.0
22+
MarkupSafe==1.1.1
23+
mistune==0.8.4
24+
nbconvert==5.6.0
25+
nbformat==4.4.0
26+
notebook==6.0.1
27+
numpy==1.17.2
28+
pandas==0.25.1
29+
pandocfilters==1.4.2
30+
parso==0.5.1
31+
pexpect==4.7.0
32+
pickleshare==0.7.5
33+
prometheus-client==0.7.1
34+
prompt-toolkit==2.0.9
35+
ptyprocess==0.6.0
36+
Pygments==2.4.2
37+
pyrsistent==0.15.4
38+
python-dateutil==2.8.0
39+
pytz==2019.2
40+
pyzmq==18.1.0
41+
requests==2.22.0
42+
Send2Trash==1.5.0
43+
simplejson==3.16.0
44+
six==1.12.0
45+
stix2==1.2.0
46+
stix2-patterns==1.1.0
47+
taxii2-client==0.5.0
48+
terminado==0.8.2
49+
testpath==0.4.2
50+
toolz==0.10.0
51+
tornado==6.0.3
52+
traitlets==4.3.2
53+
urllib3==1.25.6
54+
vega==2.6.0
55+
wcwidth==0.1.7
56+
webencodings==0.5.1

‎trainings/detection-training/.gitignore‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
data_sources.json
2+
**/*.ipynb_checkpoints/

‎trainings/detection-training/Data Sources.ipynb‎

Lines changed: 290 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,290 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# Data Source Investigation\n",
8+
"\n",
9+
"We'll use the [ATT&CK Python Client](https://github.com/hunters-forge/ATTACK-Python-Client) to manually examine the techniques, list the data sources, and build a heatmap out of our selected sources.\n",
10+
"\n",
11+
"If you're looking for less development or a more in-depth and finely-grained dive, check out:\n",
12+
"* [DeTTACK](https://github.com/rabobank-cdc/DeTTECT)\n",
13+
"* [AttackDatamap](https://github.com/olafhartong/ATTACKdatamap)"
14+
]
15+
},
16+
{
17+
"cell_type": "code",
18+
"execution_count": null,
19+
"metadata": {},
20+
"outputs": [],
21+
"source": [
22+
"# Import the packages we'll need\n",
23+
"\n",
24+
"# Some basic python and jupyter stuff\n",
25+
"from collections import defaultdict\n",
26+
"import json\n",
27+
"from IPython.display import FileLink, FileLinks\n",
28+
"\n",
29+
"# Visualization and data libraries\n",
30+
"import altair as alt\n",
31+
"import pandas as pd\n",
32+
"\n",
33+
"# ATT&CK Python Client, by @HuntersForge (https://github.com/hunters-forge/ATTACK-Python-Client)\n",
34+
"from attackcti import attack_client\n",
35+
"\n",
36+
"# Because this is in Jupyter notebooks we need to enable that renderer for the altair charts to work\n",
37+
"alt.renderers.enable('notebook')"
38+
]
39+
},
40+
{
41+
"cell_type": "markdown",
42+
"metadata": {},
43+
"source": [
44+
"## Get the ATT&CK Enterprise techniques using the client library"
45+
]
46+
},
47+
{
48+
"cell_type": "code",
49+
"execution_count": null,
50+
"metadata": {},
51+
"outputs": [],
52+
"source": [
53+
"client = attack_client()\n",
54+
"all_techniques = client.get_enterprise()['techniques'] # Note - this takes a few seconds to download and parse\n",
55+
"\n",
56+
"print(\"Got {} techniques\".format(len(all_techniques)))"
57+
]
58+
},
59+
{
60+
"cell_type": "markdown",
61+
"metadata": {},
62+
"source": [
63+
"## Analyze the data sources and build a chart to understand the most valuable ones\n",
64+
"\n",
65+
"We'll build up a dictionary that counts data sources by the number of techniques they can help detect."
66+
]
67+
},
68+
{
69+
"cell_type": "code",
70+
"execution_count": null,
71+
"metadata": {},
72+
"outputs": [],
73+
"source": [
74+
"# Collect unique data sources from the techniques\n",
75+
"techniques_by_source = defaultdict(lambda: {'count': 0, 'techniques': []})\n",
76+
"\n",
77+
"# Loop through all techniques, then through all data sources on that technique\n",
78+
"for technique in all_techniques:\n",
79+
" for ds in technique.get('x_mitre_data_sources', []):\n",
80+
" techniques_by_source[ds]['count'] += 1\n",
81+
" # External_ID in the first external reference is the T#### number\n",
82+
" techniques_by_source[ds]['techniques'].append(technique.external_references[0].external_id)"
83+
]
84+
},
85+
{
86+
"cell_type": "code",
87+
"execution_count": null,
88+
"metadata": {},
89+
"outputs": [],
90+
"source": [
91+
"# Create a pandas dataframe out of that result and show count column for the top 15\n",
92+
"df = pd.DataFrame.from_dict(techniques_by_source, orient='index', columns=['count', 'techniques']).rename_axis('source')\n",
93+
"top_15 = df.sort_values('count', ascending=False)[0:15]\n",
94+
"top_15[['count']]"
95+
]
96+
},
97+
{
98+
"cell_type": "markdown",
99+
"metadata": {},
100+
"source": [
101+
"## Show the chart in altair\n",
102+
"\n",
103+
"Altair can be used to easily turn pandas dataframes into visualizations. In this case, we just show a bar chart of the top 10 data sources (those that help you detect the most techniques)."
104+
]
105+
},
106+
{
107+
"cell_type": "code",
108+
"execution_count": null,
109+
"metadata": {},
110+
"outputs": [],
111+
"source": [
112+
"alt.Chart(df.reset_index()).mark_bar().encode(\n",
113+
" y=alt.Y(\n",
114+
" 'source',\n",
115+
" sort=alt.EncodingSortField(\n",
116+
" field=\"count\", # The field to use for the sort\n",
117+
" order=\"descending\" # The order to sort in\n",
118+
" )\n",
119+
" ),\n",
120+
" x='count'\n",
121+
")"
122+
]
123+
},
124+
{
125+
"cell_type": "markdown",
126+
"metadata": {},
127+
"source": [
128+
"## Building a Heatmap\n",
129+
"\n",
130+
"But what if you only have certain data already, and don't have flexibility to add different ones? That's the case for our exercise! How do you know what techniques you can detect based on that?\n",
131+
"\n",
132+
"We can generate a heatmap based on the data we created earlier. We can map the data sources we know we have into the data sources here."
133+
]
134+
},
135+
{
136+
"cell_type": "code",
137+
"execution_count": null,
138+
"metadata": {},
139+
"outputs": [],
140+
"source": [
141+
"# First, list the data sources alphabetically so we can figure out which ones we have\n",
142+
"\n",
143+
"df.reset_index().sort_values('source')[['source', 'count']].style.hide_index()"
144+
]
145+
},
146+
{
147+
"cell_type": "markdown",
148+
"metadata": {},
149+
"source": [
150+
"### List the data sources that are available\n",
151+
"\n",
152+
"In the list below, add the data sources that we have available in BRAWL. As a reminder, we have:\n",
153+
"* Sysmon\n",
154+
"* Windows event logs"
155+
]
156+
},
157+
{
158+
"cell_type": "code",
159+
"execution_count": null,
160+
"metadata": {},
161+
"outputs": [],
162+
"source": [
163+
"# Case sensitive!!!\n",
164+
"sources_we_have = [\n",
165+
" 'Windows event logs',\n",
166+
" 'Process monitoring'\n",
167+
"]"
168+
]
169+
},
170+
{
171+
"cell_type": "markdown",
172+
"metadata": {},
173+
"source": [
174+
"### Calculate out the techniques for which we have some detection capability"
175+
]
176+
},
177+
{
178+
"cell_type": "code",
179+
"execution_count": null,
180+
"metadata": {},
181+
"outputs": [],
182+
"source": [
183+
"techniques = defaultdict(lambda: 0)\n",
184+
"\n",
185+
"for ds in sources_we_have:\n",
186+
" for technique in techniques_by_source[ds]['techniques']:\n",
187+
" techniques[technique] += 1"
188+
]
189+
},
190+
{
191+
"cell_type": "code",
192+
"execution_count": null,
193+
"metadata": {},
194+
"outputs": [],
195+
"source": [
196+
"print(\"You can detect {} out of {} techniques\".format(len(techniques), len(all_techniques)))"
197+
]
198+
},
199+
{
200+
"cell_type": "markdown",
201+
"metadata": {},
202+
"source": [
203+
"### Generate the heatmap"
204+
]
205+
},
206+
{
207+
"cell_type": "code",
208+
"execution_count": null,
209+
"metadata": {},
210+
"outputs": [],
211+
"source": [
212+
"# Generate that heatmap!\n",
213+
"\n",
214+
"def technique_score(t):\n",
215+
" if t not in techniques:\n",
216+
" return 0.0\n",
217+
" elif techniques[t] > 1:\n",
218+
" return 1.0\n",
219+
" else: # count of sources == 1\n",
220+
" return 0.5\n",
221+
"\n",
222+
"heatmap = {\n",
223+
" 'version': \"2.1\",\n",
224+
" 'name': 'Detection Possibilities',\n",
225+
" 'domain': \"mitre-enterprise\",\n",
226+
" 'showTacticRowBackground': True,\n",
227+
" 'gradient': {\n",
228+
" 'colors': [\n",
229+
" '#ffffff',\n",
230+
" '#66b1ff'\n",
231+
" ],\n",
232+
" 'minValue': 0.0,\n",
233+
" 'maxValue': 1.0\n",
234+
" },\n",
235+
" 'techniques': [{'techniqueID': t, 'score': technique_score(t)} for t in techniques]\n",
236+
"}"
237+
]
238+
},
239+
{
240+
"cell_type": "code",
241+
"execution_count": null,
242+
"metadata": {},
243+
"outputs": [],
244+
"source": [
245+
"# Write as a JSON file and show a download link\n",
246+
"with open('data_sources.json', 'w') as f:\n",
247+
" f.write(json.dumps(heatmap))\n",
248+
" \n",
249+
"FileLink('data_sources.json')"
250+
]
251+
},
252+
{
253+
"cell_type": "code",
254+
"execution_count": null,
255+
"metadata": {},
256+
"outputs": [],
257+
"source": [
258+
"heatmap"
259+
]
260+
},
261+
{
262+
"cell_type": "code",
263+
"execution_count": null,
264+
"metadata": {},
265+
"outputs": [],
266+
"source": []
267+
}
268+
],
269+
"metadata": {
270+
"kernelspec": {
271+
"display_name": "detection-training",
272+
"language": "python",
273+
"name": "detection-training"
274+
},
275+
"language_info": {
276+
"codemirror_mode": {
277+
"name": "ipython",
278+
"version": 3
279+
},
280+
"file_extension": ".py",
281+
"mimetype": "text/x-python",
282+
"name": "python",
283+
"nbconvert_exporter": "python",
284+
"pygments_lexer": "ipython3",
285+
"version": "3.7.1"
286+
}
287+
},
288+
"nbformat": 4,
289+
"nbformat_minor": 2
290+
}

0 commit comments

Comments
 (0)