Skip to content

Commit dd9aa41

Browse files
GenEarsGenEars
committed
data-collection/README.md improvement
Signed-off-by: Lionel TAILHARDAT <lionel.tailhardat@orange.com>
1 parent 7057163 commit dd9aa41

2 files changed

Lines changed: 84 additions & 60 deletions

File tree

‎data-collection/README.md‎

Lines changed: 62 additions & 60 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
# DynaGraph / data-collection
22

3-
A set of light WebSocket servers for sending graph data to the DynaGraph app.
3+
A set of light WebSocket (WS) servers for sending graph data to the DynaGraph app.
4+
Unless using modified scripts, the WS is served at `ws://127.0.0.1:5678`.
45

56
## Requirements
67

@@ -17,86 +18,87 @@ pip install -r ./requirements.txt
1718

1819
## Usage
1920

20-
Call individual scripts from CLI depending on needs:
21-
* `ws_server.py`: a WS server that sends json messages at random intervals,
22-
* `ws_server_stdin.py`: a WS server that gets data from stdin coming from [TShark](https://www.wireshark.org/docs/man-pages/tshark.html) and sends them as json,
23-
```shell
24-
# Capture net traffic, export data as CSV, transform it to json, send it
25-
# Press Ctrl-C to stop the process
26-
tshark -T fields -l -E separator=, -E quote=d \
27-
-e _ws.col.No. -e _ws.col.Time \
28-
-e _ws.col.Source -e _ws.col.Destination \
29-
-e _ws.col.Protocol -e _ws.col.Length -e _ws.col.Info | \
30-
python ws_server_stdin.py
31-
```
32-
33-
In order to be more specific on TShark capture behavior, a *capture filter* can be set with the `-f` commutator.
34-
See [CaptureFilters](https://gitlab.com/wireshark/wireshark/-/wikis/CaptureFilters) documentation for syntax insights and example below:
35-
36-
```shell
37-
# Capture net traffic, export data as CSV, transform it to json, send it
38-
# Press Ctrl-C to stop the process
39-
tshark -T fields -l -E separator=, -E quote=d \
40-
-e _ws.col.No. -e _ws.col.Time \
41-
-e _ws.col.Source -e _ws.col.Destination \
42-
-e _ws.col.Protocol -e _ws.col.Length -e _ws.col.Info \
43-
-f "ip" | \
44-
python ws_server_stdin.py
45-
```
46-
One can also wish to send traffic data from a pcapng file. Thereon, the `tshark` will use the `-r` commutator:
47-
```shell
48-
# Capture net traffic, export data as CSV, transform it to json, send it
49-
# Press Ctrl-C to stop the process
50-
tshark -T fields -l -E separator=, -E quote=d \
51-
-e _ws.col.No. -e _ws.col.Time \
52-
-e _ws.col.Source -e _ws.col.Destination \
53-
-e _ws.col.Protocol -e _ws.col.Length -e _ws.col.Info \
54-
-r ../samples/open_github_wireshark_2021-09-04_16-52-21.pcapng | \
55-
python ws_server_stdin.py
56-
```
57-
... Firefox and Chrome can export TLS keys **for later decryption** by wireshark or TShark (i.e. no live decryption).
58-
See "[Decrypting SSL/TLS traffic with Wireshark](https://resources.infosecinstitute.com/topic/decrypting-ssl-tls-traffic-with-wireshark/)" and "[TShark TLS Encrypted](https://tshark.dev/export/export_tls/)" for operational details.
59-
As a quick hint, call Firefox, after the capture setup, with `SSLKEYLOGFILE=~/.ssl-key.log firefox` for keys exports (*don't forget to delete the file for security purposes*).
60-
61-
* `ws_server_har.py`: a WS server that parse a HAR file for 3 facets (Browser=>URL, URL=>Server, Browser=>Server) and sends data as json,
21+
Call individual scripts from CLI depending on needs ...
22+
23+
### Message generator at random intervals
24+
25+
=> `ws_server.py`: a WS server that sends json messages at random intervals,
26+
27+
### Forward TSHARK messages
28+
29+
=> `ws_server_stdin.py`: a WS server that gets data from stdin coming from [TShark](https://www.wireshark.org/docs/man-pages/tshark.html) and sends them as json,
6230

6331
```shell
64-
tshark -Q -a duration:30 \
65-
-f "tcp port http or https" \
66-
-w /tmp/http_https_netsniff.pcapng
67-
& \
68-
SSLKEYLOGFILE=/tmp/ssl-key.log firefox
32+
# Capture net traffic, export data as CSV, transform it to json, send it
33+
# Press Ctrl-C to stop the process
34+
tshark -T fields -l -E separator=, -E quote=d \
35+
-e _ws.col.No. -e _ws.col.Time \
36+
-e _ws.col.Source -e _ws.col.Destination \
37+
-e _ws.col.Protocol -e _ws.col.Length -e _ws.col.Info | \
38+
python ws_server_stdin.py
6939
```
7040

41+
In order to be more specific on TShark capture behavior, a *capture filter* can be set with the `-f` commutator.
42+
See [CaptureFilters](https://gitlab.com/wireshark/wireshark/-/wikis/CaptureFilters) documentation for syntax insights and example below:
43+
7144
```shell
7245
# Capture net traffic, export data as CSV, transform it to json, send it
7346
# Press Ctrl-C to stop the process
7447
tshark -T fields -l -E separator=, -E quote=d \
7548
-e _ws.col.No. -e _ws.col.Time \
7649
-e _ws.col.Source -e _ws.col.Destination \
7750
-e _ws.col.Protocol -e _ws.col.Length -e _ws.col.Info \
78-
-o tls.keylog_file:/tmp/.ssl-key.log \
79-
-r /tmp/http_https_netsniff.pcapng
80-
81-
| \
51+
-f "ip" | \
8252
python ws_server_stdin.py
8353
```
84-
54+
One can also wish to send traffic data from a pcapng file. Thereon, the `tshark` will use the `-r` commutator:
8555
```shell
86-
tshark -Q --export-objects http,/tmp/obj -r /tmp/myfile.pcapng \
87-
-o tls.keylog_file:$SSLKEYLOGFILE
56+
# Capture net traffic, export data as CSV, transform it to json, send it
57+
# Press Ctrl-C to stop the process
58+
tshark -T fields -l -E separator=, -E quote=d \
59+
-e _ws.col.No. -e _ws.col.Time \
60+
-e _ws.col.Source -e _ws.col.Destination \
61+
-e _ws.col.Protocol -e _ws.col.Length -e _ws.col.Info \
62+
-r ../samples/open_github_wireshark_2021-09-04_16-52-21.pcapng | \
63+
python ws_server_stdin.py
8864
```
8965

90-
Unless using modified scripts, the WS is served at `ws://127.0.0.1:5678`.
91-
92-
## HTTP Archive files (HAR)
66+
### Format and forward HAR file content
9367

94-
As an alternative to network traffic sniffing (e.g. tshark, tcpflow, tcpdump, etc.), because of TLS encryption or any other reason, browsing activity can be analyzed through HAR dumps.
68+
As an alternative to network traffic sniffing (e.g. tshark, tcpflow, tcpdump, etc.), because of TLS encryption or any other reason, browsing activity can be analyzed through HTTP Archive (HAR) files.
9569

9670
The [Haralyzer](https://github.com/haralyzer/haralyzer) package allows for HAR file parsing in Python.
9771

72+
=> `ws_server_har.py`: a WS server that parse a HAR file for 3 facets (Browser=>URL, URL=>Server, Browser=>Server) and sends data as json,
73+
74+
### A posteriori decryption of TSHARK dumps
75+
76+
Firefox and Chrome can export TLS keys **for later decryption** by wireshark or TShark (i.e. no live decryption).
77+
See "[Decrypting SSL/TLS traffic with Wireshark](https://resources.infosecinstitute.com/topic/decrypting-ssl-tls-traffic-with-wireshark/)" and "[TShark TLS Encrypted](https://tshark.dev/export/export_tls/)" for operational details.
78+
As a quick hint, call Firefox, after the capture setup, with `SSLKEYLOGFILE=~/.ssl-key.log firefox` for keys exports (*don't forget to delete the file for security purposes*).
79+
80+
```shell
81+
# Start Firefox instance with SSL key export
82+
# and a 30 seconds capture of http/https traffic to local file in background with TShark
83+
tshark -Q -a duration:30 \
84+
-f "tcp port http or https" \
85+
-w /tmp/http_https_netsniff.pcapng & \
86+
SSLKEYLOGFILE=/tmp/ssl-key.log firefox
87+
```
88+
89+
```shell
90+
# Read and decrypt local traffic dump file,
91+
# export data as CSV, transform it to json, send it
92+
tshark -T fields -l -E separator=, -E quote=d \
93+
-e _ws.col.No. -e _ws.col.Time \
94+
-e _ws.col.Source -e _ws.col.Destination \
95+
-e _ws.col.Protocol -e _ws.col.Length -e _ws.col.Info \
96+
-o tls.keylog_file:/tmp/.ssl-key.log \
97+
-r /tmp/http_https_netsniff.pcapng \
98+
| python ws_server_stdin.py
99+
```
98100

99101
## Debug
100102

101-
If using the Firefox web browser, the [WebSocket Waesel](https://addons.mozilla.org/fr/firefox/addon/websocket-weasel/) add-on can be of great help.
103+
Using the Firefox web browser, the [WebSocket Waesel](https://addons.mozilla.org/fr/firefox/addon/websocket-weasel/) add-on can be of great help.
102104
After its installation, open the add-on and connect to the WS (e.g. `ws://127.0.0.1:5678`).

‎data-collection/start-tshark-basic.sh‎

Lines changed: 22 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
#!/usr/bin/env bash
2+
3+
#
4+
# Software Name : dynagraph
5+
# Version: 1.0.0
6+
# SPDX-FileCopyrightText: Copyright (c) 2021-2022 Orange
7+
# SPDX-License-Identifier: BSD-4-Clause
8+
#
9+
# This software is distributed under the BSD-4-Clause, the text of which is available at https://spdx.org/licenses/ or see the "LICENSE.txt" file for more details.
10+
#
11+
# Author: Lionel TAILHARDAT
12+
# Software description: The DynaGraph framework: a system combining classical traces dumping tools (i.e. the tshark tool and Firefox's Network Monitor component) and a ReactJS web app for live 3D graph rendering of streamed graph data derived from traces.
13+
#
14+
15+
# Capture net traffic, export data as CSV, transform it to json, send it
16+
# Press Ctrl-C to stop the process
17+
tshark -T fields -l -E separator=, -E quote=d \
18+
-e _ws.col.No. -e _ws.col.Time \
19+
-e _ws.col.Source -e _ws.col.Destination \
20+
-e _ws.col.Protocol -e _ws.col.Length -e _ws.col.Info \
21+
-f "ip" | \
22+
python ws_server_stdin.py

0 commit comments

Comments
 (0)