Skip to content

Commit 365478d

Browse files
QmandoQmando
authored
Merge pull request YelpArchive#2474 from JeffAshton/kibana-discover-link
Adding support for generating Kibana Discover app link
2 parents 4935ac4 + 1392521 commit 365478d

9 files changed

Lines changed: 1195 additions & 0 deletions

File tree

‎docs/source/ruletypes.rst‎

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -58,6 +58,20 @@ Rule Configuration Cheat Sheet
5858
+--------------------------------------------------------------+ |
5959
| ``kibana4_end_timedelta`` (time, default: 10 min) | |
6060
+--------------------------------------------------------------+ |
61+
| ``generate_kibana_discover_url`` (boolean, default False) | |
62+
+--------------------------------------------------------------+ |
63+
| ``kibana_discover_app_url`` (string, no default) | |
64+
+--------------------------------------------------------------+ |
65+
| ``kibana_discover_version`` (string, no default) | |
66+
+--------------------------------------------------------------+ |
67+
| ``kibana_discover_index_pattern_id`` (string, no default) | |
68+
+--------------------------------------------------------------+ |
69+
| ``kibana_discover_columns`` (list of strs, default _source) | |
70+
+--------------------------------------------------------------+ |
71+
| ``kibana_discover_from_timedelta`` (time, default: 10 min) | |
72+
+--------------------------------------------------------------+ |
73+
| ``kibana_discover_to_timedelta`` (time, default: 10 min) | |
74+
+--------------------------------------------------------------+ |
6175
| ``use_local_time`` (boolean, default True) | |
6276
+--------------------------------------------------------------+ |
6377
| ``realert`` (time, default: 1 min) | |
@@ -510,6 +524,85 @@ This value is added in back of the event. For example,
510524

511525
``kibana4_end_timedelta: minutes: 2``
512526

527+
generate_kibana_discover_url
528+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
529+
530+
``generate_kibana_discover_url``: Enables the generation of the ``kibana_discover_url`` variable for the Kibana Discover application.
531+
This setting requires the following settings are also configured:
532+
533+
- ``kibana_discover_app_url``
534+
- ``kibana_discover_version``
535+
- ``kibana_discover_index_pattern_id``
536+
537+
``generate_kibana_discover_url: true``
538+
539+
kibana_discover_app_url
540+
^^^^^^^^^^^^^^^^^^^^^^^
541+
542+
``kibana_discover_app_url``: The url of the Kibana Discover application used to generate the ``kibana_discover_url`` variable.
543+
This value can use `$VAR` and `${VAR}` references to expand environment variables.
544+
545+
``kibana_discover_app_url: http://kibana:5601/#/discover``
546+
547+
kibana_discover_version
548+
^^^^^^^^^^^^^^^^^^^^^^^
549+
550+
``kibana_discover_version``: Specifies the version of the Kibana Discover application.
551+
552+
The currently supported versions of Kibana Discover are:
553+
554+
- `5.6`
555+
- `6.0`, `6.1`, `6.2`, `6.3`, `6.4`, `6.5`, `6.6`, `6.7`, `6.8`
556+
- `7.0`, `7.1`, `7.2`, `7.3`
557+
558+
``kibana_discover_version: '7.3'``
559+
560+
kibana_discover_index_pattern_id
561+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
562+
563+
``kibana_discover_index_pattern_id``: The id of the index pattern to link to in the Kibana Discover application.
564+
These ids are usually generated and can be found in url of the index pattern management page, or by exporting its saved object.
565+
566+
Example export of an index pattern's saved object:
567+
568+
.. code-block:: text
569+
570+
[
571+
{
572+
"_id": "4e97d188-8a45-4418-8a37-07ed69b4d34c",
573+
"_type": "index-pattern",
574+
"_source": { ... }
575+
}
576+
]
577+
578+
You can modify an index pattern's id by exporting the saved object, modifying the ``_id`` field, and re-importing.
579+
580+
``kibana_discover_index_pattern_id: 4e97d188-8a45-4418-8a37-07ed69b4d34c``
581+
582+
kibana_discover_columns
583+
^^^^^^^^^^^^^^^^^^^^^^^
584+
585+
``kibana_discover_columns``: The columns to display in the generated Kibana Discover application link.
586+
Defaults to the ``_source`` column.
587+
588+
``kibana_discover_columns: [ timestamp, message ]``
589+
590+
kibana_discover_from_timedelta
591+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
592+
593+
``kibana_discover_from_timedelta``: The offset to the `from` time of the Kibana Discover link's time range.
594+
The `from` time is calculated by subtracting this timedelta from the event time. Defaults to 10 minutes.
595+
596+
``kibana_discover_from_timedelta: minutes: 2``
597+
598+
kibana_discover_to_timedelta
599+
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
600+
601+
``kibana_discover_to_timedelta``: The offset to the `to` time of the Kibana Discover link's time range.
602+
The `to` time is calculated by adding this timedelta to the event time. Defaults to 10 minutes.
603+
604+
``kibana_discover_to_timedelta: minutes: 2``
605+
513606
use_local_time
514607
^^^^^^^^^^^^^^
515608

‎elastalert/elastalert.py‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,7 @@
2727
from elasticsearch.exceptions import TransportError
2828

2929
from . import kibana
30+
from .kibana_discover import generate_kibana_discover_url
3031
from .alerts import DebugAlerter
3132
from .config import load_conf
3233
from .enhancements import DropMatchException
@@ -1503,6 +1504,11 @@ def send_alert(self, matches, rule, alert_time=None, retried=False):
15031504
if kb_link:
15041505
matches[0]['kibana_link'] = kb_link
15051506

1507+
if rule.get('generate_kibana_discover_url'):
1508+
kb_link = generate_kibana_discover_url(rule, matches[0])
1509+
if kb_link:
1510+
matches[0]['kibana_discover_url'] = kb_link
1511+
15061512
# Enhancements were already run at match time if
15071513
# run_enhancements_first is set or
15081514
# retried==True, which means this is a retry of a failed alert

‎elastalert/kibana_discover.py‎

Lines changed: 192 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,192 @@
1+
# -*- coding: utf-8 -*-
2+
# flake8: noqa
3+
import datetime
4+
import logging
5+
import json
6+
import os.path
7+
import prison
8+
import urllib.parse
9+
10+
from .util import EAException
11+
from .util import lookup_es_key
12+
from .util import ts_add
13+
14+
kibana_default_timedelta = datetime.timedelta(minutes=10)
15+
16+
kibana5_kibana6_versions = frozenset(['5.6', '6.0', '6.1', '6.2', '6.3', '6.4', '6.5', '6.6', '6.7', '6.8'])
17+
kibana7_versions = frozenset(['7.0', '7.1', '7.2', '7.3'])
18+
19+
def generate_kibana_discover_url(rule, match):
20+
''' Creates a link for a kibana discover app. '''
21+
22+
discover_app_url = rule.get('kibana_discover_app_url')
23+
if not discover_app_url:
24+
logging.warning(
25+
'Missing kibana_discover_app_url for rule %s' % (
26+
rule.get('name', '<MISSING NAME>')
27+
)
28+
)
29+
return None
30+
31+
kibana_version = rule.get('kibana_discover_version')
32+
if not kibana_version:
33+
logging.warning(
34+
'Missing kibana_discover_version for rule %s' % (
35+
rule.get('name', '<MISSING NAME>')
36+
)
37+
)
38+
return None
39+
40+
index = rule.get('kibana_discover_index_pattern_id')
41+
if not index:
42+
logging.warning(
43+
'Missing kibana_discover_index_pattern_id for rule %s' % (
44+
rule.get('name', '<MISSING NAME>')
45+
)
46+
)
47+
return None
48+
49+
columns = rule.get('kibana_discover_columns', ['_source'])
50+
filters = rule.get('filter', [])
51+
52+
if 'query_key' in rule:
53+
query_keys = rule.get('compound_query_key', [rule['query_key']])
54+
else:
55+
query_keys = []
56+
57+
timestamp = lookup_es_key(match, rule['timestamp_field'])
58+
timeframe = rule.get('timeframe', kibana_default_timedelta)
59+
from_timedelta = rule.get('kibana_discover_from_timedelta', timeframe)
60+
from_time = ts_add(timestamp, -from_timedelta)
61+
to_timedelta = rule.get('kibana_discover_to_timedelta', timeframe)
62+
to_time = ts_add(timestamp, to_timedelta)
63+
64+
if kibana_version in kibana5_kibana6_versions:
65+
globalState = kibana6_disover_global_state(from_time, to_time)
66+
appState = kibana_discover_app_state(index, columns, filters, query_keys, match)
67+
68+
elif kibana_version in kibana7_versions:
69+
globalState = kibana7_disover_global_state(from_time, to_time)
70+
appState = kibana_discover_app_state(index, columns, filters, query_keys, match)
71+
72+
else:
73+
logging.warning(
74+
'Unknown kibana discover application version %s for rule %s' % (
75+
kibana_version,
76+
rule.get('name', '<MISSING NAME>')
77+
)
78+
)
79+
return None
80+
81+
return "%s?_g=%s&_a=%s" % (
82+
os.path.expandvars(discover_app_url),
83+
urllib.parse.quote(globalState),
84+
urllib.parse.quote(appState)
85+
)
86+
87+
88+
def kibana6_disover_global_state(from_time, to_time):
89+
return prison.dumps( {
90+
'refreshInterval': {
91+
'pause': True,
92+
'value': 0
93+
},
94+
'time': {
95+
'from': from_time,
96+
'mode': 'absolute',
97+
'to': to_time
98+
}
99+
} )
100+
101+
102+
def kibana7_disover_global_state(from_time, to_time):
103+
return prison.dumps( {
104+
'filters': [],
105+
'refreshInterval': {
106+
'pause': True,
107+
'value': 0
108+
},
109+
'time': {
110+
'from': from_time,
111+
'to': to_time
112+
}
113+
} )
114+
115+
116+
def kibana_discover_app_state(index, columns, filters, query_keys, match):
117+
app_filters = []
118+
119+
if filters:
120+
bool_filter = { 'must': filters }
121+
app_filters.append( {
122+
'$state': {
123+
'store': 'appState'
124+
},
125+
'bool': bool_filter,
126+
'meta': {
127+
'alias': 'filter',
128+
'disabled': False,
129+
'index': index,
130+
'key': 'bool',
131+
'negate': False,
132+
'type': 'custom',
133+
'value': json.dumps(bool_filter, separators=(',', ':'))
134+
},
135+
} )
136+
137+
for query_key in query_keys:
138+
query_value = lookup_es_key(match, query_key)
139+
140+
if query_value is None:
141+
app_filters.append( {
142+
'$state': {
143+
'store': 'appState'
144+
},
145+
'exists': {
146+
'field': query_key
147+
},
148+
'meta': {
149+
'alias': None,
150+
'disabled': False,
151+
'index': index,
152+
'key': query_key,
153+
'negate': True,
154+
'type': 'exists',
155+
'value': 'exists'
156+
}
157+
} )
158+
159+
else:
160+
app_filters.append( {
161+
'$state': {
162+
'store': 'appState'
163+
},
164+
'meta': {
165+
'alias': None,
166+
'disabled': False,
167+
'index': index,
168+
'key': query_key,
169+
'negate': False,
170+
'params': {
171+
'query': query_value,
172+
'type': 'phrase'
173+
},
174+
'type': 'phrase',
175+
'value': str(query_value)
176+
},
177+
'query': {
178+
'match': {
179+
query_key: {
180+
'query': query_value,
181+
'type': 'phrase'
182+
}
183+
}
184+
}
185+
} )
186+
187+
return prison.dumps( {
188+
'columns': columns,
189+
'filters': app_filters,
190+
'index': index,
191+
'interval': 'auto'
192+
} )

‎elastalert/loaders.py‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -257,6 +257,10 @@ def load_options(self, rule, conf, filename, args=None):
257257
rule['kibana4_start_timedelta'] = datetime.timedelta(**rule['kibana4_start_timedelta'])
258258
if 'kibana4_end_timedelta' in rule:
259259
rule['kibana4_end_timedelta'] = datetime.timedelta(**rule['kibana4_end_timedelta'])
260+
if 'kibana_discover_from_timedelta' in rule:
261+
rule['kibana_discover_from_timedelta'] = datetime.timedelta(**rule['kibana_discover_from_timedelta'])
262+
if 'kibana_discover_to_timedelta' in rule:
263+
rule['kibana_discover_to_timedelta'] = datetime.timedelta(**rule['kibana_discover_to_timedelta'])
260264
except (KeyError, TypeError) as e:
261265
raise EAException('Invalid time format used: %s' % e)
262266

‎elastalert/schema.yaml‎

Lines changed: 20 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,17 @@ definitions:
1111
type: [string, array]
1212
items: {type: [string, array]}
1313

14+
timedelta: &timedelta
15+
type: object
16+
additionalProperties: false
17+
properties:
18+
days: {type: number}
19+
weeks: {type: number}
20+
hours: {type: number}
21+
minutes: {type: number}
22+
seconds: {type: number}
23+
milliseconds: {type: number}
24+
1425
timeFrame: &timeframe
1526
type: object
1627
additionalProperties: false
@@ -203,6 +214,15 @@ properties:
203214
replace_dots_in_field_names: {type: boolean}
204215
scan_entire_timeframe: {type: boolean}
205216

217+
### Kibana Discover App Link
218+
generate_kibana_discover_url: {type: boolean}
219+
kibana_discover_app_url: {type: string, format: uri}
220+
kibana_discover_version: {type: string, enum: ['7.3', '7.2', '7.1', '7.0', '6.8', '6.7', '6.6', '6.5', '6.4', '6.3', '6.2', '6.1', '6.0', '5.6']}
221+
kibana_discover_index_pattern_id: {type: string, minLength: 1}
222+
kibana_discover_columns: {type: array, items: {type: string, minLength: 1}, minItems: 1}
223+
kibana_discover_from_timedelta: *timedelta
224+
kibana_discover_to_timedelta: *timedelta
225+
206226
# Alert Content
207227
alert_text: {type: string} # Python format string
208228
alert_text_args: {type: array, items: {type: string}}

‎requirements.txt‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -11,6 +11,7 @@ exotel>=0.1.3
1111
jira>=1.0.10,<1.0.15
1212
jsonschema>=3.0.2
1313
mock>=2.0.0
14+
prison>=0.1.2
1415
py-zabbix==1.1.3
1516
PyStaticConfiguration>=0.10.3
1617
python-dateutil>=2.6.0,<2.7.0

‎setup.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -39,6 +39,7 @@
3939
'jira>=1.0.10,<1.0.15',
4040
'jsonschema>=3.0.2',
4141
'mock>=2.0.0',
42+
'prison>=0.1.2',
4243
'PyStaticConfiguration>=0.10.3',
4344
'python-dateutil>=2.6.0,<2.7.0',
4445
'PyYAML>=3.12',

0 commit comments

Comments
 (0)