Skip to content
This repository has been archived by the owner. It is now read-only.

Commit c7430b3

Browse files
basepibasepi
authored
Merge pull request #20 from HubbleStack/develop
Merge to master (prep for v2016.10.1)
2 parents 1515416 + 32ea4e6 commit c7430b3

File tree

4 files changed

+116
-64
lines changed

4 files changed

+116
-64
lines changed

‎FORMULA‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
name: hubblestack_nebula
22
os: RedHat, CentOS, Debian, Ubuntu
33
os_family: RedHat, Debian
4-
version: 2016.7.1
4+
version: 2016.9.1
55
release: 1
66
summary: HubbleStack Nebula
77
description: HubbleStack Nebula

‎README.rst‎

Lines changed: 20 additions & 52 deletions
Original file line numberDiff line numberDiff line change
@@ -53,8 +53,8 @@ repo for updates and bugfixes!)
5353

5454
.. code-block:: shell
5555
56-
wget https://spm.hubblestack.io/2016.7.1/hubblestack_nebula-2016.7.1-1.spm
57-
spm local install hubblestack_nebula-2016.7.1-1.spm
56+
wget https://spm.hubblestack.io/nebula/hubblestack_nebula-2016.9.1-1.spm
57+
spm local install hubblestack_nebula-2016.9.1-1.spm
5858
5959
You should now be able to sync the new modules to your minion(s) using the
6060
``sync_modules`` Salt utility:
@@ -63,19 +63,6 @@ You should now be able to sync the new modules to your minion(s) using the
6363
6464
salt \* saltutil.sync_modules
6565
66-
Copy the ``hubblestack_nebula.sls.orig`` into your Salt pillar, dropping the
67-
``.orig`` extension and target it to selected minions.
68-
69-
.. code-block:: shell
70-
71-
base:
72-
'*':
73-
- hubblestack_nebula
74-
75-
.. code-block:: shell
76-
77-
salt \* saltutil.refresh_pillar
78-
7966
Once these modules are synced you are ready to schedule HubbleStack Nebula
8067
queries.
8168

@@ -100,18 +87,6 @@ it to the minions.
10087
10188
salt \* saltutil.sync_modules
10289
103-
Target the ``hubblestack_nebula.sls`` to selected minions.
104-
105-
.. code-block:: shell
106-
107-
base:
108-
'*':
109-
- hubblestack_nebula
110-
111-
.. code-block:: shell
112-
113-
salt \* saltutil.refresh_pillar
114-
11590
Once these modules are synced you are ready to schedule HubbleStack Nebula
11691
queries.
11792

@@ -120,42 +95,35 @@ queries.
12095
Usage
12196
=====
12297

123-
This module also requires pillar data to function. The default pillar key for
124-
this data is ``nebula_osquery``. The queries themselves should be grouped
125-
under one or more group identifiers. Usually, these identifiers will be
126-
frequencies, such as ``fifteen_min`` or ``hourly`` or ``daily``. The module
127-
targets the queries using these identifiers.
128-
129-
Your pillar data might look like this:
98+
These queries have been designed to give detailed insight into system activity.
13099

131-
**hubble_nebula.sls**
100+
**hubblestack_nebula/hubblestack_nebula_queries.yaml**
132101

133102
.. code-block:: yaml
134103
135-
nebula_osquery:
136-
fifteen_min:
137-
- query_name: running_procs
138-
query: select p.name as process, p.pid as process_id, p.cmdline, p.cwd, p.on_disk, p.resident_size as mem_used, p.parent, g.groupname, u.username as user, p.path, h.md5, h.sha1, h.sha256 from processes as p left join users as u on p.uid=u.uid left join groups as g on p.gid=g.gid left join hash as h on p.path=h.path;
139-
- query_name: established_outbound
140-
query: select t.iso_8601 as _time, pos.family, h.*, ltrim(pos.local_address, ':f') as src, pos.local_port as src_port, pos.remote_port as dest_port, ltrim(remote_address, ':f') as dest, name, p.path as file_path, cmdline, pos.protocol, lp.protocol from process_open_sockets as pos join processes as p on p.pid=pos.pid left join time as t LEFT JOIN listening_ports as lp on lp.port=pos.local_port AND lp.protocol=pos.protocol LEFT JOIN hash as h on h.path=p.path where not remote_address='' and not remote_address='::' and not remote_address='0.0.0.0' and not remote_address='127.0.0.1' and port is NULL;
141-
- query_name: listening_procs
142-
query: select t.iso_8601 as _time, h.md5 as md5, p.pid, name, ltrim(address, ':f') as address, port, p.path as file_path, cmdline, root, parent from listening_ports as lp JOIN processes as p on lp.pid=p.pid left JOIN time as t JOIN hash as h on h.path=p.path WHERE not address='127.0.0.1';
143-
- query_name: suid_binaries
144-
query: select sb.*, t.iso_8601 as _time from suid_bin as sb join time as t;
145-
hour:
146-
- query_name: crontab
147-
query: select c.*,t.iso_8601 as _time from crontab as c join time as t;
148-
day:
149-
- query_name: rpm_packages
150-
query: select rpm.*, t.iso_8601 from rpm_packages as rpm join time as t;
104+
fifteen_min:
105+
- query_name: running_procs
106+
query: SELECT p.name AS process, p.pid AS process_id, p.cmdline, p.cwd, p.on_disk, p.resident_size AS mem_used, p.parent, g.groupname, u.username AS user, p.path, h.md5, h.sha1, h.sha256 FROM processes AS p LEFT JOIN users AS u ON p.uid=u.uid LEFT JOIN groups AS g ON p.gid=g.gid LEFT JOIN hash AS h ON p.path=h.path;
107+
- query_name: established_outbound
108+
query: SELECT t.iso_8601 AS _time, pos.family, h.*, ltrim(pos.local_address, ':f') AS src, pos.local_port AS src_port, pos.remote_port AS dest_port, ltrim(remote_address, ':f') AS dest, name, p.path AS file_path, cmdline, pos.protocol, lp.protocol FROM process_open_sockets AS pos JOIN processes AS p ON p.pid=pos.pid LEFT JOIN time AS t LEFT JOIN (SELECT * FROM listening_ports) AS lp ON lp.port=pos.local_port AND lp.protocol=pos.protocol LEFT JOIN hash AS h ON h.path=p.path WHERE NOT remote_address='' AND NOT remote_address='::' AND NOT remote_address='0.0.0.0' AND NOT remote_address='127.0.0.1' AND port is NULL;
109+
- query_name: listening_procs
110+
query: SELECT t.iso_8601 AS _time, h.md5 AS md5, p.pid, name, ltrim(address, ':f') AS address, port, p.path AS file_path, cmdline, root, parent FROM listening_ports AS lp LEFT JOIN processes AS p ON lp.pid=p.pid LEFT JOIN time AS t LEFT JOIN hash AS h ON h.path=p.path WHERE NOT address='127.0.0.1';
111+
- query_name: suid_binaries
112+
query: SELECT sb.*, t.iso_8601 AS _time FROM suid_bin AS sb JOIN time AS t;
113+
hour:
114+
- query_name: crontab
115+
query: SELECT c.*,t.iso_8601 AS _time FROM crontab AS c JOIN time AS t;
116+
day:
117+
- query_name: rpm_packages
118+
query: SELECT rpm.name, rpm.version, rpm.release, rpm.source AS package_source, rpm.size, rpm.sha1, rpm.arch, t.iso_8601 FROM rpm_packages AS rpm JOIN time AS t;
151119
152120
.. _nebula_usage_schedule:
153121

154122
Schedule
155123
--------
156124

157125
Nebula is designed to be used on a schedule. Here is a set of sample schedules
158-
for use with the sample pillar data contained in this repo:
126+
for use with the sample queries.
159127

160128
**hubble_nebula.sls (cont.)**
161129

‎_modules/nebula_osquery.py‎

Lines changed: 85 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -30,27 +30,27 @@
3030

3131
import copy
3232
import logging
33+
import os
34+
import sys
3335
import yaml
3436

3537
import salt.utils
3638
from salt.exceptions import CommandExecutionError
3739

3840
log = logging.getLogger(__name__)
3941

42+
__version__ = 'v2016.10.1'
4043
__virtualname__ = 'nebula'
4144

4245

4346
def __virtual__():
44-
if salt.utils.is_windows():
45-
return False, 'Windows not supported'
46-
if 'osquery.query' not in __salt__:
47-
return False, 'osquery not available'
4847
return __virtualname__
4948

5049

5150
def queries(query_group,
5251
query_file='salt://hubblestack_nebula/hubblestack_nebula_queries.yaml',
53-
verbose=False):
52+
verbose=False,
53+
report_version_with_day=True):
5454
'''
5555
Run the set of queries represented by ``query_group`` from the
5656
configuration in the file query_file
@@ -73,6 +73,33 @@ def queries(query_group,
7373
salt '*' nebula.queries hour verbose=True
7474
salt '*' nebula.queries hour pillar_key=sec_osqueries
7575
'''
76+
if salt.utils.is_windows() or 'osquery.query' not in __salt__:
77+
if query_group == 'day':
78+
log.warning('osquery not installed on this host. Returning baseline data')
79+
# Match the formatting of normal osquery results. Not super
80+
# readable, but just add new dictionaries to the list as we need
81+
# more data
82+
ret = []
83+
ret.append(
84+
{'fallback_osfinger': {
85+
'data': [{'osfinger': __grains__.get('osfinger', __grains__.get('osfullname'))}],
86+
'result': True
87+
}}
88+
)
89+
if 'pkg.list_pkgs' in __salt__:
90+
ret.append(
91+
{'fallback_pkgs': {
92+
'data': [{'name': k, 'version': v} for k, v in __salt__['pkg.list_pkgs']().iteritems()],
93+
'result': True
94+
}}
95+
)
96+
if report_version_with_day:
97+
ret.append(hubble_versions())
98+
return ret
99+
else:
100+
log.debug('osquery not installed on this host. Skipping.')
101+
return None
102+
76103
query_file = __salt__['cp.cache_file'](query_file)
77104
with open(query_file, 'r') as fh:
78105
query_data = yaml.safe_load(fh)
@@ -100,4 +127,57 @@ def queries(query_group,
100127
else:
101128
ret.append({name: query_ret})
102129

130+
if query_group == 'day' and report_version_with_day:
131+
ret.append(hubble_versions())
132+
103133
return ret
134+
135+
136+
def version():
137+
'''
138+
Report version of this module
139+
'''
140+
return __version__
141+
142+
143+
def hubble_versions():
144+
'''
145+
Report version of all hubble modules as query
146+
'''
147+
versions = {}
148+
149+
# Nova
150+
if 'hubble.version' in __salt__:
151+
versions['nova'] = __salt__['hubble.version']()
152+
else:
153+
versions['nova'] = None
154+
155+
# Nebula
156+
versions['nebula'] = version()
157+
158+
# Pulsar
159+
if salt.utils.is_windows():
160+
try:
161+
sys.path.insert(0, os.path.dirname(__salt__['cp.cache_file']('salt://_beacons/win_pulsar.py')))
162+
import win_pulsar
163+
versions['pulsar'] = win_pulsar.__version__
164+
except:
165+
versions['pulsar'] = None
166+
else:
167+
try:
168+
sys.path.insert(0, os.path.dirname(__salt__['cp.cache_file']('salt://_beacons/pulsar.py')))
169+
import pulsar
170+
versions['pulsar'] = pulsar.__version__
171+
except:
172+
versions['pulsar'] = None
173+
174+
# Quasar
175+
try:
176+
sys.path.insert(0, os.path.dirname(__salt__['cp.cache_file']('salt://_returners/splunk_nova_return.py')))
177+
import splunk_nova_return
178+
versions['quasar'] = splunk_nova_return.__version__
179+
except:
180+
versions['quasar'] = None
181+
182+
return {'hubble_versions': {'data': [versions],
183+
'result': True}}

‎hubblestack_nebula/hubblestack_nebula_queries.yaml‎

Lines changed: 10 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -1,15 +1,19 @@
11
fifteen_min:
22
- query_name: running_procs
3-
query: select p.name as process, p.pid as process_id, p.cmdline, p.cwd, p.on_disk, p.resident_size as mem_used, p.parent, g.groupname, u.username as user, p.path, h.md5, h.sha1, h.sha256 from processes as p left join users as u on p.uid=u.uid left join groups as g on p.gid=g.gid left join hash as h on p.path=h.path;
3+
query: SELECT p.name AS process, p.pid AS process_id, p.cmdline, p.cwd, p.on_disk, p.resident_size AS mem_used, p.parent, g.groupname, u.username AS user, p.path, h.md5, h.sha1, h.sha256 FROM processes AS p LEFT JOIN users AS u ON p.uid=u.uid LEFT JOIN groups AS g ON p.gid=g.gid LEFT JOIN hash AS h ON p.path=h.path;
44
- query_name: established_outbound
5-
query: select t.iso_8601 as _time, pos.family, h.*, ltrim(pos.local_address, ':f') as src, pos.local_port as src_port, pos.remote_port as dest_port, ltrim(remote_address, ':f') as dest, name, p.path as file_path, cmdline, pos.protocol, lp.protocol from process_open_sockets as pos join processes as p on p.pid=pos.pid left join time as t LEFT JOIN listening_ports as lp on lp.port=pos.local_port AND lp.protocol=pos.protocol LEFT JOIN hash as h on h.path=p.path where not remote_address='' and not remote_address='::' and not remote_address='0.0.0.0' and not remote_address='127.0.0.1' and port is NULL;
5+
query: SELECT t.iso_8601 AS _time, pos.family, h.*, ltrim(pos.local_address, ':f') AS src, pos.local_port AS src_port, pos.remote_port AS dest_port, ltrim(remote_address, ':f') AS dest, name, p.path AS file_path, cmdline, pos.protocol, lp.protocol FROM process_open_sockets AS pos JOIN processes AS p ON p.pid=pos.pid LEFT JOIN time AS t LEFT JOIN (SELECT * FROM listening_ports) AS lp ON lp.port=pos.local_port AND lp.protocol=pos.protocol LEFT JOIN hash AS h ON h.path=p.path WHERE NOT remote_address='' AND NOT remote_address='::' AND NOT remote_address='0.0.0.0' AND NOT remote_address='127.0.0.1' AND port is NULL;
66
- query_name: listening_procs
7-
query: select t.iso_8601 as _time, h.md5 as md5, p.pid, name, ltrim(address, ':f') as address, port, p.path as file_path, cmdline, root, parent from listening_ports as lp left JOIN processes as p on lp.pid=p.pid left JOIN time as t left JOIN hash as h on h.path=p.path WHERE not address='127.0.0.1';
7+
query: SELECT t.iso_8601 AS _time, h.md5 AS md5, p.pid, name, ltrim(address, ':f') AS address, port, p.path AS file_path, cmdline, root, parent FROM listening_ports AS lp LEFT JOIN processes AS p ON lp.pid=p.pid LEFT JOIN time AS t LEFT JOIN hash AS h ON h.path=p.path WHERE NOT address='127.0.0.1';
88
- query_name: suid_binaries
9-
query: select sb.*, t.iso_8601 as _time from suid_bin as sb join time as t;
9+
query: SELECT sb.*, t.iso_8601 AS _time FROM suid_bin AS sb JOIN time AS t;
1010
hour:
1111
- query_name: crontab
12-
query: select c.*,t.iso_8601 as _time from crontab as c join time as t;
12+
query: SELECT c.*,t.iso_8601 AS _time FROM crontab AS c JOIN time AS t;
1313
day:
1414
- query_name: rpm_packages
15-
query: select rpm.name, rpm.version, rpm.release, rpm.source as package_source, rpm.size, rpm.sha1, rpm.arch, t.iso_8601 from rpm_packages as rpm join time as t;
15+
query: SELECT rpm.name, rpm.version, rpm.release, rpm.source AS package_source, rpm.size, rpm.sha1, rpm.arch, t.iso_8601 FROM rpm_packages AS rpm JOIN time AS t;
16+
- query_name: os_info
17+
query: select * from os_version;
18+
- query_name: interface_addresses
19+
query: SELECT interface, address FROM interface_addresses WHERE NOT interface='lo';

0 commit comments

Comments
 (0)