Skip to content

Commit 5287a68

Browse files
author
Marko Goerg
committed
add documentation
1 parent bbf4c07 commit 5287a68

File tree

19 files changed

+488
-277
lines changed

19 files changed

+488
-277
lines changed

‎.vscode/settings.json‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
{
2+
}

‎README.md‎

Lines changed: 24 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -2,17 +2,35 @@
22
WxMonitoring is a ready-to-use webMethods monitoring solution based on Open Elastic Stack (Elastic Search, Logstash, Filebeat), that provides rules-based log-file analysis and activity monitoring to optimize webMethods operations.
33

44
## Overview
5-
If running multiple webMethods productive environments it becomes critical to keep track of exceptions in near-time and leverage operational costs. WxMonitoring is a monitoring solution based on Open Elastic Stack, that provides a independent, centralized monitoring. It samples existing IS log-files and supports queries and aggregations views. Additionally automatized rule based actions allows you to react immediately after an exception occurs. They simply send mails, raise JIRA tickets or invoke a compensating service. Rules can be easily defined on existing los messages. For business users WxMonitoring provides views and reports on current and historical process executions.
6-
75
![System Architecture](ressources/img/architecture.png)
86

7+
If running multiple webMethods productive environments it becomes critical to keep track of exceptions in near-time and leverage operational costs. _WxMonitoring_ is a monitoring solution based on _Open Elastic Stack_, that provides a independent, centralized monitoring. It samples existing IS log-files and supports queries and aggregations views. Additionally automatized rule based actions allows you to react immediately after an exception occurs. They simply send mails, raise JIRA tickets or invoke a compensating service. Rules can be easily defined on existing los messages. For business users WxMonitoring provides views and reports on current and historical process executions.
8+
9+
For standalone installation you can simply import log files. This helps you to analyse logs from other environments without the need to be connected to. Together with purging data you can simulate different data sets.
10+
11+
### Features
12+
13+
* Dashboard (Overview Business Processes and Events)
14+
* Searches on Events/Processes
15+
* Aggreational View on Events
16+
* Aggreational View on Processes
17+
* Generate and Import Sample Data
18+
* Import log files
19+
* Purge Data
20+
21+
* Online Documentation
22+
923
## Jump Start
10-
To getting started just [download latest](dist/WxMonitoring-dist-0.1.zip) (current: v0.1) distribution package and unzip.
24+
Please download and use latest version (current: 0.2)
25+
* [Download](dist/WxMonitoring-dist-0.2.zip) distribution package and unzip
26+
* To getting started just see [Installation Guide](assets/IS/Packages/WxMonitoring/pub/docs/getting-started.md)
1127

12-
## Roapmap
13-
WxMonitoring started as a custom development and was ported to a community version. In November 2018 the first (beta) version 0.1 of WxMonitoring is released to give a first introduction.
28+
Further documentations
1429

15-
Several features will be added to this community edition, depending on customer requests.
30+
* [Release Notes](assets/IS/Packages/WxMonitoring/pub/docs/release-notes.md)
31+
* [Roadmap](assets/IS/Packages/WxMonitoring/pub/docs/roadmap.md)
32+
* [Users Guide](assets/IS/Packages/WxMonitoring/pub/docs/users-guide.md)
33+
* [Development Guide](assets/IS/Packages/WxMonitoring/pub/docs/development-guide.md)
1634

1735
## Authors
1836
[Marko Goerg](mailto:[email protected])

‎assets/IS/Packages/WxMonitoring/ns/WxMonitoring/docs/common/node.idf‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
3+
<Values version="2.0">
4+
<value name="node_type">interface</value>
5+
<value name="node_subtype">unknown</value>
6+
<value name="node_nsName">WxMonitoring.docs.common</value>
7+
<value name="is_public">false</value>
8+
</Values>

‎assets/IS/Packages/WxMonitoring/ns/WxMonitoring/docs/node.idf‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
3+
<Values version="2.0">
4+
<value name="node_type">interface</value>
5+
<value name="node_subtype">unknown</value>
6+
<value name="node_nsName">WxMonitoring.docs</value>
7+
<value name="is_public">false</value>
8+
</Values>

‎assets/IS/Packages/WxMonitoring/ns/WxMonitoring/docs/view/node.idf‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
3+
<Values version="2.0">
4+
<value name="node_type">interface</value>
5+
<value name="node_subtype">unknown</value>
6+
<value name="node_nsName">WxMonitoring.docs.view</value>
7+
<value name="is_public">false</value>
8+
</Values>

‎assets/IS/Packages/WxMonitoring/ns/WxMonitoring/node.idf‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,8 @@
1+
<?xml version="1.0" encoding="UTF-8"?>
2+
3+
<Values version="2.0">
4+
<value name="node_type">interface</value>
5+
<value name="node_subtype">unknown</value>
6+
<value name="node_nsName">WxMonitoring</value>
7+
<value name="is_public">false</value>
8+
</Values>

‎assets/IS/Packages/WxMonitoring/pub/docs/about.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,4 @@
1+
## About
2+
3+
* Marko Goerg
4+
* Puneet Arora

‎assets/IS/Packages/WxMonitoring/pub/docs/development-guide.md‎

Lines changed: 173 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,173 @@
1+
## Development Guide
2+
3+
This development guide gathers several topics around software development lifecycle. Therefore both content and topics may change over time and releases.
4+
5+
### Design Principles
6+
7+
1. Keep it simple
8+
9+
Keep it simple in sense of installation, configuration, usage, documentation to reduce complexity and avoid misuse
10+
11+
1. User-centric
12+
13+
All features should be motivated on users benefits.
14+
15+
1. Resilient first
16+
17+
Features should be resilient before functional complete.
18+
19+
1. Use Markdown
20+
21+
As Markdown is simple to use, Markdown should be a common way to provide projects documentation most fast and versatile
22+
23+
1. Selfdocumented
24+
25+
* Project structure (package folders, naming) should be self-explanatory, with no need to explain
26+
* assets (public IS Services) should be documented directly in flow
27+
28+
1. Use Github
29+
30+
Use GitHub to make project public available and interactive. This differs to most customer-related projects due to restricted contribution.
31+
32+
1. Selfcontained
33+
34+
??
35+
36+
### Dashboard
37+
38+
### Import log files
39+
40+
### Purge data
41+
42+
43+
### Interfaces
44+
45+
#### Filebeat
46+
47+
#### Logstash
48+
49+
```go
50+
input {
51+
beats {
52+
port => "${LOGSTASH_PORT:5044}"
53+
}
54+
http {
55+
port => "${LOGSTASH_HTTP_IMPORT_PORT:5045}"
56+
}
57+
}
58+
```
59+
60+
#### Beats
61+
62+
Logstash is listing on port :5044 to receive Beats log events.
63+
64+
#### Http
65+
66+
For debugging and importing events directly Logstash is listing on port :5045 to receive log events via REST.
67+
68+
Use this JSON-format to emit an message event:
69+
```json
70+
{
71+
"message":"2018-12-11 11:33:59 MEZ [ISS.0134.0058E] JMS Trigger test.wx.smoketest.services.edademo...",
72+
"log_identifier":"server_log",
73+
"source":"\\IS1\\server.log",
74+
"fields":{
75+
"env":"esbtest"
76+
}
77+
}
78+
```
79+
80+
81+
### Data Structures
82+
83+
#### Indexes
84+
85+
| index | description |
86+
| ------------- | ------------- |
87+
| wxmonitoring-events-original | |
88+
| wxmonitoring-events-[yyyy.MM.dd] | |
89+
| wxmonitoring-processes-original | |
90+
| wxmonitoring-processes | |
91+
| wxmonitoring-event-rules | |
92+
93+
#### Event
94+
```json
95+
{
96+
"_index": "wxmonitoring-events-original",
97+
"_type": "doc",
98+
"_id": "Zw0qmmcBRr67D9-7nTXm",
99+
"_score": 2.2693477,
100+
"_source": {
101+
"event_tags": "ISS",
102+
"complete_event_log": "2018-05-01 00:02:40 MESZ [ISS.0015.0005D] Invoking service wm.server.ns:getNode.",
103+
"event_severity": "DEBUG",
104+
"log_identifier": "server_log",
105+
"event_identifier": "unknown_IS_event",
106+
"event_timestamp_utc": "2018-04-30 22:02:40",
107+
"host": "127.0.0.1",
108+
"log_time_zone": "MESZ",
109+
"event_information": "Invoking service wm.server.ns:getNode.",
110+
"@timestamp": "2018-12-10T22:09:02.703Z",
111+
"event_code": "ISS.0015.0005",
112+
"source": """\IS\server.log""",
113+
"event_env": "serverdid2"
114+
}
115+
}
116+
```
117+
118+
#### Process
119+
```json
120+
{
121+
"_index": "wxmonitoring-processes",
122+
"_type": "doc",
123+
"_id": "202bb4e0-00e2-1a47-8335-fffffffe442c",
124+
"_score": 1,
125+
"_source": {
126+
"process_id": "202bb4e0-00e2-1a47-8335-fffffffe442c",
127+
"log_history": [
128+
"2018-11-08 14:39:34 MEZ [BPM.0102.0196I] 202bb4e0-00e2-1a47-8335-fffffffe442c:1, MID=WxSmoketestProcesses/SmokeTest3, MVer=1: process started",
129+
"2018-11-08 14:39:37 MEZ [BPM.0102.0199I] 202bb4e0-00e2-1a47-8335-fffffffe442c:1, MID=WxSmoketestProcesses/SmokeTest3, MVer=1: process failed"
130+
],
131+
"MVer": "1",
132+
"offset": null,
133+
"started_timestamp_utc": "2018-11-08 13:39:34",
134+
"process_name": "SmokeTest3",
135+
"business_domain": "WxSmoketestProcesses",
136+
"current_status": "failed",
137+
"process_env": "serverid1",
138+
"source": """\IS\server.log""",
139+
"process_instance": "1",
140+
"last_updated_utc": "2018-11-08 13:39:37",
141+
"failed_timestamp_utc": "2018-11-08 13:39:37",
142+
"event_tags": [
143+
"BPM"
144+
]
145+
}
146+
}
147+
```
148+
149+
### Design
150+
151+
### Coding Conventions
152+
153+
### Code Checks
154+
155+
* dependency checks
156+
* usage of external services
157+
* coding conventions
158+
* ...
159+
160+
#### webMethods Services
161+
162+
* wx.monitoring.services.gui.dashboard:getDashboard
163+
* wx.monitoring.services.gui.events:getEventsStats
164+
* wx.monitoring.services.admin:getEventsAgg
165+
* wx.monitoring.impl.maps:aggSourceToAggViewList
166+
* wx.monitoring.impl.persistence.handler:getESDocumentsByQuery
167+
* wx.monitoring.impl.persistence.queries.events:createGetEventAggQuery
168+
* wx.monitoring.services.gui.processes:getProcessesStats
169+
* wx.monitoring.services.admin:getProcessesAgg
170+
* wx.monitoring.impl.maps:aggSourceToAggViewList
171+
* wx.monitoring.impl.persistence.handler:**getESDocumentsByQuery**
172+
* wx.monitoring.services.adapters.es:invokeElasticSearch
173+
* wx.monitoring.impl.persistence.queries.processes:createGetProcessAggQuery

‎assets/IS/Packages/WxMonitoring/pub/docs/getting-started.md‎

Lines changed: 82 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,82 @@
1+
## Getting started
2+
3+
It is quite easy to install and run WxMonitoring. Following steps guide you through installation.
4+
5+
### Install Monitoring Environment
6+
7+
First, you need to install (at least) an instance of
8+
> **webMethods Integration Server**
9+
> * Version: 9.10 (or later)
10+
> * Packages required:
11+
> * _only_ Default, WmRoot, WmPublic
12+
> * no messaging
13+
> * no local database (use embedded database pool)
14+
15+
You may (re)use your current local webMethods environment.
16+
17+
Install the following 3rd party products on your local machine. (Skip this if already installed.)
18+
19+
> **Elasticsearch** (version: 6.2.x)
20+
> use [Installing ElasticSearch][install-elasticsearch]
21+
22+
> **Logstash** (version: 6.2.x)
23+
> use [Installing Logstash][install-logstash]
24+
25+
Start Servers
26+
27+
> **Elasticsearch**
28+
> (e.g.) run `%ES_HOME%/bin/elasticsearch.bat`
29+
30+
> **Integration Server**
31+
> (e.g.) run `%SAG_HOME%/profiles/IS_default/bin/startup.bat`
32+
33+
### Install WxMonitoring
34+
35+
1. Download the latest release of WxMonitoring from [GitHub][github-release].
36+
1. Install IS package WxMonitoring
37+
* copy WxMonitoring.zip to `%SAG_HOME%/IntegrationServer/instances/default/replicate/inbound`
38+
* open IS Administration http://localhost:5555/
39+
* start Packages > Management > Install Inbound Releases, choose WxMonitoring, install
40+
1. Open http://localhost:5555/WxMonitoring/
41+
* You should see an empty search result page
42+
1. Open http://localhost:9200/_cat/indices?v
43+
* You should see indexes - `wxmonitoring-processes`, `wxmonitoring-events-original`, `wxmonitoring-events-xxxx.xx.xx`, `wxmonitoring-event-rules` - have been created
44+
1. Start Logstash
45+
* Navigate to `...\packages\WxMonitoring\pub\elk\logstash` directory
46+
* Edit file `startLogstash.cmd` and update correct paths for following variables
47+
`SAG_HOME`: path for software ag home folder (e.g. `c:\SoftwareAG`)
48+
`LOGSTASH_HOME`: path to logstash home folder (for e.g. `C:\SoftwareAG\logstash-6.2.2`)
49+
* Run `startLogstash.cmd`
50+
1. View Logstash logs
51+
* open file `%LOGSTASH_HOME%/logs/logstash-plain`
52+
* verify following log has been recorded:
53+
`[INFO ][org.logstash.beats.Server] Starting server on port: 5044`
54+
55+
Now all components on monitoring enviroment are up and running. You can now
56+
1. start importing log files (see Users Guide), or
57+
1. install filebeat on target environment
58+
59+
### Install Log Environment
60+
61+
You need to install filebeat on each environment you want to listen to. Please repeat the following steps for ech environment.
62+
63+
> **Hint**:
64+
> Monitoring environment _should not_ be a log environment. You won't need to run WxMonitoring to receive log events anyway.
65+
66+
Install the following 3rd party products on your local machine. (Skip this if already installed.)
67+
68+
> **Filebeat** (version: 6.2.x)
69+
> use [Installing Filebeat][install-filebeat]
70+
71+
1. Copy files from `...\packages\WxMonitoring\pub\elk\filebeat` to anywhere in log environment server.
72+
* `startFilebeat.cmd`
73+
* `wxmonitoring_filebeat.yml`
74+
1. Start Filebeat
75+
* Edit file `startFilebeat.cmd` and update correct paths for following variables
76+
`SAG_HOME`: path for software ag home folder (e.g. `c:\SoftwareAG`)
77+
`FILEBEAT_HOME`: path to filebeat home folder (for e.g. `C:\SoftwareAG\filebeat-6.2.2`)
78+
`SERVER_ID`: name of this log environment (e.g. prod1)
79+
`LOGSTASH_HOST`: address URL of Logstash on monitoring environment (e.g. 127.0.0.1 or localhost)
80+
1. Run `startFilebeat.cmd`
81+
1. Wait a second and open WxMonitoring http://localhost:5555/WxMonitoring/
82+
* You should see incoming events on dashboard

0 commit comments

Comments
 (0)