README.md 13.5 KB
Newer Older
Arne Wendt's avatar
readme  
Arne Wendt committed
1
# scl - simple centralized logging
2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46
Simple logging in a central server for distributed applications, *using ZeroMQ as transport*. It allows logging of arbitrary data, handled as string internaly, with the addition of a **log-level**, **timestamp**, **client application descriptor** and a **descriptor for the host** the application is running on.
The system consists of three components
* **server application** for log "aggregation"
* **command line utility** for viewing logs (*by request and as stream*), available client applications and hosts
* **client library** for logging to the server from your application

## usage documentation
### server usage
Start the server application with following switches/options. Some options can be substituted for environmental variables.
* `-l <port>, --log-port <port>, $SCLLOG_SERVERLOGPORT` **[mandatory]** port to listen on for incoming logs
* `-r <port>, --req-port <port>, $SCLLOG_SERVERREQPORT` **[mandatory]** port to listen in for requests
* `-p <port>, --pub-port <port>, $SCLLOG_SERVERPUBPORT` port for (re)publishing incoming logs from the central server. *If omitted, log are not published!*
* `-b <n>, --blocks <n>, $SCLLOG_BLOCKCOUNT` number of blocks (logs with fixed message length) to allocate for the internal in memory buffer, buffer is used for serving requests
* `-q, --quiet` suppress console output
* `-v, --verbose` show some info messages


### command line utility usage
The command line utility can be used to query for the names of client applications and hosts with log messages currently in the servers cache, query for logs from the servers cache and, subsribe to logs from individual hosts and client applications. The utility is controlled by the following options. Examples at the end of this section
* `-s <address>, --server <address>, $SCLLOG_SERVERADDR` address of the logging server; IP or hostname
* `-l, --local` sugar to set server address to `localhost`
  
  
Supplying either `-r` or `-p` will determine operation mode as: `-r` request data from server and `-p` subscribe to log feed from client application or host. *Supply one of both exclusively!**
* `-r <port>, --req-port <port>, $SCLLOG_SERVERREQPORT` port the server is listening on for requests. **Can be supplied as a switch (without value) to determine operation mode when the environmental variable is set!**
* `-h, --hosts` supplying either `-h` or `-c` will request the list of known clients or hosts in the servers cache.
* `-h [...] <name>, --hosts [...] <name>` supplying a name identifier after (not necessarily after, but as positional parameter) either `-h` or `-c` will request the logs for the given client application or host.
* `-c, --clients` supplying either `-h` or `-c` will request the list of known clients or hosts in the servers cache.
* `-c [...] <name>, --clients [...] <name>` supplying a name identifier after (not necessarily after, but as positional parameter) either `-h` or `-c` will request the logs for the given client application or host.
* `-n <n>, --num-logs <n>` show the last *n* log messages for `<name>` client or host
* `-i, --info` request only info `[NFO]` log messages
* `-w, --warning` request only warning `[WNG]` log messages
* `-e, --error` request only error `[ERR]` log messages
* `<name>` positional parameter, see `-c` or `-h`
  
  
Subscribe to all logs of a specified host or client. Published topics are, for every `<host>`, `<client>` and `<log level>`:
```
<host>/<client>/<log level>
<host>/<log level>
<client>/<log level>
```

* `-p <port>, --pub-port <port>, $SCLLOG_SERVERPUBPORT` port number logs are published on by server
* `[<name>]` list of one or more positional parameters (devided by space), representing the topics/log-feeds to subscribe to.
Arne Wendt's avatar
readme  
Arne Wendt committed
47

48
```
49 50
EXAMPLE - requesting known client applications:
$> scl_cmd -lcr 2345
51

52 53 54 55 56
EXAMPLE - requesting last 10 error logs from host 'myhost_11'
$> scl_cmd -s 192.168.10.110 -r 2345 -hen 10 myhost_11

EXAMPLE - subscribing to log feeds
$> scl_cmd -lp 2341 hostX/clientAppX clientAppU/ERR hostY/clientAppW/NFO
57
```
58 59 60 61 62 63 64 65 66 67 68 69 70


## software documentation
### server
Collects logs from applications in a central point.
The application exposes an interface for logging and an interface for requests to the application (used by the command line utility for example). The request interface relies on the implementation of a cache module, implementing `scl::logCache` (and `scl::logSink` to gather the logs), to query this module for the requested information (logs, client application and host names).
Internally the application exposes interfaces for **sink** modules to push the collacted logs to for e.g. display, storage, further publication, you name it.
  
  
The interfaces and implemented (and included) modules are described below.


#### [in] logging interface protocol
71
Uses **ZeroMQ** `SUB`-Socket without topic framea for receiving logs. One log is one multipart message.
Arne Wendt's avatar
readme  
Arne Wendt committed
72 73
The message consists of the following 5 frames in shown order:
```
74 75
MESSAGE/PAYLOAD FRAMES:
+----------+-----------+--------+------+-------------+
Arne Wendt's avatar
readme  
Arne Wendt committed
76
|    1     |     2     |   3    |  4   |     5       |
77
+----------+-----------+--------+------+-------------+
Arne Wendt's avatar
readme  
Arne Wendt committed
78
| loglevel | timestamp | client | host | log message |
79
+----------+-----------+--------+------+-------------+
Arne Wendt's avatar
readme  
Arne Wendt committed
80 81
```

82
* `loglevel` as **8-bit integer** with `ERR = 0, WNG = 1, NFO = 10, SYS = 11`
Arne Wendt's avatar
readme  
Arne Wendt committed
83 84 85 86 87
* `timestamp`  10 digit UNIX-timestamp as **string** 
* `client` client application identifier as **string**
* `host` host identifier as **string**
* `log message` as **string**

88 89 90
**Strings** shall be zero terminated by `'\0'`!
  
  
91
#### [out][request] log and info request interface
92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113
This is a built in interface of the application, relying on the presence of a cache module, implementing `scl::logCache` (and `scl::logSink`). The server listens for requests on a `ROUTER`-socket. You shall connect using a `DEALER`-socket as you will get multiple reply-messages when requesting logs. When only requesting a list of known hosts and clients you may connect using `REQ`-sockets, as these are sent as one message.
  
**When connectin using a `DEALER`-socket, don't forget to add an empty delimiter frame before the payload, when sending your request! (Illustration below) Don't forget to remove the leading, empty routing-frame when receiving the response/s as well!* Using a `REQ`-socket will do this for you automatically, but can only handle one reply message, so no requesting logs.
```
MESSAGE FRAMES: (ROUTER / DEALER)
+-----------------+----------------------    ...    ----+
|        1        |    2         ...                  n |
+-----------------+----------------------    ...    ----+
|  routing frame  |     payload                         |
| empty, size = 0 |                 frames              |
+-----------------+----------------------    ...    ----+
|                 |               PAYLOAD               |
+-----------------+----------------------    ...    ----+
```

One reqest is one message consisting of a number of frames. **Following, only the payload frames are considered; please refer to information on `DEALER`-sockets!**

First payload frame of the request message shall be the information what is being requested, as a **zero terminated string**. Possible options are:
* `"HOSTS"` 
* `"CLIENTS"`
* `"LOGS"`

114 115
**When no matching results are found, you will get a message consisting of routing frames only and no payload.**

116
##### `"HOSTS"`, `"CLIENTS"`
117 118 119 120 121 122 123 124 125 126 127 128
Requests with a first (and only) payload-frame containing `"HOSTS"` or `"CLIENTS"` will return a list of known clients or hosts.
  
  
**The reply message** will be the list as one messge with individual host or clients as individual frames, containing a **string** with the hosts or clients name:
```
PAYLOAD FRAMES: (REPLY TO "HOSTS"/"CLIENTS" REQUEST)
+--------------------+--------------------+-  ...  -+--------------------+
| <host/client name> | <host/client name> |         | <host/client name> |
+--------------------+--------------------+-  ...  -+--------------------+
```


129
##### `"LOGS"`
130 131 132 133 134 135 136 137 138 139 140 141
Will return a number of logs matching the following, additional filter criteria.
The `"LOGS"`-frame has to be followed by a frame containing `"HOST"` or `"CLIENT"`, again followed by a frame containing the hosts or clients name as **string** `"<host/client name>"`.
```
PAYLOAD FRAMES: (REQUEST FOR LOGS)
+--------+-----------------+--------------------+
|   1    |        2        |         3          |
+--------+-----------------+--------------------+
| "LOGS" | "HOST"/"CLIENT" | <host/client name> |
+--------+-----------------+--------------------+
```
  
  
142
**The reply** to this request is a number of messages consisting each of a number of frames, with payload frames corresponding to the **logging interface**. The batch of messages is terminated by a message with an empty frame as payload.
143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159
```
PAYLOAD FRAMES: (REPLY TO LOGS REQUEST)

MESSAGE 1:
+----------+-----------+--------+------+-------------+
|    1     |     2     |   3    |  4   |     5       |
+----------+-----------+--------+------+-------------+
| loglevel | timestamp | client | host | log message |
+----------+-----------+--------+------+-------------+

...

MESSAGE n:
+----------+-----------+--------+------+-------------+
| loglevel | timestamp | client | host | log message |
+----------+-----------+--------+------+-------------+

160 161 162 163 164
MESSAGE n+1: (empty; termination)
+--+
|  |
+--+

165 166 167
```

  
168
###### Filtering requested logs
169 170
To filter the logs by log-level or only get a number *N* of the most recent logs for this host or client, your can add additional frames specifying these filter criteria. You can add no filter, one or both, the order in which filters are supplied does not matter.

171
###### Filter by log-level
172 173 174 175 176 177 178 179
To filter by log-level, append the following two frames to your request message; with `"LEVEL"` as *string* and the log-level as **8-bit integer** with `ERR = 0, WNG = 1, NFO = 10, SYS = 11`:
```
ADDITIONAL PAYLOAD FRAMES: (FILTERING REQUEST FOR LOGS)
+---------+-----------------+
| "LEVEL" | <log-level int> |
+---------+-----------------+
```

180
###### Filter to last *N* logs
181 182 183 184 185 186 187 188
To get only the *N* most recent logs for a host or client, append the following two frames to your request message; with `"NUMBER"` as *string* and the number of messages as **8-bit integer** *(max 255 logs when requesting a limited number of logs)*:
```
ADDITIONAL PAYLOAD FRAMES: (FILTERING REQUEST FOR LOGS)
+----------+-----------------+
| "NUMBER" | <count *N* int> |
+----------+-----------------+
```

189
#### [out] sinks
190 191 192
Output is implemented via pluggable *(compile time)* sinks of interface type `scl::logSink` from `scl_log_types.h`. Implemented sinks are the following:


193
##### [sink] log publication interface
194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215
**TODO: publish assigned random ID**

Using the `logsPublisher` Class in `publish_logs.cpp/.h` as a **sink**, all received logs are by default published on a **ZeroMQ** `PUB`-Socket with topic-frame preceding the log message. A message consists of the followin frames, analog to the message format of the *logging interface*:
```
MESSAGE FRAMES:
+-------+----------+-----------+--------+------+-------------+
|   1   |     2    |     3     |   4    |  5   |     6       |
+-------+----------+-----------+--------+------+-------------+
| topic | loglevel | timestamp | client | host | log message |
+-------+----------+-----------+--------+------+-------------+
```


Published topics are, for every `<host>`, `<client>` and `<log level>`:
```
<host>/<client>/<log level>
<host>/<log level>
<client>/<log level>
```
**ZeroMQ** uses prefix matching for filtering subscriptions. A subscription to `<hostX>` will thus give you all logs of that specific host independent of log level, analogously subscribing to `<hostX>/<clientY>` will give all logs independent of log level for client `<clientY>` on host `<hostX>`. The same applies to subscriptions to `<hostZ>`.
  
  
216
##### [sink] console
217 218 219
`consoleLogs` class in `console_logs.h` implements a simple demo and debug **sink**, writing all received logs to the console. No filtering, no color highlighting, no other shenanigans...


220
##### [sink,cache] inmemory log cache
221 222 223 224 225
Implemented as `inmemLogs` class in `inmem_logs.cpp/.h`. As **sink** it caches logs in an inmemory ring buffer of fixed size. Truncates host and client names to a standard of 255 byte (can be changed). Log messages longer than 1023 bytes are spread across multiple fixed size memory blocks.
It keeps track of clients and hosts with logs currently in the buffer as well.
Implementing the **cache** side allows querying for host and client names, als well as logs filtered by host, client and log level, by using the *request interface* of the server application.


226
### client library [C++]
227 228
Class `scl::sclClient` in `scl_client.cpp/.h`. Construct using a *string* parameter specifiying host-address and port to connect to. Omitting the hostname will automatically set your machines/containers hostname. Omitting the clients name will set it to `"client-<randomNumber>"`. Log by using the `log()` method.
```
229 230
EXAMPLE:

231 232 233 234 235 236
std::string	logServer = "localhost:5555", clientName = "myClient" , hostName = "myHost";
scl::sclClient	logger( logServer , clientName, hostName );

logger.log( 10 , std::string("some INFO log message" );
```

237

238 239
## building / compile
The project uses CMake to configure the build process.
240

241 242 243
**//TODO describe environmental variables**
**//TODO #defines controling buffer sizes!**
```
244
$SCLLOG_SERVERADDR 	(set by SCLLOG_SERVERADDR_ENVVAR preprocessor define)
245 246
$SCLLOG_SERVERLOGPORT 	(set by SCLLOG_SERVERLOGPORT_ENVVAR preprocessor define)
$SCLLOG_SERVERPUBPORT 	(set by SCLLOG_SERVERPUBPORT_ENVVAR preprocessor define)
247
$SCLLOG_SERVERREQPORT	(set by SCLLOG_SERVERREQPORT_ENVVAR preprocessor define)
248
```