Commit 1ef8817c authored by Arne Wendt's avatar Arne Wendt
Browse files

removed dev demos, added documentation

parent 96d16e28
......@@ -34,7 +34,7 @@ add_library(scl_client scl_client.cpp)
#demos
add_subdirectory(demos)
#add_subdirectory(demos)
......
{
"configurations": [
{
"name": "x64-Debug",
"generator": "Ninja",
"configurationType": "Debug",
"inheritEnvironments": [
"msvc_x64_x64"
],
"buildRoot": "${env.USERPROFILE}\\CMakeBuilds\\${workspaceHash}\\build\\${name}",
"installRoot": "${env.USERPROFILE}\\CMakeBuilds\\${workspaceHash}\\install\\${name}",
"cmakeCommandArgs": "",
"buildCommandArgs": "-v",
"ctestCommandArgs": ""
},
{
"name": "x64-Release",
"generator": "Ninja",
"configurationType": "Release",
"inheritEnvironments": [
"msvc_x64_x64"
],
"buildRoot": "${env.USERPROFILE}\\CMakeBuilds\\${workspaceHash}\\build\\${name}",
"installRoot": "${env.USERPROFILE}\\CMakeBuilds\\${workspaceHash}\\install\\${name}",
"cmakeCommandArgs": "",
"buildCommandArgs": "-v",
"ctestCommandArgs": ""
}
]
}
\ No newline at end of file
# scl - simple centralized logging
Simple logging in a central server for distributed applications. Using ZeroMQ as transport.
Simple logging in a central server for distributed applications, *using ZeroMQ as transport*. It allows logging of arbitrary data, handled as string internaly, with the addition of a **log-level**, **timestamp**, **client application descriptor** and a **descriptor for the host** the application is running on.
The system consists of three components
* **server application** for log "aggregation"
* **command line utility** for viewing logs (*by request and as stream*), available client applications and hosts
* **client library** for logging to the server from your application
## usage documentation
### server usage
Start the server application with following switches/options. Some options can be substituted for environmental variables.
* `-l <port>, --log-port <port>, $SCLLOG_SERVERLOGPORT` **[mandatory]** port to listen on for incoming logs
* `-r <port>, --req-port <port>, $SCLLOG_SERVERREQPORT` **[mandatory]** port to listen in for requests
* `-p <port>, --pub-port <port>, $SCLLOG_SERVERPUBPORT` port for (re)publishing incoming logs from the central server. *If omitted, log are not published!*
* `-b <n>, --blocks <n>, $SCLLOG_BLOCKCOUNT` number of blocks (logs with fixed message length) to allocate for the internal in memory buffer, buffer is used for serving requests
* `-q, --quiet` suppress console output
* `-v, --verbose` show some info messages
### command line utility usage
The command line utility can be used to query for the names of client applications and hosts with log messages currently in the servers cache, query for logs from the servers cache and, subsribe to logs from individual hosts and client applications. The utility is controlled by the following options. Examples at the end of this section
* `-s <address>, --server <address>, $SCLLOG_SERVERADDR` address of the logging server; IP or hostname
* `-l, --local` sugar to set server address to `localhost`
Supplying either `-r` or `-p` will determine operation mode as: `-r` request data from server and `-p` subscribe to log feed from client application or host. *Supply one of both exclusively!**
* `-r <port>, --req-port <port>, $SCLLOG_SERVERREQPORT` port the server is listening on for requests. **Can be supplied as a switch (without value) to determine operation mode when the environmental variable is set!**
* `-h, --hosts` supplying either `-h` or `-c` will request the list of known clients or hosts in the servers cache.
* `-h [...] <name>, --hosts [...] <name>` supplying a name identifier after (not necessarily after, but as positional parameter) either `-h` or `-c` will request the logs for the given client application or host.
* `-c, --clients` supplying either `-h` or `-c` will request the list of known clients or hosts in the servers cache.
* `-c [...] <name>, --clients [...] <name>` supplying a name identifier after (not necessarily after, but as positional parameter) either `-h` or `-c` will request the logs for the given client application or host.
* `-n <n>, --num-logs <n>` show the last *n* log messages for `<name>` client or host
* `-i, --info` request only info `[NFO]` log messages
* `-w, --warning` request only warning `[WNG]` log messages
* `-e, --error` request only error `[ERR]` log messages
* `<name>` positional parameter, see `-c` or `-h`
Subscribe to all logs of a specified host or client. Published topics are, for every `<host>`, `<client>` and `<log level>`:
```
<host>/<client>/<log level>
<host>/<log level>
<client>/<log level>
```
* `-p <port>, --pub-port <port>, $SCLLOG_SERVERPUBPORT` port number logs are published on by server
* `[<name>]` list of one or more positional parameters (devided by space), representing the topics/log-feeds to subscribe to.
## scl server
### programm usage
document options:
```
"-s", "--server"
"-l", "--log-port"
"-r", "--req-port"
"-p", "--pub-port"
"-b", "--blocks"
"-q", "--quiet"
"-v", "--verbose"
EXAMPLE - requesting known client applications:
$> scl_cmd -lcr 2345
$SCLLOG_SERVERADDR (set by SCLLOG_SERVERADDR_ENVVAR preprocessor define)
$SCLLOG_SERVERLOGPORT (set by SCLLOG_SERVERLOGPORT_ENVVAR preprocessor define)
$SCLLOG_SERVERPUBPORT (set by SCLLOG_SERVERPUBPORT_ENVVAR preprocessor define)
$SCLLOG_SERVERREQPORT (set by SCLLOG_SERVERREQPORT_ENVVAR preprocessor define)
EXAMPLE - requesting last 10 error logs from host 'myhost_11'
$> scl_cmd -s 192.168.10.110 -r 2345 -hen 10 myhost_11
EXAMPLE - subscribing to log feeds
$> scl_cmd -lp 2341 hostX/clientAppX clientAppU/ERR hostY/clientAppW/NFO
```
### [in] logging interface protocol
## software documentation
### server
Collects logs from applications in a central point.
The application exposes an interface for logging and an interface for requests to the application (used by the command line utility for example). The request interface relies on the implementation of a cache module, implementing `scl::logCache` (and `scl::logSink` to gather the logs), to query this module for the requested information (logs, client application and host names).
Internally the application exposes interfaces for **sink** modules to push the collacted logs to for e.g. display, storage, further publication, you name it.
The interfaces and implemented (and included) modules are described below.
#### [in] logging interface protocol
Uses **ZeroMQ** `SUB`-Socket without topic framea for receiving logs. One log is one multipart message.
The message consists of the following 5 frames in shown order:
```
......@@ -39,7 +88,7 @@ MESSAGE/PAYLOAD FRAMES:
**Strings** shall be zero terminated by `'\0'`!
### [out][request] log and info request interface
#### [out][request] log and info request interface
This is a built in interface of the application, relying on the presence of a cache module, implementing `scl::logCache` (and `scl::logSink`). The server listens for requests on a `ROUTER`-socket. You shall connect using a `DEALER`-socket as you will get multiple reply-messages when requesting logs. When only requesting a list of known hosts and clients you may connect using `REQ`-sockets, as these are sent as one message.
**When connectin using a `DEALER`-socket, don't forget to add an empty delimiter frame before the payload, when sending your request! (Illustration below) Don't forget to remove the leading, empty routing-frame when receiving the response/s as well!* Using a `REQ`-socket will do this for you automatically, but can only handle one reply message, so no requesting logs.
......@@ -64,7 +113,7 @@ First payload frame of the request message shall be the information what is bein
**When no matching results are found, you will get a message consisting of routing frames only and no payload.**
#### `"HOSTS"`, `"CLIENTS"`
##### `"HOSTS"`, `"CLIENTS"`
Requests with a first (and only) payload-frame containing `"HOSTS"` or `"CLIENTS"` will return a list of known clients or hosts.
......@@ -77,7 +126,7 @@ PAYLOAD FRAMES: (REPLY TO "HOSTS"/"CLIENTS" REQUEST)
```
#### `"LOGS"`
##### `"LOGS"`
Will return a number of logs matching the following, additional filter criteria.
The `"LOGS"`-frame has to be followed by a frame containing `"HOST"` or `"CLIENT"`, again followed by a frame containing the hosts or clients name as **string** `"<host/client name>"`.
```
......@@ -116,10 +165,10 @@ MESSAGE n+1: (empty; termination)
```
##### Filtering requested logs
###### Filtering requested logs
To filter the logs by log-level or only get a number *N* of the most recent logs for this host or client, your can add additional frames specifying these filter criteria. You can add no filter, one or both, the order in which filters are supplied does not matter.
##### Filter by log-level
###### Filter by log-level
To filter by log-level, append the following two frames to your request message; with `"LEVEL"` as *string* and the log-level as **8-bit integer** with `ERR = 0, WNG = 1, NFO = 10, SYS = 11`:
```
ADDITIONAL PAYLOAD FRAMES: (FILTERING REQUEST FOR LOGS)
......@@ -128,7 +177,7 @@ ADDITIONAL PAYLOAD FRAMES: (FILTERING REQUEST FOR LOGS)
+---------+-----------------+
```
##### Filter to last *N* logs
###### Filter to last *N* logs
To get only the *N* most recent logs for a host or client, append the following two frames to your request message; with `"NUMBER"` as *string* and the number of messages as **8-bit integer** *(max 255 logs when requesting a limited number of logs)*:
```
ADDITIONAL PAYLOAD FRAMES: (FILTERING REQUEST FOR LOGS)
......@@ -137,11 +186,11 @@ ADDITIONAL PAYLOAD FRAMES: (FILTERING REQUEST FOR LOGS)
+----------+-----------------+
```
### [out] sinks
#### [out] sinks
Output is implemented via pluggable *(compile time)* sinks of interface type `scl::logSink` from `scl_log_types.h`. Implemented sinks are the following:
#### [sink] log publication interface
##### [sink] log publication interface
**TODO: publish assigned random ID**
Using the `logsPublisher` Class in `publish_logs.cpp/.h` as a **sink**, all received logs are by default published on a **ZeroMQ** `PUB`-Socket with topic-frame preceding the log message. A message consists of the followin frames, analog to the message format of the *logging interface*:
......@@ -164,42 +213,36 @@ Published topics are, for every `<host>`, `<client>` and `<log level>`:
**ZeroMQ** uses prefix matching for filtering subscriptions. A subscription to `<hostX>` will thus give you all logs of that specific host independent of log level, analogously subscribing to `<hostX>/<clientY>` will give all logs independent of log level for client `<clientY>` on host `<hostX>`. The same applies to subscriptions to `<hostZ>`.
#### [sink] console
##### [sink] console
`consoleLogs` class in `console_logs.h` implements a simple demo and debug **sink**, writing all received logs to the console. No filtering, no color highlighting, no other shenanigans...
#### [sink,cache] inmemory log cache
##### [sink,cache] inmemory log cache
Implemented as `inmemLogs` class in `inmem_logs.cpp/.h`. As **sink** it caches logs in an inmemory ring buffer of fixed size. Truncates host and client names to a standard of 255 byte (can be changed). Log messages longer than 1023 bytes are spread across multiple fixed size memory blocks.
It keeps track of clients and hosts with logs currently in the buffer as well.
Implementing the **cache** side allows querying for host and client names, als well as logs filtered by host, client and log level, by using the *request interface* of the server application.
## scl client library [C++]
### client library [C++]
Class `scl::sclClient` in `scl_client.cpp/.h`. Construct using a *string* parameter specifiying host-address and port to connect to. Omitting the hostname will automatically set your machines/containers hostname. Omitting the clients name will set it to `"client-<randomNumber>"`. Log by using the `log()` method.
```
EXAMPLE:
std::string logServer = "localhost:5555", clientName = "myClient" , hostName = "myHost";
scl::sclClient logger( logServer , clientName, hostName );
logger.log( 10 , std::string("some INFO log message" );
```
## scl command line viewer
document options
```
"-s", "--server"
"-l", "--local"
"-r", "--req-port"
"-h", "--hosts"
"-c", "--clients"
"-n", "--num-logs"
"-i", "--info"
"-w", "--warn"
"-e", "--error"
"-p", "--pub-port"
## building / compile
The project uses CMake to configure the build process.
**//TODO describe environmental variables**
**//TODO #defines controling buffer sizes!**
```
$SCLLOG_SERVERADDR (set by SCLLOG_SERVERADDR_ENVVAR preprocessor define)
$SCLLOG_SERVERPUBPORT (set by SCLLOG_SERVERPUBPORT_ENVVAR preprocessor define)
$SCLLOG_SERVERLOGPORT (set by SCLLOG_SERVERLOGPORT_ENVVAR preprocessor define)
$SCLLOG_SERVERPUBPORT (set by SCLLOG_SERVERPUBPORT_ENVVAR preprocessor define)
$SCLLOG_SERVERREQPORT (set by SCLLOG_SERVERREQPORT_ENVVAR preprocessor define)
```
\ No newline at end of file
```
add_executable(demo_client demo_client.cpp)
target_link_libraries(demo_client libzmq-static scl_client)
add_executable(demo_request_if demo_request_if.cpp)
target_link_libraries(demo_request_if libzmq-static)
add_executable(demo_subscriber demo_subscriber.cpp)
target_link_libraries(demo_subscriber libzmq-static)
#include "scl_log_types.h"
#include "inmem_logs.h"
#include <time.h>
#include <vector>
std::string unixTimestampGMT() {
time_t GMTTime;
time(&GMTTime);
gmtime(&GMTTime);
return std::to_string(GMTTime);
}
int main() {
inmemLogs inmemLogDB(10);
for (int i = 0; i <= 5; i++) {
if (rand() % 2) inmemLogDB.log(scl::INFO, unixTimestampGMT(), std::string("demo_client 1"), std::string("localhost"), std::string("simple net logging ") + std::to_string(rand()));
if (rand() % 2) inmemLogDB.log(scl::INFO, unixTimestampGMT(), std::string("demo_client 2"), std::string("localhost"), std::string("simple net logging ") + std::to_string(rand()));
if (rand() % 2) inmemLogDB.log(scl::INFO, unixTimestampGMT(), std::string("demo_client 3"), std::string("remotehost"), std::string("simple network logging two blocks ") + std::to_string(rand()));
}
std::vector<std::string> clients = inmemLogDB.getClients();
std::vector<std::string> hosts = inmemLogDB.getHosts();
std::cout << "Clients: " << std::endl;
for (auto it = clients.begin(); it != clients.end(); ++it) {
std::cout << *it << std::endl;
}
std::cout << std::endl;
std::cout << "Hosts: " << std::endl;
for (auto it = hosts.begin(); it != hosts.end(); ++it) {
std::cout << *it << std::endl;
}
std::cout << std::endl;
std::vector<scl::log> logs = inmemLogDB.getLogsByClient(clients.front());
std::cout << "Log messages for " << clients.front() << ": " << std::endl;
for (auto it = logs.begin(); it != logs.end(); ++it) {
std::cout << it->logMsg << std::endl;
}
std::cout << std::endl;
return EXIT_SUCCESS;
}
\ No newline at end of file
#include <stdlib.h>
#include <iostream>
#include <zmq.hpp>
#ifdef WIN32
#include <windows.h>
#define sleep(n) Sleep(n)
#else
#include <unistd.h>
#endif
#include "scl_log_common.h"
#include "console_logs.h"
bool getPayloadFramesBlocking(std::vector<zmq::message_t>&, zmq::socket_t&);
int main() {
consoleLogs cLogs;
zmq::context_t context(1);
zmq::socket_t logReqSock(context, ZMQ_DEALER);
logReqSock.connect("tcp://localhost:2345");
std::string clientName = "logclientA";
int logsCount = 5;
scl::logLevel logLevelReq = scl::WNG;
logReqSock.send("", 0, ZMQ_SNDMORE);
logReqSock.send("LOGS", 5, ZMQ_SNDMORE);
logReqSock.send("CLIENT", 7, ZMQ_SNDMORE);
logReqSock.send(clientName.c_str(), clientName.size() + 1, ZMQ_SNDMORE);
logReqSock.send("NUMBER", 7, ZMQ_SNDMORE);
logReqSock.send(&logsCount, 1, ZMQ_SNDMORE);
logReqSock.send("LEVEL", 6, ZMQ_SNDMORE);
logReqSock.send((int8_t*)&logLevelReq, 1, 0);
std::vector<zmq::message_t> payloadFrames; //create vector for received frames and allocate memory for 5 frames
while (getPayloadFramesBlocking(payloadFrames, logReqSock)) {
if (payloadFrames.size() == 1 && payloadFrames.at(0).size() == 0) break; //last termination frame
if (payloadFrames.size() != 5) continue; //sth went wrong and this message has no 5 frames payload, next
scl::log log("",
static_cast<scl::logLevel>(*(int8_t*)payloadFrames.at(0).data()),
std::string((char*)payloadFrames.at(1).data(), payloadFrames.at(1).size() - 1),
std::string((char*)payloadFrames.at(2).data(), payloadFrames.at(2).size() - 1),
std::string((char*)payloadFrames.at(3).data(), payloadFrames.at(3).size() - 1),
std::string((char*)payloadFrames.at(4).data(), payloadFrames.at(4).size() - 1));
cLogs.log(log);
}
while (true) {
sleep(1);
}
return EXIT_SUCCESS;
}
bool getPayloadFramesBlocking(std::vector<zmq::message_t>& framesVec, zmq::socket_t& socket) {
//get all message frames and store in vector
framesVec.clear();
zmq::message_t firstFrame;
socket.recv(&firstFrame); //blocking but all frames should be available anyways
if (firstFrame.size() != 0) return false; //no delimiter frame
while (socket.getsockopt<int>(ZMQ_RCVMORE) == 1) {
framesVec.push_back(zmq::message_t());
socket.recv(&framesVec.back()); //blocking but all frames should be available anyways
}
return true;
}
#include <stdlib.h>
#include <string>
#include <iostream>
#include <zmq.hpp>
#include "scl_log_common.h"
int main() {
zmq::context_t context(1);
zmq::socket_t logSubSock(context, ZMQ_SUB);
logSubSock.connect("tcp://localhost:5556");
logSubSock.setsockopt(ZMQ_SUBSCRIBE, "S3A", 3); //subscribe
while (true) {
zmq::message_t voidFrame;
zmq::message_t levelFrame;
zmq::message_t clientFrame;
zmq::message_t timeFrame;
zmq::message_t hostFrame;
zmq::message_t logFrame;
logSubSock.recv(&voidFrame); //discard topic frame
logSubSock.recv(&levelFrame);
if (levelFrame.size() != 1) {//something is wrong and the first frame is no integer representing our logLevel
while (logSubSock.getsockopt<int>(ZMQ_RCVMORE) == 1) {//discard all frames
logSubSock.recv(&voidFrame);
}
continue; //next message please, discard this one
}
if (logSubSock.getsockopt<int>(ZMQ_RCVMORE) != 1) continue; // less frames than expected
logSubSock.recv(&timeFrame);
if (logSubSock.getsockopt<int>(ZMQ_RCVMORE) != 1) continue; // less frames than expected
logSubSock.recv(&clientFrame);
if (logSubSock.getsockopt<int>(ZMQ_RCVMORE) != 1) continue; // less frames than expected
logSubSock.recv(&hostFrame);
if (logSubSock.getsockopt<int>(ZMQ_RCVMORE) != 1) continue; // less frames than expected
logSubSock.recv(&logFrame);
if (logSubSock.getsockopt<int>(ZMQ_RCVMORE) == 1) {//more frames than expected
while (logSubSock.getsockopt<int>(ZMQ_RCVMORE) == 1) {//discard all frames
logSubSock.recv(&voidFrame);
}
continue; //next message please, discard this one
}
std::cout << "[" << scl::logLevelStr(static_cast<scl::logLevel>(*(int8_t*)levelFrame.data())) << "] [" << (char*)timeFrame.data() << "] " << (char*)clientFrame.data() << " @ " << (char*)hostFrame.data() << ": " << (char*)logFrame.data() << std::endl;
}
return EXIT_SUCCESS;
}
......@@ -108,7 +108,7 @@ int main(int, char* argv[]) {
scl::logLevel level(scl::ANY);
if (cmds[{"-i", "--info"}]) level = scl::NFO;
if (cmds[{"-w", "--warn"}]) level = scl::WNG;
if (cmds[{"-w", "--warning"}]) level = scl::WNG;
if (cmds[{"-e", "--error"}]) level = scl::ERR;
std::string entity; cmds(1) >> entity;
......
......@@ -4,6 +4,7 @@
#include <cstdlib>
#include <memory>
#include <csignal>
#include <sstream>
#include <zmq.hpp>
#include "argh/argh.h"
......@@ -110,6 +111,17 @@ int main(int argc, char* argv[]) {
}
else if (_verbose) std::cout << "Using port number for log publishing from command line: " << pubPort << std::endl;
if (blocks == 0) {
char* blocksEnvVar = std::getenv(SCLLOG_BLOCKCOUNT_ENVVAR);
if (blocksEnvVar != NULL) {
std::stringstream strStream;
strStream << blocksEnvVar;
strStream >> blocks;//from char to int using stringstream
if (_verbose) std::cout << "Using cache block count from environment: " << blocks << std::endl;
}
}
else if (_verbose) std::cout << "Using cache block count from command line: " << blocks << std::endl;
//setup zmq
context = std::make_unique<zmq::context_t>(zmq::context_t(1));
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment