04-01-2018, 05:08 AM
Hi, If you are hosting your own xonotic server you probably had an idea to make server monitoring with graphs and maybe even alerts (so you will receive notification if something goes wrong). I'd like to to show you one way to do it. I'm using it for monitoring my own servers, also most graphs are public, so you can check them here and here. I'll post screenshot for convenience.
Heart of my monitoring system is prometheus, which collects and stores metrics data, also it allows easy querying and processing of this data. It interesting system, because while most monitoring system are push based, prometheus is pull-based. It means that prometheus requests metrics from other parties and not vice versa (other party not pushes metrics). As already you have seen, I'm using graphana as UI for prometheus data. Each graph you have seen in links I have posted below are result of prometheus queries. Any data store and UI aren't useful without actual data to process and graph. Promtheus can work with applications that specially developed to be compatible with it or with gateways (called exports in prometheus) that export metrics in prometheus format, so it can collect and save those metrics. I'm using 3 exporters:
For alerting I'm using alertmanager, it is part of prometheus project.
To make you setup easier, I'll post here example prometheus config:
It's actually simplified version of my own config. Also, note that prometheus is using yaml for confgiuration file.
If you have any question about this setup you can ask me here or IRC, I will be happy to help.
Also, I want announce about second release of xrcon library. New release have few fixes also new tool: xping. Which can ping xonotic server or client, also it supports other games too (it supports Quake 3, Quake 2 and qfusion protocols).
Hope this will be helpful for you.
Heart of my monitoring system is prometheus, which collects and stores metrics data, also it allows easy querying and processing of this data. It interesting system, because while most monitoring system are push based, prometheus is pull-based. It means that prometheus requests metrics from other parties and not vice versa (other party not pushes metrics). As already you have seen, I'm using graphana as UI for prometheus data. Each graph you have seen in links I have posted below are result of prometheus queries. Any data store and UI aren't useful without actual data to process and graph. Promtheus can work with applications that specially developed to be compatible with it or with gateways (called exports in prometheus) that export metrics in prometheus format, so it can collect and save those metrics. I'm using 3 exporters:
- node_exporter — for collecting metrics about node/machine itself, it should be installed on each server you want to be monitored
- xonotic_exporter — which is my own stuff, it exports some game metrics via rcon, also measures rtt from machine it hosted to game server
- blackbox_exporter — it allows to measure rtt using different protocols, so you can compare in game rtt vs icmp ping rtt, etc.
For alerting I'm using alertmanager, it is part of prometheus project.
To make you setup easier, I'll post here example prometheus config:
Code:
global:
scrape_interval: 15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.
- job_name: 'node_exporter'
static_configs:
- targets: ['pub.regulars.win:9100']
- job_name: 'xonotic_exporter'
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: 127.0.0.1:9260
static_configs:
- targets: ['pub.regulars.win', 'mars.regulars.win', 'votable.regulars.win']It's actually simplified version of my own config. Also, note that prometheus is using yaml for confgiuration file.
If you have any question about this setup you can ask me here or IRC, I will be happy to help.
Also, I want announce about second release of xrcon library. New release have few fixes also new tool: xping. Which can ping xonotic server or client, also it supports other games too (it supports Quake 3, Quake 2 and qfusion protocols).
Hope this will be helpful for you.