Xonotic server monitoring

Hi, If you are hosting your own xonotic server you probably had an idea to make server monitoring with graphs and maybe even alerts (so you will receive notification if something goes wrong). I'd like to to show you one way to do it. I'm using it for monitoring my own servers, also most graphs are public, so you can check them here and here. I'll post screenshot for convenience.

Heart of my monitoring system is prometheus, which collects and stores metrics data, also it allows easy querying and processing of this data. It interesting system, because while most monitoring system are push based, prometheus is pull-based. It means that prometheus requests  metrics from other parties and not vice versa (other party not pushes metrics). As already you have seen, I'm using graphana as UI for prometheus data. Each graph you have seen in links I have posted below are result of prometheus queries. Any data store and UI aren't useful without actual data to process and graph. Promtheus can work with applications that specially developed to be compatible with it or with gateways (called exports in prometheus) that export metrics in prometheus format, so it can collect and save those metrics. I'm using 3 exporters:

• node_exporter — for collecting metrics about node/machine itself, it should be installed on each server you want to be monitored
  
• xonotic_exporter — which is my own stuff, it exports some game metrics via rcon, also measures rtt from machine it hosted to game server
  
• blackbox_exporter — it allows to measure rtt using different protocols, so you can compare in game rtt vs icmp ping rtt, etc.
  

For alerting I'm using alertmanager, it is part of prometheus project.

To make you setup easier, I'll post here example prometheus config:

Code:

global:
 scrape_interval:     15s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
 evaluation_interval: 15s # Evaluate rules every 15 seconds. The default is every 1 minute.

 - job_name: 'node_exporter'
   static_configs:
     - targets: ['pub.regulars.win:9100']

 - job_name: 'xonotic_exporter'
   relabel_configs:
     - source_labels: [__address__]
       target_label: __param_target
     - source_labels: [__param_target]
       target_label: instance
     - target_label: __address__
       replacement: 127.0.0.1:9260
   static_configs:
     - targets: ['pub.regulars.win', 'mars.regulars.win', 'votable.regulars.win']

It's actually simplified version of my own config. Also, note that prometheus is using yaml for confgiuration file.

If you have any question about this setup you can ask me here or IRC, I will be happy to help.

Also, I want announce about second release of xrcon library. New release have few fixes also new tool: xping. Which can ping xonotic server or client, also it supports other games too (it supports Quake 3, Quake 2 and qfusion protocols).

Hope this will be helpful for you.

That is really, really nice Slava. Thanks for sharing! Have an upvote.

Wow that looks good Slava, u the KING!!!

I will need some help to set this up:-)

cheers

kwak

Looks great! I tried installing it but when I run:

Code:

pip install -e git+https://github.com/bacher09/xrcon#egg=xrcon

i get these errors:

Code:

Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main

I would also like to set up graphs - can you help me with these issues? Thanks!

i get these errors:

Code:

Traceback (most recent call last):
  File "/usr/bin/pip", line 9, in <module>
    from pip import main
ImportError: cannot import name main

I would also like to set up graphs - can you help me with these issues? Thanks!

Perhaps this? https://stackoverflow.com/questions/4983...-name-main

Looks like that might be it... Thx.

This is a bit old (but for some reason I got email notification about this thread). Anyway, wanted to say that current exporter is a bit outdated and I made a new one. It has some api functionality, as well as prometheus exporter. That's what I'm using for regulars.win website.

This is a bit old (but for some reason I got email notification about this thread). Anyway, wanted to say that current exporter is a bit outdated and I made a new one. It has some api functionality, as well as prometheus exporter. That's what I'm using for regulars.win website.

Thank you. Looking into it.

This is a bit old (but for some reason I got email notification about this thread). Anyway, wanted to say that current exporter is a bit outdated and I made a new one. It has some api functionality, as well as prometheus exporter. That's what I'm using for regulars.win website.

Thinking of using a Docker Prometheus container and a Xonotic exporter.

Do you have any documentation?

You can check some graphs (that's public)  here .
Regarding documentation, no I don't have any documentation for it, but everything is dead simple. Old exporter has configuration sample and new one has JSON schema for the configuration file which follows same format.
You also can experiment with my instance that is running, for example players API:

Code:

https://api.regulars.win/servers

. Prometheus url in my instance is blocked from the internet (it's available for the prometheus).

Also it's open-source, so you can modify it as you want too to fit your need.

You can check some graphs (that's public)  here .
Regarding documentation, no I don't have any documentation for it, but everything is dead simple. Old exporter has configuration sample and new one has JSON schema for the configuration file which follows same format.
You also can experiment with my instance that is running, for example players API:

Code:

https://api.regulars.win/servers

. Prometheus url in my instance is blocked from the internet (it's available for the prometheus).

Also it's open-source, so  you can modify it as you want too to fit your need.

OK, thank you, I will check it out.