Demo Parser

I decided to start a new thread since this has different focus from my original thread.

What I'm trying to do here?

1. As I explained in my other thread, I believe watching demos of oneself and others, in a game one participated in, is very educational, especially for new players. I'm hoping to write code to make available a config option on the server to allow users to download the demo files, right after the game. I want to read the demo files to expose some information (such as user name) on the file name. I also want to make sure there's no private data (such as IP address) in the demo file.
   
2. This one came to me when I was adding a filter for the demo file list in the game Media menu. I think in addition to search/sort the filename, if we can parse the demo files, we can allow the users to search/sort by time, map names, servers, players, game types, etc.
   
3. With the ability to parse the console message's timestamps in the demo file, video makers like @Antibody can have easier time locating clips.
   

What demo parsers are out there?

I was able to find two existing parsers:

• Sl@va has some basic parser written in Haskell. However, I don't know Haskell well enough to create a clean and safe environment on a production server (and would like to avoid cabal hell on my system).
  
• GreEn`mArine's NexuizDemoRecorder has some JAVA code to process the demoPacket, but it mostly only reads timing data, not other data such as scores, events, chats, etc.
  

Any other recommendation?

Is there a format documentation (beside the source code)?

If I really have to invent the wheel, is there a document? Obviously if everything fails I'll go read the Darkplaces code, but I'm hoping somebody has figured that out so I don't have to.

Here's one I prepared earlier: https://github.com/TimePath/xonotic-demo-parser

It's pending a rewrite and q1vm integration (needed to parse messages the game has control over).

What demo parsers are out there?

I was able to find two existing parsers:

• Sl@va has some basic parser written in Haskell. However, I don't know Haskell well enough to create a clean and safe environment on a production server (and would like to avoid cabal hell on my system).
  
• GreEn`mArine's NexuizDemoRecorder has some JAVA code to process the demoPacket, but it mostly only reads timing data, not other data such as scores, events, chats, etc.
  

Any other recommendation?

You might not still want to use it but just for the record:

Code:

$ cabal sandbox init
$ cabal install --only-dependencies
$ cabal build

This 'sandbox' feature is fairly new addition and solves most common problems with cabal.

Here's one I prepared earlier: https://github.com/TimePath/xonotic-demo-parser

It's pending a rewrite and q1vm integration (needed to parse messages the game has control over).

Thanks so much, @TimePath. Brilliant work!

I was able to run the parser and get the messages printed out, although it chocked at some sv_bad ("case unknown" in your DemoParser.scala). I tried several demo files and all of them have sv_bad.

May I know whether there's a document on those SVC.* types? Also, how are you using this utilities?



For those of you who are also interested, I installed gradle to build TimePath's jcompile as the dependencies (api.jar and q1vm.jar are needed), and then installed scala, kotlin, Apache commons.io for the runtime libs:

(the following is for Arch Linux 64 environment)

Code:

$ java -cp build/libs/xonotic-demo-parser-1.0-SNAPSHOT.jar:../jcompile/backends/q1vm/build/libs/q1vm-0.1-SNAPSHOT.jar:../jcompile/api/build/libs/api-0.1-SNAPSHOT.jar:/usr/share/scala/lib/*:/usr/share/java/commons-io/*:/usr/share/kotlin/lib/* com.timepath.xonotic.DemoParser ~/.xonotic/data/demos/2015-05-16_03-21_afterslime.dem

You might not still want to use it but just for the record:
...
...
This 'sandbox' feature is fairly new addition and solves most common problems with cabal.

Thanks, @machine! I'm glad you told me this -- otherwise I would have really brought my machine into the cabal hell! To build it, I had to relax some version requirement (base<4.8) with "--allow-newer" during dependencies installation, and manually changed "transformer<0.4" to "<0.5" to build. At the end, the code still has some error. I think I'll wait for Sl@ava to fix those before I dig deeper.

@BuddyFriendGuy:
There are a few currently unhandled messages: (https://github.com/TimePath/xonotic-demo...scala#L405), as soon as it sees one of them it just exits with sv_bad. I can't really do much about that until all messages are handled, as the length of each message is not sent.
I wrote all of this by looking at the darkplaces source. If there's documentation, I haven't found it. The original use for this utility was to try and discover why some servers were crashing on malicious player input using server demos, but we could never get a demo file.
Congratulations on setting all that up without further instructions btw , I usually just run things from my IDE. I should just add a `./gradlew run` task

Thanks a lot, @TimePath. Your code was well written -- that's why I was able to put things together.

I guess I'll just have to go read DarkPlaces code.