The Big Benchmark

Now this is interesting. I ran the benchmark on my Windows 7 Home 64bit installation on the same machine. Of course I updated the GPU driver so it uses the same drivers as my Linux installation and: performance is significantly lower! It seems my efforts to set up my Arch Linux in a way it uses as little ressources as possible was fruitful! See for yourself:

User: Halogene
Host: Owl
CPU: Intel i5 2500K 4x3.30 GHz
RAM: 8GB
GL_VENDOR: NVIDIA Corporation
GL_RENDERER: GeForce GTX 560/PCIe/SSE2
GL_VERSION: 4.2.0
OS: Windows 7 Home 64bit
Arch: x86_64


MED: 10510 frames 33.5610000 seconds 313.1611096 fps, one-second fps min/avg/max: 196 323 660 (336 seconds)
MED: 10510 frames 35.2620000 seconds 298.0545630 fps, one-second fps min/avg/max: 188 308 674 (336 seconds)
MED: 10510 frames 39.0080000 seconds 269.4319114 fps, one-second fps min/avg/max: 163 283 660 (336 seconds)
MED: 10510 frames 43.0840000 seconds 243.9420667 fps, one-second fps min/avg/max: 146 258 608 (336 seconds)
MED: 10510 frames 49.1480000 seconds 213.8439001 fps, one-second fps min/avg/max: 134 223 470 (336 seconds)
MED: 10510 frames 60.3200000 seconds 174.2374005 fps, one-second fps min/avg/max: 46 198 484 (336 seconds)
MED: 10510 frames 76.5400000 seconds 137.3138228 fps, one-second fps min/avg/max: 37 156 301 (336 seconds)

I didn't switch off any desktop effects or anything, but I don't have excessive background tasks running (I hate a crowded system tray). The latest system updates were installed.

As it is very high framerates which have been reduced, I would have to suggest looking at compositing effects which you have suggested. Just looking at the results other Windows users have had there seem a lot that are hitting around 300fps and considering specs getting nothing like the level I would hope for with that kind of hardware. I just have to consider that my E8200 with a 9600GT can get as high as 268fps on normal and no one has yet broken 400 despite this being a 4 year old midrange system going up against Core i7's with cards that use 150W+.

RAM usage will be making no difference when you've got a system of that kind of spec. RAM can only hit framerate when you really haven't got enough for Xonotic to run. Remember that other applications not being used will be paged first so so long as you have 1Gb of RAM there is nothing to be worried about.

Yes, I figure it will be the desktop environment that is impacting on performance, as I enabled the aero effects on windows 7, whereas on Arch I run the very lightweight LXDE on Openbox. But I don't really care, the windows Xonotic installation is for testing purposes only, actually this applies to the whole Windows installation I guess.

I suspect those differences is down to bad/faulty windows install / drivers Halogene, or possibly something in the background chewing up your cpu. Also are you sure there's no overclock involved? Some boards to that dynamically when under heavy load (sometimes tough drivers). As you can see in the differences between my zen and mediabox entries on the wiki page, using the same GPU (very same card in fact), a hd 6670 on a 3.1 ghz i3 versus a 4.6 ghz i5 gave a rather large performance increase across the board. Interesting result tough, especial if you can confirm threes no OC involved (but if there is not.. why the heck did you get a K series cpu..? ; ).

Well I didn't change anything with the windows install, just updated drivers occasionally, I don't tweak my Windows as I don't need no performance from it (heck, apart from Xonotic I play maybe a UT3 game every couple of months). Actually the system is more or less as it came to me half a year ago, except updated drivers. There's not even been significant software installs on it :o)

I didn't overclock under linux, too, it's a rather clean install with nothing really INCREASING performance (except of nvidia drivers perhaps), I just tweaked the system in that way that I did not install performance eating stuff I don't need.

The K series was offered without extra cost from the shop where I got the PC so I didn't really pay attention to it. I can try disabling aero and re-run, if anyone is interested, or I can try to re-run xonotic benchmark on an ubuntu install I still have on that machine...

It would be very interesting to see some results with Aero disabled. I think this will make up a lot of the difference.

I had a thought about how we can get some more benchmark results, help spread Xonotic and also offload the maintenance of the results page on the Wiki. Phoronix is already using Xonotic 0.5 as a benchmark with a different demo and different settings in the Phoronix Test Suite. Would it be be worth reaching out to Micheal Larabel at Phoronix and seeing if he would:
- Take up 'the-big-benchmark.sh' as the standardised Xonotic test in place of his own 0.5 setup
- Allow use of PTS results for assisting Xonotic optimisation at this late stage of development
- Make a wider call for submission of benchmark results from the current rsync/git tree

Phoronix is one of the big sources of gaming and 3D info for Linux and as Xonotic is possibility the biggest open source game in development right now you would think he might be interested and would be happy to spread to word a little. He also has all the contacts within both industry and community for driver development. If a bottleneck was to be uncovered on some specific hardware we might stand a better chance of a developer tackling the issue.

I would not want to speak to him myself as it should be a dev team message but let me know your thoughts.

Ok, ran the benchmark with disabled Aero effects now. I also disabled the real-time protection of my antivirus. It's a bit faster, not that much though, and surprisingly not in all presets.

User: Halogene
Host: Owl
CPU: Intel i5 2500K 4x3.30 GHz
RAM: 8GB
GL_VENDOR: NVIDIA Corporation
GL_RENDERER: GeForce GTX 560/PCIe/SSE2
GL_VERSION: 4.2.0
OS: Windows 7 Home 64bit
Arch: x86_64
Note: Aero effects disabled

MED: 10510 frames 55.2210000 seconds 190.3261440 fps, one-second fps min/avg/max: 132 204 620 (336 seconds)
MED: 10510 frames 33.8050000 seconds 310.9007543 fps, one-second fps min/avg/max: 201 322 689 (336 seconds)
MED: 10510 frames 38.4430000 seconds 273.3917748 fps, one-second fps min/avg/max: 167 287 620 (336 seconds)
MED: 10510 frames 40.3350000 seconds 260.5677451 fps, one-second fps min/avg/max: 154 274 596 (336 seconds)
MED: 10510 frames 46.9350000 seconds 223.9267071 fps, one-second fps min/avg/max: 136 235 492 (336 seconds)
MED: 10510 frames 63.8770000 seconds 164.5349656 fps, one-second fps min/avg/max: 41 187 477 (336 seconds)
MED: 10510 frames 72.4830000 seconds 144.9995171 fps, one-second fps min/avg/max: 37 165 313 (336 seconds)

Something still isn't right there. All the way up to normal settings I can get higher with a 9600GT.

What version are the drivers on both systems? The NVIDIA driver uses the same code for each platform (one of the main arguments for NVIDIA not opensourcing their driver) so with the same version number on each the entire OpenGL implementation is identical and can be ignored. Perhaps you could look at different CPU activity across Linux and Windows during the benchmarks? While Xonotic is not particularly parallelised with only audio being a separate thread, other work being done by the kernel or the driver may pay a roll and maybe this is not being run on other cores as you might expect?

I updated the GPU driver on Arch linux before I ran the benchmark, and I downloaded the GPU driver for Windows just befor I ran the benchmark too. There's only a day or two between the two benchmarks so they should be of the same version. I'm sorry but I don't really know how to analyze CPU activity and how to look for parallelised threads, I could fire up the task manager while running Xonotic in windowed mode and have a look at the CPU usage but that's about it. On linux, you'd have to tell me what command to run in console while Xonotic benchmark runs on graphical interface, but I doubt it's worth the effort?

Edit: oh, and I did not update chipset drivers except for GPU drivers directly before the benchmark, I did, however, update them after I got the system. That might be a couple of months ago, since I barely use windows at all.

Anyway, did a run on another machine I have here:

User: Halogene
Host: FortKnox
CPU: Intel Atom 330 @ 2x1.60 GHz
RAM: 2GB
GL_VENDOR: NVIDIA Corporation
GL_RENDERER: ION/integrated/SSE2
GL_VERSION: 3.3.0 NVIDIA 295.20
OS: Arch Linux
Arch: x86_64

MED: 10510 frames 192.1251481 seconds 54.7039266 fps, one-second fps min/avg/max: 31 57 117 (336 seconds)
MED: 10510 frames 193.2106309 seconds 54.3965927 fps, one-second fps min/avg/max: 34 57 121 (336 seconds)
MED: 10510 frames 214.3742661 seconds 49.0264069 fps, one-second fps min/avg/max: 29 52 121 (336 seconds)
MED: 10510 frames 227.4606681 seconds 46.2057906 fps, one-second fps min/avg/max: 26 49 99 (336 seconds)
MED: 10510 frames 279.6205230 seconds 37.5866545 fps, one-second fps min/avg/max: 22 39 59 (336 seconds)
MED: 10510 frames 646.8815479 seconds 16.2471785 fps, one-second fps min/avg/max: 6 18 31 (336 seconds)
MED: 10510 frames 1474.8725519 seconds 7.1260395 fps, one-second fps min/avg/max: 3 8 14 (336 seconds)

I've had a look at this myself again by benchmarking Arch x86-64 va Windows XP x86. A similar picture emerged of having slower performance on Windows at the lower quality settings. What was wierd though is that I was using really old 173 series NVIDIA drivers (to test on an older driver) and it did almost twice as fast on Ultra and Ultimate as either Arch or Windows XP with 295 series drivers.


  SG03-linux-vs-windows.png (Size: 39.52 KB / Downloads: 352)

The one possible thing that comes to mind is that offset mapping and relief mapping and postprocessing are required on Ultra and Ultimate and are not being used with OpenGL 2? Therefore the results are invalid? What are the requirements for these features? Does an OpenGL 3 implementation become a requirement?

I have also expanded this to look at a comparison of average framerates on each quality settign as taken from the Wiki results page and Linux has an obvious lead on all the lower quality settings but loses out at the higher quality settings. I'm not sure something is right here as the performance gaps are big. I believe that at the higher framerates in lower quality levels there is more of an impact from a compositing window manager and Aero may account for some of this. In addition there may be some better use of multithreading under Linux but perhaps something at the gigher levels like offset mapping has a bigger impact on Linux than on Windows? I have noticed it basically halves the performance Ultra under Linux but not so much under Windows.

   

I have attached results files for the 3 setups I have run and also the spreadsheet in which I have analysed this and drawn the graphs.

Edit: Fixed for .ods and PNG files inserted as images.

.ods is now supported. Just had to add it. Max file size is 3MB, same as .xls.

Your PNG files uploaded fine.

Thanks. PNG files uploaded but weren't immediately shown, this now looks fixed.

Are you aware that Phoronix Test Suite supports Xonotic benchamrking?

http://openbenchmarking.org/test/pts/xonotic

Yes, PTS does use Xonotic and has now been updated to 0.6 but does not use the same benchmark. This means results are incomparable. I would really like Phoronix to move to our benchmark so that we have comparable data.

So, I ran this benchmark on 4 different computers, and possible I will make more tests. First of them is my gaming desktop computer with Win7, second is my student laptop with Linux, third is my brother's MacBook Air and the last (and propably the most interesting) is my old netbook ASUS EEE PC 901. Hopefully these results give nice data, when I tried to run this benchmark on very different environments.

--------------------------------------------------------------

Computer #1: Desktop gaming computer

User: Exitium
Host: KALALAATIKKO
CPU: Intel Core i7 2700k
GHz: 3,4 GHz
Cores: 4
RAM: 8GB
Vendor: GL_VENDOR: NVIDIA Corporation
Card: GL_RENDERER: GeForce GTX 285/PCIe/SSE2
Driver: GL_VERSION: 3.3.0 NVIDIA 295.73
OS: Windows 7 Enterprise, 64-bit (Service Pack 1)
Arch: x86_64
Notes: Log file is the-big-benchmark_desktop.log

Computer #2: Student laptop (Sony VAIO VPCF12M1E)

User: Exitium
Host: kalalauta
CPU: Intel Core i5 520M
GHz: 2,4 GHz
Cores: 2
RAM: 4GB
Vendor: GL_VENDOR: NVIDIA Corporation
Card: GL_RENDERER: GeForce GT 330M/PCIe/SSE2
Driver: GL_VERSION: 3.3.0 NVIDIA 295.20
OS: Xubuntu 12.04 64-bit (Beta 1)
Arch: x86_64
Notes: Log file is the-big-benchmark_laptop.log

Computer #3: MacBook Air

User: Exitium
Host: xxx-MacBookAir
CPU: Intel Core i5
GHz: 1,6 GHz
Cores: 2
RAM: 2GB
Vendor: GL_VENDOR: Intel Inc.
Card: GL_RENDERER: Intel HD Graphics 3000 OpenGL Engine
Driver: GL_VERSION: 2.1 APPLE-7.14.5
OS: Mac OS X Lion 10.7
Arch: x86_64
Notes: Log file is the-big-benchmark_macair.log

Computer #4: Netbook (ASUS EEE PC 901)

User: Exitium
Host: sintti
CPU: Intel Atom CPU N270
GHz: 1,6 GHz
Cores: 1
RAM: 1GB
Vendor: GL_VENDOR: Tungsten Graphics, Inc
Card: GL_RENDERER: Mesa DRI Intel® 945GME x86/MMX/SSE2
Driver: GL_VERSION: 1.4 Mesa 7.11
OS: Lubuntu 11.10 32-bit
Arch: x86
Notes: Log file is the-big-benchmark_eee.log

Uhm, will the new benchmark runs be added to the wiki site? Also, can we already make a statement as to what minimum system specs are recommended to play the game? I would figure it should be something that allows for the game to play at constant 60 fps (so, use the "minimum" value) at "normal" preset, or what would you think?

Some things we already can say for definite. Minimum requirements for the Windows version is Windows XP. Recommended graphics card will have to be OpenGL 2.0 compliant to enable normal and above. This benchmark is only run at 1024x768 and we can expect people to run higher resolutions in general so I would think that the 60fps at normal would make a nice recommended setting too.

My experience with an Athlon XP 2400+ with 1Gb or RAM and a 6600GT gave just over 60fps once set up properly and it is a very playable system for the game. I would think however that a Geforce FX or Radeon 9xxx of any form will no longer be very usable as OpenGL 2.0 was not supported in hardware and also these cards are no longer supported by manufacturers. So a 6600GT as recommended?

NOTES:
- I couldn't clone the repo (not enough disk space)
- I tried with the 0.6.0 release but DDS textures are not supported.
- So I used 0.6.0-low
- Darkplaces compiled from git (v0.6.0)

system: laptop
CPU: Pentium Dual-Core T4200
GHz: 2Ghz
cores: 2
RAM: 2GB
vendor: Tungsten Graphics
card: Mobile Intel GM45
driver: 2.1 Mesa 7.8.2
OS: OpenBSD
arch: x86

omg:
MED: 10510 frames 210.4531264 seconds 49.9398616 fps, one-second fps min/avg/max: 32 51 61 (336 seconds)
low:
MED: 10510 frames 345.5596711 seconds 30.4144288 fps, one-second fps min/avg/max: 4 32 49 (336 seconds)
med:
freeze

Code:

assertion "region->map_refcount == 0" failed: file "/usr/xenocara/lib/libGL/dri/i965/../../../../dist/Mesa/src/mesa/drivers/dri/intel/intel_regions.c", line 274, function "intel_region_release"

- I tried with the 0.6.0 release but DDS textures are not supported.

Have you tried using 'export force_s3tc_enable=true' to enable texture compression in Mesa? It doesn't always help performance wise as I have already tested but it gets rid of the error message. In my experience a lack of texture compression may make things slower naturally but doesn't stop it from working. Did 0.6 not work at all then?

Have you tried using 'export force_s3tc_enable=true' to enable texture compression in Mesa? It doesn't always help performance wise as I have already tested but it gets rid of the error message.

This is gone:

Did 0.6 not work at all then?

It worked but all the textures were black. I tried again (also without force_s3tc_enable) and now I can see the textures. I don't know what was wrong :S

I have now done some RAM dependant tests with 1Gb, 2Gb, 3Gb and 4Gb RAM to show memory dependency. I've never seen tests of this detail done for any game before so I thought I'd give it a go.

- Core 2 Duo E8200
- Asus P5E-VM HDMI
- Geforce 9600GT
- Samsung HD501J 500Gb SATA HDD
- Arch Linux x86-64
- KDE 4.8.1 fully loaded
- compositing automatically disabled when Xonotic starts
- 2x 1Gb GEiL PC6400, 2x 1Gb Corsair PC6400 (timings and speeds identical)

   
No real difference in framerate.

I've also taken the 'time -p' real output from Xonotic and subtracted the 4 benchmark run times to get what I am going to call the Aggregated Loading Time. This represents all of the time required to start Xonotic, load each benchmark run and exit the gaem back to the console.

  aggregatedloadingtimes.png (Size: 32.84 KB / Downloads: 211)

1Gb is slower but still not bad considering that some commercial games take minutes just to load. I would have to say that 1Gb is perfectly usable, especially considering that this is on a fully featured desktop system that isn't really intended to have just 1Gb RAM, this just keeps it more real world.

The strange ups and downs aren't a sign of unreliable testing. It is because at medium, high and ultra the texture quality goes up, increasing the RAM usage and amount of paging qith 1Gb RAM. At the next quality level the time falls simply because the memory has already been made available.

2-4Gb makes little difference. I really can't recommend any more RAM than 2Gb if you're going to spend money on an existing system. There's no real return and the money would be better spent on a graphics card. While 2x4Gb DDR3 dual channel is best value for a new system (or 3x4Gb triple channel, 4x4Gb quad channel), any more is a waste of money. Spend it on something else and if you do need more RAM later on, you can always buy more when it's cheaper.

Unfortunately I can't turn dual channel off with this motherboard, hence asymmetric and interleaved mode run automatically on 3Gb and 4Gb respectively. This does effect the results (probably more than the quantity of RAM) but can't be helped with this motherboard. If anyone has a motherboard where this is possible I attach my spreadsheet to enable it to be worked out.

At low settings, textures are compressed into memory on load, increasing the loading time. IIRC.

I've doen some more RAM testing now. This was brought about by finding out that one stick of Corsair RAM I had was very very broken so only 3Gb is usable. This gives a problem as 3Gb means no interleaved mode which makes things slower. What I have benchmarked is:

- 3Gb asymmetric, 800MHz 4-5-4-12 - highest speeds, tightest timings that could be run stabily with this configuration)
- 2Gb interleaved, 800MHz 4-5-4-12 - identical speeds/timing for comparison purposes
- 2Gb interleaved, 800MHz 4-4-3-11 - tightest stable timings possible
- 2Gb interleaved, 1000MHz 5-5-4-13 - mental speeds with RAM that's only SPD rated as 800MHz 5-5-5-18!!!

Same thing again, FPS comparisons and aggregated loading time comparisons:
   

  aggregatedloadingtimes.png (Size: 31.36 KB / Downloads: 171)

3Gb does very marginally win in terms of loading times due to less paging however 2Gb even at the same speeds manages better FPS due to interleaved mode. What makes 2Gb of RAM better in this case is that because only 2 sticks are involved, speed can be raised to 1000MHz and timings tightened down too.

Why go to the trouble? Well it shows that you DO NOT need lots of RAM. You need ENOUGH RAM, but any more is wasted. Less than 2Gb will effect performance, more than that is very minor. The order of importance when tuning is frequency, multi-channel modes then latency.

What I also can't help but notice is that this is bringing this 4 year old system higher and higher in the results others have got. I'm pretty sure that with simple overclocking of CPU and graphics card I could set the 3rd highest result.

I'm not sure how you're reading anything of significance in this last test :x

Needs some error bars in there!

There are error bars in there but very small and not really dependable. In some ways the purpose is to show that there's little difference and therefore people shouldn't bother adding more RAM. If you have 2Gb that's plenty. Les than 2Gb might slow things down but there's no reason why it can't stop you playing the game, hence I think the 2Gb minimum listed in the 0.1 release and cited on the Wikipedia article needs to go. I've run Xonotic with less than 1Gb with no real slowdown in real gameplay.