Improving Xonotic packaging and compression to make the download smaller

Xonotic 0.6 came out at 964Mb and with the improvements still coming, 1.0 could break the Gb barrier. All of this size does create a barrier to getting more people into this game and has other problems too. Contrary to the belief of teenages using their parents Internet, students in university accommodation and freeloaders hopping onto someone elses wifi, bandwidth always has a cost to someone.

Prior to Nexuiz 2.0 I did a bit of work which hacked 30Mb off the download size at the time and I thought it would be worth doing this again. This is all intended to be without loss of quality, it's just about putting a bit more effort into compression. Hopefully this may have a small performance payoff when loading the game too as when you compress a ZIP harder, it is ALWAYS faster to extract, unlike LZ78 based formats.

I've based this on the rysnc version as this is packaged up. Some of these items are just cleaning up too.

1. There are a lot of empty directories within the PK3 files under textures/. I'm sure these are left from some useful point in the past but now just add to the size of the ZIP and overcomplicate the file structure the game has to work with.

These could be found and removed with:

Code:

for c in `find . -type d -empty`
do

rm -rf $c

done

Or use this in the makefile to pass a list of exclusions to 7za.

2. Both font-nimbussansl-rsync.pk3 and font-xolonium-rsync.pk3 contain an identical unifont....ttf file which sits in data/fonts. As pk3's are read in alphanumeric order the xolonium one with replace the nimbussansl one in use. The file is quite large and might compress well but it's still 2.7Mb. This could either be removed from one PK3 or the 2 PK3s combined.

3. data-rysnc.pk3 has an autoexec/empty.cfg file in it. Does this need to be here?

4. Compression of the PK3s and the main ZIP with 7za is not as aggressive as it could be. About 1% can be gained by compressing with:

Code:

7za a -tzip -mfb15 -mpass258

5. There are a number of makefiles, perl scripts, ruby scripts in the PK3s themselves. Do these files need to be there? If they are only used to build certain parts of the game, they could be in source/ or available just on git. Having them in the PK3's adds to the filelist that the gaem is having to run with.

6. There are quite a lot of darkplaces-icon files in data-rysnc.pk3. Not sure if these are duplicated from source/darkplaces/ but they seem unneccessary where they are anyway. To build the executable with the icon, they're in the source directory, if a user wants an icon for their desktop, they're not going to look in the PK3! If they are intended as master files for making smaller icons out of then they could be left in git rather than the PK3 where they just bloat the game filestructure.

7. tuba.png and tuba.svg images are in docs/. Should these be in docs/htmlfiles/weaponimg? Or excluded?

8. fteqcc/ could perhaps be moved in misc/tools/ as it matches in better with whats in there.

9. There is an sdl.dll in the root of the game directory. Should this be moved into bin32/? There is already one in bin64/ but not bin32/. Maybe some technical reason I've missed.

10. PNG images can be optimised a little more:

Code:

for a in `find . -name *.png`
do

optipng -nb -nc -np -nz -i0 -q -strip all $a
advpng -z4 $a

done

11. JPEG images can be optimised more too:

Code:

for b in `find . -name *.jpg`
do

jpegoptim --strip-all --quiet $b
jpegtran -progressive -outfile $b $b
jpegoptim --strip-all --quiet $b
jpegoptim --strip-all -m95 --quiet $b    # Cap JPEG's at m%

done

Don't worry about the multiple jpegoptim runs, that's needed to get it done right!

12. The executables and libraries now total over 100Mb! While this does compress quite a bit, they remain a very substantial part of the package. While a universal download has to contain so many architectures this will be the case and can't be helped. If the x86 builds were to change to -march=pentium3 then a small reduction might be seen.

13. Sources. I know it's a GPL requirement for them to be available but do they need to be in the main package? Do the map sources need to be in the PK3? 99.9% of downloaders won't ever use any of the sources. Anyone else can look on git. That's one opinion on it, obviously there are a lot of different ones.

No idea on a precise figure for improved size as yet but maybe 900Mb is possible?

I'd be happy to implement any of these in git if someone wants to give me access. I realise there is a lot of 'faffing' to be done in some of these cleanup steps and quite a bit of CPU time in rebuilding everything but it may actually make maintaining and rebuilding things easier in future just by the cleanup involved.

You can apply for git access by posting a support ticket to the tracker in the repository category (example) with the pubkey of the keypair that you wish to use for authentication. With user3 access you can freely push and delete in edh/* branches.

EDIT: And perhaps be present on irc (#xonotic on freenode) when you post that tracker item, too.

2. Both font-nimbussansl-rsync.pk3 and font-xolonium-rsync.pk3 contain an identical unifont....ttf file which sits in data/fonts. As pk3's are read in alphanumeric order the xolonium one with replace the nimbussansl one in use. The file is quite large and might compress well but it's still 2.7Mb. This could either be removed from one PK3 or the 2 PK3s combined.

I'm working on an optimised version of the unifont ttf, which should be less than 10 MB (about 2 MB when compressed).
This then also could be a good opportunity to have a look at repackaging it, getting rid of redundancies, and overall save about 3.5 MB.

Thanks Mr B, I will look at getting everything set up for git, maybe not something I'm going to do immediately as I havn't used git before.

I'm working on an optimised version of the unifont ttf, which should be less than 10 MB (about 2 MB when compressed).
This then also could be a good opportunity to have a look at repackaging it, getting rid of redundancies, and overall save about 3.5 MB.

I'm curious about why the fonts are in 2 seperate pk3's. Is it by design in case an incremental release means one gets replaced?

A bigger issue I have now come across (and raised in the tracker) is that the Nexuiz tutorial sounds are still being included. There is no tutorial map now and these sounds not only include the mention of Nexuiz but also alienTRAP. For branding reasons this lot should be removed especially with sensitivies that could arise with using those names.

march=pentium 3 ... seriously do you have any fking Idea how slow that is
xonotic should be using SSE2 at the very least as I have seen 20% performance gains simply by adding -O2 -march=AMDFAM10 -mtune=AMDFAM10 -SSE2 you are suggesting a 30% loss in performance to save haft a MB .... bandwidth inst that expensive in fact I would like to see xonotic more heavy optimist-ed for recent hardware more use of SSE4 sets and more cpu dependent opflags its 2012 .. Pentium 3's and x86 are pretty much dead
you would see more of a reduction in filesize if you did away with zip-based pk3s and switched to something that's not a leftover from 1998
http://en.wikipedia.org/wiki/PAQ6
PAQ8PF would be a good algy for game data

march=pentium 3 ... seriously do you have any fking Idea how slow that is
xonotic should be using SSE2 at the very least as I have seen 20% performance gains simply by adding -O2 -march=AMDFAM10 -mtune=AMDFAM10 -SSE2 you are suggesting a 30% loss in performance to save haft a MB .... bandwidth inst that expensive in fact I would like to see xonotic more heavy optimist-ed for recent hardware more use of SSE4 sets and more cpu dependent opflags its 2012 .. Pentium 3's and x86 are pretty much dead
you would see more of a reduction in filesize if you did away with zip-based pk3s and switched to something that's not a leftover from 1998
http://en.wikipedia.org/wiki/PAQ6
PAQ8PF would be a good algy for game data

This thread was made to discuss this exact topic. Please repost there. Also, I don't understand, it's 30% slower compared to what exactly?

EDIT: Also, read the topic and reconsider your opinion after that!

march=pentium 3 ... seriously do you have any fking Idea how slow that is
xonotic should be using SSE2 at the very least as I have seen 20% performance gains simply by adding -O2 -march=AMDFAM10 -mtune=AMDFAM10 -SSE2 you are suggesting a 30% loss in performance to save haft a MB .... bandwidth inst that expensive in fact I would like to see xonotic more heavy optimist-ed for recent hardware more use of SSE4 sets and more cpu dependent opflags its 2012 .. Pentium 3's and x86 are pretty much dead

I'll answer this in the thread Mr B referenced: http://forums.xonotic.org/showthread.php...3#pid42473

you would see more of a reduction in filesize if you did away with zip-based pk3s and switched to something that's not a leftover from 1998
http://en.wikipedia.org/wiki/PAQ6
PAQ8PF would be a good algy for game data

There are some issues with doing that. A very simple non-technical one is that ZIP is simply convention for Quake derived games and this is a major reason for id's games being so moddable.

From a technical standpoint you need to remember that PK3's are not just about reducing disk footprint - they are also there to make the game load FASTER. It is incorrectly assumed that extracting on the fly slows things down with ZIP being very light in CPU and memory this is not the case. Hard disk is always the bottleneck and by reducing the disk footprint at expense of a little CPU time, it loads faster. I remember back in 2004 there were all kinds of Doom 3 'tweaking' guides which recommended extracting PK4's to speed up loading when in actual fact this would have had a detrimental effect due to the extra disk time needed.

Many more modern compression formats such as anything lz78 based, 7z and also PAQ which you mention use much higher CPU and memory when compressing and extracting which would equate to a major increase in loading time over PK3. This makes it less suitable.

I deleted autoexec/empty.cfg and finally Xonotic fits on a single floppy disk!

march=pentium 3 ... seriously do you have any fking Idea how slow that is
xonotic should be using SSE2 at the very least as I have seen 20% performance gains simply by adding -O2 -march=AMDFAM10 -mtune=AMDFAM10 -SSE2 you are suggesting a 30% loss in performance to save haft a MB .... bandwidth inst that expensive in fact I would like to see xonotic more heavy optimist-ed for recent hardware more use of SSE4 sets and more cpu dependent opflags its 2012 .. Pentium 3's and x86 are pretty much dead

I'll answer this in the thread Mr B referenced: http://forums.xonotic.org/showthread.php...3#pid42473

you would see more of a reduction in filesize if you did away with zip-based pk3s and switched to something that's not a leftover from 1998
http://en.wikipedia.org/wiki/PAQ6
PAQ8PF would be a good algy for game data

There are some issues with doing that. A very simple non-technical one is that ZIP is simply convention for Quake derived games and this is a major reason for id's games being so moddable.

From a technical standpoint you need to remember that PK3's are not just about reducing disk footprint - they are also there to make the game load FASTER. It is incorrectly assumed that extracting on the fly slows things down with ZIP being very light in CPU and memory this is not the case. Hard disk is always the bottleneck and by reducing the disk footprint at expense of a little CPU time, it loads faster. I remember back in 2004 there were all kinds of Doom 3 'tweaking' guides which recommended extracting PK4's to speed up loading when in actual fact this would have had a detrimental effect due to the extra disk time needed.

Many more modern compression formats such as anything lz78 based, 7z and also PAQ which you mention use much higher CPU and memory when compressing and extracting which would equate to a major increase in loading time over PK3. This makes it less suitable.

this is no longer the case with most modern hard drives most recent hard drives even late generation mechanical ones there seek times far exceed the benefit that packing the assets originally was intended to give you
and we are not talking about doom 3 doom 3 was a special case because of the file size of the individual assets and yes the pk3 has been the old stranded for ages but everything else in the engine has been rewriten why no the asset loading and package system ?
and granted PAQ* is heavy but even switching to LZW would provide a ~50-100MB loss in filesize

You are mixing seek times with read speed. ~90% of regular SATA300 drives don't exceed 80-100MB/s read speed (with a lot of them being even around 60MB/s). Now unless you're using an array of server grade SAS drives or one of those fancy new hdd-sized usb sticks (called ssd I believe) you don't get much better than this.

Seek time - time to find the start of the physical data - takes fractions of a second.
Read speed - a whole different story.

You are mixing seek times with read speed. ~90% of regular SATA300 drives don't exceed 80-100MB/s read speed (with a lot of them being even around 60MB/s). Now unless you're using an array of server grade SAS drives or one of those fancy new hdd-sized usb sticks (called ssd I believe) you don't get much better than this.

Seek time - time to find the start of the physical data - takes fractions of a second.
Read speed - a whole different story.

exactly compressing the data helps seek times but hurts read speed because you need to read through the pk3 before you can read the particular asset the upside however is that by condensing the data into a package you avoid nasty seek time / fragmentation slow downs those really hurt when loading lots of random bits
most modern platter drives have a full sweep seek time of 6-9Ms
assuming a 7200rpm drive total access latency at the drive is normally less then 15ms
what we need is a packaging system that is able to compress the data better then plain old zip but still provide the speed indexing the package file would help making the game engine aware of where exactly on the disk the data and package file is contained

exactly compressing the data helps seek times but hurts read speed

No. It improves read speed. The disk is only able to work in the hundreds of megabytes per second where a modern CPU is 20Gb/s+. Put more work on the CPU for less time on the disk. While you at least now accept that your propsal for PAQ wouldn't work, LZW would be a problem too as again, the more you compress, the slower it extracts. Unlike ZIP.

I think he means that the engine has to read through a lot of the archive to reach the position of a single file.

That would be true for solid archives. AFAIK non-solid archives are seekable. I even saw a FUSE filesystem that could mount a zip archive.

Speaking of compression formats - lzma is known for high compression ratio and fast decompression. If change to anything I think this would be a good option.

Speaking of compression formats - lzma is known for high compression ratio and fast decompression. If change to anything I think this would be a good option.

While it is faster extracting than many advanced formats, it still can't touch ZIP. Take a look at this data:

Decompression speed, MB/s of uncompressed data (1 MB = 1024 * 1024 bytes)
gzip bzip2 lzmash lzmash -e
1 61 12 18 -
2 61 8.4 19 -
3 61 6.9 19 20
4 61 6.3 20 20
5 63 5.9 20 20
6 63 5.7 20 20
7 63 5.6 20 20
8 63 5.4 20 20
9 65 5.3 20 20

RAM usage on decompression
gzip bzip2 lzmash lzmash -e
1 <1 MB 1 MB 1 MB -
2 <1 MB 2 MB 2 MB -
3 <1 MB 2 MB 1 MB 1 MB
4 <1 MB 2 MB 2 MB 2 MB
5 <1 MB 3 MB 3 MB 3 MB
6 <1 MB 3 MB 5 MB 5 MB
7 <1 MB 3 MB 9 MB 9 MB
8 <1 MB 4 MB 17 MB 17 MB
9 <1 MB 4 MB 33 MB 33 MB

http://tukaani.org/lzma/benchmarks.html

What you see here is ZIP being a minimum of 3 times faster than lzma. It also has substantially higher memory footprint. What is also very worthy of note is what happens when you turn of the compression settings for each format. ZIP becomes faster to extract, bzip2 and lzma become slower. Hence ZIP remains the best choice for speed and when the compression is really cranked up with the right settings using 7-zip instead of gzip will approach the other formats while offering much better memory footprint and faster speed.

Hmm... So obviously zip is faster. Anyway it would be interesting to actually try to use it in Xonotic and then compare, but maybe some other time.

blah yes .. but you manged to kill the relevant bits as well -.-

and at the drive total access latency for a operation is normally less then 15MS this is the time the drive takes to seek to the data and assemble the data into the drives own cache
and by aware I mean make the game engine aware of the physical location of the data on the disk low level not really possible but its a theroy anyway
for a random-read compressing a file is SLOWER now if you are talking about a sequential read read its a totally different animal and if you are talking about a SSD ...

Onemoar, just to let you know, I bumped your warning level to 20%. It's only symbolic for now, but bumping it any further will temporarily suspend your posting privileges.

Please cut the arrogance and use punctuation. It's a wonder why people still bother reading your messages.

compressing the data in xonotic-20120514-data.pk3 with 7z with lmza2 on the maximum preset results in ~30MB savings there is more to be had with some fine tuneing
if the engine fully-supported lmza/lmza 2 then you could have you're cake and throw it back at me

A rather overdue update, I went through most of the recrunching steps I listed initially and managed to get down to 915Mb (from 964Mb) losslessly. There's still a few Mb that could be trimmed off I'm sure without even turning to getting rid of sources. I don't really think this work is worthy of a branch in git as the trouble is that most of the changes effect vast numbers of graphics files and trying to get that lot merged (and then maintained as any changes an artist makes would then need recrunching again) would be a nightmare. It would be better to take on the code snippets I've listed in the first post and integrate them into buildscripts along with changes in structure.