[SUGGESTION] Use BitTorrent for hosting maps; or stream them

Sounds like a good idea; but how would we implement it? I've seen a few protocols which seamlessly run load-balancing systems on servers in different buildings, but in order to implement such a load-balancing system; it would be necessary to have direct access to the VMs on the servers; which would cost a lot of money. The main problem is finding a cost-effective (and a fast) way to distribute maps.

I think this bittorrent distribution is a terrible idea for the following reasons:
1) As mentioned previously, a peer could attempt to distribute bad data.
2) It requires that the server act as a tracker, which would put a lot of extra load on the server or the use of a public tracker which may have a less than 100 percent uptime.
3) It gives each client the IP address of all the other clients. This could be a security issue.
4) It requires clients to reconfigure their firewall and possibly set up port forwarding.
5) Many ISPs have hardware that actively finds and intentionally cripples bittorrent transfers.
6) Many routers designed for home use have trouble multithreading the number of connections that are used with bittorrent.

I propose the following:
1) Implement a trivial file transfer protocol (TFTP) server within or external to the game server, modified to use either zlib or lzop compression and implement CRC checking.
2) Implement a server cvar to limit the amount of bandwidth consumed by clients downloading content.
3) Replace pk3(zip) files with 7zip files for better compression. Perhaps call these files pk7 files.

I think this bittorrent distribution is a terrible idea for the following reasons:
1) As mentioned previously, a peer could attempt to distribute bad data.

BitTorrent has checksums of each data chunk. Meaning it's not really possible to poison a torrent as long as the client checks them.

3) It gives each client the IP address of all the other clients. This could be a security issue.

Happens everyday with normal torrents, yet you don't see ppl whining that someone cracked their box cause they saw his ip.

4) It requires clients to reconfigure their firewall and possibly set up port forwarding.

BitTorrent can work behind NAT.

5) Many ISPs have hardware that actively finds and intentionally cripples bittorrent transfers.

Protocol encryption bypasses most of these filters easily.

3) Replace pk3(zip) files with 7zip files for better compression. Perhaps call these files pk7 files.

Now this could be done regardless of the map distribution model ;-). 7zip/tar.xz has a great compression ratio and is fast to decompress.

I like this discussion, because Xonotic is bringing with it an increase in map sizes and that is a major problem for anyone serving maps.
I run HOCTF and HODM servers and, when activity is high, I'm using up over 600GB/month in bandwidth, with Nexuiz maps averaging somewhere around 6-7MB in size.
I get the impression that Xonotic maps are going to be more like 20-30MB in size. That is a problem.

Here is what I fear would happen with using bittorrent for maps:
If I setup a map repository today, I am serving ONLY the players on my servers. I know approximately how many that is, and I'm limited to only the servers that I am running.

If I have a map repository that is a seed for map torrents globally, then I am now serving every player on the planet (to some degree), and I don't know the limit of that. I don't think many server owners would like to opt into that unless there is a large pool of seeding.

There could be a throttle, but it would have to be selective throttling, as I don't want to throttle bandwidth to players on my own servers. I want their map downloads to be fast. I would want to use bittorrent to offset some of my bandwidth usage, but not make map downloading slower to the players on my servers.

I also test maps a lot on my servers. So I would need to have the option to do "direct" map downloads as well.

In the end, I don't know if a bittorrent implementation would offset enough bandwidth and I would be right where I started, or possibly worse.

There are a lot of things to consider with this idea. Though, I do appreciate the discussion being held.

Would it be possible to create a map pack that gets updated monthly and distributed via the Xonotic website, or would this be a software licensing nightmare with a jumble of incompatible licenses?

The question is; who would have the ability to add maps to the Xonotic website? In Nexuiz; there was sort of a central repository, but only a few people could write to it.

That's hardly a problem, have the map coordinator and the core team have access.

To recap, some open and closed questions:

• zip/7z: remains to investigate
  • The statistics of compression ratio (for a bunch of regular pk3). How much would we save switching to 7z?
    
  • The GPLity and possible inclusion in our GPLv2 with no hassle.
    
  • The 7z library and implementation issues.
    
• Implementation:
  • Has anybody used libtorrent? Should we use something else?
    
  • Are there any implementation issues?
    
  • What will the user configure and what not?
    
• Tracker/privacy issue: how can we setup trackers?
  • Would it be possible to counter Dokujisan issue with a private tracker architecture?
    
  • Can we include a mechanism to start a tracker inside the Xonotic package for servers?
    
• Who will seed?
  • How should the server admin start seeding, and how?
    
  • Shall we allow regular bt clients to download and share?
    
  • Can we turn regular map repos in web seeders? What are the requirements?
    
  • Should the players seed? How and how much?
    
• Speed:
  • What will be the typical map download size?
    
  • Can we speculate and do some tests on the download speed with the old vs. new architecture? What about server balance?
    
  
  

Link to 7zip license: http://www.7-zip.org/license.txt
As long as the unRAR portion of the source isn't copied/used it appears to be LGPL.

The clients should definitely seed. What is the point of implementing a bittorrent map download system if the clients don't seed?

I think a tracker implemented in the server would be best. If your server's public tracker goes down, clients won't be able to download maps unless there is a fall back to the traditional Nexuiz style map download system.

As for Dokujisan's concerns, libtorrent could be modified in such a way as to be passed a list of client IP addresses and only accept connections from those IP addresses. BitTornado contains some logic that blocks BitComet clients, perhaps digging into BitTornado's source would prove fruitful.

I'll post some statistics this evening on file sizes by recompressing some pk3 files from Nexuiz and some custom maps that I downloaded from various servers.

Command line showing 7zip parameters used:
7z a -m0=lzma2 -mx=9 -ms=off -mfb=128 -md=64m ./map-ctf-q3wpak4.7z ./map-ctf-q3wpak4.pk3_FILES/*

chickenteam_ctf_mappack.pk3
before: 24.9 MB (26067260 bytes)
after: 21.1 MB (22163371 bytes)

data20091001.pk3
before: 846.5 MB (887580846 bytes)
after: 720.1 MB (755075355 bytes)

desertcastles_mod.pk3
before: 21.1 MB (22082882 bytes)
after: 16.9 MB (17740981 bytes)

frankishtower14.pk3
before: 22.4 MB (23533706 bytes)
after: 20.4 MB (21417259 bytes)

greatwall_overloaded.pk3
before: 22.5 MB (23571140 bytes)
after: 18.4 MB (19314471 bytes)

imogen_beta.pk3
before: 27.2 MB (28480734 bytes)
after: 24.6 MB (25772192 bytes)

map-ctf-q3wpak4.pk3
before: 40.0 MB (41990613 bytes)
after: 32.8 MB (34381140 bytes)

Part of 7z is based on lzma, which is public domain. I had a discussion with divVerent about compression but I forgot the conclusions, and I don't have any logs. I'll point him to this post.

...Increasing map sizes: This is why we should get in the new texturepacks NOW before the release, so that a texturepack is distributed with the game, and not so that each map pk3 comes with their 20 textures all with gloss/bump/normalmaps.

Sooo, lzma is too slow to be used, and the format is more complex than zip...


Also, +10000 to FruitieX

Yes, shipping skyboxes and textures with xonotic will make compression not so useful.

BitTorrent had not seen a major upgrade in two years, but that was before the acquisition of uTorrent. BT have access to all that was missing. I have used it for hosting maps and the worked perfectly to what I was looking.