[SUGGESTION] Use BitTorrent for hosting maps; or stream them

Why not use BitTorrent for hosting map downloads? On Nexuiz; I've noticed very slow download times for large maps, and there is no good way to self-host map downloads (yes; you can set up a webserver on your own PC, point all downloads to your computer's IP address, and have people download your maps that way, but it slows down the game for everybody else just to have ONE person downloading a map).
Therefore; BitTorrent would reduce the lag time on servers hosting their own maps. Streaming maps would also be a useful alternative. Some games such as GRIDI Simulator stream the game content so that it doesn't have to be downloaded all at once (GRIDI simulator also uses some kind of P2P downloading system. Doesn't appear to be BitTorrent though); reducing the load on the server, and resulting in a much faster load time.

If doable, bittorrent would be awesome to take the load off servers and speed up map downloads. A way of giving back without lifting a finger

I agree with implementing some sort of P2P map & data sharing system in Xonotic. This would be great for improving map download speeds and decreasing server load when multiple clients are downloading. I would think that it is not the most difficult feature to work into the game.

The LibTorrent project might be useful

http://www.rasterbar.com/products/libtorrent/

I've seen some games that are just a 5MB file download, and you just open them and it streams the entire game over some kind of P2P connection. They load pretty fast too, because the textures are loaded only as-needed (so no need to pre-load textures either, but sometimes they take a while to appear in the game and are replaced with a wireframe), and the audio is also streamed; which greatly improves the load time (in Nexuiz, music is often the biggest part of a custom level; in terms of file size).

I don't think that will be good to have.
the server should have big bandwidth to serve these downloads.
let's say I'm playing and someone wants to join, download a map, according to my understanding, it will request me to upload part of the map to him/her, that will cause me to lag.
a solution would be to limit that feature to people who have low ping only, once their ping gets high (maybe above 90), they stop uploading. That way, new players won't annoy current players by making them lag.

the server should have big bandwidth to serve these downloads.

This is entirely what this is preventing.

let's say I'm playing and someone wants to join, download a map, according to my understanding, it will request me to upload part of the map to him/her, that will cause me to lag.

That's why the upload speed could be throttled.

the server should have big bandwidth to serve these downloads.

This is entirely what this is preventing.

let's say I'm playing and someone wants to join, download a map, according to my understanding, it will request me to upload part of the map to him/her, that will cause me to lag.

That's why the upload speed could be throttled.

Totally agree, removing the single point bottleneck is exactly the purpose of p2p.

Upload speed has to allow throttling, of course, and/or torrent entirely disabled (maybe no upload = falling back on http?).
Further improvements could set upload limits based on the user activity: playing, idle / spectating, chatting on a future IRC client...

This is really a good idea.

the server should have big bandwidth to serve these downloads.

This is entirely what this is preventing.

let's say I'm playing and someone wants to join, download a map, according to my understanding, it will request me to upload part of the map to him/her, that will cause me to lag.

That's why the upload speed could be throttled.

Totally agree, removing the single point bottleneck is exactly the purpose of p2p.

Upload speed has to allow throttling, of course, and/or torrent entirely disabled (maybe no upload = falling back on http?).
Further improvements could set upload limits based on the user activity: playing, idle / spectating, chatting on a future IRC client...

This is really a good idea.

Maybe this could be taken even further than downloading, and once someone creates a 'server', the HOST could actually disconnect and the game would continue to go on until all players disconnect from the 'server'. That's kind of the way that GRIDI simulator works (a 'room' is created with rules specified by the room administrator, the administrator (or server) disconnects and the game continues over P2P. If the admin re-connects, they automatically gain back their administrative powers); although I'm not sure if this host management structure would be suitable for Xonotic, but it's an interesting idea (one problem is; the way GRIDI does it, it appears to be dependent upon IPv6 for some reason).

You can be certain map downloads will be even slower with bittorent

I entirely agree! Also if I'm not mistaken, in the settings, there is already an option for selecting your connection speed. That same information can be used for determining max upload, as well as ping data. Optionally, there can just be a check box in the settings, allowing users to decide if they want to help seed or not [left on by default, unless a high ping]. Also, this would be useful for dedicated servers, since they are usually on high(er)-bandwidth connections, and it would make their serving more efficient.

I see no problems, and if people wanted, people could possibly have dedicated seeders? I'd be willing to do that.

I entirely agree! Also if I'm not mistaken, in the settings, there is already an option for selecting your connection speed. That same information can be used for determining max upload, as well as ping data. Optionally, there can just be a check box in the settings, allowing users to decide if they want to help seed or not [left on by default, unless a high ping]. Also, this would be useful for dedicated servers, since they are usually on high(er)-bandwidth connections, and it would make their serving more efficient.

I see no problems, and if people wanted, people could possibly have dedicated seeders? I'd be willing to do that.

That's a really great idea! I second that! I'd be willing to be a dedicated seeder!

You can be certain map downloads will be even slower with bittorent

That's true for small maps, but consider that on the assumption that the seeders aren't constantly overloaded with downloads (so anyone connecting gets an high priority) and the tracker is fast to respond to queries (same single point of failure), torrent sharing is going to be at least as good as regular, old, http.

I like this idea, but it's a bit flawed.

A better method would be to have the map repositories act as webseeds, so there is no real loss in speed. Game clients acting as seeders would upload at a very very miniscule speed(configurable with cvars) so that it does not affect gameplay. This way, we don't give up the speed of HTTP and clients can still contribute.

I wouldn't call that a flaw, that just more goes along with the idea of dedicated seeders (which I volunteered to do in my last post =D) Also I prefer having the options directly available in the settings menu due simply to ease of access (not solely cvars). In the settings menu, it can be adjusted on the fly, making it more ideal for the varying game-types (ie. nexball and race don't require as fast of a network response time because there aren't constantly billions of sprites being created and destroyed, and thus less content created and transmitted on the fly)

A better method would be to have the map repositories act as webseeds, so there is no real loss in speed. Game clients acting as seeders would upload at a very very miniscule speed(configurable with cvars) so that it does not affect gameplay. This way, we don't give up the speed of HTTP and clients can still contribute.

Implementing torrent would allow much more flexibility, of course; with an open tracker anyone with a torrent client always open could contribute with low effort.

Map repos acting as web seeds is a very good idea - didn't know about the web seeding specification. This way even a limited hosting can be set up to help the download of maps.

Also I prefer having the options directly available in the settings menu due simply to ease of access (not solely cvars). In the settings menu, it can be adjusted on the fly, making it more ideal for the varying game-types

Of course having a GUI is way better, but the steps are:

1. Identify the options relative to (lib?)torrent which we want to be customized
   
2. Implement the code with the relative cvars and their ranges
   
3. Design an advanced-yet-simple GUI section for them
   
4. Update the GUI
   

Some sort of easy-to-use GUI is going to be needed, but first we should discuss the options here.

I wouldn't call that a flaw, that just more goes along with the idea of dedicated seeders (which I volunteered to do in my last post =D) Also I prefer having the options directly available in the settings menu due simply to ease of access (not solely cvars). In the settings menu, it can be adjusted on the fly, making it more ideal for the varying game-types (ie. nexball and race don't require as fast of a network response time because there aren't constantly billions of sprites being created and destroyed, and thus less content created and transmitted on the fly)

Uh, what? Those things have nothing to do with transmitting a pk3 file. And the GUI would have the settings, I never said it couldn't be like that. What do you think the other settings in the GUI modify(hint: it starts with a c and ends with var)?

I was responding to you... -_- You said the idea was flawed, but my point was that it's not a flaw, it's a revision. A Web Seed is really just a dedicated seeder, as was afore-mentioned.

The second part threw me when you said, "configurable with cvars"- I know the settings all use cvars, which led me to think that maybe you were saying that it would only be viewable through the config file. I read it along the lines of "I want to brush my teeth with a toothbrush (with a handle)", instead of "I want to brush my teeth with a toothbrush"
It was unnecessary to include that because of the nature of what we're talking about, so I thought you were specifying that it was solely through the config fie, which I disagreed with, and I was explaining why.

And why yes, I am a grammar Nazi. =P

Now what you said to the different game types not being relevant to transferring pk3's, actually they are. Because games like race and nexball aren't as fast-paced nor involve anywhere near as much sprite creation, they also require less data transmission. This means the actual data being sent out is a smaller quantity, and thus you can have a higher seeding bandwidth while playing one of those games, than when you play something like deathmatch, and still have a good ping. That's how it has to do with pk3 transmission-- differing available bandwidth between different games, available for seeding from clients.

Like the idea. As long as their are seeders then map downloads would be faster and there would be less hell on the servers.

Now what you said to the different game types not being relevant to transferring pk3's, actually they are. Because games like race and nexball aren't as fast-paced nor involve anywhere near as much sprite creation, they also require less data transmission. This means the actual data being sent out is a smaller quantity, and thus you can have a higher seeding bandwidth while playing one of those games, than when you play something like deathmatch, and still have a good ping. That's how it has to do with pk3 transmission-- differing available bandwidth between different games, available for seeding from clients.

No, again this is wrong. All maps are inside a pk3(zip) container as precompiled bsp files. You don't transmit sprites, and game speed has nothing to do with file transfer. Bittorrent is a file transmission protocol. All it would replace is the method for downloading files before spawning in the map.

I understand that. But each computer has to know what the other is doing in some way, shape or form. I know the sprites themselves aren't transferred. When one machine creates an instance of a sprite, the other machines must be notified of it (unless dealing with enormous maps in which some elements aren't relevant to the character's position). When more sprites are created by one machine, information regarding that must be transmitted to the other machines (or to a server, and then to the machines) so they know what is going on. The amount of data pushed through is what determines the required minimum connection speed (why you can't play COD4 on dial-up). You're not entirely correct about game speed. If you've ever hosted a game server ie. through nexuiz, you should have noticed that there is a setting to change speed of the game according to the server:
"//sys_ticrate 0.05 // how long a server frame is. 0.05 = 20 fps, 0.02 = 50 fps. Lower settings makes things smoother but create much more traffic (known good values include: 0.05, 0.046875, 0.03125)"
That means game speed does inadvertently affect traffic, more speed, more traffic. Also, if you look at it as being a faster-paced game, it still means more traffic (unless limited by the server or another cvar listed elsewhere in config or server config [default is something like 1 mb/s, I think]). This is because as each end pc creates new sprites or changes (ie. a player moves, shoots, jumps, dies, etc). Data is sent either directly or indirectly to the other players. In a deathmatch game, sprite creation, jumps, deaths, any instructions in general would tend to be sent more frequently to other users "so the left hand knows what the right hand has done". This allows you to actually play the game. This is also why when you compare this game to something like assaultcube or tremulous, this one requires the faster connection, whereas those are both feasible of working over a dial-up connection. They are far less complex, and require much less data transmission in order to know and tell what is happening on their end. If no information was sent other than the pk3 file, you wouldn't have an online game.

And my point was that if you're consuming idle bandwidth to transmit maps, in slower games, you would have more idle bandwidth to "abuse" because again, fewer instructions, points of reference, etc will need to be sent off to other players. If this wasn't true that more action meant more bandwidth, we wouldn't have an option on both the client end, and the server end to limit client connection speed =)

An easy example of what I'm talking about is this:
Grab a friend and tell them to write down everything they see you do for the next 5 minutes. We'll set a base fps rating of 29.967 frames per second. That is supposedly about the speed that blurs still images into moving video, according to the human eye (until you introduce things other bio-chemicals, which can up it to around 60 fps). So your settings are there: 30 fps, transfer speed is how fast he can write down what you're doing, and game speed (amount of action is equal to you jumping). If you jump once, he can easily write down jump close to within the time it takes you to land. Jump twice in a row. This translates to more action. Now you see a widening gap between what you do, and his ability to translate it. Jump 3 times in a row. The gap is even bigger. That's how in-game action and sprite creation affect bandwidth consumption-- the instructions have to be relayed in some way, shape, or form in order for you to play and be aware of each other. And naturally, the more you do, the more you send to them, and the more bandwidth you consume. That's why slower games like race would produce more idle bandwidth than faster ones like deathmatch. You can also get into ping ratios based on signal conversion like with FiO connections, but that's something for later.

And yes, I fully understand that bittorrent would be used only for map transmission, and that you would have to have the models available before starting the maps (unless you prioritize pieces by their files and you introduce the idea of selectively picking out the structural map first, and then downloading and introducing textures mid-game, to enable faster access to gameplay, but through the torrent protocol it wouldn't be a good idea-- streaming in textures would be more http-style).

Um, no.

I think using bittorrent would be great + having the webseeds in the swarm to ensue a group of low upload bandwidth players wouldn't slow down the download.

I think this idea has a lot of potential and I support it. I've always been frustrated whenever I try to begin a match and then half my screen fills up with 20+ map packs/sound packs. *bangs head on table*

Um, no.

+1.

Consider this: Evil player has put an autoexec.cfg into the pk3 that screws up the target config.cfg file completely. Furthermore he has a hacked engine that fakes any CRC checks we might want to request from his client to try and prevent things like this. The malicious pk3 download sure is put into dlcache, but if the target player wants to play this map locally he'll move it out of there into data/. Easy way for an evil _PLAYER_ to simply upload any files he wants to a target pc, huh?

Another option for him would be for instance to put every single texture of the game into the pk3, but make them all black and 1x1 pixels. All of a sudden the target pc will only display blackness for all textures, models when playing on that map (since the map in the pk3 gets loaded, so does the rest of it's contents)

With the biggest problem out of the way: I doubt it's going to be significantly faster either. In order to not cause any lag for the players, the uploads would probably have to be throttled a lot. As upload speeds are generally quite slow even today, that'd lead to very slow speeds from each player. Unless we somehow make the engine immediately queue torrent packets for later transmission if there's a game packet in the queue, of course. And yeah, many users would likely want to opt out as it's not anything unusual that stupid ISPs use monthly bandwidth caps, and playing Xonotic would then be the same as seeding a torrent.

What however could be neat is having a bunch of servers seed these maps, but I think that it would just be easier to redirect users to the nearest server and download via http as normal. These servers would just mirror the maps from a central server, which I could imagine hosting an integrated map upload/rating system on the Xonotic homepage.

Um, no.

+1.

Supporting "tl;dr" behaviour is not helping the discussion.

Consider this: Evil player has put an autoexec.cfg into the pk3 that screws up the target config.cfg file completely. Furthermore he has a hacked engine that fakes any CRC checks we might want to request from his client to try and prevent things like this. The malicious pk3 download sure is put into dlcache, but if the target player wants to play this map locally he'll move it out of there into data/. Easy way for an evil _PLAYER_ to simply upload any files he wants to a target pc, huh?

Another option for him would be for instance to put every single texture of the game into the pk3, but make them all black and 1x1 pixels. All of a sudden the target pc will only display blackness for all textures, models when playing on that map (since the map in the pk3 gets loaded, so does the rest of it's contents)

I don't really get how this could be an added security flaw:

Torrent files have an "announce" section, which specifies the URL of the tracker, and an "info" section, containing (suggested) names for the files, their lengths, the piece length used, and a SHA-1 hash code for each piece, all of which are used by clients to verify the integrity of the data they receive.

The "CRC check" is actually a sha1 hash per segment, still somehow reliable - and the hash is provided by the tracker, of course.
Assuming an evil attacker with enough computing power to create "right hashed" chunks, with more than one people uploading the map there would be at least one "good" piece of data downloaded, which would probably break the zip compression.

The main issue still remains, though, and thanks for bringing that up; maps are too powerful if they can screw up everything - this is indeed a problem which requires investigation.

With the biggest problem out of the way: I doubt it's going to be significantly faster either. In order to not cause any lag for the players, the uploads would probably have to be throttled a lot. As upload speeds are generally quite slow even today, that'd lead to very slow speeds from each player. Unless we somehow make the engine immediately queue torrent packets for later transmission if there's a game packet in the queue, of course. And yeah, many users would likely want to opt out as it's not anything unusual that stupid ISPs use monthly bandwidth caps, and playing Xonotic would then be the same as seeding a torrent.

What however could be neat is having a bunch of servers seed these maps, but I think that it would just be easier to redirect users to the nearest server and download via http as normal. These servers would just mirror the maps from a central server, which I could imagine hosting an integrated map upload/rating system on the Xonotic homepage.

Now that I think about it... you can't download each chunk from more than one seeder, so it's either a huge amount of tiny parts or you have to wait until the slower seeder finishes the upload.

I agree for the player part, a few can afford to play and seed, but I'd like to see a mapserver-only version of it actually, that is servers around the world collaborate to give you a map as fast as possible.
I imagine somebody using bittorrent on a daily basis (outside the game) could also be helpful in seeding, thus reducing the servers' load. The main problem is to have map servers act as web seeds or directly torrent seeders, though.