Porting Xonotic to idTech 4

This is currently just a rumor coming from IRC, and should not be taken as an official statement from the dev team! I've recently heard discussion about porting Xonotic to a new engine, and that moving away from Darkplaces might be considered an option. Given that DP is based on idTech 1, I'm aware that it has or will reach a point where some bottlenecks simply can't be fixed, and modern features are no longer possible to add. Since a new engine implies heavy changes to the game code, a port would obviously be a great effort... but in my opinion one that's totally worth considering at this day. There is however one issue I personally have with the ideas I've heard so far, which is what I'm making this thread to discuss.

I understand that if a port will be carried out, it's meant to be toward an idTech 3 engine. As it's already been explained, this is an obvious option because it would be easier and more compatible: The same model and bsp formats would work (maps wouldn't even need to be recompiled), all important features existent in Darkplaces are supported in a similar fashion, and the game code would require fewer changes therefore making the effort to create this port smaller.

Despite these advantages however, I strongly favor a port to an engine based on idTech 4 instead (the Doom 3 and Quake 4 engine). I have recently been playing a fantastic FOSS stealth game called The Dark Mod, which extensively uses all features this engine has to offer. It has enormous advantages, which no engine based on Quake 1 / 2 / 3 is ever going to implement. I wrote a long list below... jump to the TLDR line if you don't feel like reading the details.

- Realtime map format: Maps no longer need to be compiled into a bsp file using a process like q3map2. You simply click the save button in Radiant and that's it... the engine uses your mapname.map file directly. The only compilation required is a dmap, which generates a mapname.map.prof file needed for portals and path data... but I understand the engine can handle this process automatically.

- Realtime game code interpretation: The code would no longer need to be compiled separately, with programs like fteqcc or gmqcc. The scripting system idTech 4 is interpreted in realtime, and works dynamically like Python or Lua. Among easier development, this makes it simple for anyone to develop and install mutators, by simply writing and sharing their code without anyone having to compile anything.

- Realtime lighting! idTech 4 has no such thing as lightmaps at all, because all light sources are dynamic. Apart from the possibility to move or toggle any light at any time, there are two huge visual advantages: The first is that every light casts a dynamic shadow... and unlike Darkplaces, these do not decrease FPS as much and you can have quite a few at once! The second advantage is that dynamic lights allow for bumpmapped shading... meaning that the normal map of each texture causes shadows and highlights to be much more visible, and you can see light shining and shadowing every little crease or hole in a metal or stone texture. Projectors with image textures are also possible.

- Improved VIS portal system: One remarkable advantage of the new map format is that portals are based on rooms; When the player is in one room, the rooms he doesn't see into won't be computer at all. LordHavoc confirmed that the VIS system in the bsp format is a bottleneck, which is one reason why dynamic lighting is so much slower in comparison to Quake 4's. You can also script various things to happen based on which room the player is in, and have zone based properties using the VIS portals (like ambient lighting which fades as you enter different zones).

- A physics engine: idTech 4 comes with out-of-the-box support for both rigid body physics as well as ragdolls. Darkplaces and Xonotic struggled for years to implement physics or ragdolls with barely any success... whereas here they would simply work!

- Pathfinding: With idTech 4, bot waypointing would be a thing of the past. It supports realtime path finding, and bots will find their way to any location without the mapper having to do anything.

- Advanced entity definition system: Playing DarkMod and taking a quick look in DarkRadiant was enough to see just how advanced and flexible the entity definition system of idTech 4 is; You can create any object as a def file, and script it to do whatever you want. This makes it a cinch to define bots (possibly even payers), weapons, and any form of interactive object to place in the world. An important feature is the ability to attach any entity to any other entity, including light sources to moving objects.

- Interactive surfaces: If you played Doom 3, it's impossible not to remember the computer screens on which the player could click buttons straight from the display. Since Xonotic is a FPS, it's less likely for this to be used in the world... but it still makes things like an ammo display on the weapon model easily doable.

- Volumetric fog using lights: In Darkplaces, fog is defined globally per map... and the most you can do to customize location is set a height beyond which the fog fades away. In idTech 4, the volume of a light can be solidified and even textured, allowing each room to have customizable fog.

- Extra shaders and visual effects: There are some visual enhancements which I haven't seen implemented in any of the idTech 3 forks. One are refractive particles, which can be used to make hot air distort the image behind it. Another are god rays for sky shaders, which allow clouds to cast rays coming from the sun / moon. This is one thing that can be added to any of the older idTech engines, but so far none have done it.

- Reverb and advanced sound effects: From what I read, idTech 4 should support effects such as sound reverb (echoes) for large cathedrals or metal rooms. This does however seem to rely on EAX, which may be a proprietary technology and unavailable for Linux.

TLDR: Engine works directly with map files (no bsp), realtime game code interpretation (no more compilation), fast realtime lights with dynamic shadows and bumpmap lighting, volumetric fog, object and ragdoll physics, pathfinding, interactive surfaces, scriptable object definition system.

I believe anyone can tell why I'm so excited at the idea of Xonotic running on idTech 4, and consider this one of the most awesome things that could happen! However, as this has already been pointed out to me, there are two great obstacles in the way:

- Complete rewrite of the game code: idTech 4 no longer runs QC, but a completely new scripting mechanism. Absolutely all of the game code would have to be rewritten from scratch, while aiming to reproduce the same features and results... which is a huge effort! The good thing is that, thanks to the scripting and entity definition systems being so flexible, there should be a lot less for us to code because many of the mechanics are already there or much easier to connect together.

- Existing maps would no longer be compatible: Every single Xonotic map would have to be modified and recompiled, no backward compatibility. Light sources would have to be configured much more differently. I suspect the netRadiant brush and entity format should be readable to DarkRadiant, if not we can probably write a converter easily.

These two great limitations might outweigh the benefits, depending on what everyone's opinion is. Someone could attempt such a port as a different fork, then turn mainstream Xonotic into it at a later stage. In either case, this change would be greater than even the Nexuiz -> Xonotic transition, and would mean starting all over again.

So who else thinks this great leap is worth it and doable, and wants to see a revolutionary Xonotic 2.0 happen? Or instead supports a port to a Quake 3 engine, although it would mean staying in the era of baked lighting and a rigid code with very limited features. In my opinion, the effort to rewrite all code and existing map incompatibility would be a huge problem, but so are the bottlenecks of the Q3 engines which can't support many modern features and mechanisms... which will eventually leave Xonotic looking ancient in comparison to any modern FPS. I believe this needs to be well thought of, but that both points of view are valid.

I can understand the euphoria caused by the benefits a new engine would bring, but I would vote for discussing this when you can successfully summon a team of developers that would actually want to do this. In any case it would end up (even if it succeeded) with the player base getting split up again, somehow like reaching a "higher level" of a homeopathic dosis.

@MK: Huh. Where and when was there ever any rumor about this? I think you got it all wrong.

EDIT: Also, don't present DP as ancient. You could see it another way too: DarkPlaces is an actively developed modern engine that happens to have been based on Quake 1's source code, and it's still compatible with Q1. But IMHO the comparison stops there. DP extends beyond id Tech 1 in terms of features and API, so much that porting Xonotic to id Tech 1 would be downright impossible.

EDIT2: But to be fair, DP is (currently) limited in terms of the map formats it supports.

I can understand the euphoria caused by the benefits a new engine would bring, but I would vote for discussing this when you can successfully summon a team of developers that would actually want to do this. In any case it would end up (even if it succeeded) with the player base getting split up again, somehow like reaching a "higher level" of a homeopathic dosis.

I didn't get the chance to work with idTech 4 scripting yet, although I'm familiar with QC and assume it's similar in structure. But I think the difficulty of such a port would mostly depend on how complete the builtin functions are and how much the engine can do on its own. I get the impression that the game code would be a lot smaller... because with QC we had to code and define every little detail about everything, whereas idTech 4 lets you define entities and it's those entities that you add scripts to. So if I envision this correctly, re-coding Xonotic from scratch might not be that excruciating with the Q4 engine, and work can be more easily divided across different components. It would still be a lengthy process however, which would take many months to complete, and should be started as a parallel fork so Xonotic can go on as usual until it would be ready.

I don't think such a port should divide the player base however. Since the balance and all assets (models / textures / sounds) will be the same, the primary thing that will visibly change is the lighting. The main issue could be people with hardware that can't handle dynamic lighting, even if shadows are possible to disable.

---

@MK: Huh. Where and when was there ever any rumor about this? I think you got it all wrong.

EDIT: Also, don't present DP as ancient. You could see it another way too: DarkPlaces is an actively developed modern engine that happens to have been based on Quake 1's source code, and it's still compatible with Q1. But IMHO the comparison stops there. DP extends beyond id Tech 1 in terms of features and API, so much that porting Xonotic to id Tech 1 would be downright impossible.

There was some discussion about it, that's all. I understand that other people might be in favor of a port at this stage... but to an idTech 3 engine instead. On top of that, I've already been thinking about the way Xonotic would look like if it could make this leap, and hearing other people thinking about an engine port sort of fueled my thoughts.

Darkplaces is a modern engine with many up to date features. But it too has limitations which will not be possible to surpass... and seeing modern open-source engines (such as Tesseract) helped me understand this better. I spoke with LordHavoc about some on IRC: One of my questions was why realtime lights are so slow in Xonotic compared to Doom 3... which is apparently due to the way bsp maps and the current VIS format work at their very core, which is using an outdated technique.

Another example are outdoor maps: There was a lot of discussion (much of it started by me) about creating Xonotic maps in outdoor environments, typically using patch mesh terrain. The answer was always "Darkplaces can't handle them". idTech 4 instead seems to do a really good job, with performance being very good as well! One reason might be that occlusion culling is also realtime (I think), whereas VIS as we know it is only used for area portals.

The problem is that, even if some Xonotic players aren't interested in cutting edge graphics as much as others (like myself), other FPS projects will evolve. Red Eclipse itself plans on switching engines, and going from Cube 2 to Tesseract precisely for dynamic lighting and other of its abilities. In a few years from now, Xonotic might be one of the few FPS games still held back by such limitations, whereas everything else will look like Unreal Tournament 3 or Bioshock.

just my 2 cents: i love xonotic for its fast and "old" engine which for me is something very positive. i even scale down the fx to make it look more simple. a new engine would not equal progress to me.

There was some discussion about it, that's all.

Could you be any more vague? It seems like this is just a pretext to give some legitimacy to something that only you have been thinking about.

Sounds too good to be true, really. I have howerther played q4 and seen these cool features in action. Interactive screens, advanced interactive scenery, never mind thenevermind stunning lighting with shadows and flashlights. When I was playing it, I was amazed at the level of interactivity with the world, being mostly aware of only older mapping techniques/engines.

I think that porting to a new engine is a step that all open source games would have to make in order to continue, and Xonotic must be ported too, sooner or later. If we do it latter, then we would waste a lot of time struggling with the old engine, and that would result in waste of time. If it is done now or sooner , then we would get the ability to proceed at a faster rate and higher efficiency. Also transfer at a later stage would be harder, as we would grow affection for the old engine, which howether has already happened.

The best thing to do would be to make a branch where the porting progress can be tested.

---

Edit: while I was writing the reply, the number of replies have grown from 1 to many. Some things I said I think were repeated before I said them... Please don't kill me for it.

I think it would be a good idea to have a test go at porting to id4. Port a player model, a gun and a map.... Don't that litraly, but still a quick test of sorts.

Could you be any more vague? It seems like this is just a pretext to give some legitimacy to something that only you have been thinking about.

I spoke about this in PM with one of the project maintainers, who brought it to my attention that the idea of a port has come up and could someday be considered, after I mentioned my thoughts on an idTech 4 port. I don't wish to share private conversations without asking first, and this is probably an irrelevant detail too. I am however the one who came up with the idea of a port to the Quake 4 engine specifically, but was planning to ask about this either way sooner or later.

---

The best thing to do would be to make a branch where the porting progress can be tested.

I think it would be a good idea to have a test go at porting to id4. Port a player model, a gun and a map.... Don't that litraly, but still a quick test of sorts.

That's pretty much the approach I was thinking about as well: Put the engine in a folder, a texture package + a map + a player model + a weapon in a 'data' directory, and construct the first simple script that can use them together. Once this is done and there's a basic environment running, it's a matter of defining all the weapons and players, adapting all of the maps over time, creating the rules for all game types, adding in the mutators, and all other things I might have missed. If an example from each category is up and running (one gun, one player, one map, one gametype, one menu, etc) it's guaranteed that eventually we'll get all others in there too.

There was some discussion about it, that's all.

Could you be any more vague? It seems like this is just a pretext to give some legitimacy to something that only you have been thinking about.

This idea been floating around since before the Xonotic fork; nothing new. The new thing is ppl with the skills and devotion necessary are seriously _considering_ it. I think MK are just using cautious wording not to start any tear avalanche in one way or another given its nothing official at this point.

@topic
While real time maps, interactive environment and inflatable elephant's are all really cool things, what you need to remember is that any porting will be a huge chunk of work.

With a new map format every single map we now have would be useless legacy, and have to be either remade or dropped. Same applies for models, game code and so on.

The engine considered atm, daemon , has enough common features with Darkplaces to make a _port_ feasible. Something IDT4 based would likely be more of a game remake than a port.

While real time maps, interactive environment and inflatable elephant's are all really cool things, what you need to remember is that any porting will be a huge chunk of work.

With a new map format every single map we now have would be useless legacy, and have to be either remade or dropped. Same applies for models, game code and so on.

The engine considered atm, daemon , has enough common features with Darkplaces to make a _port_ feasible. Something IDT4 based would likely be more of a game remake than a port.

I know. Weighting those disadvantages (lots of work to do the port, maps requiring a remake) against the many advantages of idTech 4 is itself a hard thing. But it indeed depends on how many people are willing to work on such a port.

Daemon is indeed the engine I heard is being proposed. I looked at that page earlier, and can't say I'm overly impressed although it has a few advantages over Darkplaces: VoIP might allow Xonotic to support voice chat in multiplayer, while in-engine IRC support is a cool idea as well. In regard to visual enhancements, it has heat haze (particle refraction) which is a feature I've wanted to see for quite some time... finally rocket explosions can cause heat refraction! I also see pathfinding in that list, which is brilliant if we can use it for our bots and remove the old waypoint system! Procedural animation blending also seems to be part of the engine, so it won't have to be done in QC. Lastly, the engine seems to hold more promise of advancement and even more features, whereas Darkplaces has likely stagnated.

What interests me most in terms of possible features for a Q3 based engine, is if the dynamic lights are any faster than the implementation in Darkplaces, so maps could rely on them and get a more modern aspect. Also if large outdoor environments stand any better themselves.

Given XreaL is in daemon's lineage, its probably safe to assume that its both more feature complete and faster render wise. To me the largest advantage seem to be an active development team behind it.

Well, given that Daemon uses QC the same way, I assume the only change Xonotic needs to do in its game code is renaming certain engine functions properly? A few grep commands in data/qcsrc/ followed by a "Replace All" could be all that's needed... or maybe there's more. In either case, there needs to be a list of which functions and properties differ between Daemon and DP, and someone with patience and GIT access to do it easily.

@tZork and MK: Oh okay then. Sorry for doubting you, but that seemed a bit unrealistic to me. I'll stay out of this discussion now

Well i just read a lot of Id Tech 3, 4, 5 and I'm sure id tech 4 sounds a lot more promising that Id Tech 1 but i think it lacks 1 feature that is essential for considering a port:

Multi-threaded processing on the CPU is done for many of its tasks, including rendering, game logic, AI, physics, and sound processing.

This is a feature of Id Tech 5 that is unavailable in Id Tech 4, and i know that Id tech 5 is not (yet) GPL like Id Tech 4 but is a feature that Xonotic really needs, after all even my tablet has 4 cores.

If Id Tech 4 and Id Tech 5 code is compatible i think that's a fantastic idea (as you explained Id Tech 3 and Id Tech 4 are not) or if there is a way to support multiple threads on Id tech 4.

I think the recompiling of maps is the most important issue. Xonotic has less than 25 official maps, but it has more than 1000 community maps. We can't just break compatibility.

A quick glance of some of the most populated servers
http://stats.xonotic.org/server/397
http://stats.xonotic.org/server/484
http://stats.xonotic.org/server/436
Shows that the official maps are definitely not the most played. If the transition happens i'll recompile my favorite maps(extreme_fight, lostspace, boxflip_dark, skybowl, minectf) and i bet some people will do the same with their favorites, but i don't see a way to have them all ported. A script to port the maps is needed to make that viable.

Also if the code for Id Tech 4 and 5 are not compatible(and Adding multi core support to Id tech 4 is not reasonable), i don't think Id Tech 4 to be a good idea, maybe Id Tech 3 is a good idea i don't know.

Well, given that Daemon uses QC the same way ...

As i understand it, it does NOT use qc. So one of the big jobs would be to either implement a QC VM in that engine or port all game code. The latter would be nice given qc is.. well qc. but the former is likely the doable solution.

Anyways, i really don't know all that much about this, just pitching in what i do know to clear some confusion.

I don't see ragdoll physics, beautiful lights or effects an improvement for the game itself really, they are extra detail that nobody has time to stop for to enjoy, Xonotic being a lightning fast FPS game after all. Making the chess table heavier and shinier is not an improvement. Sure some things on the list make modding/mapping easier and that's a valid point. However, from the way people describe the amount of work that porting requires, you might at that point as well just make a new game entirely, no? DP is an environment where players, modders and developers got years of experience on, some like to spend a lot of time with it because it's familiar for them, most of that would change when switching to idTech4 and not everyone is as enthusiastic about learning new things. There will be issues that won't necessarily have answers because it's entirely(mostly) new environment, this applies to both devs and players. DP for Xonotic is great, because it supports configuring the game in way that makes it possible for you to run in on pretty much any kind of specs, even old office computers that don't emphasize on gaming.

Another are god rays for sky shaders, which allow clouds to cast rays coming from the sun / moon. This is one thing that can be added to any of the older idTech engines, but so far none have done it.

Au contraire We've had this for quite a while in CRX.





The main portion of the source resides here if anyone is interested in porting the effect to DP: http://svn.icculus.org/*checkout*/aliena...xt%2Fplain

Clouds don't cast, but that could probably be added pretty easily actually.

I agree with Smilecythe.
We have learned from AAA games that good graphics does not equal good games;
the feel of a game is a lot more important than its looks.

Don't get me wrong, if our main developers decide that a port is the way to go, I wouldn't object to it.

But the latest and greatest ID Tech engine will hardly make the current content look better.
You'll need new content. At which point the only thing you carry over is the established brand.
Which sounds awfully familiar.

There are partial plans to port to Daemon (the engine that powers Unvanquished).
We are still looking into the feasibility of such a move, but the engines are very similar (similar enough that our maps and assets mostly work out of the box, very few changes needed there).

There are a few reasons for porting, the main one being: Daemon is (very) actively maintained.
Our current engine is stagnating, we are unable to push any patches that would alter original Quake 1 gameplay, the maintainer (LordHavoc) barely touches it anymore, and, since the move to SDL2, has become unstable (crashing and random problems almost prevented the 0.8 release, even now the Windows 32-bit build runs slower than it should).
Another reason (besides possible performance improvements) is generated navigation meshes. These remove the need to spam maps with waypoints, allowing bots to be much more advanced without using nearly as much processing power (bots are the largest bottleneck in our gamecode currently).


As for ID Tech 4... We don't have the resources or manpower to start a new game from scratch. Better to focus on the realistic goals.
Daemon is similar enough that we will not be altering the gameplay or anything in a big way, the plan is to make it virtually identical in-game to darkplaces in as many ways as possible.
If we succeed, a preview build will be made available for testing, and depending on community reviews, we will decide whether to abandon or push the port forward.

This move will also let us work more closely with the Unvanquished team, who consider Xonotic a sister project and are supporting us in the port.

My only fear is that Xonotic will be always held back by ancient crap.
qc is legacy and looks scary to me, its kinda practical FWIW but it shouldn't have never been used for any serious computations.
Other things like the way multitexturing is done is just horrifying.

Also, Xonotic is more or less feature complete now and there are actual arena games popping up all over the place.

Xonotic is built on "ancient crap". If you know enough about multi texture to call the current implementation horrifying, a replacement would be much appreciated.

.
qc is legacy and looks scary to me, its kinda practical FWIW but it shouldn't have never been used for any serious computations.

QuakeC is just C with relative entity pointers instead of absolute ones, only one global entity struct, and the only other data type is float. The compiler output on the other hand can be quite scary for some things the underlying VM doesn't support.

What I meant was texture blending and now I realized quake engines basically don't do that without relying on specialized models and shaders.

@TimePath thx for an explanation but I'm not gonna learn something so exotic.

This move will also let us work more closely with the Unvanquished team, who consider Xonotic a sister project and are supporting us in the port.

Isn't Unvanquished in the process of moving away from Daemon? See:
https://www.unvanquished.net/?p=155
http://www.phoronix.com/scan.php?page=ne...px=MTQ2MjM

EDIT: Okay, those two news items aren't clear. I understood it as if they used to use an engine called Daemon, then chose to rewrite it completely and not yet give it a name. But from this thread I gather that they are the maintainers of Daemon and are rewriting it under the same name.

From reading their wiki it seems the use C89 for client code, is this compatible with QC? Also they seem to transition their client code to C++ anyway, but it could aswell still support C89 I don't know.