[META] DarkPlaces fork

Whatever.

slow and painless

Oxymoron?

I support bringing the game to better technologies, but with my very limited knowledge of the engine, I don't know how much work it is, let alone actually contribute to the engine. I can perhaps contribute to the game code itself.

Is it possible that you or somebody layout the steps for this migration? Right now my understanding is very rough: 

1. port the engine
2. port the game code
3. profit

then why not remake everything then? https://godotengine.org/

then why not remake everything then? https://godotengine.org/

This actually sounds intriguing, even though you might be joking. Curious -- how would the performance of Godot be for a fast-paced FPS like Xonotic?



This is not the first time Lyberta voiced this idea. The fact that we haven't jumped on board already sort of shows how feasible the devs think it is, mostly because we simply don't have enough active hands on deck for such a daunting task. What's our projected timeline of completion? 5 years? 10 years? Considering our current pace, even 10 years might be a conservative estimate.

As a player who's likely to stick with this game for a long time, I don't mind the 10-year timeline if that means the game will keep on living, and new devs will have an easier time contributing.

However, do we, as a (dev) community, have enough commitment/resources to carry out this long-term project? Or is it easier to stick with Quake and its eco-system?

Another thing to think about -- remember we had to reject Nexuiz clients? That means unless the new port is really equivalent to the existing one, we are risking having to start a new community from scratch.

Here's the plan:

1. Start by forking the engine and compiling it as C++. 
   

How hard is this alone? Is it trivial to do a proof-of-concept working demo that can produce a drop-in replacement for the current binary?

then why not remake everything then? https://godotengine.org/

Godot uses GDScript for game code which is unacceptable and not gonna happen given current developer resources. This means rewriting everything in 1 go and not having playable builds. My proposal keeps the game playable during transition.

Good point.

However, do we, as a (dev) community, have enough commitment/resources to carry out this long-term project? Or is it easier to stick with Quake and its eco-system?

QuakeC is a dead end. And Xonotic QC code is riddled with macros. It requires developer who knows plain C well to understand what is going on. And C devs are rare these days.

I personally have way less experience in C++ than in C (but agree that C++'s design is better), so I'm not sure whether it's true that there are fewer C devs.

Given that about 90% of valid C is also valid C++, here are my estimates for 1 developer working 4 hours per day:

• Optimistic - within a day. DarkPlaces is written in clean C and is also a valid C++. We just change the build system and flip a switch to compile everything as C++.
  
• Realistic - within a week. There is some C insanity, it requires manual tweaks to build as C++.
  
• Pessimistic - more than a month. There is plenty of C insanity in the engine. Blood, sweat and tears ensue.
  

This is something I wanted to do myself because I plan to fork Xonotic into a military shooter but I've found that if I to fork DarkPlaces for my own purposes, I'd remove a lot of cruft as a very first step, maybe too much for what Xonotic devs would like. And I want to share as much code with Xonotic since there are not many FOSS shooters and I want everyone to benefit from my work.

Doesn't it make sense that you take the first stab and figure out how easy/difficult it is to port DarkPlaces, since you have a stake in it anyway?

I'm curious what components are considered "cruft" that you don't need in your military shooter game. Don't you want as much compatibility with other games so you can share resources?

One question though -- does it mean after this fork, it'll be very difficult for you to merge back DarkPlace's future new features (not sure how likely anyway)?

Feasibility of the engine rewrite aside: have you looked at unvanquished? They have an engine in C++ (based on the q3 engine which was in C so rewritten already and it's newer tech, although not strictly superior feature-wise for now) and they already seem to have gameplay much closer to a realistic military shooter written in C++ as well so you could share much more code with them.

The next versions Android and IOS will probably improve mouse and keyboard support. Apple will probably be a bit reluctant as they don't want to destroy their macbook market by making IOS too powerful, but Google has no need to hold back. The ports are not about playing Xonotic "on the go", but about having a port for the time when many people only have a single (portable) device and a docking station.
Imo, forking DP and customizing that fork only for Xonotic is a bad idea. The UNV devs also struggle to maintain their engine by themselves, hence they started to cooperate with us (and I think with the Smoking Guns dev(s)).

Imo, forking DP and customizing that fork only for Xonotic is a bad idea. The UNV devs also struggle to maintain their engine by themselves, hence they started to cooperate with us (and I think with the Smoking Guns dev(s)).

If I understand well, the only interest in using DarkPlaces over Daemon right now is that we can rewrite it in C++properly overtime, without having "a big update" that potentially breaks everything. Is that right?

However, if Xonotic is ultimately ported to Daemon, all this rewrite effort will go to nothing. So yes, I agree with Freddy here, why not trying to have resources in common with other similar projects from the very beginning?

Your new "gaming laptop" gets less fps than my most ancient potato, so you blame the tools and propose a complete rewrite of everything? I have a better proposal: fix your laptop.

If Quake C was as slow as you say, how can I get a cpu limited fps in the 200 to 1100 range (depending on map) on an intel Q6600 from 2007?

Edit: And I have a somewhat broken 2013 laptop with intel integrated haswell graphics, and it gets a (gpu limited) 250 fps on most maps at decent settings.

Your new "gaming laptop" gets less fps than my most ancient potato, so you blame the tools and propose a complete rewrite of everything? I have a better proposal: fix your laptop.

If Quake C was as slow as you say, how can I get a cpu limited fps in the 200 to 1100 range (depending on map) on an intel Q6600 from 2007?

Edit: And I have a somewhat broken 2013 laptop with intel integrated haswell graphics, and it gets a (gpu limited) 250 fps on most maps at decent settings.

Well he is right about the "Cache" issue. My Sandybridge has 6MB Cache. When i join a CA or a CTF Game i can get huge drops which make the game just disgusting to play sometimes.   I tested in on several similar systems with pretty much the same results and the few buddies i convinced to try the game reported similar issues and just turned xon their back because they didn't want to deal with it. I can honestly say this game never ran really perfectly smooth (at least on Windows) on my end. I mean yeah i can reach 250+fps but i also get nasty drops to the 60s that feel much worse than 60fps. It's especially frustrating when you compare that to games like Quake Champions, Unreal Tournament 4 or other AFPS which sometimes look much better and also perform much better.
I love xon and i learned to "live with its flaws" but it is a tough love sometimes.
But i guess you are right the only thing i can do might be upgrading my system or try it on Linux.

I find Xon runs smooth on computers far too old and underpowered to handle QC or UT 4, but there are situations where performance suffers.

The BSP format makes it hard work for mappers to achieve good performance and high detail in large and open maps, especially when reflection and warpzone effects multiply the amount of visible triangles. BSP was designed for boxy rooms linked by corridors that block visibility, as found in DOOM and Quake.

You can see the limitations of BSP in other games: when Valve added blacksite to CS:GO, it was much smaller than battle royale maps in other games, and its layout blocks visibilty as much as possible.  Despite these tradeoffs and a well paid, expert team to optimise, it still performs much worse than traditional (smaller, less open, BSP friendly) CS maps.

Many Xon servers run maps ported from other games such as Q3 and QL. Xon often allows water reflections on these maps, but Q3 and QL don't have fancy water and their maps were not optimised for it. The result can be a glitchy 30fps on a system that gets 300+fps on the same map with r_water 0.

There are also maps with excessive amounts of realtime lights, so on a normal map you think your PC can handle realtime lighting, then you go play one of these maps, and it's a slideshow.

Rewriting everything will do nothing to prevent map issues destroying performance.

Some weapon and damage effects are also quite slow, using too many API calls and causing the CPU, GPU and libGL to wait for each other many times per frame. The impact of this varies significantly between GPU drivers, independently of GPU power. Fortunately Xon has thousands of cvars, allowing you to reduce or disable effects as needed. Effects efficiency can be improved (contributions welcome), but rewriting in a different programming language won't help, and will greatly increase the work required.

Xon is currently using an OpenGL 2.1 renderer. Some (simpler) Quake based engines were able to improve performance by rendering with the modern Vulkan API instead. Contributions welcome...

I know that rewriting everything will be a big pain. I hope you've created the fork and copies of Xonotic project(gamecode, engine, resources, ...). I don't know if the team learders have decided this because of your idea:

First of all, they will have to divide the developer groups: one group of developers for Lyberta idea (rewriting code, searching ways to improve the things, tests with the engine, modifications, ...) and other group of developers for standing Xonotic game from now for the players (do things as much as they can, fix, enhance, add new things in the game, ... using the QuakeC programming). I said one group for QuakeC, because we will need to maintain the community and the players whose are enjoying with it.
Without this group of QuakeC that maintains the game, maybe the community won't attract the players and some of them would leave. If the fork is successful and your idea works well and is stable, then some of the group of QuakeC may move from one project to another. Some players will play the game with this engine.

Excuse me, Lyberta, I don't know if the questions that I'm doing are repeated. 
So if the project is successful and QuakeC is removed... This means that to develop gamecode we will use C++ instead QuakeC? As well as the commands, cvars, interface, mods, game modes, scoring systems, graphics settings, ... And the config / script files (.cfg) will be removed too? And won't we use scripts to create the custom configurations? I think that the scripts helped us a lot to setup and manage the servers.

Excuse me, Lyberta, I don't know if the questions that I'm doing are repeated. 
So if the project is successful and QuakeC is removed... This means that to develop gamecode we will use C++ instead QuakeC? As well as the commands, cvars, interface, mods, game modes, scoring systems, graphics settings, ... And the config / script files (.cfg) will be removed too? And won't we use scripts to create the custom configurations? I think that the scripts helped us a lot to setup and manage the servers.

Yes, QuakeC would be removed. Gamecode would either be written in C++ or Rust. With WASM it should even be possible to mix multiple languages, but that would lead to fragmentation and bugs because of inconsistent "expected behavior".
From the user perspective, not much would change (commands, cvars, etc.). Maybe there would be minor changes.
Even moving to Daemon would not cause completely new behavior.

Yes, QuakeC would be removed. Gamecode would either be written in C++ or Rust. With WASM it should even be possible to mix multiple languages, but that would lead to fragmentation and bugs because of inconsistent "expected behavior".
From the user perspective, not much would change (commands, cvars, etc.). Maybe there would be minor changes.
Even moving to Daemon would not cause completely new behavior.

Ok, Rust can be a great option, it can reduce the performance, security and memory management issues. It will make debugging less difficult. Why not?

Xonotic/Nexuiz gets excellent performance for the use case of playing a small 1-on-1 or at most 2-on-2 game on a small performance optimized map. I get over 60 frames a second (at 1280x800, low game settings) playing one bot on the seven maps I play [1] using an Intel Core 2 T8100 with Mobile Intel 865 express chipset (a 2007 era business laptop). The only time I get under 60 frames a second is when looking down at Aerowalk’s or Final Rage’s fairly open large vertical room; even here performance is 50 frames a second or higher. On my 2017 Intel Core i7 7600U with Intel HD 620 video chipeset (again, a basic video card for business non-gaming systems), I get 200-500 frames per second on medium settings, even with two or three bots.

Glancing at Lyberta’s postings, I do not see at first glance any reference to profiling tools being used to conclude that it is the QuakeC which is killing Xonotic’s performance. What I saw, back when I tried to run Xonotic on my old Intel N455 netbook (Intel GMA 3150 graphics), is that the maps were more detailed than the original Nexuiz maps, causing unacceptable performance on my then main computer.

I think Bones hits the nail on the head: Xonotic’s use case is for small, tight, hand-optimized maps, playing one-on-one games against someone on a home LAN.

Now, I think QuakeC’s real issue is the fact that it’s an unusual specialized programming language with limited documentation and a relatively small number of programmers who are comfortable with the language. If Xonotic were to be made today, it would probably use Lua, which is, last time I looked, the standard game scripting language which a lot of game developers are familiar with.

However, in the real world, to move away from QuakeC would require a good portion of the game’s content to be rewritten; the resulting game would probably be a good game, but it probably should not be called “Xonotic”, because the rewrite would inevitably change the fundamental game mechanics. Since we’re dealing with Open Source Economics (no one is getting paid to do this work), finding developers motivated enough to do the work is hard, even with the COVID-19 recession leaving developers out of work with lots of free time.

[1] For what it is worth: Nexuiz 1.1 Downer, Blood Prison, Nexuiz 2.3 Final Rage, Toxic, Soylent, the Xonotic map Graphite which I have converted in to Nexuiz format, and the originally for Quake3 Hub3 Aerowalk map.

Has the project to rewrite all the code in C++ started? Will we use the Daemon engine? If this goes well, will it also work for older computers with minimum requirements of 1 gb of RAM, 2 ghz Pentium ...? What is there for developers to do right now, do this, or continue the previous way (QuakeC, using DarkPlaces engine)?

That's a lot of question. For what I know:

• Has the project to rewrite all the code in C++ started:
  I've seen no public sign about that and enough will to wait for a start.
  
• Will we use the Daemon engine:
  Only If people work on it.
  
• If this goes well, will it also work for older computers […]:
  I don't see why c++ would raise the limit, and for Dæmon, that would be the usual requirement.
  Note: 1GB is very low, at some point people with 15 years old PC would face issues to run their operating system before trying to play a game.
  
• What is there for developers to do right now:
  Developers prepare next release. Xonotic 0.8.2 is three years old, releasing something new that works is the priority.
  

Note that anything things about “rewriting” or “port to another engine” are mostly personnal experiments at this point.

Then after the release of the new version and having done the TOS and GDPR things, maybe it's possible continue this topic and other ideas, if this gonna well.

Very related article.

The issue is this: Open Source Economics. Which means two things:

• Software will be maintained and updated for far longer than a typical commercial game
  
• Integrating new technologies is a lot slower
  

In order to integrate new technologies, such as new versions of the C++ standard, requires a non-trivial amount of developer effort (especially since Dark Places is written in C, not C++), and since no one is getting paid to do this effort, changes like this will be quite slow. Sure, if we could get some venture capital (say a million dollars) and hire a team of developers, we could get a game written using the latest C++ standard and whatever scripting language we want out the door within a year.

However, this is open source. We don’t have venture capital. We don’t have a million dollars to hire developers. We just have a lot of enthusiastic volunteers making a really great game, but since Xonotic is already a really great game, there is relatively little motivation to rewrite it using the “latest and greatest” technology.

I'm a (new) developer for Darkplaces so I feel I can speak somewhat authoritatively on this matter.

1. Start by forking the engine and compiling it as C++. We can then slowly rewrite each individual part in C++. That way every development build can be fully playable.
   

The engine already compiles as perfectly valid, warning-free C++. You don't have to rewrite anything. Darkplaces adheres to what I like to call a "hidden standard" of C that sits in the middle of a Venn diagram of compatibility between C and C++. It's ISO C minus a few C-isms that C++ doesn't like

Modify the engine so it can operate with menu and server components written in C++. First step can be as simple as adding the ability to call runtime created QuakeC "builtins" via meta builtin. So the call chain can be this: engine -> old QuakeC blob (now acting more as a bridge) -> new C++ code. This way we can slowly start rewriting menu and server code in C++ while still having fully playable builds.

I'm not sure about this. I'd prefer exporting (within reason) all of the QuakeC builtins so they can be called from native code. This has been in the cards for months and it's something I want to do, but I can't give an ETA. What this could mean is potential Quake 2 support, and QuakeC would essentially become a language binding among potential many (as such an interface would make implementing other languages much easier).

Somewhere along the lines we can drop Quake 1 client code as it is not used by Xonotic and acts as a dead weight.

I'm all for separating the Quake 1 client (and server) code from the engine, but removing Quake support altogether will never happen upstream. Darkplaces is a *Quake* engine. A better solution would be to reimplement Quake on top of the native API I mentioned above, with a bunch of hooks where Quake 1 code used to reside. That said, there's far too much generic code referencing Quake code. Untethering it would take an ungodly amount of effort.

I don't think Xonotic is likely to move away from QC any time soon, but this hypothetical native API I have in mind already would allow you to achieve the rest of the bullet points (of which I only addressed the first three). It's for these reasons I would strongly encourage collaboration with upstream, rather than forking the engine. Quake 1 doesn't have to be sacrificed.

then why not remake everything then? https://godotengine.org/

That was probably a joke, but if someone takes it seriously:
It ends up being rather pointless to port the game to something like Godot as it's basically making an entirely different game.
*hem* Liblast
Also I can't play that on my PC because it's too laggy and slow.
The focus should be improving Darkplaces slowly, as that kind of reset would basically destroy almost 20 years of work...
And it would probably kill the community too, again, it's like making an entirely new game.
Oh, and also, Darkplaces is mostly fine as it is... go add some new graphics things if you want though, idk.

then why not remake everything then? https://godotengine.org/

That was probably a joke, but if someone takes it seriously:
It ends up being rather pointless to port the game to something like Godot as it's basically making an entirely different game.
*hem* Liblast
Also I can't play that on my PC because it's too laggy and slow.
The focus should be improving Darkplaces slowly, as that kind of reset would basically destroy almost 20 years of work...
And it would probably kill the community too, again, it's like making an entirely new game.
Oh, and also, Darkplaces is mostly fine as it is... go add some new graphics things if you want though, idk.

I think that is just an extremely pessimistic perspective and it doesn't at all recognize the benefits to community and future of Xonotic, albeit there might also be downsides in this direction as you mentioned. It is a difficult question and difficult decision, with many upsides and downsides - and I think clearly the benefits are outweighing the problems in the end.

Porting the game precisely wouldn't be making an entirely different game. It would be re-making the same game in a different engine, so that it looks the same, feels the same and has as many features as you can reasonably copy by hand to fit most people's needs. If this cannot be done satisfactorily (due to e.g. shortage of manhours), such that it appeals to the community, then it should never be attempted at all. The very premise of making a port is to replicate the very same player experience, true to the game for it to live on in the sphere of modern gaming and to have the community test and approve of it.

Liblast is a great attempt with good media attention, but I think a bad example even as work in progress. Other multi-player shooters have been done in Godot 3, that have cloned perfect and trialed networking code from Id and Valve games, and the code is much more efficient and the quality is much better also. Liblast is made in Godot 4, which is still early "beta" (beta as in: no more radical new changes are made) and this produces performance issues. Though you can't say at this point there will be a 2x, 5x or 10x improvement - it will certainly substantially change when the beta is over, even when assuming that Liblast code quality rather is the source of it.  That isn't to say Godot won't be slower overall, but due to QuakeC inefficiency, you can bet that there will also be considerable performance gains in certain aspects (e.g. with bots >10x gains are realistic).

From the community side, the biggest issue is that most or all of the current dev team will probably not switch to a new engine, so this will basically "destroy" the current dev community which is very unworkable and inacceptable. On the other hand, assuming everyone was simply well-minded of the idea, then Godot would be a HUGE advantage because it would connect Xonotic to the wider Godot dev community. Currently there are just a hand full of people able to work with QuakeC, worse than how there are no COBOL programmers. But with Godot there are thousands, maybe tens of thousands of people who know it and can code GDscript and have already coded Counter Strike clones, etc. with plenty of experience and other code as reference. Everything is much simpler and everything is easier and more intuitive with the UI, the barrier is much lower, like with Python vs. LISP or Jupyter Notebook vs. LaTeX. So from a dev perspective, switching to Godot is well-worth the investment on the long run.

Darkplaces comes from the DOS era, and in many ways, metaphorically speaking Xonotic is running still on DOS with it. At some point it should make the switch to Windows 10, it needs little explaining why. If it continues to run on old tech, it creates issues, like increasing programming overhead, more bugs due to engine limitations, and grave shortcomings in the new features you can realistically implement. Also it is important to remember that QuakeC has been abandoned by Id Software for the performance issues, which is as I understand it, the rock-bottom blockade that prevents new mods and desirable features from being developed and it also drains huge amounts of CPU when there are dozens of players in a match.

With a new engine code complexity would dramatically decrease and freedom to implement new things will dramatically increase.

From this point on the code can be adapted to the next and next version of Godot, and Xonotic will profit from the new features and improvements hence forth.

It is true that a lot of work will be lost, but this is largely because it concerns features that already ship with Godot, or features that are not commonly used by many people. And Xonotic is extremely bulky in such rarely used features.

Some work will be lost entirely though and compromises have to be made. Mods will probably never return, maybe even not all game modes will be copied over. Such decisions are tough choices, but like Mozilla did a complete rewrite of Firefox 2017, which broke many old and long-loved addons such as Tabmix Plus and Gopher support, it was still the right choice. They were mindful of the future and they bit the bullet and did what was best to keep Firefox alive and popular, rather than just a relict and retro experience, something stuck in time but not up to the market. I used Palemoon (the Firefox Classic clone) for almost 5 years, and shamed Mozilla for the Quantum release all the time, until I finally had to admit that I was in error all this time, trying to just stick to what I am used to and what worked for me in the past while ignoring the future path ahead. And of course the new Firefox got better with time too, old features came back, bugs disappeared and everything just improved.

Porting Xonotic to Godot is maybe a lot of work in the beginning that maybe no one really has time to invest. But all things considered and overly sceptical newcomers aside, I think it is fair to say that this is just the switch from DOS to Windows, or the switch from PPC to x86. 

Will Xonotic be on the frontline of free software gaming, competing with modern games and appealing to the player base on the current gaming market? 

Or will Xonotic be a retro game like Quake 2, a game that is on the grand scale frozen in time and has a niche playerbase that grows older and smaller every year?


I don't think there is anything wrong with either of those pathways. Certainly though you can't have them both or that the former can be accomplished with the old engine.


You can read more about the current situation with Godot in this post I made.


Like you can see there are shortcomings, but the overall situation is still very profitable.

Porting Xonotic to Godot is maybe a lot of work in the beginning that maybe no one really has time to invest.

Good summary. I made one small correction.

It is less work though than any other option. Like people here were floating the idea to port to other Id-tech variants or moving away from QuakeC to C++ or something.

Looking at how the game is implemented right now, it really might sound like a lot of work. But actually, just making a networked shooter is child's play in Godot.

The only devil is in the details and making it very very close to original.

To get to a workable intermediate result, if a couple of people just jumped in on the idea, with maybe 2-4 being serious about it, there wouldn't be a shortage of manhours.