q3map2 alternatives for NetRadiant

I've been talking with Div today about how slow compiling with q3map2 is. Anyone who mapped for Xonotic likely knows the pain of compiling a map with maximum settings, even a small one (generally, it takes hours with CPU used to 100%). It seems q3map2 was not optimized well when ID software made it, and it would be difficult for anyone to improve its code at this point.

I heard that LordHavoc is working on an implementation that will compile lightning Xonotic-side. As I said on IRC, I personally disagree with such a change because I don't believe a mapper should have to use many editors, and compile half of the map in NetRadiant and the other half in Xonotic (already happens with rlights and waypoints, but at least they are optional). However, if that part of the engine can be made a separate exe file instead (that works like q3map2) and included with NetRadiant, I completely agree with that change, so we can stick to only one mapping / compiling utility.

Apart from that, I managed to find two alternatives to q3map2 that could be used with NetRadiant. Both seem to be discontinued, but if they are faster than q3map2 they could be adapted to Xonotic and used for NetRadiant instead. I didn't test either of them with Xonotic yet.

http://map3bspc.sourceforge.net/ - MAP3BSPC: Seems to be discontinued since 2003, but I seen many old posts recommending it as a q3map2 alternative. Source code seems to be available too, so in case it could be re-continued for Xonotic that would be an option.

http://sourceforge.net/projects/rmapc/ - BSP Map Compiler: This seems most promising to me. It appears to be discontinued since 2008. Only issue is that it doesn't come with a Linux version, but that can probably be fixed with some new code. I gave it a quick test and it compiled a map very quickly, although I didn't try it Xonotic (just watched it generate the bsp file). This seems the best option from the two to me.

What do you think? Is q3map2 the best thing to stay with, even if it's so very slow and hard on resources? Or should we wait for LH's in-engine compiler... or would one of these two q3map2 replacements do for NetRadiant (my vote is on this at the moment)?

Well, a compiling step that's taking its time is the calculation of the lightmaps (esp. with bounce), and the results are not too impressive.

From a quality (and efficiency) point of view, it would be logical to use the best open source program to do the work, i.e. Blender. If it in some way would be possible to calculate the lightmaps with Blender (using Blender light entities), I would expect some impressive results. I then wouldn't even mind working with 2 programs.

If it in some way would be possible to calculate the lightmaps with Blender (using Blender light entities), I would expect some impressive results. I then wouldn't even mind working with 2 programs.

For example using Yafray or Sunflow for boucing light rendering or even baking the Blender's Radiosity.

I'am not sure but Blender 2.5 seems to come with some nice indirect lighting rendering. We might also use some render farming like BURP for our purposes then. That would speed up whole process.

Sounds interesting, for an optional solution imo. I was thinking of something that would work with NetRadiant, so again, mappers wouldn't be forced to use more than one mapping or compiling tool unless they want to. So I think a blender script would be something separate, but the idea does sound nice

At present I have a powerful enough machine to just hammer maps through in q3map2 so I can't say I'm hugely bothered. Doesn't take long for me to compile even at the highest settings.

What would be nice would be a distributed compiler. A q3map2(at this point it would probably be q3map3 with such a drastic change) server would run on a machine, and when a q3map2 client is run(on the machine which has the map to compile), connects to all q3map2 servers in a list and distributes render jobs among them.

This is essentially what distcc does, except with raytracing, not code.

One way to improve this concept would be also to broadcast the q3map2 server on the network with avahi, and you could set up large render farms with just the computers in your own home without any additional setup beyond installing the server. You could compile a map on any machine in your house and get the same high performance benefit.

Unlike distcc, however, since bsp compilation has no ties to cpu architecture, you could send jobs to all servers and get the same results out of all of them regardless of the cpu.

I was thinking of something that would work with NetRadiant, so again, mappers wouldn't be forced to use more than one mapping or compiling tool unless they want to. So I think a blender script would be something separate, but the idea does sound nice

It would be even better if it were possible to build and compile a map in Blender only, as an alternative to Radiant+q3map2. Especially when it would give more control like: all visible surfaces in scene 1, all collision surfaces in scene 2, all vis surfaces in scene 3, and so on. This would make it a lot easier to optimise a map.

Doesn't take long for me to compile even at the highest settings.

Maybe you'd like to start a little render-farm for Xonotic needs?

Compiling from Blender may be an idea, but there aren't many people who could do that.

I for example won't - I know python, but Blender's user interface is so horrible it's simply not viable at all for me. Can't even move the camera around, always have to pivot around a point and moving forward does not work and just zooms... HOW CAN ANYONE MAP WITH THAT?

I seriously don't see the appeal in using Blender for making maps. Or is there a hidden feature for a FPS-like camera mode in the editor (NOT only when you press a key combo, which then disables all other editing controls and moves around continually, but in the actual editing mode)?

Most people who can code, hate Blender, and vice versa... won't get you a in-blender map compiler anytime soon that way

Compiling from Blender may be an idea, but there aren't many people who could do that.

I for example won't - I know python, but Blender's user interface is so horrible it's simply not viable at all for me. Can't even move the camera around, always have to pivot around a point and moving forward does not work and just zooms... HOW CAN ANYONE MAP WITH THAT?

I seriously don't see the appeal in using Blender for making maps. Or is there a hidden feature for a FPS-like camera mode in the editor (NOT only when you press a key combo, which then disables all other editing controls and moves around continually, but in the actual editing mode)?

Most people who can code, hate Blender, and vice versa... won't get you a in-blender map compiler anytime soon that way

Well, you can switch from orthographic to perspective view mode (NumPad 5), then you can move "through" objects.
There's also a completely new user interface with 2.5, though I didn't work with that yet, it looks a lot cleaner and easier to use.

For now, I simply do the main brushwork with NetRadiant, and detail models with Blender...

Hmmm I feel that the blender user interface is the best user interface I have ever encountered in ANY software. All the functions follow the same principle, every function is absolutely easy and quick to access. Actually, all the functions are where I expect them to be, which is very satisfying. Also, it is very easy to move the camera around in this tool, and very comfortable too.

I only recently started to work with blender, out of curiosity and the learning curve is very steep indeed.

But then again I can't code (except LaTeX but that doesn't really count, does it).

Hmmm I feel that the blender user interface is the best user interface I have ever encountered in ANY software. All the functions follow the same principle, every function is absolutely easy and quick to access. Actually, all the functions are where I expect them to be, which is very satisfying

I agree (though it probably isn't the best ever). Once you know it, it's faily easy to use. The getting to know it is the difficult part though. Without reading a tutorial, you won't get anywhere. And that's not ideal for a user interface.

I agree with divVerent there. I could never map with Blender either... nothing like NetRadiant where it's easy to move around and drag all the brushes. I don't hate Blender though... it's a great modeling tool, just not good for maps imo.

At present I have a powerful enough machine to just hammer maps through in q3map2 so I can't say I'm hugely bothered. Doesn't take long for me to compile even at the highest settings.

From what I've seen, that must be a *really* powerful machine. I still wait 45 minutes to compile a map like my Pushme (small map like evilspace) at max settings, with a core i7 socket 1366 CPU running at 2.8 ghz, and 9GB of triple channel RAM.

What's more shocking and hard to understand is that q3map2 is used for over 10 years (Quake 1 2 and 3 used them iirc), times when 128 MB RAM and 700 mhz processors were a luxury. I can't explain how someone could compile a map under one week back then

I agree with divVerent there. I could never map with Blender either... nothing like NetRadiant where it's easy to move around and drag all the brushes. I don't hate Blender though... it's a great modeling tool, just not good for maps imo.

At present I have a powerful enough machine to just hammer maps through in q3map2 so I can't say I'm hugely bothered. Doesn't take long for me to compile even at the highest settings.

From what I've seen, that must be a *really* powerful machine. I still wait 45 minutes to compile a map like my Pushme (small map like evilspace) at max settings, with a core i7 socket 1366 CPU running at 2.8 ghz, and 9GB of triple channel RAM.

What's more shocking and hard to understand is that q3map2 is used for over 10 years (Quake 1 2 and 3 used them iirc), times when 128 MB RAM and 700 mhz processors were a luxury. I can't explain how someone could compile a map under one week back then

q3map2 was used for q1 and 2? wow i hand no idea anyway, yes back then you did wait much longer for a final compile. days and even weeks was not uncommon. ppl just wasn't such bitches abt things taking a bit of time you see your compile times with q3map2 is depended on your mapping skillz, your shader-fu and your q3map2-fu. pushme for example should not take much time at all if you tweak your settings for it. That said, its far to complicated and confusing, but theres no real alternative atm (afaik)

I can't explain how someone could compile a map under one week back then

That software is stable

From what I've seen, that must be a *really* powerful machine. I still wait 45 minutes to compile a map like my Pushme (small map like evilspace) at max settings, with a core i7 socket 1366 CPU running at 2.8 ghz, and 9GB of triple channel RAM.

Evilspace(-ish)? At max settings? On an i7 chip? 45 minutes?? What about at, say, [Q3Map2] -light, low quality? I usually find that sufficient (at least for testing).

For example, I'm running an i5, and to compile (guiltless self plug) evermore_v1r3 on -light, low quality, it takes just under 2 minutes. I've never tried that [Q3Map2] (final) build, but I can't imagine that it would take much longer...

From what I've seen, that must be a *really* powerful machine. I still wait 45 minutes to compile a map like my Pushme (small map like evilspace) at max settings, with a core i7 socket 1366 CPU running at 2.8 ghz, and 9GB of triple channel RAM.

Evilspace(-ish)? At max settings? On an i7 chip? 45 minutes?? What about at, say, [Q3Map2] -light, low quality? I usually find that sufficient (at least for testing).

For example, I'm running an i5, and to compile (guiltless self plug) evermore_v1r3 on -light, low quality, it takes just under 2 minutes. I've never tried that [Q3Map2] (final) build, but I can't imagine that it would take much longer...

I have to say that 45 minutes for EvilSpace is obviously an exaggeration(or, someone doesn't know about the -threads flag). I can compile maps far more complex than that in less than 5 minutes on my AMD Phenom II X4. EvilSpace is a cakewalk for q3map2.

Don't be pussies. I remember final builds that took 24 hours back in the q3 days

Wow. Q3 must be really old if build took that long...

Anyway, we're getting off-topic a bit.

Another idea would be to drop q3map2 and include a tiny map compiler in the engine, which runs the bsp and vis stage, then renders all the lights as if they were realtime lights, and bakes the shadows into lightmaps. All of this would be done in the map loader and be totally transparent to the player.

You'd still need a way to compile the bsp first.

Interesting idea, though.

You'd still need a way to compile the bsp first.

Interesting idea, though.

The engine would load the .map and compile it into a bsp, like doom3.

Touché. Actually, I kinda like that idea. But idk how much that would affect load times, especially for complex maps. You need to compile lightmaps and such too, you know, and that eats time like termites eat wood. It would also be affected by the available processor: that would have to be handled server-side and then sent to everyone else, so if the server host has a slow machine or you have a huge ping, there might be a problem.

However, some mappers might also not like to freely distribute their .maps to the world (I don't really understand why, I just know that some of my mapper friends are adamant that their .maps are private.)

Touché. Actually, I kinda like that idea. But idk how much that would affect load times, especially for complex maps.

Little if at all anything. The BSP and VIS stages of Q3Map2 are already incredibly fast. You'd be trading raw BSP loading(which takes some time in itself) with .map loading which is substantially faster, then running a simple bsp compile on the .map data. Remember, Q3Map2 is rather old and every person I've talked to who has had experience dealing with its code has told me that it's a complete mess and should be re-written.

You need to compile lightmaps and such too, you know, and that eats time like termites eat wood.

As I said, it would simply generate realtime lights for every shader light, render the entire map for one frame, then bake the resulting shadows into a lightmap, which is then loaded by the map loader sequence. All the generated shader lights are then removed.

It would also be affected by the available processor: that would have to be handled server-side and then sent to everyone else, so if the server host has a slow machine or you have a huge ping, there might be a problem.

No, it would all be done client-side with options available for tweaking the light settings. CPU would hardly be used by the light compilation stage as it's all done by the GPU at that point.

Another idea would be to drop q3map2 and include a tiny map compiler in the engine, which runs the bsp and vis stage, then renders all the lights as if they were realtime lights, and bakes the shadows into lightmaps. All of this would be done in the map loader and be totally transparent to the player.

By doing that you'd need to generate huge lightmap textures, and lightmap swapping is one of the biggest slowdowns when rendering using lightmaps. It's not impossible, but even when it sounds simple, it's very hard to get right.