04-04-2010, 05:51 AM
(This post was last modified: 04-04-2010, 05:53 AM by GreEn`mArine.)
Recently when I looked at a forum thread where people were showing their available hardware+software configurations, I came across this idea:
Developers often want to know how changes did impact performance on different systems. However, testing this is error prone and tedious most of the time, and the worst of all, it usually requires people with above-average-knowledge, because they will have to know how to download, install, and use a GIT client, how to build the game.dat files and the engine (for a specific revision) and then also need to know how to give the results to the developer.
So, since ages, the process was that someone who was in the developer's IRC channel was compiling binaries manually for one revision, running a timedemo, and then copying the resulting line (with min/avg/max fps) into the channel together with the revision information, and then doing the same thing for another revision. All this is a quite manual, time intensive process, and it's bad because only a limited number of people are actually able to do that.
So, now finally my suggestion:
- Have a cross-platform application consisting of a server component (running on a server hosted by the Xonotic team) and a client binary (GUI or non-gui).
- The purpose of the client is that it has to be able to check out (fetch) engine and game repositoriy data and compile game and engine binaries
- The client must be able to connect to the server component and ask the server for new performance test profiles that should be performed. The Xonotic developers are the ones who specify what data they want. A profile could consist of the following information: (a) update to DP revision X and xonotic game revision Y, (b) after the client compiled the binaries, execute the benchmark-demo A with config B (A and B are obviously present in xonotic's respository at revision X)
- After the client executed the profile, the resulting data (given from DPs benchmark file, namely that min/avg/max fps) will be extracted and sent back to the server. Of course for the information to be meaningful, the client also has to specify additional information, such as his username, OS and version, CPU type, RAM, Graphic card model, other details...
So, in case such a system would be done, imagine how powerful it would be. End-users who want to help the developers simply have to start their test-client, connect to the server and select the new "performance test profiles" they want to execute on their machine, click on start. The client will automatically fetch the required revision/commit, compile the binaries, run the test and send the data back to the developers. Since this is an easy step, more users with even more versatile HW/SW configurations would participate, since all they need to do is to do a few clicks and then let their PC/Mac do the work, e.g. over night.
A few other aspects that come to my mind or that I haven't explained yet:
- The client must of course be able to download several profiles from the server and allow the user to select which ones to actually execute. This batch execution is what makes collecting data powerful
- In case that compiling binaries on the client's machine is too complicated (e.g. for Mac OS X), the game/engine binaries could actually be sent to the client from the server along with the profile. Of course, the downside of this is that it leaves the Xonotic developer who creates the profile and uploads it to the server has more work
- One might think about an actual account/authorization system, it might even be "invite only". Advantage of this is that test results could be clearly mapped to one user, or maybe actually have one account per user+HW+SW configuration)
- End-users should also be able to run custom profiles which have not been fetched from the server, but are simply based on the interests of the user him/herself.
Developers often want to know how changes did impact performance on different systems. However, testing this is error prone and tedious most of the time, and the worst of all, it usually requires people with above-average-knowledge, because they will have to know how to download, install, and use a GIT client, how to build the game.dat files and the engine (for a specific revision) and then also need to know how to give the results to the developer.
So, since ages, the process was that someone who was in the developer's IRC channel was compiling binaries manually for one revision, running a timedemo, and then copying the resulting line (with min/avg/max fps) into the channel together with the revision information, and then doing the same thing for another revision. All this is a quite manual, time intensive process, and it's bad because only a limited number of people are actually able to do that.
So, now finally my suggestion:
- Have a cross-platform application consisting of a server component (running on a server hosted by the Xonotic team) and a client binary (GUI or non-gui).
- The purpose of the client is that it has to be able to check out (fetch) engine and game repositoriy data and compile game and engine binaries
- The client must be able to connect to the server component and ask the server for new performance test profiles that should be performed. The Xonotic developers are the ones who specify what data they want. A profile could consist of the following information: (a) update to DP revision X and xonotic game revision Y, (b) after the client compiled the binaries, execute the benchmark-demo A with config B (A and B are obviously present in xonotic's respository at revision X)
- After the client executed the profile, the resulting data (given from DPs benchmark file, namely that min/avg/max fps) will be extracted and sent back to the server. Of course for the information to be meaningful, the client also has to specify additional information, such as his username, OS and version, CPU type, RAM, Graphic card model, other details...
So, in case such a system would be done, imagine how powerful it would be. End-users who want to help the developers simply have to start their test-client, connect to the server and select the new "performance test profiles" they want to execute on their machine, click on start. The client will automatically fetch the required revision/commit, compile the binaries, run the test and send the data back to the developers. Since this is an easy step, more users with even more versatile HW/SW configurations would participate, since all they need to do is to do a few clicks and then let their PC/Mac do the work, e.g. over night.
A few other aspects that come to my mind or that I haven't explained yet:
- The client must of course be able to download several profiles from the server and allow the user to select which ones to actually execute. This batch execution is what makes collecting data powerful
- In case that compiling binaries on the client's machine is too complicated (e.g. for Mac OS X), the game/engine binaries could actually be sent to the client from the server along with the profile. Of course, the downside of this is that it leaves the Xonotic developer who creates the profile and uploads it to the server has more work
- One might think about an actual account/authorization system, it might even be "invite only". Advantage of this is that test results could be clearly mapped to one user, or maybe actually have one account per user+HW+SW configuration)
- End-users should also be able to run custom profiles which have not been fetched from the server, but are simply based on the interests of the user him/herself.
