No announcement yet.

Trying to understand darkplaces source code

  • Filter
  • Time
  • Show
Clear All
new posts

  • // just a thought...

    Have you ever tried to pick up someone else's work, gotten halfway through, and then realized how much more controlled and sensible the whole project would be if YOU had actually done it to begin with?

    I deal with this all the time at work. I pick up where someone left off and spend a shit-ton of time 'reverse engineering' their thought process so that I can keep moving forward.

    I don't see why it's any different with a game engine. How many countless hours did LordHavoc spend connecting all the dots in his engine? Would you rather disconnect all of his dots, just to reconnect them again? Or just draw your own dots and connect them from the get-go?

    Analogies aside, I would imagine learning the foundations and basics of C and building an exectutable would be more manageable than hacking apart darkplaces and re-pasting it back together kindergarten-style. Maybe not as quick but it would be a more enjoyable and knowledgeable ride.
    'Replacement Player Models' Project


    • The trick with large software is to remember that large doesn't need to mean complex.
      Sure, you'll have a load of different parts of it, but there's no reason for the bridges between those parts to be a nightmare of inter-dependence.
      Basically, what I'm saying is that if you're making a large program like a game engine then just keep it modular.

      For instance, keep the renderer and the 'client' separate. The 'client' subsystem just reads network messages and builds lists of entities, and passes those over to the renderer. The renderer then renders that stuff, job done. Yes, input is tied to which window has focus etc, there's not much you can do about that other than having the video code take input events from the system and pass them over to the input code (which is what in_generic.c is all about in fte, for instance). You also have the server, which is of course pretty much entirely separate from the client. The renderer and sound subsystems are basically just slaves to the client subsystem. The nightmare comes when you start to create cvars and console commands and of course consoles and other such things that are re(ab)used by every single other subsystem. Which causes issues because it means you can't thread anything without mutexing all those cvars and that would just be horribly slow.
      So what have we learnt from all this? Keep code modular. Don't use globals (including cvars at least beyond initialising each subsystem). Keep shared code mutexable (read: few entry points) or reenterant (ie: the code only accesses arguments, no globals).

      And of course, DP and FTE both suck, at least in terms of threading - they're NOT 'modern' engines because of that. The whole compatibility and legacy thing means that most subsystems can't be rewritten cleanly for threads, and as such they'll always favour single-core performance and basically ignore all other cores. We can't just throw the server onto another thread to run it async, and there's all sorts of things like that. With vulkan and d3d12 (and frankly other rendering APIs too if you're doing it right, just with two passes) your renderer needs to be multithreaded if you want to make the most of the gpu (for instance, FTE does actually use two threads for vulkan, and this doubles the framerate in certain circumstances).
      An example - once you have the client just passing the renderer a complete scenegraph, you can have the renderer and the rest of the client running on different threads entirely. Due to quake's legacy of cvar abuse, this simply cannot be done reliably in DP/FTE. One such trick around this is to use fte's 'mapcluster' feature to have the server running inside an entirely different process, thus using two copies of every cvar as a solution to thread safety. Of course, there's a large number of single player mods that this breaks, and its hardly user friendly to have to two separate consoles that need any cvar changes...
      So yeah... Cvars. They're evil. By all means use a similar construct for initialisation, but beyond that avoid them like the plague. That also includes console commands. They work in quake because they're executed at a time when its guarenteed that nothing else is running, but in a modern engine that means you need to wait for all the various threads to finish too, and that SUCKS, frankly its better to use cvars (or some kind of message-passing to the subsystem that's actually responsible for it).
      Basically what I'm trying to say is that you should get VERY familiar with networking, and to use that sort of thing even within your engine, because threading is a nightmare otherwise.

      memory allocation is a fairly straight forward thing. Allocator Frees. If you call malloc, you should call free. If you call CreateFoo, you should call DestroyFoo. If foo has blocks of memory or textures or whatever, it can then free those too. If you're refcounting then Get/Create/Dupe vs Release is fine, or just use C++'s smart pointers or whatever (constructor and destructor, so same idea, just a bit more automatic).
      The singular except to this is with message passing between threads, in such cases its perfectly fine to re-assign ownership as it were - the client passes its entity list over to the renderer, and the renderer then frees it, but remember that its usually best for the renderer to call some callback so the client can do the actual free - especially if you're crossing module boundaries where the malloc/free implementations may actually differ. Besides, your client may actually care to know when the previous frame has finished, if only so that it doesn't try rendering 20 different frames before the first completes. It would be fine to do it as X slots in a ring buffer, for instance.
      Taking simple rules like 'allocator frees' and applying them to your entire codebase (API-willing) helps ensure a level of consistency across all of your internal APIs, making it easier to remember how to actually use them all. Everything becomes easier when you know what to expect.
      The main issue with memory is to avoid making large allocations. If you need to make one then don't bother freeing it after if you're likely to reuse it after. Reallocating large blocks of memory will just fragment your address space when mixed with smaller allocations. If possible use VirtualAlloc or mmap or whatever to reserve address space from the start, then allocate+release pages as needed instead of fragmenting your heap. Vanilla Quake's hunk memory is fast to allocate and avoids fragmentation, but it makes it hard to release the other memory associated with an object, like textures or whatever.

      And yes, other people's work never makes sense. Much of that is because you can still half-remember your own work, which makes it so much easier to work with (eg: you know who's responsible for freeing everything without needing to look it up).
      Sometimes the code uses bizarre paradigms (like C's '-->' operator hack) that can take a little bit of time to get comfortable with. And quite often, its because the person writing it just wanted something done quick, and couldn't be arsed to document the limitations and performance issues of their quickie hack...
      So yeah, when taking over someone else's project, it can take quite some time before you're fully comfortable... Especially if it was me that wrote it. Ultimately if it takes a long time to get a proper overview of it then it's just shitty code... Or uses lots of maths (but even then it should still be easy enough to trace when something is calculated and then used, and why, even if you can't figure out what its actually doing, you may just have to be prepared to take a step back and look at the calling function instead.
      Some Game Thing


      • darkplaces and video.

        The cl_video_libavw.c file is the only .c file that is included, instead of being compiled separately like all others. (There is even a confused comment about this inside.) I probably could change it, but I don't see video played anywhere and I don't really need it, so I deleted it instead. It also caused problems with codeblocks (nothing too serious, but still annoying, every c file is compiled separately if it's included in the project), another reason to get rid of it.

        delete or comment out CL_Video_Init in cl_main.c/CL_Init()

        comment out those functions in prvm_cmds.c

        in mvm_cmds.c/vm_m_builtins, replace this:

        VM_cin_open, // #461
        VM_cin_close, // #462
        VM_cin_setstate, // #463
        VM_cin_getstate, // #464
        VM_cin_restart, // #465

        with this:

        #if 0
        VM_cin_open, // #461
        VM_cin_close, // #462
        VM_cin_setstate, // #463
        VM_cin_getstate, // #464
        VM_cin_restart, // #465
        NULL, // #461
        NULL, // #462
        NULL, // #463
        NULL, // #464
        NULL, // #465

        You can now delete cl_video.c cl_video.h cl_video_libavw.c cl_video_libavw.c cl_video_libavw.c cl_video_jamdecode.c files. Also delete cl_video from makefile.

        It works with a wrapper, which was written specifically for darkplaces. It actually uses libav wrapper and libavw.dll, and I don't have those anyway and don't know where to get them.

        just an extra:

        in cl_video_libavw.c

        const char* dllnames_libavw[] =

        static dllfunction_t libavwfuncs[] =
        {"LibAvW_Init", (void **) &qLibAvW_Init },
        {"LibAvW_ErrorString", (void **) &qLibAvW_ErrorString },
        {"LibAvW_AvcVersion", (void **) &qLibAvW_AvcVersion },
        {"LibAvW_Version", (void **) &qLibAvW_Version },
        {"LibAvW_CreateStream", (void **) &qLibAvW_CreateStream },
        {"LibAvW_RemoveStream", (void **) &qLibAvW_RemoveStream },
        {"LibAvW_StreamGetVideoWidth", (void **) &qLibAvW_StreamGetVideoWidth },
        {"LibAvW_StreamGetVideoHeight",(void **) &qLibAvW_StreamGetVideoHeight },
        {"LibAvW_StreamGetFramerate", (void **) &qLibAvW_StreamGetFramerate },
        {"LibAvW_StreamGetError", (void **) &qLibAvW_StreamGetError },
        {"LibAvW_PlayVideo", (void **) &qLibAvW_PlayVideo },
        {"LibAvW_PlaySeekNextFrame", (void **) &qLibAvW_PlaySeekNextFrame },
        {"LibAvW_PlayGetFrameImage", (void **) &qLibAvW_PlayGetFrameImage },
        {NULL, NULL}

        looks like libav is pretty much just another name for ffmpeg,

        darkplaces dpv


        • Okay, managed to compile darkplaces with codeblocks, without makefile.

          Create an empty project.

          Add all .c files into the project. You can ignore .h files, compiler will find them as long as they are in the same folder. You also need to add .rc file.

          If you want to add or rename build targets (Debug Release things), or just something about how your project is built, Project->Properties->Build targets

          If you want to change compiler parameters, it's either in Settings->Compiler, or in Project->Build options

          If you want to change compiler parameters for a single file, right click file->Properties->Advanced

          In Project build options->Compiler settings->Other compiler options, add those:



          The -Wall and -O3 work weirdly

          In Project build options->Compiler settings->#defines, add those:


          In Project build options->Linker settings->Other linker options, add those:


          and maybe those:

          -static-libgcc -static-libstdc++ -static

          In Project build options->Linker settings->Other linker options, add those:


          and maybe those:


          If something went wrong, you probably added some .c file that isn't used, or skipped some .c file that is used. I may upload my current version of darkplaces sources, if anyone is interested.

          for file mod_skeletal_animatevertices_sse.c , use this compiler command:
          $compiler $options -msse $includes -c $file -o $object

          and for dpsoftrast.c , this:
          $compiler $options -msse2 $includes -c $file -o $object

          The builddate.c no longer works as it should, but whatever.

          >Needing to use a pre-processor is not a valid reason to use a makefile as CB has a pre/post build option. From the menu project->build options there appears a tab with pre/post build steps that can be used for this purpose.


          • Found client_state_t cl, but can't inspect it in my debugger. It's too big, and gdb autoexpands all fields, and so debugger hangs. I need to find how to disable autoexpand, or write a pretty printer, like those people say. Too lazy for that right now though.

            You though this thread is about darkplaces? Too bad, it's about me struggling with my tools.


            • I wonder how Spike debugs memory leaks in his FTE. I'm asking because darkplaces does interesting magic with it's memory. I can write more details if anyone is interested.

              Need to understand FTE makefile and create a codeblocks project for it too. Maybe I'll start tomorrow.


              • ok, so I figured out why my potato showed such a low fps.

                the r_speeds 2 was priceless for that. After typing it, I saw that "shadowmaps" takes suspiciously much of rendering time. The code for rendering is in gl_rmain.c/R_RenderScene(). Statistics are gathered with this snippet:

                if (r_timereport_active)

                I tried to see which part of drawing shadows was the slowest by placing my own probes, and saw that the chain R_Shadow_DrawShadowMaps()/R_Shadow_ClearShadowMapTexture()/GL_Clear()/qglClear() takes all of rendering time. This thread suggested that it means nothing, because driver groups opengl calls.

                So instead of figuring this code out I just commented the whole shadow rendering. I couldn't spot the difference, honestly. Levels are still shaded, monsters get darker then go into the dark areas. I guess monsters don't cast shadows now, but I never seen that anyway for some reason. LATER check if dlight from rockets still works.

                Now my potato can play it, it got to 60 fps.

                I wonder if there is a way to disable it with just cvars, without commenting out and recompiling.

                One weird thing I spotted, inside R_Shadow_ClearShadowMapTexture(), there is R_Mesh_ResetTextureState() and R_Shadow_RenderMode_Reset() called one after another. If you look inside the second one, you'll see first one called again. Dunno if it's supposed to be like that or not, but it's pretty suspicious.




                • By the way, thank you for your replies. Feet kinda reluctant to write just a simple thank you, without anything else, that's why it's so slow.


                  • sys_shared.c/Sys_DirtyTime()

                    It looks like there is no need to call QueryPerformanceFrequency on every frame, it will always be the same. Windows will fix the value of QueryPerformanceCounter by itself if internal cpu frequency changes. Or something.

                    Wonder if it works the same on linux.

                    Wonder if it even worth creating an extra global variable for, considering it's not even the default timer used.


                    Tried to email darkplaces author(s) some time ago, not about this, got no reply so far. Could be wrong email, or bad spamfilter. Torturing you for answers isn't that productive.

                    Tried arcane dimensions on darkplaces, it's noticeably slow. On quakespasm it's pretty fast. Maybe because darkplaces is 32 bit, and BSP2 uses 64 bit data. Or I did something wrong with compilation, who knows.

                    Still haven't looked at physics, curious how they are done.

                    ogre3d has some usable code for timer. Valve did something weird with their timer, but it's not fully open source, only headers.

                    static cvar_t sys_usequeryperformancecounter = {CVAR_SAVE, "sys_usequeryperformancecounter", "0", "use windows QueryPerformanceCounter timer (which has issues on multicore/multiprocessor machines and processors which are designed to conserve power) for timing rather than timeGetTime function (which has issues on some motherboards)"};


                    • >Basically anything more complicated than RLE, i.e. Huffman would have been covered by patents like GIF, JBIG, JPEG2000, etc.

                      Huh, Carmack would've probably used more advanced compression, if he made quake today. I wonder what would fit, for textures and maps.


                      • Do not try.



                        • wha


                          • Problem I have right now is, I have three different quake engines, and they all require almost the same data (music, levels). And it's copied for each one of them. I wonder if there is a way to make them read from the same source, I need to save harddrive space.

                            Darkplaces also keeps trying to write something in %USERPROFILE%, and I want to use it as a portable app.


                            • -basedir


                              • sys_win.c/WinMain()
                                Strange that darkplaces parces arguments manually instead of calling CommandLineToArgv.
                                Strange that darkplaces actually stores hInstance, instead of just using 0x400000 everywhere. As far as I know, it's always 0x400000 in 32bit apps.

                                Wish somebody actually paid me for this.
                                Last edited by vibok; 08-30-2017, 02:12 AM.