Trying to understand darkplaces source code

[vibok]

I wonder how Spike debugs memory leaks in his FTE. I'm asking because darkplaces does interesting magic with it's memory. I can write more details if anyone is interested.

Need to understand FTE makefile and create a codeblocks project for it too. Maybe I'll start tomorrow.

[vibok]

Found client_state_t cl, but can't inspect it in my debugger. It's too big, and gdb autoexpands all fields, and so debugger hangs. I need to find how to disable autoexpand, or write a pretty printer, like those people say. http://forums.codeblocks.org/index.p...22036.new.html Too lazy for that right now though.

You though this thread is about darkplaces? Too bad, it's about me struggling with my tools.

[vibok]

Okay, managed to compile darkplaces with codeblocks, without makefile.

Create an empty project.

Add all .c files into the project. You can ignore .h files, compiler will find them as long as they are in the same folder. You also need to add .rc file.

If you want to add or rename build targets (Debug Release things), or just something about how your project is built, Project->Properties->Build targets

If you want to change compiler parameters, it's either in Settings->Compiler, or in Project->Build options

If you want to change compiler parameters for a single file, right click file->Properties->Advanced

In Project build options->Compiler settings->Other compiler options, add those:

-Wall
-Wold-style-definition
-Wstrict-prototypes
-Wsign-compare
-Wdeclaration-after-statement
-Wmissing-prototypes

-O3
-fno-strict-aliasing
-fno-math-errno
-ffinite-math-only
-fno-rounding-math
-fno-signaling-nans
-fno-trapping-math

The -Wall and -O3 work weirdly

In Project build options->Compiler settings->#defines, add those:

_FILE_OFFSET_BITS=64
__KERNEL_STRICT_NAMES
CONFIG_MENU
CONFIG_CD
SVNREVISION=-
BUILDTYPE=release_sure

In Project build options->Linker settings->Other linker options, add those:

-mwindows

and maybe those:

-static-libgcc -static-libstdc++ -static

In Project build options->Linker settings->Other linker options, add those:

ws2_32
winmm
user32
gdi32

and maybe those:

comctl32
dinput
dxguid

If something went wrong, you probably added some .c file that isn't used, or skipped some .c file that is used. I may upload my current version of darkplaces sources, if anyone is interested.

for file mod_skeletal_animatevertices_sse.c , use this compiler command:
$compiler $options -msse $includes -c $file -o $object

and for dpsoftrast.c , this:
$compiler $options -msse2 $includes -c $file -o $object


The builddate.c no longer works as it should, but whatever.

http://wiki.codeblocks.org/index.php..._and_Makefiles

>Needing to use a pre-processor is not a valid reason to use a makefile as CB has a pre/post build option. From the menu project->build options there appears a tab with pre/post build steps that can be used for this purpose.

[vibok]

darkplaces and video.

The cl_video_libavw.c file is the only .c file that is included, instead of being compiled separately like all others. (There is even a confused comment about this inside.) I probably could change it, but I don't see video played anywhere and I don't really need it, so I deleted it instead. It also caused problems with codeblocks (nothing too serious, but still annoying, every c file is compiled separately if it's included in the project), another reason to get rid of it.

delete or comment out CL_Video_Init in cl_main.c/CL_Init()

comment out those functions in prvm_cmds.c
VM_cin_open
VM_cin_close
VM_cin_setstate
VM_cin_getstate
VM_cin_restart

in mvm_cmds.c/vm_m_builtins, replace this:

VM_cin_open, // #461
VM_cin_close, // #462
VM_cin_setstate, // #463
VM_cin_getstate, // #464
VM_cin_restart, // #465

with this:

#if 0
VM_cin_open, // #461
VM_cin_close, // #462
VM_cin_setstate, // #463
VM_cin_getstate, // #464
VM_cin_restart, // #465
#else
NULL, // #461
NULL, // #462
NULL, // #463
NULL, // #464
NULL, // #465
#endif

You can now delete cl_video.c cl_video.h cl_video_libavw.c cl_video_libavw.c cl_video_libavw.c cl_video_jamdecode.c files. Also delete cl_video from makefile.

It works with a wrapper, which was written specifically for darkplaces. It actually uses libav wrapper and libavw.dll, and I don't have those anyway and don't know where to get them.

just an extra:

in cl_video_libavw.c

const char* dllnames_libavw[] =
"libavw.dll",

static dllfunction_t libavwfuncs[] =
{
{"LibAvW_Init", (void **) &qLibAvW_Init },
{"LibAvW_ErrorString", (void **) &qLibAvW_ErrorString },
{"LibAvW_AvcVersion", (void **) &qLibAvW_AvcVersion },
{"LibAvW_Version", (void **) &qLibAvW_Version },
{"LibAvW_CreateStream", (void **) &qLibAvW_CreateStream },
{"LibAvW_RemoveStream", (void **) &qLibAvW_RemoveStream },
{"LibAvW_StreamGetVideoWidth", (void **) &qLibAvW_StreamGetVideoWidth },
{"LibAvW_StreamGetVideoHeight",(void **) &qLibAvW_StreamGetVideoHeight },
{"LibAvW_StreamGetFramerate", (void **) &qLibAvW_StreamGetFramerate },
{"LibAvW_StreamGetError", (void **) &qLibAvW_StreamGetError },
{"LibAvW_PlayVideo", (void **) &qLibAvW_PlayVideo },
{"LibAvW_PlaySeekNextFrame", (void **) &qLibAvW_PlaySeekNextFrame },
{"LibAvW_PlayGetFrameImage", (void **) &qLibAvW_PlayGetFrameImage },
{NULL, NULL}
};

https://gitlab.com/xonotic/darkplace...0e7975a14fa1bb

https://github.com/paulvortex/DpLibAVW

https://libav.org/

looks like libav is pretty much just another name for ffmpeg, https://en.wikipedia.org/wiki/Libavcodec

darkplaces dpv

http://forums.insideqc.com/viewtopic.php?f=1&t=2946

http://svn.icculus.org/twilight/trunk/dpvideo/

http://quakeone.com/forums/quake-mod...arkplaces.html

[Spike]

The trick with large software is to remember that large doesn't need to mean complex.
Sure, you'll have a load of different parts of it, but there's no reason for the bridges between those parts to be a nightmare of inter-dependence.
Basically, what I'm saying is that if you're making a large program like a game engine then just keep it modular.

For instance, keep the renderer and the 'client' separate. The 'client' subsystem just reads network messages and builds lists of entities, and passes those over to the renderer. The renderer then renders that stuff, job done. Yes, input is tied to which window has focus etc, there's not much you can do about that other than having the video code take input events from the system and pass them over to the input code (which is what in_generic.c is all about in fte, for instance). You also have the server, which is of course pretty much entirely separate from the client. The renderer and sound subsystems are basically just slaves to the client subsystem. The nightmare comes when you start to create cvars and console commands and of course consoles and other such things that are re(ab)used by every single other subsystem. Which causes issues because it means you can't thread anything without mutexing all those cvars and that would just be horribly slow.
So what have we learnt from all this? Keep code modular. Don't use globals (including cvars at least beyond initialising each subsystem). Keep shared code mutexable (read: few entry points) or reenterant (ie: the code only accesses arguments, no globals).

And of course, DP and FTE both suck, at least in terms of threading - they're NOT 'modern' engines because of that. The whole compatibility and legacy thing means that most subsystems can't be rewritten cleanly for threads, and as such they'll always favour single-core performance and basically ignore all other cores. We can't just throw the server onto another thread to run it async, and there's all sorts of things like that. With vulkan and d3d12 (and frankly other rendering APIs too if you're doing it right, just with two passes) your renderer needs to be multithreaded if you want to make the most of the gpu (for instance, FTE does actually use two threads for vulkan, and this doubles the framerate in certain circumstances).
An example - once you have the client just passing the renderer a complete scenegraph, you can have the renderer and the rest of the client running on different threads entirely. Due to quake's legacy of cvar abuse, this simply cannot be done reliably in DP/FTE. One such trick around this is to use fte's 'mapcluster' feature to have the server running inside an entirely different process, thus using two copies of every cvar as a solution to thread safety. Of course, there's a large number of single player mods that this breaks, and its hardly user friendly to have to two separate consoles that need any cvar changes...
So yeah... Cvars. They're evil. By all means use a similar construct for initialisation, but beyond that avoid them like the plague. That also includes console commands. They work in quake because they're executed at a time when its guarenteed that nothing else is running, but in a modern engine that means you need to wait for all the various threads to finish too, and that SUCKS, frankly its better to use cvars (or some kind of message-passing to the subsystem that's actually responsible for it).
Basically what I'm trying to say is that you should get VERY familiar with networking, and to use that sort of thing even within your engine, because threading is a nightmare otherwise.

memory allocation is a fairly straight forward thing. Allocator Frees. If you call malloc, you should call free. If you call CreateFoo, you should call DestroyFoo. If foo has blocks of memory or textures or whatever, it can then free those too. If you're refcounting then Get/Create/Dupe vs Release is fine, or just use C++'s smart pointers or whatever (constructor and destructor, so same idea, just a bit more automatic).
The singular except to this is with message passing between threads, in such cases its perfectly fine to re-assign ownership as it were - the client passes its entity list over to the renderer, and the renderer then frees it, but remember that its usually best for the renderer to call some callback so the client can do the actual free - especially if you're crossing module boundaries where the malloc/free implementations may actually differ. Besides, your client may actually care to know when the previous frame has finished, if only so that it doesn't try rendering 20 different frames before the first completes. It would be fine to do it as X slots in a ring buffer, for instance.
Taking simple rules like 'allocator frees' and applying them to your entire codebase (API-willing) helps ensure a level of consistency across all of your internal APIs, making it easier to remember how to actually use them all. Everything becomes easier when you know what to expect.
The main issue with memory is to avoid making large allocations. If you need to make one then don't bother freeing it after if you're likely to reuse it after. Reallocating large blocks of memory will just fragment your address space when mixed with smaller allocations. If possible use VirtualAlloc or mmap or whatever to reserve address space from the start, then allocate+release pages as needed instead of fragmenting your heap. Vanilla Quake's hunk memory is fast to allocate and avoids fragmentation, but it makes it hard to release the other memory associated with an object, like textures or whatever.

And yes, other people's work never makes sense. Much of that is because you can still half-remember your own work, which makes it so much easier to work with (eg: you know who's responsible for freeing everything without needing to look it up).
Sometimes the code uses bizarre paradigms (like C's '-->' operator hack) that can take a little bit of time to get comfortable with. And quite often, its because the person writing it just wanted something done quick, and couldn't be arsed to document the limitations and performance issues of their quickie hack...
So yeah, when taking over someone else's project, it can take quite some time before you're fully comfortable... Especially if it was me that wrote it. Ultimately if it takes a long time to get a proper overview of it then it's just shitty code... Or uses lots of maths (but even then it should still be easy enough to trace when something is calculated and then used, and why, even if you can't figure out what its actually doing, you may just have to be prepared to take a step back and look at the calling function instead.

[Dutch]

// just a thought...

Have you ever tried to pick up someone else's work, gotten halfway through, and then realized how much more controlled and sensible the whole project would be if YOU had actually done it to begin with?

I deal with this all the time at work. I pick up where someone left off and spend a shit-ton of time 'reverse engineering' their thought process so that I can keep moving forward.

I don't see why it's any different with a game engine. How many countless hours did LordHavoc spend connecting all the dots in his engine? Would you rather disconnect all of his dots, just to reconnect them again? Or just draw your own dots and connect them from the get-go?

Analogies aside, I would imagine learning the foundations and basics of C and building an exectutable would be more manageable than hacking apart darkplaces and re-pasting it back together kindergarten-style. Maybe not as quick but it would be a more enjoyable and knowledgeable ride.

[MadGypsy]

@large programs in general.

ALL of my very extensive studies covering numerous languages has led me to decide and agree that you should actually write a bunch of smaller programs that have as few dependencies as possible and then write a class that combines and coordinates them.

For one, if you do it that way you end up with a lot of reusable script. An example would be my math expression parser. I can drop that parser in any project and it's ready to use. My content manager is the same way. Actually, everything I've made is this way... Lots of idea specific API's that can be used as is in any project.

Let's assume you write a solid renderer. It shouldn't be married to anything. You should be able to drop the renderer in any project without dragging along an entire game engine.

Quake engines are generally one big dependent package and look how hard it is to track everything down. Imagine if all the concepts we're their own API and they were being combined and coordinated by a main script or 2.

Another perk is testing. With mini independent APIs you don't end up with situations where thingA has to be completed and functional before you can test/work on thingB, thingD or any other thing. Rip progs parser out of Quake and tell me it's still rendering the BSP. You can't and it won't.

[vibok]

I actually already wrote a small window in winapi, and text rendering in opengl. I paused to look at how real engines do these things before moving forward.

I need to learn how they organize memory, and details about rendering, and how to write large programs in general. So far I only found keyboard handling.

[MadGypsy]

You are starting to remind me of SSBoxer. That's not meant as a bad thing...just a true thing. I knew infinitely more than him regarding what he should do, he kept trying to go his own way and he hit wall after wall. You have a super talented and accomplished programmer (spike) giving you guidance, you are trying to go your own way and you are going to hit wall after wall if you aren't hitting them already.

I think the point here is clear. Start building an engine and use SDL2. Actually, even more accurately, forget the engine for right now and just start practicing with SDL2. You aren't going to sit down and just build an engine from A to Z no matter how much you learn about other engines. Get one textured poly to render in a 3D context and build from there.

For the record, I have built a huge chunk of a working engine and although I haven't accomplished as much as people like spike, Baker, etc... with it, I am light-years ahead of where you are. I got that way by sitting down and actually programming. I didn't spend any time trying to understand everything before I start. I got a poly to render, then I turned that poly into a bsp, then I learned more and turned the bsp render into a better bsp render... on and on.

[vibok]

There will be so much trash inside my executable if I'll just use libraries blindly. Both programmers and users usually ignore that. Who really cares if your exe is 1 MiB or 10.

GLFW looks just a bit better than SDL2. Haven't driven into it either yet. Raw winapi is the best, and also the biggest waste of time.

[Spike]

the point is that you don't need to understand the SDL2 sourcecode, same as you don't need to understand the winapi sourcecode.
What you do need to understand is the API that SDL2 provides to you, and that is much simpler than the alternatives.

here's a good place to start: https://wiki.libsdl.org/APIByCategory
You'll need to call SDL_Init to init whatever subsystems you're actually going to use, but after that you're free to call any function you want.
eg call sdl_createwindow to create a window, or SDL_PollEvent to try to read the next event for you to handle.
if you want to use opengl, create a window then call SDL_GL_CreateContext (you'll find some example code there).
It doesn't show EVERYTHING, but if it did there would just be too much noise.
It doesn't document opengl itself, which is an entirely different beast, SDL just shows the helpers that it provides to create an opengl context, you have to do the rest yourself.

[vibok]

My main points of interest currently are: rendering, physics, memory handling, file handling. If somebody knows how to find them, please share.

[vibok]

Guys, have you managed to understand SDL2 source code? I tried some time ago with SDL1, but failed. One of it's comments just said "if you don't like seeing complicated #ifdef code in the headers, then don't look".

Also, how to use memory mapped file if not with winapi?

[MadGypsy]

lol...Spike said a bunch of stuff that I had decided not to. My last post was actually pretty close to what spike said with links to a few "simpler" engine sources. At the last minute I changed it to what is there now. I had decided it would be easier to give you the most simple code I know of and let you come to your own conclusion that the overview method is the only one that is going to work with it because, there isn't really anything to strip.

Also...didn't LordHavoc use a bunch of macros that make it complicated to tear apart/modify his engine? I seem to remember Baker commenting about that around a year ago.

[Spike]

okay, a few things. if you just want to get things to a point where you understand them then ditch the whole windows api bullshit. learn SDL2 stuff only. its simpler, AND its more portable. Yes its an extra dependancy, but then so too is the winapi stuff, in a way.

spending your time finding and isolating the parts that you DON'T want is basically a waste of that time. If you're planning on forking DP then fine, but if you're trying to write your own engine then its purely wasted time. If you're trying to make a mod for DP then forking the engine to strip stuff is in the long run just going to make it harder to maintain. At the end of the day, an entire engine is a really complex beast, by the time you've learnt one subsystem, you'll have forgotten the previous one. Either way its really only the overview that you need to learn, anything more than that and you should know where to look up anything more that you need.

Not to contradict Baker, but if you really want to learn then building something from scratch and then stripping+copying over parts of the individual subsystems into an otherwise-from-scratch engine would be much more interesting and therefore more effective way to learn how a game engine works. Nothing shows that you understand something better than building an alternative to it.