Announcement

Collapse
No announcement yet.

Decompiling every progs.dat

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Decompiling every progs.dat

    There are a few quake mods out there that are "closed" source. FvF, CAx, CRMod- to name a few. I thought I'd take a look into how they prevent decompilation by looking into the source code for compilers (frikqcc, qccx), but it turns out the decompiling code is not actually included in the source code for these... That is, the binaries you can download (.exe programs) can decompile a simple progs.dat into human-readable code, but if you use the released source code to build your own .exe, that particular .exe cannot decompile any progs.dat.

    This situation really piques my interest, and I might have to make my own decompiler from scratch, but I'm wondering what the ramifications of this would be. Someone(s) already decided that a decompiler out in the wild would be bad news, and I'm not sure how to continue, in terms of philosophically. Should code be free, or are there reasons to protect closed source mods?

    I don't really have free time until the semester ends next month, so there's not a big hurry at the moment. I'd like to hear some opposing viewpoints before I dig in.

  • #2
    This is interesting,
    I thought that anyone using Quakes source would under the GPL agreement have to provide the source either with the mod or on request as per the licence.

    Unless they have paid the owners for a closed licence.

    Have you tried asking the developers for a copy of their source code?

    Disclaimer : I could be completely wrong about this.
    Username : Atomic Robokid on Steam

    Please check out my Quake made things:

    https://www.indiedb.com/games/run-over
    https://adam-freeman.itch.io/hazard
    https://adam-freeman.itch.io/diver
    https://adam-freeman.itch.io/beyond

    Comment


    • #3
      In the past, I've enhanced frikdec to be able to better cope with certain exotic mods without crashing. it can't make sense of many of the functions, so its practical value is somewhat low, but it can be handy for quickly verifying that a qc compiler is generating something that makes sense.
      I could officially release it, but I personally feel that *ANY* encouragement for people to use decompiled source is a disservice to those people who will then be left trying to maintain what is essentually gibberish.
      Additionally, as a maintainer of a qc compiler, there's a conflict of interest.

      from a legal perspective, the gpl doesn't say anything to allow decompiling, it only makes it a bit pointless to persue.
      Many mods are based upon the original code base and are not tied to the GPL (but are tied to commercial quake). A later version being relicensed does not invalidate the previous licenses (unless the previous licenses included a clause to that affect). You cannot blanket-say that all/most/interesting QC mods are legally decompilable, because that simply isn't true.
      Additionally there are from-scratch mods that were written purely to escape the gpl's restrictions.

      from a personal perspective, releasing something to decompile any mod ever made isn't likely to go down well.

      Edit: more stuff.
      There are decompilers out there that can decompile most mods. fteqcc's default optimisations are chosen to avoid making too much of a mess when run through frikdec, but that doesn't mean that all the output will make sense.
      And that's kinda the problem. At some point, you have to give up and accept that the decompiler will basically be ouputting unreadable gibberish if the compiler is optimising everything to the maximum extent possible. Essentually, with full optimisations you cannot reliably tell whether a variable was a temp, a local, or a global. And because of that, you either have to resort to dumping assembler instead of actual statements, or to splitting up the statements in an arbitrary and probably unreadable fashion.
      Of course, current qc decompilers don't accept(and can't detect) this and just output buggy gibberish in such cases. Either way, the names of any locals will be long gone.

      There are also tools out there that randomize certain parts of the progs.dat. By reordering or stripping information, decompilers that just scan stuff linearly started to crash. If you do write a debugger, it would probably be appreciated to detect this case and abort with a message along the lines that the original creator doesn't want people to decompile their mod.

      I've decompiled closed source mods when people have reported issues, told them how to fix their mod to avoid the issue, and they've been more put out by the fact that its a bug in their mod more so than the fact that I so easily decompiled it.
      Last edited by Spike; 11-27-2014, 11:39 AM.
      Some Game Thing

      Comment


      • #4
        Well the Quake server can 'decompile' every working mod, so I'd make a mod analyzer from the Quake server.

        QuakeC logic flow is totally event driven, events call a function using:
        void PR_ExecuteProgram (func_t fnum)
        https://github.com/id-Software/Quake...uake/pr_exec.c

        Event list
        Code:
        sv_main.c: void SV_ConnectClient (int clientnum)
        sv_main.c: void SV_SaveSpawnparms (void)
        host.c: void SV_DropClient (qboolean crash)
        host_cmd.c: void Host_Kill_f (void)
        host_cmd.c: void Host_Spawn_f (void)
        sv_phys.c: qboolean SV_RunThink (edict_t *ent)
        sv_phys.c: void SV_Impact (edict_t *e1, edict_t *e2)
        sv_phys.c: void SV_PushMove (edict_t *pusher, float movetime)
        sv_phys.c: void SV_PushRotate (edict_t *pusher, float movetime)
        sv_phys.c: void SV_Physics_Pusher (edict_t *ent)
        sv_phys.c: void SV_Physics_Client (edict_t	*ent, int num)
        sv_phys.c: void SV_Physics (void)
        world.c: void SV_TouchLinks ( edict_t *ent, areanode_t *node )
        As the Quake server with mod analyzing features is running it can record key information that helps one deduce what a unnamed variable or function does by noting what is occurring as it's being accessed. Then you can create a memory map from this information for the assembler code.

        https://github.com/id-Software/Quake...uake/pr_comp.h
        has details on the data located in progs.dat

        dprograms_t is the overview of statements,globaldefs,fielddefs,functions,strings, globals,and entityfields
        dfunction_t details functions

        Decompile progs.dat to assembler code, use assembler to label functions,variables, and add comments, then compile the assembler back to machine code (progs.dat) Granted assembler code isn't as easy to work with as a C style code, but it's better than nothing. When I say machine code I of course mean virtual machine code, not something a CPU would recognize.

        Decompiling any mod has many positive aspects:
        -It can be used to fix bugs that otherwise would never get fixed.
        -It can be used to remove limitations that otherwise would never get removed.
        -It can be used to update to newer progs versions.
        -Perhaps it could even be improved upon.

        Many mods have been abandoned long ago. As long as proper credit is given it shouldn't be a problem. I suppose doing a file compare of progs.dat could detect if someone copied code without due credit, or if permission was denied from the original author and they went ahead and used it anyways.

        Comment


        • #5
          (If) I was coder,and I abandoned my baby only to find out some one later on found that same baby and raised it into something more matured. I would be proud.
          Want to get into playing Quake again? Click here for the Multiplayer-Startup kit! laissez bon temps rouler!

          Comment


          • #6
            an easy way to make the source non-decomplie-able is to define a function at
            the bottom of the defs.qc
            for instance in cax i have this

            Code:
            void () decompile_error =
            {
                 local string dopefish;
                 dopefish = ("dont smoke fish, drink Bud.\n");
            };
            as for gpl concerns i used the 1.06 progs.dat so it is kind of a grey area as to that licence. plus portions of the code were provided from proprietary code from JPG
            so without his permission i dont release the source openly. that and any admin could make built-in cheats for themselves adding special impulse commands etc.
            Last edited by R00k; 11-28-2014, 06:34 PM.
            www.quakeone.com/qrack | www.quakeone.com/cax| http://en.twitch.tv/sputnikutah

            Comment


            • #7
              I find it ironic how J. Carmack purposely opened up, completely, a commercially released game/engine and yet mod authors would purposely keep their hobbyist sources closed!
              Name's damage_inc, and killing is my business. Don't worry though, it's nothing personal! Oh wait... maybe it is

              Comment


              • #8
                @hypersonic, that's not decompiling, that's disassembling.
                if you want to disassemble a mod, fteqcc should be able to parse asm statements/functions mixed with regular qc code (read: re-assemble). you'll need to figure out prototypes (or specify things as variants or so), but this should simplify modifying such disassembled mods (use -Fwasm at compile time to see some vauge example, although this does omit a few things like temps such that it won't be directly compilable. the asm statements themselves should have the right form+names though, hopefully).

                the engine api is event driven. the language itself is not. you can write a mod without any consideration for the standard events. just make sure you provide the fixed entry points and you can do whatever the heck you want.
                function names and things need to be present in order for saved games to work. this means that quakec has rich reflection features and contains much more info than would be present in equivelent C code. function, fields, and globals are not a problem. locals are, however.
                you may be able to generate meaningful names for those locals, but they should at least only be used in a single function so just using numbered names will probably work too.

                fixing bugs can be done without changing the progs itself. the advantage of this is that the same fix can be applied to any mod with a bug without having to explicitly fix every mod by hand. this can be done either with engine hacks (see some of baker's fixes), or via something like fteqw's addons/mutator feature which can override and thereby wrap functions in other progs.dat files loaded in the same vm instance (note that this can also help get around potential license limitations).

                @rook, I'd prove your undecompilable claim wrong, but I'm too lazy.
                that said, 'for' loops will generally confuse decompilers enough that the function that they're contained in will make no sense or at least not recompile. creative use of gotos can also really confuse things in the same way. the ternary operator can need special handling too.
                Any admins could make built-in cheats for themselves by editing the engine too. Changing the engine constantly reset their health field to 6000 should just about do it. The 'aim' builtin can also be quite interesting too.

                @damage_inc, yes, its a shame that they do, but it is an option available to them, and many feel very strongly that they should do this. see r00k's concern about not having permission to release the source of parts of his mod.
                Last edited by Spike; 11-28-2014, 08:25 PM.
                Some Game Thing

                Comment


                • #9
                  @Spike, true its not a rock solid deterrent its something that will break frik s decompiler without any modification. i have provided the source to people who have asked for no other reason just to learn from. I'm always open to share code for simple learning purposes. the game is so old the competitive players have all but vanished anyways. i had planned on recoding jpgs portions and migrate the code to fte but the way the online scene has dwindled it seems like the effort would go without notice.
                  www.quakeone.com/qrack | www.quakeone.com/cax| http://en.twitch.tv/sputnikutah

                  Comment


                  • #10
                    Oh yea, I did confuse disassemble with decompile. I'm surprised that you can get source code from any progs.dat, even for unprotected progs.dat files. I'm surprised function and variable names are stored in progs.dat, I just noticed the first 1/4 or so of the dat file are names. I suppose one could obfuscate code by changing names to non-descriptive versions, if they wanted to make it difficult for others to figure out what's going on with their code. Google does this all the time with the JavaScript on their pages.

                    I'm used to dis-assembly. I've made a few CLEO mods for GTA3 engine which is just like assembly. Apparently Rockstar games has a high level code compiler that compiles to virtual machine code, but they never released it. CLEO is basically an assembler for game scripting. What is awesome about CLEO is the ability to run multiple scripts at the same time. It would be cool to do the same with QuakeC mods!

                    So functions are a series of statements. Each statement has an op code with 3 vars (a,b,c) where c is usually the result. I assume -fwasm converts to these opcode/operand statements?
                    Last edited by Hypersonic; 12-05-2014, 12:15 PM.

                    Comment


                    • #11
                      Originally posted by damage_inc View Post
                      I find it ironic how J. Carmack purposely opened up, completely, a commercially released game/engine and yet mod authors would purposely keep their hobbyist sources closed!

                      wise words

                      maybe some people maintain the closedsource because is full of ugly hacks
                      the invasion has begun! hide your children, grab the guns, and pack sandwiches.

                      syluxman2803

                      Comment


                      • #12
                        Originally posted by nahuel View Post

                        wise words

                        maybe some people maintain the closedsource because is full of ugly hacks
                        In this case I agree. I see no need for closed source in Quake mods.
                        Scout's Journey
                        Rune of Earth Magic

                        Comment


                        • #13
                          @hypersonic, -Fwasm causes fteqcc to write out the assembler code for the functions that it generates into some qc.asm file in the working directory.
                          you can also enable it on a per-function basis with '#pragma wrasm 1' and then switch it off again by setting to 0.
                          I did toy with the idea of getting fteqcc to initially load up an existing progs.dat and compile new code into it, but I gave up when it didn't work properly. part of the problem is that there's no reliable way to determine argument types and stuff, so existing functions etc are not prototyped reliably enough.

                          @nahuel... how very true...
                          Some Game Thing

                          Comment


                          • #14
                            I just manually assembled the function ImpulseCommands (machine code to a form of assembly) from the progs.dat in pak0.pak.

                            One thing I noticed is that although ImpulseCommands function has no parameters, locals, nor return values, it still declares a parm_start. It uses the parm_start address as the start of a scratchpad. It doesn't erase anything from the scratchpad as it uses it. This makes sense as it might need info from the scratchpad when it calls then returns from another function.

                            Another thing I noticed is that when compiled if statements turn into if_not statements with a statement offset if not true, similar to CLEO for GTA.

                            Yet another thing I noticed is that it doesn't store immediate values in the statements, rather it refers to an address where they are stored.

                            I plan to study ImpulseCommands further before I move on to functions with parameters, locals, and returns. I want to find out where these immediate addresses refer to.

                            Comment


                            • #15
                              params+locals form a single contiguous block, starting with the first param. if there's no params, then of course it refers to the first local instead.
                              when a function is called, the engine copies the values in the OFS_PARM* offsets into the new function's locals, so the param sizes are important while the types themselves are not interesting and may not be known.
                              qc's stack is basically an after-thought. each function has its own set of variables. the engine copies the current values out when its called, and restores them when it returns. this provides support for recursion, and has the side effect of restoring locals to their initialised/0 value. locals can thus fail to be initialised properly if there is another instance of the function higher on the stack (thus causing that parent instance's locals to override the child's initialised values).
                              often, temps are not marked as locals, but can appear as if they're hidden globals used in a single function. this does not harm the calling convention so long as they're not live over the course of a function call (fteqcc refers to such temps as 'locked' in that case). trying to keep temps as globals instead of locals means function calls are a little faster as less needs to be preserved.
                              some qcc optimisations can completely strip all local+param info, making their types hard to determine and their names. this also prevents a decompiler from determining if a local is a temp or not.
                              functions have no specified return type. the vm just doesn't care. the statements will refer to OFS_RETURN using the appropriate type, you may also be able to determine a type for the a arg of op_return/op_done, but don't expect it to always be correct.

                              the only values in instructions that are immediates are the offsets for jump instructions (and conditional jumps). jump tables cannot be implemented (except as a sea of conditional jumps).

                              not really sure if there's anything else helpful to mention.
                              Some Game Thing

                              Comment

                              Working...
                              X