Announcement

Collapse
No announcement yet.

Verify checksums for quake files

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Verify checksums for quake files

    Hey,

    I realize this may not exactly fit here but I thought others could use it. Quaddicted.com has a list of checksums and filesizes for Spirit's single player archive:

    http://www.quaddicted.com/filebase/f..._checksums.txt
    http://www.quaddicted.com/filebase/filebase_sizes.txt

    I wrote a small perl program that takes those checksums and sizes and uses the system's locatedb to find matching filenames. It runs a SHA1 checksum if the files are the same size. It outputs which files have the same name but aren't the same size.

    It's more likely that the ones that are different sizes are corrupt but it gave me too many false positives because some maps have generic names. In a new version, I will probably let the user specify which directories to look in so I can assume that files named the same must be the same checksum. There will still be false positives but much less than checking your whole system.

    It's kind of neat to verify you have the same copy of maps and also you can tell how many maps you have that someone else does. You can also tell if you have duplicate copies. I'm always hesitant to delete files named similar things without verifying checksums because one could be different than the other. This tells you if they are the same using the checksum.

    Would anyone here be interested in this? It's not just quake specific. You could use this with any set of checksums and filesizes. It's Unix only at this point. I'm not aware of a Windows locate program that's standard.

    Baker: If you think this would be useful, could you host it? I have no webspace. It's GPL v2 or later. There are two versions: One that uses all system commands ("locate", "sha1sum", etc) and another that's much faster but requires extra perl modules.

  • #2
    Originally posted by kbdrate
    Baker: If you think this would be useful, could you host it? I have no webspace. It's GPL v2 or later. There are two versions: One that uses all system commands ("locate", "sha1sum", etc) and another that's much faster but requires extra perl modules.
    We'll give it a home. I can't imagine it being that large either. It might be tomorrow or Sunday before I get back with you on that.
    Quakeone.com - Being exactly one-half good and one-half evil has advantages. When a portal opens to the antimatter universe, my opposite is just me with a goatee.

    So while you guys all have to fight your anti-matter counterparts, me and my evil twin will be drinking a beer laughing at you guys ...

    Comment


    • #3
      Send the file to me and I can throw it up... is it just a basic perl script or does it require any sort of extra packages or anything?? steve _AT_ quakeone _DOT_ com

      Comment


      • #4
        Vandal!
        Last edited by Sajt; 05-29-2006, 12:32 AM.

        Comment


        • #5
          Originally posted by Solecord
          is it just a basic perl script or does it require any sort of extra packages or anything?
          Both. compare_files_easy.pl only uses the core module File::Spec to break up file path components.

          compare_files_fast.pl uses File::Spec, Digest::SHA1, Digest::MD5, File::Locate, and IO::File. It's faster because there's less overhead using these perl modules instead of calling executables and parsing their output.

          The files are independent of each other. On my system, the easy one takes 18 minutes and the fast one takes 8 minutes. It takes so long because it has to go through every file locate reports with the same name, check the filesize, and then run a checksum on any file with the right name and size. There are about 1,000 files in his list so it takes a while.

          In my test runs, I found one file that Spirit and I both have with a different checksum. It's the same name and filesize. I'm waiting to hear from him if they are just differences in versions (one newer or older) or if one of ours is corrupted.

          Comment


          • #6
            Heh, I already upped it as soon as I had it
            http://www.quaddicted.com/filebase/c...uake_files.zip

            My tigcat02.zip was corrupted, I replace it with a valid zip.
            Quake 1 Singleplayer Maps and Mods

            Comment


            • #7
              Just wanted to tell you that Quaddicted.com will have checksums for all files on their page/entry soon. At the moment I am working on getting filelistings (what is inside the .zip) and the appropriate checksums. md5 though, since I found a nice and easy Windows tool for that.
              Quake 1 Singleplayer Maps and Mods

              Comment


              • #8
                Originally posted by Spirit View Post
                Just wanted to tell you that Quaddicted.com will have checksums for all files on their page/entry soon. At the moment I am working on getting filelistings (what is inside the .zip) and the appropriate checksums. md5 though, since I found a nice and easy Windows tool for that.
                I just went to Quaddicted.com... and I can't even figure out what the site IS! Someone needs to work on basic site layout and structure.

                Comment


                • #9
                  Originally posted by Spirit View Post
                  md5 though, since I found a nice and easy Windows tool for that.
                  In general, people are moving away from MD5. It's not really an issue here because we aren't too concerned about maliciously modified files. Most new apps will be using SHA-1 or another algorithm (or SHA-1 with more bits).

                  There have to be easy to use SHA-1 tools if you're not against using something different. What makes it easy about the windows MD5 tool?

                  Comment


                  • #10
                    TheMachine - Quaddicted underwent a hacking attempt recently (this summer) and was down for a while. It is not currently in a transitory state, and thus not very apparent in its design. It is a well regarded site for its archive of Quake single player maps, as well as texture wads and random cool stuff.
                    Inside3d - Because you can't be Outside 3D!

                    Comment


                    • #11
                      Well, I did choose md5 because I found this program which is just perfect: http://www.brandonstaggs.com/filecheckmd5.html
                      It needs no installation and works fine.
                      I am playing around with JSummerX (http://sha-1.qarchive.org/) at the moment. It's nice because it runs in Java and it is GNU GPL. And it supports md5, sha-160 and sha-256. Very negative is the very high cpu (100%...) usage when checking files from a checkfile.

                      TheMachine:
                      Scar3crow: Thanks.
                      Quake 1 Singleplayer Maps and Mods

                      Comment


                      • #12
                        Spirit - Not accepted! I demand a million thanks for the capitalization of the s!

                        BTW, come to #qc sometime, we must discuss tanz metal!
                        Inside3d - Because you can't be Outside 3D!

                        Comment


                        • #13
                          scar3crow: http://unix.lt/thankyou.swf
                          I decided not to use IRC much anymore. It ate so much of my precious time.

                          JSummerX is a very stupid program. I need to find something else. Or I'll just stick to md5 as I can't see any advantage of a different checksum anyways.
                          Quake 1 Singleplayer Maps and Mods

                          Comment


                          • #14
                            Fsum will do both MD5 and SHA-1, as well as a few others. The only disadvantage is that it's only free as in beer, not free as in speech...
                            16:03:04 <gb> when I put in a sng, I think I might need nails
                            16:03:30 <gb> the fact that only playtesting tells me that probably means that my mind is a sieve

                            Comment


                            • #15
                              Originally posted by Spirit View Post
                              Or I'll just stick to md5 as I can't see any advantage of a different checksum anyways.
                              Just stick with MD5.

                              It's not really an issue in this case. It would be if you were designing a peer to peer network because you want to make sure someone sends valid blocks. In this case, we have a level of trust in you and we're just using MD5 to verify the file is the same.

                              I was just saying in general people are moving away from MD5 but it's fine in this case.

                              Comment

                              Working...
                              X