Cross platform binary files


Depending on what kind of data you have in your structs, it can be very not difficult or a bit tricky. Bitfields, or data being reinterpreted from a different type are especially tricky. Linux reader to successfully read the copied over file? Unlike the hacky approach, this is hard to get wrong. XP machine, on my Linux machine. Not to mention, not difficult to get wrong. This is a number with a known value for a given endianness. Not really an OpenGL question, but I hope it helps a bit. Keep all these typedefs in a single header, with conditional compilation blocks for different platforms. Microsoft used a similar approach anyway, probably because it was proven and tried.


Apple PCs, and even there it seems like they only used them because they needed to transition from PowerPC to x86. And standards like POSIX or Linux Standard Base also help. Win16 a third subsystem was a straightforward choice. The costs of bandwidth favors delivering just the minimum necessary bits for a particular software package. FatELF exists but is not very successful. Fat binaries made more sense when software was shrink wrapped physical media. With all these changes, it was better to keep the 32 and 16 bit worlds separate. On Linux, both criteria would still apply if you want to support both 32 and 64 bit on a single Linux distribution.


But why bother, if you already have to support multiple distributions? Shipping a fat binary down the wire degrades both customer experience and seller infrastructure efficiency. Interoperability, transparency, extensibility, and storage or transaction economy: these are the important themes in designing file formats and application protocols. When you feel the urge to design a complex binary file format, or a complex binary application protocol, it is generally wise to lie down until the feeling passes. IO code with a member named Disch for about a year ago. Interoperability and transparency demand that we focus such designs on clean data representations, rather than putting convenience of implementation or highest possible performance first. What I learn is that if you need to store data in files that will be read and written to on different machines, you have to define in the program what endianness should be used. Extensibility also favors textual protocols, since binary ones are often harder to extend or subset cleanly.


The above has simply copied the literal contents of memory to a file. Have the rules changed since last year? Correspondingly, input is the conversion of a sequence of human readable characters into the internal representation of an object. IO functions detect what endianness should be used on the current machine that is running the program? For example, if I needed to store 4 bytes, I had to do this manually with my own functions and define in those which endianness is used. Within a more general context, a file can be considered memory and have endianness. Each chunk or run of at least ten equal bytes is looked up in the tree. Other tags encode references to data present in the original file. Before expanding the match, the linked list is checked.


Binary file comparison is useful for many applications. From the byte values with a mean block length between 20 and 80 bytes, I choose the one with the lowest standard deviation. The Symantec project file should contain: BinDiff. The algorithm in BinDiff tries to find equal chunks in the two files being compared. The source file contains multiple versions of the program; you can change compilation variables according to the version being compiled. BinDiff simply displays a message after startup. The new match is better because it is bigger. Dividing files into lines lets you index the lines in one file, then use the index to find matches with lines in another file.


In such cases, it may be desirable to create the diff file on another platform and use the resulting file under DOS. If there is no such byte, I gradually loosen the limits on the block length, trying values from 20 to 130, 20 to 180, and so on. Once both files are divided into chunks, BinDiff builds an index and begins matching chunks from both files. DIF as file type. The linked list of matches is kept sorted on starting position in the updated file. Finally, the diff file could be compressed. Such were the requirements when I created BinDiff, a utility that intelligently compares two versions of a binary file and creates a diff file. Macintosh files differ in that they are composed of the data fork and the resource fork. DOS, and the Macintosh. Figure 1 Matching both forward and backward.


Some tags are used as headers to data included in the diff file. Next, consider the possibility of using the zero delimiter when no good delimiter is found. Overlapping matches are cut off, and if a match drops below six bytes, it is removed from the list. If six or more bytes match, the match is stored. Also, more data could be read into memory. The zero default is probably one of the worst choices, but it is very rarely used, normally occurring only on very small files.


DumpDiff writes the tagged diff file. Due to the file format used, matches of less than six bytes increase the size of the diff file instead of reducing it. For instance, the unbalanced tree can become a balanced tree. The first few bytes to compare do not have to be read from disk. One match can enclose another without being expanded: Enclosing matches have unequal distances; expanded matches have equal distances. On the other hand, using a diff file to update the file is very lightweight and fast. The algorithm to choose a suitable delimiter involves simple statistics.


On most UNIX systems, you can compile with the following command line: cc bindiff. The diff file consists of a header and a sequence of tagged entries. The data fork corresponds to a file on the other platforms. The diff file contains two separate, sequential diff files. The tags are four bits in size and are contained in the lower four bits of a byte. This prevents linking problems.


On a low level, the resource fork can be seen as just another file. BuildTree scans a file and builds the chunk index tree. My version of fopen has an extra option that lets you specify which fork to open. Macintosh diff files from the data fork are usable on other platforms; those from the resource fork are not. The updated file is read and chunked using the same delimiter. Choosing the lowest standard deviation yields the byte that most evenly divides the file; the blocks are more or less the same length. In each node, I also store some bytes from the file to speed up comparisons. CRCs give more security against using a diff on a nonmatching original file, and against diff file corruption. After the match list is built, the complete match list is reviewed.


Larger chunks are encoded by using more than one tag. There is also a block of bytes or two, three, or four extra bytes to encode a chunk file position within the original file. Under DOS, such a process can not difficult exceed the 640K limit. BinDiff then uses an indexing algorithm to find matching chunks so equal chunks need not be in the same sequence in both files. The upper four bits, together with zero, one, or two extra bytes, contain a chunk size. You can optimize BinDiff in several ways. If the match is already present in an expanded form, it is not expanded or added; if no expanded form of the match is present, it is expanded and inserted into the linked list. At the receiving side, this diff file could be used to update the original file.


FindDelimiter checks the tables built by ScanFile and chooses a suitable delimiter. Because BinDiff is insensitive to Endianness, diff files created on one platform can be used on another. Finally, the diff file is written. Another optimization is to use CRC instead of the simple checksums used for checking the files. When a lot of memory is available, more of the file should be read into the tree. ExtendMatch extends a match forward and backward, until the first nonmatching bytes or the file limits. Each tree node contains a left and right pointer. In BinDiff, the resource fork is viewed as a second file.


One example is sending updates of large files over a communications line: Instead of sending a complete update each time, you could send the complete file once, then create a diff file containing the differences between the original file and the updated file. While inserting the new match, all matches enclosed by it are removed from the list. For each possible byte value, I calculate the mean length and standard deviation of the lines in the file. In the rare cases that this does not help, 0 is used as the byte value. If an ANSI compiler is available on a particular UNIX platform, it can be activated with one of the conditional compilation switches. Every match of sufficient length is expanded as much as possible and stored in a linked list of matches. Creating a diff file is processor and memory intensive. Any information would be good at this point, but the specific thing I would like to know is: Considering the lines below, why is the binary not being read in as the same value as it is written out as? How can I fix this? How do I use this?


SQLite require a disk file to operate on. You cannot stop a determined hacker. First there is the problem of what you do if someone gets access to your script. And does it make it safer? With this concept something as simple as a serialized hash could contain the encrypted password data. Changing the XOR key with approximately the same frequency as the passwords are changed defeats this. Why would you ever need to decrypt it? Expat and do not have root access to install sun freeware packages. Much more serious is the fact that one time pads only work if you only use them once. If you encrypt individual parts of file separately, you make it easier to break the encryption. SQL engine is overkill for this.


If the file is repeatedly acquired by other people, and changes often, and changes in ways which allow multiple guesses of passwords to expose some of the XOR key, then yes, parts of the XOR key can be obtained. XOR text if you XOR the file with banana, since there is no repeated XOR text. Why not use the good old one time pad. OP here: just to provide some additional info on the audit requirement. When in doubt you can always hire a specialized company to assess your security to profit confidence in it. Once, that is, for every time the file is modified. Homebrew crypto is a bad idea. DB on one server and distribute the DB file to the others without having to modify it. With all that in mind, I think the best solution is what moritz suggested: use YAML or Storable and encrypt the whole file.


You will have to change the password with some frequency and device a method for that. Next option would be similar to above but wrap around an SDM_File instead. You could then limit access to the key files with ACLs. YAML or Storable, encrypt the whole thing and dump it to disk. If so, the obfuscation is really superfluous. The systems implementation, procedure and policy usually left gaping holes to exploit before needing to crack the crypto. For the record: the DES algorithm is often implemented in hardware because it outperforms the software implementations by a factor.


Last but not least there is the problem of Key management which is normally the most difficult component of the entire cryptographic system! LDAP, web server, database password are encrypted with seperate keys. Check out past polls. Edit: yes, using another crypt module is probably a better idea but perhaps harder to achieve given his platform constraints. So that you could use the password? Just encrypt the file using an algorithm that is available on all of the platforms you need to support. But that is minor. There are standard encryption mechanisms that are so good as to be effectively unbreakable. This is a more legitimate concern.


During the cold war the hotline between Washington and Moscow was based on something similar. If it is changed, it will no longer match the signature. Suppose someone gets access to your data. DB file, so that meets the encryption requirement. Your sysadmins will, and its their job to install stuff that users need, after checking that you really do need them. Thanks to all for your help!


You will have the XOR key but not know it. Then you would access whatever password you needed by name via hash element. CPAN module to implement it. Does it really make sense to update the key in the script every time the file is modified? But where do I store the key to decrypt that file? Does anyone know of solutions for this that might already exist in CPAN? So here are my current thoughts: Create a module to wrap around an SQLite DB. Or do you want to encrypt the passwords in that file, but not the file itself? ACLs to control access to the file, in addition to the other safeguards in place. When I majored on this stuff DES was safe.


BTW: the security requirements of DES were kept secret by instigation of the NSA! If you do encrypt the whole file, you may have issues using SQLite. Anyone who has access to the script has access to the data, whatever scheme is used. If you decrypt an encrypted SQLite database file and write a temporary plaintext version to disk, you render all the encryption pointless. All they need to do is take a small dictionary of common passwords, xor it against 100 passwords, and look for some piece of xor text popping up more than once. Essentially if my code needs a set of credentials to access some service, those credentials should be stored in a separate file not subject to version control and not in the path of the web server. Password, but as far as I can tell the credentials are stored in plain text in perl modules, and this would not help for web services, LDAP, etc.


Gotchas here would be the length restriction of the DBM files, and is SDM_File always going to be portable across environments? Well then, if you need compilers to do your job then your boss will need to get you compilers. XOR your data file with that. And is it reliable? You can, however, help to keep honest people honest. The scripts could only access the keys corresponding to the passwords they need. If you are doing this anywhere, be assured that any serious security audit should find the fact that you are doing this, and you will get a lecture about how wrong you are. Lots of people use very bad passwords.


IBM Power SystemsTM, Linux, and Windows servers. Keep in mind there are a lot of libraries that already provide platform headers for things like integer widths. OpenGL will take them in as GLubytes and GLuints anyways, which is going to be platform specific. SDL also has functions for dealing with endianness. One of the things I want to store in my binary files are OpenGL color values. IEEE 754 floating point types may have incompatible binary representations. Aside from the endian issues for versions of OSX still running on PowerPCs, should that be pretty safe for the platforms above? SDL both contain integer width typedefs.


But is there a way? Endianess: Obviously this is an issue. And in this case, a byte would be at least 32 bits. Are there any other common platforms that people would hope to play a game on? Also rather than rely on packing pragmas to serialize structures, you can just serialize and deserialize the individual members one at a time. Currently I have every piece of data casted to chars, and write the chars. That will fail to compile because of no definition for uint8_t. Individuals values in binary stream may not be properly aligned for processor. Is that pretty good coverage for a game?


Is a GL typedef for four byte unsigned integers that can vary by platform. Unfortunately, the only portable way to serialize floats is to do so by converting them into text. Possible solution: Remake the datafiles on each platform I compile on? This is a perfectly reasonable thing to do. Blender Community to have an AI system, with a particle engine, GUI HUD system, to implement into the BGE. After you mention of using the SDL_Image formats I looked up the book Focus on SDL. PhysFS thanks for the advice I am certainly going to check that out. So I was trying to kill several birds with one stone. Sounds like a really good idea and makes a lot of sense. The file is an uncompressed TGA and it runs and works in windows. Thanks for setting me straight.


Mac OS X to test the code on it, but it should work on a Mac also. One more thing I just noticed. So I suspect the frame rate will go way up. The file path problem took care of other programs like the OBJ loader and the FBX Loader. For more advanced trainees it can be a desktop reference, and a collection of the base knowledge needed to proceed with system and network administration. Open source interface designer for windows for cross platform programming? Sorry to waste your guys time on such a stupid mistake on my part but your comments did solve the problem. Note that registered members see fewer ads, and ContentLink is completely disabled once you log in. OpenGL books for the SDL as I prefer it over GLUT. Having a problem logging in? If you think of anything else I would be interested in hearing it. When I check it in debug mode in Eclipse I get 0 with nothing more in regards to the file variable.


If you need to reset your password, click here. Linux system and network administrator, trainer and consultant. They hope these examples will help you to get a better understanding of the Linux system and that you feel encouraged to try out things on your own. Is there some performance profit by using SDL_Image? Bullet into the system. Oh my gosh it was the path. Join our community today! What do I need to do in order to make my code compatible with Linux so that it will open binary files, and also be compatible still with Windows? You are currently viewing LQ as a guest.


Thank so much, who knows how many other days I would have been going crazy over such a simple mistake. Below is a bit of code where the problem is occurring. Windows and Linux, to eliminate a source of confusion. The uncompressed TGA file opens in Windows compile but not in Linux compile. This guide was created as an overview of the Linux Operating System, geared toward new users as an exploration tour and getting started guide, with exercises at the end of each chapter. Click Here to receive this Complete Guide absolutely free.


If you have any problems with the registration process or your account login, please contact us. Linux and Python in Linux before now. Are you new to LinuxQuestions. Of course you would never make a game in GLUT. Holycow I feel stupid. No error messages and no warnings for the entire project. The if statement is proving to be true when I run the program and get the output. SDL so I think I might give them a whirl later.


The sad part of this is that I rarely ever post questions on a forum about code, and I finally gave in after pulling my hair out over this error for some days now in my spare time. If this is not possible please let me know. Registration is quick, simple and absolutely free. By joining our community you will have the ability to post topics, receive our newsletter, use the advanced search, subscribe to threads and access many other special features. Linux, Windows, and Mac. Adding a passphrase to a file. Enter and confirm your passphrase for the encryption and click OK. Selecting the encryption key for your file.


EncryptPad on elementary OS. EncryptPad is an open source application that allows you to view and edit symmetrically encrypted text. EncryptPad generating a encryption key. EncryptPad can make that happen. EncryptPad generate a keyfile for us. EncryptPad offers the ability to encrypt with a passphrase and a keyfile. Save As and walk through the process of saving your file. If you know you want to use this same key for every file, you can check Persistent Key Location In The Encrypted File; if you want to use different keys for different files, do not check this box. Key In Repository and give your key a name.


After you verify your password, click OK and the file will be encrypted. EncryptPad entry in your desktop menu. Launch it, and get ready to encrypt your file. Your file is protected by both passphrase and keyfile. Open a terminal window. EncryptPad will also let you encrypt binary files on your local drive. Do that and click OK. You can click either No or Yes. Here is a thread that discusses some of the problems I ran into. Maybe a dumb question, but is that defined behavior?


Thanks as well, I tried looking up streambufs inner workings, but that got confusing. Standard C library FILEs.

Comments