demo file loading & saving with pointers.
category: code [glöplog]
Hi, just scraped up my old demotool or engine-sources.
I turned off ASLR (Address space layout randomize). but im doing a trick for this because basepointer is still changed: get pointer-address (of anything in the memory) and subtract it with process heap location (GetProcessHeap()). when loading, do the same trick. not sure if it will work in the long run, but for now its working.
anyone done this before? and is it safe or
and/or what do you use, ID's? it doesnt matter what it is for (texture, generators etc.) it should be used for anything.
I turned off ASLR (Address space layout randomize). but im doing a trick for this because basepointer is still changed: get pointer-address (of anything in the memory) and subtract it with process heap location (GetProcessHeap()). when loading, do the same trick. not sure if it will work in the long run, but for now its working.
anyone done this before? and is it safe or
and/or what do you use, ID's? it doesnt matter what it is for (texture, generators etc.) it should be used for anything.
Is putting your data in a .dll (or .so for UNIX-ish platforms) and letting the system relocate it and adjust the pointers an option?
Why would you ever do something like this?
What about doing what a normal person would do and load your data from disk before the demo starts (from separate files or, if you want, cooked/packed formats).
If you still insist on being weird, take a look at how executables are composed and use the OS API facilities to get to the right chunk of memory. But really, just don't.
If you still insist on being weird, take a look at how executables are composed and use the OS API facilities to get to the right chunk of memory. But really, just don't.
Serialization is a bitch, isn't it.
Basically that kind of calculation should be resolved by the linker and pe loader:
Never had a problem with using incbin or some binary-to-c-hexcode-array tool for including static data directly into executables so far.
Never had a problem with using incbin or some binary-to-c-hexcode-array tool for including static data directly into executables so far.
.incbin?
rudi, per vognsen has a video showing your idea: https://www.youtube.com/watch?v=qZM2B4D7hZs&t=2749s (A memory mappable, zero-copy asset format for real-time graphics)
One cool trick I picked up from some Jon Blow's stream could be useful here. It's somewhat related to the technique used by Per Vognsen in the video linked above.
When serializing some struct allocated on the heap you also write its this-pointer to the file. When reading the data back from disk you then keep a mapping from old this-pointers to new ones that are valid in the current heap:
So you are effectively using this-pointers as object ids that are guaranteed to be unique.
After deserializing some object, you add it to the map:
During deserialization you can then easily query the correct addresses:
I haven't really tried it in practice so I can't tell if there are some gotchas you need to know :)
When serializing some struct allocated on the heap you also write its this-pointer to the file. When reading the data back from disk you then keep a mapping from old this-pointers to new ones that are valid in the current heap:
Code:
// 64-bit addresses
unordered_map<uint64_t, void*> pointer_map;So you are effectively using this-pointers as object ids that are guaranteed to be unique.
After deserializing some object, you add it to the map:
Code:
Object* obj = new Object(stream); // constructor reads new fields from stream
uint64_t old_this = stream.read_uint64(); // assume this-pointers are always stored after each object instance
pointer_map[old_this] = (void*)obj;
During deserialization you can then easily query the correct addresses:
Code:
struct Object {
Object* other;
Object(some_stream_object_t stream) {
uint64_t old_other = stream.read_uint64();
// we assume the child objects are serialized before parents
other = (Object*)pointer_map[old_other];
}
}
I haven't really tried it in practice so I can't tell if there are some gotchas you need to know :)
thanks for the replies.
_-_: ok video. he mentioned converting pointers to references or offsets. that something in the line of what i was looking at..
cce: yes, the map-trick seems interesting. i didnt think about that.
_-_: ok video. he mentioned converting pointers to references or offsets. that something in the line of what i was looking at..
cce: yes, the map-trick seems interesting. i didnt think about that.
the map trick is used by blender for reading/writing its files.
Its serialization stores objects with their native pointers + a descriptor of all the datastructures, and loading documents upgrade them to the current version using the descriptor (only if necessary) and maps old pointers to new pointers as necessary.
Its serialization stores objects with their native pointers + a descriptor of all the datastructures, and loading documents upgrade them to the current version using the descriptor (only if necessary) and maps old pointers to new pointers as necessary.
ah and Per Vognsen video is supported by this gob.h file:
https://gist.github.com/pervognsen/c25a039fcf8c256141ef0778a1b32a88
https://gist.github.com/pervognsen/c25a039fcf8c256141ef0778a1b32a88
Quote:
So you are effectively using this-pointers as object ids that are guaranteed to be unique.
So why use pointers then?
Because you don't need to explicitly serialize, just load int mem (or even map with COW which is way faster) and fix up the ptrs.
Honestly that seems like a minimal amount of gain for something that ultimately end up being fiddly code.
Yeah, personally I'd go for "arrays of structs and indices+magic instead of pointers" too but I can understand the appeal of basically not having to adhere to any specific structure for your data.
kb: Is really mmap faster than just loading? (And if your data set is huge and you're not going to use nearly all of it: Explicit load-on-demand.) mmap is one of those ideas that seem really attractive until you try it in practice and find all the quirks.
In demos, are there benefits in storing *all* resources as single file / in executable?
Waffle: yes. Depending a bit on what you do, it can be faster to read the entire file into memory and then perform whatever unpacking/decoding stuff that you need than to load individual files. For my uses, I've never bothered though.
Also, the mmapping thing: wouldn't that be a royal pain in the ass to debug if/when something goes wrong somewhere and you're trying to cram some random data into a data structure which ends up being filled with garbage? Also, portability might not be an issue but it sounds to me like there's a high chance of things not working when moving to an another platform. Or an operating system?
Also, the mmapping thing: wouldn't that be a royal pain in the ass to debug if/when something goes wrong somewhere and you're trying to cram some random data into a data structure which ends up being filled with garbage? Also, portability might not be an issue but it sounds to me like there's a high chance of things not working when moving to an another platform. Or an operating system?
Sesse: At least under Windows mmaping is way faster than simple loading, at least when I've tried, yes. Consider it - when you use ReadFile, the system has to allocate some cache space, load from disk into cache, then memcpy into your destination buffer. With mmap, that last memcpy (and the associated cache shenanigans) doesn't need to happen.
Next advantage (especially on x64 where virtual address space is like almost infinite, man): Your loader implementation, or let's say Snappy decoder :) doesn't need to care about buffer boundaries and can just assume all the data is in contiguous memory. That also translates to a slight speed advantage due to reduced complexity. And especially in demo coding it saves a shitload of time spent on the annoying parts of your hobby :)
Or, as an alternative, you can use uncached overlapped reads. Those are also as fast as the disk gets. But you get the buffer handling problems and some nice new lignment issues back, and it takes some time to get it right.
Preacher: mmap is really just a fancy way to fread(f, buffer, file_size), and this should at least work within the same CPU architecture (alignments, endianess, etc). Switching OSes shouldn't make a difference.
Next advantage (especially on x64 where virtual address space is like almost infinite, man): Your loader implementation, or let's say Snappy decoder :) doesn't need to care about buffer boundaries and can just assume all the data is in contiguous memory. That also translates to a slight speed advantage due to reduced complexity. And especially in demo coding it saves a shitload of time spent on the annoying parts of your hobby :)
Or, as an alternative, you can use uncached overlapped reads. Those are also as fast as the disk gets. But you get the buffer handling problems and some nice new lignment issues back, and it takes some time to get it right.
Preacher: mmap is really just a fancy way to fread(f, buffer, file_size), and this should at least work within the same CPU architecture (alignments, endianess, etc). Switching OSes shouldn't make a difference.
But eg. the pack file in Masagin works this way - just mmap the whole pack, and set a pointer to the header with the directory in it. When "opening" a file from the packfile, just return the part of the memory where that file resides. That one was really easy to do, and it loads so fast that the loader bar at the beginning was entirely fake :)
https://bitbucket.org/paniq/masagin/src/23349a9ba924f6341bd7f1ed907b1b413b298342/libs/msglib/packfile.cpp?at=default&fileviewer=file-view-default
https://bitbucket.org/paniq/masagin/src/23349a9ba924f6341bd7f1ed907b1b413b298342/libs/msglib/packfile.cpp?at=default&fileviewer=file-view-default
I like how there's a big-endian fixup function and yet at the same time a (MSVC-specific) multi-character constant in the same source file :)
That constant isn't THAT specific. Let's say that particular piece of code also ran on one of the last-gen consoles (even if the mmap idea is pretty stupid there) :)
Quote:
I can understand the appeal of basically not having to adhere to any specific structure for your data.
Isn't having to fix up pointers specific enough though? Seems like a generic unserialization task.