http://www.youtube.com/watch?v=MPQDEKMHk5s

InPlace is 100% efficient, but it is very hard to maintain across platforms and between runtime & editors. You have to save all the class (although in lot of scenarios you could save only a small percentage). Compression probably will eliminate all that redundant information, but anyway it doesn’t seem the right way.

My current approach is a Serialization System, where you Serialize the member that you want and where a schema file is generate for debug versions of the data. The final version doesn’t use schemas, is a raw of the data in the same order they are read and where a lot of optimizations can be applied, for example a std::vector of POD can be read with a single fread operation. I have to test it, but I am almost sure I will reach almost the same speed that the inplace with this approach.

Actually, my main concern sounds like the opposite; raw data is typically taking more space than complex C++ objects, so load-in-place performance effect, positive or negative, might be negligible (compared with loading-in-place only parts of objects). I would then prefer the most productive approach in day-to-day programming.

I believe in IDL solutions like Bruno mentionned; it can generates a lot of useful code for both tools and engine. But I still doubt full load-in-place as a single solution. The IDL can actually be used to do the exact opposite: control how to serialize a class. Programmers can then end up with more flexibility in their code by dealing only with IDL for serialized members, whatever how. Combined with some form of C++ reflection (can be generated, can be meta-prog, can be both combined), it’s possible to save any object in C++ engine in tool-mode. In a sense, C++ reflection, IDL and load-in-place are all orthogonal.

I now work in an environment with mixed solutions like you describe, and I tend to prefer it. It’s probably why data management is not a popular topic as you said; we just can’t agree on one ideal solution;)

Cheers,
Nic

We load compressed packages of fairly optimized data structures, and in the future we’ll try to use as much procedural data as possible. You are right, it is good to balance CPU time with loading time. Compression is crucial, in any form, procedural techniques or generic LZ, and that will make good use of all those processors we now have. Something we’ll need to improve in time, but (I think) orthogonal to LIP.

I think the I/O problems many people have are due to seek times, and the parsing-less nature of load-in-place is still desirable.

Yeah, this article bears more than a passing resemblance to the talk you guys did at GDC.

I’ve come to regard load-in-place as somewhat of a lost cause lately, though, since we’re loading so much data this round, that it’s probably not as effective as it could be. Of course, it all depends on your requirements, but as the author points out, this method has a huge IO overhead and virtually zero CPU overhead. When the player is staring at the loading screen, I think we should do whatever we can to reduce the IO load even if it costs us 100% of our CPU. As such, loading less data (e.g. “prototype data”) and constructing the objects from that may be more effective going forward.

For “large streaming worlds” (which we started doing on a PS1 game I was working on in 1996), you should do everything you can to have zero CPU overhead, so then I’d say you should try to load data that functionality consumes rather than full-blown objects. The requirement to call placement new on every object is overhead that’s just wasted in my opinion. Load PODs off of disc, and do as much or as little processing as you need to get that data into your game system.

I think that data management is a serious topic for modern games, and sadly one that goes undiscussed during a game’s development, but the solutions are going to vary by the problems, and I would not be surprised if the “best” solutions for a game were as varied as the scenarios that need to load data. For my current game, for instance, some data gets cooked right into the executable, some gets loaded out of bare files, some gets loaded out of WADs. Some of the loaded data is further processed after load, and some isn’t. I don’t believe that a “one size fits all” solution is generally appropriate.

Load-in-place definitely has its place, though, so my hats off to all of you guys for doing the presentations and articles.

Simple data representation, fast loading, runtime memory defragmentation, we’re happy.

The system also needs to have a complementary xml-binary-like format that is backward-compatible. The tool-mode engine needs to be able to save both this format and the load-in-place format. Keep up the good work, it would be nice if at some point we stop reinventing the wheel and share such code.

Regards,
Nicolas

The point of this article was to release a simple implementation of Load-In-Place that anybody could download and start playing with it. I spent more than a month working in the source code (after trying several implementation) starting from scratch. So please, don’t say this is a ripof of anything.

When you publish something like you did, I think you expect others to continue or take some of your ideas to avoid reinventing the wheel. Your article gave me a lot of nice information to solve this problem and due to that I am including a link to your Presentation. So everybody reading my article can read yours. I think this is honest but if you think you deserve more credit for that, we can definitely talk about that.

About the references to your article and others in Gamasutra: it has been an unintentional error that is going to be fixed. I already sent the information to the Gamasutra staff. Sorry for that.

I started with a simple implementation with only simple structs (no polymorphism, no pointers, etc) but it was not too useful. We indeed already had lot of data that was represented using C++ object. So we would have had to do conversions from simple structs in disk to complex C++ objects.

I am using this to implement continuous streaming of data. At the end, you cannot stream continuosly at the byte level. You pack your data in packages and your unit of streaming is the package. That package is loaded in place.

I am going to write an article about this.

When you say “the tools use one that rebuilds a nicer class structure that has STL containers embedded in them”. How is that class generated? Dynamically or it is statically defined by the author or the class?

I started to implement a version of this that encapsulate vector, list and other containers with the same interface that the stl equivalents but your idea seems interesting.