template <class Archive>
static void load_and_allocate(Archive & archive, cereal::allocate<Clickable> &allocate){
Draw2D* renderer = nullptr;
MouseState* mouse = nullptr;
archive.extract(
cereal::make_nvp("renderer", renderer),
cereal::make_nvp("mouse", mouse)
require(mouse != nullptr, MV::PointerException("Error: Failed to load a mouse handle for Clickable node."));
allocate(renderer, *mouse);
archive(
cereal::make_nvp("rectangle", cereal::base_class<Rectangle>(allocate.get())),
cereal::make_nvp("eatTouches", allocate->eatTouches),
cereal::make_nvp("shouldUseChildrenInHitDetection", allocate->shouldUseChildrenInHitDetection),
cereal::make_nvp("ourId", allocate->ourId)
I'll probably implement this using a light-weight version of boost::any (working in a simple test) and throw all of this stuff under "advanced usage" for cereal. Debating the names for the two functions for adding/removing these things right now. I don't want the names to imply that this function actually serializes things.
Sounds like a good plan. Naming is hard, add and extract were the best I could come up with that satisfied a few constraints:
single word description
indicated the non-symmetrical nature
conveyed setting and getting information. I could have used set and get, but those words are commonly used for hidden member access in other situations and I didn't want to use them.
I think the one issue they have is:
Just reading the words a person might think they are added to the archive.
Maybe something more along the lines of specifying these as "parameters" would be good.
set_parameters
get_parameters
This kind of implies they are separate from the archive process. They are a bit more wordy, but are probably also more accurate.
Sorry for posting so late, but I don't get the point of this.
You (M2tM) want to load a bunch of objects and set pointers to other (external, unserialized) objects.
IMHO, this is NOT a thing to be included in ANY way into a serializer. And doing so would only encourage users to write worse/entangled code.
There are (at least) two standard methods to cope with the singletons myRenderer and myMouse:
Make them available as static/global variables and let the default constructors copy/reference those pointers (if they need it)
Have those singletons NOT referenced inside all your objects, but rather make them part of function calls like Draw( myRenderer ) and handleMouse( myMouse )
For other 'wiring' purposes, you can still come up with other ways to do it. E.g. just call
Init( yourPointersandStuff ) for the whole tree. Or more advanced, if needed.
Your other 'parameter' request, I do not understand atm., due to lack of time.
But in general my statement is:
Writing code compatible with 'standard' serialization == writing good code.
Just imagine you have to switch to another serializer someday...
Dude, they are not singletons. I could easily have two renderers, one for one window and one for another in a desktop environment (way more common on a Mac than PC, look at photoshop for example). They are also NOT GLOBAL.
For those two reasons your suggestions do not cover my use case. How about when I have game controllers? Like 4 controllers? A mouse can easily be considered candidate for global single instance, but when you drill deeper you really come to see multiple mice are possible, and multiple controllers are common.
Sorry, but that's just absolutely unhelpful.
A suggestion to make something global for no reason other than to get around an interface problem in loading/saving is unacceptably bad. This is counter to basic encapsulation and data hiding which is fundamental in software engineering for managing complexity in real world projects. If I were writing a toy app, fine, I can make it global. But I have about 20k lines of code (not a huge project by any means, but big enough and stable enough that it's a great test for a serialization library) and am making dozens of types serializable (for networking and for saving), making shit global to hook up references during the loading process does NOT cut it.
And finally, you cannot have a closed loop load/save and assume every aspect of a class can be written to a file and loaded back when you want to save different aspects of your application. That ideal does not exist in most cases. You're always going to have external dependencies at some point in your chain unless it's just a toy test case.
Apologies for the heated response, but your own was dismissive and uninformed. I am absolutely open to other suggestions to get around this issue, but I think you really need to see it as an issue before we can get to that. If you disagree, fine, I don't know what your needs are.
And if these changes don't make it in to Cereal that is also fine. I need the feature, however, and the beauty of open source projects is the ability to actually implement something you need.
I'd like to offer a bit of an addendum. I did not address the "Draw(renderer)" consideration, but the renderer itself contains an interface to draw to a window, but it also contains world to local transformation calls which are useful in creating bounding boxes (useful for collision detection, or mouse click detection), window dimensions which is useful for GUI node anchors, and framebuffer information which can be useful for render to texture functionality going on under the hood.
I construct a node with a reference to the Renderer because it is a legitimate reference requirement, passing it in through the Draw call is not enough, and I don't think it's an unreasonable requirement even if you might.
There are an infinite number of ways to design a system, I could do C style object oriented programming with passing dependencies in every function call as you suggest, but (in my opinion) a serializing library's job is not to force major re-factoring or framework redesign, but to facilitate serializing and reconstituting serialized objects and to provide an interface that makes this less of a pain in the ass than just directly using something like rapidjson.
Cereal is excellent, but unless you want to pretend objects don't occasionally require additional resources to perform correct construction, this (or something addressing this problem) is a necessary feature, and indeed is part of the serialization problem.
Consider this: Cereal could do less than it does right now, it could not handle std::shared_ptr for example, or we could have just said "Cereal doesn't handle circular references, handle those during the save process", and just say "oh, this is a user problem, have them recreate the objects" and it would still be a serialization library.
There is no one true definition of what is and is not in scope. But it's important to understand what Cereal is. It is an abstraction for saving and loading things to one of several formats, and it is a convenience tool to remove much of the boilerplate from that process and provide a stream-lined solution in that domain. My contention is that if it would be minimal effort to offer a clean interface to solve a problem in the domain of object serialization or reconstitution, and the work-arounds are hideous or difficult (and again, are only specific to the loading process)... Maybe that's worth considering.
Minimalism at the cost of function and the addition of complexity elsewhere in user code is really in contention with the purpose of a convenience wrapper library like Cereal.
Dude, don't call me 'dude' and calm down.
OK, say you have 2 different renderers. Say, they are not globally accessable. What you are trying to to is make them accessable by putting them into cereal. At least this is how I understand your posts and this, in my very humble opinion, cannot be the way to go.
I am also quite sure that there is a simple refactoring for you to handle this. And that it won't take more effort than passing things though cereal.
Just trying to help...
So I talked with @randvoorhies about this today and am a bit hesitant to implement this into cereal right now. His opinion, which I mostly share, is that this is stepping down dangerous feature creep, though I do see the benefit of such a mechanism, especially for things such as pointers or references.
If we did have something like this in cereal, we see a few ways of doing it:
Using the style described earlier in this discussion with a boost::any like mechanism, which would be a weakly-typed mapping indexed by strings (though we would throw exceptions for invalid conversions).
Allowing users to store a single struct, which they are responsible for filling, in the archive, instead of individual fields ala method 1 (this is weakly typed once again).
Allowing users to store a single struct, but making it an additional template parameter for the input archive. This makes it strongly typed but makes this a big refactoring job. This is simultaneously the most and least elegant approach, in my opinion. Elegant because we get strong typing, inelegant because it makes the code look worse.
The other option is just to leave this feature out for now, pushing it to a 1.x or even 2.x release, depending on demand from other users.
Other obvious solutions for this issue push the burden onto the user to implement similar functionality using some global object, or to just locally patch cereal as they see fit.
I'm still a fan of option 1 and haven't fully made up my mind regarding what to do about this.
I can totally appreciate that. I do not think rushing into an implementation is the right answer necessarily. I do think this is a valid problem within the scope of cereal. I can say that I've been using option 1 now for a while well, and have not needed any further feature, or additions to this feature, but so long as the form of the solution is reasonable I am agnostic to the implementation.
I've definitely gone over all of my thoughts on this so far, but that's from one specific user's biased perspective. I think it's a good idea to see this in use across a few larger projects, but I'm not sure how to directly influence this beyond advertising the library... And I think pushing it out to a 1.x release is probably alright. I do not think pushing it out to a 2.x release would be a good plan unless a general suggested approach to solving this can be offered for user code
Ultimately I don't see a way around the issue except by modification of cereal in some way to open up the closed loop, and the obvious solution is the one I've already taken, but I'm only one user case.
Compiles but not stable yet, no interface to use Any within cereal yet. Will be kept internal and not exposed for use
outside of whatever the load/store names will be.
So here's the current plan: I'm punting this issue until a 1.1 release but am including an implementation of a light-weight boost::any type thing inside of helpers, so you can implement this yourself using that if you'd like. This will essentially be purely stagnant code until we come to a consensus on the actual solution.
Any situation in which you need to perform some kind of factory construction of an object with supplied parameters would also qualify. In a general sense, this could be any value or reference type.
It doesn't have to be in the constructor, just in the construction process (or more accurately, reconstruction.) Basically if you can imagine a scenario where cereal cannot recreate an object without touching that object immediately after to "fix" it, that's what I want to avoid.
If we start to get too many restrictions, I'll probably be keeping with the boost::any style approach as it is a) dirt simple, and b) pretty flexible.
I could see an external shared_ptr dependency being reasonable as well (above raw pointer/reference)
This is a solution for #46 that uses RTTI to allow get_user_data to work
and throw an error when used in an archive that doesn't actually have user
data. Unfortunately this is a run-time check and uses a dummy virtual function
that is never actually called (so the overhead of this will be very low, a few bytes
for the vtable, no runtime cost).
Another solution I'm going to play around with involves re-arranging some templates and
typedefs.
Took another look at this and I'm not completely satisfied with either solution that I've thought of.
The first solution involves using a "cast" (get_user_data) to retrieve the user data from an archive template parameter. The problem with this approach is that the template will never be instantiated with the wrapper type (see above for example), but instead the wrapped type itself, meaning that the userdata is not accessible without the cast. This also means that the cast will blindly let you try and get to userdata even on instantiations which do not actually wrap a type. This will ultimately lead to undefined behavior.
This solution can be made a bit better by adding a dummy private virtual function to the base archives which will never be called to allow us to use dynamic_cast instead of static_cast. This will let us throw an exception if you try to use get_user_data on an archive that doesn't have user data. I have this solution tested and working. There shouldn't be any overhead from having this dummy virtual function other than a few bytes for a vtable since it never actually gets called.
The second solution, which involves adding template parameters to existing archives, will make everything too ugly for too niche of a use case (even though it would probably give us the compile time constraints).
I'll have to think this over and discuss it with @randvoorhies. If we're fine with not having compile time guarantees we can put this in.
I noticed you marked the light weight solution as experimental.
I've been using my original solution for some time now but I'm toying with the idea of updating cereal to the latest build, I'm curious if there have been any developments on this enhancement since? If not, I may re-apply my "any" based changes locally, but I'd prefer a built in solution that isn't experimental. :)
I could also use this feature. I have a component orientated system in a game I've been working on and the system uses custom "Smart Pointers" to maintain references to entities and components. The references have two data members and look something like this:
class EntityHandle {
size_t entity_id;
Scene* scene;
At the moment, I'm just setting a global in the load function of the scene so that I have it available when any SmartPointers are encountered. This works but does feels a little messy.
EDIT: I realized I was passing in the construct object, not the archive, this might resolve the issue.
I haven't made use of it yet, but decided to give it a try before you mark it non-experimental. Since I had a working solution which was to use add and extract on the archives themselves I decided to create a "ServiceLocator" style class I could pass in which would take that responsibility on and refactor my old code to make use of the User Data type that cereal expects.
Based on this, I wanted to actually give it a try converting my code base to make use of this feature. I'm having issues right now:
1>c:\git\external\cereal\include\cereal\archives\adapters.hpp(153): error C2683: 'dynamic_cast': 'cereal::construct<MV::Scene::Button>' is not a polymorphic type (compiling source file Source\Render\Scene\button.cpp)
The code that is kicking off these issues looks like this:
std::shared_ptr<Node> Node::load(const std::string &a_filename, MV::Services& a_services, const std::string &a_newNodeId, bool a_doPostLoadStep) {
std::ifstream stream(a_filename);
require<ResourceException>(stream, "File not found for Node::load: ", a_filename);
LoadOptions nodeOptions(a_services, a_doPostLoadStep);
cereal::UserDataAdapter<MV::Services, cereal::JSONInputArchive> archive(a_services, stream);
std::shared_ptr<Node> result;
archive.add(cereal::make_nvp("services", a_services));
archive(result);
if (!a_newNodeId.empty()) {
result->id(a_newNodeId);
return result;
Then trying to use it in this fashion:
template <class Archive>
static void load_and_construct(Archive & archive, cereal::construct<Node> &construct, std::uint32_t const version) {
auto& services = cereal::get_user_data<MV::Services>(construct);
Note, I am making use of polymorphic deserialization as well as versioning so have calls like:
CEREAL_REGISTER_TYPE(MV::Scene::Drawable);
CEREAL_CLASS_VERSION(MV::Scene::Drawable, 3);
My services class looks like this:
#ifndef _MV_SERVICES_H_
#define _MV_SERVICES_H_
#include <typeindex>
#include <unordered_map>
#include "boost/any.hpp"
#include "Utility/log.h"
namespace MV {
template <typename T>
class Service {
public:
Service(T* a_service) :
service(a_service) {
Service(const Service& a_rhs) :
service(a_rhs.service) {
Service& operator=(const Service<T>& a_rhs) {
service = a_rhs.service;
return *this;
T* self() {
return service;
private:
T* service;
class Services {
public:
Services() {}
template<typename T>
T* connect(T* serviceObject) {
types[typeid(T)] = Service<T>(serviceObject);
return serviceObject;
template<typename T, typename V>
T* connect(V* serviceObject) {
types[typeid(T)] = Service<T>(serviceObject);
return dynamic_cast<T*>(serviceObject);
template<typename T>
void disconnect() {
types.erase(typeid(T));
template<typename T>
T* get(bool a_throwOnFail = true) {
auto i = types.find(typeid(T));
if (i != types.end()) {
try {
return boost::any_cast<Service<T>>(i->second).self();
} catch (boost::bad_any_cast&) {
auto message = std::string("Error: Tried to get service [") + i->second.type().name() + "] as [" + typeid(T).name() + "]\n";
if (a_throwOnFail) {
throw MV::ResourceException(message);
} else {
MV::warning(message);
auto message = std::string("Error: Failed to find service [") + typeid(T).name() + "]\n";
if (a_throwOnFail) {
throw MV::ResourceException(message);
} else {
MV::warning(message);
return nullptr;
template<typename T, typename V>
V* get(bool a_throwOnFail = true) {
auto i = types.find(typeid(T));
if (i != types.end()) {
try {
if (V* result = dynamic_cast<V*>(boost::any_cast<Service<T>>(i->second).self())) {
return result;
} catch (boost::bad_any_cast&) {
auto message = std::string("Error: Tried to get service [") + i->second.type().name() + "] as [" + typeid(T).name() + "]\n";
if (a_throwOnFail) {
throw MV::ResourceException(message);
} else {
MV::warning(message);
auto message = std::string("Error: Failed to find service [") + i->second.type().name() + "] as [" + typeid(T).name() + "]\n";
if (a_throwOnFail) {
throw MV::ResourceException(message);
} else {
MV::warning(message);
return nullptr;
template<typename T>
T* tryGet() {
auto i = types.find(typeid(T));
if (i != types.end()) {
try {
return boost::any_cast<Service<T>>(i->second).self();
} catch (boost::bad_any_cast&) {
return nullptr;
template<typename T, typename V>
V* tryGet() {
auto i = types.find(typeid(T));
if (i != types.end()) {
try {
if (V* result = dynamic_cast<V*>(boost::any_cast<Service<T>>(i->second).self())) {
return result;
} catch (boost::bad_any_cast&) {
return nullptr;
private:
Services(const Services&) = delete;
Services(Services&&) = delete;
Services& operator=(const Services&) = delete;
std::unordered_map<std::type_index, boost::any> types;
#endif
The official documentation does not seem to mention CEREAL_FUTURE_EXPERIMENTAL nor UserDataAdapter.
I just used it and I think it is worth adding it to the doc.
