JSONify All Things

The nlohmann/json library is everything a developer can expect from a modern library -- easy to integrate and JSON objects are treated as first class citizens with a very intuitive API.

However, it has one problem that is widely mentioned across the internet, which I'll tell you about below. Various solutions to the problem have been developed and shared, but none seem to be easy-to-use.

In this blog post, we will see how one can serialize and deserialize almost anything by extending the library a bit.

The Problem

I particularly like how you can easily define serialization/deserialization for your own type:

But what about for a more complex data structure, using a std::optional or std::variant , like this:

Unfortunately, it's not supported out-of-the-box by nlohmann/json; see some of the issues here:

• https://github.com/nlohmann/json/issues/1261
• https://github.com/nlohmann/json/issues/1749

I'll tell you what I want, what I really, really want: being able to write:

Variants

nlohmann/json gives you everything you need to write down your own serialization code; see the documentation for arbitrary type conversions . In our case, this means writing an adl_serializer for std::variant .

We just need to fill the TODO. The issue I linked earlier gives one solution. Unfortunately, it requires storing the index of the type close to the value in the JSON file. This is only possible if you control the whole chain. If you need to integrate in an existing protocol, it may not be possible.

The solution below will focus on the case where you have only one value for the variant and no indication of the type.

Serialization

To serialize the to_json method, we just want to automatically set into j the type that is in the variant. Fortunetaly, std::visit comes to the rescue and it ends up being a one liner:

Deserialization

To deserialize is a bit more complex as we don't know the exact type. So, we need to try them all to find the right one. If you can use C++17, this can be done quickly with fold expressions:

The fold expression (line 11) allows us to handle variadic templates without the need of writing recursive code. The ... means that it's going to repeat what is on the left of the comma for all types.

Be careful; this solution has some issues:

• The types need to be "exclusive," meaning you shouldn't be able to convert one from another. For example, std::variant<int, long long> won't work as expected.
• During deserialization, if your type variant has n types, you will raise (and catch) n-1 exceptions.

Optionals

Optionals are more complex, because we can't have a to_json / from_json at the level of the property, as the property may not exist at all. So we need to go one level up. There are actually some detailed explanations on the issue I linked before, thanks to all the people who've shared their solutions.

First Step

The first thing is to write down code to serialize/deserialize an optional. This code will be called later on in the parent json value to_json / from_json :

But we still need to write down, explicitly, the code to serialize/deserialize the structure using the optionals. At this point, we have only done half of the work.

Second Step

To go further, we need to look at how the macro is implemented. Looking at the code, this is what is done for all properties you pass in the macro NLOHMANN_DEFINE_TYPE_NON_INTRUSIVE :

In the case of a std::optional , we want to call our own optional_[from|to]_json rather than the default. Again, with C++17 to the rescue, using if constexpr we can write:

We then use our extended version to create our own macro, still copying what his done in the nlohmann/json library:

Conclusion

That's it! With this code, we are now able to write down:

and it just works out of the box !

Now why is it not by default in the library? In my opinion, there are multiple reasons for that:

• the current implementation for std::variant has a hard pre-requisite that all types must be exclusive;
• the current implementation for std::variant is not the most performant one, as it requires multiple exceptions;
• it would be better to store the index of the type's variant in the json, if possible;
• the current implementation for std::variant expects that an empty optional does not exist, while for some it may be on a null value in JSON.

Overall, the problem is complex and I don't think one solution will fit them all, meaning the chances for inclusion in the library are probably very low.

A big thanks to Niels Lohmann for creating this amazing piece of the library! I love it and am using it when I can.

Annex: full code all together

A big thank to Andrew for finding an issue in the code in the blog (now fixed).
You'll find the whole code all together below, if you want to just copy/paste it.