New Technology and the Need to Deserialize JSON

As part of the ongoing development effort to create Statsplash, an Omnified technology that creates a screen overlay showing raw, data mined game statistics for display on my stream, I needed to formalize a type of data contract and method for communicating raw game data from hacked assembly code to higher .NET level processes.

When deciding on which data format to use, it made sense to use JSON; something much simpler and faster than XML. Basically everything is using it these days, all the hip and cool kids included.

Because official support for dealing with JSON can now be found in .NET in the System.Text.Json namespace, I decided to make use of that to turn data exported from a hacked game process into beautiful and coherent (in comparison to the horrors of raw memory one faces when working with assembly) .NET objects.

I’ll be writing a number of Omnified Design articles detailing the experiences I’ve had with the creation of the Statsplash system, however for this article I just wanted to share a problem I had with a “shortcoming” of the System.Text.Json namespace and what my solution was.

Saying Bye to the Third Party, Then Missing Them

Since .NET 5.0 just came out, at the time of writing at least, it seemed like an appropriate time to port my army of very useful Bad Echo .NET libraries to this latest and seemingly important release of .NET.

When creating these universal .NET libraries (basically libraries I will use in anything I write, as they solve so many general-purpose problems), I like to keep third party dependencies to a minimum. Since .NET had official support for JSON now, that means no more reliance on ye olde Json.NET, from Newtonsoft.

In years past, if you wanted to deal with JSON in your .NET application, you had to grab ahold of that third party library and reference it. Well, now we apparently don’t need to, so I went ahead and:

  1. Constructed a JSON schema that the hacked assembly code would follow in writing the data we mined, and
  2. Wrote some code in Statsplash using System.Text.Json to deserialize the JSON output into a hierarchy of objects deriving from a Statistic base type.

Fail.

If you take a gander over at the listing of differences between the two libraries by Microsoft, you will see that polymorphic deserialization and serialization are both not supported out of the box. A workaround is provided, which requires you to write your own converter deriving from System.Text.Json.Serialization.JsonConverter<T>.

This workaround easily ends up being a lot of work however, and there’s is really no “generic” solution provided.

The lack of support for polymorphic serialization isn’t due to any sort of ineptitude on Microsoft’s part. It was actually quite intentionally left out, as allowing for the data to be solely in control of its own type instantiation is indeed quite a potential security threat.

When using Newtonsoft’s libraries, one would add support for polymorphic deserialization by storing type information for a JSON object in a property named $type. Technically, the data could have us instantiating whatever type from whatever assembly it wanted!

Of course, one can tighten down on these security concerns by providing an appropriate ISerializationBinder instance, which adds a measure of control to what types are actually going to be initialized. That’s good to know if you are set on using Newtonsoft’s stuff, but I was more interested in seeing if we could get some use out of the new System.Text.Json in my software.

Further into the GitHub discussion linked above, the same .NET developer provides an acceptable solution that would avoid the potential security pitfalls encountered when letting data have a say in its own type information. Specifically, separating the type information from the payload, and then only allowing type information to be expressed as an integer that the controlling serialization program would have mapped to a constrained list of types.

In order to get Statsplash to read our JSON game data while appeasing the .NET gods, we’re going to create a base converter type that does exactly that. The core universal library in the Bad Echo ecosystem is my Odin framework, which contains the many general purpose .NET constructions and helper functionalities I get so much use out of.

So, we’ll be stuffing it in there and then referencing that from Statsplash.

The Data We’ll Be Working With

Before we get into how we’ll be reading the data from JSON, let’s go over quickly what we’ll be reading from JSON, as well as a sample of some JSON that will be outputted by our injected Omnified assembly code.

All of these data types are native to the Statsplash project; be aware that these types as they are presented here should all be considered as preliminary. They will undoubtedly undergo changes prior to the release of Statsplash.

The Statistic Objects

The data of interest collected from our injected code take the form of individual (typically numeric) statistics, each of which we’ll be displaying to the viewer in one form or another.

Base Statistic Class

/// <summary>
/// Provides an individual statistic exported from an Omnified game.
/// </summary>
public abstract class Statistic
{
    /// <summary>
    /// Gets or sets the display name of the statistic.
    /// </summary>
    public string Name
    { get; set; } = string.Empty;
}

This is the root of all the various objects we’ll be deserializing from JSON, and is indeed the base type we’ll be setting up our converter to work with.

WholeStatistic Class

/// <summary>
/// Provides an individual statistic exported from an Omnified game concerning a whole, numeric value.
/// </summary>
public sealed class WholeStatistic : Statistic
{
    /// <summary>
    /// Gets or sets the whole numeric value for the statistic.
    /// </summary>
    public int Value
    { get; set; }
}

The first specific statistic type is also the simplest. It is a single number, and will be used to express statistics such as “Number of Deaths” or “Number of Souls”, etc.

FractionalStatistic Class

/// <summary>
/// Provides an individual statistic exported from an Omnified game concerning a fractional,
/// numeric value.
/// </summary> 
public sealed class FractionalStatistic : Statistic
{
    /// <summary>
    /// Gets or sets the current numeric value for the statistic.
    /// </summary>
    public int CurrentValue
    { get; set; }

    /// <summary>
    /// Gets or sets the maximum numeric value the statistic can be.
    /// </summary>
    public int MaximumValue
    { get; set; }
}

This next type of statistic is probably the most commonly used one for expressing information pertaining to the player or another character. It is a fractional value, where the “denominator” represents the maximum value of a stat, and the the “numerator” represents the current value.

An example of this is the player’s health; typically, there is a maximum health amount known, and then of course the current health, like so:

Player Health: 1230/1400

There are of course, innumerable other types of information that are fractional in nature: stamina, mana pool, etc.

CoordinateStatistic Class

/// <summary>
/// Provides an individual statistic exported from an Omnified game concerning a coordinate
/// triplet value.
/// </summary>
public sealed class CoordinateStatistic : Statistic
{
    /// <summary>
    /// Gets or sets the value for the X coordinate.
    /// </summary>
    public float X
    { get; set; }

    /// <summary>
    /// Gets or sets the value for the Y coordinate.
    /// </summary>
    public float Y
    { get; set; }

    /// <summary>
    /// Gets or sets the value for the Z coordinate.
    /// </summary>
    public float Z
    { get; set; }
}

This final type of statistic object, at least as far as this article is concerned, deals with Cartesian coordinate triplets. This is how the location of all types of entities is described in your typical 3D game.

The triplet consists of X, Y, and Z axis coordinate values, and because they are typically always floats in the game’s memory, they are represented as floats here in the class. I know there’s a tendency to use double in .NET code for any non-integer number (well, it is the default type for such a thing), but they are all float here because that’s what they actually are.

So these objects are basically what we’ll be wanting to produce from the JSON, in the form of an IEnumerable<Statistic> instance. Time to take a look at what kind data we’ll actually be deserializing.

Sample JSON Game Data

Our injected assembly code will be doing its thing and outputting mined game data to a statistics JSON file, and it’ll look a little something like this:

stats.json

[
  {
    "Type": 1,
    "Statistic": {
      {
        "CurrentValue": 1700,
        "MaximumValue": 2045,
        "Name": "Player Health"
      }
    }
  },
  {
    "Type": 1,
    "Statistic": {
      {
        "CurrentValue": 3520,
        "MaximumValue": 5000,
        "Name": "Enemy Health"
      }
    }
  },
  {
    "Type": 0,
    "Statistic": {
      {
        "Value": 120,
        "Name": "Last Damage Taken"
      }
    }
  },
  {
    "Type": 0,
    "Statistic": {
      {
        "Value": 50000,
        "Name": "Max Damage Taken"
      }
    }
  },
  {
    "Type": 0,
    "Statistic": {
      {
        "Value": 50120,
        "Name": "Total Damage Taken"
      }
    }
  },
  {
    "Type": 0,
    "Statistic": {
      {
        "Value": 200,
        "Name": "Last Damage Done"
      }
    }
  },
  {
    "Type": 0,
    "Statistic": {
      {
        "Value": 400,
        "Name": "Max Damage Done"
      }
    }
  },
  {
    "Type": 0,
    "Statistic": {
      {
        "Value": 2000,
        "Name": "Total Damage Done"
      }
    }
  },
  {
    "Type": 2,
    "Statistic": {
      {
        "X": -1566.7635,
        "Y": 0.10857524,
        "Z": 1997.6492,
        "Name": "Coordinates"
      }
    }
  }
]

The actual Omnified hacking code used to achieve the above output is outside the scope of the article, and will be provided when we actually dig deep into Statsplash upon its release.

So, looking at the above JSON, we can see an array containing our various statistic objects, with the type information and payload data separated for each one.

Let’s get to creating a general purpose polymorphic converter using Microsoft’s new libraries, and then a derived one showing how to use it to serialize these objects.

Polymorphic Deserialization With a Base Converter

So how to support working with data like the above JSON snippet with Microsoft’s new System.Text.Json namespace? Well we’ll start off like they say to do and make a class deriving from JsonConverter<T>.

This new class is going to itself be a base class that will require a deriving class to get any use out of it with specific datasets. This new mystery class shall heretofore be known as the JsonPolymorphicConverter<TTypeDescriptor,TBase> class.

The code itself is fully documented, so I hope that the code’s own documentation will ensure everything is clear to the reader.

JsonPolymorphicConverter<TTypeDescriptor,TBase> Class

/// <summary>
/// Provides a base class for converting a hierarchy of objects to or from JSON.
/// </summary>
/// <typeparam name="TTypeDescriptor">
/// A type of <see cref="Enum"/> whose integer value describes the type of object in JSON.
/// </typeparam>
/// <typeparam name="TBase">The base type of object handled by the converter.</typeparam>
public abstract class JsonPolymorphicConverter<TTypeDescriptor,TBase> : JsonConverter<TBase>
    where TTypeDescriptor : Enum
    where TBase : class
{
    private const string DEFAULT_TYPE_PROPERTY_NAME = "Type";

    /// <summary>
    /// Gets the expected property name of the number in JSON whose
    /// <typeparamref name="TTypeDescriptor"/> representation is used to determine the specific
    /// type of <typeparamref name="TBase"/> instantiated.
    /// </summary>
    protected virtual string TypePropertyName 
        => DEFAULT_TYPE_PROPERTY_NAME;

    /// <summary>
    /// Gets the expected property name of the object in JSON containing the object data payload.
    /// </summary>
    protected abstract string DataPropertyName { get; }

    /// <inheritdoc/>
    public override TBase? Read(
        ref Utf8JsonReader reader, Type typeToConvert, JsonSerializerOptions options)
    {
        if (reader.TokenType != JsonTokenType.StartObject)
            throw new JsonException(Strings.JsonExceptionNotStartObject);
            
        reader.Read();
        if (reader.TokenType != JsonTokenType.PropertyName)
            throw new JsonException(Strings.JsonExceptionMalformedText);
            
        string? typePropertyName = reader.GetString();
        if (typePropertyName != TypePropertyName)
            throw new JsonException(Strings.JsonExceptionInvalidTypeName.CulturedFormat(typePropertyName!, TypePropertyName));

        reader.Read();
        if (reader.TokenType != JsonTokenType.Number)
            throw new JsonException(Strings.JsonExceptionTypeValueNotNumber);

        var typeDescriptor = reader.GetInt32().ToEnum<TTypeDescriptor>();
        reader.Read();

        if (reader.TokenType != JsonTokenType.PropertyName)
            throw new JsonException(Strings.JsonExceptionMalformedText);

        string? dataPropertyName = reader.GetString();
        if (dataPropertyName != DataPropertyName)
            throw new JsonException(Strings.JsonExceptionInvalidTypeName.CulturedFormat(dataPropertyName!, DataPropertyName));

        reader.Read();

        if (reader.TokenType != JsonTokenType.StartObject)
            throw new JsonException(Strings.JsonExceptionDataValueNotObject);

        TBase? readValue = ReadFromDescriptor(ref reader, typeDescriptor);
        reader.Read();
            
        return readValue;
    }

    /// <inheritdoc/>
    public override void Write(Utf8JsonWriter writer, TBase value, JsonSerializerOptions options)
    {
        if (writer == null) 
            throw new ArgumentNullException(nameof(writer));

        if (value == null)
            throw new ArgumentNullException(nameof(value));

        TTypeDescriptor typeDescriptor = DescriptorFromValue(value);

        writer.WriteStartObject();
            
        writer.WriteNumber(TypePropertyName, typeDescriptor.ToInt32());

        writer.WriteStartObject(DataPropertyName);
        JsonSerializer.Serialize(writer, value, value.GetType(), options);
        writer.WriteEndObject();

        writer.WriteEndObject();
    }

    /// <summary>
    /// Reads and converts the JSON to a <typeparamref name="TBase"/>-derived type described by the provided
    /// <typeparamref name="TTypeDescriptor"/> enumeration.
    /// </summary>
    /// <param name="reader">The reader, positioned at the payload data of the object to read.</param>
    /// <param name="typeDescriptor">
    /// An enumeration value that specifies the type of <typeparamref name="TBase"/> to read.
    /// </param>
    /// <returns>The converted value.</returns>
    protected abstract TBase? ReadFromDescriptor(ref Utf8JsonReader reader, TTypeDescriptor typeDescriptor);

    /// <summary>
    /// Produces a <typeparamref name="TTypeDescriptor"/> value specifying a converted value's type.
    /// </summary>
    /// <param name="value">The converted value to create a type descriptor for.</param>
    /// <returns>
    /// A <typeparamref name="TTypeDescriptor"/> value that specifies the type of <c>value</c> in JSON.
    /// </returns>
    protected abstract TTypeDescriptor DescriptorFromValue(TBase value);
}

If there is any call above being made to methods that don’t compile for you, they are most likely extension methods that are part of my Odin framework. The source for that framework will be put up someday, but otherwise the actual things that these methods do can be easily figured out (although my extension methods typically are the result of long bits of research to find out the best way to do whatever it is doing).

To use the above base converter class, you need to create a derived class targeting a specific base object, providing support for the target object by providing implementations for the ReadFromDescriptor and DescriptorFromValue methods.

You also need to create an enumeration type that will essentially link the integer values in the JSON to type specifications.

All of this is very easy to do, and we’ll show an example that uses our statistics objects now.

Creating the Omnified Statistics Converter

We’re going to go ahead then and first create an enumeration type that will specify the particular type of statistics object we’re working with.

StatisticType Enum

/// <summary>
/// Specifies the type of statistic exported from an Omnified game.
/// </summary>
public enum StatisticType
{
    /// <summary>
    /// A whole, numeric value statistic.
    /// </summary>
    Whole,
    /// <summary>
    /// A fractional, numeric value statistic.
    /// </summary>
    Fractional,
    /// <summary>
    /// A coordinate triplet value statistic.
    /// </summary>
    Coordinate
}

Very simple, this basically establishes a mapping of integer values 0, 1, and 2 to Whole, Fractional, and Coordinate respectively.

Now to implement our specific statistics converter. Here you will see that the implementation of the required methods are essentially just some simple-to-write pattern matching expressions.

StatisticConverter Class

/// <summary>
/// Provides a converter of <see cref="Statistic"/> objects to or from JSON.
/// </summary>
public sealed class StatisticConverter : JsonPolymorphicConverter<StatisticType,Statistic>
{
    private const string STATISTIC_DATA_PROPERTY_NAME = "Statistic";

    /// <inheritdoc/>
    protected override string DataPropertyName
        => STATISTIC_DATA_PROPERTY_NAME;

    /// <inheritdoc/>
    protected override Statistic? ReadFromDescriptor(
        ref Utf8JsonReader reader, StatisticType typeDescriptor)
    {
        return typeDescriptor switch
        {
            StatisticType.Whole => JsonSerializer.Deserialize<WholeStatistic>(ref reader),
            StatisticType.Fractional => JsonSerializer.Deserialize<FractionalStatistic>(ref reader),
            StatisticType.Coordinate => JsonSerializer.Deserialize<CoordinateStatistic>(ref reader),
            _ => throw new InvalidEnumArgumentException(nameof(typeDescriptor), 
                                                        (int) typeDescriptor, 
                                                        typeof(StatisticType))
        };
    }

    /// <inheritdoc/>
    protected override StatisticType DescriptorFromValue(Statistic value)
    {
        return value switch
        {
            WholeStatistic => StatisticType.Whole,
            FractionalStatistic => StatisticType.Fractional,
            CoordinateStatistic => StatisticType.Coordinate,
            _ => throw new ArgumentException(Strings.ArgumentExceptionStatisticTypeUnsupported, 
                                             nameof(value))
        };
    }
}

It’s very simple to create a derived polymorphic converter, as you can see above. Let’s wire it up with some sample code, and see what we get.

Sample Usage

            string jsonString = File.ReadAllText("stats.json");
            var options = new JsonSerializerOptions {Converters = {new StatisticConverter()}};

            var statistics = JsonSerializer.Deserialize<IEnumerable<Statistic>>(jsonString, options);

Running this, we get…

Shows the serialized objects via a debugger.
Here’s our Statistic objects, freshly deserialized from JSON data.

Fabulous. There you go. Simple polymorphic deserialization with the System.Text.Json namespace.

Until next time.