Saving System Upgrade: Replacing BinaryFormatter with Json.NET [ON HOLD]

Important Update: ON HOLD

Unfortuntely, we have discovered that while this saving system works flawlessly in the Editor, it fails under build conditions. I’m working on a solution to this problem, but for now this conversions should be considered on hold.

-------------------------------------------------------------------------

Over the last month or so, I’ve been working on a replacement for the BinaryFormatter we currently use in our saving system. As some may be aware, Microsoft has officially declared the BinaryFormatter unsafe as there is a potential for an exploit that could allow a hacker to intercept a save file, inject data, and actually instantiate the hacker’s own classes into our project.

This thread will serve as a replacement model for the existing saving system. It is probably best read on my repo/wiki here: Converting the Saving System To JSon · Wiki · Brian K. Trotter / RPGMods · GitLab

Why Convert?

When the RPG Course Saving System was introduced in the first course, Core Combat, we used the BinaryFormatter to serialize and deserialize the needed fields. Unfortunately, Microsoft recently announced that using the BinaryFormatter was not safe, and recommends that it no longer be used. This issue presents a problem for companies around the globe who used this system not knowing the potential threats. I won’t go into the specific threats, and to be clear, while we’re in the prototyping stage (up until we release games to the public using this system) there is little to no risk. It’s when those save files start going over the internet that the trust level declines, and the risk of a hack that can compromise your system could be introduced. Bottom line ***Don’t Panic!*** All of that being said, I’m presenting some modifications to the saving system that will allow us to save in the Json format. As a side effect of teaching how to convert the system, I’ll even be leaving the original saving system more or less in place (you’ll be able to deactivate it and remove it easily once we’re done).

Getting Started

You may want to start by learning how the Saving System works. This is covered in depth in the Core Combat Course, both with a “Here’s an asset Pack” style approach and a “Let’s build this from the ground up approach”. If you’re starting in one of our other courses, you should review [How The Saving System Works](SavingSystemPrimer)

Installing Json.NET

While Unity does come with some rudimentary Json serialization, it’s not sufficient to keep up with the demands of our Saving System. For starters, it doesn’t support Dictionaries, let alone what are essentially nested Dictionaries.

Fortunately, there is another solution. There is a package in the Asset store called Json.NET for Unity. If you’re not familiar with Json.NET, it is a very robust serialization system that is more than powerful enough for our needs. It even works with IC2CPP in Android and iOS. Best of all, it’s free!
Follow this link to the Unity Asset Store Listing and add it to your assets. Then you can install it from the Package Manager in your project under Packages: My Assets.

For this tutorial, I’m going to create a new interface IJsonSaveable, so that at least for now, we can keep the original code intact. At the end of the tutorial, you’ll be able to remove the original save code and simply progress with the new Saving System.

IJsonSaveable

using Newtonsoft.Json.Linq;

namespace GameDevTV.Saving
{
    public interface IJsonSaveable 
    {
        public JToken CaptureAsToken();
        public void RestoreFromToken(JToken state);
    }
}

This is the Interface we will be using to replace the CaptureState/RestoreState from the original save system. You can, if you wish, simply edit ISaveable directly and replace the object in CaptureState and RestoreState with JToken. This will, of course, mean you’ll need to change everything at once to compile the code, which is why I’m taking this approach.
JToken is a special class that can serve the same way as Object for passing information around. Essentially, you can serialize the state to a JToken by using JToken.FromObject(T item), and you can convert it back by using the method ToObject(). I’ll demonstrate this now.

So let’s start with our first IJsonSaveable, we’ll start with Mover.cs. Here’s the original code in Mover.cs

        public object CaptureState()
        {
            return new SerializableVector3(transform.position);
        }

        public void RestoreState(object state)
        {
            SerializableVector3 position = (SerializableVector3)state;
            navMeshAgent.enabled = false;
            transform.position = position.ToVector();
            navMeshAgent.enabled = true;
            GetComponent<ActionScheduler>().CancelCurrentAction();
        }

SerializableVector3 is a special class that was created for the purpose of capturing and restoring Vectors. This is because for the BinaryFormatter, Vector3 is not Serializable. Fortunately, Json.Net has no such trouble. You can directly convert a Vector3 to and from a JToken.

Here’s the code for our IJsonSaveable classes. Be sure to add ,IJsonSaveable to your class declaration (right after ,ISaveable), and to add using NewtonSoft.Json.Linq; to your usings clauses.

// this is the added using declaration
using NewtonSoft.Json.Linq;

//this is the new class declaration
        public class Mover : MonoBehavior, IAction, ISaveable, IJsonSaveable
        
// Add these classes to fulfill the IJsonSaveable
        public JToken CaptureAsToken()
        {
            return JToken.FromObject(transform.position);
        }

        public void RestoreFromToken(JToken state)
        {
            navMeshAgent.enabled=false;
            transform.position=state.ToObject<Vector3>();
            navMeshAgent.enabled=true;
            GetComponent<ActionScheduler>().CancelCurrentAction();
        }

So for CaptureAsToken, we’re simply returning the position as a JToken… Since JToken can manage a Vector3, it’s just a simple matter of using the JToken static method FromObject().
For RestorFromToken, all we need to do is convert the JToken back into a Vector3 .

Ok, time for your first challenge. Head on over to Health.cs, and take a look at the CaptureState and RestoreState. Add IJsonSaveable to the class declaration, the using clause, and implement the the CaptureAsToken() and RestoreFromToken() methods.

3 Likes

How did you do on the challenge?

In the first entry of this thread, I challenged you to write the Health.cs IJsonSaveable code. Were you able to meet the challenge?

Let’s look first at the Health.cs ISaveable code:

        public object CaptureState()
        {
            return healthPoints.value;
        }

        public void RestoreState(object state)
        {
            healthPoints.value = (float) state;        
            UpdateState();
        }

That should be straightforward enough. We’re returning the value portion of our LazyValue healthPoints, and when we restore it, we’re simply changing the state back to a float.
So here’s what we need to do for the IJsonSaveable

//Add this to the using clauses
using NewtonSoft.Json.Linq;

//Modify the class declaration
public class Health : MonoBehaviour, ISaveable, IJsonSaveable

//Implement the missing members
        public JToken CaptureAsToken()
        {
            return healthPoints.value;
        }

        public void RestoreFromToken(JToken state)
        {
            healthPoints.value = state.ToObject<float>();
            UpdateState();
        }
    }

Notice that in this case, all I did was return the float. That’s because many conversions can be done implicitly. Assigning a value like an int, float, or string to JToken will automatically convert the item to JToken. In RestoreFromToken, we still need to explicitly declare the type. This is important, as even if there was an implicit conversion from JToken to our native types, it’s the implicit conversions that are behind much of the danger in the BinaryFormatter.

Let’s try another challenge, this time in Experience.cs:
Looking at Experience, we see the Capture and RestoreState methods look like this:

        public object CaptureState()
        {
            return experiencePoints;
        }

        public void RestoreState(object state)
        {
            experiencePoints = (float)state;
        }

Take a shot at writing the IJsonSaveable version. I’ll hide the solution

Click this to view/hide the solution

//Add this to the using clauses
using NewtonSoft.Json.Linq;

//Modify the class declaration
public class Experience : MonoBehaviour, ISaveable, IJsonSaveable

//Finally the methods
        public JToken CaptureAsToken()
        {
            return experiencePoints;
        }

        public void RestoreFromToken(JToken state)
        {
            experiencePoints = state.ToObject<float>();
        }

Once again, we see that we can return experiencepoints (it is simply a float value) and it is auto converted. You could also have done an explicit declaration

return JToken.FromObject(experiencePoints);

When converting back, we need to explicitly convert the state back to a float using

state.ToObject<float>();

Let’s take care of the last remaining ISaveable from the Core Combat course, Fighter.cs before we jump in to changing the SaveableEntity.

Note that if you have completed the entire course, you’ll likely have removed the ISaveable (as the equipped weapon will really be managed by Equipment anyways). If you look at the commit for this lesson, you won’t see Fighter.cs set up as an ISaveable or IJsonSaveable

Here’s the code from Fighter.cs before the end of Shops and Abilities

public object CaptureState()
{
    return currentWeaponConfig.name;
}
public void RestoreState(object state)
{
    string weaponName = (string)state;
    WeaponConfig weapon = UnityEngine.Resources.Load<WeaponConfig>(weaponName);
    EquipWeapon(weapon);
}

Since string is one of the things that can be implicitly converted as a JToken, we can copy CaptureState’s contents into CaptureAsToken

public JToken CaptureAsToken()
{
    return currentWeaponConfig.name;
}

and all we need to change in RestoreFromToken is how we convert state

public void RestoreFromToken(JToken state)
{
    string weaponName = state.ToObject<string>();
    WeaponConfig weapon = UnityEngine.Resources.Load<WeaponConfig>(weaponName);
    EquipWeapon(weapon);
}

In the next comment, we’ll start making the needed changes to SaveableEntity.cs

1 Like

Converting to JSon Part III

SaveableEntity

In the previous pages, we did some work on the replacements for ISaveables in the components from the Core Combat course. I promise, we’ll get to the components in the rest of the courses, but at this point, it’s a good time to start making the changes to SaveableEntity.cs and the Saving System itself so that we can see this JSON saving system in action.

We’re not going to worry about the extra code in SaveableEntity built around ensuring that every entity has a unique UUID, that’s not important to what we’re doing now, and we’re not going to change any of that code. For a good explanation of how that code works, be sure to study the second saving section in the Core Combat course.

So here is the section we’re interested in changing, and the good news is that there’s really not a lot to change in here:

        public object CaptureState()
        {
            Dictionary<string, object> state = new Dictionary<string, object>();
            foreach (ISaveable saveable in GetComponents<ISaveable>())
            {
                state[saveable.GetType().ToString()] = saveable.CaptureState();
            }
            return state;
        }


        public void RestoreState(object state)
        {
            Dictionary<string, object> stateDict = (Dictionary<string, object>)state;
            foreach (ISaveable saveable in GetComponents<ISaveable>())
            {
                string typeString = saveable.GetType().ToString();
                if (stateDict.ContainsKey(typeString))
                {
                    saveable.RestoreState(stateDict[typeString]);
                }
            }
        }

If you’re not super familiar with how this works, the SaveableEntity gathers all of the ISaveable components attached to the GameObject and asks them for their state. This state is then saved in a Dictionary<string, object> which is returned at the end of the method to the Saving System. When the Saving System restores, RestoreState decodes the dictionary and distributes the states back to the ISaveable components attached to the GameObject. These are the methods that call our Capture and Restore in the ISaveables.

CaptureAsToken

So let’s start with CaptureState… we’re going to make a change and demonstrate a nifty trick in the process:

        public JToken CaptureAsToken()
        {
            JObject state = new JObject();
            IDictionary<string, JToken> stateDict = state;
            foreach (IJsonSaveable saveable in GetComponents<IJsonSaveable>())
            {
                stateDict[saveable.GetType().ToString()] = saveable.CaptureAsToken();
            }
            return state;
        }

So that second line in the code may seem a bit confusing… IDictionary<string, JToken> stateDict = state; To start, we’re using a JObject instead of a JToken. JObject is a special type of JToken that is designed to easily encode collections and Dictionaries of JTokens. That IDictionary<string, JToken> allows us to work on the JObject as if it were a Dictionary. Now the rest of the method can work on stateDict (the IDictionary), and everything we do to it is automatically reflected in the JObject. When we’re done, we don’t have to do any special conversion, because it’s already done. You can just return state;

RestoreFromToken

We’re going to use this JObject approach once again with the RestoreFromToken.

        public void RestoreFromToken(JToken state)
        {
            IDictionary<string, JToken> stateDict = state.ToObject<JObject>();
            foreach (IJsonSaveable saveable in GetComponents<IJsonSaveable>())
            {
                string typeString = saveable.GetType().ToString();
                if (stateDict.ContainsKey(typeString))
                {
                    saveable.RestoreFromToken(stateDict[typeString]);
                }
            }
        }

We’re starting by converting the state to a JObject, which we’re immediately mapping as an IDictionary. We could have done this in two lines, but this is something we should be able to do on one line. Then, because it’s mapped to an IDictionary<string, JToken>, the rest of the method is virtually identical to the original RestoreState, just swapping out the interface name and methods.

It’s really that easy. Don’t be fooled, however, under the hood, a lot of great stuff is happening, but fortunately for us, we’re not doing the heavy lifting.

In the next comment, we’re going to make the changes to SavingSystem.cs. This is when it gets real. We’ll be saving JSON instead of a binary file.

1 Like

Converting to JSon Part IV

SavingSystem Changes

Ok, so far, we’ve converted the ISaveables in the Core Combat course to IJSonSaveables and we’ve modified our SaveableEntity to accept and distribute the JTokens that contained the saved data from each component. It’s time to make the needed changes to the SavingSystem and really start to commit ourselves to saving in JSon instead of the unsafe BinaryFormatter format.

Backup Your Project and Commit Your Repo

First of all, if you’re not using Source Control, then at the very least make a copy of your project. The changes we’re about to make could break the game if done incorrectly, and you definitely want a way back if things go wrong. This isn’t a course on using Git, and this is your big reminder.

Enough preaching, on to modifying the Saving System. Luckily, there’s not a lot to do, but it’s important to get it right.

Replacing CaptureState and RestoreState

As usual, we’ll start with the original code for CaptureState and RestoreState. We’ll leave these in for now, and put in their inevitable replacements.

        private void CaptureState(Dictionary<string, object> state)
        {
            foreach (SaveableEntity saveable in FindObjectsOfType<SaveableEntity>())
            {
                state[saveable.GetUniqueIdentifier()] = saveable.CaptureState();
            }

            state["lastSceneBuildIndex"] = SceneManager.GetActiveScene().buildIndex;
        }

        private void RestoreState(Dictionary<string, object> state)
        {
            foreach (SaveableEntity saveable in FindObjectsOfType<SaveableEntity>())
            {
                string id = saveable.GetUniqueIdentifier();
                if (state.ContainsKey(id))
                {
                    saveable.RestoreState(state[id]);
                }
            }
        }

Much like SaveableEntity, the SavingSystem gathers all of the SaveableEntities and asks them for their state, saving them in a Dictionary<string, object>. RestoreState distributes the states back to their respective SaveableEntities. One big difference is that instead of CaptureState returning a Dictionary, we’re actually passing the Dictionary directly to the method to modify. This allows us to save the data from multiple scenes in the same Dictionary. No need for a separate save file for each scene, it all goes in the one save file. For our replacement methods, instead of passing a Dictionary<string, object> we’re going to pass a JObject. Remember that JObjects are actually a collection of JParameter. A JParameter represents a string and a JToken. Because of this, we can map our JObject to an IDictionary<string, JToken>. Let’s take a look at our new Capture:

        private void CaptureAsToken(JObject state)
        {
            IDictionary<string, JToken> stateDict = state;
            foreach (SaveableEntity saveable in FindObjectsOfType<SaveableEntity>())
            {
                stateDict[saveable.GetUniqueIdentifier()] = saveable.CaptureState();
            }

            stateDict["lastSceneBuildIndex"] = SceneManager.GetActiveScene().buildIndex;
        }

So the JObject that is fed into CaptureAsToken may or may not contain data from other scenes (or even this scene that need to be overwritten). Fortunately, the IDictionary interface comes into play allowing us to treat this JObject as if it were, in fact, a Dictionary(). This means that aside from changing a few method names, we literally don’t have to change anything else in the method! Just the header and that first line to setup the IDictionary.

This leads us to RestoreFromToken() which is also just a straightforward a change:

        private void RestoreFromToken(JObject state)
        {
            IDictionary<string, JToken> stateDict = state;
            foreach (SaveableEntity saveable in FindObjectsOfType<SaveableEntity>())
            {
                string id = saveable.GetUniqueIdentifier();
                if (stateDict.ContainsKey(id))
                {
                    saveable.RestoreFromToken(stateDict[id]);
                }
            }
        }

Just like in SaveableEntity, we’re mapping an IDictionary over the JObject.

SaveFile and LoadFile

This is where the actual serialization happens. Up until now, the JTokens and JObjects are just being collected, they’re not actually JSon yet. Now we’re going to actually write the JSon to disk and read it back. Once again, we’ll start with the original code:

        private void SaveFile(string saveFile, object state)
        {
            string path = GetPathFromSaveFile(saveFile);
            print("Saving to " + path);
            using (FileStream stream = File.Open(path, FileMode.Create))
            {
                BinaryFormatter formatter = new BinaryFormatter();
                formatter.Serialize(stream, state);
            }
        }

        private Dictionary<string, object> LoadFile(string saveFile)
        {
            string path = GetPathFromSaveFile(saveFile);
            if (!File.Exists(path))
            {
                return new Dictionary<string, object>();
            }
            using (FileStream stream = File.Open(path, FileMode.Open))
            {
                BinaryFormatter formatter = new BinaryFormatter();
                return (Dictionary<string, object>)formatter.Deserialize(stream);
            }
        }

This is the part of our saving system that Microsoft has complained about. BinaryFormatter is unsafe, because in the process of Deserializing, any objects within the stream are actually instantiated, and can get their initialization run on them. Not only that, if an object is simply embedded alongside our carefully curated Dictionary, but it’s class doesn’t exist in your project, the object can provide the information to download it into your computer and run it. It means a malicious hacker could inject code into a system by infecting the save file.
While we’ve been prototyping, and the savefile is safely in the PersistentDataPath, not being shared on the web, etc, the danger is virtually zero that it could get infected, but the moment that Save goes out into the world (say you email the save file to me to examine what’s going on, or you put the save file on an AWS or Google Cloud server)… then the trust barrier is broken, and the potential for this hack always exists.

Here’s our new SaveFile method:

        private void SaveFileAsJSon(string saveFile, JObject state)
        {
            string path = GetPathFromSaveFile(saveFile);
            print("Saving to " + path);
            using (var textWriter = File.CreateText(path))
            {
                using (var writer = new JsonTextWriter(textWriter))
                {
                    writer.Formatting = Formatting.Indented;
                    state.WriteTo(writer);
                }
            }
        }

Instead of a binary writer, we’re using a textWriter. We’re setting the formatting to indented so that the file will be readable. This is optional, and you can remove that line if you wish to make the entire JSon go on a single line.

Here’s the LoadFile replacement method:

        private JObject LoadJSonFromFile(string saveFile)
        {
            string path = GetPathFromSaveFile(saveFile);
            if (!File.Exists(path))
            {
                return new JObject();
            }
            
            using (var textReader = File.OpenText(path))
            {
                using (var reader = new JsonTextReader(textReader))
                {
                    reader.FloatParseHandling = FloatParseHandling.Double;

                    return JObject.Load(reader);
                }
            }
        }

This is fairly similar to our original LoadFile method as well. If there is no save file, an empty JObject is returned, which can then essentially become our new save file later on.

This leads us to the moment we’ve been waiting for, the actual Load and Save… Here’s the original:

        public void Save(string saveFile)
        {
            Dictionary<string, object> state = LoadFile(saveFile);
            CaptureState(state);
            SaveFile(saveFile, state);
        }

        public void Load(string saveFile)
        {
            RestoreState(LoadFile(saveFile));
        }

This may shock you, but at this point, the changes are almost minimal… and this is the point of no return, the point at which we commit our saving system to switch from BinaryFormatter to JSON. From here on out, we’re committed. So committed, we’re going to just edit the method directly.

Did you Commit?

Just another reminder, commit your code or at least back it up!

Here’s our modified code. By the end of this post, you’ll be saving and restoring Json

        public void Save(string saveFile)
        {
            JObject state = LoadJSonFromFile(saveFile);
            CaptureAsToken(state);
            SaveFileAsJSon(saveFile, state);
        }

        public void Load(string saveFile)
        {
            RestoreFromToken(LoadJSonFromFile(saveFile));
        }

Finishing touches

There are a few more tidbits in SavingSystem.cs to take care of to make everything work. The first is in the GetPathFromSaveFile method. Right now, we’re using the extension “.sav”. You can leave it at that if you wish, but it might be better to change that extension to “.json” for clarity.

        private string GetPathFromSaveFile(string saveFile)
        {
            return Path.Combine(Application.persistentDataPath, saveFile + ".json");
        }

If you’ve completed the Main Menu section of the course, you’ll also need to change the extension in the ListSaves method to “.json”

Finally, there is one last method to change, and that’s LoadLastScene.

        public IEnumerator LoadLastScene(string saveFile)
        {
            JObject state = LoadJSonFromFile(saveFile);
            IDictionary<string, JToken> stateDict = state;
            int buildIndex = SceneManager.GetActiveScene().buildIndex;
            if (stateDict.ContainsKey("lastSceneBuildIndex"))
            {
                buildIndex = (int)stateDict["lastSceneBuildIndex"];
            }
            yield return SceneManager.LoadSceneAsync(buildIndex);
            RestoreFromToken(state);
        }

We’re starting the method out by getting our state as a JObject. As we’ve already seen through this tutorial, that JObject can be mapped as an IDictionary<string, JToken>. We need this to get the lastSceneBuildIndex, which tells the method which scene to load. Once we’ve loaded the correct scene, we pass the state on to RestoreFromToken() and things are restored.

At this point, you should be able to test the saving system out. Bear in mind, we’re not saving much, just the stuff from the Core Combat course. Start your game, maybe take on an enemy or just move to a new spot and save the game. Then quit the game and return, and you should find yourself in the position you left, not the starting position.

This might be a good time to navigate to the .json file and check out what’s been saved. Unlike the results from BinaryFormatter, the .json file is a human readable document.

So believe it or not, you now have the tools you need to convert the rest of the ISaveables into JsonSaveables. Don’t worry, though, the next comment will go over the remaining sections.

1 Like

Converting to JSon Part V

Converting the rest of the ISaveables

There’s not going to be a great deal of explanation on these. This will be a rundown of all of the ISaveables in the course repo that we haven’t already covered before. You may see classes here you haven’t encountered, depending on what courses you’ve taken/completed. If you do get to those classes in one of the courses, then head here for the code.

Note that if you have made substantive changes to your project, the code here may not work (for example, you may have a Purse with 10 different currency types, so you’d need to capture/restore a collection instead of a single float). I’m thinking that if you’re ready to make those changes, you’ll probably

Remember that for all IJsonSaveables, you’ll need to include using NewtonSoft.Json.Linq; to your using clauses, and add the ,IJsonSaveable to the class declaration.

Inventory.cs

        public JToken CaptureAsToken()
        {
            var slotStrings = new InventorySlotRecord[inventorySize];
            for (int i = 0; i < inventorySize; i++)
            {
                if (slots[i].item != null)
                {
                    slotStrings[i].itemID = slots[i].item.GetItemID();
                    slotStrings[i].number = slots[i].number;
                }
            }
            return JToken.FromObject(slotStrings);
        }

        public void RestoreFromToken(JToken state)
        {
            var slotStrings = state.ToObject<InventorySlotRecord[]>();
            slots = new InventorySlot[inventorySize];
            for (int i = 0; i < inventorySize; i++)
            {
                slots[i].item = InventoryItem.GetFromID(slotStrings[i].itemID);
                slots[i].number = slotStrings[i].number;
            }
            inventoryUpdated?.Invoke();
        }

Equipment.cs

        public JToken CaptureAsToken()
        {
            var equippedItemsForSerialization = new Dictionary<EquipLocation, string>();
            foreach (var pair in equippedItems)
            {
                equippedItemsForSerialization[pair.Key] = pair.Value.GetItemID();
            }
            return JToken.FromObject(equippedItemsForSerialization);
        }

        public void RestoreFromToken(JToken state)
        {
            equippedItems = new Dictionary<EquipLocation, EquipableItem>();

            var equippedItemsForSerialization = state.ToObject<Dictionary<EquipLocation, string>>();

            foreach (var pair in equippedItemsForSerialization)
            {
                var item = (EquipableItem)InventoryItem.GetFromID(pair.Value);
                if (item != null)
                {
                    equippedItems[pair.Key] = item;
                }
            }
        }

ActionStore.cs

        public JToken CaptureAsToken()
        {
            var state = new Dictionary<int, DockedItemRecord>();
            foreach (var pair in dockedItems)
            {
                var record = new DockedItemRecord();
                record.itemID = pair.Value.item.GetItemID();
                record.number = pair.Value.number;
                state[pair.Key] = record;
            }

            return JToken.FromObject(state);
        }

        public void RestoreFromToken(JToken state)
        {
            var stateDict = state.ToObject<Dictionary<int, DockedItemRecord>>();
            foreach (var pair in stateDict)
            {
                AddAction(InventoryItem.GetFromID(pair.Value.itemID), pair.Key, pair.Value.number);
            }
        }

ItemDropper.cs

        [System.Serializable]
        private struct JDropRecord
        {
            public string itemID;
            public Vector3 position; //No need for SerializeableVector3
            public int number;
        }
        
        public JToken CaptureAsToken()
        {
            RemoveDestroyedDrops();
            var droppedItemsList = new JDropRecord[droppedItems.Count];
            for (int i = 0; i < droppedItemsList.Length; i++)
            {
                droppedItemsList[i].itemID = droppedItems[i].GetItem().GetItemID();
                droppedItemsList[i].position = droppedItems[i].transform.position;
                droppedItemsList[i].number = droppedItems[i].GetNumber();
            }
            return JToken.FromObject(droppedItemsList);
        }

        public void RestoreFromToken(JToken state)
        {
            var droppedItemsList = state.ToObject<JDropRecord[]>();
            foreach (var item in droppedItemsList)
            {
                var pickupItem = InventoryItem.GetFromID(item.itemID);
                Vector3 position = item.position;
                int number = item.number;
                SpawnPickup(pickupItem, position, number);
            }
        }

PickupSpawner.cs

        public JToken CaptureAsToken()
        {
            return isCollected(); //implicit conversion on primitive type!
        }

        public void RestoreFromToken(JToken state)
        {
            bool shouldBeCollected = state.ToObject<bool>();

            if (shouldBeCollected && !isCollected())
            {
                DestroyPickup();
            }

            if (!shouldBeCollected && isCollected())
            {
                SpawnPickup();
            }
        }

Mana.cs

        public JToken CaptureAsToken()
        {
            return mana.value; //implicit conversion
        }

        public void RestoreFromToken(JToken state)
        {
            mana.value = state.ToObject<float>();
        }

Purse.cs

        public JToken CaptureAsToken()
        {
            return balance;
        }

        public void RestoreFromToken(JToken state)
        {
            balance = state.ToObject<float>();
        }

QuestStatus

Before QuestList will Capture/Restore correctly, we need to change QuestStatus first. Don’t make QuestStatus an IJSonSaveable, as Questlist will call these methods directly without an interface.

        public QuestStatus(JToken objectState)
        {
            var state = objectState.ToObject<QuestStatusRecord>();
            quest = Quest.GetByName(state.questName);
            completedObjectives = state.completedObjectives;
        }
        public JToken CaptureAsToken()
        {
            QuestStatusRecord state = new QuestStatusRecord();
            state.questName = quest.name;
            state.completedObjectives = completedObjectives;
            return JToken.FromObject(state);
        }

QuestList

Be sure to properly set up QuestList or this won’t work

        public JToken CaptureAsToken()
        {
            JObject state = new JObject();
            IDictionary<string, JToken>  stateList = state;
            foreach (QuestStatus status in statuses)
            {
                //We're adding a key of the instance ID just to ensure unique dictionary entries
                stateList.Add($"{status.GetQuest().GetInstanceID()}", status.CaptureAsToken());
            }
            return state;
        }

        public void RestoreFromToken(JToken state)
        {
            IDictionary<string, JToken> stateList = state.ToObject<JObject>();
            
            statuses.Clear();
            foreach (KeyValuePair<string, JToken> objectState in stateList)
            {
                //We're not concerned with the key, just the value
                statuses.Add(new QuestStatus(objectState.Value));
            }
        }
    }

Shop

        public JToken CaptureAsToken()
        {
            Dictionary<string, int> saveObject = new Dictionary<string, int>();

            foreach (var pair in stockSold)
            {
                saveObject[pair.Key.GetItemID()] = pair.Value;
            }

            return JToken.FromObject(saveObject);
        }

        public void RestoreFromToken(JToken state)
        {
            Dictionary<string, int> saveObject = state.ToObject<Dictionary<string, int>>();
            stockSold.Clear();
            foreach (var pair in saveObject)
            {
                stockSold[InventoryItem.GetFromID(pair.Key)] = pair.Value;
            }
        }

TraitStore

        public JToken CaptureAsToken()
        {
            return JToken.FromObject(assignedPoints);
        }

        public void RestoreFromToken(JToken state)
        {
            assignedPoints = state.ToObject <Dictionary<Trait, int>>();
        }

This completes the basics of converting the saving system to use Json. The sections following this will contain some more advanced topics, including applying the Strategy Pattern to our saving system so that we can implement whatever sort of encoding/encryption we choose.

In the next section, we’ll go over some options to save our files as BSON files, and to save them in an encrypted format.

1 Like

Converting to Json Part VI

The pitfalls of Json

So one of the pitfalls of writing to a Json file is that a Json is a human readable file. Actually, for our purposes during prototyping, I consider this to be a strength. A human readable save file means you can actually see what’s going on in the save file when things don’t seem to be quite right. That doesn’t mean, however, that we want to distribute our games using Json. A player could easily edit the files to give themselves inventory items or other forms of mischief.

Here’s a fantastic example, in just a fragment of a save file:

    "RPG.Stats.Experience": 2929.0,
    "GameDevTV.Inventories.ActionStore": {
      "0": {
        "itemID": "9d0578ce-a8ca-4634-9631-dfcc7002476a",
        "number": 1
      }

What this represents (in my game, it may be different in yours) is the Experience of the player, and a single potion in slot 0 of the ActionStore…
Now… what would happen if I edited this section to read:

    "RPG.Stats.Experience": 100000.0,
    "GameDevTV.Inventories.ActionStore": {
      "0": {
        "itemID": "9d0578ce-a8ca-4634-9631-dfcc7002476a",
        "number": 99
      }

Our player would now have 100,000 experience points (gaining a LOT of levels) and would have 99 health potions.

Let’s explore a few options to make our save files a little bit trickier for the users to edit:

Bson - Binary Json

We’ll start with the most obvious. There is another form of Json called Bson, or Binary Serialized Object Format. The structure is very similar, and is actually somewhat readable. Capture|attachment

Anybody familiar with basic hexadecimal mathematics may find that they can alter many of the same attributes, as all of the classes prefix the values that need to be changed… Ultimately, this probably shouldn’t be our final product, but we’ll use this as the starting point to get to a more secure format.

There are actually only a few changes needed to write to Bson instead of Json. I’ll include these as new methods:

        private void SaveFileAsBSon(string saveFile, JObject state)
        {
            string path = GetPathFromSaveFile(saveFile);
            Debug.Log($"Saving BSon to {path}");
            using (FileStream fileStream = File.Open(path, FileMode.Create))
            {
                using (BsonWriter writer = new BsonWriter(fileStream))
                {
                    state.WriteTo(writer);
                }
            }
        }

        private JObject LoadBSonFromFile(string saveFile)
        {
            string path = GetPathFromSaveFile(saveFile);
            Debug.Log($"Loading BSon from {path}");
            if (!UnityEngine.Windows.File.Exists(path))
            {
                return new JObject();
            }

            using (FileStream fileStream = File.Open(path, FileMode.Open))
            {
                using (BsonReader reader = new BsonReader(fileStream))
                {
                    return (JObject) JToken.ReadFrom(reader);
                }
            }
        }

You’ll need to replace LoadJSonFromFile with LoadBSonFromFile in LoadLastScene(), Save() and Load() You’ll need to replace SaveFileAsJSon in Save() You’ll also need to change the extension to a new extension from .json to avoid confusion. I’ve used .bson If you’ve finished the Menu section of the Shops and Abilities course, you’ll also need to change the extension there.

Once you’re done, you should be able to save and restore the game to a Bson file. You’ve probably noticed, at this point, that it may be a bit cumbersome to switch methods… if you want to save in Json for testing and Bson for production all that switching may get very frustrating.

A Strategy For Saving - The Strategy Pattern

In Dialogues and Quests, Sam introduces a new pattern called the Strategy Pattern. This innovative pattern lets you define a behavior or family of behaviors in the main class, but the actual implementation of that behavior in a separate class. You can swap out the implementation (strategy) class to get a different implementation of the behavior. Setting aside the original BinaryFormatter (really, just don’t use it once this is working), we currently have two different strategies for saving and restoring the data collected by CaptureAsToken… we can save and restore to human readable JSon or we can save and restore to Bson… ultimately, we will want to be able to save to an encrypted format (encode to Bson and encrypt the results before saving). We may also want a pattern to save to an encrypted format that can be saved as a text file. That’s a lot of potential methods to use and swap and edit…

What we’re going to do is introduce a new class SavingStrategy. This will serve as a template class for our actual strategy classes.

using System.IO;
using Newtonsoft.Json.Linq;
using UnityEngine;

namespace GameDevTV.Saving
{
    public abstract class SavingStrategy
    {
        public abstract void SaveToFile(string saveFile, JObject state);
        public abstract JObject LoadFromFile(string saveFile);
        public abstract string GetExtension();
        public string GetPathFromSaveFile(string saveFile)
        {
            return Path.Combine(Application.persistentDataPath, saveFile +GetExtension());
        }
    }
}

This is an abstract class with three abstract methods.
SaveToFile will take a JObject and save it in the format associated with the strategy LoadFromFile will deserialize the save file and return the JObject, using the format associated with the strategy GetExtension will return the correct extension to go with the strategy. Finally, we have the GetPathFromSaveFile() method copied from the SavingSystem. This will give the strategies access to the path information they need to properly save/load the files.

So far, we have two viable strategies for saving… we can save to Json or we can save to Bson. For each of these strategies, we’ll write a new class inherited from SavingStrategy JsonStrategy.cs

using System.IO;
using Newtonsoft.Json;
using Newtonsoft.Json.Linq;
using UnityEngine;


namespace GameDevTV.Saving
{
    public class JsonStrategy : SavingStrategy
    {

        public override void SaveToFile(string saveFile, JObject state)
        {
            string path = GetPathFromSaveFile(saveFile);
            Debug.Log($"Saving to {path} ");
            using (var textWriter = File.CreateText(path))
            {
                using (var writer = new JsonTextWriter(textWriter))
                {
                    JsonSerializer serializer = new JsonSerializer();
                    serializer.Serialize(writer, state);
                }
            }
        }

        public override JObject LoadFromFile(string saveFile)
        {
            string path = GetPathFromSaveFile(saveFile);
            if (!File.Exists(path))
            {
                return new JObject();
            }
            
            using (var textReader = File.OpenText(path))
            {
                using (var reader = new JsonTextReader(textReader))
                {
                    reader.FloatParseHandling = FloatParseHandling.Double;

                    return JObject.Load(reader);
                }
            }
        }

        public override string GetExtension()
        {
            return ".json";
        }
    }
}

BsonStrategy.cs

using System.IO;
using Newtonsoft.Json.Bson;
using Newtonsoft.Json.Linq;
using UnityEngine;

namespace GameDevTV.Saving
{
    public class BsonStrategy : SavingStrategy
    {
        public override void SaveToFile(string saveFile, JObject state)
        {
            string path = GetPathFromSaveFile(saveFile);
            Debug.Log($"Saving BSon to {path}");
            using (FileStream fileStream = File.Open(path, FileMode.Create))
            {
                using (BsonWriter writer = new BsonWriter(fileStream))
                {
                    state.WriteTo(writer);
                }
            }
        }

        public override JObject LoadFromFile(string saveFile)
        {
            string path = GetPathFromSaveFile(saveFile);
            Debug.Log($"Loading BSon from {path}");
            if (!UnityEngine.Windows.File.Exists(path))
            {
                return new JObject();
            }

            using (FileStream fileStream = File.Open(path, FileMode.Open))
            {
                using (BsonReader reader = new BsonReader(fileStream))
                {
                    return (JObject) JToken.ReadFrom(reader);
                }
            }
        }

        public override string GetExtension()
        {
            return ".bson";
        }
    }
}

So each of these strategies use the same basic methods we outlined for saving the Json and Bson, but as overrides to the SavingStrategy abstract class.

The next step is to inject one of these strategies into our saving system.

At the top of the SavingSystem class, add a field for a SavingStrategy

SavingStrategy savingStrategy;

You’ll also need to initialize the saving strategy. For now, we’ll just use Awake(). Create an Awake() method and initialize the savingStrategy with the strategy of your choice:

void Awake()
{
    savingStrategy = new JsonStrategy();
}

Now we need to change the methods for saving and loading… In LoadLastScene():

JObject state = savingStrategy.LoadFromFile(saveFile);

In Save():

JObject state = savingStrategy.LoadFromFile(saveFile);
...
savingStrategy.SaveToFile(saveFile, state);

In Load()

RestoreFromToken(savingStrategy.LoadFromFile(saveFile));

In GetPathFromSaveFile()

return Path.Combine(Application.persistentDataPath, saveFile + savingStrategy.GetExtension());

and finally for those of you who have completed the Menu section of the Shops and Ability section, in IEnumerable ListSaves()

if(Path.GetExtension(path) == savingStrategy.GetExtension())

Here’s the final SavingSystem.cs after the modifications:

using System.Collections;
using System.Collections.Generic;
using System.IO;
using Newtonsoft.Json.Linq;
using UnityEngine;
using UnityEngine.SceneManagement;

namespace GameDevTV.Saving
{
    /// <summary>
    /// This component provides the interface to the saving system. It provides
    /// methods to save and restore a scene.
    ///
    /// This component should be created once and shared between all subsequent scenes.
    /// </summary>
    public class SavingSystem : MonoBehaviour
    {
        private SavingStrategy savingStrategy;

        private void Awake()
        {
            savingStrategy = new JsonStrategy();
        }

        /// <summary>
        /// Will load the last scene that was saved and restore the state. This
        /// must be run as a coroutine.
        /// </summary>
        /// <param name="saveFile">The save file to consult for loading.</param>
        public IEnumerator LoadLastScene(string saveFile)
        {
            JObject state = savingStrategy.LoadFromFile(saveFile);
            IDictionary<string, JToken> stateDict = state;
            int buildIndex = SceneManager.GetActiveScene().buildIndex;
            if (stateDict.ContainsKey("lastSceneBuildIndex"))
            {
                buildIndex = (int)stateDict["lastSceneBuildIndex"];
            }
            yield return SceneManager.LoadSceneAsync(buildIndex);
            RestoreFromToken(state);
        }

        /// <summary>
        /// Save the current scene to the provided save file.
        /// </summary>
        public void Save(string saveFile)
        {
            JObject state = savingStrategy.LoadFromFile(saveFile);
            CaptureAsToken(state);
            savingStrategy.SaveToFile(saveFile, state);
        }

        /// <summary>
        /// Delete the state in the given save file.
        /// </summary>
        public void Delete(string saveFile)
        {
            File.Delete(GetPathFromSaveFile(saveFile));
        }

        public void Load(string saveFile)
        {
            RestoreFromToken(savingStrategy.LoadFromFile(saveFile));
        }

        public IEnumerable<string> ListSaves()
        {
            foreach (string path in Directory.EnumerateFiles(Application.persistentDataPath))
            {
                if (Path.GetExtension(path) == savingStrategy.GetExtension())
                {
                    yield return Path.GetFileNameWithoutExtension(path);
                }
            }
        }
      
        private void CaptureAsToken(JObject state)
        {
            IDictionary<string, JToken> stateDict = state;
            foreach (SaveableEntity saveable in FindObjectsOfType<SaveableEntity>())
            {
                stateDict[saveable.GetUniqueIdentifier()] = saveable.CaptureAsToken();
            }

            stateDict["lastSceneBuildIndex"] = SceneManager.GetActiveScene().buildIndex;
        }

        private void RestoreFromToken(JObject state)
        {
            IDictionary<string, JToken> stateDict = state;
            foreach (SaveableEntity saveable in FindObjectsOfType<SaveableEntity>())
            {
                string id = saveable.GetUniqueIdentifier();
                if (stateDict.ContainsKey(id))
                {
                    saveable.RestoreFromToken(stateDict[id]);
                }
            }
        }

        private string GetPathFromSaveFile(string saveFile)
        {
            return Path.Combine(Application.persistentDataPath, saveFile + savingStrategy.GetExtension());
        }
    }
}

Once these changes are made, you should be able to select saving in Json or Bson simply by changing the strategy we initialize in Awake(). No other modifications to SavingSystem will be necessary, as all changes will be made in the strategies.

Try it first with JsonStrategy(), make sure you can save and load the scene. Then change the strategy in Awake() to BsonStrategy() and do it again. The Json save won’t be available, but you’ll find in a new game you can save and load in Bson.

Now that we’ve established a strategy pattern for saving, in the next section we’ll explore a couple of options for encrypting the save file. One pattern for simple encryption, and another pattern to encrypt the file and convert it to a text file that can be sent to text only cloud systems.

1 Like

So far, we’ve seen how to capture and restore the JToken version of object, and how to serialize and deserialize in both human readable Json and quasi human readable Bson. We’ve also seen how to set up a strategy pattern to allow you to choose at compile time which of these formats we wish to use.

This last section will extend that strategy pattern concept to give us a couple more options when it comes to saving/restoring while still making the file more difficult (but never impossible) to alter.

A quick note on encryption: Aside from implementing NSA level encryption, there is no such thing as a file that cannot be decrypted. Very smart eggheads have literally spent decades coming up with new ways to encrypt messages, and more to the point, coming up with ways to decrypt them. It’s a bit like locking your front door. It will stop honest people from breaking into your home, but it’s not a guarantee.

I’m going to present the simplest of cyphers, the XOR Cypher. This cypher takes a simple key and character by character applies an XOR with the corresponding letter of the key and the text to encrypt… For this example, I’m using a modified version of an example from Codingame.com

using System.IO;
using System.Text;
using Newtonsoft.Json.Linq;
using UnityEngine;

namespace GameDevTV.Saving
{
    public class XorStrategy : SavingStrategy
    {
        private const string key = "TheQuickBrownFoxJumpedOverTheLazyDog"; //This is a TERRIBLE KEY
        
        public string EncryptDecrypt(string szPlainText, string szEncryptionKey)  
        {  
            StringBuilder szInputStringBuild = new StringBuilder(szPlainText);  
            StringBuilder szOutStringBuild = new StringBuilder(szPlainText.Length);  
            char Textch;  
            for (int iCount = 0; iCount < szPlainText.Length; iCount++)
            {
                int stringIndex = iCount % szEncryptionKey.Length;
                Textch = szInputStringBuild[iCount];  
                Textch = (char)(Textch ^ szEncryptionKey[stringIndex]);  
                szOutStringBuild.Append(Textch);  
            }  
            return szOutStringBuild.ToString();  
        } 
        
        public override void SaveToFile(string saveFile, JObject state)
        {
            string path = GetPathFromSaveFile(saveFile);
            Debug.Log($"Saving to {path} ");
            using (var textWriter = File.CreateText(path))
            {
                string json = state.ToString();
                string encoded = EncryptDecrypt(json, key);
                textWriter.Write(encoded);
            }
        }

        public override JObject LoadFromFile(string saveFile)
        {
            string path = GetPathFromSaveFile(saveFile);
            if (!File.Exists(path))
            {
                return new JObject();
            }
            
            using (var textReader = File.OpenText(path))
            {
                string encoded = textReader.ReadToEnd();
                string json = EncryptDecrypt(encoded, key);
                return (JObject)JToken.Parse(json);
            }
        }
        public override string GetExtension()
        {
            return ".xor";
        }
    }
}

So there’s a bit to unpack here, as we can’t serialize and deserialize directly to the filestreams. The first method is our modified encryption cypher. Because this particular cypher uses the key to both encode and decode the file, the same method can be used to flip the bits on the file. It’s imperative that once you’ve chosen a key, you never change it, as it will render any future files unreadable.
If you use a text editor to read the encoded xor file, you’ll note some deficiencies in the algorythm… you’ll find places where you can see sections of the key… even if I were not to have shown you the key, it might not take you very long to figure out that it is the phrase used to teach young typists how to use every key in the alphabet. You’ll see fragments of the phrase in places due to the nature of the algorithm when certain complementary values are encountered. You can minimize this risk by using keys with more random characters. Try copying a UUID from one of your characters and using that as the key. The important thing is once you’ve set a key, you can’t change it or old save files will be unreadable.

The final strategy I’m going to leave you with is what I’m going to call the XorTextStrategy. Base64 encoding is a way of using only “readable” text characters to represent a binary file. It allows sending the file via email or over http as plain text (doing this with the results of Bson or the XorStrategy could cause accidental control characters to be interpreted, so transferring the files has to be done as binary files). While transferring binary files isn’t ordinarily a problem, some 3rd party cloud solutions will only accept pure text files. The XorTextStrategy will add an additional layer of encoding by converting the final product into Base64 encoding.

using System;
using System.IO;
using System.Text;
using Newtonsoft.Json.Linq;
using UnityEngine;

namespace GameDevTV.Saving
{
    public class XorTextStrategy : SavingStrategy
    {
        private const string key = "TheQuickBrownFoxJumpedOverTheLazyDog"; //This is a TERRIBLE KEY
       
        
        public string EncryptDecrypt(string szPlainText, string szEncryptionKey)  
        {  
            StringBuilder szInputStringBuild = new StringBuilder(szPlainText);  
            StringBuilder szOutStringBuild = new StringBuilder(szPlainText.Length);  
            char Textch;  
            for (int iCount = 0; iCount < szPlainText.Length; iCount++)
            {
                int stringIndex = iCount % szEncryptionKey.Length;
                Textch = szInputStringBuild[iCount];  
                Textch = (char)(Textch ^ szEncryptionKey[stringIndex]);  
                szOutStringBuild.Append(Textch);  
            }  
            return szOutStringBuild.ToString();  
        } 

        string EncodeAsBase64String(string source)
        {
            byte[] sourceArray = new byte[source.Length];
            for (int i = 0; i < source.Length; i++)
            {
                sourceArray[i] = (byte)source[i];
            }
            return Convert.ToBase64String(sourceArray);
        }

        string DecodeFromBase64String(string source)
        {
            byte[] sourceArray = Convert.FromBase64String(source);
            StringBuilder builder = new StringBuilder();
            for (int i = 0; i < sourceArray.Length; i++)
            {
                builder.Append((char) sourceArray[i]);
            }

            return builder.ToString();
        }
        
        public override void SaveToFile(string saveFile, JObject state)
        {
            string path = GetPathFromSaveFile(saveFile);
            Debug.Log($"Saving to {path} ");
            using (var textWriter = File.CreateText(path))
            {
                string json = state.ToString();
                string encoded = EncryptDecrypt(json, key);
                string base64 = EncodeAsBase64String(encoded);
                textWriter.Write(base64);
            }
        }

        public override JObject LoadFromFile(string saveFile)
        {
            string path = GetPathFromSaveFile(saveFile);
            if (!File.Exists(path))
            {
                return new JObject();
            }
            
            using (var textReader = File.OpenText(path))
            {
                string encoded = textReader.ReadToEnd();
                string decoded = DecodeFromBase64String(encoded);
                string json = EncryptDecrypt(decoded, key);
                return (JObject)JToken.Parse(json);
            }
        }
        public override string GetExtension()
        {
            return ".xortext";
        }
    }
}

While this will add an extra layer of obscurity to the save file (Base64 encoding uses 6 bits per character meaning that an invididual byte of the original save spans 2 characters, but in different ways depending on it’s position in the file), anybody familiar with Base64 encoding will be able to decode the file back to it’s original format. This is why I’m using it combined with the XOR, to allow an extra layer of security.

This concludes the Converting to Json tutorial. Hopefully, you’ve learned how to adapt your game’s Captures and Restores to be JTokens, and how to save and load your game data to and from Json files.

This tutorial’s final commit

2 Likes

An attempt to replace saving system with XML
I have just posted this. I don’t know if this works better than the Json but it may worth looking into.
Thanks.