A More Generic way of retrieving resources

So I was working on another topic altogether, replacing enums with ScriptableObject place holders (just trust me, this is a great way to decouple your architecture, and give control to the designer) when I came upon the problem of saving an EquipmentItem along with its slot. It occurred to me that I could use the same mechanism used to save/restore an InventoryItem reference to save the EquipmentItem reference.

     static Dictionary<string, InventoryItem> itemLookupCache;
     public static InventoryItem GetFromID(string itemID)
        {
            if (itemLookupCache == null)
            {
                itemLookupCache = new Dictionary<string, InventoryItem>();
                var itemList = Resources.LoadAll<InventoryItem>("");
                foreach (var item in itemList)
                {
                    if (itemLookupCache.ContainsKey(item.itemID))
                    {
                        Debug.LogError(string.Format("Looks like there's a duplicate GameDevTV.UI.InventorySystem ID for objects: {0} and {1}", itemLookupCache[item.itemID], item));
                        continue;
                    }

                    itemLookupCache[item.itemID] = item;
                }
            }

            if (itemID == null || !itemLookupCache.ContainsKey(itemID)) return null;
            return itemLookupCache[itemID];
        }

So I copied the logic from InventoryItem into my ScriptableEquipSlot and adapted it to load the SES instead of the InventoryItem…

static Dictionary<string, ScriptableEquipSlot> itemLookupCache;
     public static InventoryItem GetFromID(string itemID)
        {
            if (itemLookupCache == null)
            {
                itemLookupCache = new Dictionary<string, ScriptableEquipSlot>();
                var itemList = Resources.LoadAll<ScriptableEquipSlot>("");
                foreach (var item in itemList)
                {
                    if (itemLookupCache.ContainsKey(item.itemID))
                    {
                        Debug.LogError(string.Format("Looks like there's a duplicate  ID for objects: {0} and {1}", itemLookupCache[item.itemID], item));
                        continue;
                    }

                    itemLookupCache[item.itemID] = item;
                }
            }

            if (itemID == null || !itemLookupCache.ContainsKey(itemID)) return null;
            return itemLookupCache[itemID];
        }

That’s when I started thinking… what if I need to use this yet again… this is starting to look like what we call a “WET” solution… (“We Enjoy Typing”/“Write Everything Twice”/“Waste Everyone’s Time”)… Whenever possible, we should be using DRY solutions (“Don’t Repeat Yourself”).

So I created an alternate solution using Generics. Generics allow you to declare a class using a generic placeholder… it’s saying “Ok, I have this class, but I don’t want to determine what it is right now, I’ll determine what it is when I use it.” Traditionally, that generic type is represented with a T.

     public class List<T>{} 

This is probably the most well known example of a generic class. It makes whatever class you want and turns it into a powerful flexible array.

I decided to use this powerful tool to make a Resource Retriever capable of loading any ScriptableObject from the Resources folder (subject to a simple contraint, The SO must contain the method GetItemID();

First, I need a simple interface to help with the GetItemID() function.

     public interface IHasItemID
    {
        string GetItemID();
    }

Just make any ScriptableObject you want to be able to load from a resource folder implement this simple interface, and the rest of the code will work flawlessly.

Now here’s the class that does the work:

// The Where T: ScriptableObject, IHasItemID means only in item that has 
//BOTH of these things qualifies.
public class ResourceRetriever<T> where T : ScriptableObject, IHasItemID
    {
        static Dictionary<string, T> itemLookupCache;
        public static T GetFromID(string itemID)
        {

            if (itemLookupCache == null)
            {
                itemLookupCache = new Dictionary<string, T>();
                var itemList = Resources.LoadAll<T>("");
                foreach (var item in itemList)
                {
                    if (itemLookupCache.ContainsKey(item.GetItemID()))
                    {
                        Debug.LogError(string.Format("Looks like there's a duplicate  ID for objects: {0} and {1}", itemLookupCache[item.GetItemID()], item));
                        continue;
                    }

                    itemLookupCache[item.GetItemID()] = item;
                }
            }

            if (itemID == null || !itemLookupCache.ContainsKey(itemID)) return null;
            return itemLookupCache[itemID];
        }
    }

So now, when you want to load a resource, you simply have to call the ResourceRetriever. You can remove the GetFromID() from the Inventory Item (or any other class you’re using it in), or in the case of InventoryItem, you might want to change the function to read:

public static InventoryItem(string ItemID)
{
     return ResourceRetriever.GetFromID(ItemID);
}
5 Likes

Thanks, Brian. Neat solution. I have not created any Generics, but with inventory soon to have multiple item types (weapons, consumables, armor, etc), I was thinking about ways to declare something as storable (e.g. IInventoryItem) to “wrap” my existing items (my WeaponSO, for starters). I know, I know…we’ll get there, but I think about/try to do things first, THEN watch the video. More challenging and I learn more this way.

Question (to make sure I fully understand your solution): With your generic call:
var = ResourceRetriever.GetFromID(ItemID)
that will load everything from every Resources folder that is a SO that implements the IHasItemID interface, correct? If so, if you wanted to restrict the search to just your ScriptableEquipSlots, would you do it like this:

ResourceRetriever<ScriptableEquipSlot> sesResourceRetriever = new ResourceRetriever<ScriptableEquipSlot>();
var = sesResourceRetriever.GetFromID(ItemID);

?

Actually, no. The GetFromID() is a static function. This means that you don’t need (or in this case want the overhead of) an instance. From outside of the class, you call a static method by using the class name.method name, so ResourceRetriever<yourclass>.GetFromID()

In the case of my ScriptableEquipSlot example, you’re looking to get a class reference of type ScriptableEquipSlot so the syntax would be like this:

ScriptableEquipSlot myScriptableEquipSlot = ResourceRetriever<ScriptableEquipSlot>().GetFromID(itemID);

Suppose you wanted to get a WeaponConfig using this:

WeaponConfig weaponConfig = ResourceRetriever<WeaponConfig>().GetFromID(itemID);

Excellent. Thanks for the clarification. Bookmarked this.

I research that in class ResourceRetriever, itemLookupCache is static,
but

ResourceRetriever<ScriptableEquiSlot>.itemLookupCache

and

ResourceRetriever<WeaponConfig>.itemLookupCache

will hold the separate values

That’s correct. With generic classes, the compiler actually creates a hidden class for each useage type found in the program. This means that each of these classes holds it’s own variables, including statics. Without this funcitonality, itemLookupCache would have huge conflicts, because ScriptableEquipSlot != WeaponConfig.

an error log when implementing class ResourceRetriever
“ReleaseAllScriptCaches did not release all script caches!”
I don’t know why the console just shows the message like that, nothing more

Hmm, researching this, it seems to be associated with Resources.Load (or with bad Editor Windows). I’ve never had it crop up personally, however, so I’m not sure what the exact cause is.

I have read a topic in:

I saw a comment with the keyword “OnAfterDeserialize”. So, I guess this issue relates to ISerializationCallbackReceiver.OnAfterDeserialize() and ISerializationCallbackReceiver.OnBeforeSerialize () at class InventoryItem.
In unity editor, I click into InventoryItem objects, like Boots, Bow, … and unity will recall these methods.
Luckily, the issue disappears, and it may not be relevant to class ResourceRetriever. Sorry about that.

EDIT: I was dumb. I didn’t need a <TU> on my LoadAll(). Revised and lightly tested. It seems to work.

How would one implement a cached version of Resources.LoadAll<T>()

So for example I’d want to do something like the following to lookup not just by ID but to have a list of all resources of a given type. I have a use case, where I want to pick a set of items/quests/etc that fit certain criteria so lookup by id won’t solve that. I need the bigger population first.

In doing so, it would be good to also populate the ResourceRetriever’s cache so that future calls to GetFromID just pull from the cache.

List<Quest> allQuest =  ResourceRetriever<Quest>.LoadAll();

I came up with the following enhancement to ResourceRetriever but not sure I’m doing things the right way.

    public class ResourceRetriever<T> where T : ScriptableObject, IHasItemID
    {
        public static List<T> LoadAll()
        {
            List<T> newList = new List<T>();
            
            if (itemLookupCache == null)
            {
                itemLookupCache = new Dictionary<string, T>();
                var itemList = Resources.LoadAll<T>("");
                foreach (var item in itemList)
                {
                    if (itemLookupCache.ContainsKey(item.GetItemID()))
                    {
                        Debug.LogError(string.Format(
                            "Looks like there's a duplicate  ID for objects: {0} and {1}",
                            itemLookupCache[item.GetItemID()], item));
                        continue;
                    }

                    itemLookupCache[item.GetItemID()] = item;
                    newList.Add(item);
                }
            }

            return newList;
        }
    }

I rewrote my original class a bit to provide all items of the specified type, and all itemIds as the specified type. I went with IEnumerables instead of Lists because 9 times out of 10, you’re most likely going to be using this as an IEnumerable anyways (generally my preference when dealing with collections is to work with them as IEnumerables when dealing with them as a one off, but store them as Lists when long term storage is needed.

public class ResourceRetriever<T> where T : ScriptableObject, IHasItemID
    {
        //Dictionary already holds all items, no need for separate list.
        static Dictionary<string, T> itemLookupCache;
        
        /// <summary>
        /// Retrieves item represented by itemID.  
        /// </summary>
        /// <param name="itemID">Must be unique across all instances of T</param>
        /// <returns>cached item represented by itemID or null if not in Dictionary</returns>
        public static T GetFromID(string itemID)
        {
            BuildLookup();
            if (itemID == null || !itemLookupCache.ContainsKey(itemID)) return null;
            return itemLookupCache[itemID];
        }

        /// <summary>
        /// All items of the Retriever's type (readonly). 
        /// </summary>
        public static IEnumerable<T> Values
        {
            get
            {
                BuildLookup();
                return itemLookupCache.Values;
            }
        }
        /// <summary>
        /// ItemIDs of all items of the Retriever's type (readonly)
        /// </summary>
        public static IEnumerable<string> Keys
        {
            get
            {
                BuildLookup();
                return itemLookupCache.Keys;
            }
        }
        
        private static void BuildLookup()
        {
            if (itemLookupCache == null)
            {
                itemLookupCache = new Dictionary<string, T>();
                var itemList = Resources.LoadAll<T>("");
                foreach (var item in itemList)
                {
                    if (itemLookupCache.ContainsKey(item.GetItemID()))
                    {
                        Debug.LogError(string.Format("Looks like there's a duplicate  ID for objects: {0} and {1}",
                            itemLookupCache[item.GetItemID()], item));
                        continue;
                    }
                    itemLookupCache[item.GetItemID()] = item;
                }
            }
        }
    }
1 Like

Brian,

I found a potential issue associated with this line of code.

var itemList = Resources.LoadAll<T>("");

If you’re loading an SO and it contains a reference to a Sprite or Texture (or something that could take up a lot of memory), then the itemLookupCache will conceivably load a ton of heavy stuff into memory that you may never use - yikes! I found this thanks to the Memory profiler and suspected this to be an issue due to some game crashes.

Solution? I guess use Addressables? Or make the particular offending SO class not implement IHasItemID. Example - I have all the flags of the world in my game. Each flag takes about 4MB when loaded in memory (2MB native + 2MB graphics memory). 4MBx200 = 800MB just for flags. I haven’t even gotten to the other stuff.

For my current situation I’m trying to figure out what’s easier (separate topic I know) but ongoing I think I have no choice but to go to Addressables.

You’re right. This could be an issue. The Resources.LoadAll is far from perfect, and this could be a big pileup with lots of large texture resources.

Speaking of large pileups… Yikes! In most cases, you won’t need a 1048x bit map or larger unless you are planning on the flag taking up the full screen (even then, better to rely on anti-aliasing). People tend to focus on mesh size as a bottlenecking issue with the graphics card, but it’s usually the textures… so I would definitely start with some size reduction there… For example, dropping to 512x would put those images at around 440k. But that’s not the real issue here… I will note that those flags are only in graphics memory while you have a model/quad that is displaying said flag. If they’re all going to be on the screen at once, you can get away with 256 or 128. As a side note: For icons (like the InventoryItems) I usually set my max resolution at 64x64.

Ideally, yes, addressibles is the way to go. It’s something I’ve looked into but not truly explored (though I do plan on taking them on sooner rather than later). It should be noted, though, that when you load an addressible, anything that is in the same “block” as that addressible is also loaded and not released until everything in that block is released…

We’re actually using Resources to make it possible to create a dynamic lookup at runtime to get Inventory Items quickly… Unity has actually been recommending for quite some time to move away from Resources.

Another solution might be to reference all of the InventoryItems on a GameObject in the Persistent Object Prefab, building the Dictionary there instead of in a static InventoryItem Loadall… (or more generically, as this hack I wrote has). For this, it might be best to have some Editor code to load all of the InventoryItems and store them as an array on the object. InventoryItem could then find the lookup and get the information there.
That was actually my original model before Sam introduced looking them up through LoadAll.

Yes. I think the simplest solution is to use the Asset Import Settings and resize to 256 on WebGL and 512 otherwise. The weird thing with flags is they have obscure dimensions so they are not in a format that is compatible with normal compression techniques and WebGL seems to have limited compatibility with the more advanced formats. I think this should allow me to defer looking into Addressables.

Interesting - the profiler said they’re in graphics memory after the Resources.Load call. This is running it in the Editor and I understand the Editor profiling isn’t always representative of a built game… but I am having trouble getting the remote profiler to work. Perhaps a firewall issue I’ll solve another time.

I actually have something not exactly like what you mention but has some similarities. I call it DataPrefetchManager and it is a child of the PersistentObjectsPrefab. The nice thing here (depending on how you look at it) is when a new scene loads, the unused assets get dumped but the look ups are still present. It helps address the problem below.

So here’s a weird finding Resources.LoadAll<T> will load ALL resources even not of type T. Again as per my profiling and memory snapshot taking. And again I know not to fully trust Editor based profiling. But I at least did validate that in a built WebGL game when I inspected the browser’s console that other assets were loaded as a result of Resources.LoadAll<Quest> being called.

Do you remember a while back a few folks were trying to figure out why Resources.LoadAll caused a major stutter in some cases. This could be it.

Loading a new scene should dump the unused assets so that should address part of the problem caused by the LoadAll. Because I have this in the PersistentObjectsPrefab it gets loaded on the main menu, then the unused stuff gets cleared when I load my first non-main menu scene.
This means Resources.LoadAll should never get called again.

public class DataPrefetchManager : MonoBehaviour
{
    private void Start()
    {
         ResourceRetriever<Quest>.BuildLookup();
         ResourceRetriever<Foo>.BuildLookup();
         // and so on
    }
}

It sort of has to… In order to know it’s of type <T>, it has to load the resource and check it’s type. (Not the most efficient feature in Unity. :slight_smile: That’s likely part of why they recommend not using it. That and Resources are always bundled with the executable, which eats into file size (like, say, Google’s 150MB limit for the primary AAB/APK!).

We know we need to move away from Resources, we just have to take the time to learn addressables right and get them taught well.

I guessed it might have discarded the stuff not of type <T> before it returns but I guess that would come with its own issues of constantly loading and unloading.

Looking forward to it!

Well, the Resources.LoadAll should only be being called once for each given <T> for the course of the game, that’s the whole idea of the lookup.

I can’t remember the Editor tool I tried to use earlier this year where the the author was querying the AssetDatabase every frame in the OnGUI() multiple times to present you with a list of assets with a particular tag… it was horribly unuseable. When I sent him a suggestion to cache the results, I was met with “you don’t know what you’re talking about” and ignored… I don’t use that tool anymore. :stuck_out_tongue:

Brian, After reading through these posts, I’m confused (as a relatively inexperienced coder) if I should implement the generic class with the memory overload potential problem? Do you still recommend your latest version using IEnumerable?

For now, the Generic class is the best solution I have to the issue. It can be an issue if your SO has very large textures in it. In our InventoryItems (from which this solution was based), we generally only have a small icon, but @Cat_Hill’s specific circumstances lead to a major bottleneck.

Thanks. I’ll try replacing the code in our RPG game.

Privacy & Terms