Resource System

This is the design document for the resource system. It's currently a work in progress, but it should be followed for future work on the resource system.

Overview of the Resource System

The resource system is our own internal platform-independent way to read files from disk that are platform-independent game data. This includes things like textures, sound effects, and game object models. While it would be straightforward to only focus on platform-independent path handling, we require a certain amount of information to be associated with resources, and some resources even contain game logic, such as maps. Resources can also be downloaded and/or installed automatically at the direction of a game server or the user. Therefore, we need a complete resource-handling system capable of much more advanced work while hiding underlying platform details from the rest of the game.

We have tResourceManager. In my view, it handles retrieving files from the cache/downloading, and will handle timing out cached resources and deleting them. It transparently handles bundled resources, installed resources, and cached resources, so none of the other components need care about it.

To help tResourceManager do it's work, we have tXmlResource. Currently there's nothing in tXmlResource, it's a placeholder. What it *should* contain, however, is the ability to read tags that are standard across all resources, such as the parent <Resource> tag. It should also understand tags that reference external files and, imo, provide methods for opening filetypes that are common amongst many components. This includes textures and models, as far as I'm concerned. So the code path to loading files into memory passes through tXmlResource. For an initial implementation, we can just put the actual code into tXmlResource, later we can work out a fancy way to register mimetype handlers to allow components to hook in their own special formats.

tResourceManager, then, will scan all resource directories creating its own internal list of tXmlResources, and it will use that list to do its work. Same for installing, when tResourceManager is called upon to install a resource, it will be given the current location of the resource, load it into a tXmlResource, and then do what it needs to do to install it.

Components will create their own classes as needed that inherit tXmlResource. Then they'll register that class with tResourceManager. When it's time for a component to request a resource, it will ask tResourceManager for an instance of that resource and tResourceManager will provide it, either by creating a new one, or by dynamic_casting the one from its list, or whatever else we dream up.

So the rest of the game, then, will only access resources through tResourceManager. No component will create new instances of a resource, all will always get points from tResourceManager. tResourceManager will always retain ownership of the resource instances, so components will not delete them, ever.

Design Goals

The design goals of the system are as follows.

• Low overhead for implementing new resource types
• Low overhead for using resources from outside code
• Scripts should be able to use the same interface to implement and use resources
• Insure integrity of resources
• Platform independence for all other code that uses resources (i.e. isolate platform-dependent code to the fewest possible places within the resource system)
• Separation from the game itself, to be provided as a shared object for other programs to use in a support role

Organization in the Source Tree

Ok, resources need their own directory. Right now, each directory under src represents a layer and they all stack on top of each other. I'd like to move it to where each represents a component, and they may be stacked, maybe not. Anyway, the resource system doesn't belong in tools, it needs access to stuff tools don't have access to.

Organization of Resources in the Source Tree

Todo: Write this

Organization of Resources in an installation

Todo: write this

Detailed Overview of Resource System

The resource system needs to be something where new resources can be added with a very minimum amount of work. It also needs to be flexible, since there's no point in adding resources if the resource system itself is going to restrict what types of resources can exist in the first place. We will hit a soft upper bound on useful resource types, hopefully before Bacchus gets out the door. But that won't mean we'll be done creating resource types.

Resources should be able to exist in script. So when we add scripting, we need it to have access to resources, and it needs to be able to create new resources.

A resource consists of several things. There are binary files of arbitrary format, some of which we'll support on our own, some of which we'll expect people that really want them to write scripts to support them. Several examples of binary files are jpg, png, ogg, mpg, wav, flac, tiff. The list goes on.

So we have these xml files that provide all the meta-data needed for the resource system to manage them. That's part of the resource.

In order to give the resource system the lowest overhead for people to use it, it needs to have a simple interface to the rest of the game. It should be as simple as: tResource* theResource = tResourceManager::getResource(myresource); and you get the xml file loaded into memory, you get the binary files all loaded that can be loaded (streamed files are an exception by their nature). Basically after this line, you can send textures from the resource to openGL for rendering with no intermediate steps required.

To support such a simple interface to the rest of the game, the resource system needs to be able to load any arbitrary binary file, and it needs to be able to do it in a generic manner. So you can't have, for example, LoadTiff, LoadJpg, LoadOgg, and so forth for methods. You need one method with one return type that is capable of loading any arbitrary binary file. Obviously that return type will have to be a class pointer, this is a problem solved in the cockpit already that can be copied into the resource system.

Normally, when you need one method with one return type in a bunch of classes you use inheritance to get that method there. So for gMap and gCockpitParser and gTheOthers they'll need to subclass tResource in some way or other, and tResource contains the LoadFile method (or whatever it winds up being called). So what does this LoadFile do?

tResource::LoadFile will call tResourceManager::GetFileLoader which will return a function pointer to the loader tResource has asked for, or it will return NULL for error, maybe it'll throw an exception. tResource's subclasses will use this file to load the actual binary data that goes with the resource. So for models this could be textures, could be another xml file that describes the model or part of the model, whatever. For sound effects this could be vorbis-encoded sound.

So tResourceManager keeps a list of loaders internally, indexed by type and mimetype, such as (texture, png) or (sound, mp3). Whatever. The other game modules will register their loaders with tResourceManager using a static nonsense-named object whose constructor just registers the loaders with all the other global variables. When tResourceManager scans its list of loaders, if none is found, it throws an exception. So if it sees, say, (texture, mp3), it can say "I don't have a loader that loads mp3s as textures". It might say "I have an mp3 loader, but it thinks mp3 is music and I can't use it". The loader itself needs to verify that the file is the right kind and the match has been made and so check the integrity of the file. If it fails, it also needs to throw an exception.

Game components should already know what they're loading, why, and what they intend to do with it. So as long as they ask for the right stuff, they should be fine. tResource::LoadFile should take two arguments, then. It should take a string that contains a type, such as "texture", "sound", "song", "model". Then it should take an identifier for the file.

The files are identified in a special section at the top of the resource xml file. Under the <Resource> tag should be a <Files> section that contains all referenced files and what they are. tResource understands these tags and is able to load them as described.

So it should be possible to implement a new resource with 3 steps:

1. Create the xml file and subclass tXmlParser 2. Create special binary file loaders 3. Register binary file loaders *and* the new subclass created in step 1

So after all this, we need tResourceManager to provide a single API that allows the rest of the game to retrieve, create, sort, delete, and do other things to resources. (Mind you, tResourceManager might ultimately find a vacation home as a shared object in a python library for other applications to use in resource creation) tResource needs to provide an interface that makes it possible for tResourceManager to do everything it needs to do, which is moving files around, expiring its cache, and so forth. The subclasses of tResource can extend the interface for their specific components, and they should do so. There's no reason gCockpit should be using a gMap to set itself up, after all. So gCockpitParser would have stuff in its interface that gMap doesn't have, and vice versa. But tResourceManager doesn't know about them, nor does it care for its work. (Except that it needs to be able to create instances of the subclass to give when asked, but it can do that and then dynamic_cast them for internal use. The cockpit/arena/whtever will have to dynamic_cast it back to the subclass for its own use)

Components of the Resource System

tResourceManager

This is the top-level interface. All outside code will retrieve resources from tResourceManager. tResourceManager will automatically provide these features:

• Automatic download and installation of resources - this is an on-demand service provided. When a game server specifies a resource the client should use, tResourceManager will download and install that resource and make it available to the code that requires it on the client side. The game server itself should also automatically download and install resources in the same fashion from the repository specified in its configuration files.
• Cache management - just like a web browser, tResourceManager should expire resources in the cache and manage the disk space used by the cache.
• Generate lists of resources - this is an on-demand service as well. Its primary purpose is to provide information needed to build UI elements for the user. Use cases include moviepack selectors, resource browsers, and map rotators.
• Loaders for files associated by file type and mimetype.

Features provided that generally require user action of some sort:

• Resource creation - Used by resource authoring programs, like a map editor.
• Resource packaging - Used by resource developers to create distributions of their resources
• More (I seem to have forgotten some of this momentarily)

tResource

tResource is the base class for all xml resources. It is also used as a standalone class by tResourceManager to perform all of the functions needed during the day of managing resources. All resources should inherit this class.

tResource provides a generic way to handle all resources using standard tags that exist in all xml files. It also provides a base class for components to use to implement their own special resources.

An internal registry will exist eventually that components will use to tell the resource system about their own special resources, so that when they query tResourceManager for their own special resources, tResourceManager can provide.

These two aspects combine to make it easy to add new resources, both in new code and potentially in scripting. All that's needed is to make your subclass of tResource and register it with tResourceManager. We also get comprehensive resource management within tResourceManager via tResource, and this is true for even new resource types, and we get this without sacrificing extensibility.

Registering your subclass of tResource is a matter of registering a callback that returns an instance of your subclass. Ideally, this callback would be a static method in your subclass. We should consider making that a requirement. Is it possible to make a pure virtual static method? If so, then that's all we need to do, and in your derived class's constructor, call something like tResourceManager::RegisterResource with it, similar to the mechanism for loaders.