Memory Woes

Discuss anything programming related.
Post Reply
User avatar
Grimdoomer
Admin
Posts: 1835
Joined: Sun Dec 09, 2007 9:09 pm

Memory Woes

Post by Grimdoomer »

Well while recoding Mutation I removed this gaint switch block that was atleast 1k lines, as well as two other switch blocks about the same size. I coded the class below in replacement. But after looking at it I think having an array of all the tag definitions might be a bit hard on memory, as each one is a complete layout with fields.

Code: Select all

public class TagCollection
    {
        public TagDefinition[] TagDefinitions = new TagDefinition[]
        {
            new  fx (), new adlg(), new ant (), new bipd(), new bitm(), new bloc(), new bsdt(), new HaloPlugins.Xbox.Char(),
            new clwd(), new coll(), new colo(), new cont(), new crea(), new ctrl(), new deca(), new DECR(),
            new effe(), new egor(), new eqip(), new fog (), new foot(), new fpch(), new garb(), new gldf(),
            new goof(), new hlmt(), new hudg(), new itmc(), new jmad(), new jpt (), new lens(), new ligh(),
            new lsnd(), new ltmp(), new mach(), new matg(), new mdlg(), new MGS2(), new mode(), new mulg(),
            new nhdt(), new phmo(), new phys(), new pmov(), new pphy(), new proj(), new prt3(), new PRTM(),
            new sbsp(), new scen(), new scnr(), new sfx (), new shad(), new sily(), new skin(), new sky (), 
            new sncl(), new snd (), new snde(), new snmx(), new spas(), new spk (), new ssce(), new stem(),
            new styl(), new tdtl(), new trak(), new udlg(), new ugh (), new unic(), new vehc(), new vehi(),
            new vrtx(), new weap(), new weat(), new wgit(), new wgtz(), new wigl()
        };

        public string[] TagTypes = new string[]
        {
            "<fx>", "adlg", "ant!", "bipd", "bitm", "bloc", "bsdt", "char", "clwd", "coll", "colo", "cont",
            "crea", "ctrl", "deca", "DECR", "effe", "egor", "eqip", "fog ", "foot", "fpch", "garb", "gldf",
            "goof", "hlmt", "hudg", "itmc", "jmad", "jpt!", "lens", "ligh", "lsnd", "ltmp", "mach", "matg",
            "mdlg", "MGS2", "mode", "mulg", "nhdt", "phmo", "phys", "pmov", "pphy", "proj", "prt3", "PRTM",
            "sbsp", "scen", "scnr", "sfx+", "shad", "sily", "skin", "sky ", "sncl", "snd!", "snde", "snmx",
            "spas", "spk!", "ssce", "stem", "styl", "tdtl", "trak", "udlg", "ugh!", "unic", "vehc", "vehi",
            "vrtx", "weap", "weat", "wgit", "wgtz", "wigl"
        };

        public string[] TagExtensions = new string[]
        {
            "sound_effect_template", "ai_dialogue_globals", "antenna", "biped", "bitmap", "crate", "breakable_surface",
            "character", "cloth", "collision_model", "color_table", "contrail", "creature", "control", "decal",
            "decorator", "effect", "screen_effect", "equipment", "fog", "material_effect", "fog_patch", "garbage",
            "global_lighting", "multiplayer_variant_settings", "object_properties", "hud_globals", "item_collection",
            "model_animation_graph", "damage_effect", "lens_flare", "light", "looping_sound", "lightmap", "machine",
            "match_globals", "ai_mission_dialogue", "light_volume", "render_model", "multiplayer_globals", "hud_interface",
            "physics_model", "physics", "particle_physics", "point_physics", "projectile", "particle", "particle_model",
            "scenario_structure_bsp", "scenery", "scenario", "sound_effect_collection", "shader", "ui_option",
            "user_interface_list_skin_definition", "sky", "sound_class", "sound", "sound_environment", "sound_mixture",
            "shader_pass", "speak", "sound_scenery", "shader_template", "style", "beam_trail", "camera_track", "unit_dialog",
            "sound_diagnostics", "unicode_string_list", "vehicle_collection", "vehicle", "vertex_shader", "weapon",
            "weather_system", "user_interface_screen_widget_definition", "user_interface_globals_definition",
            "user_interface_shared_globals_definition"
        };

        public int FindTag(string Key)
        {
            // Check Types
            for (int i = 0; i < TagTypes.Length; i++)
            {
                if (TagTypes[i] == Key)
                    return i;
            }

            // Search Extensions if null
            for (int i = 0; i < TagExtensions.Length; i++)
            {
                if (TagExtensions[i] == Key)
                    return i;
            }

            // Return
            return -1;
        }
    }
The tag defs then look something like this:

Code: Select all

public class itmc : TagDefinition
    {
       public itmc() : base("itmc", "item_collection", 12)
       {
           Fields.AddRange(new IMetaNode[] {
           new TagBlock("Item Permutations", 16, 32, new IMetaNode[] { 
               new Value("Weight", typeof(float)),
               new TagReference("Item", "item"),
               new StringId("Variant Name"),
           }),
           new Value("Unused Spawn Time   (in seconds, 0 = default)", typeof(uint)),
           });
       }
    }
Does anyone have a sugestion as to how I could eliminate the array, and replace it with something else other than a switch block?
Don't snort the magic, we need it for the network.
User avatar
XZodia
Staff
Posts: 2208
Joined: Sun Dec 09, 2007 2:09 pm
Location: UK
Contact:

Re: Memory Woes

Post by XZodia »

have an array of delegates which create objects
Image
JacksonCougar wrote:I find you usually have great ideas.
JacksonCougar wrote:Ah fuck. Why must you always be right? Why.
User avatar
Grimdoomer
Admin
Posts: 1835
Joined: Sun Dec 09, 2007 9:09 pm

Re: Memory Woes

Post by Grimdoomer »

I might be able to use reflection to create an instance based on the class name or some such. Would allow me to replace the TagDefinition[] with another string[].
Don't snort the magic, we need it for the network.
User avatar
JacksonCougar
Huurcat
Posts: 2460
Joined: Thu Dec 06, 2007 11:30 pm
Location: Somewhere in Canada

Re: Memory Woes

Post by JacksonCougar »

No idea what this is, but what I do is I have my method to create the layouts. And have them named the same with the only thing that changes being the name of the class. Use another method to return the structs. If that method creates a struct it adds it to a cache of them and returns the cached version if it is asked to get another copy of the same class.

Also having a copy of all the structs in memory will in no way rape memory as bad as some of the crap I do. That's like less than 5 Mb

Code: Select all

public static Tag_Block get_class_tag_block(string Class)
        {
            string TagBlockClass = TagEntry.FilterClass(Class);

            if (CachedTagBlocks.ContainsKey(TagBlockClass))
                return (Tag_Block)CachedTagBlocks[TagBlockClass];
            else
            {
                string MethodName = string.Format("get_{0}_tag_block", TagBlockClass);
                if (Methods.Length == 0)
                    Methods = typeof(Tag_Block_Classes).GetMethods(BindingFlags.Public | BindingFlags.Static);
                MethodInfo TagBlockMethod = null;
                foreach (MethodInfo Method in Methods)
                    if (Method.Name == MethodName)
                    {
                        TagBlockMethod = Method;
                        break;
                    }
                Tag_Block Tagblock = (Tag_Block)TagBlockMethod.Invoke(null, new object[0]);
                CachedTagBlocks.Add(TagBlockClass, Tagblock);
                return Tagblock;
            }
        }
User avatar
kornman00
Posts: 104
Joined: Wed Jan 20, 2010 7:48 pm

Re: Memory Woes

Post by kornman00 »

For the library I wrote for dealing with Halo based games I created something akin to Bungie's definition system that they use in their engine code (just tuned for .NET)

That is, I have actual tag_group objects which handle all actions and such for a specific group tag.

Code: Select all

/// <summary>
/// sound_cache_file_gestalt
/// </summary>
public static TagGroup ugh_ = new TagGroup("ugh!", "sound_cache_file_gestalt");

...

/// <summary>
/// All tag groups in Halo 2
/// </summary>
public static TagGroupCollection Groups = new TagGroupCollection(
// ...bunch of other TagGroup objects...
ugh_,
// ...bunch of other TagGroup objects...
);

...

ugh_.Definition = new Tags.sound_cache_file_gestalt_group().State;
That's it. Since all tag_groups are accessed via a specific game's TagGroupCollection, all the definitions will be initialized (one time process) in the game's static TagGroup class's cctor (static constructor).

Field editor names and other editor related data are kept separate from the actual definition field objects (as the way you're doing it only introduces data that will be reduntantly processed). The way I did this was via attributes but if I had the time I would go back redo a the systems for separating the various layers of the tag_group definitions as this is an implementation from 2006/2007 (and I don't like it anymore). All definitions of any sort (tag groups, blocks, etc) have their own runtime state object which handles everything from versioning (if the engine requires it) to field editor information.

Supporting 6 Blam! based games in this manner takes ~1 second of app startup time (on a crappy 2ghz laptop) and I don't inccur any performance hits later on from definition processing as it's all done on a one-time preprocess basis..
User avatar
kornman00
Posts: 104
Joined: Wed Jan 20, 2010 7:48 pm

Re: Memory Woes

Post by kornman00 »

Also, the way you add your fields to the owner definition isn't something I would consider optimal.

You're allocating an array with all the fields which you're going to add. That array will only get used once: in the ctor. After that it will (hopefully) be passed to the first generation of the GC. You'd be better off having a base ctor which has a parameter for how many fields the inheriting definition is going to add to the overall definition. That way the underlying definition system can allocate "Fields" to the correct starting capacity there by eliminating any overhead from reallocations. From there, you could just use single Field.Add calls there by eliminating that extra array allocation which takes time and memory to process.

The complexity of List(T)'s Add vs AddRange is great. The current implementation of AddRange just makes a call to InsertRange which has some extra processing itself. Add just ensures the current capacity then appends the item; a pretty small method which could probably get inlined in the JITed code in a release build.
User avatar
Grimdoomer
Admin
Posts: 1835
Joined: Sun Dec 09, 2007 9:09 pm

Re: Memory Woes

Post by Grimdoomer »

Well this array is only to get the definition, once I have it I'm going to create a new instance using the Activator class. So I wrote it planning to use it like so:

Code: Select all

int Index = TagCollection.FindTag("bitm");
Meta m = TagCollection.TagDefinitions[Index];
m = (Meta)Activator.CreateInstance(...);
I plan to make this a built in function in the TagCollection class. I will return a new instance of the tag def, and will add indexers to make it even more simple.
Don't snort the magic, we need it for the network.
User avatar
kornman00
Posts: 104
Joined: Wed Jan 20, 2010 7:48 pm

Re: Memory Woes

Post by kornman00 »

However, that array of "IMetaNodes" that you define in that ctor for the tag group is just used as an IEnumerable(T). It's tossed after the ctor ends. Even with the type activator it is still going to call that ctor and thus run that code again. Again, AddRange will impact the execution with more processing and allocations. If you're doing this for all tag_groups and tag_blocks, imagine how big of an impact that this will explode to during regular use of the code when you're loading hundreds of tags with thousands of block elements.
Post Reply