Halo 3 Retail Game Research

[Supermodder911]

This post will be kept updated with the latest findings related to Halo 3 files. Currently:

• Fri Feb 15, 2008 - Topic started.
• Sat Feb 23, 2008 - Updated BLF Image

Note that code representations are done in C#, and so follow the C# syntax. This can be slightly confusing with the namings of some datatypes, so just to clear them up:

• Int: 32bit integer, or 'long'.
• String: ASCII/Unicode-encoded text of variable length.

Root -> maps -> images
File Extension: .blf
File Description: The body of these files is either in PNG, or JFIF format. The header has not yet been researched.
Extra: The JFIF file format is part of the JPEG family. The PNG and JFIF are also both recognised formats.
Structure:

Code: Select all

    struct BLFImageFile
    {
        string _blf; // Len4
        byte[] unk1; // Len48
        int fileSize; // Size of the File
        byte[] unk2; // Len8
        int dataSize;  // Size of IMGBuffer in bytes
        byte[] ImgBuffer;  // Either PNG or JFIF
        byte[] hash; // Len 272?
    }

Contributors: Prey, LuxuriousMeat, Supermodder911

Root -> maps -> info
File Extension: .mapinfo
File Description: The body of these files contain the name and description of the corresponding map (ie. salvation.mapinfo -> salvation.map), in the several different languages supported by the 360, along with an extra variation of Spanish. The header and footer have not yet been researched.
Structure:

Code: Select all

    struct MapInfoFile
    {
        byte[] _blfHeader;     // Len68
        byte Zero1;
        string EnglishName;    // All strings are unicode. All are Len64
        string JapaneseName;
        string GermanName;
        string FrenchName;
        string SpanishName;
        string LatinAmericaSpanishName;
        string ItalianName;
        string KoreanName;
        string ChineseName;
        byte[] Zero2;
        string PortugueseName;
        byte[] Zero3;
        string EnglishDesc;    // All strings are unicode. All are Len256
        string JapaneseDesc;
        string GermanDesc;
        string FrenchDesc;
        string SpanishDesc;
        string LatinAmericaSpanishDesc;
        string ItalianDesc;
        string KoreanDesc;
        string ChineseDesc;
        byte[] Zero4;
        string PortugueseDesc;
        byte[] Zero5;          // Len255
        string InternalName1;  // Len256
        string InternalName2;  // Len256
        byte[] Zero6;          // Varying length
        string Word_eof;       // Len4
        byte[] Unk1;           // Len269
    }

Contributors: Prey, Tural

Root -> maps
File Extension: .map
File Description: These files, for example, contain within them all the content used within that specific level (ie. guardian.map will contain the textures used in Guardian). Although some "special" maps contain assets that are used within more than one map, and thus referenced by others. These special, or "shared" maps are unplayable (ie. shared.map). Everything is sorted into "tag" structures within the map file, and these tags each have a pointer to their defining properties, or 'meta' (ie. the Mauler weapon tag stores how much ammo it may carry at any one time). Other miscellaneous stuff that is contained includes the Locale tables, Scripts, etc.
Extra: These files are of the byte order 'Big Endian'. Also near all objects within are padded to 4096. The padding is calculated by doing (4096 - (ObjectSize % 4096)) % 4096.

What's what:

Code: Select all

.map FileName | Corresponding Map
====================================
005_intro ----- Arrival
010_jungle ==== Sierra 117
020_base ------ Crow's Nest
030_outskirts = Tsavo Highway
040_voi ------- The Storm
050_floodvoi == Floodgate
070_waste ----- The Ark
100_citadel === The Covenant
110_hc -------- Cortana
120_halo ====== Halo
130_epilogue -- Epilogue
chill ========= Narrows
construct ----- Construct
cyberdyne ===== The Pit
deadlock ------ High Ground
guardian ====== Guardian
isolation ----- Isolation
mainmenu ====== Main Menu
riverworld ---- Valhalla
salvation ===== Epitaph
shrine -------- Sandtrap
snowbound ===== Snowbound
zanzibar ------ Last Resort
campaign ====== [Unplayable]
shared -------- [Unplayable]

General overview of map layout:

• Header - Len12288
• StringTableIndex - Padded
• StringTable - Padded
• FileTable - Padded
• FileTableIndex - Padded
• Assets..?
• Meta
• TagInfoHeader
• TagInfo
• TagClassIndex
• IndexHeader
• Unk
• Meta
• Locale Tables and Indices - Padded

Structures:
Header: The very first thing in the map file.

Code: Select all

    struct Header
    {
        string WordHead;            // Len4
        int Version;                // 11 For H3 Maps
        int Filesize;
        byte[] Zero1;               // Len4
        int IndexOffset;            // Address
        int VirtSegmentStart;       // Guess
        int VirtSegmentSize;        // Guess
        byte[] Zero2;               // Len256
        string BuildInfo;           // Len32
        short MapTypeIndex;         // 0=SP 1=MP 2=MM 3=Shared
        byte[] Unk1;                // Len26
        int StringTableCount;
        int StringTableSize;
        int StringTableIndexOffset; // Address
        int StringTableOffset;      // Address
        byte[] Zero3;               // Len4
        int Unk2;
        int Unk3;
        byte[] Zero4;               // Len24
        string InternalName;        // Len32
        byte[] Zero5;               // Len4
        string ScenarioName;        // Len256
        int Unk4;
        int FileTableCount;
        int FileTableOffset;        // Address
        int FileTableSize;
        int FileTableIndexOffset;   // Address
        int Checksum;               // Every 4 bytes xor'ed together after the header
        byte[] Unk5;                // Len32 - Constant
        int MapMagicBaseAddr;
        byte[] Unk6;                // Len128
        byte[] Hash;                // Len256
        byte[] Unk7;                // Len4
        int MapMagicAddrMod1;
        byte[] Unk8;                // Len4
        int LocaleTableAddrMod;
        byte[] Unk9;                // Len12
        int MapMagicAddrMod2;
        byte[] Unk10;               // Len11120
        string WordFoot;            // Len4
    }

StringTableIndex: Reached via StringTableIndexOffset found in the Header. Contains pointers to the starts of each string in the StringTable. They are in the same order as the strings, and are 4 bytes in length (int(32)) each time. They are also relevant to the StringTable (ie. string1's pointer will be 0, string2's pointer will be the length of string1 + 1 for the null terminator, etc).

StringTable: Reached via the StringTableOffset found in the Header. Comprised of ASCII strings, which are each ended with a null terminator (0x0).

FileTableIndex: Reached via the FileTableIndexOffset found in the Header. Follows the same structure as the StringTableIndex.

FileTable: Reached via the FileTableOffset found in the Header. Follows the same structure as the StringTable.

Index: Reached via the IndexOffset found in the Header.

Code: Select all

    struct Index
    {
        int TagClassCount;
        int TagClassIndexOffset;  // Address
        int TagCount;
        int TagInfoOffset;        // Address
        int TagInfoHeaderCount;
        int TagInfoHeaderOffset;  // Address
        int TagInfoHeaderCount2;
        int TagInfoHeaderOffset2; // Address
        byte[] Unk1;              // Len4
        string WordTags;          // Len4
    }

Extra: TagInfoHeaderOffset and TagInfoHeaderOffset2 are normally equal, and in those cases the second count is 0 (which, seeing as the count is always 0, may mean the value is completely depreciated.)

TagClassIndex: Reached via the TagClassIndexOffset found in the Index.

Code: Select all

    struct TagClass
    {
        string Class;            // Len4
        string ParentClass;      // Len4
        string GrandParentClass; // Len4
        int Identifier;
    }

TagInfo: Reached via the TagInfoOffset found in the Index.

Code: Select all

    struct Tag
    {
        short ClassIndex;
        int Identifier;   // Read as 16bit (short)
        int MetaOffset;   // Address
    }

Extra: ClassIndex references the TagClass at that index in the TagClassIndex.
Also because tag identifiers are actually 32bit, we have to convert our identifier from 16bit using the following method:
[code]Identifer <<= 16;
Identifier |= TagIndex; // Index in TagInfo list[/code]

TagInfoHeader: Reached via the TagInfoHeaderOffset(2) found in the Index.

Code: Select all

    struct TagInfoHeaderItem
    {
        string Class; // Len4
        int Unk1;     // Len4
    }

Magic
Addresses must have other modified addresses added/subtracted from them in order for them to point at their corresponding object in the map file.. opposed to in xbox memory. Current methods that have been found that calculate these "magic" values are as follows:

• HeaderMagic: StringTableIndexOffset - HeaderLength (12288).
  Usage In Header:
  • VirtSegmentStart -= HeaderMagic
  • StringTableIndexOffset -= HeaderMagic
  • StringTableOffset -= HeaderMagic
  • FileTableOffset -= HeaderMagic
  • FileTableIndexOffset -= HeaderMagic
• MapMagic: MapMagicBaseAddr - (MapMagicAddrMod1 + MapMagicAddrMod2)
  Usage In Header:
  • IndexOffset -= MapMagic
  Usage In Index:
  • TagClassIndexOffset -= MapMagic
  • TagInfoOffset -= MapMagic
  • TagInfoHeaderOffset -= MapMagic
  • TagInfoHeaderOffset2 -= MapMagic
  Usage In TagInfo:
  • MetaOffset -= MapMagic

Tag Meta
The meta of a tag is reached by following it's MetaOffset. Several structures exist inside the meta, here is a list of what you can expect: (Note that this list is currently incomplete)

TagRef: A reference to another tag.

Code: Select all

    struct TagRef
    {
        string Class;   // Len4
        byte[] Zero1;   // Len8
        int Identifier; 
    }

LoneID: A reference to another tag, but only by ID.

Code: Select all

    struct LoneID
    {
        int Identifier;
    }

Reflexive: A pointer to a list somewhere else in the map file, or 'tag block'. The pointer is translated by subtracting the MapMagic. The count of items in the list is also included in the structure. These items are usually referred to as 'chunks', and it is also possible to get reflexives inside of chunks; these are normally referred to as nested or inner reflexives.

Code: Select all

    struct Reflexive
    {
        int ChunkCount;
        int Pointer;    // Address    
        byte[] Zero;    // Len4
    }

Locale Tables
To reach these, you must first go to the "matg - globals\globals" tag meta + 0x1C4, and then there will be 12 structures, one after the other, each holding information for a different language. The structure is as follows:

Code: Select all

    struct Locale
    {
        int Count;
        int Size;
        int TableIndexOffset; // Address
        int TableOffset;      // Address
        byte[] Unk1;          // Len52
    }

The addresses are translated by adding the LocaleTableAddrMod found in the Header. The TableIndex's structure is as follows:

Code: Select all

    struct LocaleTableIndex
    {
        int Unk1;
        int StringIndex;
    }

The TableOffset must be added to the StringIndex to point to the string in the map. The Table follows the same structure as the StringTable. The strings also contain with-in them 'codes', which are translated to images by the engine before being displayed on screen. The codes are as follows: (incomplete)

Code: Select all

   Start: EE848C20
A Button: EE848020
B Button: 20EE8481
X Button: EE848220
Y Button: 20EE8483

Scripts
To reach these you must first go to the only "scnr" tag meta, and read in the following:

Code: Select all

0x3E0 Int Table Size
0x3E8 Int Table Offset
0x3F4     Scripts Reflexive
0x4A4     ScriptSyntaxes Reflexive

The Script's chunks structure is as follows:

Code: Select all

    struct ScriptChunk
    {
        string Name;
        short ScriptType;
        short ReturnType;
        short ExpressionIndex;
        short ChunkIndex;
        byte[] Unused; // Len12
    }

The ScriptType is an index into the following list:

Code: Select all

enum ScriptType : short
    {
        Startup,
        Dormant,
        Continuous,
        Static,
        Stub,
        Command_Script
    }

And as is the ReturnType into this list:

Code: Select all

enum ReturnType : short
    {
       UnParsed,
       SpecialForm,
       FunctionName,
       PassThrough,
       Void,
       Boolean,
       Real,
       Short,
       Long,
       String,
       script,
       string_id,
       unit_seat_mapping,
       trigger_volume,
       cutscene_flag,
       cutscene_camera_point,
       cutscene_title,
       cutscene_recording,
       device_group,
       ai,
       ai_command_list,
       ai_command_script,
       ai_behaviour,
       ai_orders,
       starting_profile,
       conversation,
       structure_bsp,
       navpoint,
       point_reference,
       style,
       hud_message,
       object_list,
       scenary,
       effect,
       unit,
       looping_sound,
       animation_graph,
       object_definition,
       bitmap,
       shader,
       UNK,
       render_model,
       structure_definition,
       lightmap_definition,
       game_difficulty,
       actor_type,
       hud_corner,
       model_state,
       value,
       network_event,
       value2,
       unit_name,
       vehicle_name,
       weapon_name,
       device_name,
       scenery_name,
       object_name
    }

The ScriptSyntaxes' chunks structure is as follows:

Code: Select all

    struct ScriptSyntaxChunk
    {
        short ExpressionID;
        short Identity;
        short ValueType;
        short ExpressionType;
        short SiblingPointer;
        short SiblingIndex;
        int ScriptStringOffset;
        byte[] Value; // Len4
        short Unk1;
        short Unk2;
    }

For now, further help with Scripts can be gained from the following Halo 2 resources: (Just keep in mind these were made for Halo 2, not Halo 3)
Soldier of Lite's Halo 2 Scripting Guide
xbox7887's Completed Script Database

Assets.. or 'raw'
Definite pointers and sizes have yet to be found for the assets, but this is a compilation of all the posts in this topic that may help us find them: (newest to oldest)
http://forums.halomods.com/viewtopic.ph ... 219#724219
http://forums.halomods.com/viewtopic.ph ... 872#713872
http://forums.halomods.com/viewtopic.ph ... 101#712101
http://forums.halomods.com/viewtopic.ph ... 849#710849
http://forums.halomods.com/viewtopic.ph ... 999#709999
http://forums.halomods.com/viewtopic.ph ... 904#709904
http://forums.halomods.com/viewtopic.ph ... 852#709852
http://forums.halomods.com/viewtopic.ph ... 351#707351
http://forums.halomods.com/viewtopic.ph ... 800#703800


Map Security
The map file is protected by a series of hashes, and only one/a select few know where they are and how to re-produce them (non-bungie employees that is). But they are keeping the knowledge to themselves. Whether or not they intend to one day release the info is unknown to me, but anyway here are the posts in this topic that may help the rest of us to figure it out: (newest to oldest)
http://forums.halomods.com/viewtopic.ph ... 952#708952
http://forums.halomods.com/viewtopic.ph ... 942#708942
http://forums.halomods.com/viewtopic.ph ... 495#701495
http://forums.halomods.com/viewtopic.ph ... 752#700752
http://forums.halomods.com/viewtopic.ph ... 640#700640
http://forums.halomods.com/viewtopic.ph ... 619#700619
http://forums.halomods.com/viewtopic.ph ... 436#700436
http://forums.halomods.com/viewtopic.ph ... 355#700355
http://forums.halomods.com/viewtopic.ph ... 080#695080

Contributors: Prey, Iron_Forge, Anthony, shade45, LuxuriousMeat, -DeToX-

Applications
Currently released applications that open at least one of the above:

5.
Name: Johnson

Click to be directed to post + download

4.
Name: Engineer

Click to be directed to post + download

3.
Name: Johnson
Author(s): Prey
Version: 1.3
Released: Mon Oct 22, 2007
Description: Can open map files, files from the 'images' folder, and files from the 'info' folder. Main purpose is to aid research into the map file, so expect lots of conveniences.

Download: Binary, Source

2.
Name: Engineer [Beta]

Click to be directed to post + download

1.
Name: Mango
Author(s): Prey
Version: 1.0
Released: Mon Sep 24, 2007
Description: Can open files from the 'images' and 'info' folders, as well as .map files.

Download: Binary, Source

E
Feel free to post any of your findings in this thread (of course related to Halo 3 files), and if I deem appropriate: I'll update this post with your research and add your name to the appropriate Contributors list(s).

[Supermodder911]

Updated.

[Supermodder911]

One hell of an update. o.O

lol, We gotta keep this topic on-topic.

[DarkShallFall]

One hell of an update. o.O

lol, We gotta keep this topic on-topic.

Testicles, that is all.

[Supermodder911]

In some weird way...I agree. lol.

[DarkShallFall]

There like gum on my shoe. srlsy.

[Aumaan Anubis]

Hypocritic member is hypocritic.

[Tural]

Let's stop talking about penises and start talking about vaginas.

[Grimdoomer]

Yea we're not all gay

[Avarorn]

Yea we're not all gay

We Already know you are

[NotZachary82]

Maturity. Srsly.

[((N8))]

Maturity. Srsly.

Haha, sadly, I agree.

Note: I love your JQ (Jackson Quote) btw