Problem with H2Core...

User avatar
Click16
Posts: 1941
Joined: Mon Dec 31, 2007 4:36 am
Location: United States

Re: Problem with H2Core...

Post by Click16 »

Also, once I have tag extraction and injection working perfectly, Abide will be allowed to transfer SBSP and LTMP tags between maps. It will be allowed to allocate space in the BSP Raw tables where the various BSP and Lightmap raw sections are stored, and allocating space in the BSP Meta table for the new BSP meta to go.
Image
User avatar
JacksonCougar
Huurcat
Posts: 2460
Joined: Thu Dec 06, 2007 11:30 pm
Location: Somewhere in Canada

Re: Problem with H2Core...

Post by JacksonCougar »

sbsp and ltmp data are discontinuous so you can really just allocate the space and write it without having to recalculate anything.
User avatar
Click16
Posts: 1941
Joined: Mon Dec 31, 2007 4:36 am
Location: United States

Re: Problem with H2Core...

Post by Click16 »

The Raw section (after the Model) The Lightmap and BSP raw seem to be independent of each other, however, the BSP and Lightmap Meta seem to be linked. The length that you find in the Scenario is both the BSP and Lightmap meta. The int at 8 in the BSP Meta is the pointer to the Lightmap meta. So when expanding BSP Meta, the subsequent BSP and lightmap metas will have to be adjusted.
Image
User avatar
JacksonCougar
Huurcat
Posts: 2460
Joined: Thu Dec 06, 2007 11:30 pm
Location: Somewhere in Canada

Re: Problem with H2Core...

Post by JacksonCougar »

Click16 wrote:[...] So when expanding BSP Meta, the subsequent BSP and lightmap metas will have to be adjusted.
I thought the same thing until I actually looked into it. In virtual memory all the sbsp and ltmp meta are in the same allocation—not in different allocations but in the same virtual memory. They all start at the same virtual address.

So all you need is a new chunk in the scenario pointing to a new file-location of the new sbsp and ltmp and you don't have to fix anything.
User avatar
Click16
Posts: 1941
Joined: Mon Dec 31, 2007 4:36 am
Location: United States

Re: Problem with H2Core...

Post by Click16 »

Well, in light of this information, I'm going to try to import a BSP...
Image
User avatar
XZodia
Staff
Posts: 2208
Joined: Sun Dec 09, 2007 2:09 pm
Location: UK
Contact:

Re: Problem with H2Core...

Post by XZodia »

You will have to fix the raw pointers of course...
Image
JacksonCougar wrote:I find you usually have great ideas.
JacksonCougar wrote:Ah fuck. Why must you always be right? Why.
xXF3RcHoXx
Posts: 338
Joined: Sat Sep 06, 2008 2:36 am
Location: Mexico

Re: Problem with H2Core...

Post by xXF3RcHoXx »

I want to get started in H2 app programming :c shall I read the "White paper" or someone would explain the map file format for me? Click16 already told me some things that are on the header and how they are related to other portions of the map but that's it

btw, I got a workaround, I made a new partition and installed Win 7 x64, right now, it seems to be working good, I have rebuilt a map with a shitload of new data and I didn't get any exception, plus, at some point, H2Core took a whole 2 GB of memory while rebulding, which surpasses it's previous record.

Click, if you can build a BSP tag into the map, then you can swap it on the scnr and voila, instant BSP conversion without rebuilding.
We are the last stand.
Aumman Anubis wrote:I'm still trying to figure out how a cat grooming defeats a failing economy.
User avatar
JacksonCougar
Huurcat
Posts: 2460
Joined: Thu Dec 06, 2007 11:30 pm
Location: Somewhere in Canada

Re: Problem with H2Core...

Post by JacksonCougar »

Rebuilding does much more than just building in the sbsp data however. It also adds in the shaders that bsp used and all the bitmaps. The halo 2 file format is pretty simple when you get right down to it.
xXF3RcHoXx
Posts: 338
Joined: Sat Sep 06, 2008 2:36 am
Location: Mexico

Re: Problem with H2Core...

Post by xXF3RcHoXx »

Click16 wrote:Well, in light of this information, I'm going to try to import a BSP...
Click, y u no answer to me in skype? :c
We are the last stand.
Aumman Anubis wrote:I'm still trying to figure out how a cat grooming defeats a failing economy.
User avatar
Click16
Posts: 1941
Joined: Mon Dec 31, 2007 4:36 am
Location: United States

Re: Problem with H2Core...

Post by Click16 »

I get your messages, but most of the time you're offline when I get on
Image
User avatar
Grimdoomer
Admin
Posts: 1835
Joined: Sun Dec 09, 2007 9:09 pm

Re: Problem with H2Core...

Post by Grimdoomer »

You guys are only thinking about this from a binary file perspective, and not a memory perspective. Sure you can add a new bsp to the binary map file and everything will seem okay based on UAT and being able to view the bsp and etc. But you need to be thinking about this from a memory perspective.

Anyone remember the time old issue of using entity to build in a bsp that was larger than the existing bsp causing the map to black screen? Remember using Gemini to do bsp conversions because it was the mp map with the biggest bsp? There is a reason for all of this. Halo 2 only allows for one bsp to be loaded into memory at a time, and the bsp meta comes before all of the other tags in memory. So if you build in a bsp that is larger than the existing bsp in the map, you will end up overwriting the bsp meta in memory with other tags, or vice versa, causing your map to black screen.

Lets open up a clean ascension.map and anyone interested can follow along. The memory layout for ascension is as follows:

0x80061000 - Memory Header
This is the start of memory for tag data in RAM. This includes the tag index, the currently loaded bsp and lightmap tags, and all of the other tags listed in the tag index. This value is constant throughout the game. Peeking this address in memory will get you the 32 byte header of the tag index.

0x80061020 - Tag Index + 32 bytes
This address is the start of the tag index (ex IndexOffset + 32), it is also constant throughout the game. It's the first int in the tag index header. Navigate to 0x007C8301 in ascension.map and you will see this value. Note: Halo 2 is written in little endian so you will need to byteflip the data to get the same address I have posted here.

0x800AB000 - BSP/Lightmap meta (of the currently loaded bsp)
This address is the start of the bsp and lightmap meta block. This value can be found in the "Structure BSPs" tag_block in the scenario tag. The bsp and lightmap meta are in one block, so they both get loaded at the same time. Each bsp/lightmap block has a 16 byte header added to it which is as follows:

Code: Select all

0 - BspBlockSize (size of the 16 byte header, bsp meta, and lightmap meta)
4 - BspMetaAddress (memory address of the bsp meta, ex BspOffset + 16)
8 - LightmapMetaAddress (memory address of the lightmap meta, ex BspOffset + 16 + BspMetaSize)
12 - BspGroupTag ("sbsp")
Halo 2 only allows for one bsp meta block to be loaded into memory at a time, and the bsp meta block comes before all other tags in memory. Anyone notice a problem with this? Remember trying to use entity to build in a bsp that was larger than the existing one and having your map freeze in game? Remember using Gemini to do bsp conversions because it's the mp map with the largest bsp?

Well here is the reason for both of those. Since the bsp meta block comes before the rest of the tags in memory, trying to replace it with a bsp that is larger will end up with you overwriting the bsp meta with another tags meta (starting with the matg tag) when the game tries to load the map. Loading the map into UAT and entity wont show you this, and you would never know! Your map would freeze and you would just chalk it up to shitty ol' entity.

So how do you fix this issue? Once the new bsp is in the map you have to recompute the memory address of all of the tags in the tag index. Then you recompute the secondary magic using the new memory address for the first tag in the tag index, the matg tag. Once you have your new magic value you can recompute EVERY, SINGLE, tag_block address in every non-bsp/lightmap tag in the tag index. Yup, every last single fucking one. Sorry guys who want to just shove data here and there and tweak a tag_block address, but that's not going to work with bsp! If you want to increase the size of the existing bsp or build in a larger bsp you best believe rebuilding the map is the easiest and, atm, the most reliable.

0x801AF000 - Meta Table (all other tags)
This address is really the memory address of the first tag in the tag index, the matg tag. It can be found in the tag index.

So now we can see that trying to increase the size of the bsp or building in a larger bsp will result in the map being broken at runtime, even though UAT or entity won't show you that. So how can you easily tell if your map is fucked up without having to load it in game? Simple, you use this tool I attached to the post. But first, lets do some algebra so you can understand what this tool actually computes, and how you can fail the arithmetic checks and still have a working map.

There are two equations we need to compute in order to tell if the index, bsp block, and meta table are all positioned in a way such that they do not overwrite each other. For the first we will using the memory header constant, 0x80061000, to check that the index does not overwrite the bsp block. For the second we will use the address of the first tag in the tag index, the matg tag, to check that the bsp does not overwrite any other tags.

We know that the Bsp comes after the index in memory, and our memory constant is the start of the index. So we know that:

Code: Select all

MemoryConstant = BspBlockAddress - IndexSize - 32
If we subtract MemoryConstant from both sides, thus setting the equation equal to zero we get:

Code: Select all

0 = BspBlockAddress - IndexSize - 32 - MemoryConstant
Viola! We can use this equation programmatically to check if the index overwrites the bsp meta block. If (BspBlockAddress - IndexSize - 32 - MemoryConstant) is not equal to zero then something is wrong.

Now to check if the bsp block overwrites other tags. We know that the address of the first tag in the meta table comes after the bsp block in memory. So we know that:

Code: Select all

Tags[0].Address = BspBlockAddress + BspBlockSize
Just as we did with the first equation we will subtract Tags[0].Address from both sides, thus setting the equation equal to 0 we get:

Code: Select all

0 = (BspBlockAddress + BspBlockSize) - Tags[0].Address
So now we have two equations we can use to check if memory will get corrupted when loading the map. But wait, we are only worried about overwriting memory, what happens if we build in a smaller bsp? Well this is where we can fail the arithmetic checks and still have a working map. Looking at the two equations:

Code: Select all

0 = BspBlockAddress - IndexSize - 32 - MemoryConstant
0 = (BspBlockAddress + BspBlockSize) - Tags[0].Address
For the first equation if (BspBlockAddress - IndexSize - 32) is less than MemoryConstant will we get a negative result. Similarly with the second equation if (BspBlockAddress + BspBlockSize) is less than Tags[0].Address we will also get a negative result. But we wont overwrite anything in memory which is what we are concerned with! Hello Gemini bsp conversion solution!

So that is how the tags work in memory in a nutshell. At this point you should understand why we can't just make the bsp larger, why rebuilding using H2C&G for bsp conversions is way more reliable than using entity, and why we use to use Gemini to do bsp conversions.

Now that some of you understand this, it's time I repost an app I made that no one understood what the hell it did. It does everything I just explained, and runs the two equations we found for you. It will spit out two lines and tell you if the index check passed and if the bsp check passed. But keep in mind, that the checks can fail when you have a negative result even though your map will still work!! As long as the numbers it spits out are less than or equal to 0, your map should work.
Attachments
MemoryChecker.rar
(4.09 KiB) Downloaded 295 times
Don't snort the magic, we need it for the network.
xXF3RcHoXx
Posts: 338
Joined: Sat Sep 06, 2008 2:36 am
Location: Mexico

Re: Problem with H2Core...

Post by xXF3RcHoXx »

Oh, so, in that Gemini case, what happens if you overwrite Gemini's BSP with a larger one? It won't work because of a memory issue or because the extra data actually overwrote data in the block next to BSP? I remember before H2Core, building a smaller SP map bsp into gemini was the only way.

Si, if you rebuild the map, H2Core or any other rebuilding tool if existed would recalculate the bsp size and start offset so you won't have anything to worry about?

Also, the fact that campaign maps have multiple BSP's, do they have a "special way" or something like that as the game loads them? of subsequent BSP's are smaller than the previous one to prevent memory overwriting? I know the games swaps the BSP using a script
We are the last stand.
Aumman Anubis wrote:I'm still trying to figure out how a cat grooming defeats a failing economy.
User avatar
XZodia
Staff
Posts: 2208
Joined: Sun Dec 09, 2007 2:09 pm
Location: UK
Contact:

Re: Problem with H2Core...

Post by XZodia »

Ah the Gemini rule that time forgot. I never did understand why that was the case, thanks for the explanation grim.

An interesting take away from what your saying grim, is that we could build multiple mp maps into gemini and they would all load in game. And (with the exception of gemini) they would all have extra space to add AI or whatever else to there bsp.
But the really fun thing which would be achievable, is adding bsp loading triggers so you could switch bsp in-game.
This is something I have dreamed of ever since I started modding...although I appreciate there are still issues to be resolved as all the maps would be in the same place. Even so, a map that just randomly changed every few minutes would be hilarious...
OMG awesome idea. The scnr spawns change when the bsp changes....so you could move objects around the map by changing bsp index (even if all the bsps used the same raw data)...

OMG super extra awesome idea. Might be able to use the aforementioned bsp loading technique to poke scnr object placements....

[/random thoughts]

@xXF3RcHoXx

In any case, if you add a larger bsp to a map (even it you add the space required) when the xbox loads the map, the extra bsp data will overlap with the other tag data.

Using H2C resolves this issue by fixing every pointer in every tag, so that they don't overlap when the xbox loads them.

Campaign maps have multiple bsps cause they were built that way.
Image
JacksonCougar wrote:I find you usually have great ideas.
JacksonCougar wrote:Ah fuck. Why must you always be right? Why.
User avatar
JacksonCougar
Huurcat
Posts: 2460
Joined: Thu Dec 06, 2007 11:30 pm
Location: Somewhere in Canada

Re: Problem with H2Core...

Post by JacksonCougar »

XZodia wrote:[...]
OMG awesome idea. The scnr spawns change when the bsp changes....so you could move objects around the map by changing bsp index (even if all the bsps used the same raw data)...

OMG super extra awesome idea. Might be able to use the aforementioned bsp loading technique to poke scnr object placements....
[...]
In theory I don't see why you couldn't just chunk-clone the reflexive in scnr without changing anything and setup your bsp-specific placements to that new index. Not sure what would happen behind-the-scenes but heyy
xXF3RcHoXx
Posts: 338
Joined: Sat Sep 06, 2008 2:36 am
Location: Mexico

Re: Problem with H2Core...

Post by xXF3RcHoXx »

XZodia wrote:Ah the Gemini rule that time forgot. I never did understand why that was the case, thanks for the explanation grim.

An interesting take away from what your saying grim, is that we could build multiple mp maps into gemini and they would all load in game. And (with the exception of gemini) they would all have extra space to add AI or whatever else to there bsp.
But the really fun thing which would be achievable, is adding bsp loading triggers so you could switch bsp in-game.
This is something I have dreamed of ever since I started modding...although I appreciate there are still issues to be resolved as all the maps would be in the same place. Even so, a map that just randomly changed every few minutes would be hilarious...
OMG awesome idea. The scnr spawns change when the bsp changes....so you could move objects around the map by changing bsp index (even if all the bsps used the same raw data)...

OMG super extra awesome idea. Might be able to use the aforementioned bsp loading technique to poke scnr object placements....

[/random thoughts]

@xXF3RcHoXx

In any case, if you add a larger bsp to a map (even it you add the space required) when the xbox loads the map, the extra bsp data will overlap with the other tag data.

Using H2C resolves this issue by fixing every pointer in every tag, so that they don't overlap when the xbox loads them.

Campaign maps have multiple bsps cause they were built that way.
So I can assing multiple BSP's in H2Guerilla and afterwards rebuild the map and everything will be ok?
We are the last stand.
Aumman Anubis wrote:I'm still trying to figure out how a cat grooming defeats a failing economy.
User avatar
XZodia
Staff
Posts: 2208
Joined: Sun Dec 09, 2007 2:09 pm
Location: UK
Contact:

Re: Problem with H2Core...

Post by XZodia »

xXF3RcHoXx wrote:So I can assing multiple BSP's in H2Guerilla and afterwards rebuild the map and everything will be ok?
Difficult to say without testing. However, I suspect H2C wouldn't like there being more than 1 BSP...
My multiple BSP idea would require a new tool which could build in a new BSP without removing the old one. (This is relatively simple compared to rebuilding the entire map, but still difficult)
If Click can get a tag builder working, it would be easy for him to add this feature.
Image
JacksonCougar wrote:I find you usually have great ideas.
JacksonCougar wrote:Ah fuck. Why must you always be right? Why.
xXF3RcHoXx
Posts: 338
Joined: Sat Sep 06, 2008 2:36 am
Location: Mexico

Re: Problem with H2Core...

Post by xXF3RcHoXx »

XZodia wrote:
xXF3RcHoXx wrote:So I can assing multiple BSP's in H2Guerilla and afterwards rebuild the map and everything will be ok?
Difficult to say without testing. However, I suspect H2C wouldn't like there being more than 1 BSP...
My multiple BSP idea would require a new tool which could build in a new BSP without removing the old one. (This is relatively simple compared to rebuilding the entire map, but still difficult)
If Click can get a tag builder working, it would be easy for him to add this feature.
Hey Xzodia, I played around with H2Core and I was able to rebuild a map with two bsp's :P Haven't tested if it works in game but if it does i'm gonna crap myself

I decompiled a sp bsp, then a mp map, then in guerilla, duplicated the only bsp chunk it has and referenced the sp bsp in the new chunk.

The map got rebuilt with no issues, plus I was able to load both BSP's in entity's bsp viewer.

Time to see if it works in game...
We are the last stand.
Aumman Anubis wrote:I'm still trying to figure out how a cat grooming defeats a failing economy.
User avatar
XZodia
Staff
Posts: 2208
Joined: Sun Dec 09, 2007 2:09 pm
Location: UK
Contact:

Re: Problem with H2Core...

Post by XZodia »

Can't wait to hear the results =)
Image
JacksonCougar wrote:I find you usually have great ideas.
JacksonCougar wrote:Ah fuck. Why must you always be right? Why.
xXF3RcHoXx
Posts: 338
Joined: Sat Sep 06, 2008 2:36 am
Location: Mexico

Re: Problem with H2Core...

Post by xXF3RcHoXx »

The map loaded with the first BSP (headlong), i'll try to switch bsp's in-game via scripting
We are the last stand.
Aumman Anubis wrote:I'm still trying to figure out how a cat grooming defeats a failing economy.
User avatar
XZodia
Staff
Posts: 2208
Joined: Sun Dec 09, 2007 2:09 pm
Location: UK
Contact:

Re: Problem with H2Core...

Post by XZodia »

Some Interesting ideas:

If you build a Singleplayer sbsp with a Multiplayer scenario into a Singleplayer map, does this get around the Gemini Rule?

If you Rebuild a Singleplayer map removing all but 1 sbsp then do a Mystery conversion, would we have a Multiplayer map sized (~50mb) file for that sbsp?
Image
JacksonCougar wrote:I find you usually have great ideas.
JacksonCougar wrote:Ah fuck. Why must you always be right? Why.
User avatar
Grimdoomer
Admin
Posts: 1835
Joined: Sun Dec 09, 2007 9:09 pm

Re: Problem with H2Core...

Post by Grimdoomer »

XZodia wrote:Some Interesting ideas:

If you build a Singleplayer sbsp with a Multiplayer scenario into a Singleplayer map, does this get around the Gemini Rule?

If you Rebuild a Singleplayer map removing all but 1 sbsp then do a Mystery conversion, would we have a Multiplayer map sized (~50mb) file for that sbsp?
Those are really round about ways to get around the problem.

I took a look at the source I have for H2Core, it's the 4-5-07 build. It uses the size of the first bsp to compute the secondary magic:

Code: Select all

int secondarymagic = primary + bsp_sizes[0];
I tried to modify H2Core to use the size of the largest bsp, or use a user specified size, but ran into a few issues. I'm using the source for an older version, and from looking at the code I'm not even sure H2Core could ever compile a working map with more than 1 bsp. I believe only the first bsp would load. I need to find a sp map and so some testing, however, I just don't have the time to do it tonight.
Don't snort the magic, we need it for the network.
xXF3RcHoXx
Posts: 338
Joined: Sat Sep 06, 2008 2:36 am
Location: Mexico

Re: Problem with H2Core...

Post by xXF3RcHoXx »

Grimdoomer wrote:
XZodia wrote:Some Interesting ideas:

If you build a Singleplayer sbsp with a Multiplayer scenario into a Singleplayer map, does this get around the Gemini Rule?

If you Rebuild a Singleplayer map removing all but 1 sbsp then do a Mystery conversion, would we have a Multiplayer map sized (~50mb) file for that sbsp?
Those are really round about ways to get around the problem.

I took a look at the source I have for H2Core, it's the 4-5-07 build. It uses the size of the first bsp to compute the secondary magic:

Code: Select all

int secondarymagic = primary + bsp_sizes[0];
I tried to modify H2Core to use the size of the largest bsp, or use a user specified size, but ran into a few issues. I'm using the source for an older version, and from looking at the code I'm not even sure H2Core could ever compile a working map with more than 1 bsp. I believe only the first bsp would load. I need to find a sp map and so some testing, however, I just don't have the time to do it tonight.
Well Grim I was able to rebuild a map with two bsp's (headlong and deltaapproach_bsp0), the map loaded with the only issue of having a fucked up sky. I added a script chunk to later attempt a bsp switch via scripting. Will try it l8r
We are the last stand.
Aumman Anubis wrote:I'm still trying to figure out how a cat grooming defeats a failing economy.
Post Reply