What causes Prop Pox (and how to avoid it)

[owlsinger]

I am computer stupid and know nothing about modding, but I am interested in what's happening here, because I have one infected city, and one I think will get the pox soon. Both are large city tiles with populations over a million. The city that is poxed has a .sc4 file size of 50.8mb, and a 2977aa47 file size of 18.5mb. There is one other large file in that city - 6a0f82b2, which is 24mb. (I have no idea what that means.) The city that I think will fall victim soon has a .sc4 file size of 59.9mb and a 2977aa47 file size of 19.4mb. The 6a0f82b2 file in that one is 30.6mb. There are no other files over 5mb in either city. I hope that info can help. I use downloads from both Simpeg & SC4D.

I have another question - in poxed cities, there is a phenomenon of peeps showing up in a crowd in the upper left corner of the tile. How might this be related? I went into my poxed city last night & observed that they are time dependent - a huge crowd shows up at 8am, and there are only a few in the corner at night. And, well, this really sounds nuts, but if you look at the bottom edge of that corner of the tile, you can see peeps appearing, and some sort of weird shadow. I have pics, if anyone is interested. It sounds crazy, but that's what I saw.

Thanks.
Kendra

[metarvo]

 

Honestly, I'm starting to worry for my cities now, and this has begun to scare me just a little.  It's one thing to have a temporary glitch like the Water Bug affect a city; at least that is not permanent.  It's another to have something like Prop Pox occur that could permanently ruin a good city.  If the props disappear as a result, are they gone for good?  At least my cities are small enough in file size (despite being large quads) that they won't be affected by this crazy epidemic.  Whatever it is, it sounds positively frightening.  Thankfully, I have enough advance warning to start monitoring my file size before anything bad happens, but I do have and use some of the mentioned content.  Does using it at all automatically set me up for Prop Pox, or should I just not use too much of it?

[allan_kuan1992]

Okay...

All three cities currently are in the 3000 - 4000 KB range (total filesize), which means 3 or 4 MB I think. Not bad I guess.

However expansion is planned for two cities (Oceania and Vanier), which are both around 3922 to 3942 kb. There's still some space left for it to grow.

The other city (Mount Oceania) may grow still but I tend to have trouble building on hills with skyscrapers xD. However, if I do (probably with suburban housing) the file will certainly bloat quickly (it's at 4230 kb at the moment).

I have replopped all beach lots after changing the exemplars (although i didn't destroy them beforehand).

I'll stay on the lookout for other potential problems... but so far so good...

- Allan Kuan

[BarbyW]

I am now totally confused. On 26th Feb Pegasus stated:

The game save file is an XML file and one of the sub files within it is the network file that stores the street and street prop data.

On 27th Feb he said:

You can remove or add almost anythig you want to the save file... and the city will still load. Its in XML format so its very flexible... and certainly does not use predefined memory locations or data block sizes.  XML just don't roll that way

After being asked how to display the file in XML Peg posted a screenshot of a Hex Editor screen and the following:

Oh... any old sector editor will do. There's one in the reader. No so great... but its integrated so helpful. I've used it to create HLS files for random query sound effects.

You are so not from the DOS era.

Yea!! Chalk one one for the Old School!! 

I have used a Hex Editor many times in the past - yes I am from the DOS era - and this does not appear to be the same as XML which looks like this:

<recipe name="bread" prep_time="5 mins" cook_time="3 hours">
   <title>Basic bread</title>
   <ingredient amount="8" unit="dL">Flour</ingredient>
   <ingredient amount="10" unit="grams">Yeast</ingredient>
   <ingredient amount="4" unit="dL" state="warm">Water</ingredient>
   <ingredient amount="1" unit="teaspoon">Salt</ingredient>
   <instructions>
     <step>Mix all ingredients together.</step>
     <step>Knead thoroughly.</step>
     <step>Cover with a cloth, and leave for one hour in warm room.</step>
     <step>Knead again.</step>
     <step>Place in a bread baking tin.</step>
     <step>Cover with a cloth, and leave for one hour in warm room.</step>
     <step>Bake in the oven at 180(degrees)C for 30 minutes.</step>
   </instructions>
</recipe>

Ref: http://en.wikipedia.org/wiki/XML#.22Well-formed.22_and_.22Valid.22_XML_documents
Could one of our more knowledgeable members please explain if these formats are the same or different?

[callagrafx]

HEX and XML are distinct and no, XML doesn't need to be read with a HEX editor.  Obviously the person in question is simply making it up as he goes along, without really knowing what he's talking about.    XML (or Extensible Markup Language) was developed as a way to create simple databases whose data could be re-purposed...in effect a repository of data that could be read by a variety of different sources and displayed.  I have used it in the past to create a database of product that was part of a Director program, at the same time as a web app.  All you need is Notepad.  I'm pretty sure the gamesave file is proprietary to Maxis....it just includes SOME markup.

Sir Arthur Conan Doyle wrote "When you have eliminated all which is impossible, then whatever remains, however improbable, must be the truth."  Bap has done that and come up with the theory that Maxis props, remodded to be timed, but retain the original TGI will cause the issues (and it is an issue with props, after all).  This has been accepted by pretty much all of the modding community with the glaring exception of the person who's releases are suspected of causing the issue.  It takes a big man to admit when they're wrong.....

Yes... absolutely. And its my humble opinion that that should have been done before the folks at SC4D went nutso with the doom and gloom, started a panic, changed their files and pressured other developers to do the same. I feel that was irresponsible to do based solely on a single post by an unknown individual before any attempt to validate the claims was made.

To clarify, no one's gone "nutso" or are scaremongering or are pressuring anyone to change their files...If Peg wants his stuff to be the cause of the loss of peoples' work, then that's his perogative.

[BarbyW]

Stage 2 Testing with BSC works. In each case city was run then saved and exited after each addition or bulldoze.

Added deadwood's Raised Parks plus all dependencies. Plugins is now 87.4MB

Opened test city. Plopped nothing.    Network File 747        Savegame file 243
Plopped one corner piece.      Network File 910        Savegame file 237
Added single tile with tree      Network File 946        Savegame file 237
Added 3x3 market          Network File 2497        Savegame file 248
Added 2x2 with azaleas         Network File 3328        Savegame file 247

Found I had missed a prop pack so added it. Plugins now 92.3MB   
Bulldozed and replopped all above   Network File 4588        Savegame file 250
Bulldozed all            Network File 748        Savegame file 238

At this stage there are no roads or any other form of transport network in the city.

Stage 3 Testing with BSC Basic Parks set plus all dependencies. Plugins is 66.6MB

Nothing plopped            Network File 748        Savegame file 237
Single tile with bench         Network File 1056        Savegame file 247
Added single tile with flowerbeds   Network File 1225        Savegame file 240
Added 3x3 with SG fountain      Network File 3913        Savegame file 246
Added 2x2 Azalea Park         Network File 5606        Savegame file 259
All bulldozed            Network File 748        Savegame file 240

There are still no transit networks of any type in this test city and I believe that the sub file is misnamed and should be Prop File.

[RippleJet]

There are still no transit networks of any type in this test city and I believe that the sub file is misnamed and should be Prop File.

That seems pretty obvious.
Especially after Adam's testing:

I made also some other experiments. 1 average building increases the 2977aa47 filesize by 1,5 - 2 KB. Transit networks without buildings doesn't increase this file size.

[RippleJet]

With the help of Dustin, Barby posted a comment on Simpeg explaining what she had corrected in the one prop pack where she had included modded Maxis props with unchanged TGI addresses.

She had added the property Building/prop family to them, which increases the size of the prop exemplar with 5 bytes.

Since...

Note -  No further comments or propaganda from the BSC will be accepted in this topic or anywhere else on this site regarding this matter. They will be deleted immediately.

...I will have to reply to Pegasus here and correct him on one particular statement:

Barby... Dear... changing a prop's family or adding it to a family... has ab-so-lute-ly  nothing to do with what Bap stated about the issue with modified props. You changed the material and propagated further panic for absolutely no reason at all. That's not "erring on the side of caution"... that's an irresponsible  knee-jerk reaction to mass panic.

This is what bap has said about the size of the prop exemplar:

Face to face with a cause of Prop Pox
Prop Pox seems to be caused by the program reading again an already stored prop the exemplar file of which is of different size at the second read.

How to avoid the problem
While the time dependent props of Pegasus beach pack are a cause of Prop Pox, they may not be the only cause of it. Any other modding of Maxis prop which changes (=increases) the prop exemplar file size will equally lead to Prop Pox.

I agree with Alex (Tarkus) and Andreas (as posted on Simpeg), can we please keep this civilized, and only discuss bap's findings?

[Ennedi]

Some of you say that the savegame file subfiles are of XML format. Would it be possible to decode them and see what data exactly are written in them?

[RippleJet]

Some of you say that the savegame file subfiles are of XML format.

The savegame format (like all other game files) is in Maxis proprietary DBPF format.

Would it be possible to decode them and see what data exactly are written in them?

That would be a test with plopping a lot with a single prop and saving the city...
Then checking the prop subfile (2977aa47) and comparing any newly added data with the prop exemplar itself...

[RippleJet]

There seems to be a misunderstanding with what the XML format is.

Instead of rewriting it here, I'll copy a post that Andreas made,
comparing the HEX representation of the prop subfile's DBPF content,
and the content of an XML file, also included in a Maxis game file:




The structure of the prop subfile (2977aa47) is so far unknown, as the case is with most subfiles in a saved game file.
Because of that Reader doesn't provide a decoder for it, and can only show it in HEX or garbled text format.

[paroch]

Well, I have to say I've found all of this very interesting.  Bap's approach to the testing impressed me very much and gives much credence to his findings - well done on this piece of work.

Luckily I've never been effected by PropPox but of course am concerned by it.  I did some checking that RippleJet suggested to see the size of my network sub folder in my West End cities.  I have three that have exceeded the 6MB file size:

Kingly:  18,700,726  (lots of  P-E-G canals and custom lotted buildings - I spent three months on this one quad so am not surprised it is so big)
Carnaby: 16,654,648  (contains a modular airport with quite a few custom lots)
Piccadilly:  6,362,324   ("normal large quad with lots of custom lots)

None of these cities have been infected/affected.  I've checked my plugins (I keep a seperate plugin folder for each city to allow for diversity of building styles) and can see that I do not have any of the  P-E-G OWW2, CDK-IND or Trail Parks and nor do I have BSC Mega Props Misc Vol2.  So even though these cities are larger than the 6MB size, the absence of the prop pack already identified with problems seems to suggest that I do not have any props with the PropPox problem.  But it should be noted that I did use  P-E-G's canals extensively in Kingly but assume that these do not use any Maxis props that have been modified.


My concern however is with the quad I am currently working on which is almost complete.  The city of Hanover has a network file size of 6,214,290 which is VERY close to exceeding 6MB.  This is the first quad I've developed using virtually all of the content from the LEX disc (I just wanted to see how it would look).  One of the prop packs in the plugins folder I created for Hanover is BSC Mega Prop Pack Vol2.

Now given this - the fact that my city is close to going over 6MB and it has one of the infected prop packs, I am happy if this quad is used for testing.  I will make a back up of the quad as it stands and not do anything else with it yet.  If someone would like to test it please PM me and I can email you the quad file plus a list of the plugins folder, or even just send the whole file (if that's something that's allowed).  If there is a simple test that I can perform then let me know and I'll happily share the results here.


Once again, great work Bap - I know this has been one of the unsolved mysteries of SC4.

[snorrelli]

In one of these posts I seem to remember reading that if the original maxis props load after the modified versions, the problem would be avoided. Is this correct or did I misunderstand? In other words, if I placed simcity1.dat in a ZZZ folder, would that solve the problem?

[Ennedi]

In one of these posts I seem to remember reading that if the original maxis props load after the modified versions, the problem would be avoided. Is this correct or did I misunderstand? In other words, if I placed simcity1.dat in a ZZZ folder, would that solve the problem?

If it would really solve the problem, you should extract only props data from simcity1.dat and place it into zzz subfolder. In other case you would lose the possibility of using lot of valuable things such as terrain mods, custom transit networks and many others.

[wouanagaine]

Paroch I think you'll need to post your file somewhere on the net so we can investigate, first only the savegame

If I can't load it, I'll ask for a plugin list. I'm also interest in seeing a screen shot of your city with network visible, so I can get a rough idea of what is needed to get a subfile close to 6mb

At least you proved half of Bap's assumption. I hope you're known enough in the community to be trusted and no one will discuss your report

[callagrafx]

I just want to go on record to answer this comment made by Pegasus on his site (he is free to post here, but seemingly hides behind his admin tools  ):

I will again ask that the BSC keep their speculations to themselves and stop trying to use this unjustified scare to further their own agenda.

First, this has absolutely nothing, zero, nada, the square root of sod all, to do with BSC....these findings were made by a member not affiliated with any team, organisation or alliegance.  To suggest otherwise smacks of a persecution complex.  Second, this is not an unjustified "scare"....the Pox is real and all evidence to date points to mis-modding practices.  Third, the BSC "agenda" hasn't changed from when it was founded, to provide assistance, advice, guidance and quality to the hundreds of BATers, MODers, LOTers and game players for SC4.  The BSC is not the Illuminati, despite what people and their "acolytes" mistakenly believe. 

[BarbyW]

Sticking purely to facts, I am posting the latest round of testing which convinces me even more that the network file should be called the prop file.

Using the same city as before I added in my BSC Props, BSC Textures and my BLS folder. These are all uncompressed and made a plugins of nearly 2GB. The three original Maxis items - the small park, wind turbine and community garden have been left in place throughout all the previous tests and also this one. As before after each line the city was saved and exited.

City opened but nothing added.   Network file 747  Filesize 248
Added single tile with water spout Network file 804 Filesize 246
Added swi rail crossing bridge  Network file 3148  Filesize 247
Added SNM MAAM Halifax Park Network file 5392  Filesize 261
Bulldozed all including the 3 Maxis items. There was no network file in the savegame file and the Filesize was 244.

Paul, so far as I can see, and I admit I haven't checked all the CSK files as I don't have them all, the only one with Maxis props that have been modded but with the original TGIs is the CSK Fountains. These are the same in the Trail Park fountain as well.

[Pat]

Hey Lee I could have sworn I seen this around floating 




Sorry I couldn't resist it... 

I think its pure well you know what!!!  But anyways I know first hand how hard you all work to help everyone including myself on several dozen occasions even when I was part of BLaM/LBT and now on ITC when I needed help when a team mate didn't know the answer someone from BSC was there helping me out!!!

Thank you to all of BSC!!!

[RippleJet]

Pat, the biggest thank you in this thread belongs to bap!

Would it be possible to decode them and see what data exactly are written in them?

That would be a test with plopping a lot with a single prop and saving the city...
Then checking the prop subfile (2977aa47) and comparing any newly added data with the prop exemplar itself...

With the help of Barby I decided to try to take a look at the file structure of the prop subfile (2977aa47).

Barby created a custom park with one single Maxis prop on it, the portable toilet, TGI 0x6534284A, 0xC977C536, 0x03040000.

It's a static prop that looks like this in Reader's Exemplar Analyser:



Barby plopped this lot in a new city, saved it and sent the SC4 file to me.
The size of the prop subfile (2977aa47) size was 88 (0x58) bytes.

After this Barby made a mod that changed this Maxis' portable toilet prop into a timed one.
However, she kept the TGI address unchanged:



Back in the city she plopped another of these parks, and saw that the prop was now timed.
She saved the city, exited, and sent it to me again.
The size of the prop subfile had now increased to 120 (0x78) bytes.

Comparing the prop subfiles to each other was the next step, and this is where it might become interesting.
The first DWORD clearly indicates the size of the record (0x00000058 and 0x00000078):



There are two chunks of numbers that are identical, underlined in yellow.
There is also clearly a reference to the TGI address of the prop exemplar:

0x6534284A - underlined in green is the Type ID
0xC977C536 - underlined in blue is the Group ID
0x03040000 - underlined in red is the Instance ID

However, and this is the alarming bit, it seems as if the instance ID has changed to 0x05040000 in the second file.
There are also fewer bytes between the yellow chunks and the TGI address in the middle.

If the Instance Address has changed to an invalid one, the prop exemplar would of course not be found.
- There is no prop exemplar with the address TGI 0x6534284A, 0xC977C536, 0x05040000.
- There is a prop exemplar with the address TGI 0x6534284A, 0x47BDDF12, 0x05040000 though.

On the other hand, we do know that Barby could clearly see the now timed props in her city...

I'll stop speculating here...
As we don't know what the other numbers in the prop subfile are, we cannot draw any final conclusions.
None of those numbers seem to match any property values in the prop exemplars at least.

[Ennedi]

(...)
However, and this is the alarming bit, it seems as if the instance ID has changed to 0x05040000 in the second file.
There are also fewer bytes between the yellow chunks and the TGI address in the middle.

If the Instance Address has changed to an invalid one, the prop exemplar would of course not be found.
- There is no prop exemplar with the address TGI 0x6534284A, 0xC977C536, 0x05040000.
- There is a prop exemplar with the address TGI 0x6534284A, 0x47BDDF12, 0x05040000 though.

On the other hand, we do know that Barby could clearly see the now timed props in her city...
(...)

1. Which prop is described by this second exemplar with changed Group ID (0x47BDDF12)?
2. Maybe it wouldn't be difficult to get to know which props are collected in this group?
3. Maybe changing a prop to timed one forces moving it to another group, and the game do it automatically? Everything would be OK in this case.
4. I understand that the whole TGI must be different for different objects, ie. if the Instance ID is the same, but Group IDs are different, addresses are different?