What causes Prop Pox (and how to avoid it)

[JoeST]

right, the Prop Pox are now on the wiki, [here]. I personally know very little, would be nice if anyone interested would add to it whenever they are free, just PM me for permissions if you dont already have them

Joe

[wouanagaine]

Thanks a lot Kendra

[bob56]

something I came across...

I was playing a city and got prop pox. I have a theory that the problem is PEG's Area 52. It has timed, modded, Maxis props. I am testing this theory in a new city using only that plugin. It will take time to get results, so I'm just letting everyone know I might have a cause.

Bob


EDIT: It was simply disappearing props. I will stay out of posting before thorough testing and leave it to the pros. 

[RippleJet]

It has timed, modded, Maxis props.

It does indeed include one modded Maxis prop:

• Street2x6x2_FedSUV, which is an untimed version of Maxis' Street2x6x2_$$$LuxerySUV_1D73

However, it has been given a unique Instance ID, and thus won't replace the ingame Luxery SUVs.
In other words, it won't lead to the prop pox.

Another prop included in Area 52 is Aper2x2x4_military_Armed Guard, which is a timed prop (set to appear between 13:00 and 12:00).
However, the RKT0 model (TGI 0x29A5D1EC,0x49A593E7,0x0AD00C00) referenced to by the RKT4 is not one by Maxis, and it is not included in the dat file either.

[BarbyW]

There are s few people who seem to think that solutions have been found and fingers pointed. This is not so. Many hours have been invested by some members of SC4D trying to find out how the savegame file works, what happens and how to prevent future incidences of prop pox. We know for certain that one prop causes prop pox - in every test this has been shown to be true. This is not to say that others may do this too but until we have far more cities to examine - both poxed and non-poxed - no one can say exactly what causes it.
No one is pointing fingers at anyone in particular. It is unfortunate that this is seen to be a an exercise in proving one person wrong. This is far from the truth and unfortunately the people who could explain better cannot post where they need to.
It is disheartening to see posts that demonstrate a complete lack of understanding of the problem. It is known that prop pox existed before the release of the BDK file but so far we have not identified any specific cause. As I said earlier we can only say that as one specific prop does cause prop pox then it appears that if there are other props similarly modded then they too could cause it.
I can also tell you that today I could easily have reported an outbreak of prop pox in one of my cities in GRV II. It wasn't prop pox as I checked the file in SC4Save but it gave the same appearance. I found out why. I had two folders for GRV II as I want one area to be predominately agricultural and one industrial and commercial. One folder did not have my farms. I had the other folder in place and entered my farming city to add a neighbour connection. That was all I did then I saved. When I returned to the city with the other folder in place all my farms were missing textures and props. Fortunately I had a back up from two days ago so was able to restore that city but it could easily have been reported a prop pox.
The tests that have been done DO confirm prop pox as no other custom lots have been used. Tests have been done with other props and textures in place but so far the only custom content identified is the umbrella prop from the BDK. This is fact not conjecture.

[RippleJet]

I had two folders for GRV II as I want one area to be predominately agricultural and one industrial and commercial. One folder did not have my farms. I had the other folder in place and entered my farming city to add a neighbour connection. That was all I did then I saved. When I returned to the city with the other folder in place all my farms were missing textures and props.

A clear case of Disappearing Props - not to be confused with Prop Pox

[RippleJet]

2. If the game changes the memory management to the "big" version (network file above 6 MByte) there is a high chance (aprox 33%) this will lead to prop pox. However it's possible that the management changes and no prop pox will appear. In fact this happens more often.

3. If the "big" memory management is active and you bulldoze enough buildings the game will change back to the "small" memory management. And this will always (!) lead to prop pox!

The appearance of the prop pox is very much related to what you're calling the "memory management".
Thus, I feel we need to explain why the size of the prop subfile suddenly can jump from 6 MB to 16 MB and v/v.

Thanks to Wouanagaine's excellent tool, SC4Save (an easier-to-use version of which is available on the LEX),
we are able to read the structure of the prop subfile in a lot simpler way than using Reader's HEX dump.
The images below are all from his tool, showing your different versions of Berlin.

In your large, unpoxed city, sized 24 MB, the prop subfile is uncompressed, having a size of 16,143,668 bytes (0xF65534):



Usually, the prop subfile is always compressed. However, when the size approaches 16 MB, it always should become uncompressed.
This is the reason for the jump from a compressed size of about 6 MB to 16 MB.

In the second city you uploaded (renamed City_-_Berlin 2 by me), the prop subfile is compressed.
Looking at the raw hex dump of the subfile, we see the following:

• The size of the compressed file is given by the first four bytes (0x005CF3DD = 6,091,741)
• The size of the uncompressed file is given by the three bytes at offset 0x06 (0xF0C9E8 = 15,780,328)

The reason for the city always becoming uncompressed when surpassing 16 MB, is to be found in the above.
For some strange reason, in the compression algorithm, Maxis only reserved three bytes to indicate the size of the uncompressed file.
The maximum value that can be stored in these three bytes is 0xFFFFFF = 16,777,215.
This is thus the maximum size of a subfile that can be compressed.



Now, to explain why we can say if a city is poxed, without even playing with it:



Wouanagaine's tool says there's a buffer corruption at offset 0x00DD6444.
Let's take a look at this offset in a "structured view", right down at the bottom of the prop subfile:



The record at offset 0x00DD6444 is the last one having a normal record size (0x58).
There are only a few possible record sizes available, 0x58, 0x68, 0x69, 0x6C and 0x94.

The last record, at offset 0x00DD649C has a record size of 0x28462CB4.
This is of course rubbish, and indicates that the savegame has stored some records at incorrect offsets.

This always happens when the city becomes poxed, erroneously stored records overwriting existing ones.
For each subsequent save after this, the number of erroneously stored records increase in numbers,
overwriting more and more properly stored prop records, which all disappear in the game.

The erroneously stored record size above is actually the memory address (third field) of a prop record
that has been erroneously stored 8 bytes earlier, at offset 0x00DD6494.

We know that each prop record normally occupies 104 bytes (0x68) in memory,
e.g. the difference between the last two records, 0x27F4B674 - 0x27F4B60C = 0x68.

Thus, comparing this last memory address (0x28462CB4) with the previous one (0x27F4B674),
we get a difference of no less than 0x517640. Dividing that by 0x68 would indicate some 51,333 missing props...

All props appearing with erroneous offsets in the end of the prop subfile also always seem to be flagged as "disabled" (or deleted).

At the moment we cannot explain exactly why the savegame screws up the records in the end of the file.
It is quite obvious that it does happen when the prop subfile's size approaches 16 MB (or 0xFFFFFF).
It most likely is due to the savegame trying to compress the prop subfile when surpassing this limit.
And it has also been confirmed that the savegame screws up only if there are modded Maxis props involved.

[Diggis]

Tahill, can you do me a favour (or anyone else with access to Pegs site)? Can you put a post up, feel free to say it's from me, pointing out that Pegs theory of file size and custom content being installed has been disproven by Bap's original set of testing, and all subsequent testing on the files Bap provided.  He developed a city up to the 6mb limit using soley MAXIS CONTENT.  These cities are the ones linked in the original post now and are available for download.  He then added in ONLY the BDK resource kit (or what ever the hell it's called) and got the pox.  Thats it, nothing else.  He proved, beyond doubt to most people, that those files DO cause the Pox.  We are now trying to find out what else does, as it has been pointed out the Pox is older than the BDK. 

[FrankU]

Yes, and it might be nice to stress that nobody blames him for that, because the problem is very weird and nobody could ever have foreseen that.
So no blame, but please believe us PEG, and maybe you will be so nice to reedit the BDK set?

[RippleJet]

Tahill, let me quote you from over there:

combination of plugins size and city size?

Tahill, I think you need to read bap's post below once more.
This is also the post where you should have found the links to his saved cities:

Testing Prop Pox by yourself

Basic Information

For those willing to check my experiments with Prop Pox, I uploaded three versions of the test city #2 at fileden.com. They can be downloaded at the following addresses:

Riquelandia 4

Riquelandia 5

Riquelandia 6

The relevant information about these cities is as follows:

filename         total size(Kb)   2977aa47 size (Kb)
Riquelandia4       15967              5926
Riquelandia5       16159              6162
Riquelandia6       15923              5742

Riquelandia4 was build exclusively with Maxis vanilla lots & props up to this point. If developed in a plugins without Maxis-modded infected props it will not show Prop Pox. There is room for further zoning in the south and east outer bounds of the city, just inside the highway belt. There is also room for development at the southeast corner outside of the highway belt.

Riquelandia5 is a further development to Riquelandia4 made while the infected props were in the plugins folder. Thus, it will get Prop Pox if you let it grow just a little bit more -- even if the infected props are no longer in the plugins folder. No big development effort is needed here because the city is on the verge to show signs of Prop Pox. Because only custom prop packages were in the plugins (not the lots), anyone can play with this city without being in risk of seeing brown boxes. All installed lots up to this point are Maxis vanilla ones.

Riquelandia6 is a saved version of Riquelandia5 just after it got Prop Pox. As you can see, the city file and the 2977aa47 file both shrank. Several props are missing at different places around the city (look for city parks, water treatment plants, farms).

When developing Riquelandia 4, bap had a plugins size of 0 bytes.
For Riquelandia 5 and 6, bap only had four props by Peg in the plugins folder.

The size of the plugins folder does not matter... unless four props (a few bytes) is what you'd consider sizeable.

Tahill, you may copy that post by bap in its entirety over to Pegs,
so that he can have a city to test that doesn't require any custom content, other than his own.

[RippleJet]

There is nothing that really indicates that any particular type of prop... default custom or modified... would cause this problem with the game's save routines. But there is quite a bit of evidence that indicates that having too many props in a city will trigger the glitch.

Yes, there is evidence that the city prop subfile must be close to 16 MB (uncompressed) for the prop pox to appear.
There is also evidence that the prop pox can be triggered with certain specified props.

As we stated before, growable lots are the largest collective source of props in a city. We know the game can handle all of the game-default lots no matter how large and well developed the city. So logic & simple deduction then points us toward custom growable lots that use a large amount of extra custom props.

What would be considered too many props in a city?

We know that each prop (whether it's an in-game prop by Maxis or a custom prop), is saved in the prop subfile.
We also know that if you have 100 instances of the same prop in the city, they will each occupy one record in the prop subfile.
Having 100 separate custom props will occupy exactly the same size in the prop subfile, as 100 identical props, whether by Maxis or custom ones.

We know that each prop has a record size of 88, 104, 105, 108 or 148 bytes.
104 bytes is the record size of a prop being timed between two dates.
148 bytes is the record size of arrow props indicating neighbour connections.

To reach a filesize of 16,777,216 bytes, we'd need 190,650 props having a size of 88 bytes.
In a large city (256×256 tiles) that would mean an average of 2.9 props per tile.

Now, let's take a look at some of Peg's own recent creations:

PEG Corner Starbucks Growable	Lot Size 2×2	26 props	6.50 props per tile
PEG Corner Starbucks Ploppable	Lot Size 2×2	29 props	7.25 props per tile
PEG Area 52	Lot Size 6×4	108 props	4.50 props per tile
PEG MTP TLC Rail Transfer Yard	Lot Size 4×5	69 props	3.45 props per tile
PEG MTP Log Town Cross Roads Center	Lot Size 3×3	66 props	7.33 props per tile
PEG MTP Log Town Fuel Depot	Lot Size 2×2	49 props	12.25 props per tile
PEG MTP Gazebo Park	Lot Size 1×1	15 props	15.00 props per tile
PEG UT Large Plaza	Lot Size 3×3	64 props	7.11 props per tile
PEG-UT Mall Canal Corner	Lot Size 3×3	70 props	7.78 props per tile

Most in-game lots also have more than 3 props per tile. Below are a few extreme examples:

Grass Park	Lot Size 1×1	0 props	0.00 props per tile
Gazebo	Lot Size 1×1	29 props	29.00 props per tile
Large Plaza	Lot Size 3×3	52 props	5.78 props per tile
Urgent Care Clinic	Lot Size 1×2	15 props	7.50 props per tile
Small School	Lot Size 2×3	13 props	2.17 props per tile
City College	Lot Size 5×3	59 props	3.93 props per tile
University	Lot Size 12×10	577 props	4.81 props per tile
R$1_1x2	Lot Size 1×2	12 props	6.00 props per tile
R$1_1x2	Lot Size 1×2	19 props	9.50 props per tile
CO$$$8_4x4	Lot Size 4×4	56 props	3.50 props per tile
CO$$$8_4x4	Lot Size 4×4	8 props	0.50 props per tile

Peg's MTP gazebo is better than the in-game one.
Otherwise, I cannot really see any advantage in using e.g. ploppable log town lots over growable ones.

It's another matter that nobody would be filling a complete large city tile with ploppable lots.
However, it isn't a question of which lots give the most props and which to avoid,
it's a question of loosing a city after you've developed it to a point having more than 150,000 props.

Of course, I'm sure it hasn't escaped anyone's notice that the group trying to accuse a handful of modified props as being the source of the issue... are also the number one providers of custom growable lots that use a very large amount of custom props.

Nobody is accusing you, Peg.
However, bap was accused of presenting the prop pox as a hoax.

As i said above, the number of custom props is irrelevant.
All props, whether custom, in-game or modded in-game ones occupy one record in the prop subfile.

The final conclusion we can draw from all this is that to avoid "Prop Blight" or issues related to the game save bug... just use some common sense in what custom growable material you install or use.

Indeed, I think most of us know which files to avoid by now.

[Lord_Morpheus]

@RippleJet

I already opened a topic on simpeg. Thanks for excellent explanation about the compressed and uncompressed network file.

[Lord_Morpheus]

Well now my topic on simpeg got deleted^^

Oh my god. Nice one. Well I just tried to help and this is the answer. Nice 

[RippleJet]

I did manage to download Helmstedt before the thread disappeared though!
Maybe you'd like to post the links to RapidShare here as well?

The unpoxed Helmstedt is a perfect test ground,
having an unpoxed prop subfile sized 0x1005198 (only just over the limit of 0xFFFFFF).

[Lord_Morpheus]

Helmstedt unpoxed: http://rapidshare.com/files/214572578/City_-_Helmstedt.sc4
File Size over 26Mbyte, uncompressed network file

Helmstedt poxed: http://rapidshare.com/files/214578158/City_-_Helmstedt.sc4
File Size under 16Mbyte, compressed network file

If you want to test the issue yourself:
- delete your plugin folder (make a backup elsewhere  )
- download the large city file and verify you can load the city
- quit the game
- install OWW2 set
- load the city again
- there are four maxis beaches in the city, bulldoze them!
- save the city
- quit the game and take a look at the savegame size
- restart the game
- load the city
- go to the east side of the city and look around a little bit, it will be poxed now!

[RippleJet]

Thanks, Lord_Morpheus!

I'd say what you posted on Simpeg was all in line with this post by Pegasus:

Note -  No further comments or propaganda from the BSC will be accepted in this topic or anywhere else on this site regarding this matter. They will be deleted immediately. However... any legitimate member of the community who has been a victim of this scare tactic, may post about their concern and will we alleviate your fears with documented facts.

I suppose your fears have been alleviated by documented facts now:



Yes, the issue is rare, but that won't comfort those who've lost their largest cities.
Knowing that 10,000+ other players have not experienced the pox won't help much once you're there.
Besides, the solution to keep the pox from spreading in the future is so incredible simple, as bap showed already in his first posts.

[tahill79]

It's obvious that the creator in question will not modify his work, so I think the safest thing is either to mod it yourself or simply not use it or any other that has been identified as a cause.

Oh.  Hey look everyone,  a simple answer for a small problem.  Or just dont use the content.  Its pretty simple when it comes down to it.  There are some things in the LEX that i wont use.  Reasons being too many dependencies.  The community found out about this over 2 years ago,  so why the all of a sudden freaking out over it now?  

Listen,  Im a big fan of what everyone does for this game.  Im the lucky one who has never had PP.  So to all of you that have had it happen,  sorry.  Demolish the building and put something else there.  

Morpheus,  im sorry again that your thread was deleted.  Im not a mod over at SimPeg.com.  Just a PCG.  A Beta Tester.  Thats all I am there.  Peg posted a reason and I hope you read it.  I dont want to make any cyber enemies so please dont be mad at me for what I have posted here and at SimPeg.


This is my last post in this thread.  If you guys have any questions or just wanna say hi, please PM here, at ST,  or on SimPeg.

thank you

[Andreas]

Im the lucky one who has never had PP.  So to all of you that have had it happen,  sorry.  Demolish the building and put something else there.

Maybe you should re-read what has been posted so far. Bulldozing a building that might cause prop pox doesn't help, it isn't even necessary to build it at all, as the mere presence of it in the plugins folder might lead to prop pox. Also, prop pox is not reversible, so bulldozing will do nothing. It's sad to see that something that simple to fix is called "stirring panic" and denied eventually. However, I have to admit, it doesn't surprise me, after all what happened recently.

[callagrafx]

No one is freaking out....Why does Peg still insist that we're creating a scare? The Pox exists and it's a fact, and instead of burying our heads in the sand and deleting anything of relevance, certain members have been labouring away really quite hard to understand WHY it's happening. Wouanagaine has written a program that helps better understand the savegame file, Ripplejet has done the most exhaustive research into the raw data I've ever seen, Barbyw, Ennedi, Bap and Lord Morpheus have done test after test after test and that has resulted in Barby discovering a potential problem with a BSC release, but instead of denying that one of her files could be a cause, she promptly fixed it and re-released, which is all that has been asked of Pegasus.  Instead they are presented with a diatribe of nonsense posts from Peg and his acolytes who have no understanding of the undertaking.  The Pox is rare yes, but so is Ebola...should people stop trying to cure that too, as it only affects a handful of people.  And NOWHERE has Bap "recanted" that any Pegasus releases are a contributory factor....in fact it has been proven by Lord Morpheus that one is.  Why is Pegasus so against this research?  Maybe it's because it has been proven that they are responsible and he is not man enough to admit when he's wrong.  

[BarbyW]

Once again Pegasus is denying that there is a problem and doesn't just lock a thread but deletes it as if what has been posted is unimportant. He will not log in here to discuss this subject and Ripplejet, Wouanagaine, Andreas and I are all banned from his site so cannot post there.
To answer his points in the above post:

1. Panic? I see no panic being generated here nor is there any finger pointing as I explained in my last post. In fact I posted that I had tested with all other Peg lots and packs in place apart from the BDK and had not had a poxed city. I have been testing with a variety of other custom content but as I have also explained before it is a long and tedious process.

2. Pegasus has obviously not seen the previous posts at Simtropolis and here about the problem. He has not seen the posts of CJers who have lost whole cities to this problem. bap reported a problem last year and was interested enough to conduct tests over a long period of time. These have been dismissed as not valid as bap does not appear to be a known member of the community.

3. bap has never recanted his staement and it has been corroborated by others independently. I posted my findings on 20th March.

4. None of the props are corrupted per se. The problem seems to occur when a prop that is widely used on Maxis growables - especially the low density R$/R$$ - is modded to a different timing but without changing the IID of the exemplar. I have checked many of Pegasus' dats and the vast majority of time when he has modded Maxis originals have been given new IIDs. I simply do not understand why the four props in the BDK weren't.

5. I have never had a poxed city. I have never had a city with a savegame file anywhere near 25MB in size. That does not mean I deny the problem nor that I would not like to find a solution. I feel it is encumbent upon all custom content providers to ensure that their work does not and cannot cause problems - so far as they are able. If problems are found it is then responsible of the provider to correct the problem.

To date once again BSC have been villified and blamed for this whole situation. The original reporting of the problem was by snorelli - not BSC. Others then joined in with their experiences: Fledder, Nardo69, BruceAtkinson - none of them BSC. bap is not BSC. Lord Morpheus is not BSC.
To turn to the people who have been investigating - wouanagaine is a respected member of NHP first and foremost. He works in the computer gaming industry and has programmed a number of useful tools for the SC4 community. He programmed the tool for decyphering savegame files that he has now released to the LEX. Ripplejet is one of the foremost technical modders in the community. He has analysed and posted about the blank lots problem; he discovered how to increase the growth stages from 8 to 15; he discovered how to have upgradeable custom airports and seaports without CTDs. As for me, I have just been testing in accordance with bap's instructions and also testing other combinations of custom content. I am not a number cruncher but can do tests as requested so that the number crunchers can look at them.
It is time that certain people stopped blaming BSC for the woes of the world and co-operated with the community. We don't do this for money or for any other reason than trying to make sure a game which should have been consigned to the rubbish bin long ago is kept fresh, alive and stable to use. Problems reported should be dealt with - as with the Mayor Mode ploppables that are based on cycledogg's template but not corrected to only plop in Mayor Mode.

To reply to tahill69, you suggest "Demolish the building and put something else there."
That does not make any difference as if a savegame is corrutped there is nothing you can do about it execpt use a back up. Please go back and read again through this thread. The reason this community is trying to deal with it is because bap did so much meticulous testing and after being rudely told it was rubbish by Pegasus posted his findings here.