What causes Prop Pox (and how to avoid it)

[Ennedi]

(...)
Obviously the game saves space by merging the prop records, and omitting at least the Type ID and Group ID for those props that have them in common. In this case only the IID is recorded for the custom, timed prop:
(...)

OK, it was an intention of the test suggested by me, and this is an answer for my question (is it something bad in having two props with the same Instance ID but different Group ID? Looking at your result the answer is yes. The IID should be unique in any case.)

[bap]

It is good to see you, bap. I hope you realise that here we are trying to help and find a solution.

Sure, I never doubt it.

The thread at [url=http://sc4devotion.com/forums/index.php?topic=7066.0]simpeg[/url] is now locked but I am posting here a copy of his final post:

Two things annoys me concerning the behaviour of such a key batter/modder when it comes to the Prop Pox:

1) his insistence in adhering to wild speculative theories that bear no support in the facts. Facts are not prone to sympathy/antipathy or to acceptance/dismissal. One may freely propose and throw away explanations if they don't fit the facts. But one cannot throw away the facts when they don't fit ones convenience of beliefs.

2) his obstination (and clearly his only concern) in convincing the community that his creations has absolutely nothing to do with Prop Pox.

The facts are: (1) a group of props in his oww2-bdk-resource file cause Prop Pox, and (2) there must be other affected props around in the community because there were players affected by Prop Pox before the oww2-bdk-resource file was released. This is not the Santa Inquisicion. Nobody is tring to burn witches. Nobody is pointing fingers accusing any one to be "responsible" for the Prop Pox. It is absolutely irrelevant who modded Maxis props in a way they cause Prop Pox. This is the past. The only thing that matters is to find all occurrences of infected props (wherever they are, whoever modded them) and correct them at least to avoid new city developers from being infected by Prop Pox.

There are a few distracting and erroneous affirmations and insinuations made at simpeg that I have to refute for the records.

1) "Prop Pox is a random effect".

False. I developed test cities 31 times. In 15 of these tests, I ended up having Prop Pox. It appeared always at the same 2977aa47 file size, just beyond 6 Mby. These tests were done at different computers, with different screen resolutions, some with & some without the EP1 and BAT updates, and with different combinations of custom prop packs & lots. The 4 infected props were in the plugins folder in all 15 tests. For the other 16 tests I got no Prop Pox. In none of these 16 times the 4 infected Maxis modded props were in the plugins folder. Therefore, there is no random behaviour. And there is a statistically significant correlation between the presence of those 4 specific props and the appearence of Prop Pox.

2) "Prop Pox may well be caused by NAM".

False. None of the modds, bats and lots in my plugins folder causes Prop Pox. I performed two tests will all my custom content bats & lots in the MyDocuments/plugins folder without the 4 affected props in the ProgramFiles/plugins folder. I got no Prop Pox in both cases. Thus, I can say that NAM, SAM & RHW mods cause no Prop Pox. To the benefit of Pegasus, I can also say that his RiverTool kit, Pond kit, Stream kit, CDR-recreational kit, CDK-OWW kit, CDK3 kit and CSK2 kit cause no Prop Pox.

3) "Using CSK, CDK, OWW2 and related stuff helps prevent Prop Pox".

False. Having sizeable parts of a city covered by water helps to prevent Prop Pox because in this case there is not enough land space to develop the city up to the point where it will show Prop Pox. Zoning mostly high density residential areas also helps as there will be, on average, less props per lot in the city (see one of my previous posts here). It certainly has nothing to do with the custom lots one install or not in the city, although most custom lots have, on average, more props than Maxis ones and will contribute to reach the undesired 6 Mby 2977aa47 subfile limit at an earlier city development stage. Prop Pox is a matter of modded props and city 2977aa47 subfile size.

Bap

"Everyone is a fine person when every thing goes fine. The real character of a person only appears when he/she is under pressure."

[bap]

@bap, you made your experiments very correctly from the scientific point of view   and you made an excellent preparation to the next research work. But we must remember that we are in the beginning stage of researching now. We are far from final conclusions.As for your theory about fixed-length arrays in the memory is very probable but not proven yet.

But I'm sure we go in the good direction. We not only want to find an ultimate answer ie. get to know the mechanism which acts here, but we also want to know what to do in the meantime. Looking at everything what was written here I think we should use the method of thinking similar to Pascal's proof of the existence of God  : "We can't prove if He exists or not, but it will be better for us to act as He would exist". And we just do it in this case. Assigning an unique Instance ID number to every game object modified by us will not cause any harm in any case. So it will be safe to take it as a principle in future and to improve existing game objects according to this principle.

.....

Bap, one of good methods of testing the theory is trying to falsify it. So don't be surprised if I will try to dio it, it is simply a testing method as I said  . I also suggest others to take another possibilities into account, it can only increase our knowledge, maybe we will get to know another important things too.

Ennedy, I feel confortable with the scientific method as I am doing this for living over the past quarter of century.
I would not call that a theory but simply a working hypothesis. One may put an alternative hypothesis on the table by saying that it is not the increase in prop size that matters, but the transformation of a static prop into a timed one. If this second hypothesis is true, then the umbrella prop (which is originally already a timed prop) alone would not cause Prop Pox. I am able to test this.

Please be assured I will not take it personally if you (or anyone else) proves my working hypothesis is wrong. I will certainly not die defending an explanation if it doesn't fit the facts.    I will happily accept any other explanation if it gives a better match to the facts and help us to find a cure to this pox. It really doesn't matter who proposed what, but only if our game of proposing-testing hypotheses lead us to answer the questions we are up to.

Edit: At this stage we can't eliminate the possibility that other Prop Pox sources can be collected in other subfiles too. I will try to find another subfiles where particular data are collected.

This is an interesting approach. Good luck!

Time for me to stop interacting (pleasant thing) and do some more testing.

Bap

"Everyone is a fine person when every thing goes fine. The real character of a person only appears when he/she is under pressure."

[BarbyW]

Just out of interest I checked in simcity_1.dat for 0x6534284a, 0x47bddf12, 0x05040000 and found it to be the building exemplar for a growable building: CO$$56x36_8ChiOfficeBldg5_0504
Why this should show a crowd of people the corner or a poxed city I have no idea. unless they are waiting for the office to open to complain.

[daeley]

Thanks for clearing that up bap with some straight facts bap. Although his credibility did go down the drain when he posted a screenshot of raw hex and said, quote, that's how XML used to be read in the DOS days I had to pick myself up from the floor after that one...

alright, back to the topic at hand...

[Diggis]

Nope. One of the first experiments was developing a city from scratch only with Maxis lots, but will all custom content props (including the infected ones) in the plugins folder. It got Prop Pox when 2977aa47 subfile reached 6 Mby. I went back to a version of this same city prior to showing Prop Pox signs, removed every custom content (props files) from the plugins (including the affected ones) and further developed the city. It got Prop Pox again at about the same stage.
One may conclude the problem is not changing the definition of a prop that is used in a city after the city started (this would only affect the new occurrences of that prop), but changing the properties/exemplar size of a given prop while reading files at the start of the game (this occurs every time you start the game with Maxis modded props).

Bap

Rats, I was afraid you would say that...    Thanks for all your hard work on this, you really have done an amazing job on this.

Andreas, I am sorry to see one of the last members at Pegs with any real experience and knowledge leave there.  It concerns me that new players will go there looking for support and find none.

[spa]

A lot of drama over what should be a simple situation.

"Hey I found what causes this rare bug."
"Great lets fix our files so that this doesn't happen again. Thanks so much for your hard work."

Isn't that how things are suppose to go? It's what Barby did with the BSC file that could cause problems. It's a real pity that others refuse to participate in the SC4 community. Oh well, at least we know which files to avoid.

[tahill79]

aint it against SC4D rules to post personal matters from another site here?  im pretty sure it is.  just pointing that out.  i remember the fall out from the seaport controllers it was stated there that it would not be tolerated by any member.  i do believe that your post and mine should be deleted from this topic as they violate Devotion rules.  good luck andreas and i will stick to my word in my post at peg.  i respect you as a member of this community.

[Andreas]

tahill79, I really hate to do this, but it has to be done. bap was posting here about his findings about the prop pox, and were were trying to set up a communication with PEG this way, since most of the BSC members are banned from there. I did take the opportunity to post at PEG's, but one of my postings was removed there, and now I'm completely banned after officially resigning from being a staff member. I was told that my posting at PEG's was reduced to the last paragraph, therefore I posted the unabbreviated version here, since it deals with a problem that I can't discuss any longer at PEG's. Maybe I should have posted in a thread about Mayor Mode ploppables, but this place seemed appropriate, as it was dealing with PEG's modding as well. Since I cannot continue my work at PEG's, I will try to contribute my knowledge here instead.

EDIT: I just realized that you can barely read the screenshots, as the size was reduced by PEG's forum automatically, so here are the original ones:

     

[Ennedi]

I see I have two reasons to write something here  

Ennedy, I feel confortable with the scientific method as I am doing this for living over the past quarter of century.
I would not call that a theory but simply a working hypothesis. One may put an alternative hypothesis on the table by saying that it is not the increase in prop size that matters, but the transformation of a static prop into a timed one. If this second hypothesis is true, then the umbrella prop (which is originally already a timed prop) alone would not cause Prop Pox. I am able to test this.

Please be assured I will not take it personally if you (or anyone else) proves my working hypothesis is wrong. I will certainly not die defending an explanation if it doesn't fit the facts.    I will happily accept any other explanation if it gives a better match to the facts and help us to find a cure to this pox. It really doesn't matter who proposed what, but only if our game of proposing-testing hypotheses lead us to answer the questions we are up to.
(...)
Bap

"Everyone is a fine person when every thing goes fine. The real character of a person only appears when he/she is under pressure."

Bap, it's a real pleasure to share an experience and knowledge with people like you. We don't know one to another, but as we can see, there is a common language that can be used by us in our contacts. We would only wish ourselves to see scientific discussions in the real life made in similarly objective and respectful manner  . I am sure our science would be much better and useful than it is now. I don't know how are your own experiences connected with your work, but personally I am happy that scientific research is not my job and I can use my mind for my personal pleasure and for pleasure of some of my friends  
One of the best things caused by this game and this site is an opportunity to contact with other people in a very different way than most often happens in RL - in an objective, repectful and often enjoyable and even funny atmosphere  
I hope we will continue our experiments and maybe we will learn something new and interesting together  

@Andreas, I see your actions at many places costantly from the day of my access to SC4D and I must say you have my full confidence in everything you do and everything you say - and I can say this about very few people  . One must be incredibly stupid or maybe fully isolated from the reality   to don't see and appreciate an objective and quiet manner of your argumentation.
You did it very good showing this example with tree controllers. It was fully reasonable in this case.

Even bad situations can be used to create positive things, if only people think positive  . Reading your post in Simpeg I experienced a (little  ) illumination - I have an idea how to make creating tree controllers easier... (the graphical representation will be helpful, I'm going to make a table identical as the Terrain Texture Map Table for every kind of flora instead of the single row with 256 repetitions. Then some simple formulas in Excel and we will be able to automatize most of the process. In addition it is possible to harmonize flora with terrain textures this way).

Adam

[T Wrecks]

Jeez, you should think that this was a common thing to happen in a modding community that has been forced to rely on trial and error rather heavily due to lack of official support - someone finds a method for making something new that seems to work, others adopt it, but then problems occur. Finally these problems are traced to that method, and quite often these findings lead to a method that really works.

Happened with the RCI blank lots before, and today we know better. So what? I guess nobody blamed RalphaelNinja (or the people who used his method) because all knew how much he had contributed and stuff like that was just bound to happen - after all, you can't have trial without error, it's that simple. And the current situation is exactly analogous to the one with the RCI blank lots - well, except for the reaction, that is. 

What a sad moment.

[callagrafx]

Not sure just how to take that post, but nobody was actually pointing fingers and shouting "it's all your fault!"...Bap identified a prop pack that was a cause of the pox...note "a" cause, not "the" cause.  Did the creator go and fix it and say "thanks, I'll quickly update it to prevent a lot of people losing months or years of work"?  No, he basically said it was a hoax and we're all scaremongering.  I think the reaction to that, given the nature of this community, was and still is only natural.  Sad day yes, but at least we now know how to avoid losing carefully crafted cities....simply don't use any content identified as being a cause of this problem.

[catty]

I come from a background in database management, so apologies if I've gone off on a tangent here re Prop Pox or repeating whats already been worked out and has already been stated in a previous PM.

If Maxis wanted to save file space it would make sense rather than for example having 100 beach chair props on 100 different LOTs, have one beach chair prop based on the TGI with the 100 LOTs all referring back to that one prop.

The problem would come if you have 99 beach chairs and the same TGI with one set of identical contents and 1 beach chair with the same TGI, but with slightly different contents the system would still try to merge to one prop with a unique TGI reference but the prop data would be slightly different so the prop data would start getting screwed up and you would end up with missing props on the LOTs as the referencing system would no longer work.

EDIT: its called a One to Many Relationship or maybe in this case One Prop to Many LOTs   

[simsamerica]

ty, now I know 

[wouanagaine]

  need to rewrite the long post, I hit close

So to get back on Prop pox topic, because at least here we're looking for explanation and deep analysis ( and by "we" I don't mean BSC, but the whole SC4 community that have some membership at SC4D )

I have read quickly over the last pages, and I come to a theory. feel free to correct me if I'm wrong, I may have read last bit of information to quickly

On the old Prop pox thread, and Ripplejet post some of my screenshots and some thougths on it, we discovered that the timed prop are not save in the same subfile as normal prop, and this is a fact, everyone can do what I did if they don't believe me, no denial can be done on this

Now on my theory, and following informations from here which is at least where we can find informations
When a prop is saved, it is saved either in the 'normal prop' subfile or either in the 'timed prop' subfile ( there might be other subfile with different kind of props ).
We also can easily guess that a timed prop need much more information than a normal prop to be saved and I clearly think this is why there are 2 differents subfiles
Now programming wise, I expect the prop object to handle the code to save, I also expect the code to be optimized for reading/saving. What does it means ?
We already have a proof of optimisation, maxis clearly handle different code path for saving when the number of prop hit a certain number ( the subfile change at 6mb )
For optimization , It means that to prevent even longier time to load, the code expect data to be in certain format, ie the code for reading the normal props subfile expect the data to be valid normal prop data, the code for reading the timed props subfile expect the data to be valid timed prop data. This will allow the code to be as efficient as possible.
Now I also expect maxis programmers to code efficiently and as game coder are used to write code, ie using lookup table to speed up things a lot. It means they certainly have write code to categorize kind of prop ( timed or normal ) to speed the code. It will in effect speed up the rendering, a timed prop need to be redrawn when the condition of the timed prop are meet, whereas a normal prop is need to be redrawn only when a screen refresh occurs. it will be very efficient to only iterate along the timed props instead of all prop in that case.
As for savegame ( and back to prop pox ) we indentified that timed and normal prop are saved in different subfile. The same lookup table can be used to filter which saveing code path are used for each prop ( save normal or save timed ).
Now what happen if a prop is not correctly categorize ? bad thing
Now how can a prop be not correctly categorize ? if a prop is categorized at SC4 initialisation time when it parse maxis file, it is categorized at a normal prop, but when reading your plugin folder it is then categorized as a timed prop because an old modding practice has been done, how the code will handle that ? I don't know, maybe correctly
Now back on the prop pox.
The normal prop code reading of normal prop subfile expect data in a certain way ( and it might be different format for different number of prop, where with low number of prop, some values can be store with 2 bytes, but with higher number of props, it need to be store with 4 bytes - this is only an example, but a very common practice in gamedevelopement ). Now what happen if for some reason a prop data in the normal prop subfile is not in the correct expected format ?
-CTD ? maybe sometimes
-discard of all subsequents props in the subfile ? yes, it leads to prop pox
A CTD will only occurs if the data don't make anysense to the game. What happen if a timed prop data are stored where a normal prop data is expected ? maybe the data are valid enough - just because I expect all data of a normal prop to be in a timed prop  but not the contrary -  but then the data after the length are still there ( the specific data of a timed prop ), which the code can't handle, and it won't look like a new prop at all. So maybe in that case maxis decide to discard all the following bytes in the subfile and stop reading it instead of just crashing, leaving the player with prop pox city



Disclamer:
Of course this is a theory,  as I and noone have the SC4 source code. This theory can't be prove yet, but at least it can be tested, and the result may confirm it. I just know that even in that case some members will still find the theory wrong and won't do anything to find what the prop pox problem is as they don't experience it anyway. For the record I never experienced a prop pox, lot of people in this thread activily researching the prop pox root cause have never expereinced it. It does't refrain them to look for the problem and search a solution

[callagrafx]

If Maxis wanted to save file space it would make sense rather than for example having 100 beach chair props on 100 different LOTs, have one beach chair prop based on the TGI with the 100 LOTs all referring back to that one prop.

Correct, that's the same as what we do when we create prop packs (well, when I say "we", I mean Barby, Hex to me is a clockwork computer with an "Antill Inside" sticker on the side  )....the content is re-used over and over, irrespective of how long ago it was created.

[wes.janson]

"Antill Inside"

This will be a wee bit off topic, but great Prachett reference!

[dedgren]

...I know, I know.  I used that pic in 3RR a couple of years ago.  I just read the level of tech detail in this thread, and my head starts to hurt.

One thing I'm sure of is that, if a definitive answer can be found, the folks posting here will be the ones to find it.


David

[sithlrd98]

While I've never had PP hit any of my cities , and since there has been so much info lately , I am wondering if this is limited only to Maxis props? If I understand this correctly , what has changed is static props were turned into timed props without changing the TGI values. Can this also be caused by T-21 mods where a prop (Maxis or otherwise) did not have the TGI (mainly the instance) changed , ie , a seasonal tree or traffic prop?  I'm sorry if my question is confusing since , as far as I understand, by not creating a new instance ID, the new mod can  "contaminate" the original prop info (and possibly create even more instances than the original and the newer instance of the prop ie , 2 becomes 4)? I am sorry if this is a dumb question , as I only understand a fraction of a fraction of what you "old timers"   have been able to uncover by looking under the hood of this game!

Jayson

[wouanagaine]

Maybe, I'm pretty sure that T21 props are stored elsewhere ( as seen in my pic Ripplejet reposted, we can see T21 props ( I think they are T21 props I'm not expert in those ) ). Maybe they can instead of produce prop pox on lot, produce prop pox on network which might be less visible