Using SC4Tool to identify problems with missing dependencies

[doug_wesson]

One of the first lots I was looking at was Deadwoods Ministry of Garbage and it identifies that

BSC MEGA PROPS - SG Vol01.dat is missing

That this has been replaced by

BSC MEGA PROPS - SG Vol01_v2.dat

How does this affect the use of this lot and is there a way of rectifying it so it recognizes the new dat file

Many thanks in advance

[mgb204]

I have to say (sorry @Tyberius06, I know you are working very hard on this), but renaming dependency files like this, even if just to denote a version number, is really not ideal. Because once you do that, things like Cleanitol files and file listings like this, will no longer recognise them as the correct file. It sucks, but some things once they have existed, become necessary to keep that way or you run into all sorts of problems and confusions. When we're talking about some of the most commonly used files in the community, continuity is a must IMO. The problem of balancing user friendliness with retaining backwards compatibility is always a tricky line to walk though.

One the other side, by giving each file a clear version number, I can envisage a number of scenarios where that might be helpful. For example, in the event of a user having problems, anyone helping could ask you which version you have?, making it easy to see if you've the update or not. Also, having a distinct version number for the files, provides clarity to end users, should they find they have both files.

In an ideal world a user would rid themselves of any old versions when updating, so they never accidentally used it, the old one is after all obsolete. Likewise if troubleshooting, the first thing would be to tell the user to ensure they are using the newer version. This is quite simply established by the date in the file MetaData which Windows easily reveals. In the event of confusion, just advise to download the new one and use it instead to be 100% certain. The problem here is that users don't always want to handle things so rigidly, so as creators we are always asking ourselves, how can we avoid this or that problem.

Don't get me wrong, I know there is a need to differentiate one file from another, but in cases like this I think it would be better to do that with the download itself (.zip file), not the filename of its contents. If nothing else because the sheer amount of existing content that now fails to work due to changing it, given it's easy to mitigate against the potential problems.

[Tyberius06]

@doug_wesson :

Deadwoods Ministry of Garbage and it identifies that

BSC MEGA PROPS - SG Vol01.dat is missing

It is not even listing any versions of the BSC Mega Props SG vol01, not on the LEX download page and not on the STEX download page, and not in the readme. The proper dependency list was listed in the dependency tracker which is currently offline.
If you are refering to the given list based on the LD property ("Required Files (LTD Files)" - as it shows up in the SC4 Tool), you should generally be careful with that as that information can be fals. In this case it is not false and it seems that back in the time they failed to list the SG prop pack as dependency OR the older version of the BSC Essentials (released on STEX at that time) could have contained those props which later got into the SG Mega Props. For 16 years there was an outdated SG Mega Props version available on the STEX under the same name as the then-updated version of the same prop pack re-released on LEX. It was me last year who got rid of the older versions and on both sites now we have the latest v2 version.
With all the updates what I've been doing, they are backwards compatible. If an older upload is listing a certain prop pack, but you have a version with a higher version number, the newer version is also working fine with that upload. You should simply remove the one without the version number (or lower version number) and replace it with the newer or highest version number. The biggest change is, that now v2 (and yes, there will be even a v3 as well, which I'm already working on) contains mostly all the commonly used props from simgoober, which were originally dumpped into the BSC Mega Props MISC vol02. It might seem confusing, but the older contents with the newever prop pack versions are completley backwards compatible.

But for sure, the LTD-file list (or LD property if you check the file in ilieve Reader) is something which is generated by the Maxis Lot Editor and often contains fals information and for exemple with newer lot files or files created with SC4PIM-X and never touched by Maxis LE, you won't even have this list as PIM-X never generates this LD property. Sometimes it is useful with these older contents because it can lead to actual dependencies, but many/most of the times it has fals information.

@mgb204 : hm... most of the cleanitol files are only for removing stuffs and a very few is using the *.txt-s as collecting the dependency links.

Because once you do that, things like Cleanitol files and file listings like this, will no longer recognise them as the correct file.

If you are refering to private listings or custom cleanitols made by the players for themselves, well, that's bad luck. IF they were able to work with those by themselves, then probably as unconfortable as it might be it's up to them to update it again for themselves.
I'm doing my best, but we are still talking about 1000+ uploads which needs to be checked and occasionally updated/fixed and since we temporary lost the dependency tracker I need to update the readme files for the proper dependency links too. That was NOT the plan, as the DT should have taken care of that...
Thanks for mentioning the cleanitol files, I'll take extra care to check what they are refering to, but so far if I recall correctly only the two real mf BSC Airport and BSC Seaport uploads are using the cleanitol to collect dependencies and not just removing obsolete files. And I put dependency links into the updated cleanitol of the BSC Mega Props MISC vol02 (as it has dependencies) and some of the CSX MEGA Props.
Now I'm going through one by one on most of the BSC items on LEX and check them against obvious issues etc... I've already found a handfull of stuffs which were inhereted from original releases back in 2004-2006 still on the STEX, but were failed to be updated and fixed later on when they were re-released on LEX. And I'm updating the readme files too to reference the latest versions.

I can tell, that many of the most popular dependency packs will be updated to a new version once again (v2-s to v3 and some of the v3-s to v4 and if everything goes well, then there will be a final version of the BSC Essentials as v202x it depends on which year I get there), when I finally finished with this whole project.

At the end, a database like Nos17's prop and texture catalogue after getting an update with the new and finalized prop packs, could list most of the props even those ones which are currently hiding in sc4lot files.

[doug_wesson]

I have solved the problem I just used the LotEditor and saved it and it now points to the right .dat file

[Andreas]

As Tyberius06 said, the LD-/LTD-file that the Lot Editor creates isn't always very helpful and could be misleading. It is important to understand that this file is only a help for the user in order to track down dependencies, but it's neither complete (i. e. texture files are not listed there at all) nor does it reflect renamed or merged files. The game doesn't need this file, it solely relies on the actual IDs of the properties that are read during the game starting up.

What SC4Tool's Dependency Scanner does is a mixed approach in order to track down dependency files. It shows you the content of the LD file, so you might recognize necessary dependencies based on the filename, but it also shows a list with all IDs, including colored markers (green = ID found, red = ID not found; there's also a blue marker, but right now, I can't remember exactly what it stands for). So if you want to check a lot's dependencies, just make sure there are no red markers, then you should be safe.

It is not necessary to re-save a lot in order to update the LD file; you can do this to update the file list in the LD file, but for the game, it doesn't make any difference how the files are named. If you're a custom content creator, it might be a good idea to have an LD file that lists all the files with the proper filenames of a prop pack rather than a huge list of SC4Desc files before merging them, but when I published the SimForum BAT team stuff, I simply removed the LD file entirely in order to avoid confusion.

[mgb204]

If you are refering to private listings or custom cleanitols made by the players for themselves, well, that's bad luck. IF they were able to work with those by themselves, then probably as unconfortable as it might be it's up to them to update it again for themselves...

I'm doing my best, but we are still talking about 1000+ uploads which needs to be checked and occasionally updated/fixed and since we temporary lost the dependency tracker I need to update the readme files for the proper dependency links too. That was NOT the plan, as the DT should have taken care of that...

As I said, the intention is most definitely not to criticise the sterling effort you've put into this process, truly it's heroic how much you've done. But changing the file name is a deliberate choice as part of that process, it's not a mandatory one. I'm just stating that personally I think that's not the way to go and that it is sufficient to add this only to the filename of the download. By simply not changing the names, you'd actually save any potential work re-linking or fixing things broken by the change.

Cleanitol's aren't the most prolifically used things, but they are bloody useful and if the file names don't match, then for what benefit are we breaking things?

[doug_wesson]

Sorry to have caused any trouble. all I really wanted to know was will the affected lots still work ok

[Tyberius06]

Cleanitol's aren't the most prolifically used things, but they are bloody useful and if the file names don't match, then for what benefit are we breaking things?

The problem is the user usually. I was thinking about not changing the names by adding the new version number (I did this with the BSC MEGA Props JES Vol08, which doesn't contain any new stuffs, but I had to remove unused ploppable building exemplars), but I did not find it fool proof enough. I wanted originally the new updated versions to override the old ones, but the player can opt out the override when the new version would collide with the old one. And then the questions would appear just from the other way... "can I just override the old version with the new one". "I have a version but I don't know if I have the latest one..." The version numbers in the file name give away immediately which version they are having. So when a readme (the updated readmes are written following this idea) lists a v3 and specifically states that the player needs the v3 for the given content in order to make it functioning properly and they only have a v2 or one without version number, they will know that they need to download that v3 version.

The cleanitols of the new versions are always listing the old versions to be removed. That means that a Whatever Mega Props vol01 v3 is listing the whatever mega props vol01.dat and the _v2.dat too and whatever versions were before the most recent one.
This is quite critical exactly with the BSC Mega Props SG vol01, because of the older conflicting versions (STEX version vs. LEX version) had the exact same name and file path from the exe: BSC MEGA Props - SG Vol01.dat. Player downloaded the LEX version, then a STEX upload linked the STEX version, and they downloaded and installed that too, because yeah why not... Then they found a version from the simcity kurier, which might be the STEX version or the LEX version or predates both of them, I have no idea, but it is running under the same name.
Now the BSC MEGA Props - SG Vol01_v2.dat (and later BSC MEGA Props - SG Vol01_v3.dat) will definitely load after these versions, and the cleanitol also lists them to be removed.

We are not breaking things, it seems we are breaking, because I'm at the beginning of this update process on the contents, but I can not do anything with LD properties listing non-existing anymore prop packs besides removing the LD properties (which I'm doing quite thoroughly by the way after I determined all the proper dependencies for given content). Older the contents the higher the chance that someone will run into prop packs which pre-dates the mega prop packs. Many of the BSC readmes that I've been updating links back to non-existing STEX links (after the BSC migrated from STEX) and quite few of them actually links the PLEX, which is even funnier. These files were locked into the installers and nobody bothered to update them and reassemble the installer.
That's also true for some pretty nasty f. ups - has anybody noticed that technically the BSC BLS Housing 6x4 GC, 6x6, 8x8 and 10x10 Euro lots (most if not all of them) could/CAN - can at this moment the currently available versions - potentially cause Immortal Lot Syndrome because of the inhereted poor modding between building and lot config exemplars since 2004? And they are actually can cause ILS because I tested the theory. When the ILS came to light with ONLY ONE CAM Bixel lot, the whole pack got locked down for a decade, yet we have dozens of lots through several lot packs from 2004/2007 (if I count them from the so called updated LEX release) which do the very same, but might be harder to trigger.
Honestly I did not expect that I need to deal with these stuff too... So the process is even slower than I expected, because now I can not trust in a single upload and I need to double check all of them one by one and fix them if they needed to be fixed...

Sorry to have caused any trouble. all I really wanted to know was will the affected lots still work ok

@doug_wesson

You did not cause any trouble. In short just to make it sure. The newer BSC prop packs (newer versions) which come with this Project ZIP effort always compatible backwards. They are replacing the previous versions; sometimes they are fixing underlaying issues, sometiems they add new items, but on the LOT sides they remain dependencies as the older version was.

In the following months there's gonna be many updates with LEX contents (mainly with the still installer based BSC contents), where the readmes will be updated to reference the proper dependencies (tested and checked one by one).