Modifying .dat files

[Mandarin(a)]

Hello everyone!

I have a question that have bothered me for quite some time now: if I want to remove a model from a mega pack, is this possible? Or is it at least possible to extract everything, sort things out and then pack them again?
I know that with ILives' Reader I can delete exemplars and so prevent props to appear in L.E., but the unused models still require time to load (for game itself also).
I would also like to make a similar things with texture packs, but I found doing that much easier; in Reader I just sort textures by "Group" and all five .fsh files of each texture are put together, so deleting them is easy. With models ... to manually delete all .fsh and .s3d files, I don't know ...

Is there a tool or a method to do this? Tutorial anywhere? I have thoroughly searched Simtropolis and SC4D for tutorials, tips, tools, anything, but I had no luck. Everything I am currently aware of is implying that my wishes are not realistic ...


Thank you for helping!

[vester]

That is possible, but will be a huge detective job, to find all trace of it.
Haven't heard of any tools to do it.


You might try to see if there is pattern to the TGI's in the sc4model files. It would make it easier to sort out the which files to take out.

[rivit]

@Mandarin - the secret to doing this is to have ALL of the dependencies of the item you want - then remove them from wherever they're lurking.

You do run the risk that something is used in more than one spot or more than one .dat and that removing it will break something else. Its hard to reverse search for all uses of a texture for instance.

At the risk of advertising, SC4DataNode (on Simtropolis) will give you a list of all of the dependencies of a lot, building or prop down to the S3D Model instances, and they contain the FSH references. (In fact they're usually found together) . If you select the detail display CTRL-c will put this list into the Clipboard from where you can paste it into Notepad or Excel... (see the Manual)

You will still need to remove items from the .dats manually.

Make sure you backup before removing things - its easier to recover from mistakes this way.

[Mandarin(a)]

Ah yes, that should probably work. I will take a look at your tool, thank you very much.
I still have to consider if it is not too much of a Sisyphean task ... But this is probably a method I was looking for. Thank you again.
Cheers!

[Andreas]

Depending on how the various models have been packed into a single .dat, you can often find the FSH, S3D and possibly other files neatly grouped together when you open a .dat in the Reader. Each model consists of a set of 20 S3D files (4 directions, 5 zoom levels), with a corresponding number of FSH files (can be more than one per S3D, and there are also nightlighting textures). There might be Exemplar files, LTEXT files and more associated to the models, keep an eye on the IID references in the files. You can safely remove any BMP, JPG and similar files if present, they're only used for preview purposes.

As said before, you have to extract individual file sets manually, I usually do this by opening two Reader instances, open a new (empty) .dat in the second one, select and copy the files in the source .dat, and insert them into the target .dat, then save the file. Personally, I don't bother to separate small props and such, but only larger building models that have been merged with other stuff. You can reduce the size of many "Mega Prop Packs" considerably with this method, though.

[mgb204]

It's not as easy as Unzipping a file, for sure. But if you really value your plugins setup being kept as small as possible, doing this can be a worthwhile investment.

When files are packed together, usually they are grouped in sets. For example, a typical model has 20 S3D files and at least 20 FSH textures. If these are grouped together, it's easy to work with them once you've figured out the ID schemes that bind them together. Let's use a popular file as an example here.

BSC MEGA Props - SG Vol 01.dat

If you open this file with Reader, the first block of files are all cohort files. In essence they are prop families, but we don't need to do anything here.

The next block is a bunch of prop exemplars. Select the first one and on the right pane, see the property Resource Key Type 1. Note for most SC4 Models, the Type and Instance numbers are unimportant. It's the Group ID links together the parts of a model. The first prop exemplar "prop_SG_BarnHouse01Blue" has the TGI: 0x5AD0E817,0xAE9038D5,0x00030000.

In total there are eleven props, the final prop links to the model ID 0x5AD0E817,0x8E917B3D,0x00030000.

Now if you look at the first groupings of S3D and FSH files, you can see that has the Group ID 0xAE9038D5. So if you wanted to remove this one model, select the entire group of files using this GroupID. The next 10 models by group ID, ending in 0x8E917B3D, follow the same IDs from the prop exemplars above them. After that, the file seems to have the S3D and FSH files with the prop exemplar placed after them. But this is a good point, don't assume the organisation structure is consistent. If you cross reference all the parts correctly, after a few it will be easy to locate groups of items.

Not all files are organised in the same way. But the basic process is the same. Find all the groupings you don't want to keep. Start by finding the prop or buildings exemplars for the models to be removed. Make a note of the Group IDs of all the models. Then go through the S3D and FSH groups to select those objects you want to remove. Lastly remove the prop exemplars. Grouping the list of files by Group ID can be very useful here.

One last tip, use the navigator function of the reader to help making finding objects easier. Click on Navig. and a new window appears, "Object" contains props, so open that. Here we can see a list of props by name. Let's take the second object as an example here, "BSC_Prop_BananaCrateLg". Left click to select the item, the properties are now shown on the right. Click the little + next to the prop name, revealing it's linked to a model, with four sides. Right click on the Prop Name and select Synchronise, on the main reader screen this prop will now be highlighted/selected. Do the same where it says Side 1, now it's moved the main display to select the model. Using this feature makes it much easier to locate items, especially if a file is not well organised. The other benefit is that it will show you other linked objects, such as Icons, LTexts etc.

[twalsh102]

You have set yourself a daunting task.  Everything you want to do is definitely doable, but as others have said here, it will all have to be done manually.  Rivit gave you the absolute must-do advice of "have  backups" if you attempt this.  Deleting the wrong thing can definitely mess up your game.

How daunting of a task is this?  Let's just start with some stats from my Plugins folder garnered from Rivit's SC4DataNode (and this only includes prop / texture packs that I've identified as being needed for the lots/buildings I currently have installed):

# of Props:  23576 (and the Prop list shown in DataNode apparently only shows the last override of any specific prop, so this number doesn't include any duplicates)

# of Lot Textures:  52631 (this number does include some duplicates, but not more than 100 or so).

Let's dig a little deeper (I'll use HKABT-bixel Vol01.dat as an example):  This .dat file consists of 5 building prop exemplars and their associated model files.  Between exemplars, bmp, jfif, xml, fsh, and s3d subfiles, this one .dat file consists of 759 entries -- for 5 props.  So you've got some work ahead of you.

Depending on the size of your Plugins folder, and what you desire for its end state, this could be a several month- (or several year-) long project

DataNode and Reader can do a couple of things to help "clean up" your Plugins folder:

1.  Identify prop packages, or parts of prop packages (if the prop pack includes multiple files) that are redundant and can be safely deleted (making sure they are truly redundant).

2. I've found several prop packs (sorry I can't recall which specific ones) that had multiple copies of the same prop exemplar and associated models within the .dat files (the worst had 4 copies of everything).   Once you are sure of what you're doing, you can safely delete any copies you run up against.

3.  I've run into situations where a needed prop is part of a lot download (as opposed to part of someone's prop pack).  The single dat files included exemplars for the lot, the building, and all props, as well as all associated model files. If I know that I'm not likely to use any of the lot associated items, I'll go in and delete everything but the prop exemplars and associated models.

Having said all of the above, there are always risks doing something like this. 

First, this is not really beginner stuff.  Unfortunately the only way to learn is by doing (with nothing other than included manuals available as any reference), and by asking questions that hopefully someone will answer for you.  Expect mistakes.  Luckily, as long as you have backups, you're not likely to permanently break anything. 

Second, versions of the Reader earlier than the most current version (1.54) are famous for corrupting subfiles other than one you may make changes to, when saving the .dat or .sc4lot file.   The corruption often happens to LTEXT and PNG files.

Third, trying to cut down the size of a prop pack by deleting things you don't think you need is always problematic, unless you intend to never download any more custom content (yet another reason to keep backups).

A final thought since one of your stated objectives was to cut down on the number of props available in LE:  Assuming you have the storage space available, consider having 3 different copies of any specific prop/texture pack:  the original (archived someplace), an LE copy that has been modified to only include those props you are likely to use, and a game-play copy (possibly modified, but not to the extent of the LE copy) to accommodate any future downloads of custom content.

OK, I'll stop rambling now!

[Mandarin(a)]

Hey, thanks for all those useful tips. I started playing a little bit and succesfully deleted few things I don't like. I am aware of horrendous dimensions of that task and I really don't plan to do it all at once. But gradually I may be able to delete some more models for which I am 100 % sure I won't use them.
Of course if it appears that they have been used somewhere else, I can always open LE and modify the new lot ... so this is not a problem.
Regarding backups, I am already practising the method with three copies of everything for sime time now. And I have also experienced the LTEXT corruption even with the most recent version of Reader. Luckily with all those backups I have been able to dismiss all failures.
As for now, everything seems allright, the game still works (almost) like a Swedish bearing 
Thank you for your help and patience.