SC4 Dat Packer

[dragonshardz]

geez, i lost 4+ hours today getting this thing to work. I found that it is not a good idea to put ALL lots/BATS into one folder as stated by the readme. not only does doing that take up a lot of RAM, but it create errors as weel and sometimes the app crashes. It is best to put your lot/Bats/props/textures/models into "chunk" folders and pack that way. there are far fewer errors and less RAM is used.

[wouanagaine]

Yes that is what is written in the readme, split your folder

[dragonshardz]

yeah, but the tips @ the bottom say to remove modds like nam, cam, etc and split folder into two folder: lots/bats & props/textures/models. i had to split my folder into 15 different folders (not counting mods) to work, because about 1/2 way through the packer would start getting errors like crazy and then crash.

it dorsn't matter reall, just noticed that it is better to have many smaller dats (<20) than to have 1 or 2 big dats becasue trying to make 1 or 2 big dats on a system with 2 gigs of ram and a 2 GHz processor WILL make DP crash.

[wouanagaine]

From the tips section :

A subfolder bigger than 1Gb should be dispatch to help SC4Datpacker

[dragonshardz]

so each folder to be turned into a dat shouldn't be larger than 1 Gb? i think i missed that part....i'm going to go and reread the readme

[wouanagaine]

Yes, my bigger folder is the BSC with 900 mb ( packed ), and I'm pretty sure I'll need to split it in near future

In fact, the theorical limit is bigger than 1Gb, but I forgot the formula to compute it, and as Maxis made the SimCity_x.dat files around 300Mb, I think it is better to have packed file around that size when possible


And I will edit the readme for futur release to be less confusing

[dragonshardz]

yeah, the qoute you made is kinda confusing.

and the "a word from soulstealer" section should be a given, y'know?

[mittfh]

The BSC folder is a fun one to split up...

DatPacker just about coped with mine split into two (it produced an exception en-route, but was happy to continue after being told to ignore it):

BSC-Basics: BSC Parks, BSC Props, BSC Textures, Day of Peace plus a handful of tiny files (e.g. save alarm, opera house fix, reward essential, region counter etc.) - 696MB uncompressed.
BSC: everything else - 867MB uncompressed.

I also have folders called automata, road, rail, water, fx, utilities and buildings (251MB), plus the various CAM (22MB) and PEG (134MB) folders, as well as SimGoober (316MB - moved out of BSC due to sheer quantity of his lots downloaded/installed). Anything by prominent BSC members heads to BSC, PEG stuff goes in PEGPROD (since he's so prolific!), and pretty much everything else goes into the relevant general folder. I prefer to organise stuff by what it is, rather than who developed it (although BSC stuff goes into the developer's subfolder).

As others have said, it helps to run Cleanitol's cleanuplist beforehand - don't forget to add .ico and .url to the file if you want to avoid "cannot read" errors - and possibly also plugins.ini, which it doesn't like either.

By the looks of things, superseed warnings aren't necessarily anything to worry about - so far, BSC Park Textures, SOMY Industrial Ports, Euro road textures, JRJ Sidewalks and BSC Simcity Rail Utils GA have all triggered it.

EDIT: OK, so it's choked on Buildings - a petrochemical refinery tower 4 description (Chapapotex?) - with an exception. Tried loading the description in DatGen - that claimed an error in the file - deleted the description and DatPacker's happy again.

[wouanagaine]

EDIT: OK, so it's choked on Buildings - a petrochemical refinery tower 4 description (Chapapotex?) - with an exception. Tried loading the description in DatGen - that claimed an error in the file - deleted the description and DatPacker's happy again.

Yes Datgen is the best tool to find out if a file is screwed or not

[mrbombit]

Hi all. So i am having the same problem as some. I have about 2 gigs divided into about 6 folders. When I start the dat packer it gets about half way through then freezes and stops responding. SOooo... if i go this correctly I should divide my folders even more, then run dat packer on the smaller folders, then place all of those (once they have been packed) back into the plugin folder. I hope this makes sense. Thanks again guys for your help!

[dragonshardz]

i had to split my plugins folder into 15 folders for datpacker to run effectively.

what you need to do is split your plugins into an amount of folder that will allow datpacker to work. make sure to remove NAM, CAM, and any mods. They tend to not like datpacking. then run datpacker, and when it's done, move the split plugins to a new folder in your SimCity folder. i recommend calling it "Protected_Plugins". after you have moved the unpacked plugins out, replace your mods.

I also recommend that before packing you run SC4 to make sure there are no mods or lots that conflict with each other.

[High5Tower]

I got around to cleaning up the file with the "Warning." Its amazing what you can over look. Got all my ducks in a row again and have a plugin folder that sets at 2.74 GB. The game runs nice and smooth! The data packer has revitalized this great game for everybody. Thank you.   

[wouanagaine]

I'm playing with CAM NAM etc datpacked ( each on their own folder, so on their own dat file ) without any problem
It was advised to make them external because not everybody understand correctly the correct loading order and its consequences on the game. As CAM, NAM and al are very sensitive to correct load order, it is easier to move them outside

The freeze when datpacking may occur because of a screwed file, as I rewrote the latest version from scratch, without any 'bad' files in my plugins it may not be totally bugproof.

[High5Tower]

Here are some stats on my data packing.
The new version picked up 153 items that totaled 31 mb of extra stuff.
This extra load took the data packer an extra 5 minutes to complete its task. Total time 13 minutes it now takes 8.
The othere big difference is that the new version does not cause such a big defrag occurrence.
After a major data pack like I just did (2.74gb of packing) the computer is very slow, I call it bogey. I do a system restart to bring performance back on line. Very fast and easy fix.
This is a major tool for SC4, couldn't live without it.   
Now about those menues.....

[ander121212]

Quick quetion on removal of duplicates.  I downloaded the HCC lite tool as suggested in the read me.  I see I have 270 Same content and 264 Full clones are these duplicates and am I safe to go to all groups and remove all but one instance?  Thanks for the help.  I did a search of the forums first and could not find so my apologies if this has been answered before.

[SC4BOY]

Good question Ander.. was wondering the same thing.]

I also have another question. What exactly does "superseed" mean? Does it mean that a previous copy was found and loaded first? does it mean that the other file is a later version? if a version is loaded, but another copy is found with a later date, does the later date take precedence? could the previous version be from a megapak while this is a single copy? just curious about the in's and out's of what your program is actually reporting so I can make the most intelligent action based on the report.

For the items that scroll past the 2 windows during the packing process, does the program write a copy of those files somewhere?

[wouanagaine]

Superseed means that the file can be removed from your plugins ( pay attention, I said removed not deleted )
it means that every entry in the mentionned file has been found in one or multiple other files and that SC4 will never use the data of that file.

Precedence is only based on the lexicographical sort of subfolder. there is a section in the readme explaining the load order of SC4, there is nothing base on file date.

As for true duplicates, it is waste of space, waste of loading time

For the items that scroll past the 2 windows during the packing process, does the program write a copy of those files somewhere?

Sorry I don't understand what you're asking

[snorrelli]

Superseed means that the file can be removed from your plugins ( pay attention, I said removed not deleted )
it means that every entry in the mentionned file has been found in one or multiple other files and that SC4 will never use the data of that file.

Hmmm... That seems pretty strange considering many of the files it identifies I'm pretty sure are new and unique creations. I have attached a shot of just some of the BSC files it thinks are "superseed"



Surely it's not telling me I should remove the JENX Burg Eltz building? What about all those jestarr lot files? If I remove them from my plugins, how will the lots grow?

[SC4BOY]

Superseed means that the file can be removed from your plugins ( pay attention, I said removed not deleted )
it means that every entry in the mentionned file has been found in one or multiple other files and that SC4 will never use the data of that file.

Precedence is only based on the lexicographical sort of subfolder. there is a section in the readme explaining the load order of SC4, there is nothing base on file date.

As for true duplicates, it is waste of space, waste of loading time

Do you know if there is an ordered way to export the dups to a directory with a .bat file similar to what you do with Cleanitol?

What I mean here is I'd love to find a less labor-intensive way to take out the dups to a structure that recreates the directory structure. This would mean I could then "message" the data to be sure I have the most current or "best" copy to be used for the datapack process? I want to remove the dup's but I want to not really lose either the actual files nor their "directory structure" so I can reconstruct it if needed. Similar to what is done with cleanitol and it's move/batch-restore process. LOL.. well this is a LONG WAY OF SAYING do you have an automated way to do what you mean in your first paragraph above..

Sorry I don't understand what you're asking

What I mean is .. "is the information shown below saved or savable to a txt file?"

Surely it's not telling me I should remove the JENX Burg Eltz building? What about all those jestarr lot files? If I remove them from my plugins, how will the lots grow?

I believe that the program is telling you exactly that.. the information contained in those files are already in the "DataPack load" before the file you named there is loaded. It may have been found in any number of places.. for example in a GROWABLE folder rather than a PLOP folder .. or visa versa.. or maybe in a MEGA file.. etc.. .

HOWEVER I don't believe DatPacker has the intelligence to TELL YOU that that is the CORRECT file to load. Only you really can tell that.. Further I believe that this is the reverse of what SC4 actually does.. If Datpacker finds a file, it loads it and discards later copies of the information.. On the other hand SC4 itself insteads OVERWRITES the previously loaded information. So I believe you need to eliminate these files at least with a thought to this end.. I'm sure if I'm wrong someone will correct this.. If in fact the files are TRUE DUPLICATIONS then it doesn't matter..

Also remember Wou said "remove" not "delete" .. If you have to recover from an error, but don't have the data, you will be in a bit of a mess. This all is really what all these messages I am writing are about.. how to "best" and "recoverably" edit my SC4 Plugins (and if desired even the install plugin file "Program Files\....\Plugins")

If I am not mistaken, all of this is somewhat of an academic exercize..MOST people can just find a configuration of SC4 that works successfully, then break it up so no folder tree in plugins is greater than a gigabyte or so, THEN just run DATPACKER and run the compressed file.. ALL DONE.. even tho the actual plugins folder is still rather a mess..
Only anal retentives such as myself probably need to be much concerned.. hehe..

[High5Tower]

Remove them and get on with enjoying the game. When I first used the new data packer I had a monster list of things to remove. After about three hours I was finished and  put all the items in a quaranteen folder and dated and packed them away. I haven't had to re-install anything. Believe the data packer its telling you good information. Get on with it and start playing again. Just my view on things. Thanks for this great tool.