Datpacking and DAMN

[joshua43214]

I am having trouble getting both DAMN and datpacking to work properly together. I've tried a bunch of different combinations, and spent alot of quality time on the forum here trying to find a solution.

I am sure this is a tired old topic, I seem to remember this topic coming up a bit in the past. All the posts I can find though predate the current versions of both SSPTool and Datpacker. I am sure my issue is not unique, so its probably a good time for the SC4 gurus to discuss it again

My problem is that I can get the DAMN to work properly after I datpack unless I datpack my entire plugins folder - then I am missing alot of custom menus when I click on something like a CAM lot.

I run only a handful of mods - NAM, CAM, and the BSC port mod. No slope mods, demand or growth mods, etc.

I've tried different variations of not datpacking NAM, DAMN_Menu, DAMN-Remove, etc., packing and not packing z-files. Moving stuff to c:/programfiles/maxis/.../plugins. Did this for several hours trying to find a combo that gives me a DAMN menu and hides stuff in the Maxis menus.
I always rerun the file indexer in SSPTools after datpacking.

I tried following the readme in datpacker as closely as possible, but Soulstealers method seems a bit counter intuitive to me

Step 5.5 (Optional)
I move all texture, prop files I can find into my "program file/maxis/simcity/plugins/props_and textures/" folder as these get updated often and its easier to find duplicates and remove oldies when necessary. I also use different sets of plugin according to the city I'm making (like Urban city and Rural town) so each time I load a different set of BATs and lots but the props and textures are shared by them both so it's better to keep them in a central place e.g the "program files\maxis..." folders).

This looks to me like updating files would be difficult since Cleanitol only checks the mydocs/.../plugins folder, and it would require updating each file my hand.

Of particular difficulty seems to be hiding the NAM walls, They only seem to stay hidden if NAM is datpacked...

Part of my problem is that I am not sure what files in particular are hurting performance, is the the lot and model files, or the DAT files?

My plugins folder is only about 2.5gigs, the game runs fine except for switching between above and below ground views, and then it takes forever to make the switch. I prefer the lag to scrolling through a hundred or so canal mods so I can plop a couple of BSC parks...

Any and all comments would be greatly appreciated, It would be especially nice if someone took the time to do a step by step tutorial type thing for integrating datpacking and DAMN, along with NAM and CAM.

Thanks Again in advance

[joshua43214]

^^bump^^

I'm still living with this issue, if anyone has any hints or a direction.

Thanks.

[ZonZaucker]

that's my issue too!! waiting for this to be discussed, since I don't have a clue!
morever, ssptool crash anytime i try to read my plugin compressed folder!
thanks in advance...

[ScottFTL]

Joshua, can you provide more details on what exactly you are doing?  I understand that you're having problems with DAMN menus after using DatPacker, but I don't know exactly what the problem is or what you are doing with your files.  While these are both powerful tools, they take a little work to master.

I use both without issues.  I keep all Plugins under Documents rather than Program Files to start, then I keep a packed and unpacked version of my Plugins.  I've heard some say that DAMN works better on packed files, but I've seen some strange behavior in SSP Tool with packed files so I always use the unpacked files.  Once you have created your DAMN menus, then you can run DatPacker.  After packing, you have to move everything except the Plugins Compressed folder and any folders that you skipped out of Plugins.

I do pack the zzz_DAMN Remove folder, but it will not work properly unless it is moved from Plugins Compressed to zzz_DAMN Remove.  Obviously, the unpacked zzz_DAMN Remove folder has to be moved before this step.  You want to create a new zzz_DAMN Remove folder and move the zzz_DAMN Remove.dat file into that folder.  Or you can simply leave the zzz_DAMN Remove folder unpacked.  The important thing is that the files in this folder load last, which they may not do in the Plugins Compressed folder.

[pezzajn]

@ScottFTL:  I posted this question yesterday over at SimPeg perhaps you might be able to have some insight too how you get it to work (if you do of course).  Does Datpacking the DAMN Remove folder result in allowing the DAMNed items to remain hidden in the game menus?  When I tried to datpack the ZZZDAMNRemove the dat file blows up to a Gig.  So I decided to just leave it uncompressed.  Should I have started this whole thing off with DAMNing my menus using "Use One DAT file for hiding each menu" checked?

_______________
Has anyone run through this same anomaly (a.k.a. problem) and know of a solution for it, have any insight on it, or to get around it?  Or is this a byproduct I will have to live with?

The issue is that I finally bit the bullet to use the SC4 DATpacker, after using it on select folders, and getting back into the game I ran into a problem that all my hidden DAMNed content reappered in the game menus.  Selecting the lot from the DAMN menu still works just fine.  But lot icons are just not hidden anymore. 

A little background and some observations to set all you expert helpers up (and I apologize up front for this post's length)...

I'm using Win XP Pro with SP3, and have a Pentium 4 processor.  Learned recently Pentium 4 emulates dual core and about setting affinity to one CPU, and the priority to high.  Definitely makes an improvement on the game.  However, XP does not recognize START AFFINITY command, and I have to set affinity manually with Task Manager.  At least that was up until last week when I figured out how do this automatically using Microsoft Application Compatibility Toolkit 5.6.  Works like a charm and now setting priority HIGH & affinity to just CPU 0 are working automatically and all seems more stable.  So, all cool on the front end.

Now onto the software stuff and my OCD for organization.

I've meticulously organized my plugins folder & effectively DAMNed near everything, into similar folder structure (except those things I don't mind scrolling game menus for, mostly MMP paintbrush flora stuff, police, fire, utilities, stuff like that, etc.). 

Since installing DAMN about a year ago, I've kept with the default install setting for hiding files which is to use a separate folder for hidden lots (folder named "ZZZZ_DAMN Remove" in the plugins folder), and I have unchecked the "Use One DAT file for hiding each menu" in the DAMN Options.  Also, unchecked is modifying the original exemplar (I don't think it wise to mess with the original).  However, I have observed some pros & cons with these settings, which can be a post in-of-itself if you would like me to elaborate.
 
All hidden icons with these settings and my setup are hiding from in-game menus & working properly with DAMN.

After hearing all good things, I finally decided to try the DAT packer with my setup, SC4DatPacker 2008 version to be exact, since on LEX the "newer" version is locked.   I selected to DATpack folders in my plugins that, for me, seemed appropriate to compress.  After compressing a selected folder into a single dat file, I removed the original folder/files from the plugins folder.  Created a new folder with the same name and added "_compress" to the end of the folder name and placed the single corresponding dat file in that folder.  I did this figuring it would maintain the load order for my plugins even after the compression.

Here is a screen shot of my plugin folder after compression.  Folders tagged "FolderName_compress" now just have the single dat file (e.g., "FolderName.dat") instead of my original structure of sub-folders/files that were in that folder prior to compression.  I did not compress ZZZ folders with the mods I wanted to load last, or the ZZZZ_DAMN Remove folder.



Rebooted computer, defragged, rebooted computer.  Old Plugin folder size was 3.03 Gigs, 2,142 Folders, 14,702 Files.  Now after packing it is down to 2.64 Gigs, 105 Folders, 3,626 Files.

Then went back into the game.  Without having a stopwatch to time it, I really did not notice a difference on first load time, but I am sure there has to be something.   Once in game, I did notice the stability of the game environment improved and navigating DAMN menus was a little more "speedy", to use the non-technical term.

Clicked to my in-game menus to see if lots were still all hidden and....well I'll direct you back to my first two paragraphs.  All my DAMN hidden icons were now back in the game menus again.

Aaaaargh, Good Grief!  As Charlie Brown would say!

So can anyone clue me in on what I did, or didn't do, that could have been done differently to keep my DAMN icons hidden?  Is this typical of others setups, meaning there is no workaround, and you live with cluttered menus? 

Worst case, I can still just close up Dat packer and swap back to my original plugins folders setup and go back to the way it was before compressing.  At least there I know they stayed hidden.  And I will have to learn to live with a little lag in game play.

Thanks in advance for anyone who can provide help or insight.

-   Jack

[ScottFTL]

@ScottFTL:  I posted this question yesterday over at SimPeg perhaps you might be able to have some insight too how you get it to work (if you do of course).  Does Datpacking the DAMN Remove folder result in allowing the DAMNed items to remain hidden in the game menus?  When I tried to datpack the ZZZDAMNRemove the dat file blows up to a Gig.  So I decided to just leave it uncompressed.  Should I have started this whole thing off with DAMNing my menus using "Use One DAT file for hiding each menu" checked?

What happens when you load the uncompressed version of your Plugins including the DAMN Remove folder?  Do your DAMN menus works properly, and all are lots hidden as they should be?  I would do this to verify that the problem is DatPacker and not SSP Tool.  I've noticed that SSP Tool creates hidden items properly the first time around, but changes to the menu will not create the proper files in the DAMN Remove folder.  I haven't found a way to fix this, aside from deleting the DAMN menu completely and starting over.

Also, are you seeing problems with everything or just certain lots?  It could be something about the originals that is causing problems.  It shouldn't be widespread, but you may have to choose between DatPack or DAMN in some cases.  I think with enough work you should be able to make anything work, but sometimes it's just not worth the effort.   

In general, you can datpack the DAMN Remove folder.  The files in this folder have to load last to hide items in the game menu, but there is otherwise nothing unusual about them.  Are you saying that the datpacked version of this folder is larger than the original?  Do you have the Optimize space and Use only useful TGI options selected in DatPacker?  I can't imagine how it could make a bigger file, but I wouldn't datpack the DAMN Remove folder if this is the case.

As for the Use One DAT file for hiding each menu in SSP Tool... I've had mixed results with this.  As I said above, SSP Tool does not always refresh these files properly as you edit menus.  There seems to be some situations where it can resolve certain problems, as shoktherapy mentioned in this post on SimPeg.  I've also seen plenty of situations where it can cause problems.

On a side note... The organization of your compressed files is not necessary.  DatPacker maintain the load order when it compresses your files, so you can leave everything in Plugins Compressed.  The only one you need to move is the zzz_DAMN Remove.dat file because it has to load after any uncompressed content you might have.

[ZonZaucker]

@ScottFTL: thank you very much!
I missed the part in which I could use ssptool with unpacked plugin and THEN use datpacker!!!

[pezzajn]

What happens when you load the uncompressed version of your Plugins including the DAMN Remove folder?  Do your DAMN menus works properly, and all are lots hidden as they should be? 

Using uncompressed version of plugins and DAMN Remove folder work fine.  All hidden items are as they should.  When using the DAMN Remove folder with compressed plugins it is set to load last, and it is uncompressed.

Also, are you seeing problems with everything or just certain lots?

Appears all lots are unhiding themselves.  But I have so much content it would take some time to really get down into which might or might not be working properly.  Also, if I scroll menus to fast game locks up and I have to use Task Manager to end process.

In general, you can datpack the DAMN Remove folder.  The files in this folder have to load last to hide items in the game menu, but there is otherwise nothing unusual about them.  Are you saying that the datpacked version of this folder is larger than the original?  Do you have the Optimize space and Use only useful TGI options selected in DatPacker?  I can't imagine how it could make a bigger file, but I wouldn't datpack the DAMN Remove folder if this is the case.

When I ran datpacker on the DAMN Remove folder it bombed (ceased to complete), and during the process created two dat files in the compressed plugin folder one that "blew up" over a gig and one that stayed around 32 Mb.  The original DAMN Remove folder prior to compression only has 3,333 files.  With windows reporting size to be 2.16 MB (and size on disk is reported as 13.3 MB).  I decided to forego trying to compress this folder and leave it uncompressed.

Just to give it another go, last night I started from scratch with my original setup (uncompressed) all working fine.  Compressed only the folders that did not start with "z".  Did this in a single run by selecting all the files at once and compressing.  Took about 30min.  Both Optimize Space and Include Only Useful TGI were checked/selected.  Did not check/select force repack.

Removed the uncompressed folders and left the compressed DAT counterparts in the plugins_compresssed folder.  DAMN Remove folder is still uncompressed and loads last.  Started up SC4 and same problem, lots all show up in the menus again.

As for the Use One DAT file for hiding each menu in SSP Tool... I've had mixed results with this.  As I said above, SSP Tool does not always refresh these files properly as you edit menus.  There seems to be some situations where it can resolve certain problems, as shoktherapy mentioned in this post on SimPeg.  I've also seen plenty of situations where it can cause problems.

I ran across that problem a while ago with some of the datpacked sets and came up with workarounds.  For example Simgoobers retail set has tons of stuff (like 100 items) I wanted to breakout into separate folders (like food stores in one folder and restaurants in another).  If I created one folder and DAMNed it I was fine (lots selected were hidden).  Then if I created another folder with say the restaurants.  All lots but those indexed during the creation of that folder would reappear in game.  If I created everythign and then indexed all would be hidden.  However, my workaround was to create a dummy folder for all lots in the dat file.  So regardless of whether they all appeared again (say if I added a lot to the DAMN Folder and reindexed), I would just go to the dummy folder, remove all the SG lots, index, add all the lots and index and all would be fine in game, ergo all hidden and DAMN menus working fine.

However, I wonder if I initially had set the DAMN hidden settings to Use One DAt file per menu if that issue would've stopped manifesting itself.  And if this is why I am having problems with the lots not hiding correctly.

Also wondering if I compress with Optimize Space or Include Only Useful TGI unchecked if perhaps one of those settings will fix problem.  If I get time I might try various combinations later this week.

Also wondering if I was to successfuly DAMN teh DAMN Remove folder if that would work.  Though trying to figure out why it blows up like it does may take some extra time.

Any thoughts?  Suggestions?

[ScottFTL]

Also wondering if I compress with Optimize Space or Include Only Useful TGI unchecked if perhaps one of those settings will fix problem.  If I get time I might try various combinations later this week.

Also wondering if I was to successfuly DAMN teh DAMN Remove folder if that would work.  Though trying to figure out why it blows up like it does may take some extra time.

Any thoughts?  Suggestions?

Hmmm... I have to say I'm stumped.  I don't understand how they can be hidden when your plugins are not compressed but not when compressed.  DatPacker just makes a bunch of small files into a big file.  You should try without the Include only useful TGI option.  This will ensure that DatPacker doesn't remove anything it thinks is unnecessary.

[SC4BOY]

In the overwhelming percent of the time a DatPacked file won't operate it is the result of trying to do maintenance on a datpacked plugin's file. The datpacked setup should never be used for "maintaining" your SC4 game.. do ALL maintenance in the UNPACKED pluging files (including things like changing NAM parameters, using the program to adjust simulator speeds or network parameters, making changes to your DAMN setup, etc). I've never come across the legitimate case where a CLEAN datpacked load will not work where it is also true that the exact UNPACKED plugin will also not fail.

To test it.. delete your datpacked setup... run the game.. does it work properly? if so, then datpack (doing absolutely no changes other than datpacking).. it should run fine.

[Lowkee33]

Also wondering if I compress with Optimize Space or Include Only Useful TGI unchecked if perhaps one of those settings will fix problem.  If I get time I might try various combinations later this week.

Also wondering if I was to successfuly DAMN teh DAMN Remove folder if that would work.  Though trying to figure out why it blows up like it does may take some extra time.

Any thoughts?  Suggestions?

Datpacking with the Optimize Space option off might actually make a difference.  The optimize space function converts files to binarized form (EQTZI to EQTB1).  I've had a few issues solved by optimizing space, so I guess it makes sense that it could cause problems too.

Did you already try it?  Did it work?  You might want to try putting each DAMN file into its own folder with the same name, and then DatPacking with Optimize on.  This would at least make all of your file types the same.

Sorry if its no help.

[pezzajn]

Would have posted earlier but RL has been busy.

Thank you all for the input so far.    Just a quick update & recap from this past week. 

I took a closer inspection of which lots are hiding and un-hiding themselves.  I noticed that maybe 1-3% are remaining hidden.  Why?...I can't say for sure.  Considering I hid all lots using the same method when creating my DAMN menus.  Of those lots hidden, the original file for said lots is sometimes from a file ending ".dat" or ".sc4lot file".  So it still means 97% are not conforming to the expected, which of course is unexpected.  Percentages of course are my estimate.

Have now experimented by starting with the original configuration and datpacked the same folders with only selecting Include Only Useful TGI.   Unfortunately the result ended with the same results.  With this configuration majority DAMN lots set hidden do not stay hidden in game menus.

Went one step further, started again with the original configuration and datpacked the same folders with no options selected (i.e., all boxes unchecked).  Again results ended the same.  With this configuration majority (of DAMN lots set hidden do not stay hidden in game menus.

Note when running datpacker on those folders mentioned above I have no errors during the runs.  At least that is a positive on packing my originals.

During these experiments, I left the ZZZZDAMNRemove folder unpacked given the issue I noted earlier.  I may this weekend try and datpack it in smaller groups to pinpoint which files are given it a problem during compression.  Wondering if by getting the ZZZDAMNRemove folder compressed if the lots will start hiding again in the menus.

If that fails, perhaps I will try to recreate my DAMN menus with DAMN Settings selected for using one dat for each menu.  Which I never did before.  Shokthrpy mentioned this in another forum, and it may be worth my time to investigate.

Last resort of course is just playing with my files uncompressed.  Though I could probably compress the folder that I keep props and textures separated.

[pezzajn]

Beware A Shallow Double Post Ahead...

Eureka a Breakthrough!

I found the culprits in my ZZZZ_DAMNRemove folder that were the cause of the file "blow up" during compression with Datpacker, and was able to neutralize/correct them.  After doing so I then was successful in compressing the ZZZZ_DAMNRemove folder with datpacker.

Loading the game up with both the compressed plugins and the compressed ZZZZ_DAMNRemove folder (loading last of course) provided a successful end to this Merry-go-Round ride.  This combination resulted in the DAMN menu items set hidden to remain hidden again in the game menus!


The issue with the ZZZZ_DAMNRemove folder was with two files.  Found them quickly using the usual routine to split the folder's contents in half to separate folders, and datpack both  separately to determine which one compressed fine and which one blew up.  Continued doing this until I narrowed it down to the two files.

The first file belonged to a lot for Jules78 Art Museum (.SC4Lot file) that always gave me problems when trying to hide the lot during DAMNing, as it kept showing up in the game menu despite my attempts.  I just given up on it and decided it was fine in both game and DAMN menus.  The original lot showed no problems compressing, so I am guessing when DAMN created the hidden duplicate in the ZZZ_DAMNRemove folder it must have corrupted it somehow.  As I have a pretty long list of other museums, I decided to just delete that museum & its corresponding hidden duplicate from the plugins altogether to avoid possibly creating the same issue again in the future.

The second file belonged to the Horse_MMP_V100.dat from Paeng.  Good thing I took a closer look at what happened to this file.  There is no issue with the original or with hiding it with DAMN, if done correctly.  The cause was a byproduct of my own experimentation with copying a folder in the DAMN Root folder...specifically...After release of SPAM, I created a DAMN menu folder for Livestock under a parent menu folder Agriculture (SPAM).  I then wanted to have a duplicate of this Livestock folder under my DAMN menu folder called 02_Mayor Mode Ploppables.  Trying to save time from recreating the menu in DAMN, I just copied it in the Root Folder directly from the SPAM folder to the MMP folder.  I then must of added the Horse_MMP later to this folder and indexing had locked up.  A closer inspection of the contents of the folder Root\02_MMP\ shows it contains two DAMN-Menu_######.dat files.  Also when viewing the same folder in the DAMN Manager shows a duplicate of all menu items (strangely I had not noticed all this before).  I gather that when trying to add the Horse_MMP to the menu items, must have put it through the spinner not knowing which Dat menu file to associate it to.  Thus indexing corrupted the associated duplicate file on its creation in the ZZZZ_DAMNRemove folder.

Deleting one of the two menu files (particularly the one identical in the other location), removing the Horse_MMP from the menu items, running indexer, and then adding Horse_MMP and reindexing corrected the situation.  Checked by playing game with uncompressed plugin configuration and later with compressed configurations.

Lesson learned don't copy Menu folder items in the Root folder.   
Though it is OK and works to move them around or rename them without problems (just remember to run the Indexer after doing so to update changes).

So long story short, I am now successfully able to compress my ZZZZ_DAMNRemove folder.

I was also curious and toyed around with the following combinations and in-game results:

Setup 1) Uncompressed plugin folders with an Uncompressed ZZZZ_DAMNRemove folder works correctly hiding lots in game menus.  (i.e. the original setup)

Setup 2) Uncompressed plugin folders with a Compressed ZZZZ_DAMNRemove folder works correctly hiding lots in game menus.

Setup 3) Compressed plugin folders with a Compressed ZZZZ_DAMNRemove folder works correctly hiding lots in game menus.

Setup 4) Compressed plugin folders with an Uncompressed ZZZZ_DAMNRemove folder still does not work correctly as hidden lots keep showing up in the game menus.


Interesting is Setup 4 is still a mystery :shrug:, but one that I can let go of now that I can get Setups 2 and 3 to work.

Overall Game stability is definitely noticed, as well as the lack of game lag experienced in the uncompressed configuration (i.e. scrolling or zooming, navigating through both the game menus as well as the DAMN menus).

Though long to get there, and made my head    I think at least it was worth going through the process to find and correct those two culprits in my plugins.   

Thank you all for listening to my rants and raves, and for all the good input.  Now I am gonna go get my game on.

- Jack     

[packersfan]

I am having a major problem.  I datpack my plugins folder...including the DAMN remove...I pull the DAMN.dat out of that compressed folder...but when I try to load a city the game just Crashes to Desktop?  Very disappointing.  I would love to get more stability and better performance, but it seems my game is just going the other direction with everything I try.