SC4 Dat Packer

[tsitrana]

Thanks k808j, now to get packin'!

[Rady]

Hi Wouanagaine,

I just compressed all of my plugins. Two of them produced an error message:

********************* ERROR *********************
in examplar 0x6534284AL 0xD3D5AF87L 0x1491265EL
located in C:\Dokumente und Einstellungen\Morawetz\Eigene Dateien\SimCity 4\Plugins\CAM\SFBT\Pre-Fabs\R$9_4x4_SFBT Pre-Fab Low Rise Double Block.SC4Lot

*************************************************
********************* ERROR *********************
in examplar 0x6534284AL 0x2D2D42D9L 0xD658243L
located in C:\Dokumente und Einstellungen\Morawetz\Eigene Dateien\SimCity 4\Plugins\Dusktrooper\Keaton Plaza\DT_Keaton Plaza Growable.dat
0x00000010:{"Exemplar Type"}=Uint32:0:{Buildings}
*************************************************
********************* ERROR *********************
in examplar 0x6534284AL 0x2D2D42D9L 0xD658243L
located in C:\Dokumente und Einstellungen\Morawetz\Eigene Dateien\SimCity 4\Plugins\Dusktrooper\Keaton Plaza\DT_Keaton Plaza Growable.dat
0x27812832:{"Wealth"}=Uint8:0:{High Wealth}
*************************************************
********************* ERROR *********************
in examplar 0x6534284AL 0x2D2D42D9L 0xD658243L
located in C:\Dokumente und Einstellungen\Morawetz\Eigene Dateien\SimCity 4\Plugins\Dusktrooper\Keaton Plaza\DT_Keaton Plaza Growable.dat
0x27812833:{"Purpose"}=Uint8:0:{Office}
*************************************************
********************* ERROR *********************
in examplar 0x6534284AL 0x2D2D42D9L 0xD658243L
located in C:\Dokumente und Einstellungen\Morawetz\Eigene Dateien\SimCity 4\Plugins\Dusktrooper\Keaton Plaza\DT_Keaton Plaza Growable.dat
0xAA1DD396:{"OccupantGroups"}=Uint32:6:{Building: Commercial,Building: CO$$$,Style: Chicago,Style: New York,Style: Houston,Style: Euro}
*************************************************
********************* ERROR *********************
in examplar 0x6534284AL 0x2D2D42D9L 0xD658243L
located in C:\Dokumente und Einstellungen\Morawetz\Eigene Dateien\SimCity 4\Plugins\Dusktrooper\Keaton Plaza\DT_Keaton Plaza Growable.dat
0xAA83558F:{"Crane Hints"}=Uint8:0:{Outer Then Inner Crane}
*************************************************

Pls advice how to proceed?

thx!

[k808j]

I believe the Keaton Plaza files may have been updated by a mega pack. Use Cleanitol before you compress.

[BarbyW]

The Keaton Plaza files look like the original files which, if you are using CAM, should be deleted. For CAM you only need the model files as the original lots were modded to grow at no more than stage 8. There may also be modding errors as these are old files.

[debaser]

Well I gotta say that for a while I was a little intimidated of using the Datpacker, but after finally encountering some major lag over the weekend (I'm just getting back into the game after a long layoff and of course a reinstall), I took the plunge.

The results are definitely noticeable, but I guess I was expecting a miracle and unfortunately didn't get quite that. 

It's funny, I have a fairly large city of almost 700k people.  Now when I try to move around it, it will still lag pretty heavily if I'm zoomed out.  There's some improvement but not what I'd call a lot.  However, if I'm zoomed in pretty tight I see a pretty big improvement.  I don't see an improvement with loading/saving cities.  Maybe a little but nothing really that noticeable.  Of course this is a very large region (mostly undeveloped however) with all large tiles.  I downloaded it and that's what I got.  Not sure if that makes a difference.  I don't like playing huge regions and I especially don't like playing all large cities, but I liked the map and that's how it came.

I guess it's just the game itself.  I've always heard it was poorly written.  The computer I'm playing it on these days is basically a supercomputer compared to what I used to have.  Quadcore Intel, 8 gigs ram.  Vista runs seamlessly on it.  I am a photographer and work with extremely large files in photoshop (larger than my plugins folder would ever get) and it's very snappy.  It's just SC4.  Maybe it can't take advantage of all the RAM and processors?  I'm guessing that's what it is because even if I get away from the developed areas of a city it moves around pretty freely.   When I hit downtown, though, it sure hiccups a lot.  Was hoping to get rid of that.

But it's improved somewhat.  Yesterday it was difficult even to start dragging a road.  The cursor wouldn't settle in.  I'm sure you all know what I mean.  Now it still does that but it's improved enough that I can deal with it. 

Pretty sure that I did everything correctly.  I'm not going to lie -- I didn't real all of this threat but did read a lot of it.  Read the readme fully and think I understand it.  Kept all my mods and NAM and the AC Airports away from the Datpacker.  Files labeled as essentials were left in the root. 

I've never had to wait four minutes for a city to load, much less some of the times reported in the early part of this discussion.  I would guess that the datpacker is helpful to everyone but probably those with machines that aren't spec'd out all that well will benefit the most. 

Anyway I don't mean to rain on the parade.  Like I said yesterday it was a serious challenge to play and now things are definitely workable.  Real good job with this tool.  Before it came along how in the hell were you guys working with these big cities on these big maps?  Can't say enough about you guys who make CAM, Cleanitol, and this one.  Reminds me of a guy I know that has a 1969 Camaro.  It has a new corvette engine in it, new front end and high tech suspension.  He says it's an amazing car and certainly better than anything produced back then, but it cannot compare with, say, the new ZR1 (maybe on pure power but not on handling).  What I'm saying is you guys are like whoever build this guy's car.  You take something and it is what it is and it will always be that, but you push the envelope by improving every single component that can be improved and it ends of being a hell of a ride. 

[City Builder]

I wasn't aware this tool is supposed to actually get the game to perform better in terms of moving around the screen and lessening the lag.  I've always assumed (maybe wrongly) that it allows the game to load faster since it can load larger one file plugins that used to perhaps be hundreds or thousands of files since it packs them all into one pack.  It does lower the load times for me, but I dont' have any issues with large lands and large cities, it plays the way it plays.  Now when the game first came out and I had my old 486 dx25mhz, then lag city, but on todays hardware it seems to be just fine (for me anyway).

The only time the game feels laggy to me now is when I find something really intensive running in the background (like antivirus deciding to run when Im playing), other than that, game runs rather smooth on my 6 month old computer.

[debaser]

I wasn't aware this tool is supposed to actually get the game to perform better in terms of moving around the screen and lessening the lag.  I've always assumed (maybe wrongly) that it allows the game to load faster since it can load larger one file plugins that used to perhaps be hundreds or thousands of files since it packs them all into one pack.  It does lower the load times for me, but I dont' have any issues with large lands and large cities, it plays the way it plays.  Now when the game first came out and I had my old 486 dx25mhz, then lag city, but on todays hardware it seems to be just fine (for me anyway).

The only time the game feels laggy to me now is when I find something really intensive running in the background (like antivirus deciding to run when Im playing), other than that, game runs rather smooth on my 6 month old computer.

Are you saying that it's as easy for you to scroll around the CBD of a 600k person city as it is to move around in an undeveloped area?

This is confusing.  I'm not bragging but my computer's not a year old and I built this thing to be an absolute beast, and it is with every other program.  Maybe I have something with the game that's not set up correctly?  I mean I deal with 2 gig image files in photoshop regularly and this computer throws them around like they were a little jpeg.  It can't be hardware....something I've done to my game has done this.

[City Builder]

No, obviously it's not going to perform as well on a 600k population city as it is on an undeveloped landscape, however with shadows to minimum or off (i believe it's the lowest setting), scrolling around a 600k pop city is not any great bother.  Is there lag, Im sure there must be atleast a little, but it's really not a bother for my computer.  If however I simply turn on or raise the level of shadows and have too many trees on the map, then yes, I will experience some heavy heavy graphical lag on even a unbuilt landscape (although it takes a ABSOLUTELY HUGE amount of trees, like filling the landscape with them).

But my main point was that I wasn't aware that datpacker was supposed to do anything besides help the game to load faster, is this not the primary purpose of datpacker?

My computer was setup to be mainly for video and photo editing (hence the 8GB of RAM) but it plays all the games that I own first class if I simply turn off shadows or turn them to the lowest settings.  I've never owned a computer (and I've owned quite a few let me say) that was able to deal with shadows properly without taking a hit to performance, SC4 is no different in this regards from what I recall, it's been atleast 2 or 3 months since I've played SC4.

The only time I turn on shadows in any game that I play is to take pretty screenshots.  Perhaps shadows are slowing down the game for you too.

[debaser]

No, obviously it's not going to perform as well on a 600k population city as it is on an undeveloped landscape, however with shadows to minimum or off (i believe it's the lowest setting), scrolling around a 600k pop city is not any great bother.  Is there lag, Im sure there must be atleast a little, but it's really not a bother for my computer.  If however I simply turn on or raise the level of shadows and have too many trees on the map, then yes, I will experience some heavy heavy graphical lag on even a unbuilt landscape (although it takes a ABSOLUTELY HUGE amount of trees, like filling the landscape with them).

But my main point was that I wasn't aware that datpacker was supposed to do anything besides help the game to load faster, is this not the primary purpose of datpacker?

My computer was setup to be mainly for video and photo editing (hence the 8GB of RAM) but it plays all the games that I own first class if I simply turn off shadows or turn them to the lowest settings.  I've never owned a computer (and I've owned quite a few let me say) that was able to deal with shadows properly without taking a hit to performance, SC4 is no different in this regards from what I recall, it's been atleast 2 or 3 months since I've played SC4.

The only time I turn on shadows in any game that I play is to take pretty screenshots.  Perhaps shadows are slowing down the game for you too.

It could be.  I'll try turning them off.  Datpacker is purported to speed up a number of things aside from that which it was intended.  Just from what I've seen so far, it speeds up everything just a little but loading time, ironically, not so much.  But things like going from the zones data view back to the regular view, opening the parks menu or whatever...these things have sped up most noticeably for me. 

[City Builder]

It could be.  I'll try turning them off.  Datpacker is purported to speed up a number of things aside from that which it was intended.  Just from what I've seen so far, it speeds up everything just a little but loading time, ironically, not so much.  But things like going from the zones data view back to the regular view, opening the parks menu or whatever...these things have sped up most noticeably for me. 

Yes things like opening the parks menu, if you have a lot of content and it's not packed, then it's slower than if the files are packed.  For me on my last install of SC4 when I had about 4GB of plugins, it dropped my load times from many minutes to only a couple minutes.  Of course if someone has no plugins or only 100 or some such small amount then the load time isn't going to be all that noticible since SC4 is perfectly capable of handling a reasonable amount of plugins when they are all loose files, it's only when you really start to collect the plugins that it starts to really slow down SC4's load time.  Right now Im up again at about 3.2GB Packed and the load times are still reasonable.  I dare say that if I were to put back in all the loose files instead of the packed files my load time would go up again.

[debaser]

Yep.  I think I have this all straightened out now.  I took your suggestion to turn the shadows down and that worked okay.  But then I thought about software rendering.  I have a good graphics card but then you know SC4 is just screwy with that stuff and with the kind of software power that you and I must have, it should be able to do that, right?  Well, I tried it and it works great.  I even turned shadows back to high and it still works great.  Everything looks fine, and just as importantly there is NO lag at all.  Not even when zipping through the densest part of a big city...it's as smooth as it could possibly be. 

I say it looks fine, but there may be a little bit of a drop off in the graphics quality.  Honestly I can't tell but that would make sense.  But still if you're playing the game as in building a region, this solution works very well.  If you're doing a mayor diary then, yeah, you probably don't care as much about being able to move around and you can set it back and restart to make your illustrations or whatever.

So with this simple option clicked, cleanitol having dealt with my plugins file (which was not as big as yours but it's pretty big), and now the datpacker having it's way with the beast, SC4 is for the first time in my life working the way it ought to work.  So far anyway.

[High5Tower]

I pay attention to threads like this since my machine is getting older now. Dell 3gh, 2gb ram, and a good video card back from 1985 when I bought this computer. Its been a great machine for SimCity. I had to start data packing when my plugins got into the 700mb area. Now I have 4.3gb of plugins and this machine runs fantastic. The only thing that never gets updated is the video card driver. Same one since May of 85. I have tried more current drivers but SimCity does not like them. I can scroll around pretty well in my bigger cities but I can see the improvement that the latest machines can give. I plan to buy or build a new machine in 2011 but I well still keep this baby in tip top running order just as backup. My how costs have come down on parts.  I upgraded to 1gb of ram in late 85 and it cost around $200 back then. Now I even have 2 extra sticks of ram (1gb each) for $50. One thing for sure though without data packer we would all be dead in the water.

Need to correct the date above its 2005 and not 1985. Did buy my first computer in 85 though a Victor and it cost a good $5000 back then. Was a great machine but the floppies didn't like the dust.

[City Builder]

Need to correct the date above its 2005 and not 1985. Did buy my first computer in 85 though a Victor and it cost a good $5000 back then. Was a great machine but the floppies didn't like the dust.

Yeah I had a feeling that was something like that.  I remember, when SC4 first was announced (or shortly after that, I remember seeing screenshots and thinking I'd have to buy a new computer to play it), so I bought one on time on my Mobil Gasoline card (yeah they used to send offers like "buy this 486 dx25 with monitor for only $2500.00 on installments using your Mobil Gasoline card) so that I'd have a computer capable of playing SC4.  Prior to that I was still running on an old 386 computer used mostly to just play shareware games and or chat on old dialup BBS's.

Anyway, yeah datpacker is  great as it does allow us to use many more plugins than we might other wise be able to do.  I'm positive the developers never had any ideas that we might have as many plugins as we have available to us today when they were developing SC4.

[allan_kuan1992]

Umm... I know that Wou is currently away and has limited time to work on this... but is there a way to change the folder paths in the program? I have most of my downloaded programs in "I Plugins" and the DATpacked files in the "Plugins". Every time I run the program I have to rename the folders to "Plugins" and "Plugins I" respectively. If there was a way to customize the paths a little it would save me some time trying to find the folders and renaming them before and after running the program.

- Allan Kuan

[SC4BOY]

I certainly am not arguing against configurable paths (in fact I'd say that FOR TOOLS I'd support that they ALL have configuration control... through .ini files or whatever) but in the case for SC4 the difficulty is that the world of SC4 basically revolves around the "standard path" and changing datpacker would have minimal impact on the overall issue.. for example many programs use "installers" that are singularly aimed, etc. just my 2 cents..

[brooksy10366]

hi,
   i was wondering if we could use the datpacker on the lex dvd? thank you for your help

[z]

i was wondering if we could use the datpacker on the lex dvd? thank you for your help

Sure - it will pack anything.

But this brings up a related question I have about the DATPacker and the LEX DVD.  Many files on the DVD are duplicates of ones in my Plugins folder.  If I just DATPack everything together, will it essentially get rid of the duplicates, so that the resulting compressed files are no bigger than if I had culled the duplicates manually?  I wouldn't have thought so initially, but I've seen some hints that this might be the case.

[jmyers2043]

will it essentially get rid of the duplicates,

To some extent yes.

The Dat Packer will look at my BSC folder and all its subfolders. Examine, compress, and write the BSC.dat file. And I believe it will deal with duplicates within the BSC folder. It doesn't 'get rid' of them per se, but simply will not write the same file twice into the dat file.

But what if I have the SimGoober_Props_Vol 01 in my BSC and SimGoober folders? Not likely. The packer will examine the SimGoober folder in due course. Compress and write the SimGoober.dat file. In that case I will have the duplicated prop file in both dot dat files.

There are probably other good programs that find duplicate files. I wouldn't use the dat packer for that purpose as its primary function is to speed the loading process of the game. 

[SC4BOY]

... Examine, compress, and write the BSC.dat file. And I believe it will deal with duplicates within the BSC folder. It doesn't 'get rid' of them per se, but simply will not write the same file twice into the dat file.

I think that the effect you suggest is generally true, however unless the DatPacker has been modified it will simply overwrite (not refuse to overwrite as you've suggested above) the first info. This is why the ordering of files is important .. for example simulators or mods like the HSRP or the BSC Park Textures, etc needing to be in "z files or folders". DatPacker used to give a running output of "duplicate info" in the bottom window as it ran, but Wou removed that somewhere along the way.. perhaps because it was causing so many support questions from people that didn't understand what to do..

Further as you suggest, datpacker basically does the same thing as the SC4 game at load time, so you are in effect "preprocessing" it.. It does however save a HUGE number of file handles.. cutting it from 100's of thousands sometimes to under 10 thousand. Some folk argue that in theory the datpacker should only help for "load times" but I and others have shown that it does in fact improve performance under certain conditions and the file handles count is the only thing I can think of that changes significantly..

[High5Tower]

To keep items in order I have 24 different main file folders that can be data packed individually.A new download is put in its appropriate folder and just that folder then needs to be data packed.
All Props (no BSC)
All Rewards & Landmarks
Big Name BATS
Big Box & Small Retail Stores
BSC BAT Props; BSC Mega Props; BSC Props; BSC Textures
CAM; CAM Commercial;  CAM Mega Props
City Wide Services
Commercial
Commercial stage 8
Current Likeables
Farm stuff
Industry
Mods & other important standards
Parks & Other related stuff
Pegrod
Residential
Residential stage 8
SG Mega packs
Transportation & Road
For the last couple of years this method has worked very well for me with around 4.5gb of plugins.
Of course there are many subfolders in each of the above listed main folders but you get the idea.
Gotta keep your ducks in a row.