• Welcome to SC4 Devotion Forum Archives.

Reverse engineering SC4, help welcome.

Started by speeder, January 20, 2016, 03:52:29 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

speeder

I am reverse-engineering SimCity 4.

Now that lots of people are shocked by the previous phrase, I would like to comment on the legality of it, also keep in mind I won't discuss it again in the rest of the thread, and I ask people to also not do it, if you are not ok with SC4 reverse engineering, don't complain on my thread.

So, is reverse-engineering SC4 legal? It depends, in US the heavy handed DCMA laws make this a complex issue, still the DRM-less GOG.com version avoid this, then the problem in US is the fact that you will be breaking the EULA, and US considers the EULA binding.

I am not in the US, here not only we don't have a DCMA equivalent, click-wrap EULA are not binding (EULA are only binding here if you make it an actual contract, where people physically sign a paper), also our laws state specifically that learning about software for interoperability,  competition, or learning, is allowed.

This mean that in my country even doing something like OpenTTD or OpenRCT2 (creating a engine out of the reverse engineered code) is completely legal.


That said, this is NOT my aim here, SC4 game is too complex, and badly coded anyway, if someone ever made a clone of SC4, it would make more sense to code from scratch, and as we know even that has been failing.

So, why I am reverse-engineering SC4? Mostly, to find what it can do, and maybe fix bugs, and make interesting mods.


Now to the interesting bits:

What I found so far?

1) Non-documented command line switches... unfortunately I don't found out how to use them properly (I managed to make some switch even create a random file on my folder, but not write anything on it), one of the switches actually loads Lua files and executes them, but I don't found out how to test (ie: how I make the Lua file do something meaningful so I know it worked?)

2) Some "unknown" stuff in the Wiki are not unknown to me anymore, for example the page about input of graphs, state it found some inputs they are unsure if they work, I stumbled into those by accident, and I can say that they DO work, and I even know what simulator they track, but it was accidental, and I don't chased down to know what exactly they track (example: the two first unknown values on the wiki come from the power simulator).

3) The game engine supports much more stuff than we have seen, it has a fully working weather simulator, that is just not used (it calculates things, heavy calculations even, but nothing uses it, thus it waste CPU), many, many, many more dialog types, including pizza charts, a window named "LuaConsole" (I am curious about that one!), and a couple of other interesting tidbits.

4) The game loads the config file by copy and pasting the entire file to the memory! It makes figuring what it do kinda hard (it requires tracking the individual part that accessed a part of the file, and then figuring what it wanted with the information).

5) The game supports more music than it was shipped with, without any modifications (up to 30 tracks per "radio" it seems, meaning you can have 5 new mayor tracks, and lots more terrain tracks).

6) The game also support threads, in fact it shouldn't crash when multicore is active, probably a bug can be fixed somewhere that make the game work with threads smoothly, I even activated 8 cores on my machine, and the game don't crashed (Because of this at least) on me yet.

7) Many exemplars cannot have extra fields used, the parts that read exemplars do it in a fairly rigid manner, looking for specific things in specific places, if you use the properties list to add extra properties to some exemplars, the game will just ignore them, that said, there are some properties that are not documented.

8) The game actually reads the Ingred.ini file, but I don't found out yet what it do with it.

9) It supports many more options than the interface let you change, one obvious mod to make eventually is a configuration app to change those.

10) There are more cheats than the list people know, also unfortunately some promising cheats are disabled, for example the cheat named "GZLog" reaches on the code a function that is only written "return 1", meaning the game understand a cheat ran, but nothing actually happens.


Now some interesting history and technical tidbits:

The game engine is nicknamed Rizzo-Gonzo, because of the Muppets (yep... O.o), it was used not only in Maxis games, but also in the prototype for the cancelled Ultima Online 2.

The game engine is quite old, seemly the first game that used it was SimPark (and yes, back then it also supported dll plugins on the style of cheats dll!!)

SimCity 4 version of it, and some other games of the same era, used STLPORT on Windows, the Mac version use GNU::STL, and that might be the source of some of the Mac build bugs.

Recent versions use EASTL, that is EA amazing take in rewriting STLPORT, EASTL is also partially open-sourced, you can find it here: http://gpl.ea.com/ (it is inside any of the webkit projects, EAWebkit needs the opensourced part of EASTL to be compiled).

Parts of Rizzo-Gonzo are floating around on internet, we have partial sources to the GZCOM system, and partial sources to their compression engine (someone put this here in another thread, it came from an attempt of modding SC3000 and some nice employee at EA handed over that), also the comments in the EAWebKit sometimes explain design decisions of Rizzo-Gonzo or how some things work (for example the reference counter used in Rizzo-Gonzo is partially released with EAWebKit).

There are some GDC talks about Rizzo-Gonzo, the most interesting version is this one: http://pisa.ucsd.edu/cse125/2006/Papers/Game_Development_in_C++.pdf


Also the engine was designed to be easily extensible and pluggable, not for modding, but to make things easier to develop.... unfortunately it backfired, made developing hard (Ultima Online 2 team complained about the issues of GZCOM), but we can mod SC4 easily, with enough information we can call any SC4 class, and invoke any function we want by using a DLL, meaning we can do whatever we want without modding the .exe, we don't even need to "hook" and edit the .exe on the fly, the game has proper interfaces to do proper calls.


And now I am tired of typing, and help is welcome... the engine is a mess, hard to understand, and I don't think I can go much farther without help.

APSMS

Quote from: speeder on January 20, 2016, 03:52:29 PM
I am reverse-engineering SimCity 4.

That said, this is NOT my aim here, SC4 game is too complex, and badly coded anyway, if someone ever made a clone of SC4, it would make more sense to code from scratch, and as we know even that has been failing.
Just so you know, discussion about EULA violations (along with redistributing plugins w/o author permission) and reverse-engineering (& piracy) are generally violations of site rules, not anyone's particular attitude towards piracy or reverse-engineering in general (although admittedly plugin redistribution is a bit of a...um, I can't say it here, but you know what I mean).

That being said, since the purpose of this thread is mainly for the discussion of DLL creation, and not .exe modification (which is expressly illegal in the US and prohibited by the site, rules 8-11), I think we can excuse the supposed "reverse-engineering", especially since the last time that DLL flags were released to the community by MAXIS, the implicit statement made was that the community could reverse engineer the two official DLLs currently available, so that more might be made.

You finally cracked the code (so to speak) a few weeks ago, and here we are. I think you are in the clear, given you made the contact with MAXIS in the first place. The anti-piracy and anti-.exe modification rules are mostly in place so that SC4D can continue to host the vital SC4 files that EA took off of their site years ago (despite continuing to sell disk versions of the game, but I digress)

QuoteSo, why I am reverse-engineering SC4? Mostly, to find what it can do, and maybe fix bugs, and make interesting mods.

<snip>

3) The game engine supports much more stuff than we have seen, it has a fully working weather simulator, that is just not used (it calculates things, heavy calculations even, but nothing uses it, thus it waste CPU), many, many, many more dialog types, including pizza charts, a window named "LuaConsole" (I am curious about that one!), and a couple of other interesting tidbits.

5) The game supports more music than it was shipped with, without any modifications (up to 30 tracks per "radio" it seems, meaning you can have 5 new mayor tracks, and lots more terrain tracks).
Both 3 and 5 are known. I think the music modification is extremely simple, and involves editing some config files in the plugins folder and some renaming of the requisite music files, but other than that I think it is straightforward. Given that the track pauses every time you leave the game window, and that the actual music files are in MP3, I usually just open the playlist of the tracks in Windows Media Player, and run them that way, It adds a little bit of overhead, sure, but not too much, and this way the music doesn't restart every time I tab back to the game from looking at NAM manuals.

As for 3.), you should look at Ennedi (actually his MD, here, though the images were hosted on ImageShack and are lost) and Lowkee's terrain threads. Ennedi did extensive testing for his terrain mods, to try and work out the kinks with moisture based snow textures. Apparently wind, rain (ground moisture), and temperature are all calculated, and depending on the assigned parameters in the terrain .ini, will show different textures. Lowkee's Appalachian terrain mod actually includes a parameter to suppress the weather simulation, to keep the terrain more "stable" in appearance and prevent unwanted discrepancies. Unfortunately I think Lowkee's images are mostly lost in the Imageshack reshuffle as well, which is a shame. You'll have to use your imagination, I guess.

Quote6) The game also support threads, in fact it shouldn't crash when multicore is active, probably a bug can be fixed somewhere that make the game work with threads smoothly, I even activated 8 cores on my machine, and the game don't crashed (Because of this at least) on me yet.
Interesting. I usually let my computer sort out the threading (at the risk of extra crashes in exchange for a more even overhead), but I wasn't aware that the game supported this natively, and always figured that I'd have to rely on my computer's native scheduling. If we can somehow activate this it would go a long way to keeping the game alive in the new paradigm of hyper-threading CPUs

Quote9) It supports many more options than the interface let you change, one obvious mod to make eventually is a configuration app to change those.
Well, I first though this was somehow related to Menus and things like the DAMN, but now I realize you are referring to things like disabling shadows, which for some extremely comedic reason we are forced to change in the config files rather than in the game (Sadly, we have no off), but a few minor conveniences are always welcome, even if they are not critically needed.

QuoteAlso the engine was designed to be easily extensible and pluggable, not for modding, but to make things easier to develop.... unfortunately it backfired, made developing hard (Ultima Online 2 team complained about the issues of GZCOM), but we can mod SC4 easily, with enough information we can call any SC4 class, and invoke any function we want by using a DLL, meaning we can do whatever we want without modding the .exe, we don't even need to "hook" and edit the .exe on the fly, the game has proper interfaces to do proper calls.
It's interesting that all of this is available and we don't even need to "hack" it, so to speak; that the functionality is there is amazing, and I wish I knew enough about programming to help. As it is, developments like these are exciting, especially since DLLs offer universal functionality, unlike a modified .EXE.
Experience is something you don't get until just after you need it.

My Mayor Diary San Diego: A Reinterpretation

speeder

About how the engine dll plugin system work:

The engine has almost all classes to be derived from a class named IGZUnknown, that is a copy of Microsoft's IUnknown

IGZUnknown has exactly 3 functions, AddRef, Release, and QueryInterface.

This mean that almost all classes also have those functions, AddRef and Release are used to control memory, a sort of hacked garbage collector, that don't work properly (Paul Pedriana, the inventor of these, talks about this problem in a GDC talk).

What interest us here, is QueryInterface, almost all classes have this function, that is always the first function of the class, what this function do is give a pointer to the class data if you request the class GUID

Similar to the DBPFs TGI system, every single class on SC4 engine have a ID, and if you know the ID, you can invoke them, you don't even need to know where the game stores them, you just "ask around" by using the class ID, the game engine sort stuff out and find it for you.

Then you can use it as you wish.

The Cheats DLL use this: It invokes the ID for the class that store the app configurations, and enable the debug mode by calling the appropriate function.



Random unrelated note: I sent an e-mail to the guy that invented the information windows, he quit Maxis before SC4 was released, but I hope he can at least give some information :)

mgb204

I certainly wish you all the best with this. Like many I suspect, I won't be of any help to you unravelling things at this level. Still, interesting stuff and I'm sure many would welcome the ability to start changing things. You'd be a hero if we could have for example extra menus or put sets of items (non-transit) into tab rings for example. Anyways, I hope your experimenting bears fruit, whatever that may be.  :thumbsup:

InvisiChem

If you are able to gather enough class information, I may be able to help with some things. The idea of possibly calling the classes by ID also leads to the possibility of some old school inheritance in mod development. Of course, I am in the United States, so I have to be very careful on what I do, but I would be more than willing to help in any way I can. Not an expert dll developer, but functional enough with GCC and VS to help out with enough documentation.
Everyone has something to offer, most do not possess the courage to offer it.

speeder

The "class information" part is mostly easy actually, but time consuming.

The Mac build of the game (I downloaded a legal copy of it by getting the "patch" available on SimTropolis http://community.simtropolis.com/maxis_files_for_simcity_4.html/ ) has all debug symbols enabled, and RTTI enabled too.

This means a symple python script can scour the file and make a list of all classes and inheritance, and you can easily make a list of functions in each class.

The problem is that it is the Mac build... Windows build is very similar, but different (example: Mac build use pthreads for threading, windows build use Microsoft's "Critical Section" system, Mac build use GCC STL, Windows use STLPORT... I am not sure why Mac build use GCC STL, since STLPORT work natively on Mac too, and replacing the STL is a pain the butt, also I suspect most of the Mac bugs are due to this replacing).

speeder

Quote from: mgb204 on January 21, 2016, 07:21:17 AM
I certainly wish you all the best with this. Like many I suspect, I won't be of any help to you unravelling things at this level. Still, interesting stuff and I'm sure many would welcome the ability to start changing things. You'd be a hero if we could have for example extra menus or put sets of items (non-transit) into tab rings for example. Anyways, I hope your experimenting bears fruit, whatever that may be.  :thumbsup:

I decided to look into your request.

So, I looked around, and found out what might create the menus, but the menus are complex and I wasn't understanding rubbish, so I decided to see how the game create a simple dialog box, and choose the boundary reconciliation box.

I looked in the code for it, foudn some number there, and looked in the Reader for it... didn't work, but then I decided to do the inverse, look into the reader, and see what I found in the source... to my surprise, not only I found out something, in a unexpected place, I discovered that "TGI" has a official name, and the game DOES use TGI system, just not explicit files, TGI that it uses are mostly coded into the source.

So, the name of TGI is GZPersistResourceKey


Also, the game consider anything that you create with "UI" files on the Reader, to be a window, like a dialog box, even non-windowed shaped stuff... seemly it is a hack of The Sims Window system (when The Sims was being made, Maxis hired Don Hopkins to make the Linux version of it, Don Hopkins then created the "Sim-Antics" system, taht control Sims behaviour in the The Sims series, and also created the Window UI System for the Rizzo-Gonzo engine)

Tarkus

Very interesting thread, here--thanks for sharing your findings, speeder!  On the menu front, I don't know if you've run across this in your research, but another non-US user, GoaSkin, did some poking around back in 2007 and actually did manage to add to the menu system--his old thread for it can be found here.  Sadly, the images are gone, but it's still an intriguing read.

-Alex

epicblunder

Very interesting!

As APSMS said re: (3);
The game does use the weather simulation. Weather simulation outputs a moisture value for each game tile that changes over the course of the year.  That moisture value actively controls what terrain texture each tile has and which props in a tree controller will be planted on that tile. Some of that we can tweak with the Weather Tuning Exemplar, but some of it we can't.  3 years ago or so i would have killed to be able to control the hard-coded side (doing so could possibly double the range of what you could achieve with a tree controller), but now .... eh, who has the time?

There are some other minor things involving it, like ground fog and clouds. Not really sure playing with the weather would be worth the time though.

It will be interesting to see where this leads.  Good luck.   :popcorn:

simmaster07

If you're going to be reverse-engineering the DLL framework, you'll probably find this helpful:
http://github.com/nsgomez/gzcom-dll

This is a project that I started years ago and tried to refine on-and-off since then. It contains some original code samples from Paul Pedriana at EA related to DLLs and DLL loading, as well as an attempt to reverse-engineer the framework in full for the game to load it. As of right now I think the game crashes if you try to load this DLL because the classes' vtables don't match what SC4 is expecting, and it crashes with an "illegal instruction" exception.

NCGAIO

#10
Quote
EASTL is also partially open-sourced, you can find it here: http://gpl.ea.com/ ........


or here - https://github.com/paulhodge/EASTL


old doc - http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2007/n2271.html

speeder

About EASTL: I cited the gpl site, because in it you can find a very early version of EASTL, that is the code closest to SC4 (although still post-SC4, since SC4 used STLPORT, many parts of EASTL were "copy and paste" of STLPORT, at least in design, thus you can read it to understand what SC4 is doing).


Now two news: One, I decided to try to focus only on what is relevant to my hardmode mod, the game is too big, and I ended going in several wild goose chases while finding discoveries (example: days wasted trying to trigger the in-game editor, and yes, the game has a in-game editor, I found the code that assembles a lot editor, a network lot editor, and a Lua debugger, and I suspect it also has a "Reader" too, also more days wasted trying to redo GoaSkin's work and add submenus).

Second news: my mod intention is to modify many simulator exemplars, I decided to start with the traffic simulator, because the "commute time" bug really annoys me, and I found out that the engine reads a couple properties that weren't used in the .dat and aren't documented anywhere (ie: not on ingred.ini, not on XML, and not on the wiki, since noone noticed them before).


This is a incomplete list, but I will add the ones I found so far:

Property 0x29136789, it is a floating point number, don't accept being array.
Property 0xA92356B6, unsigned int 32 number, not array.
Property 0xA92356B7, unsigned int 32 number, not array.

speeder

#12
Nagged Paul Pedriana again.

Unfortunately, the things I really wanted to know, he couldn't help because he can't remember, despite being the one that coded the thing I wanted to know (ie: how the Cheats dll work).

But he did gave some useful info:

QuoteThe lot editor was built into the application, and I think it was accessible via a cheat code. I don't recall if that was enabled in the shipping binary though.

So yes, I am not crazy! Don't figured how to trigger it yet though.

QuoteI don't recall ingred, but I think it meant In Game Resource Editor, and it let you tweak app data.

Now we know what Ingred means! (I thought it was related to "ingredients"). Also we know why the game loads Ingred.ini (because the editor, that is inside the game, needs it).

QuoteIf you want to know about automata and path finding, ask Cisco Lopez-Fresquet. You can find him.

NAM team, go nag that guy ;) (I only want to know about commute time... I don't wanna muck around the pathfinder).


EDIT: Also, forgot to mention, several ex-EA employees are trying to convince EA to release some code they made because they wanna use it.

Paul Pedriana mentioned this in the e-mail to me (he wants to use EASTL, and some stuff derived from Rizzo and Gonzo at his current job), Bob Summerwill also wrote about it on his blog, and in an e-mail to me (he also currently maintains this EASTL partial repository: https://github.com/kitsilanosoftware/EASTL

Some other current EA employees also talked on CPPCon they want to release stuff for their ex-colleagues to use outside EA.

Let's hope it happens, if it do, bright times ahead (not only for us fans, but for all game programmers).

Simcoug

That's cool that Mr. Pedriana was kind enough to get back to you, especially since it's probably been a decade since he's even thought about SC4 :)
I'm impressed by your persistence on this project!  If it were up to me, I'd say a Karma point was in order.

gn_leugim

Quote from: Simcoug on February 02, 2016, 06:55:32 AM
That's cool that Mr. Pedriana was kind enough to get back to you, especially since it's probably been a decade since he's even thought about SC4 :)
I'm impressed by your persistence on this project!  If it were up to me, I'd say a Karma point was in order.

I second this, for the persistance and for the potential that this has! way to go!  :thumbsup:

speeder

Since Simmaster07 sort dropped off the face of the earth again or something, I am trying to use his source code to see if I can fix the problems I have in my own computer (game detecting CPU speed wrong, and detecting my videocard wrong).

I still don't managed to track down the commute time bug :(

romualdillo

Thank you for starting this project and sharing your results! I'm not a modder but I can see the potential it has. Perhaps you could even recover the multistage props, which were planned but never used (yes, it was a not-well-hidden suggestion)  :D.

Please keep it up!  &apls

speeder

Quote from: romualdillo on February 02, 2016, 10:22:55 AM
Thank you for starting this project and sharing your results! I'm not a modder but I can see the potential it has. Perhaps you could even recover the multistage props, which were planned but never used (yes, it was a not-well-hidden suggestion)  :D.

Please keep it up!  &apls

I saw that sunday, while looking for Mr. Pedriana contact, I accidentally found the blog of one of the artists, where he showed the multistage stuff he planned, but didn't made to the code because of memory issues.

It is probably easy to re-enable this, but I doubt it can work decently, mostly because the memory concerns still exist and can't be easily fixed...

The memory concerns are related to the fact that all these features are not automatic, you need to create separate texture for each stage, variation, zoom level and so on... and the game must load them all, the result is you running out of memory rather quick if you overdo it.

Also, I never made a lot, so I dunno how textures are called and so on, I would have to check that...

simmaster07

Quote from: speeder on February 02, 2016, 08:18:52 AM
Since Simmaster07 sort dropped off the face of the earth again or something

Still here, just a lot less active at the moment due to college. SC4Fix was just something I made with the time I had during winter break. During the academic year it'll be a lot more challenging to work on SC4 stuff.

speeder

EA just dumped EASTL out in the open =D

https://github.com/electronicarts/EASTL

\o/

Now makes understanding the game a bit faster.