Simtitude: A SimCity Package Manager

[simmaster07]

There are packages available.
Download

• STEX (0.0.1-beta): http://www.simtropolis.com/stex/details.cfm?id=24171
  
  
  
  • Support and Development Thread: http://www.simtropolis.com/forum/messageview.cfm?catid=42&threadid=115424&enterthread=y

It's happened to a good majority of us at least once: you're downloading a lot when you get links to NDEX and BSC dependencies, which have dependencies associated with them, making you spend a good ten to twenty minutes to download dependencies. Some of us can't even find one of the dependencies we need and we're forced to see that ugly box in SimCity until we go back, check over all the dependencies one-by-one, etc.

There is SC4Tool, which can help you trace a lot's dependencies. There's the LEX Downloader, which downloads lots, BATs and other plugins from your download list. And Cleanitol removes duplicate dependencies. Now, the problem is, there is no tool that can directly download a lot or BAT and all of its associated dependencies and all the dependencies associated with those dependencies without having to go to the LEX, select all of them in your download list, and fire up LEX Downloader. Some dependencies may not even be available on the LEX.

However, I am working on a tool that serves a similar purpose to Ubuntu's apt-get install program. Simtitude is a .NET (C#) application that uses server-side repositories and XML files to track packages associated with a repository. By using PHP for server-side administration and MySQL for tracking package status and information, the C# front-end is able to add the repositories, download the XML metafile, and use elements in the XML metafile to find and download -- you guessed it, both the dependencies and the original package.

The process of download a plugin looks something like this:

• The user searches for a plugin's name and a list of plugins found shows up with checkboxes to select plugins to install
• The user selects plugins by checking the boxes.
• The Web Interface, when the Apply Changes button is selected, installs the selected plugins
• For each plugin checked, the name of the plugin is searched for in all metafiles until a match comes up
• The match has dependency entries in the XML, taking the installer back to step four
• The list of files and dependencies are stored in an array and are all downloaded
• The files are downloaded. The installer assumes we're using the current user's My Documents\SimCity 4\Plugins directory.
• All files are unzipped. Each file's postinstall script is run. This makes plugins such as the NAM feasible for repositories if the postinstall script removes sub-plugins not selected by the user (such as Left Hand Drive). The postinstall script must be an executable (.exe)

Example metadata XML:

Code Select Expand


<repo>
<metadata>
<title>Simtitude Main</title>
<author>simmaster07</author>
<description>The official Simtitude repository</description>
<hash>[a really long and random hash]</hash>
                <baseurl>http://whatisthisdomain.com/Simtitude/repofiles/blah/fish/</baseurl>
</metadata>
<packages>
<plugin-name>
<title>My Plugin</title>
<version>1</version>
<desc>Just an example</desc>
<size>5</size>
<preinst />
<postinst />
<serv-filename>plugin.zip</path>
</plugin-name>
</packages>
</repo>


Screenshots (command-line):

[thingfishs]

While I don't fully understand all the implications of such an approach, it certainly is an interesting idea. And no doubt there are hoards of villagers waiting to hoist you over there heads should you succeed. Personally I already have a near complete collection of BSC stuff and rarely have to worry about dependencies, but mistakes happen.
I'm keen to see what those more knowledgeable about all this have to say and, whether it can realistically be achieved or not, kudos to you for giving it a go.

[simmaster07]

While I don't fully understand all the implications of such an approach, it certainly is an interesting idea. And no doubt there are hoards of villagers waiting to hoist you over there heads should you succeed. Personally I already have a near complete collection of BSC stuff and rarely have to worry about dependencies, but mistakes happen.
I'm keen to see what those more knowledgeable about all this have to say and, whether it can realistically be achieved or not, kudos to you for giving it a go.

Well, it would require either the BSC, NDEX, any other major teams, and independent developers to make or access repositories and upload to them, or, though extremely unlikely, switch over the license to allow redistribution. Like that's ever going to happen.

And thanks for the feedback!

[callagrafx]

Couple of things...

1) How are you going to create the database of pre-existing content and their related dependencies?  Are you expecting content creators to do it for you? They won't.

2) What happens when a dependency file is not found?  Some older works have dependencies that have been superceded.

While it's a nice idea in theory and you are obviously a very talented young man, I don't think you've thought through the enormity of the task....There are over 14,000 files on the STEX, 2000+ files on LEX plus all the other exchanges out there.  It simply isn't feasible. 

[simmaster07]

1) How are you going to create the database of pre-existing content and their related dependencies?  Are you expecting content creators to do it for you? They won't.

The PHP backend is simple enough to integrate directly into the LEX, PLEX, and SimCityKurier exchanges, so pre-existing files are taken care of. Dependencies are a different case, though. No, not all authors are going to care to update their plugins to list the dependencies, but those with not many uploads probably will. This could also be a solution that would apply to new uploads.

2) What happens when a dependency file is not found?  Some older works have dependencies that have been superceded.

It's planned to show a dialog asking if you're sure you have the dependencies pre-installed. Yes, it continues. No, it skips the files in queue associated with the missing dependencies.

While it's a nice idea in theory and you are obviously a very talented young man, I don't think you've thought through the enormity of the task....There are over 14,000 files on the STEX, 2000+ files on LEX plus all the other exchanges out there.  It simply isn't feasible. 

There are well over 3,000 packages on other repositories for Linux files (Ubuntu multiverse, as an example), some over 150 MB like the Linux kernel and GNU C Compiler data files, but the APT and Yum repo management tools manage to cycle through those repositories. I can't code in ColdFusion either, so the STEX isn't possible.

What if we looked at a different solution where instead of parsing dependencies, we parsed HTML? Using regular expressions to find dependencies and download links for different exchanges, we can have one program for the STEX, LEX, PLEX, and Kurier. Filtering through the search results and dependencies with regex should be more feasible, since it wouldn't involve massive metafiles. (and yes, it has been done before)

[BarbyW]

One of the major problems of aproject like this can be seen if you download, to an empty plugins folder, from the official EA SC4 exchange. This searches for and installs missing dependencies but the problem is that many of the dependencies are out of date. This can result in multiple instances of props if care is not taken to keep things organised.
The system that is in use on the LEX where dependencies are linked in Readme files is more than adequate for the majority of users.

[callagrafx]

But you are still going to have to MANUALLY go through every file and make a note of its dependencies to add to your database, unless you are planning to read files inside ZIPs...and not every file has a HTML readme with current URL links to dependency files...some readmes will be horrendously out of date, others will follow dead links (all BSC stuff was pulled from STEX a couple of years ago, so not all dependency links are still valid).  Lastly, because of the sheer volume of files at ST and the fact they still use Cold Fusion (which is the biggest programming abomination on the planet) meaning you cannot integrate means the vast majority of content will not be catalogued, making this tool only partially functional.

This could also be a solution that would apply to new uploads.

But as a lot of people use pre-existing releases as dependencies (Prop packs etc) then how can it find it and add it to the database?

In all honesty I really don't think the dependency issue is as bad as you make out, as we've survived and thrived quite well for the last seven years without it.

[Andreas]

I concur with what has been said above. While your proposal is sound, you're not the first one who suggested a Linux-style repository system with automatic dependency detection and retrieval. As stated above, the main problem is that there are so many files out there, and a large portion of them is either poorly documented (because the uploader was too lazy or not knowledgeable enough) or outdated due to superseeded files. On top of that, there are about a dozen exchanges/download sites spread all over the world (STEX, LEX, PLEX, SimCityKurier, CSC, TSC, SimCityPolska, to name a few), some of them open to the public, others maintained by the website admins.

As an example, SimCityKurier is a closed site, and I'm the only person in charge for uploading new stuff, so every single upload is in the same format, and wherever possible, I'll also upload the needed dependency files to this site as well. Each file description and the readme file include download links to the dependencies, so it's a matter of a few seconds to retrieve those files. I'd assume it would be possible to create a script that collects the links and creates a repository list for your proposed system, so implementing it wouldn't be much of a problem.

But then again, why would one need such a system if all links are posted at the proper place already, so people just need to look at their plugins folder in order to see if they already have the file? If they used the default installer paths, locating them on their HD should be easy. Problems only arise if the user's plugins are not maintained well enough, if they simply dump the files into it from a zip file, and don't read the readme file, if there is one. But how would you approach such messy files and add those to your repository? Some custom content creators are gone since years, so the only way to add those would be to inspect them manually, one by one. That's truly a Sisyphean challenge...

[simmaster07]

Now, let's keep in mind that this is a proposal for such a solution, and it is only a solution. It's not something as big as an HIV vaccine, and is intended to expand gradually, and not be done all at once. Going through superseded and old dependencies would, indeed, be truly Sisyphean, but taking this one step at a time, it's not as complicated and confusing as it may appear now.

Starting off with the dependencies themselves (prop packs), we can then expand on to lots with one or two dependencies, to multiple dependencies, etc. Again, this isn't intended to be taken in all at once. Now, of course a well-maintained plugin folder wouldn't lead to this type of problem, but not everyone will take the time to organize their plugins folder, making it harder to search the plugins folder one by one.

Also, while there's absolutely nothing wrong with keeping your plugins folder organized, that approach has been used since SC4 dependencies themselves. We have a multitude of advanced programming languages and techniques to accomplish the goal of this type of project, so why not embrace the technology available to us and try making everything a little more user-friendly for old and new members alike? The project may seem unfeasible, but it can be done if the community does chip in, since that's the whole purpose of a community.

[callagrafx]

What we are trying to say is it's a colossal task that's a bit of a waste of time.  Don't you think if there really was any easier way to track dependencies, we would already have it?  It would be fine if everyone sang off the same hymnsheet, but we don't....

[xxdita]

Don't you think if there really was any easier way to track dependencies, we would already have it? 

We already have Wouanagaine's Cleanitol program, which is highly used here, to help gather dependencies as needed, from whatever site they may already be on.

Cleanitol can also be used to parse through the end user's plugin folder to check for outdated items that will be replaced with a Prop Pack, or items that will conflict, or whatever. It's just a matter of someone making the proper Cleanitol txt files to get the job done. The link in my signature is to a board where I've worked on several Cleanitols for the CAM Starter packs, among other things, to make the dependency search easier for anyone new to CAM. I've found it quite useful when rebuilding my plugins recently as well.

[simmaster07]

As skeptical as most of you were about the concept, I've released a command-line version of Simtitude on the STEX with about 80 packages available as of now and still expanding.

(link)

[thingfishs]

good on you for keeping on with it. It may not be a cure-all solution but it's got its place.

I am just always encouraged by people trying new things.