Allow me to begin with:
A ) Request:
QuoteThis can readily be a hot-button topic within the Sim City community, so I must reiterate that this is a public forum. As such, the discussion needs to remain civil - in spite of how strongly one may be entrenched with either extreme.
B ) Recognition:
QuoteThis discussion is in no way intended to be regarded as a slight against those who so readily offer up their hard work to the Sim City community - be it as LOTs, MODs, BATs, or whatever incarnation their contribution takes. Mere words cannot fully express my respect and gratitude regarding the skill, artistry, and generosity of contributors.
And now, the discussion:
Will Wright's innocent little game,
Sim City, has grown light-years beyond all possible expectations in the quarter century of its existence and turned an unknown fledgling game developer, Maxis, into a legend of the gaming industry. The game succeeded for the very reason it almost never was - there simply
is no other game like it! Numerous communities, with countless members spanning the globe, remain glued to their monitors in support of a game that is older than many of them are - the newest incarnation is approaching a decade in age and is likely the oldest thing most of us own. But until Maxis/EA releases a worthy successor,
Sim City 4 will remain in our hearts the de-facto standard of what a Sim game should be.
Until a true
Sim City 5 comes to pass, we have a plethora of options in expanding the bounds of
Sim City 4 - far too many to even attempt to list. We, as a community, indeed owe a great debt of gratitude to those who so generously invest their time and efforts developing both the expansions themselves and the tools to create them.
But herein lies the rub:
With so many expansion possibilities, a great deal of care must be taken on all fronts to:
- Avoid internal conflicts (ie - duplicated resource addresses)
- Minimize overhead
- Eliminate redundancies
- Simplify accessibility for the less-technically-inclined
in addition to maintaining the communal spirit which has allowed
Sim City 4 to thrive as it has. Yet, in some ways, we're victims of our own successes.
Countless Mega-Packs of fabulous textures and intricate props enable individuals to create fantastic new content for the game - but only if the users already have these Mega-Packs. If not, the user needs to find and acquire these dependencies - in some cases adding more than 20MB (a sizeable load for SC4's internal coding) in order to properly implement a 65KB lot. For the average user, even one with a considerable amount of custom content, it's fair to say that far less than 30% of any particular Mega-Pack dependency ever gets utilized.
The simplest solution would be that content creations being offered include these dependencies - though this leads to other problems. Increasing the likelihood of internal conflicts and redundancies, in addition to a lack of proper credit where it is due.
Given the availability of open-source installers and utilities (many of which have their own dependency headaches), it shouldn't be that difficult for us to develop a one-size-fits-all distribution installer/manager.
Such would actually consist of two programs: the developer utility and the user utility, each of which would be available as readily-locatable separate downloads:
The Developer Utility:This program would organize all required assets (props, textures, models, etc) from their respective sources and compress it all into a single, compact file for distribution. In doing so, it would preserve the hierarchy of those sources.
It would also distinguish between unique assets (those exclusive to this particular expansion) and common assets (ie - those distributed in Mega-Packs and reused by multiple expansions).
The User Utility:This program would read the distribution file produced by the Developer Utility to reconstruct the source files, complete with reproducing the source hierarchy. In doing so, it would check each asset in turn and, if absent, install it - if a particular asset already exists, it goes to the next. This functionality should also include an over-write option (just in case).
The User Utility would:
| ⚫ | Install the unique assets in the appropriate location |
| ⚫ | Notify the user that it uses a given dependency - giving the creator(s) proper credit |
| • | Check to see if the given dependency exists |
| ⚬ | If not, it creates the file with all necessary assets |
| ⚬ | If so, it checks each asset in turn - adding those not present (or overwriting, if enabled) |
| ⚫ | Repeat for each additional dependency |
While probably not the
best potential solution, it seems to cover all the bases:
- Eliminates having to hunt down and manually install multiple dependencies
- Eliminates cluttering things up with extraneous assets (minimizing overhead)
- Polices itself in regards to installing assets (eliminating duplications)
- Streamlines the distribution of custom content
- Simplifies things enormously for the user, making individual expansions far more appealing
If I were more adept with programming (as well as SC4's inner workings), I'd take a crack at this myself. Though I'm sure there are dozens (if not hundreds) in this forum who could write these utilities in less time than I've spent describing them - preferably as self-sufficient programs
without dependencies of their own.
Your topic of discussion: Streamlining the process of downloading plugins.
QuoteThe Developer Utility:
This program would organize all required assets (props, textures, models, etc) from their respective sources and compress it all into a single, compact file for distribution. In doing so, it would preserve the hierarchy of those sources.
Kinda sounds like the DatPacker. Nifty for making prop packages or texture packages, for those wannabe or seasoned developers out there. Or to just cram all the junk in your Plugins folder into one single compact file so that it doesn't make a mess out of things.
Oftentimes, there's a readme that comes with these dependency packages, which includes things such as the developers in question, details on the items, and a link to a forums thread to report problems to or to suggest new things. The readme says, "READ ME!!!" (Why else is it called a readme? :D )
QuoteThe User Utility:
This program would read the distribution file produced by the Developer Utility to reconstruct the source files, complete with reproducing the source hierarchy. In doing so, it would check each asset in turn and, if absent, install it - if a particular asset already exists, it goes to the next. This functionality should also include an over-write option (just in case).
Sounds like the Reader. Many of the plugins made for Simcity 4 (And many of the files that originally came with the game) are packaged into single files: DAT files. The Reader's able to read an entire DAT and organise it into the individual files that make it up. Plus you can edit the values set in them. (Actually it doesn't matter how they're organised within the DAT; The game can tell what's a texture, what's an exemplar, and what's a prop and so on; Organisation is for when you're looking THROUGH the DAT's contents.)
If you're DatPacking a number of plugins, their readmes (and all other irrelevant files) will be excluded, and the resulting DAT is placed in a "plugins_compressed" folder in the plugins folder (if you didn't specify another location). Some plugins (like the NAM) cannot be DatPacked because it can screw up any update processes. Others don't need any packing, as they're small enough already or deemed unnecessary.
---
Many people would balk at the sheer number of dependencies needed for a specific download. Problem is that in some cases, a certain plugin (Let's say it's a landmark) made by, for example purposes, myself, may use props made by one person, and textures made by another, and models made by yet another person. Two ways to simplify the download process:
- The unrecommended way: Copy all three things needed, pack it, and submit it on the file exchange in question, and face potential termination.
- The "Coalitionist" way: The three people and myself may unite into a group, unify their respective models, props, textures, et cetera, into a combined DAT and submit the Dependency packages. That way, if I (or another in that group) make more lots that require their props, people won't be burdened by downloading things they already have or have to download again. Dependency packages are one-time downloads, unless it needs to be updated; Everything you need is already available for everything that group makes. I guess that best describes what goes down for BSC and SFBT.
Doesn't work best when there are solo developers that like to use "deps" made by two different groups, unless you already have the dep packages made by the two groups. I see what you mean there.
There's the CAM for example: It adds more growth stages to the game, but has no "CAMeLOTs" except for the ones derived from the game's existing buildings; You'll have to go fish for them yourself. There's a lot of them anyway, but there exists several "starter packs" and yes, they have their own dependencies. Chances are, you will have already downloaded many or all of the prop and texture dependencies, leaving only the model (The actual building) to be downloaded. It's just as intense as downloading individual CAMeLOTs, only there are readmes you could use as checklists. (That was an absolute nightmare for me to do... It's just that I have incredible determination.)
There's one more thing:
There's a SC4 custom-created tool, Cleanitol, that can identify what files you have and don't have, and can also delete them if you're replacing them with an updated one. (I never tried that tool before; I just run on full faith that I have all the plugins I need, until I get brown boxes in the game, then I just read the readme to determine what's missing. Plus, I just use the Search feature in Windows Explorer as a close-enough equivalent.)
Also, I've never seen a tool that required a dependency, except for the BAT (It requires GMAX) and the Lot Editor (I think it's more of a patch).
Actually, I don't see anything wrong with the current system; You just gotta read the readmes and use some special tools. And a REALLY sharp eye.
-----
I actually want to compare this to the Network Addon Mod, just because I'm more familiar with it.
Does the NAM require a dependency? No, unless you don't have Simcity 4 Deluxe or Rush Hour. Hole-digging lots and slope mods are more like accessory mods; Technically not required, but highly recommended.
Is the NAM a dependency? Yes; Many NAM plugins (RHW, NWM, HSR, SAM, Rural Roads) require it to work properly, and special TE lots (GLR stations, for example) also require it. (The NAM Retexture and Cosmetic Mod only modifies some of the game's original textures; It can technically work without the NAM.)
Who makes the NAM? A large group of people, and a number of others who test it. (What else would all those names in the NAM readmes be for?) Many people make special and different features for the NAM (Superhand's NAM RCM for example; It's pretty much him who made the entire RCM); Think of it as the individual prop/model/texture makers in my aforementioned coalition example. It's essentially specialisation and division of labour that occurs here, and I would bet it also happens in all the other SC4 groups here.
Actually, only the NAM has ever had such a distributor made for it: You tell it what plugins you want, and it creates the entire NAM folder containing what you requested. But what happens if the NAM creates new content, like with the currently in-development (and currently stalled) RHW v4.2? Yes, the distributor will then be out of date, until it obtains the updated files.
Now that I bring it up, a sort of "Omni-Distributor" would have to keep up with hundreds of files being updated, and who would ever maintain such a distributor anyway? At least, in theory, it could be applied in small-scale. Proof of concept: The NAM distributor.Another idea that just struck me. Have you ever been on an online shopping site and the things you ordered are in a "cart"? Perhaps if the STEX had such a feature, it would make downloading easier. The downside is when your "cart" gets "full". But even with the "cart" size limit being X amount of megabytes, what's three large downloads compared to 30 smaller ones? At least then, the ability to download individual plugins would be an option, as is the cart option.
To hit some of the highlights in response:
You specifically listed DatPacker and iLive's Reader, which are indeed wonderful tools - if you have the technical background to understand their functioning. Not everyone does. And let's face it... Reader is an incredibly powerful tool - and one innocent change could inadvertently trickle down like Apollo 13. v1.4 is an immense improvement over its predecessor, but is less-readily obtainable - I forget the reasoning (I think it has to do with the filesize) as to why it's not available through the usual channels - and it's far too easy to mess something up in v0.9.
As for the "Read Me" files... I consider myself fairly tech-savvy and some of them have me wondering if they were written by Machiavelli. That's more on the individual creators than 'the system', as that degree of technicality is more representative of bad technique. And, as you said yourself,
you have to keep a sharp eye out - and you're far more tech-savvy than the average player.
As for Cleanitol, I can't comment one way or the other - most every version I've found in the repositories has been locked for one reason or another. As with the numerous dependencies, some of the comments I've seen in the locked versions have dissuaded me from investigating this program further.
But the source availability of these and other programs is why I feel a unified distribution method is a reasonable 'next step'. In some form or another, most all the work has already been done - it just needs someone properly knowledgeable to put it together.
Quote from: GDO29Anagram on May 08, 2011, 06:27:48 PM
Many people would balk at the sheer number of dependencies needed for a specific download. Problem is that in some cases, a certain plugin (Let's say it's a landmark) made by, for example purposes, myself, may use props made by one person, and textures made by another, and models made by yet another person. Two ways to simplify the download process:
- The unrecommended way: Copy all three things needed, pack it, and submit it on the file exchange in question, and face potential termination.
- The "Coalitionist" way: The three people and myself may unite into a group, unify their respective models, props, textures, et cetera, into a combined DAT and submit the Dependency packages. That way, if I (or another in that group) make more lots that require their props, people won't be burdened by downloading things they already have or have to download again. Dependency packages are one-time downloads, unless it needs to be updated; Everything you need is already available for everything that group makes. I guess that best describes what goes down for BSC and SFBT.
Doesn't work best when there are solo developers that like to use "deps" made by two different groups, unless you already have the dep packages made by the two groups. I see what you mean there.
Which is but one facet to the problem.
I may be misunderstanding things slightly but, to me, the numerous assets available are being offered under a GPL-like license - the example you offered being akin to a 'derivative work'.
So long as you make it clear that a given expansion includes external components - as a common example, textures made by BSC - and convey proper credit to their sources, there shouldn't be any problem with including them in the distribution itself. With the utilities I'm proposing, such recognition would inherently be encoded into the distribution (similar to the common .CAB file) and displayed to the user when installed - they cover these bases themselves.
Quote from: GDO29Anagram on May 08, 2011, 06:27:48 PM
Also, I've never seen a tool that required a dependency, except for the BAT (It requires GMAX) and the Lot Editor (I think it's more of a patch).
I've encountered quite a few which require various MS Runtimes (VB & VC being the most common), along with ones demanding
specific versions of .NET.
Quote from: GDO29Anagram on May 08, 2011, 06:27:48 PM
I actually want to compare this to the Network Addon Mod, just because I'm more familiar with it.
Personally, I categorize NAM itself more akin to Rush Hour - it's a fundamental improvement/expansion to the game itself (as are most traffic/pathfinding improvements). Most of the NAM's core is stuff that should have been in SC4 to begin with.
Yes, additional NAM components (RHW, NWM, SAM, etc) require the NAM Core but, again, more comparable to Rush Hour requiring SC4. Not to mention that they're specifically 'marketed' as add-ons to NAM.
Quote from: GDO29Anagram on May 08, 2011, 06:27:48 PM
Actually, only the NAM has ever had such a distributor made for it: You tell it what plugins you want, and it creates the entire NAM folder containing what you requested. But what happens if the NAM creates new content, like with the currently in-development (and currently stalled) RHW v4.2? Yes, the distributor will then be out of date, until it obtains the updated files.
Now that I bring it up, a sort of "Omni-Distributor" would have to keep up with hundreds of files being updated, and who would ever maintain such a distributor anyway? At least, in theory, it could be applied in small-scale. Proof of concept: The NAM distributor.
What I'm proposing is similar, yes, but differs in several significant ways:
- It would be offered as a freely-distributed tool (vs 'in-house'), with both developers and users having full access to the utilities themselves (even if the sources are kept proprietary)
- It wouldn't be as 'compartmentalized' as NAM's installer - checkboxes would only be offered on a distribution-basis.
- Rather than being an end-all 'catalog', it installs whatever distributions (created by the Developer Utility) it finds. ie - Each .DST (or whatever extension used) found has a single checkmark (see below)
Simply as an example (using random offerings from the LEX):
(https://www.sc4devotion.com/forums/proxy.php?request=http%3A%2F%2Fi134.photobucket.com%2Falbums%2Fq103%2FTwyla_Naythias%2Finstaller_demo.jpg&hash=144499ead4edebe9e3b81d9a7fda072de4113de3)
As you can see, one could download whatever LOTs/MODs/BATs they desired and install them all at once
without having to worry about dependencies.
And since the User Utility can throw a pop-up or other notice when a dependency is encountered, the creator(s) of that dependency still receives full and proper credit for their work.
Seems a very equitable solution to me.
Making it like the package manager in many Linux distros would simplify things. You can for starters create a command-line tool and then allow people to build GUIs on top of it, all open-source of course.
Quote from: riiga on May 10, 2011, 11:26:59 AMMaking it like the package manager in many Linux distros would simplify things. You can for starters create a command-line tool and then allow people to build GUIs on top of it, all open-source of course.
A true package manager would be nice, but that would also involve creating a catalog/index of every add-on and plugin available (and
maintaining it indefinitely). While, yes, this would be the ideal, this would have an inherent tendency to 'squeeze out the little guys' and lend itself to favoritism. Not implying the bias would be intentional (though it possibly could be), but it's simply the nature of the beast.
Does an actual licensing arrangement for custom content even exist? So far as I know, it's mainly an issue of common courtesy (along with site-specific ToUs regarding uploads).
i came up with a somewhat similar idea here, http://sc4devotion.com/forums/index.php?topic=13126.0
i dont see how it could squeeze the little guys out, make it so they system would be a snap to register, were if your on the LEX you can register easily and get a unique ID for your dependency.
if an auto system could be developed that would be great, auto install, auto update, and easy registration for new content.
Heh, isn't it great to have such optimism :D
I'm currently looking at doing something similar, but rather than changing the current system, I'm looking into ways of wrapping the current exchanges with api's so that developers can query them easily, and then building simple, UNIXy tools that can be combined easily to do stuff like automatic dependency gathering, reporting duplicate TGI's (similar to what DATpacker already does, but without the packing, so it can be used with other tools)
RE a package manager, my ideas are what you'd initially need to do, since the Exchange databases already contain much data, they dont contain dependencies though, so that would have to be user stipulated/contributed#
The 'include dependencies in everything' approach is non-viable, the creators stipulate non-GPL licenses usually, the click-through license on BSC (and other) installers isnt GPL, and PEG has a weird, very restrictive EULA/license. I think it's also true that SimCity/Maxis's EULA also stipulates some conditions on custom content, though I dont know. I think the particulars of the most common license is 'personal modification is allowed, but you cannot distribute without explicit permission from the creator, in any form'/
The Tool dependencies are things that cannot be helped by these kinds of systems because the libraries they use are usually proprietary/closed source/licensed in such a way as to disallow distribution though inclusion.
The current system is used and understood by many, most of which are set in their ways (somewhat unfortunately).
Good ideas, if a little unwieldy. I will be attempting to attempt my ideas when I'm free from exams, if I remember by then. I'm willing to hear and to try other's ideas, if any of mine get off the ground...
Joe
Quote from: superchad on May 17, 2011, 11:02:35 AMi dont see how it could squeeze the little guys out, make it so they system would be a snap to register, were if your on the LEX you can register easily and get a unique ID for your dependency.
See, here's the problem...
Trying to maintain a complete database cross-linking dependencies is an onerous task - even for a single group. Trying to do so for multiple groups, plus all the 'independents', gets deep into the realm of diminishing returns - exponentially increasing work for exponentially decreasing returns. Not to mention the fact that the hoops a small-time contributor would need to leap through in order to submit would be discouraging. In short - you're creating a Union.
I hate Unions.
Quote from: JoeST on May 17, 2011, 11:48:43 AMThe current system is used and understood by many, most of which are set in their ways (somewhat unfortunately).
And that's hitting the nail between the eyes.
The .DST approach works fully within the existing system - it's an additional tool vs the replacement a package manager would be.
Maybe I'm just too old-school, being from the days of ye olde 640K barrier - long before bloatware. I simply do not see the return of installing thousands of additional resources (every one of which puts a strain on SC4's old-school coding) when an offering utilizes less than a score of them. I've seen offerings (here and elsewhere) with a dozen (or more) resource dependencies - that's a
LOT of dead weight. I took one of these apart to discover that the only thing the 'creator' contributed was the 'recipe' - NONE of the actual resources used were theirs.
Admittedly, that's more on the LOT creators than those who provide Mega Packs - but the point remains.
As to some of the 'restrictive EULAs' you mentioned... Rather than start a war, lemme use the analogy of why I hate Pepsi.
It's not because of the product (I actually like a good ol' Pepsi Throwback once in a while), but because of their 'marketing strategy':
1 ) They buy up popular fast-food chains and force them to serve/sell their products exclusively vs allowing them to follow customer preference
2 ) They offer at-cost distributing to retailers willing to not carry
any competing products (yes, this is FACT)
3 ) They use these 'forced purchases' to artificially inflate their sales statistics - using half-truths vs actual consumer preferences
The whole reason the SC4 community has grown as large as it has is because of the openness of it. A lot of the offerings of additional and improved content exist
because of the availability of Mega Packs.
But, again, there's only so much baggage SC4 can handle. And only so much benefit utilities like DAT packers can offer.
There is also one additional aspect which needs to be taken into consideration:
Let's say Person-X (who could as easily be a small group as an individual) has a fair amount of resources available - props, textures, models, etc. Good stuff. Stuff that a lot of people make use of.
And, for some reason, they decide to pull their content... Maybe they got in an argument with someone here, or found a different SC4 site, or just got their panties in a wad for no particular reason, etc.
Now what?
If you don't already have these resources, where do you get them?
Even without this, some dependencies are pretty hard to track down - some have been around a long while, and the sheer volume of offerings frequently makes any given file a needle in a haystack. Some creators automatically assume that everyone already has them (anyone worth speaking of), etc.
Not to mention a number of 'legacy' dependencies... Their creators have moved on - making their inclusion in a Linux-type distribution manager a tentative affair, at best.
Believe me... I've studied just about every angle out there - SNAFUs, FUBARs, and other red tape included. As much as I would truly
LOVE a full-bore package manager, it simply isn't a feasible solution - partially owing to the huge investment required to establish/maintain an exhaustive database (as well as certain offerings being locked/removed for whatever reasons), but mainly owing to the limitations of SC4's internal coding.
I don't profess to be a person of many talents, though one of my modest 'claims to fame' is a bent for combining seemingly contradictory ideas into a cohesive whole in such a manner as it seems inevitable.
Not saying that there aren't other solutions, or better ones, but none that I've seen or heard that I believe to be more compatible with the existing situation.
I've ummed and ahhed about responding to this topic. It comes up every few months, people have great ideas, but no one is willing or able to actually do anything about it.
As I understand it, you want someone to develop a program that everyone will upload their dependencies into. This program will break each part of the mega pack into individual models and textures and ID them.
Then creators will add a dependency file to their upload which will link to this program/depository and download only the necessary ones, and only if you don't have it?
I can foresee a large number of issues....
- There are already somewhere in the vicinity of 860 dependency packs out there. I know cos I prepared the list of them. This was up to December 09, so there are more now obviously. Many of these are already compiled. I don't know if you have ever tried to pull one model out of a compiled pack, but it's not easy. Textures are easier, but only marginally.
- Many of the creators of these packs are no longer active in the community. Who takes responsibility for their work? Or do we ignore it and start from now? While there are many great artists still working out there I doubt we could NOT include SG's stuff for example. Now, BSC can manage anything by one of their creators, but solo artists or smaller groups could struggle.
- I took from your opening post that you can't write this package yourself, so would be requiring someone else to write this. There aren't many people in this community I know of that could write such a programme, and non that I can think of that are active.
- There are 3 major exchanges plus a number of minor ones in the community. Relations between members of all these sites are always on great terms. Who hosts this? What if someone is banned from one of the other sites?
While I appreciate that the current system isn't great I don't believe it's as bad as you make it out. Most responsible lotters use common packs, such as the BSC Mega Packs, and they try to minimise the number they reference by looking for alternatives. Not only that, they only time you'll get a brown box is if you don't have the building model. Missing props just don't show up so the lot is a little bare. If it's too bare, you need to find the dependencies.
I created the dependency list because I was sick of not knowing what I had or where it was. Now I check off what I have downloaded and then I don't have to redownload it. It also links to the latest version of each pack so you know you are getting the latest version.
in my idea the program has no database, there is a server database, and a small database of needed files that the downloaded content needs, the database on the server is a mere reference to files on the LEX, the database for the user tells the server what is needed based on ID numbers
Adding a separate manifest-like file to every item on the LEX listing anything it uses is as big a job as maintaining a distinct database of links. Especially since the updating would have to be done by the owners of the items (or the LEX staff), rather than just anyone, which stops them from having time to do other stuff, and all for a system that is new/untested.
A hybrid approach could be a third party make these manifest files for each file on the LEX/STEX/etc... but that's just the same as a database in a way....
Actually, the Hybrid might work: given an id of a lot (such as STEX-2780 (http://www.simtropolis.com/forum/files/file/2780-dusktroopers-keaton-plaza/) or LEX-443 (http://sc4devotion.com/csxlex/lex_filedesc.php?lotGET=443)) we can make a file:
{
"name":"Keaton Plaza 1",
"author":"DuskTrooper",
"description":"a skyscraper blahhhh<h1>OMG THIS IS AWESOME</h1>",
"dependencies":[
"LEX-443",
"http://sc4devotion.com/csxlex/lex_filedesc.php?lotGET=583"
]
}
which is actually kind of what I would have had the API display for each item, if I could work the dependencies database in somewhere behind the API. Maybe having a site with a collection of these in an editable form, so that anyone can come and edit the dependencies at least, maybe with a thing that makes sure someone 'in charge' can check over edited ones, or something.
QuoteMaybe I'm just too old-school, being from the days of ye olde 640K barrier - long before bloatware. I simply do not see the return of installing thousands of additional resources (every one of which puts a strain on SC4's old-school coding) when an offering utilizes less than a score of them. I've seen offerings (here and elsewhere) with a dozen (or more) resource dependencies - that's a LOT of dead weight. I took one of these apart to discover that the only thing the 'creator' contributed was the 'recipe' - NONE of the actual resources used were theirs.
This might be solved by having some smart software similar to datpacker, that analyses LOT's and things, and strips out anything it finds that isn't used by anything else.
The people who stop us indexing their contributions, or those who have left (unless they've said it is ok to do so) should not be included. That is their choice,
Sorry I forgot to make this clear, I wasn't intending to build software that downloaded items from exchanges, merely create tools to make dependency tracking simpler. All my tool might do is provide a link to where to find the items it knows you need. That's the power of UNIX philosophy: simple tools, easy interfaces. It would mean that I could interface this tool easily with another one that could download things from the LEX (similar to Wou's download tool), or check to see if you already have it installed, or some other yet to be thought of application. The same goes for a duplicate detector, or an unused detector. Simple, extensible, awesome!
Joe
EDIT: Indeed Diggis' dependency list is a fantastic resource, what I am attempting/suggesting/etc. is merely an extension to this and stuff. Thank you diggis, you're awesome :D
Quote from: JoeST on May 17, 2011, 02:22:31 PM
EDIT: Indeed Diggis' dependency list is a fantastic resource, what I am attempting/suggesting/etc. is merely an extension to this and stuff. Thank you diggis, you're awesome :D
The problem with the dependency list is it relies on someone (read: Me :'( )to update it. I tend to get a bug every maybe year to do that. I can't see an automated way to do it without having it linked to the Exchanges.
Quote from: superchad on May 17, 2011, 02:19:31 PMin my idea the program has no database, there is a server database, and a small database of needed files that the downloaded content needs, the database on the server is a mere reference to files on the LEX, the database for the user tells the server what is needed based on ID numbers
Umm... that's a lot of databases for a program that doesn't need a database...
Quote from: Diggis on May 17, 2011, 02:17:40 PMAs I understand it, you want someone to develop a program that everyone will upload their dependencies into. This program will break each part of the mega pack into individual models and textures and ID them. Then creators will add a dependency file to their upload which will link to this program/depository and download only the necessary ones, and only if you don't have it?
Nope.
The utilities I'm proposing are more akin to the way a typical archive manager works, treating the dependency files (by name) more-or-less as folders:
~ The Developer Utility would build a compressed 'archive' which includes all assets utilized by the offering - pretty much the same as it is on their system
~ The User Utility would rebuild that structure of the user's system - adding if it doesn't exist, with the option to over-write (for updates, etc)
Quote from: Diggis on May 17, 2011, 02:17:40 PMI can foresee a large number of issues....
- There are already somewhere in the vicinity of 860 dependency packs out there. I know cos I prepared the list of them. This was up to December 09, so there are more now obviously. Many of these are already compiled. I don't know if you have ever tried to pull one model out of a compiled pack, but it's not easy. Textures are easier, but only marginally.
- Many of the creators of these packs are no longer active in the community. Who takes responsibility for their work? Or do we ignore it and start from now? While there are many great artists still working out there I doubt we could NOT include SG's stuff for example. Now, BSC can manage anything by one of their creators, but solo artists or smaller groups could struggle.
- I took from your opening post that you can't write this package yourself, so would be requiring someone else to write this. There aren't many people in this community I know of that could write such a programme, and non that I can think of that are active.
- There are 3 major exchanges plus a number of minor ones in the community. Relations between members of all these sites are always on great terms. Who hosts this? What if someone is banned from one of the other sites?
As you've pointed out - the voice of experience - even just trying to maintain a simple listing of dependencies is a monumental task; comprehensively cross-referencing them (as a full-blown distribution manager would need to), not to mention keeping them up-to-date, would be a full-time task for multiple people.
What I'm proposing bypasses that mess entirely - by simply packaging the individual dependencies with the offering.
And it isn't that I
CAN'T write it myself... I just don't have a great deal of experience tinkering with SC4's inner workings - and experience counts! Someone already knowledgeable could likely churn out the entire program in less time than it would take me to track down and establish the necessary data fields.
Quote from: Twyla on May 17, 2011, 03:30:37 PM
The utilities I'm proposing are more akin to the way a typical archive manager works, treating the dependency files (by name) more-or-less as folders:
~ The Developer Utility would build a compressed 'archive' which includes all assets utilized by the offering - pretty much the same as it is on their system
~ The User Utility would rebuild that structure of the user's system - adding if it doesn't exist, with the option to over-write (for updates, etc)
As you've pointed out - the voice of experience - even just trying to maintain a simple listing of dependencies is a monumental task; comprehensively cross-referencing them (as a full-blown distribution manager would need to), not to mention keeping them up-to-date, would be a full-time task for multiple people.
What I'm proposing bypasses that mess entirely - by simply packaging the individual dependencies with the offering.
OK, sorry you are using terms that have lost me. The developers utility builds an archive of all the dependences, yes? How do they get there? How do new ones get there as we go on?
And the User Utility then interfaces with this and downloads the necessary elements of the archive to the users computer, yes? How does it know what they are? Does a lotter have to list the ID's of the elements?
Bear in mind we already have people who don't list dependencies with links.
Also, who would host the archive?
I like the ideas coming out here, so I don't want to halt the conversation, but.
I'll throw my "stuck in the old ways" idea out there.
I would say that all of this can be done without changing the current format. The work is re-doing every exemplar so that models across every pack share the same Building/Prop Families. Each MEGA pack has "exemplar packs" associated with it (one for buildings, another for props...). A set of core lots are also made.
The player would download the Lot pack. Say this person wants to use CP vol 01, so this person downloads that MEGA pack and the exemplar packs. Since all of the exemplars are separated, no lot is "dependent" on the player having a specific MEGA pack, only the more packs the more models (the same amount of families), and the less repetition.
As far as programmed lists of things: I would like to see some database that works the other way around. When a dependency is downloaded, so too is every exemplar associated with it. The models take up the most file size, so getting the most of them is more important to me.
Twyla i mean by that the program doesnt have its own database, the database is provided with custom content, within a file included in the lot, the file just has the ID numbers that reference in the server database, the server database then link to the correct files on the exchange, it would have a system so content that has been merged links to the merged packs, and a system for determining if existing content is up to date, maybe the plugin folder has a database of installed dependencies, and updates ones needed updating. the program wouldnt have free access to the exchange, you would have to login to the exchange on the program.
regardless if you take my approach or another approach i belive we need to come up with simpler ways of tracking down dependencies, installing them, and updating them.
Its 2011 so I think most people have something better than dial-up for downloading, and have a better computer than the 256 mb RAM Windows ME dinosaur they started playing SC4 with in 2003. It's my opinion that if ain't broke, don't fix it, or things get even more complicated. I have virtually all the major packs on this site, I keep some in an inactive backup plugins folder, etc. The real solution is for creators to think about what they are doing first, and avoid using one-off dependencies if it means their creation could otherwise be dependency light or free.
QuoteI took one of these apart to discover that the only thing the 'creator' contributed was the 'recipe' - NONE of the actual resources used were theirs.
To be fair, some lots are really good. Some dependency packs with far more bits and pieces than any one thing could use. They seemingly exist for this purpose. Jestarr's industrial props and their use towards building railyard lots and industrial complexes are a great example.
@ superchad
Quoteregardless if you take my approach or another approach i believe we need to come up with simpler ways of tracking down dependencies, installing them, and updating them.
Interesting. ??? I don't find them hard to track down at all. I can see a scenario when someone goes on a massive download spree that they may have some dependency searching and extra work afterwards.
@ Twyla. I'd say go for it if you want to write a program. If not? Then the discussion should properly fizzle out and will have served no purpose.
My rules of thumb are to use the lot as is. Delete. Fix it in the LE. Download the dependency when critical to the function or look of the lot. Four choices and none require programming knowledge, time searching the past, or future upkeep.
- Jim
In my honest opinion, I would have to say that this entire converstion is useless. &mmm
I mean sure this will benefit new players to the game who have to download all these dependencies but for those of us who already have them it won't matter. Besides once you download them, you won't have to download them all again and with all these dependency superpacks (like BSC Essentials). Before you know it you won't have to download anything ever again. I have to agree with Zaphod :
Quote from: Zaphod on May 17, 2011, 05:33:33 PM
It's my opinion that if ain't broke, don't fix it, or things get even more complicated.
I believe that doing this will just cause major problems, but that's just my thought. Which is better, spending a huge amount of time to create a program and database, never mind updating it, or spending a few minutes just to go to a dependency and download it?
If you decide to go ahead with this then I wish you all the best of luck. :thumbsup:
Regards,
Jack
i dont see how a system could hurt
jmyers2043
that is the major scenerio that happens alot, people dont just download one or two things, i always downloads large amounts of content, and extract it into a single folder, label the folder say May162011 run the install files, and hunt for dependencies if needed
it would make doing this much simpler, and updating them simpler too
also sometimes people have to rebuild there plugins, say from a hard drive crash/corruption
Like everyone else using custom content with SC4, I'd love to see a simpler way of managing what I have and what I need when I get a new lot from one of the repositories. And while I don't want to stifle enthusiasm and inititive, I've been through many of these discussions over many years (both logical/civil and very heated) and we always seem to come back to the same position.
We (as a community) have looked to improve the dependency process; that's where the mega pack solution came from. We have tried to make it easier to find dependencies with linkable HTML-based readme's, Diggis' brilliant list, the LEX DVD's etc. We have looked at the concerns around a large volume of plugins through the DatPacker.
But the reality is that the SC4 community is not as large and active as it was 5 years ago. Tools are great when the owner/developer is still around, but once they move on (and everybody does) the tools often fall into disarray (luckily iLive's reader is the exception here).
We have a huge volume of existing custom content, and much of it belongs to folks who are no longer here. Creating any form of index/digest/restructuring exemplars etc. etc. etc. is just too big a job and not feasible for anyone.
If you re-distribute every model, desc etc. with each lot, you will often be redistributing the same content multiple times (because we all love the same park benches or trees) which is why we went down the prop-pack path in the first place.
Not all custom content in these dependencies are Models and Desc's, there are custom queries, scripts, graphics files, etc. etc. etc. that are interrelated and work best when packaged together.
And, as we've seen with the recent upgrade with ST, any work we could do now may be invalidated with a major upgrade killing links.
And at the end of the day "as soon as you make something idiot proof, someone invents a new type of idiot"! You will never solve every problem, so it becomes a risk vs. reward exercise and I think that after all these years we've found a reasonable balance.
I had a long, detailed, and intricately-wrought post.... that a glitch inadvertently wiped - and it took me a while to get back to this subject.
Then, this morning, a chat friend really hit the nail between the eyes as to where the *REAL* division concerning this topic lies:
"Old Guys" vs "New Guys"
So far as the "Old Guys" are concerned, conversations like this are 'a waste of forum space'. After all - they've been here since SC4 came out and already have all the dependencies (as well as many extremely useful organizational utilities which are no longer available). They know all the ins and outs intimately (not to mention having at least three expansions which utilize every single asset in the Mega Packs).
Not to name names, but jacksunny's post (http://sc4devotion.com/forums/index.php?topic=13079.msg379361#msg379361) very much reflects this attitude.
The "New Guys" (like me) are a bit more selective about custom content. I can't speak for all the "New Guys", but dependencies make me stop and consider - "Is this particular offering desirable enough to justify downloading the dependencies?" And naturally, the more dependencies a particular offering requires, the faster that question earns a 'no'.
I've spoken with - and heard from - enough of the "New Guys" to be convinced that my perspective is far from being unique.
Yet people don't see how this affects the community as a whole...
As deadwoods pointed out, people tend to move on to other pursuits (for whatever reasons), leaving SC4 (and its communities) to gather dust in some forgotten corner as a (hopefully) cherished memory.
Which means that these communities rely on "New Guys" to keep going - and a lot of the "Old Guys" tend to forget that they were "New Guys" themselves at one point. SC4 is hardly alone in this attitude - though hardly the worst 'offenders' (I could offer a HUGE list from personal experience).
Back when they were "New Guys", dependencies weren't an issue - they were a 'solution'. It certainly wasn't the ONLY option - just what seemed to be the best one at the time.
Well... "Times, they are a-changin'."
We need a new 'solution' - one that both addresses the "New Guys" problems (which is - and has been - causing stagnation in the community) and also remains compatible with the status-quo ("Old Guys").
What I proposed in the OP fits the bill. Again - it's not the ONLY potential solution, and may not be the BEST one, but I haven't seen or thought of a better one.
Also - just for the record - I am solidly AGAINST the database/indexing notion (as is readily obvious in the OP). Myself and others have outlined - in considerable detail - as to why such an approach is beyond impractical.
Additionally: It isn't that I CAN'T write these programs myself... It's that I'm not intimately familiar with SC4's inner workings - and that familiarity counts for a lot. Someone who already possesses that familiarity could likely write these programs in less time than it would take me to track down and assign the variable elements these programs would need (and with far fewer mistakes).
if you want info on DBPF file structure, check out ModTheSims (http://www.modthesims.info/wiki.php?title=DatabasePackedFile) or SimsWiki.info (http://www.simswiki.info/DatabasePackedFile)
If you know Java, check out DBPF4J (http://sourceforge.net/projects/sc4dbpf4j/) by our own Stephan79 and you could always try your hand understanding OR_DAT from ilive's reader (http://sf.net/projects/ilive-reader/) (I think I remember being told that is the part that does dbpf stuff).
To add on to JoeST. Reader 1.4 can also use LUA to perform mass modding. You will have to be knowledgeable about the way Models/Exemplars work (and the limitations of 1.4), but there are very few things that can not be done with the scripting.
All you need is MAXIS LE (http://www.sc4devotion.com/SC4D_Maxis_files_LE.html), or SC4PIM/PIMX (http://sc4devotion.com/csxlex/lex_filedesc.php?lotGET=2260). You then can re-lot things so that you need no extra dependencies. In many cases you may find that all you want is the model that comes with the download (The Maxis Props/Textures are quite decent).
I think that if you spend time learning about the way all of these mods connect, then you will see that the line between "new guy" and "old guy" is very thin. I have been around for a little more than a year, but there is not much I haven't touched as far as general modding is concerned. I have to say, the most valuable things that people offer to the community is time.
So, suggestions for tools that would be useful (and feasible):
- Duplicate TGI discovery/removal:
Scans a set of DAT files for conflicting TGI's and warns the user where they are. Allows the user to automatically strip them out from where they arent wanted.
note: The removal task is done by DatPacker already.
- Unused TGI discovery/removal:
A deep scan which builds a graph/database of TGI's you dont need. Allows the user to automatically strip unused TGI's.
- Dependency TGI discovery:
A deep scan which builds a graph/database of TGI's you need. Also alerts on TGI's that it cannot find.
- Small DAT packer/unpacker:
Allows you to select which files you wish to merge into a DAT. Maybe allows you to insert/extract raw files to particular places/types/etc.
example: I have a DAT with lots of LTEXT's in it, this tool allows me to pull all of them out into a set of .txt files, and then change them in any program before putting them back where I found them.
note:I guess this is already done by Reader's Inport/Export features, but it would be nice to have a nice interface around the task.
The first three could be bundled together into a tool that builds a graph/database of TGI's as the game sees it, and allows you to see where there are duplicates or unused or missing TGI's/etc.
What would be really nice as a toolmaker would be a set of libraries that would allow you to rebuild all the current tools on a common base, this would allow you to make quick tools as described above really easily.
Any other simple tools/actions people want?
Joe
I would say the stripping of TGI's needs to have the type of file next to them. I wouldn't want to remove a query when I aimed for a prop. If one is removing models, only one of the 20+ TGI's should be visible, and a zoom 4 FSH being visible might be nice too.
For me, going into Reader and filtering by Exemplar/Ltext to remove them is rather quick. Models not so quick. Something that could display an image of a model with a check box next to it ("remove?") would be much appreciated by me.
I don't think the removed models should be deleted from the computer though (so remove can also mean keep). Perhaps move the models/exemplars to the program's folder. Then next time it was run it would check that folder if it found you needed more dependencies.
SG Vol_01 makes props out of Maxis Effects, so it is important that the program knows when something is in Maxis.
indeed, they would scan/store/"know about" the maxis stuff, and would allow you to add stuff to this 'special' list, and also they wouldn't delete stuff, merely maybe stop it from being put in game, the tool outputting DAT's of your stuff akin to what you've asked for.
Possibly the nicest way I can think to keep what you have downloaded and allow for subsets to be used in game would be a program that pulled out every file in a DBPF to a "Windows"-able equivalent (LTEXT<->.txt, Exemplar<->.csv?, etc..) and then stored in a source control system (git, mercurial etc), and store a manifest of what the user wants in game along with the graph/db so that it can select which parts it wishes to pack up for gameplay.
And the grouping of models/textures/etc into something visible and atomic, along with other things, is something obvious.
Quote from: Twyla on May 19, 2011, 12:00:01 PM
The "New Guys" (like me) are a bit more selective about custom content. I can't speak for all the "New Guys", but dependencies make me stop and consider - "Is this particular offering desirable enough to justify downloading the dependencies?" And naturally, the more dependencies a particular offering requires, the faster that question earns a 'no'.
I've spoken with - and heard from - enough of the "New Guys" to be convinced that my perspective is far from being unique.
First off, edited for relevance, I try not to make extremely long posts.
Now then, the part I am responding to...being on 'old guy' from the days of the STEX and an active player of the game since its release, I can tell you that what you have pointed out here even gives me pause. Especially if the dependency list is long. I *probably* have all the dependencies, but I still don't want to take that chance. I have passed up on a lot of really beautiful lots due to the number of dependencies required to make it work.
Currently I have some lots that have brown boxes, so I know this well. Yes I am an old timer, but I also lost a HD and had to go back to a three year old copy of my plugins...so when I reinstalled the newer content, I have dependencies missing and am still having problems tracking them all down (especially with the changes at STEX).
As for a solution, if there was simply a 'clearing house' of sorts for all the dependencies to be found, it would simplify the matter some. LEX pretty much has this already, in that you can pull a search for dependency. Easy enough. The problem is that finding dependencies that aren't available here at the LEX can be a challenge, especially since the links are now outdated due to the STEX upgrade. Those are probably the files I am missing, but finding them is the problem.
I know there is a master dependency list here in the forums, but again, the links may be outdated due to the STEX upgrade, or sites that no longer exist.
In the end, there really isn't an easy way to do this. If there were, some one would undoubtedly have done it by now. I'm sure you can make a program to search for what is missing (but not being a programmer myself, no clue how to do it), but then actually pointing you where you need to go to get it is another thing entirely.
Quote from: Shark7 on May 20, 2011, 09:08:29 AM
I know there is a master dependency list here in the forums, but again, the links may be outdated due to the STEX upgrade, or sites that no longer exist.
The list is mostly still updated. I've checked a few of the STEX links and they still work. I have had to change them all in the past, but it appears the new upgrade has left them mostly in tact.
Don't know that I agree with you going all "George W" on us, turning this into an "us vs. them" discussion.
I'm just passing on what has been done/discussed in the past. There's the old adage "those who ignore the lessons of the past are destined to repeat them".
That being said, if you can come up with a more effective way to handle all the custom content out there, good luck to you.
Quote from: Diggis on May 20, 2011, 03:14:59 PM
The list is mostly still updated. I've checked a few of the STEX links and they still work. I have had to change them all in the past, but it appears the new upgrade has left them mostly in tact.
That's a good thing then, I'll be sure to double check it. After today, I think I may have gotten most of the ones I have been missing. Or rather I've taken care of the brown boxes, I may still be missing some non-essential props.
Quote from: JoeST on May 20, 2011, 02:39:52 AM
So, suggestions for tools that would be useful (and feasible):
- Duplicate TGI discovery/removal:
Scans a set of DAT files for conflicting TGI's and warns the user where they are. Allows the user to automatically strip them out from where they arent wanted.
note: The removal task is done by DatPacker already.
- Unused TGI discovery/removal:
A deep scan which builds a graph/database of TGI's you dont need. Allows the user to automatically strip unused TGI's.
- Dependency TGI discovery:
A deep scan which builds a graph/database of TGI's you need. Also alerts on TGI's that it cannot find.
- Small DAT packer/unpacker:
Allows you to select which files you wish to merge into a DAT. Maybe allows you to insert/extract raw files to particular places/types/etc.
example: I have a DAT with lots of LTEXT's in it, this tool allows me to pull all of them out into a set of .txt files, and then change them in any program before putting them back where I found them.
note:I guess this is already done by Reader's Inport/Export features, but it would be nice to have a nice interface around the task.
The first three could be bundled together into a tool that builds a graph/database of TGI's as the game sees it, and allows you to see where there are duplicates or unused or missing TGI's/etc.
What would be really nice as a toolmaker would be a set of libraries that would allow you to rebuild all the current tools on a common base, this would allow you to make quick tools as described above really easily.
QuoteAny other simple tools/actions people want?
Me, myself and I would be over the moon to be able to do what you suggest above! :thumbsup: :o
need a beta tester??? &bis&