Dependencies - Blessing or Bane? And possible solutions?

[deadwoods]

Like everyone else using custom content with SC4, I'd love to see a simpler way of managing what I have and what I need when I get a new lot from one of the repositories. And while I don't want to stifle enthusiasm and inititive, I've been through many of these discussions over many years (both logical/civil and very heated) and we always seem to come back to the same position.

We (as a community) have looked to improve the dependency process; that's where the mega pack solution came from. We have tried to make it easier to find dependencies with linkable HTML-based readme's, Diggis' brilliant list, the LEX DVD's etc. We have looked at the concerns around a large volume of plugins through the DatPacker.

But the reality is that the SC4 community is not as large and active as it was 5 years ago. Tools are great when the owner/developer is still around, but once they move on (and everybody does) the tools often fall into disarray (luckily iLive's reader is the exception here).

We have a huge volume of existing custom content, and much of it belongs to folks who are no longer here. Creating any form of index/digest/restructuring exemplars etc. etc. etc. is just too big a job and not feasible for anyone.

If you re-distribute every model, desc etc. with each lot, you will often be redistributing the same content multiple times (because we all love the same park benches or trees) which is why we went down the prop-pack path in the first place.

Not all custom content in these dependencies are Models and Desc's, there are custom queries, scripts, graphics files, etc. etc. etc. that are interrelated and work best when packaged together.

And, as we've seen with the recent upgrade with ST, any work we could do now may be invalidated with a major upgrade killing links.

And at the end of the day "as soon as you make something idiot proof, someone invents a new type of idiot"! You will never solve every problem, so it becomes a risk vs. reward exercise and I think that after all these years we've found a reasonable balance.

[Twyla]

I had a long, detailed, and intricately-wrought post.... that a glitch inadvertently wiped - and it took me a while to get back to this subject.

Then, this morning, a chat friend really hit the nail between the eyes as to where the *REAL* division concerning this topic lies:


"Old Guys" vs "New Guys"


So far as the "Old Guys" are concerned, conversations like this are 'a waste of forum space'.  After all - they've been here since SC4 came out and already have all the dependencies (as well as many extremely useful organizational utilities which are no longer available).  They know all the ins and outs intimately (not to mention having at least three expansions which utilize every single asset in the Mega Packs).

Not to name names, but jacksunny's post very much reflects this attitude.

The "New Guys" (like me) are a bit more selective about custom content.  I can't speak for all the "New Guys", but dependencies make me stop and consider - "Is this particular offering desirable enough to justify downloading the dependencies?"  And naturally, the more dependencies a particular offering requires, the faster that question earns a 'no'.

I've spoken with - and heard from - enough of the "New Guys" to be convinced that my perspective is far from being unique.


Yet people don't see how this affects the community as a whole...

As deadwoods pointed out, people tend to move on to other pursuits (for whatever reasons), leaving SC4 (and its communities) to gather dust in some forgotten corner as a (hopefully) cherished memory.

Which means that these communities rely on "New Guys" to keep going - and a lot of the "Old Guys" tend to forget that they were "New Guys" themselves at one point.  SC4 is hardly alone in this attitude - though hardly the worst 'offenders' (I could offer a HUGE list from personal experience).

Back when they were "New Guys", dependencies weren't an issue - they were a 'solution'.  It certainly wasn't the ONLY option - just what seemed to be the best one at the time.

Well...  "Times, they are a-changin'."


We need a new 'solution' - one that both addresses the "New Guys" problems (which is - and has been - causing stagnation in the community) and also remains compatible with the status-quo ("Old Guys").

What I proposed in the OP fits the bill.  Again - it's not the ONLY potential solution, and may not be the BEST one, but I haven't seen or thought of a better one.

Also - just for the record - I am solidly AGAINST the database/indexing notion (as is readily obvious in the OP).  Myself and others have outlined - in considerable detail - as to why such an approach is beyond impractical.

Additionally:  It isn't that I CAN'T write these programs myself...  It's that I'm not intimately familiar with SC4's inner workings - and that familiarity counts for a lot.  Someone who already possesses that familiarity could likely write these programs in less time than it would take me to track down and assign the variable elements these programs would need (and with far fewer mistakes).

[JoeST]

if you want info on DBPF file structure, check out ModTheSims or SimsWiki.info

If you know Java, check out DBPF4J by our own Stephan79 and you could always try your hand understanding OR_DAT from ilive's reader (I think I remember being told that is the part that does dbpf stuff).

[Lowkee33]

To add on to JoeST.  Reader 1.4 can also use LUA to perform mass modding.  You will have to be knowledgeable about the way Models/Exemplars work (and the limitations of 1.4), but there are very few things that can not be done with the scripting.

All you need is MAXIS LE, or SC4PIM/PIMX.  You then can re-lot things so that you need no extra dependencies.  In many cases you may find that all you want is the model that comes with the download (The Maxis Props/Textures are quite decent).

I think that if you spend time learning about the way all of these mods connect, then you will see that the line between "new guy" and "old guy" is very thin.  I have been around for a little more than a year, but there is not much I haven't touched as far as general modding is concerned.  I have to say, the most valuable things that people offer to the community is time.

[JoeST]

So, suggestions for tools that would be useful (and feasible):

• Duplicate TGI discovery/removal:
  Scans a set of DAT files for conflicting TGI's and warns the user where they are. Allows the user to automatically strip them out from where they arent wanted.
  note: The removal task is done by DatPacker already.
  
• Unused TGI discovery/removal:
  A deep scan which builds a graph/database of TGI's you dont need. Allows the user to automatically strip unused TGI's.
  
• Dependency TGI discovery:
  A deep scan which builds a graph/database of TGI's you need. Also alerts on TGI's that it cannot find.
  
• Small DAT packer/unpacker:
  Allows you to select which files you wish to merge into a DAT. Maybe allows you to insert/extract raw files to particular places/types/etc.
  example: I have a DAT with lots of LTEXT's in it, this tool allows me to pull all of them out into a set of .txt files, and then change them in any program before putting them back where I found them.
  note:I guess this is already done by Reader's Inport/Export features, but it would be nice to have a nice interface around the task.
  

The first three could be bundled together into a tool that builds a graph/database of TGI's as the game sees it, and allows you to see where there are duplicates or unused or missing TGI's/etc.

What would be really nice as a toolmaker would be a set of libraries that would allow you to rebuild all the current tools on a common base, this would allow you to make quick tools as described above really easily.

Any other simple tools/actions people want?

Joe

[Lowkee33]

I would say the stripping of TGI's needs to have the type of file next to them.  I wouldn't want to remove a query when I aimed for a prop.  If one is removing models, only one of the 20+ TGI's should be visible, and a zoom 4 FSH being visible might be nice too.

For me, going into Reader and filtering by Exemplar/Ltext to remove them is rather quick.  Models not so quick.  Something that could display an image of a model with a check box next to it ("remove?") would be much appreciated by me.

I don't think the removed models should be deleted from the computer though (so remove can also mean keep).   Perhaps move the models/exemplars to the program's folder.  Then next time it was run it would check that folder if it found you needed more dependencies.

SG Vol_01 makes props out of Maxis Effects, so it is important that the program knows when something is in Maxis.

[JoeST]

indeed, they would scan/store/"know about" the maxis stuff, and would allow you to add stuff to this 'special' list, and also they wouldn't delete stuff, merely maybe stop it from being put in game, the tool outputting DAT's of your stuff akin to what you've asked for.

Possibly the nicest way I can think to keep what you have downloaded and allow for subsets to be used in game would be a program that pulled out every file in a DBPF to a "Windows"-able equivalent (LTEXT<->.txt, Exemplar<->.csv?, etc..) and then stored in a source control system (git, mercurial etc), and store a manifest of what the user wants in game along with the graph/db so that it can select which parts it wishes to pack up for gameplay.

And the grouping of models/textures/etc into something visible and atomic, along with other things, is something obvious.

[Shark7]

The "New Guys" (like me) are a bit more selective about custom content.  I can't speak for all the "New Guys", but dependencies make me stop and consider - "Is this particular offering desirable enough to justify downloading the dependencies?"  And naturally, the more dependencies a particular offering requires, the faster that question earns a 'no'.

I've spoken with - and heard from - enough of the "New Guys" to be convinced that my perspective is far from being unique.

First off, edited for relevance,  I try not to make extremely long posts.

Now then, the part I am responding to...being on 'old guy' from the days of the STEX and an active player of the game since its release, I can tell you that what you have pointed out here even gives me pause.  Especially if the dependency list is long.  I *probably* have all the  dependencies, but I still don't want to take that chance.  I have passed up on a lot of really beautiful lots due to the number of dependencies required to make it work.

Currently I have some lots that have brown boxes, so I know this well.  Yes I am an old timer, but I also lost a HD and had to go back to a three year old copy of my plugins...so when I reinstalled the newer content, I have dependencies missing and am still having problems tracking them all down (especially with the changes at STEX).

As for a solution, if there was simply a 'clearing house' of sorts for all the dependencies to be found, it would simplify the matter some.  LEX pretty much has this already, in that you can pull a search for dependency.  Easy enough.  The problem is that finding dependencies that aren't available here at the LEX can be a challenge, especially since the links are now outdated due to the STEX upgrade.  Those are probably the files I am missing, but finding  them is the problem.

I know there is a master dependency list here in the forums, but again, the links may be outdated due to the STEX upgrade, or sites that no longer exist.

In the end, there really isn't an easy way to do this.  If there were, some one would undoubtedly have done it by now.  I'm sure you can make a program to search for what is missing (but not being a programmer myself, no clue how to do it), but then actually pointing you where you need to go to get it is another thing entirely.

[Diggis]

I know there is a master dependency list here in the forums, but again, the links may be outdated due to the STEX upgrade, or sites that no longer exist.

The list is mostly still updated. I've checked a few of the STEX links and they still work.  I have had to change them all in the past, but it appears the new upgrade has left them mostly in tact.

[deadwoods]

Don't know that I agree with you going all "George W" on us, turning this into an "us vs. them" discussion.

I'm just passing on what has been done/discussed in the past. There's the old adage "those who ignore the lessons of the past are destined to repeat them".

That being said, if you can come up with a more effective way to handle all the custom content out there, good luck to you.

[Shark7]

The list is mostly still updated. I've checked a few of the STEX links and they still work.  I have had to change them all in the past, but it appears the new upgrade has left them mostly in tact.

That's a good thing then, I'll be sure to double check it.  After today, I think I may have gotten most of the ones I have been missing.  Or rather I've taken care of the brown boxes, I may still be missing some non-essential props.

[Gringamuyloca]

So, suggestions for tools that would be useful (and feasible):

• Duplicate TGI discovery/removal:
  Scans a set of DAT files for conflicting TGI's and warns the user where they are. Allows the user to automatically strip them out from where they arent wanted.
  note: The removal task is done by DatPacker already.
  
• Unused TGI discovery/removal:
  A deep scan which builds a graph/database of TGI's you dont need. Allows the user to automatically strip unused TGI's.
  
• Dependency TGI discovery:
  A deep scan which builds a graph/database of TGI's you need. Also alerts on TGI's that it cannot find.
  
• Small DAT packer/unpacker:
  Allows you to select which files you wish to merge into a DAT. Maybe allows you to insert/extract raw files to particular places/types/etc.
  example: I have a DAT with lots of LTEXT's in it, this tool allows me to pull all of them out into a set of .txt files, and then change them in any program before putting them back where I found them.
  note:I guess this is already done by Reader's Inport/Export features, but it would be nice to have a nice interface around the task.
  

The first three could be bundled together into a tool that builds a graph/database of TGI's as the game sees it, and allows you to see where there are duplicates or unused or missing TGI's/etc.

What would be really nice as a toolmaker would be a set of libraries that would allow you to rebuild all the current tools on a common base, this would allow you to make quick tools as described above really easily.

Any other simple tools/actions people want?

Me, myself and I would be over the moon to be able to do what you suggest above!   
need a beta tester???