• Welcome to SC4 Devotion Forum Archives.

Can I Upload Base Packs (no mods) to the LEX?

Started by ChiefZDN, January 07, 2018, 09:30:38 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

ChiefZDN

Hello,

In part of the Modpacc project, I want to upload base packs (no mods included, just a reference in the Deptracker) to the LEX. One question, can I do it?

Thanks.

Tarkus

#1
The idea of using the LEX Dependency Tracker and Custom Group options to assemble complete collections of plugins--essentially the same as the MODPACC idea--is one that has been discussed internally among the staff.  Theoretically, if one could assemble a working, satisfactory set of plugins taken entirely from the LEX catalog, the software is currently capable of taking care of every other step along the way.  The technology is there. 

The one thing on the LEX end that would probably need to be changed is some repackaging of items, as the prevalence of installers would actually complicate installation.

The other issue is when off-site dependencies come into play, as our Dependency Tracker can't do any more than provide links in those cases.  There are a lot of lots (play on words fully intended) that use dependencies from other sites, and some rather prominent ones, to boot, and that complicates the prospect of producing a self-contained, hassle-free package.  ST Staff has indicated that Dependency Tracker functionality is beyond their reach for the foreseeable future, so cross-site deployment is not an option at present.

As far as individuals making their own "dummy uploads" that are lists of plugins, that's probably a matter that we'd have to discuss internally before coming to a final decision.  My gut feeling, however, is that I'd like to see some additional oversight--namely, at least some basic quality assurance that the set works--before such an "upload" would get the go-ahead to be on the LEX.  It's an idea with merit, however.

-Alex

ChiefZDN

Quote from: Tarkus on January 08, 2018, 04:27:36 AM
The idea of using the LEX Dependency Tracker and Custom Group options to assemble complete collections of plugins--essentially the same as the MODPACC idea--is one that has been discussed internally among the staff.  Theoretically, if one could assemble a working, satisfactory set of plugins taken entirely from the LEX catalog, the software is currently capable of taking care of every other step along the way.  The technology is there. 

The one thing on the LEX end that would probably need to be changed is some repackaging of items, as the prevalence of installers would actually complicate installation.

The other issue is when off-site dependencies come into play, as our Dependency Tracker can't do any more than provide links in those cases.  There are a lot of lots (play on words fully intended) that use dependencies from other sites, and some rather prominent ones, to boot, and that complicates the prospect of producing a self-contained, hassle-free package.  ST Staff has indicated that Dependency Tracker functionality is beyond their reach for the foreseeable future, so cross-site deployment is not an option at present.

As far as individuals making their own "dummy uploads" that are lists of plugins, that's probably a matter that we'd have to discuss internally before coming to a final decision.  My gut feeling, however, is that I'd like to see some additional oversight--namely, at least some basic quality assurance that the set works--before such an "upload" would get the go-ahead to be on the LEX.  It's an idea with merit, however.

-Alex

Why we not using a blockchain API? Yes, I'm not expert on C or non-scripting language but there is an API for do that, Ethereum API (not to be confused with the cryptocurrency). The benefit here is easy to kickstart the Modpacc project. But, we need some coders here. I think Mak can implement that because he recently created Poppy Multiplayer Region.

Thanks.

Tarkus

At least in terms of everything under the LEX umbrella, the Dependency Tracker can already do the job.  As already mentioned, the reason ST isn't jumping in on the tracker is because they're not in position to implement new APIs, which would also preclude them from implementing some speculative blockchain-based API.  One would need to have a completely separate exchange, and then permission matters and catalog depth would come into play. 

Beyond that, one can throw all the programmers and new web technologies at it that they want, but the packaging issue would still be front and center.  Blockchain, as promising of a technology as it is, doesn't make ClickTeam installers go away.  They can't be cracked open in 7-zip like NSIS installers, and while I'm sure that someone with enough time could rig up a script to run a bunch of them in a row, I don't believe that'd be a particularly efficient setup.  I really do think it would be quicker to repackage existing LEX assets, as appropriate (and which one would need to do anyway for this other proposed route), and use what we already have. 

-Alex

mattb325

Quote from: Tarkus on January 14, 2018, 03:08:13 AM
....I really do think it would be quicker to repackage existing LEX assets, as appropriate (and which one would need to do anyway for this other proposed route), and use what we already have....

I moved away from the installers a long time ago (same went for html readmes and all of the other superfluous packaging that isn't necessary with run-of-the-mill uncomplicated files), but on my looooong, looooong list of things to do is to go through my old work and strip away the installer packages and re-submit the files....but with probably over 100 files in installers floating around just from me, it won't be quick  ::)


Kitsune

Interestingly with the click team installers I was able to extract just the model using 7zip. (.... this made installing IRM a lot easier). Anything else was seemingly encrypted.
~ NAM Team Member

matias93

Quote from: mattb325 on January 14, 2018, 02:33:36 PM
Quote from: Tarkus on January 14, 2018, 03:08:13 AM
....I really do think it would be quicker to repackage existing LEX assets, as appropriate (and which one would need to do anyway for this other proposed route), and use what we already have....

I moved away from the installers a long time ago (same went for html readmes and all of the other superfluous packaging that isn't necessary with run-of-the-mill uncomplicated files), but on my looooong, looooong list of things to do is to go through my old work and strip away the installer packages and re-submit the files....but with probably over 100 files in installers floating around just from me, it won't be quick  ::)

May I suggest a bundle approach for some things? Just take the installed files from the plugins folder, zip them together by categories (for example, the catalogue houses or the commercial parks), and release them packaged. Honestly, once one downloads one of your creations, is simply unavoidable to go on a rampage and download them all, so this will produce the same effect in less time and with less effort for you and the upcoming downloaders.

"Lets be scientists and as such, remember always that the purpose of politics is not freedom, nor authority, nor is any principle of abstract character,
but it is to meet the social needs of man and the development of the society"

— Valentín Letelier, 1895

ChiefZDN

Quote from: Tarkus on January 14, 2018, 03:08:13 AM
At least in terms of everything under the LEX umbrella, the Dependency Tracker can already do the job.  As already mentioned, the reason ST isn't jumping in on the tracker is because they're not in position to implement new APIs, which would also preclude them from implementing some speculative blockchain-based API.  One would need to have a completely separate exchange, and then permission matters and catalog depth would come into play. 

Beyond that, one can throw all the programmers and new web technologies at it that they want, but the packaging issue would still be front and center.  Blockchain, as promising of a technology as it is, doesn't make ClickTeam installers go away.  They can't be cracked open in 7-zip like NSIS installers, and while I'm sure that someone with enough time could rig up a script to run a bunch of them in a row, I don't believe that'd be a particularly efficient setup.  I really do think it would be quicker to repackage existing LEX assets, as appropriate (and which one would need to do anyway for this other proposed route), and use what we already have. 

-Alex

The first thing we need to do is allow large file (up to 100 M) to be added to exchanges (particularly LEX) without permission. 100 MB is enough for 97% of most repacked mods. The second thing is to develop an app to auto-install all mods installed in a computer. This may involving keystroke automation or sending some message to the system to select things by code. This may require knowledges about machine learning because it's pain to tag installers as Clickteam by hand. The third thing is to modularize enough (not less or more) mods. This is to reduce hassle and reducing data usage too, especially in the emerging markets like India or Indonesia.

mattb325

Quote from: Kitsune on January 14, 2018, 03:23:07 PM
Interestingly with the click team installers I was able to extract just the model using 7zip. (.... this made installing IRM a lot easier). Anything else was seemingly encrypted.

That would be a great find! Which option did you use? Everything I try just extracts the installer....

Kitsune

There are two ways to do it:

First is to open the zip and then extract using # :



Second to extract the exe and use #:



When looking inside you'll notice everything is numbered. The model will be in a bz2 with the largest size and is usually the last number. You'll need to give a name and add .dat to the end of it. iLives reader is also capabable and of reading what is the first bz2 - however it'll give you thousands of lines with 0000000 exemplars that are giberish.
~ NAM Team Member

mattb325


Tarkus

Kitsune, that's really clever stuff there! :thumbsup:  It looks like it does require the latest version of 7-zip (16.04), or at least a newer one, as I didn't have those menu options until I upgraded.

Even with the workaround, though, I do think repackaging is likely to be a necessary part of the solution, and as someone who is quite interested in seeing this through, I'd be happy to help with that process.  The simplest solution would be to simply put the contents of the existing installers into zip files, as it'd minimize disruption to the existing system, while still offering a substantially more streamlined experience.  If "nested zips" start to become a stumbling block, then it may be time to look at some file mergers, and/or allowing straight up .dat files in certain cases.

-Alex

Kitsune

Repacking stuff from exe to zip will need to answer a simple question... what do you do with install path? What I mean, currently with the above exe the install path is C:\Users\xxxxx\Documents\SimCity 4\Plugins\BSC\JMyers\Simmons Industrials. I think instead of dumping the files into a zip. we should atleast keep the integrity and zip it under BSC\JMyers\Simmons Industrials. Also - will the repackage now have two versions - one exe and one zip (and this would have a physical cost due to increasing the innodes / number of files), or replace the exe with the zip? Also, this should be totally doable by script, something that runs the installer and then zips the results. I do have a old school 6 core gulftown xeon x5650 gathering dust that can be reput into use.
~ NAM Team Member

ChiefZDN

#13
Quote from: Kitsune on January 15, 2018, 12:51:54 PM
There are two ways to do it:

First is to open the zip and then extract using # :



Second to extract the exe and use #:



When looking inside you'll notice everything is numbered. The model will be in a bz2 with the largest size and is usually the last number. You'll need to give a name and add .dat to the end of it. iLives reader is also capabable and of reading what is the first bz2 - however it'll give you thousands of lines with 0000000 exemplars that are giberish.

Yes, Clickteam Installer uses zlib and bzip2 to compress their file per https://community.clickteam.com/threads/34055-Packing-method?highlight=extract. Also, the installer might UPXed to reduce the size (https://community.clickteam.com/threads/35462-Copy-Protection-of-Installer-File?highlight=extract)