• Welcome to SC4 Devotion Forum Archives.

Automated Grass/Sidewalk texturing.

Started by mgb204, May 04, 2015, 04:23:45 PM

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

mgb204

Well it's been quite a year or so for me with SC4, from very simple beginnings with some lots I'm starting to get a handle on how everything works under the hood. One of the things that attracted me to modding in the first place was unifying my game textures together, I wanted to be able to switch sidewalks on the fly without loosing the uniform textures that the Maxis ones have. As a result of both my SEN (sidewalk) and TGN (grass) mods I got talking to Ron (Rivit) and those discussions have led to the developments I wish to share here.

Before I even got far with SEN development last summer, I already knew a simple Sandstone replacement wouldn't cut it, I wasn't looking to make a Sandstone mod, but my work on it would lead to a way to create a mod for any given sidewalk. Whilst my efforts in this regard held much promise, a better solution was proposed by Ron, one that would pretty much automate the entire process. In essence instead of a single-colour alpha, by utilising multi-colour alphas it's possible to replace both sidewalks and grass (and/or transparency) based on the colours used within the alphas. Here is an example to illustrate what I'm talking about:



This texture is from the Turbo Roundabouts, White areas (RGB 255,255,255) denote where parts of the original texture will be kept, whilst Green (RGB 0,255,0) denotes the sidewalk, Cyan (RGB 0,255,255) for grass and Black (RGB 0,0,0) for Transparency. Here's how this works in practise:



To process them requires a custom in-development version of GoFSH which Rivit designed for this purpose, my part in all this has been in trying to create a complete NAM repository, partially so I can make custom sidewalks in no time, but also with the intent of uploading it for others to make use of. Here I've used Paeng's sidewalks and SV grass, but you can process any sidewalk or grass you wish this way.

Due to the differences in NAM textures, this process had to be robust enough to interpret different types of texture wealthing, so for example the green channel was used for these puzzle-based textures. However, for regular wealthed textures by simply switching to another RGB colour channel (i.e. using different colours for the Alphas), it's possible to do more than simply replace sidewalks and grass, it's possible to generate automatically all the wealth textures based on nothing more than a modified alpha. The additional benifit of simple customisation of grass/sidewalk (or the possibility of releasing such sets for users to customise themselves) is one huge additional benefit in respect of texture creation. Here's an example of a suitable texture from Rivit's TSR files:



This time using Blue to Denote where Grass appears for wealth 1-3 and optionally sidewalk for wealth 5-7, note, the final textures use the Alphas for transparency to avoid texture bleeding hence the sidewalks. All this is created from a single texture/alpha, in this case you can simply take the regular 0-wealth alpha and add the blue parts for the grass. Naturally the final textures will incorporate all wealth grasses and sidewalks as required, although it's possible to make a unified set by adjusting the textures used by GoFSH to a single-wealth. This whole system is exceedingly flexible and for the most part can be adapted to fit a creators requirements for how the wealth textures should appear. I have to admit that whilst I've a good overview of how to put this into practise, some of the theory alludes me still, I'm really more of a technical guy and image editing is something I've had to learn from the ground up, it doesn't exactly come naturally. The point being, hopefully Ron will jump in here and explain the more precise details of how the different RGB channels work to build up the necessary alphas/textures, because I won't be able to explain it in more depth.

Using this technique, the EU Alphas work just fine for the US textures so long as the networks geometry is consistent, for the most part this requires no additional work to create US or EU versions. Similarly when I made these Alphas for the TuLEPs, I only had to modify a small number to fully support all the EU TuLEP variants.

Right now I've completed the TuLEPs, Turbo Roundabouts, FLUPs and some other pieces. I will expand this to hopefully include every texture with a baked-in sidewalk in a repository. I don't plan to do this right now for the other NAM networks, even though the process is simple in practise, when you start dealing with 1000's of textures, there is a limit to where I want to set my goals here. However, the grass can already be automated for replacement on other networks if you are determined enough, with the addition of sidewalks it does open the possibilities up for users to seriously consider such modding without taking weeks or months to achieve. In this sense I want to personally thank Ron for creating the tools to make something that I've wanted to realise since those beginnings.

eggman121

Great discovery  :thumbsup:

From my understanding the turbo roundabouts are puzzle pieces so this is great for dealing with those situations.

kudos and +1 for your discovery.

I think Mandelsoft had a repository of all the road textures so you may be able to get the textures from there.

This will be great for alternative sidewalks due to the wealth which will give much variety

-eggman121

noahclem

Wow, great work with this  &apls  Automating the process of generating variations of the wealthification would be hugely valuable and I'd have loved to have something like this to easily keep track of the grassy areas when I did the tram and other textures with grassy areas that I've already done. It's of extra importance to me since, like you, I used an SV-based texture for my grass without different shades for wealth and I've recently switched to a custom texture from Vortext for grass after he noticed tiling effects which were exacerbated be me scaling the texture down to reduce the size of the blades of grass or whatever those details are exactly. In case his change is of interest to you with your textures here's my modified SV compared to Vortext's with subtle shading around the edges for depth:



I've not been looking forward to going in and redoing the grass, to say the least, but it will be much nicer to do if I'm at least adding the ability to easily make them compatible with other wealth/grass systems :)

vortext

Wow indeed, this seems very promising developments!!   &apls

So, if I understand correctly, it would be quite simple to render a full wealth & zone dependent texture set, i.e. 7 unique combinations of grass and sidewalk?

Do the multi-color alphas handle transparency? Hope so because I like noisy gradients and the like. ;D

Anyway, think I'll hold of making alphas for the cobble stone roads just a bit to see how this pans out.  :thumbsup:
time flies like a bird
fruit flies like a banana

mgb204

When I say create a repository, one of the problems with Marteen's is the lack of Euro Texture support, but what I'm doing is a little differently focussed. I'm planning to include the full required Alpha Set and NAM textures complete with scripts for auto-generation in GoFSH. In theory you'd download the repository and GoFSH, then you simply need to follow a few instructions to generate your own custom set. All you need are three sidewalk/grass wealth textures, ideally with colour-corrected darkened variants. These textures will then be incorporated into a series of DATs ready to place in your plugins folder. In the first instance this should cover only those NAM pieces with baked-in sidewalks, since I've mapped that out with my SEN mod, much still unreleased which will be included once I've finished converting my personal repository to this new system.

That leaves a whole that my TGN mod currently fills for three grass variants, hopefully in future support can be extended. Making the Alphas is pretty easy, in some cases it appears automation will be possible too, ideally at some point I'm hoping others will contribute, but better still I'd love to see this adopted during texture development, that way such changes are made easy. From a development standpoint, it's actually less work overall with huge benefits in terms of flexibility. For example if you were testing different grass types, you could automate generation of different sets by changing a single grass texture GoFSH references.

@noahclem, taking that Tram Roundabout as an example, the changes to the edges are not something such an automated tool can handle. It simply takes a grass texture and applies it as a layer, so each separate texture would have the same grass pattern.

@vortext, Indeed, although it assumes that wealths 1-3 and 4-6 will keep grass/sidewalk in the same configuration, otherwise it generates the various $, $$ and $$$ variants for you from just the 0 wealth and one alpha texture, gradients are indeed possible too.

MandelSoft

I actually do have a full repository with all Euro Textures I made so far:

https://drive.google.com/file/d/0B8Tb6LVuRM4Qa1NBODY2aURTaTg/view?usp=sharing
(Warning: this file is 480MB!)

Best,
Maarten
Lurk mode: ACTIVE

memo

This really is a great idea by Ron. :thumbsup:

My concern is keeping your repository up-to-date. If something in the NAM changes, it will be really hard to update your complete repository with all those changes. It's a similar problem the Euro textures already have to fight. I am not entirely sure how your process of creating the repository works, but, assuming there are simple rules for creating the multi-color alphas, automating this entire process might be worth considering, perhaps?

mgb204

#7
Part of the reason for keeping it quite focussed right now to the baked-in sidewalk pieces, they don't change too much from version to version. OK take NAM 32 for example and you'd have to add El-Train over Ave/RD-4, Turbo Roundabouts and Draggable Road Viaducts. The Turbo Roundabout file took around 3 hours, the others are much smaller and only 10-15m jobs. I think this approach is manageable, in any case it's a mirror of something I will be doing anyway given my desire for uniformity. When you try to expand this to every NAM texture, that's a whole new kettle of fish I'm all too familiar with, I also don't think that would be easy to keep on top of.

From my initial testing though, regular network textures seem quite automatable, for example I did the whole Bikepath set without making a single alpha. The problem with automating puzzle pieces is that there are no wealth textures to build the new Alphas from, so these pieces will remain a manual job. From my work on TGN, with the exception of RHW, in theory at least, the automation should be able to cover all the non-pp textures.

rivit

Probably a good moment to chime in (sorry Robin I was a bit slow) - hereunder a summary diagram of the Wealthing and Edit Auto processes now supported in GoFSH (I'm still writing the documentation). This is worthy of study if only to let the principle sink in - if you can easily think in RGB channels this is a doddle, otherwise you need to take the colour approach to understanding how it works.

GoFSH now has a scripting capability - in effect you can tell GoFSH how to construct an entire modd as far as the textures are concerned. My last TSR modd took 15min to assemble from its C,D files. Robin made his variant of it from the masters and its scripts without touching the files - he supplied his grass and paving textures.   

In working my recent TSR, and with other sets Robin and I have tried out, we've seen a space saving of about 60% making the colour and definition (C,D) bitmaps which zip to about the same size as the output .dats that will be used in game. Most of the effort saving is in the edit process, 2 maps per texture to get up to 40 mips saves a lot of work. Being able to modify "fixed textures" instead of manually editing them is a real advance too.


When we thought about how this could work at large scale, we concluded that a certain discipline would need to be introduced in the repository - someone would probably have to be the Keeper of the Textures (one who issues the ids, and locks the door), a register of used textures (texture sets) would be needed (I'm not aware that one exists), as also a check in/out process for the repository. This isn't really any different than what is needed for any sub-project of NAM today when thinking about how a project, and then its .dats comes together. It does require everyone to buy in to have any chance of success.

Once made, textures don't change very much - they are made, fixes applied and then they only see large scale revision, if at all.  If a master repository exists then derived works (i.e. special grass/paving variants) can always be keyed back to the masters - which need to be stored at high colour resolution given how the .fsh process is lossy. Maarten's US/EU road repositories are probably the best place to start - there are almost certainly others with NAM Masters too. The repository could be the public master copy and should become the basis for derivative works.

However, right now I see all of this as fodder for thought and discussion - it wont come about overnight, and in any case,  I think more people should try out the tools and methods to see what else we have missed. The product is reasonably stable, is being tested all the time but isn't finished. Also there's NAM33 to get out of the door...

Questions are welcome anytime.

mgb204

#9
Progress continues on the main repository, in the meantime I've put together a small preview for the RTMT textures so people can see how this all works. You can find this small repository here, note that the in-development version of GoFSH is required to utilise this which can be found here. You can download GoFSH on the STEX, you need the latest version (May 16) or newer for the scripting capability.

I've cobbled together some basic instructions for how to do this, although the process is quite intuitive, expect more detailed documentation with the final product.

:update:

- Link to release version of GoFSH now included.
- Updated link.

GDO29Anagram

I have a few requests:

1. Proper handling of wealth 7 textures. As it stands, 0 and 7 (and in most cases, 4-6) are all sharing the same exact thing. I've had an idea of integrating proper wealth 7 textures by making them work the same way as 1-3 wealth textures: dirt sidewalks.

As it stands, integrating that feature, as far as I can tell, may entail commandeering one of the colour channels for wealth 7 and adding another separate alpha for wealth 0.

2. This goes hand in hand with wealth 7 textures: selectively choosing which wealth textures to generate via checkboxes; IE, one for wealth 0 by itself, one for wealth 1-3, another for wealth 4-6, and one for wealth 7. That way, one could choose to generate wealth 0 and wealth 7 only, or any other combination of the four.
<INACTIVE>
-----
Simtropolis | YouTube | MLP Forums

mgb204

#11
Quote from: GDO29Anagram on May 16, 2015, 08:27:44 PM
I have a few requests:

1. Proper handling of wealth 7 textures. As it stands, 0 and 7 (and in most cases, 4-6) are all sharing the same exact thing. I've had an idea of integrating proper wealth 7 textures by making them work the same way as 1-3 wealth textures: dirt sidewalks.

As it stands, integrating that feature, as far as I can tell, may entail commandeering one of the colour channels for wealth 7 and adding another separate alpha for wealth 0.

Just off the top of my head (Ron would be able to give you a better answer), this is difficult to achieve, certain exceptions to the general accepted rules of wealth have to use workarounds to process correctly. Ultimately such automation must work by making certain assumptions, however sufficient options exist to have enough flexibility to work around such issues, here's an example of that in action:



So here is a regular SAM1 texture (C0) and the definition (D0) (blue will be grass for 1-3, black is sidewalk). However, this will assume the 0 Wealth is blank with no grass (as is typical), but for SAM1 all the 0 wealth textures should have full grass instead. The Additional D0 was generated by simply removing both the Green and Blue channels from the original D0, this can be bulk automated with Batch in Photoshop for a set of Alphas (I guess similar automation exists in other programs too). I simply generate the usual 0-3 textures with the first D0, then overwrite just the 0 Wealth textures (in the same script, running later) with the new textures generated with the Additional D0's.

Quote2. This goes hand in hand with wealth 7 textures: selectively choosing which wealth textures to generate via checkboxes; IE, one for wealth 0 by itself, one for wealth 1-3, another for wealth 4-6, and one for wealth 7. That way, one could choose to generate wealth 0 and wealth 7 only, or any other combination of the four.

I believe the scripting system has sufficient flexibility to do this right now, but so long as you are comfortable with files/folders again it's possible to have as much control as most users would need. Automation will always have some limitations ultimately, I've been striving to find a solution to this for a year, I had done pretty well, but my systems had two flaws, one relying on having a "blank" template texture to automate adding the sidewalk layer and secondly having photoshop to process them.

I'm very excited at the developments Ron has managed to realise here, right now I don't know of a quicker way to do this. GoFSH is the Swiss Army Knife of texturing now, oddly though it seems to have been overlooked by many, I wish I'd known about it a year ago when almost every forum post pointed me towards using PNGs and certain other tools. One of the key benefits of GoFSH is the quality of the finished textures, great efforts were made to keep everything at the highest quality whilst keeping file sizes efficient.

rivit

Ganaram, thank you for your feedback -

   both of the items you mention are possible but will have to go on the todo for now.

1. The wealth 7 texture has not been tackled except in theory at the moment. Right now its the same as Wealth 0 but can be filled with a different dirt. Because this is not a common requirement as yet I had contemplated adding an ALPHA (real one) to the D0 explicitly for this (applying the same rules as the other components) when the need arose. Doing this will affect the display of such textures.  I have used special alphas to make rural textures but have done these as a separate run with common C0 and A0. An exception thus.

2. Same goes for the 0,1-3,4-6, 7 choices. They're not usually separates so this is why the options evolved as 1-3,1-6 and 1-7. All quite possible and will only be implemented in Scripts (which are the way to use the C,D methods)

Ron

mgb204

#13
After 3 months of hard work, I am finally getting ridiculously close to finishing the automated sidewalk repository for the NAM. I've checked every piece that can be plopped and dragged and am pretty confident that if a texture currently exists with baked-in sidewalks, it can be taken care of.

Over 2000 replacement textures take about 10-12m to process, when finished you will find your sidewalks and grass on the following networks:

- Pedmalls


- MHW Interchanges

- FLUPs


- RHW Interchanges

- TuLEPs (choose from one of 6 TuLEP variants)


- Viaduct Rail

- Draggable Road Viaducts


- URail

- Turbo Roundabouts


- El-Rail over Road, Ave and RD-4 Networks

- Complete unified GLR set - that's GLR Drag & Puzzle, Rural GLR, TiA, TiR, ToR, TiS and even the extended dragable GLR too.

- RTMT / NAM station textures


- Other misc. textures.


Options to generate a set for either US or EU textures, both with either LHD or RHD support and options to cover Maxis, Peg 205 and RRW crossings exist (Note: rail crossing support is Maxis only for GLR).

To make your own set set is very simple, download my resources (200MB+, zips nicely to 20MB though) and a copy of GoFSH (needed to process them). Add your custom textures to GoFSH (it can handle multiple sets if desired), then simply point the program at one of my included scripts and click Run. When finished, just copy the newly generated folder containing your custom mod into your plugins directory. I will include step-by-step instructions for finding/copying any existing textures from your game for this purpose, absolutely no modding experience is required, everybody can have a custom NAM if they want it.

Here are some more examples of those textures in action, here I've used a custom sidewalk from a texture by GTC and Terrain matching grass:





This is a tiny fraction of the whole that I'm able to show off here, a few textures fixes and finalising the documentation is all that's left to do right now. If you've one of my TGN mods installed, combined with the final output you should have a complete set of NAM textures (excluding RHW) that match your grass in addition to whatever custom sidewalks you might be using.

APSMS

This. Is. Awesome! I can't wait for you to release this. :popcorn:

Making a modding tutorial is hard enough but a package that helps us do it all ourselves? Priceless.  :thumbsup:
Can someone send him a K-point?
Experience is something you don't get until just after you need it.

My Mayor Diary San Diego: A Reinterpretation

eggman121

Quote from: APSMS on June 21, 2015, 07:23:03 PM
Can someone send him a K-point?

Done  :thumbsup:

Again great work mgb  ;D

-eggman121

mgb204

#16
And for anyone eager to get their hands on it early, here's a preview release.

Currently the documentation is complete, but needs some work. Some texture issues may exist, but it's 99% there I'm certain.

:Edit:

- And the first derp appears... the developer-intended script "BuildAll" contains errors, please replace it with the attached version. The regular scripts are all fine.
- Updated Version contains fixed script (Oct 20). No changes, I just needed a new link since I moved the file.

noahclem

Congratulations on getting all this to the point of release! Looks great and is a huge achievement &apls

trabman11

Looks like I'll be saying goodbye the the Japanese FaceLift mod very soon  :crytissue:. Great work so far!  &apls
RISE UP AND FIGHT WITH SUPER BUNNY AND BOB!!!
  (\_/)           ☻/       /  \       
\(='.'=)/        /▌        |☢|       
(")_(")         / \       //||\\
*whispers* ...For what I don't know...I think they are anti-Google+ anarchists  ???

mgb204

Quote from: trabman11 on August 06, 2015, 04:11:35 PM
Looks like I'll be saying goodbye the the Japanese FaceLift mod very soon  :crytissue:. Great work so far!  &apls

Don't throw it all away just yet... from the beginning I've been using this mod, so most of the work I do compliments it rather than replacing it's functionality. This automation will not cover the original Ploppable viaducts for example, eventually those may be relegated to legacy support in the NAM when the draggable replacements are better supported. I've also yet to work on the WRC/FAR pieces, for much the same reasons, since the draggable versions simply don't require a mod for sidewalks, it seemed redundant. Once I get to the point of official release I will document which components of the Jap NAM FL you might still require in more detail, but by and large it does make a lot of it unnecessary.