new traffic experiments

[z]

Commute time shows around 40 minutes, which I thought was odd too. I don't have out of towners so I thought the numbers were supposed to be somewhat accurate still.
I used a multiplier of 1 so I thought I should get 1 to 1 for my commute time units. Obviously this is not the case since my max commute is 24. Also as there is no congestion whatsoever and we are cruising at network speed x 1.3 I don't see why the commute time would be long.

Now the internal documentation doesn't make a lot of sense on it. I can't remember it offhand well enough to quote it. As is common, it is an 8-bit register, and that is why we see that number we see so often in the computer world mentioned, 255 (count 0 as a number and we get 256).  At first I thought what was meant was that it would go up to 255 minutes. So one could divide 255 by their max commute time and use that number as a multiplier. That doesn't seem to work. I thought let me just take it as the raw number and leave it at 1. That doesn't seem to work either.

Based on the documentation you're quoting, it appears that you're making the mistake that RippleJet points out.  The game never looks at that property in the simulator.  I already pointed you to the explanation of the commute time graph which is in this post; it sounds like you read part of it, but you may want to reread it.  The commute time graph can be made perfectly accurate for a region containing a single city, but you've got to figure out by experiment what the proper scaling factor is, which must be done by experiment for your particular traffic simulator.  There may be a formula for determining the scaling factor, but I've never seen it.  To quote Tropod:

About the Commute Graph; ignore it. It serves no real purpose really, especially more so if you have neighbour connections. The fact that there might be such a low Commute according to the Commute Graph, is pointless really, as it is by no means accurate nor indicative of the actual Commute Times undertaken by individual Sims. All it is, is a city average.

[ldog]

If you by that mean the property Trip Length to Minutes Display Multiplier,
then I believe several tests have shown that it isn't used at all.
Whatever value you give it, the multiplier stays at 25.

Based on the documentation you're quoting, it appears that you're making the mistake that RippleJet points out.  The game never looks at that property in the simulator.  I already pointed you to the explanation of the commute time graph which is in this post; it sounds like you read part of it, but you may want to reread it.  The commute time graph can be made perfectly accurate for a region containing a single city, but you've got to figure out by experiment what the proper scaling factor is, which must be done by experiment for your particular traffic simulator.  There may be a formula for determining the scaling factor, but I've never seen it.  To quote Tropod:

That would answer that part. I am sure I do need to reread that. I think I understand better now, that is what the seperate traffic exemplar I asked you about the other day is for correct?
I was starting to catch on since you and Jason both left it at 25.

The thing is I thought I also read that even though we cannot trust the numbers that the shape of the graph was still somewhat accurate, at least until we get intercity commuters.
Or like I was saying in the other reply, the relative values should still have some kind of accuracy? At least in relation to each other.
eg. the commute time, whatever it is, goes up..we can trust it that traffic is getting worse...we throw down a highway and now the commute time goes down, because we just provided a faster route.

Are you saying we can't even trust it when it indicates the above?

What Brian was refering to is my prior post right before the one Ripplejet is answering.
I guess I shouldn't make posts right after each other, I was trying to keep unrelated information in a seperate reply.
First one was a seperate test, second one is first impressions to the bigger test I just started.

[b22rian]

What Brian was refering to is my prior post right before the one Ripplejet is answering.
I guess I shouldn't make posts right after each other, I was trying to keep unrelated information in a seperate reply.
First one was a seperate test, second one is first impressions to the bigger test I just started.

  Naw you did fine IDog-- it was my fault really for being lazy and not putting the relevant quote I was replying
to in my posting.. i do appreciate both tage and steve's replies of course.. But I was hoping to get a reply back
from Steve , concerning the experiments you and I are running  as to whether or not the commute times are
calculated only based on the morning commutes rather than some kinda average of both the morning and evening
commutes.. In the latter event it would have to be than the "buggy" commute time graph which they are referring to in the above posts,  which would be calling the confusion on this..
OK, now as long as you trust these regular queries which show the
  (quantitative commute lengths.. the short, medium, and long commute trips) early indications just from checking through some of the commutes in my largest city do seem to indicate that the traffic sim does in fact
calculate the commute trip lengths based on the morning commutes only.. Although i didnt find enough to test
to be absolutely sure of this.. More precise testing is needed unless Steve already know the answer to this ?

Thanks, brian

[z]

I think I understand better now, that is what the seperate traffic exemplar I asked you about the other day is for correct?

Yes.

The thing is I thought I also read that even though we cannot trust the numbers that the shape of the graph was still somewhat accurate, at least until we get intercity commuters.

Yes, as a citywide average.

Or like I was saying in the other reply, the relative values should still have some kind of accuracy? At least in relation to each other.
eg. the commute time, whatever it is, goes up..we can trust it that traffic is getting worse...we throw down a highway and now the commute time goes down, because we just provided a faster route.

Are you saying we can't even trust it when it indicates the above?

With only a single city in a region, what you say is generally true.  But at one point, Tropod says that adding mass transit tends to make commute times go up.  He doesn't elaborate, but I have always found his analyses to be extremely accurate.  If this is indeed the case, then it certainly does cast even more doubt on the usefulness of this graph - which is what Tropod said in my previous quote.

I guess I shouldn't make posts right after each other, I was trying to keep unrelated information in a seperate reply.

In general, double posting is heavily discouraged on this site.  I think there's a rule regarding it, but I can't find it right now.  Double posting is defined as two adjacent posts in the same thread on the same day.  Instead, people are encouraged to simply edit their first post.  I think it's even acceptable to repost it, if you want people to see your new addition.  The fact that I can't find this rule certainly shows that such a rule is easy to miss, but now you know.

[RippleJet]

In general, double posting is heavily discouraged on this site.  I think there's a rule regarding it, but I can't find it right now.  Double posting is defined as two adjacent posts in the same thread on the same day.  Instead, people are encouraged to simply edit their first post.  I think it's even acceptable to repost it, if you want people to see your new addition.  The fact that I can't find this rule certainly shows that such a rule is easy to miss, but now you know.

Actually, there is no such specific rule... ...just this general one:

SC4D does not allow inappropriate member conduct on the forums. This includes, but is not limited to: posting spam, posting hostile comments, "flaming" of any sort, posting simply to increase post count, and so on.

As long as "enough" time has elapsed from the previous post, and considering whether the content in the new post provides new (as opposed to additional or supplementary) information, I'd say that Lenny's double posts in general have been according to the rules (otherwise I would have intervened by now). Maybe the "mini-update" earlier today could have been added to the previous post though...

Also, considering the lengths of some of the posts that Lenny (and Jason and particularly Steve) have made, it sometimes would even be better to break them up into shorter posts, which would be easier to grasp, one subject at a time...

[ldog]

  Naw you did fine IDog-- it was my fault really for being lazy and not putting the relevant quote I was replying
to in my posting.. i do appreciate both tage and steve's replies of course.. But I was hoping to get a reply back
from Steve , concerning the experiments you and I are running  as to whether or not the commute times are
calculated only based on the morning commutes rather than some kinda average of both the morning and evening
commutes.. In the latter event it would have to be than the "buggy" commute time graph which they are referring to in the above posts,  which would be calling the confusion on this..
OK, now as long as you trust these regular queries which show the
  (quantitative commute lengths.. the short, medium, and long commute trips) early indications just from checking through some of the commutes in my largest city do seem to indicate that the traffic sim does in fact
calculate the commute trip lengths based on the morning commutes only.. Although i didnt find enough to test
to be absolutely sure of this.. More precise testing is needed unless Steve already know the answer to this ?

Thanks, brian

I know, I was just snapping that off before I left work so I got into a rush.
Ripplejet answered the second posts question and Steve confirmed and expounded on it. Reading between the lines a little he also says "Hey dumbass, we went over this already, I thought you did your homework" ...I did it, I just didn't understand it lol. I think I grasp it now. At least the part Ripplejet very clearly explains. Of course some of why I ask stuff that was explained in other posts is because Steve really does explain pretty well what he is doing, but without going into specifics.
What I really meant to say was, "Hey Steve, I hope you saw the post before as well, cause I'm looking for your commentary on that as well"

Also I just didn't have time to reply to your prior post (not going to requote it) before I ran out the door but I think if I understand you correctly then I think it is a great idea.

@Ripplejet
Thanks for clearing the rules up.
Yeah, like I said I originally was going to make them one post but then I thought for clarity I should seperate them.
Sometimes it is a hard call to make about when to break up a post or not.
Hopefully I will get into a rhythm eventually where I am not posting all day and in 20 different directions at once.

uhh almost forgot @ Steve thanks for clearing those up. What do you think of the Omnibus thing? I know we talked a little about it the other night but I don't know how coherent I was.

[z]

@ Steve thanks for clearing those up. What do you think of the Omnibus thing? I know we talked a little about it the other night but I don't know how coherent I was.

What do I think about it?  Well...

The "Commute Times Ten Bug" appears to be intentional on the part of Maxis.  They didn't really want people to know that Sims were limited to only three minutes to get to their jobs.  Now multiply that by ten and you still have on ly 30 minutes.  But throw in the neighbor connections and their effect, and that's enough to sufficiently confuse everyone.

The "Pathfinding Engine Shortcomings" are what happens when you don't use Perfect Pathfinding (by definition).

As for the "Unrealistic Mass Transit Speeds," the goal is to make the results of the game look realistic, and the internal numbers are of secondary importance.  But coincidentally, Simulator Z does take the basic advice here, although not using the exact numbers.  Cars are now faster than buses, and rail speeds have been reduced.  This was done because it produces a more accurate simulation, and a better-running simulator.

[z]

The commute time is always counted cumulatively (door-to-door); the pathfinder wouldn't be able to compute the best paths otherwise.  This was proven conclusively by RippleJet's experiment a year ago.  In my own experiments with the commute time graph, I also verified this.

The problem with MT's making the commute times rise is that the pathfinder is always supposed to find the fastest path (and with perfect pathfinding, it does).  The only explanation I can think of for a rise in commute times due to MT use is yet another bug in the commute time graph, which would not be surprising.

[RippleJet]

For whatever it's worth... when we tried to cure the prop pox earlier this year, we decoded parts of the savegame format (SC4 files).
The commute path for each lot is saved in the end of the Lot Subfile (Type ID = 0x09BD5D4A), with the following structure:

DWORD	Count of Commute Blocks  (number of destinations commuted to from here)
DWORD	Count of Commute Paths  (usually 0x00000002 = morning and evening, 0x00000001 for freight)
DWORD	Count of Bytes in the Path
BYTES	Commute Path  (1
DWORD	Number of Commuters
BYTE	Unknown, 0x00 or 0x03
BYTE	Unknown, 0x00, 0x01 or 0x02
BYTE	Unknown, 0x08, 0x09, 0x0A or 0x0B  (0x00 if the commute path has disappeared)
BYTE	Unknown, 0x00 or 0x03  (0x04 if the commute path has disappeared)
SINT16	Commute Destination, X Tile  (2
SINT16	Commute Destination, Z Tile  (2
FLOAT32	Trip Length
DWORD	Unknown  (3

1)
Structure of Commute Paths:

BYTE	Starting Traffic Type  (see below)
BYTE	Path Start Tile X (network in front of the lot)
BYTE	Path Start Tile Z (network in front of the lot)
	Commute legs, repeated for each transit switch
BYTE	Transit Switch 0x4T  (T = Traffic type, see below)
BYTE	Length of Path (Count of quads)
QUAD	Path Direction  (0 = West, 1 = North, 2 = East, 3=South)
QUADS	00b  (repeated 0, 1, 2 or 3 times, in order to fill the last BYTE)

Traffic types are in the same order as in the Traffic Simulator:

• 0x00 = Walk
• 0x01 = Car
• 0x02 = Bus
• 0x03 = Passenger Train
• 0x04 = Freight Truck
• 0x05 = Freight Train
• 0x06 = Subway
• 0x07 = Light Rail
• 0x08 = Monorail

2)
Destination Tile

A job located in a neighbouring city has the XZ coordinate of the border crossing.

• A job located in the western neighbour has an X coordinate of -1 (0xFFFF)
• A job located in the northern neighbour has a Z coordinate of -1 (0xFFFF)
• A job located in the eastern neighbour has an X coordinate of 64 (0x0040), 128 (0x0080) or 256 (0x0100), depending on the size of the city
• A job located in the southern neighbour has a Z coordinate of 64 (0x0040), 128 (0x0080) or 256 (0x0100), depending on the size of the city

If the commute path has disappeared (e.g. work demolished or redeveloping), the X coordinate becomes 0xDFA8 (-8,280).


3)
Unknown

For residentials; often 2, but I've also seen e.g. 9 and 11. Always 16 for industrials.
If the commute path has disappeared (e.g. work demolished or redeveloping), this DWORD becomes 0x0012FC70.

[sumwonyuno]

...

QUAD	Path Direction  (0 = West, 1 = North, 2 = East, 3=South)

...

Well that validates my findings earlier in the Query.txt thread.  I wonder why an entry of four possible values needs a quad for storage.

[RippleJet]

A quad is two bits, which can take only four values, 00b, 01b, 10b and 11b.
Four quads make up a BYTE, hence it's naming.

[sumwonyuno]

Ahh, thanks.  I was thinking of quad words (I need to be more specific on terminology).

[catty]

An awful lot got lost last year when Simtropolis was hacked, have a read of this topic in the Omnibus (unfortunately the links its referring to I haven't been able to locate anywhere, but it does show the testing they were doing)

http://www.simtropolis.com/omnibus/index.cfm/Main.SimCity_4.Commute_Time_and_Pathfinding_Report

I continued to have a look around for the links this report was referring to, and found information concerning this report and other problems on the official SimCity site in the BBS from the @the7trumpets

I am one of the people who did a lot of work testing the commute engine, and compiled the work of simtropolis members in the 'commute time and pathfinding report' which described many issues in the commute engine.

http://simcity.ea.com/bbs/messages.php?&openItemID=&threadID=0f9d1749cdef413a65f10da4ebbd5044&directoryID=261&startRow=1#49308a9b642b1988479cb0a79a33c5cb

continuing to have a rummage around the old posts, there are a number of bugs being reported including this one

Lastly, the commute time that is displayed in the query box for residential buildings has been determined to contain a bug. Once a building gets a 'long' commute time, it will always have it, even if all of the sims in that building were to get jobs across the street. Maxis is aware of the problem, but it is only visual. It does not actually affect the commute time.

just how relevant any of these old posts are       but it makes for some interesting reading 

[b22rian]

I continued to have a look around for the links this report was referring to, and found information concerning this report and other problems on the official SimCity site in the BBS from the @the7trumpets

http://simcity.ea.com/bbs/messages.php?&openItemID=&threadID=0f9d1749cdef413a65f10da4ebbd5044&directoryID=261&startRow=1#49308a9b642b1988479cb0a79a33c5cb

continuing to have a rummage around the old posts, there are a number of bugs being reported including this one

Quote
Lastly, the commute time that is displayed in the query box for residential buildings has been determined to contain a bug. Once a building gets a 'long' commute time, it will always have it, even if all of the sims in that building were to get jobs across the street. Maxis is aware of the problem, but it is only visual. It does not actually affect the commute time.



just how relevant any of these old posts are       but it makes for some interesting reading 

Oh this is very relevant Catty...
in that it throws a bit of a "wrench" into using the commute lengths displayed when you preform a "query", in terms of using that to ascertain whether length of commute times are a product of both the evening and morning commutes or whether the game just calculates all the morning commutes alone and than assumes the return commutes from work to be the same routes..
Thanks a lot for this Catty !

Brian

[z]

That's a real treasure, Cathy - thanks for bringing it to our attention!  I found a few especially interesting nuggets that I'll report on a little later on.

A word of caution - these posts were written before the release of Rush Hour, and the patch that followed that release.  So a lot of the bugs reported in the posts no longer exist.

[catty]

A word of caution - these posts were written before the release of Rush Hour, and the patch that followed that release.  So a lot of the bugs reported in the posts no longer exist.

  its still a bit like rummaging around in a old attic, you know most of its junk or out-of-date, but its also interesting history at the same time   

[z]

Oh this is very relevant Catty...
in that it throws a bit of a "wrench" into using the commute lengths displayed when you preform a "query", in terms of using that to ascertain whether length of commute times are a product of both the evening and morning commutes or whether the game just calculates all the morning commutes alone and than assumes the return commutes from work to be the same routes..

Neither, actually.  As mentioned above, the round trip commute times are always double the morning commute time; this was verified by experiment by ldog, and is mentioned in the Prima Guide.

A word about the Prima Guide:  It has many mistakes in it (I've found a few myself), but I'd say it's about 95% accurate, and it contains a huge amount of information about the game all together in one place.  I don't know of any other single source that does this.  I would strongly recommend it as a beginning text for understanding how the game really works.

I continued to have a look around for the links this report was referring to, and found information concerning this report and other problems on the official SimCity site in the BBS from the @the7trumpets...

As I mentioned before, this is really fascinating stuff - I wish I had more time to read it all.  At least as important as the posts from the7trumpets (one of the game's all-time greats) are the posts from MaxisFrank.  He was quite possibly the last Maxis employee to communicate officially with the SC4 community.

Since there's a lot of material in these posts, I thought I'd summarize some of the things that caught my eye that are particularly relevant to what we're doing now:

Everybody in these posts seems quite fed up with the Maxis traffic simulator, for reasons which should be quite obvious to us.  A suggestion is made by the7trumpets that Maxis use a pathfinding algorithm based on something like A* - the original simulator works so poorly that it wasn't obvious that that was what they were already doing.  MaxisFrank's response is apparently the first time the community found out that A* was being used.  From MaxisFrank:

We're already using a modified version of A*. The problem is that intersections as node based calculations won't work due to the insane number of possible nodes in a city. This is due to the fact that most buildings have jobs and thus are possible destinations (nodes). We considered this, but had to rule it out.

Unfortunately, he doesn't say anything more about what they actually do, but this should give some insight into Maxis' A* implementation.

The pathfinding heuristic was a topic of great debate even back then, and it appears that the7trumpets was actually the author of the first "perfect pathfinding" patch, a term that appears in these posts.  Specific values as low as .005 are mentioned, and it is likely based on other posts here that the7trumpets patch actually used .003, and that this value was determined experimentally.  I have not come across the exact specifics to support this, and I am not sure that they exist on this board.

The question as to why Maxis used a value of .09 is answered directly by Maxis Frank, though:

By the way, the reason why the heuristic is set at 0.09 instead of lower is for performance reasons. The lower you make this the smarter the traffic sim gets, but the longer it takes to complete a traffic cycle. If you've got a buff machine, drop it and be happy. If your machine is emitting smoke and screaming at you to make it stop, turn it up a bit

And an interesting response from toroca:

I can understand the pathfinding heuristic being set higher to speed up performance, but surely you could find a middle ground that allows for both good pathfinding and good performance? As it is now, that value is so high that pathfinding virtually doesn't exist. Sims do NOT take the fastest route to work in almost any situation; rather, they take the most direct, in terms of shortest possible distance. this results in ridiculously overused streets, crowded roads, empty highways, and underused mass transit systems.

Surely there's a better way to improve performance than by crippling the pathfinder? the7trumpets' pathfinding modd doesn't have THAT much of an impact on performance... I think the worst was a 45% slowdown, and that was only with the PERFECT pathfinding mod. In my case, the slowdown was MAYBE 15%, which is MORE than tolerable since it means sims actually look for the fastest way to work instead of the most direct. Few of my streets are overcrowded, and my highways are heavily used, for the first time since I bought the game.

[ldog]

So much good new info.
Thanks everyone joining in the discussion.
I'll have to go read that article.
I'm also glad to see a bit of the query.txt topic showing up here; I've been following that with fascination since Cathy brought it to my attention and thinking about how it relates to traffic.
Of course so has Steve who has already answered some of the questions that topic raised.

[b22rian]

Neither, actually.  As mentioned above, the round trip commute times are always double the morning commute time; this was verified by experiment by ldog, and is mentioned in the Prima Guide.

   Thanks Steve,
I knew you had mentioned it some place.. for some reason I was having trouble finding it.
Course it wasnt the answer myself or probably any of us were hoping for.. Obviously we would like the
game to me as accurate as possible.. But it is, what it is..

    I know for a fact i have routes where the return trips from work are different routes from those going to work.  I admit i do not know the % of the route discrepancies versus identical routes to and from home..
Also I suspect the distances of the route discrepancies would not vary much.. I suppose if there was a large
variance between them, than this would indicate a highly inefficient transit system 

thanks again brian ..

[ldog]

Course it wasnt the answer myself or probably any of us were hoping for.. Obviously we would like the
game to me as accurate as possible.. But it is, what it is..

Yeah, we are all a bit obsessed. Considering that little gem my A* research pointed me to on SC4s site, Maxis had much less lofty goals.
This was the first SC that even had pathfinding in the traffic engine, so they were happy just to add it, forget perfecting it.
If we were actually getting a SC5 we might get better.
Although honestly I think what the community has done with SC4 over the years it probably is still more than we would have gotten out of an SC5.

    I know for a fact i have routes where the return trips from work are different routes from those going to work.  I admit i do not know the % of the route discrepancies versus identical routes to and from home..
Also I suspect the distances of the route discrepancies would not vary much.. I suppose if there was a large
variance between them, than this would indicate and highly inefficient transit system 

Well the reason it was relevant to me was mainly because of the kind of things you could do with one way roads. Otherwise it is a fairly safe assumption they made that most people return home the same way they go to work. Jason being a notable exception Although...I admit even I take a slightly different route home generally. Of course this is mainly due to local congestion patterns. Sometimes it is just over a 5 minute difference but it is enough for me.  I guess as much as possible I try to use perfect pathfinding myself

Once again, for practical purposes it is probably irrelevant but something I found interesting (summary for those among us not interested in theory ).

Test mini-update for the day (last night's play actually). I didn't make a lot of progress. Was still noting anomolies. So I started a couple other tests (I had the sense not to slag the started one this time though). I still don't understand why I am seeing abandoned due to commute time when I have a highway and train system that has no congestion and the zones are far less than half a large tile apart. No Steve, I'm not using .003 for ph. Using .009 ...incidentaly 82x1.3=106.6 and 1/106.6~0.009 ... theoreticly if we find an open highway we stop looking for a better route. I still get a fair amount of train use as well. Even though I am still not convinced about .003 at some point I will switch it up and see if it makes a difference. I think some of my problem is trying to force such an unnatural growth and I wonder, if there actually is no job available for those sims does it still report due to commute time? I have not seen any commercial or industrial abandonment yet, just residential, mostly R$ , some R$$ but no R$$$ (paying better attention this time). Of course I am still far from saturating the train stations; adding more MT at this time would only make it worse. I've still got quite a bit of building to do before I get any kind of MT related results.

Anyway, in the middle of an assets audit at work so updates will be kinda sporadic from me this week.