środa, 2 października 2013

Activity duration distribution - how to get it?

Problem:

We got distribution for 'time-of-day' choices for Home-Work trips and Work-Home trips. What we need is distribution of work-time duration. We needed it to populate activity chains. Here's the solution which I particulary like as a smart way to solve complex issue and exploit sources that are available.

 Background:

Our Uni (Cracow University of Technology) jointly with (Poznan University of Technology) gave another try for activity based models (it's a broader project incorporating MATsim + electric taxi dispatching, etc.), but our small part is to create damand input for agent-based model (MATsim).
What we had is an out-dated survey made for activities (year 2000) and quite recent classic travel survey. We need to make use of this as seamless as possible.

 

Input:

 
  

Algorithm:

Let’s define following linear program.
Inputs are HW and WH time-of-day vectors and variable is the duration distribution vector Dw.
Let’s define the mapping m(HW, Dw) à WHt  as follows:

a)      let’s calculate cross product of vector HW  and transposed Dw shifted left by respective travel start time t(HW(i)):    WHt = HW×((Dw + t(HW(i))T

b)      Then compute output: theoretical  WH vector, as column sum of resulting matrix WHt

c)      Objective function is likelihood between  WHt  and WH (either r2, either MRSE, or log- likelihood, whatever).
 

Solution of this problem gives results as follows:

 Summary:

We got what we needed: duration of time spent at work, from weel founded input that we had. We didn't need to make extra surveys - we just exploited what was available.
 

 



środa, 15 maja 2013

Accessibility as a measure of distance in transport model - distance = seconds, not meters

Mapping is almost always done with distances, but mapping areas based on travel time between them can be much more valuable information.
Results of modeling are usually given in flows, travel times over network elements, or heat maps. But sometimes it'd be nice to show different dimension.
Belowe we can see Małopolska Region in Poland, but from three differen perspectives. First is normal map in euclidean space. But then we see completely different view on the same area. Distance from towns to the capital - Cracow, is proportional to travel time in Public Transport. 3rd picture shows the same but for three regional capitals competing with each other.


It was done with simple mathematical operations based on PTV Visum data model with elementar python script. Script simply loops over each zone and each skim matrix Value of Perceived Journey Time. Contact me at info@intelligent-infrastructure.eu to receive free support with such a script.

wtorek, 9 kwietnia 2013

Multiple Visum licences -> Multiple COM Objects

Problem:
Well, that's kind of a problem when you have many Visum licences and you want to use them variantly via COM.
By default latest installation is available under: ("Visum.Visum") key in registry and any new installation of Visum overwrites previous keys.
When you have may licences (Uni, Commercial, Teaching, etc.) it becomes confusing.
There's an option "Register as COM server" under "Options/Options" in Visum GUI - but it doesn't work for me.
So what's the solution?:
You simply run your Visum125.exe with '/regserver' flag from the command line - that's it. Below I created a small wrapper for my multiple licences:

import os   
def register_Visum(nazwa):
    def regVisum(path):
        command = "cd " + path + "& " + "Visum125.exe /regserver"       
        print command
        os.system(command)
       
    licencedict = {"edu_GN": "C:\Program Files (x86)\PTV_Vision\VISUM125_student_GN\Exe\ " ,
            "edu_45min": "C:\Program Files (x86)\PTV_Uni\VISUM125\Exe\ ",
            "PK": "C:\Program Files\PTV_Vision\VISUM125\Exe\ "}
   
    regVisum(licencedict[nazwa])
 
register_Visum("edu_GN")
register_Visum("edu_45min")
register_Visum("PK")


It will save a lot of my time and nerves now
PS. Another issue would be to register them under different names, ie. "Visum.Visum.Uni", "Visum.Visum.Commercial" - but I believe that's compiled within dll / exe and I won't have access to this, unless PTV exposes it.

środa, 20 marca 2013

why python for Visum scripting - short answer

that's why

task:
check if UDA exists

code:
def CheckAttr(obj,attr):
       return attr in [e.Code for e in o.Attributes.GetAll]


usage:

CheckAttr(Visum.Net.Links,"ToNodeNo")


with VB, or anything lower level it'd be at least 20 lines of code :P


Update - the same with VB:

Sub UpdateVisumUDAs(ByVal tableName As String _
   , ByVal keyFields As String() _
   , ByRef ds As DataSource _
   , Optional ByVal CreateUDA As Boolean = True _
   , Optional ByVal ValueType As Integer = 5)


Dim o As Object = ds.StringToVisumObject(tableName, ds)
Dim Attributes As Object()
Attributes = o.Attributes.GetAll
Dim AttrCodes As New List(Of String)
For Each Attribute As Object In Attributes
   AttrCodes.Add(Attribute.Code.ToString.ToUpper)
Next
For Each key As String In keyFields
   If CreateUDA AndAlso Not AttrCodes.Contains(key.ToString) Then
   o.AddUserDefinedAttribute(key.ToString, key.ToString, key.ToString, ValueType)
End If
Next


PS. Don't take me as an ignorant - I know more or less pros and cons for python vs. any 'proper' programming language, but I just love the way you can do things in python


czwartek, 24 stycznia 2013

Change Node/Link/Zone Numbers in Visum via COM + limit of Visum

I believe anyone who has created once Visum network out of big extrenal datasets (mainly GIS - .shp) came across problems with numbering duplicates. Here I will show you how to overcome those issues. There are two ways: one is scripting (snippet below), another is via Access database.

Case:
We have model of Kraków (city ca. 1mln inhabitants), we are creating regional model of Małopolska - Kraków is it's capital.
We have detailed GIS network from regional municipality, we have our own Kraków model with Visum network. We want to merge them together.
In most cases we will have problems with duplicate keys in database (i.e. "Node with ID 301 already defined in the network"). That's mainly due to very decent DB architecture behind visum data. Each network object belongs to specific table with well defined 'primary key' and 'foreign key' - which basically cannot have duplicates (http://en.wikipedia.org/wiki/Foreign_key).
It goes like this:
  • each node is identified by it's NO
  • each link is identified by its FromNodeNo and ToNodeNo (those are foreign keys, plus implicitly it has it's own primary key NO - I assume)
  • each turn has its FromNode, ViaNode and ToNode
If during import of any external objects those numbers will get confused we get rubbish results (either all network is mixed up, either it's not imported at all.

Toy-network example:

1. I created following network and exported it to .shp:


Network no 1


2. I created another network like this:
Network no 2
3. I tried to import network 1 into network 2:
a) result without setting "Offset parameter":
No network import at all.
b) when You set "Offset" to a number greater than max NodeNo, you will get something
first you get error for some links (which were already defined in network no 2)

and finally the resulting network will be as follows:
network 1 imported into network 2 (marked added links) all together network makes no sense
As we see there's a lot of confusion.

Solution:

there are two methods: one is scripting, another is via Access DB.


Scripting is easier and faster and can be automated, but it doesn't work for many objects (it works for nodes, doesn't work for links). See examples below(all scripts in python):

a) Bulk (didn't work with older versions of Visum) - much faster:

def get_Nodes(Attr):
       return Visum.Net.Nodes.GetMultiAttValues(Attr)


Nodes=get_Nodes("No")
Nodes=list(Nodes)
Nodes=[list(node) for node in Nodes]
for i,node in enumerate(Nodes):
       node[1]=i+100000000 #put max node no in your network here

# we need to do it twice: once we add big number to all elements and the we can give them ordered numbers starting from whatever
Visum.Net.Nodes.SetMultiAttValues("No",Nodes)

Nodes=get_Nodes("No")
Nodes=list(Nodes)
Nodes=[list(node) for node in Nodes]
for i,node in enumerate(Nodes):
         node[1]=i+1000000 #give your threshold here (nodes will start fron No=1000000

Visum.Net.Nodes.SetMultiAttValues("No",Nodes)


b) element by element - can be very slow:

Nodes=Visum.Net.Nodes.GetMultiAttValues("No")
Nodes=[node[1] for node in Nodes]
for node in Nodes:
       Visum.Net.Nodes.ItemByKey(node).SetAttValue("No",node+
100000000 )
       i+=1
for node in Nodes:
      Visum.Net.Nodes.ItemByKey(node+
100000000 ).SetAttValue("No",i)
      i+=1


2) Access DataBase:
I'm not ver familiar with Access so just a picture of what I do:
a) Export network into Access DB (can take a lot of time)
b) Open access file and edit tables
c) Change keys in tables (for links it's good to change name of "No" column into anything else and create additional column named No, then set this column as a key.
d) make the new column "Autoincrement" - so the number will be integers starting from 1
(for links it needs to be different, links have directed numbers so it shall be like this: 1 1 2 2 3 3 4 4 5 5 6 6 ... - I did it in excel and pasted.
e) you save the file and Import it into Visum

This way we merged four different shapefiles with overlapping numbers and two different Visum networks.
Everything is neatly numbered starting from one onwards.


PS. We came across another issue which seemed strange but we found very easy explanation. During import we got following message that node number exceeds: 2 147 483 648. Which happened to be max of LongInt datatype, so actually that's the limit of Visum :) You cannot go any further. Simply reduce umbers by means of the solutions above.

Hope it helps!




niedziela, 6 stycznia 2013

Geocoding places from paper traffic surveys

Long time since the last post - it doesn't mean however that i^2 was relaxing and playing guitar on the beach, contary: big projects - interesting, challenging, innovative.

I hope soon at least few more posts will be published.
Now just snippet from small bit I was playing with yesterday. I'm pretty proud of how clever complex issues can be solved ;)

Problem:
To geocode origins and destinations from their names and place them on the map.
Background:
We got survey results in xls files. There were around 5000 surveys from Małopolska (region in Poland with Kraków being it's capital) - respondents were asked to specify their origin and destination (classic OD survey). Perfect - but what's next? XLS has infinite capacity for data mining, but not for me - I just don't get it: why don't you program if it's that much easier, elegant and straighforward than horrible "vlookups" in excel... Nevermind: I created nice SQLite DB with neat primary, foreign keys, indexes, etc. - works quite fast. But for those trips we needed details: distance, where it started, what country, which województwo ~= county, etc. Adn that was the major problem to be solved. Solution 1: place it on the map, and for each trip determine it's distance (yuck!) Solution 2:
Solution:
~2hrs of python coding and full info on 5000 trips gathered.
a) xlrd to get all the trips data into SQLite
plik = xlrd.open_workbook(path)
arkusze = plik.sheet_names()
for nazwa_arkusza in arkusze:
   arkusz = plik.sheet_by_name(nazwa_arkusza)
    for rownum in range(9,arkusz.nrows):
    rzad=arkusz.row(rownum)
    

b) use python set() to define list of unique places observed during survey
Places.add(str(rzad.value).upper())
c) create 3 tables in SQL: 'places' , 'relations' , 'trips' . relation has two foreign keys from places and trip has one foreign key to relation
d) use googlemaps.pyhttp://pypi.python.org/pypi/googlemaps/ ) very easy, but clever library to use googlemaps data in your code (it uses json to gather data, and probably can be easily expanded to cover more queries):
for Place in Places:
   punkt=gmaps.geocode(Miejsce)
   lng, lat = punkt['Placemark'][0]['Point']['coordinates'][0:2]
   country=result['Placemark'][0]['AddressDetails']['Country'] ['CountryName']
   wojewodztwo=result['Placemark'][0]['AddressDetails']['Country']['AdministrativeArea']['AdministrativeAreaName']
   city=result['Placemark'][0]['AddressDetails']['Country']['AdministrativeArea']['SubAdministrativeArea']['SubAdministrativeAreaName']


e) to get distance I used the same  googlemaps.py with different query:
for relation in relations:
         result=gmaps.directions(relation.from,relation.to)
    dystans = result['Directions']['Distance']['meters']

Results:
See two screenshots below:
1) before (xls file)



Original xls file with place names

2) after (Visum network with origins/detinations placed in proper geocoding)
Geocoded places on Visum network
PS. It shows olny the bright side of the solution. There were many problems with proper geocoding (i.e. first row in DB was: 'Kosocice' suburban area in Kraków, which was suggested by google to be "Kosice" city in slovakia ~400km away. And such problems were numerous. Next: there is around 50 places in PL named: "Nowa Wieś" (means new village in polish).  etc. etc.


piątek, 20 lipca 2012

Plate number survey add-in for Visum


From raw input files to Visum OD and travel time matrices.



 i2 - Intelligent-infrastructure was working hard last months. We struggled to integrate powerful SQL engine with Visum. We struggled to process errors, to interpret results properly, to create user-friendly gui and we come back even more experienced. Below you can see latest development: From raw plate number results to OD matrix in Visum- that's what was definitely needed. Have a look!

1. Idea

Plate Number survey (APNR) is always a chance to improve your model. However it always comes along with data processing problems - thousands of records stored in numbers of files and all need to be processed to gather information. That's why we integrated data processing within VISUM. Now all standard steps between APNR and OD matrix in Visum are automated in user friendly graphical interface.


2. Functionalities

APNR Support will support you with every step of APNR data processing:
1. Flexible data importer will create SQL database from your records.
2. Powerful database engine will organize results
3. Full Visum integration will import counting points' locations
4. Filtering engine will show data you need (i.e. list of truck crossing two count locations during morning peak hour).
5. Data processing machine will calculate OD matrices with several error detection procedures.
6. You will be able to export travel time skim matrices, paths, OD matrices to Visum.
7. You will see your results on histograms and charts.


fig. 2. flexible filtering engine showing you whatever you need to know

 3. Summary

- take advantage of integrating APNR data and VISUM network model in one flexible Add-In
- speed up your calculations with highly efficient database engine
 - work with user friendly GUI to simply see the data in tables, lists and plots or export ones to Visum and Excel.

APNR created by i2 runs as a script from Visum. It uses data from Visum network and APNR data to provide not only advanced queries to APNR database but also calculation of the characteristics based on Visum network and Count Locations.

fig.3. graphical presentation of observed travel times histogram for OD matrix cell

Contact us at info@intelligent-infrastructure.eu for more details. Visit our website at http://intelligent-infrastructure.eu

PS. Project was done with great help and encouragement from Dipl.-Ing. Timotheus Klein from ARGUS Hamburg, who is co-author of the project.