The power of the OSM community

It all started one Saturday night sitting at a cafe with my laptop having a coffee 1000s of kilometers away from home. I decided to fire up JOSM (Java OSM Editor) and take a peek at the transit data of my hometown of Laval. The available transit data roughly looked like the image below. Out of ~45 routes only 1-2 were mapped and very incompletely.

I then proceeded the download the latest GTFS dataset (of the time) and pick out a trip of the route that I used to take every day to go to university and then later on in life to work - a route close to my heart. I pick out the stops a create a GeoJSON file to import into JOSM as a visual aid. That’s actually the 46S route from the image above. I read up on PTv2 to refresh my knowledge and start constructing/mapping out the route in JOSM complete with nodes and route ways. I map out one direction of the route and call it a day with ideas brewing in my head of eventually planning out and doing a full import as a walk to my apartment.

A few days later, I get a message from another mapper saying:

“Hello, I noticed that you have created in OSM the first bus route based on public_transport_v2 in Laval (line 46S) and would like to inform you that a project to add all the routes of the ARTM (including the STL) has been proposed.”

Enthused, I reply to the message and we start discussing on how to proceed. Eventually permission is granted by the STL to use the GTFS data and do the import. The import project was actually planned for all the operators and routes of the ARTM as a whole so permissions for each one was obtained. Within a few days, our team of 2 became 3 volunteers and a couple of weeks in a 4th mapper joined us.

To me, this project was a good testament to the great global community that is the OSM community; always ready to come together and create for the love of mapping and providing the world with important data and information.

General methodology used for the import

We split the mapping according to geographies and operators that we were most familiar with but also put up systems to standardize the way to present GTFS specific information that doesn’t necessarily have a direct equivalent in the OSM node-way-relation system summarized here. I wrote a Python script osm-gtfs-laval to automate the process of creating the XML file for import through JOSM. As per OSM import guidelines, the import was done using a separate import account. The script requires GDAL to run and roughly does the following steps:

1. Unzip GTFS archive and load data in to memory.
2. Using the calendar.txt file/data, it finds the latest service_id in order to ensure only the latest data will be imported.
3. Filters the whole GTFS dataset keeping only data that corresponds to the latest service_id.
4. Writes the GTFS stops to GeoJSON for visualization (to spot apparent problems and potential problems).
5. It downloads existing OSM data using overpy. This includes existing bus stops (nodes), route relations, and route_master relations along with the member nodes and ways. It converts the data into an internal format to be able to work with it in the script.
6. Handles conflation within the GTFS dataset stops.
7. Creates new OSM nodes for the stops making sure to not duplicate or delete existing stops.
8. For each trip (trip_id) for a given route (route_id), it finds the longest trip (the one with the most number of stops).
9. It creates OSM route and route_master relations using the identified longest trip respecting PTv2 requirements. It handles conflation with existing relations.
10. Write the JOSM XML files (nodes and relations)

It does not automatically add the ways in the relations. This is one limitation that would be quite difficult to solve programmatically in an automated way; the primary challenge being adding the wrong ways into the relation, specifically the way representing the road in the opposite direction of the trip. It’s actually a very interesting problem to tackle.

For the import, I created GeoJSONs for each route-direction combination and added them as a background layer to help with selecting the ways to manually add to the route relations. The pt_assistant JOSM plugin was actually a very big help to cut the end ways in a clean and easy way. In the video below, I show how to use the pt_assitant plugin to accomplish that.

The detailed description of the technical details of the script and import process are summarized in this wiki post. The script is quite rudimentary but I’m hoping to turn it into something more usable when I get a chance.

Animation of route relations (without way members) ready for import seen in GIF above!