-
Notifications
You must be signed in to change notification settings - Fork 3.5k
Cucumber Test Suite
The correct functionality of OSRM is supported by a test suite based on Cucumber (http://cukes.info). If you don't want to run tests, you can build and run OSRM without doing any of the following.
Latest test results can be seen at http://project-osrm.org/cucumber.html.
Cucumber itself is written in Ruby, and is distributed as a Ruby gem. All tests are written in Ruby using the natural language Gherkin syntax.
- osmosis
- ruby 1.9+: Some system come with 1.8, so you might need to update.
- gems: cucumber, rake, osmlib-base, sys-proctable, rspec-expectations. The easiest is to first install the bundler gem using
gem install bundler, then runbundle install` from the project folder. This will automatically install the needed gems.
vi /var/lib/gems/1.8/specifications/term-ansicolor-1.0.7.gemspec
and change the s.date = line to the following: s.date = %q{2011-10-13} and save it.
Tests are located in a sub-directory called 'features/' and organized into feature files, each containing scenarios.
Actual test runs are performed inside the folder 'test/'. The folder contain a .stxxl config file using a small 10MB disk.
For each test scenario, an .osm data file is constructed based on the description in the test scenario. The .osm file is then converted to .pbf format using the osmosis command line tool.
It's then preprocessed with osrm-extract and osrm-prepare. The relevant profile script is copied to 'test/profile.ini' before each preprocessing, so the binaries pick it up.
The resulting .osrm files are cached, so that preprocessing is only repeated if either the osm data, the profile or the binaries have changed. This helps to speed rerunning of tests.
'test/server.ini' is now rewritten to ensure that the right configuration is used. osrm-routed is then launched, and routes are requested via http calls to localhost:5000, and the response is compared to the expected result. All request are timed out after 10 seconds. both timeouts and crashes are caught and reported. To ensure a clean run of every test, all processes named osrm-extract, osrm-prepare and osrm-routed are killed before and after each test, using a signal 9, no matter who launched them.
For Cucumber documentation, see https://github.com/cucumber/cucumber/wiki/A-Table-Of-Content.
The OSRM tests currently come in three main flavors: node maps, lat/lon tests, and routability tables.
With node maps you geometrically place nodes on a mini map. Each node is identified by a letter. You then string together nodes to create way, and optionally add relations. Finally you specify a list of routes to be tested, described by start and end points, and the expected route:
Scenario: Zig Zag
Given the node map
| a | | c |
| | b | |
And the ways
| nodes |
| ab |
| bc |
When I route I should get
| from | to | route |
| a | c | ab,bc |
| c | a | bc,ab |Under 'Given the node map', the node map is constructed using the pipe character '|' to space nodes. The node map can have any size you want, but must be rectangular. Each node must be specified by a letter a-z or a number 0-9. By convention, ways can only be constructed between nodes specified with letters, while nodes with numbers can only be used as start/end points in the route tests. Only one node can exists at each location, and a node can only be present once on the map.
You can add OSM tags to nodes by adding using a 'And the nodes' table:
And the nodes
| node | barrier |
| a | bollard |
| b | gate |The 'node' column identifies the node, while additional columns are added as tags to the corresponding node.
Under 'And the ways', you create ways by stringing together nodes in the node map. For example, 'ab' is a way starting at a and ending at b, while 'abc' is a way starting at a, passing b, and ending at c. A ways can have two or more nodes. All nodes of the way must exist in the node map.
You can add OSM tags to ways by adding columns to the 'And the ways' table:
And the ways
| nodes | oneway |
| ab | yes |
| bc | no |Note that 'Given...' and 'And...' is interchangeable in cucumber, and you should just use what creates a natural language flow.
An OSM file is constructed based on the given nodes, ways and relations, and then used for testing. Using 'When I route I should get' you specify routes to be tested, each with a start and an end node, and the ways you expect the route to follow:
When I route I should get
| from | to | route |
| a | c | ab,bc |
| c | a | bc,ab |Note that you cannot specify nodes to be visited, only the ways to be follow. This is because OSRM does not return nodes in query results, only ways. For each row, osrm-routed is queried on port 5000, and the result parsed and compared to the expected route.
In addition to the route itself, you can test for total length or time (or both) by adding the corresponding columns to the test table:
When I route I should get
| from | to | route | time | distance |
| a | b | ab | 24s | 100m |
| c | d | cd | 72s | 200m |Time is specified in seconds, while length is specified in meters. (Note that OSRM currently quantized length to the nearest 10m, and time to the nearest second.)
OSRM uses spherical coordinates. Distances in node maps deviate from the naive flat geometry when you move away from the equator.
Fuzzy ranges for expected numerical values can be specified using either a percentage range, or an absolute +- range:
When I route I should get
| from | to | route | distance |
| a | d | abcde | 300m +-1 |
| a | e | abcde | 700m ~1% |You can add a comment column by using the column name "#". The column will be ignored, but can be useful if you need to explain individual rows:
When I route I should get
| from | to | route | time | \# |
| a | d | abcde | 30s | we're going downhill |
| a | e | abcde | 60s | we're going uphill |Lat/lon tables are quite similar to node maps, except the nodes are placed using a table with absolute latitude and longitude values:
Scenario: Longitudinal distances at latitude 45
Given the node locations
| node | lat | lon |
| c | 45 | 80 |
| d | 45 | 0 |
And the ways
| nodes |
| cd |
When I route I should get
| from | to | route | distance |
| c | d | cd | 6028844 ~0.5% |This is especially useful for testing things related to the Mercator projection and huge distances.
This flavor of tests is used for checking individual isolated ways, and is often useful for testing access and barrier tags. You can test each direction (forward/backward) separately:
Scenario: Simplest possible oneway
Given the speedprofile "car"
Then routability should be
| highway | oneway | forw | backw |
| primary | | x | x |
| primary | yes | x | |
| primary | -1 | | x |Column headers are used to specify tag keys. In the example above, the 'highway' and 'oneway' tags are used.
You can also set tags on a central node of the way, by using columns with titles in the form of 'node/...':
Scenario: Car - Barriers
Then routability should be
| highway | node/barrier | bothw |
| primary | | x |
| primary | bollard | |
Columns forw, backw and bothw has special meaning. If an forw column is present the way is tested for forward routability. If an backw row is present, it's tested for backwards routability. If an bothw row is present, it's tested for both forwards and backwards routability.
A single OSM file is constructed, containing an isolated way for each row. The ways are labelled w1, w2, w3, etc and placed in a line, but with spacing in between:
--w1-- --w2-- --w3--
##Customizing Tests For both types of tests, nodes are by default spaced on grid with origin at 1,1 and a grid size of 100m. The grid size can be changed using:
Given a grid size of 500 metersBy default, tests use the "bicycle" profile in profiles/bicycle.lua, but this can be changed for each scenario by adding:
Given the speedprofile "car"Note that you can use a 'Background:' section if you need to specify things for all test scenarios in a .feature file:
Background:
Given the speedprofile "car"
Given a grid size of 500 meters#Running Tests You can run all tests, or specific sets or scenarios.
- To run all test: cucumber
- To run test with a specific tag, use: cucumber --tags @restrictions
- To exclude tests withs a specific tag, use: cucumber --tags ~@restrictions
- To run a specific scenario: cucumber features/bicycle/restrictions.feature:6
- To run all scenario in a specific folder: cucumber features/foot
To verify that a change did not introduce regression errors, it's recommended to run all tests, except @todo and @stress:
cucumber -t ~@todo -t ~@stress###Output Tests are evaluated by parsing the instructions hash returned by osrm-routed. Results are shown in the terminal. A passed routed will be shown using a green row. A failed row will be shown with two rows: The expected result in a yellow row, followed by the actual result in a grey row.
(Note that there's currently an cucumber issue, causing rows below the first failed row to incorrectly by marked as failed.)
###Log files During the tests, several log files are used. Output from osrm-extract and osrm-prepare are redirected to test/preprocess.log. Output from osrm-routed is redirected to test/osrm-routed.log.
Additional info about failed tests is logged to fail.log, containing details about the osm data used, the profile, expected and returned routes, as well as actual query and response involved.