Graph Cleaning

Last update

October 15, 2024

Install and update cityseer if necessary.

# !pip install --upgrade cityseer

See the guide for a preamble.

Please also see the graph corrections guide for an alternative approach.

Downloading data

This example will make use of OSM data downloaded from the OSM API. To keep things interesting, let’s pick London Soho, which will be buffered and cleaned for a 1,250m radius.

from shapely import geometry
import utm

from cityseer.tools import graphs, plot, io

# Let's download data within a 1,250m buffer around London Soho:
lng, lat = -0.13396079424572427, 51.51371088849723
# lng, lat = 2.166981, 41.389526 -- Barcelona - which is a complex case
buffer = 1250
# creates a WGS shapely polygon
poly_wgs, _ = io.buffered_point_poly(lng, lat, buffer)
# use a WGS shapely polygon to download information from OSM
# this version will not simplify
G_raw = io.osm_graph_from_poly(poly_wgs, simplify=False)
# whereas this version does simplify
G_utm = io.osm_graph_from_poly(poly_wgs)

# select extents for clipping the plotting extents
easting, northing = utm.from_latlon(lat, lng)[:2]
buff = geometry.Point(easting, northing).buffer(1000)
min_x, min_y, max_x, max_y = buff.bounds


# reusable plot function
def simple_plot(_G, plot_geoms=True):
    # plot using the selected extents
    plot.plot_nx(
        _G,
        labels=False,
        plot_geoms=plot_geoms,
        node_size=4,
        edge_width=1,
        x_lim=(min_x, max_x),
        y_lim=(min_y, max_y),
        figsize=(6, 6),
        dpi=150,
    )

INFO:cityseer.tools.io:Converting networkX graph from EPSG code 4326 to EPSG code 32630.
INFO:cityseer.tools.io:Processing node x, y coordinates.
100%|██████████| 12429/12429 [00:00<00:00, 303680.47it/s]
INFO:cityseer.tools.io:Processing edge geom coordinates, if present.
100%|██████████| 13773/13773 [00:00<00:00, 705866.92it/s]
INFO:cityseer.tools.graphs:Generating interpolated edge geometries.
100%|██████████| 13773/13773 [00:00<00:00, 29654.56it/s]
INFO:cityseer.tools.graphs:Removing filler nodes.
100%|██████████| 12429/12429 [00:02<00:00, 4186.89it/s]
INFO:cityseer.tools.graphs:Ironing edges.
100%|██████████| 5741/5741 [00:02<00:00, 2305.88it/s]
INFO:cityseer.tools.graphs:Merging parallel edges within buffer of 1.
100%|██████████| 5741/5741 [00:00<00:00, 94840.76it/s]
INFO:cityseer.tools.io:Converting networkX graph from EPSG code 4326 to EPSG code 32630.
INFO:cityseer.tools.io:Processing node x, y coordinates.
100%|██████████| 12429/12429 [00:00<00:00, 360898.07it/s]
INFO:cityseer.tools.io:Processing edge geom coordinates, if present.
100%|██████████| 13773/13773 [00:00<00:00, 387534.04it/s]
INFO:cityseer.tools.graphs:Generating interpolated edge geometries.
100%|██████████| 13773/13773 [00:00<00:00, 53904.99it/s]
INFO:cityseer.tools.graphs:Removing filler nodes.
100%|██████████| 12429/12429 [00:02<00:00, 5440.01it/s]
INFO:cityseer.tools.graphs:Removing dangling nodes.
100%|██████████| 4170/4170 [00:00<00:00, 129302.62it/s]
INFO:cityseer.tools.util:Creating edges STR tree.
100%|██████████| 4781/4781 [00:00<00:00, 529102.04it/s]
INFO:cityseer.tools.graphs:Splitting opposing edges.
100%|██████████| 3358/3358 [00:00<00:00, 59286.57it/s]
INFO:cityseer.tools.graphs:Squashing opposing nodes
INFO:cityseer.tools.graphs:Merging parallel edges within buffer of 60.
100%|██████████| 4781/4781 [00:00<00:00, 95049.92it/s]
WARNING:cityseer.tools.graphs:Be cautious with large buffer distances when using crawl!
INFO:cityseer.tools.util:Creating nodes STR tree
100%|██████████| 3358/3358 [00:00<00:00, 76015.59it/s]
INFO:cityseer.tools.graphs:Consolidating nodes.
100%|██████████| 3358/3358 [00:00<00:00, 69445.95it/s]
INFO:cityseer.tools.graphs:Merging parallel edges within buffer of 60.
100%|██████████| 4781/4781 [00:00<00:00, 114110.43it/s]
INFO:cityseer.tools.graphs:Removing filler nodes.
100%|██████████| 3358/3358 [00:00<00:00, 23833.85it/s]
INFO:cityseer.tools.util:Creating edges STR tree.
100%|██████████| 4428/4428 [00:00<00:00, 30054.39it/s]
INFO:cityseer.tools.graphs:Splitting opposing edges.
100%|██████████| 3005/3005 [00:00<00:00, 61930.68it/s]
INFO:cityseer.tools.graphs:Squashing opposing nodes
INFO:cityseer.tools.graphs:Merging parallel edges within buffer of 40.
100%|██████████| 4428/4428 [00:00<00:00, 94363.69it/s]
WARNING:cityseer.tools.graphs:Be cautious with large buffer distances when using crawl!
INFO:cityseer.tools.util:Creating nodes STR tree
100%|██████████| 3005/3005 [00:00<00:00, 47486.74it/s]
INFO:cityseer.tools.graphs:Consolidating nodes.
100%|██████████| 3005/3005 [00:00<00:00, 68326.32it/s]
INFO:cityseer.tools.graphs:Merging parallel edges within buffer of 40.
100%|██████████| 4428/4428 [00:00<00:00, 102651.79it/s]
INFO:cityseer.tools.graphs:Removing filler nodes.
100%|██████████| 3005/3005 [00:00<00:00, 504215.85it/s]
INFO:cityseer.tools.util:Creating edges STR tree.
100%|██████████| 4428/4428 [00:00<00:00, 388316.01it/s]
INFO:cityseer.tools.graphs:Splitting opposing edges.
100%|██████████| 3005/3005 [00:01<00:00, 2599.51it/s]
INFO:cityseer.tools.graphs:Squashing opposing nodes
INFO:cityseer.tools.graphs:Merging parallel edges within buffer of 40.
100%|██████████| 4595/4595 [00:00<00:00, 14529.93it/s]
WARNING:cityseer.tools.graphs:Be cautious with large buffer distances when using crawl!
INFO:cityseer.tools.util:Creating nodes STR tree
100%|██████████| 3005/3005 [00:00<00:00, 72605.53it/s]
INFO:cityseer.tools.graphs:Consolidating nodes.
100%|██████████| 3005/3005 [00:00<00:00, 11083.19it/s]
INFO:cityseer.tools.graphs:Merging parallel edges within buffer of 40.
100%|██████████| 3917/3917 [00:00<00:00, 49944.64it/s]
INFO:cityseer.tools.graphs:Removing filler nodes.
100%|██████████| 2604/2604 [00:00<00:00, 190354.46it/s]
INFO:cityseer.tools.util:Creating edges STR tree.
100%|██████████| 3873/3873 [00:00<00:00, 125609.23it/s]
INFO:cityseer.tools.graphs:Splitting opposing edges.
100%|██████████| 2591/2591 [00:00<00:00, 28315.30it/s]
INFO:cityseer.tools.graphs:Squashing opposing nodes
INFO:cityseer.tools.graphs:Merging parallel edges within buffer of 30.
100%|██████████| 3880/3880 [00:00<00:00, 62688.85it/s]
INFO:cityseer.tools.util:Creating nodes STR tree
100%|██████████| 2591/2591 [00:00<00:00, 46261.17it/s]
INFO:cityseer.tools.graphs:Consolidating nodes.
100%|██████████| 2591/2591 [00:00<00:00, 45728.77it/s]
INFO:cityseer.tools.graphs:Merging parallel edges within buffer of 30.
100%|██████████| 3843/3843 [00:00<00:00, 87162.56it/s]
INFO:cityseer.tools.graphs:Removing filler nodes.
100%|██████████| 2568/2568 [00:00<00:00, 498448.46it/s]
INFO:cityseer.tools.util:Creating edges STR tree.
100%|██████████| 3843/3843 [00:00<00:00, 393628.91it/s]
INFO:cityseer.tools.graphs:Splitting opposing edges.
100%|██████████| 2568/2568 [00:00<00:00, 10225.61it/s]
INFO:cityseer.tools.graphs:Squashing opposing nodes
INFO:cityseer.tools.graphs:Merging parallel edges within buffer of 30.
100%|██████████| 3863/3863 [00:00<00:00, 46736.46it/s]
INFO:cityseer.tools.util:Creating nodes STR tree
100%|██████████| 2568/2568 [00:00<00:00, 47990.22it/s]
INFO:cityseer.tools.graphs:Consolidating nodes.
100%|██████████| 2568/2568 [00:00<00:00, 21975.16it/s]
INFO:cityseer.tools.graphs:Merging parallel edges within buffer of 30.
100%|██████████| 3730/3730 [00:00<00:00, 76070.96it/s]
INFO:cityseer.tools.graphs:Removing filler nodes.
100%|██████████| 2476/2476 [00:00<00:00, 392260.50it/s]
INFO:cityseer.tools.util:Creating edges STR tree.
100%|██████████| 3725/3725 [00:00<00:00, 582411.93it/s]
INFO:cityseer.tools.graphs:Splitting opposing edges.
100%|██████████| 2472/2472 [00:00<00:00, 11019.44it/s]
INFO:cityseer.tools.graphs:Squashing opposing nodes
INFO:cityseer.tools.graphs:Merging parallel edges within buffer of 25.
100%|██████████| 3734/3734 [00:00<00:00, 50643.76it/s]
INFO:cityseer.tools.util:Creating nodes STR tree
100%|██████████| 2472/2472 [00:00<00:00, 10906.68it/s]
INFO:cityseer.tools.graphs:Consolidating nodes.
100%|██████████| 2472/2472 [00:00<00:00, 8319.65it/s]
INFO:cityseer.tools.graphs:Merging parallel edges within buffer of 25.
100%|██████████| 3591/3591 [00:00<00:00, 75974.26it/s]
INFO:cityseer.tools.graphs:Removing filler nodes.
100%|██████████| 2368/2368 [00:00<00:00, 343897.78it/s]
INFO:cityseer.tools.util:Creating edges STR tree.
100%|██████████| 3581/3581 [00:00<00:00, 469339.50it/s]
INFO:cityseer.tools.graphs:Splitting opposing edges.
100%|██████████| 2360/2360 [00:00<00:00, 55182.98it/s]
INFO:cityseer.tools.graphs:Squashing opposing nodes
INFO:cityseer.tools.graphs:Merging parallel edges within buffer of 30.
100%|██████████| 3581/3581 [00:00<00:00, 67408.09it/s]
WARNING:cityseer.tools.graphs:Be cautious with large buffer distances when using crawl!
INFO:cityseer.tools.util:Creating nodes STR tree
100%|██████████| 2360/2360 [00:00<00:00, 44965.24it/s]
INFO:cityseer.tools.graphs:Consolidating nodes.
100%|██████████| 2360/2360 [00:00<00:00, 75735.53it/s]
INFO:cityseer.tools.graphs:Merging parallel edges within buffer of 30.
100%|██████████| 3581/3581 [00:00<00:00, 71812.51it/s]
INFO:cityseer.tools.graphs:Removing filler nodes.
100%|██████████| 2360/2360 [00:00<00:00, 554106.44it/s]
INFO:cityseer.tools.util:Creating edges STR tree.
100%|██████████| 3581/3581 [00:00<00:00, 25060.15it/s]
INFO:cityseer.tools.graphs:Splitting opposing edges.
100%|██████████| 2360/2360 [00:00<00:00, 19685.03it/s]
INFO:cityseer.tools.graphs:Squashing opposing nodes
INFO:cityseer.tools.graphs:Merging parallel edges within buffer of 25.
100%|██████████| 3585/3585 [00:00<00:00, 57240.78it/s]
INFO:cityseer.tools.util:Creating nodes STR tree
100%|██████████| 2360/2360 [00:00<00:00, 41612.96it/s]
INFO:cityseer.tools.graphs:Consolidating nodes.
100%|██████████| 2360/2360 [00:00<00:00, 28141.85it/s]
INFO:cityseer.tools.graphs:Merging parallel edges within buffer of 25.
100%|██████████| 3561/3561 [00:00<00:00, 25893.68it/s]
INFO:cityseer.tools.graphs:Removing filler nodes.
100%|██████████| 2350/2350 [00:00<00:00, 533251.16it/s]
INFO:cityseer.tools.util:Creating edges STR tree.
100%|██████████| 3560/3560 [00:00<00:00, 413861.87it/s]
INFO:cityseer.tools.graphs:Splitting opposing edges.
100%|██████████| 2350/2350 [00:00<00:00, 24410.49it/s]
INFO:cityseer.tools.graphs:Squashing opposing nodes
INFO:cityseer.tools.graphs:Merging parallel edges within buffer of 25.
100%|██████████| 3560/3560 [00:00<00:00, 50944.30it/s]
INFO:cityseer.tools.util:Creating nodes STR tree
100%|██████████| 2350/2350 [00:00<00:00, 55926.91it/s]
INFO:cityseer.tools.graphs:Consolidating nodes.
100%|██████████| 2350/2350 [00:00<00:00, 34726.81it/s]
INFO:cityseer.tools.graphs:Merging parallel edges within buffer of 25.
100%|██████████| 3553/3553 [00:00<00:00, 76546.81it/s]
INFO:cityseer.tools.graphs:Removing filler nodes.
100%|██████████| 2348/2348 [00:00<00:00, 455830.86it/s]
INFO:cityseer.tools.util:Creating edges STR tree.
100%|██████████| 3551/3551 [00:00<00:00, 225372.60it/s]
INFO:cityseer.tools.graphs:Splitting opposing edges.
100%|██████████| 2346/2346 [00:01<00:00, 1800.82it/s]
INFO:cityseer.tools.graphs:Squashing opposing nodes
INFO:cityseer.tools.graphs:Merging parallel edges within buffer of 25.
100%|██████████| 3863/3863 [00:00<00:00, 11449.89it/s]
INFO:cityseer.tools.util:Creating nodes STR tree
100%|██████████| 2346/2346 [00:00<00:00, 24471.91it/s]
INFO:cityseer.tools.graphs:Consolidating nodes.
100%|██████████| 2346/2346 [00:00<00:00, 3424.32it/s]
INFO:cityseer.tools.graphs:Merging parallel edges within buffer of 25.
100%|██████████| 2913/2913 [00:00<00:00, 28242.28it/s]
INFO:cityseer.tools.graphs:Removing filler nodes.
100%|██████████| 1926/1926 [00:00<00:00, 39995.00it/s]
INFO:cityseer.tools.graphs:Ironing edges.
100%|██████████| 2754/2754 [00:00<00:00, 4255.90it/s]
INFO:cityseer.tools.graphs:Merging parallel edges within buffer of 1.
100%|██████████| 2754/2754 [00:00<00:00, 19277.37it/s]

The automated graph cleaning may give satisfactory results depending on the intended end-use. See the steps following beneath for an example of how to manually clean the graph where additional control is preferred.

print("The graph before simplification.")
simple_plot(G_raw, plot_geoms=True)

print("The graph after simplification")
simple_plot(G_utm, plot_geoms=True)

INFO:cityseer.tools.plot:Preparing graph nodes
INFO:cityseer.tools.plot:Preparing graph edges

The graph before simplification.

100%|██████████| 5705/5705 [00:00<00:00, 15899.38it/s]

INFO:cityseer.tools.plot:Preparing graph nodes
INFO:cityseer.tools.plot:Preparing graph edges

The graph after simplification

100%|██████████| 2754/2754 [00:00<00:00, 15020.47it/s]

Manual cleaning

The automated simplification uses a number of steps and should generally give a solid starting point for further manual cleaning.

It is also possible to clean the network with an entirely manual sequence of steps if you need greater control. The following is a basic example with the tools.graphs module.

The io.osm_graph_from_poly convenience method used for this demonstration has already converted the graph from a geographic WGS to projected UTM coordinate system; however, if working with a graph which is otherwise in a WGS coordinate system then it must be converted to a projected coordinate system prior to further processing. This can be done with io.nx_wgs_to_utm.

# remove dangling nodes: short dead-end stubs
# these are often found at entrances to buildings or parking lots
# The removed_disconnected flag will removed isolated network components
# i.e. disconnected portions of network that are not joined to the main street network
G = graphs.nx_remove_dangling_nodes(G_raw)
simple_plot(G)

INFO:cityseer.tools.graphs:Removing dangling nodes.
100%|██████████| 4170/4170 [00:00<00:00, 194706.03it/s]
INFO:cityseer.tools.plot:Preparing graph nodes
INFO:cityseer.tools.plot:Preparing graph edges
100%|██████████| 4742/4742 [00:00<00:00, 16334.55it/s]

Refining the network

Things are already looked much better, but we still have areas with large concentrations of nodes at complex intersections and many parallel roadways, which will confound centrality methods. We’ll now try to remove as much of this as possible. These steps involve the consolidation of nodes to clean-up extraneous nodes, which may otherwise exaggerate the intensity or complexity of the network in certain situations.

Merging parallel roadways: graphs.nx_split_opposing_geoms is used to intentionally split edges in near proximity to nodes located on an adjacent parallel roadway.

G1 = graphs.nx_split_opposing_geoms(
    G,
    buffer_dist=30,
    prioritise_by_hwy_tag=True,
    osm_hwy_target_tags=["trunk", "primary", "secondary", "tertiary"],
    osm_matched_tags_only=True,
    contains_buffer_dist=40,
)
simple_plot(G1, plot_geoms=True)

INFO:cityseer.tools.util:Creating edges STR tree.
100%|██████████| 4742/4742 [00:00<00:00, 404190.16it/s]
INFO:cityseer.tools.graphs:Splitting opposing edges.
100%|██████████| 3354/3354 [00:01<00:00, 3052.71it/s]
INFO:cityseer.tools.graphs:Squashing opposing nodes
INFO:cityseer.tools.graphs:Merging parallel edges within buffer of 40.
100%|██████████| 4919/4919 [00:00<00:00, 39594.95it/s]
INFO:cityseer.tools.plot:Preparing graph nodes
INFO:cityseer.tools.plot:Preparing graph edges
100%|██████████| 4803/4803 [00:00<00:00, 21649.77it/s]

Consolidating nodes: Cleanup of intersections will be performed with the graphs.nx_consolidate_nodes function. The arguments passed to the parameters allow for a number of different strategies, such as whether to ‘crawl’; whether to use intersection through routes for determining new centroids; and to set the direct or indirect neighbour policies according to which nodes and edges are consolidated. These are explained more fully in the documentation.

G2 = graphs.nx_consolidate_nodes(
    G1,
    buffer_dist=12,
    crawl=True,
    prioritise_by_hwy_tag=True,
)
simple_plot(G2, plot_geoms=True)

INFO:cityseer.tools.util:Creating nodes STR tree
100%|██████████| 3354/3354 [00:00<00:00, 65556.46it/s]
INFO:cityseer.tools.graphs:Consolidating nodes.
100%|██████████| 3354/3354 [00:01<00:00, 2448.76it/s]
INFO:cityseer.tools.graphs:Merging parallel edges within buffer of 25.
100%|██████████| 3171/3171 [00:00<00:00, 20202.23it/s]
INFO:cityseer.tools.plot:Preparing graph nodes
INFO:cityseer.tools.plot:Preparing graph edges
100%|██████████| 3009/3009 [00:00<00:00, 23708.96it/s]

Finally, the edges are “ironed” to straighten out artefacts introduced by automated cleaning, which will sometimes bend the ends of edge segments to the locations of new centroids.

G3 = graphs.nx_remove_filler_nodes(G2)
G4 = graphs.nx_iron_edges(G3)
simple_plot(G4, plot_geoms=True)

INFO:cityseer.tools.graphs:Removing filler nodes.
100%|██████████| 2059/2059 [00:00<00:00, 28881.64it/s]
INFO:cityseer.tools.graphs:Ironing edges.
100%|██████████| 2796/2796 [00:00<00:00, 5151.34it/s]
INFO:cityseer.tools.graphs:Merging parallel edges within buffer of 1.
100%|██████████| 2796/2796 [00:00<00:00, 166103.99it/s]
INFO:cityseer.tools.plot:Preparing graph nodes
INFO:cityseer.tools.plot:Preparing graph edges
100%|██████████| 2790/2790 [00:00<00:00, 29996.48it/s]