Fair point. I managed to find some time to make a MWE. Here's the link to a zip file containing both the input data (network and costs as csv dumps from our postgres db) and the code: https://www.dropbox.com/s/3iby8nil7zyi7q3/MWE.zip?dl=0To use: just run "python mwe_main.py". The last line calculates a shortest path tree until 60mins for cost structure "49" (these are costs associated with car travel, the specific id has no meaning). Because this is only Belgium, mem usage when I run this is manageable (under 1Gb).In general my question is: can this code be made more performant, both in terms of mem and speed ? Am I right in using GraphViews for each cost structure, where the view only contains the edges which have costs associated with them ? Does this make gt.shortest_distance() faster ?The most production-critical part of the code here is point_drivetimes(). Any gain there in calculation time would be very valuable. The loading of the network and cost structure only need to happen once, so can be a bit slower.Thx for helping me !On Tue, Jun 22, 2021 at 9:43 AM Tiago de Paula Peixoto <tiago@skewed.de> wrote:Dear Mathias,
It is not reasonable to expect us to make this kind of evaluation just
from partial code. As with anything, we need a minimal working example
to be able to say something concrete.
I would recommend you to try to separate the pandas dataframe
manipulation from the graph-tool side in order to determine which is
consuming more memory.
Best,
Tiago
Am 22.06.21 um 09:24 schrieb Mathias Versichele:
> Hi all. Anyone can provide me with some insights here ? I know it's
> quite an open question here, and it might take some effort of course.
> Would anyone be available/willing to do an actual code audit of the code
> that I have ? This would be compensated of course. Feel free to contact
> me to discuss.
>
> Kind regards
>
> On Tue, Jun 15, 2021 at 8:45 PM Mathias Versichele
> <mathias.versichele@gmail.com <mailto:mathias.versichele@gmail.com>> wrote:
>
> Hi, I've been using graph-tool for the last year or so for
> calculating shortest-path trees on large-scale road networks. We
> used to do this in a postgres database with the pgrouting extension,
> but were continually confronted with unacceptable startup costs. The
> switch to a python module using graph-tool has considerably sped up
> our routing queries, but we are suffering from this service using
> too much memory. I have the feeling I might be using graph-tool in a
> wrong way, but before I dive into that, it would be good to know
> what is the expected memory footprint for my use case.
>
> Take for example a road network with 30Mio edges and 31 Mio nodes
> (the combined road network of Belgium, Netherland, France and
> Germany in OSM). For this road network, I need to calculate shortest
> paths using different edge weights (edge property map). What would
> be a very rough estimate how much memory this would use ? For the
> network only + per edge-property-map. In our setup, there would be
> one edge-property-map with edge weights per country. We're currently
> seeing usage of over 50Gb easily, spiking even higher when we're
> loading extra cost structures or networks. Is that expected ? Or am
> I experiencing memory leaks somewhere ?
>
> How I'm using graph-tool right now:
>
> *1) loading network*
> /nw = dataframe with edges info in the structure: startnode-id,
> endnode-id, edge-id, country/
>
> G = gt.Graph(directed=True)
> G.ep["edge_id"] = G.new_edge_property("int")
> G.ep["country_id"] = G.new_edge_property("int16_t")
> eprops = [G.ep["edge_id"], G.ep["country_id"]]
>
> n = G.add_edge_list(nw.to_numpy(), hashed=True, eprops=eprops)
> G.vertex_properties["n"] = n
>
> *2) loading edge costs: I'm using GraphViews*
> *
> *
> /countries = list of country-codes/
> edge_filter = np.in1d(G.ep["country_id"].a, [get_country_id(c) for c
> in countries])*
> *
> GV = gt.GraphView(G, efilt=edge_filter)
>
> edges = GV.get_edges([GV.edge_index])
> sources = G.vertex_properties["n"].a[edges[:,0]]
> targets = G.vertex_properties["n"].a[edges[:,1]]
> idxs = edges[:,2]
>
> /db_costs = pandas dataframe with structure: source, target, cost
> /
>
> sti = np.vstack((idxs,sources,targets)).T
> sti_df = pd.DataFrame({'idx': sti[:, 0], 'source': sti[:, 1],
> 'target': sti[:, 2]})
> m = pd.merge(sti_df, db_costs, on=['source', 'target'], how='left',
> sort=False)[['idx', 'c']]
> wgts_list = m.sort_values(by=['idx']).T.iloc[1].to_numpy()
> wgts_list = np.where(wgts_list==np.nan, np.iinfo(np.int32).max,
> wgts_list)
>
> wgts = GV.new_edge_property("int32_t")
> wgts.fa = wgts_list
> wgts.fa = np.where(wgts.fa==-2147483648, np.iinfo(np.int32).max,
> wgts.fa)
> GV.edge_properties[cs_ids_str] = wgts
>
> GV2 = gt.GraphView(GV, efilt=wgts.fa != np.inf)
>
> *3) I then use GV2 for calculating Dijkstra and such...*
>
>
> I could of course work on an MWE of some sorts. But would be very
> nice to get an estimate on mem footprint, and to see if I'm doing
> sth really silly in the code above.
>
> Thx!
>
>
>
>
>
>
> _______________________________________________
> graph-tool mailing list -- graph-tool@skewed.de
> To unsubscribe send an email to graph-tool-leave@skewed.de
>
--
Tiago de Paula Peixoto <tiago@skewed.de>
_______________________________________________
graph-tool mailing list -- graph-tool@skewed.de
To unsubscribe send an email to graph-tool-leave@skewed.de