Am 16.07.2018 um 03:34 schrieb cmos:
Hi, I have the same question. Upon running code attempting to use GraphViews, I get an error during pickling:
error: 'i' format requires -2147483648 <= number <= 2147483647
More specifically, it looks like a line inside joblib is unhappy: CustomizablePickler(buffer, self._reducers).dump(obj)
And this takes us to a struct packing line: header = struct.pack("!i", n)
So, if I had to guess, I'd suspect joblib is trying to pickle the whole graph rather than the GraphView reference, or something like this. Was either of you able to get code to successfully parallelize using GraphViews to avoid copying?
It is impossible to say anything, without a minimal and self-contained example that shows the problem.
Below is a minimal breaking example, if it helps. I am also happy to provide other information such as tracebacks.
def toy_func(g): return g.vertex_properties['skim'][0][1]
vmr = [0, 1] g = load_graph(path) # 22,000 vertex directed graph (a road network) skim_table = shortest_distance(g, weights=g.edge_properties["weight"]) g.properties['skim'] = skim_table p(joblib.delayed(toy_func)(GraphView(g)) for i in range(10))
That is not a complete minimal example; the function 'p' is undefined and there are other errors. Please provide one that actually runs, and does not depend on external data.
Best, Tiago