I am trying to outsource some of my calculations to the Univeristy's cluster as the compute times for some of my datasets are getting very lengthy. I had a couple of questions arising out of this and was wondering whether you had any thoughts/advice on them:
1. The University normally limits runtimes of a given job to 12h and suggests checkpointing to get around this so that the calculation can simply resume as a new job. Apparently the cluster has "DMTCP: Distributed MultiThreaded CheckPointing" installed in order to allow checkpointing "for some applications which do not have their own support for this". Is this compatible with graph-tool or are you aware of any other ways in which I could stop and restart a job? (What I am after is stopping and restarting a graph-tool function which takes more than 12h to complete as I can obviously pickle results of a calculation straightforwardly already.) If there is no way of currently doing this is this something you might consider doing at some point in the future?
2. As far as I understand using OpenMP I can only ever use a single node at a time and am thus limited by how many CPUs and how much RAM this node supplies. To be able to use more CPUs I would need to use e.g. MPI. Are there any plans to implement this at some point? (I realise this may not be straightforward but was interested in your thoughts.)
3. I would sometimes find it useful to be able to get a progress update for some of the functions to carry out a rough order-of-magnitude estimate of required compute time. Two use cases: a) If calculating the betweenness centrality for a large network this can take a long time. Having an idea of how many nodes have been covered already would be useful to extrapolate time roughly remaining to see whether the calculation is even feasible in the time available. b) If collecting data using mcmc_equilibrate it would be potentially less meaningful as the process is stochastic, however, information on how many sweeps have been completed would still be useful to give a rough estimate of whether completion will take hours, days, weeks, or months.