On 06/15/2011 12:11 PM, Sebastian Weber wrote:
Strange, I don't see this problem at all. Are you using the standard python from macports? The DLFCN module should be there, since it is an standard module which should be defined for MacOS.
Nope, it is not. At least on my machine macports does not install any DLFCN.py.
Well, this is obviously a problem with the python installation of macports, right? This is a standard python module, which should be installed...
Here is the issue in detail: We need to make sure that the C++ RTTI (run-time type information) works properly when the compiled modules are imported from python. The only way to do this, is by ensuring the RTLD_GLOBAL flag is passed to the dlopen() function, when python loads the module. This is not a bug, either in graph-tool or python, it is just how life is (see http://gcc.gnu.org/faq.html#dso). Python provides an interface for doing this, namely the sys.setdlopenflags() function. This is all nice and well, but we need to know the numeric values of the flags we want to pass, and we need to pass always two flags: RTLD_LAZY | RTLD_GLOBAL (or RTLD_NOW | RTLD_GLOBAL, for immediate symbol resolution). Thus, these two flags must be defined somewhere, since their numeric values will vary across OS's. These two values used to be inside the "dl" module, which is now deprecated and has even been removed in python 3. To close this hole, they have created the DLFCN module, whose _only purpose_ AFAIK is to contain these important definitions. Thus, without this module we are completely stuck. The ctypes module, for whatever reason, does not include either RTLD_LAZY or RTLD_NOW (as you have already noticed).
So, we really need the DLFCN module. I've checked, and in my macports installation the module is also not installed... However it does have the dl module, which I then used as a fall-back.
I guess we have to submit a bug report the macports people.
BTW, I saw this
http://www.boost.org/doc/libs/1_46_1/libs/python/doc/tutorial/doc/html/pytho...
I guess it is quite tedious to split every function declaration in a separate cpp file (and you do already to some extent in graph-tool), but using this technique a little more would help in cutting down compilation efforts. I am using here a machine with 8GB of RAM for the compilation and when I use 2 compilation processes in parallel my machine is swapping a lot! Apart from the fact that it takes really long to compile things.
These techniques are already extensively used... But the compile time/memory behavior in graph-tool has little to do with boost::python, and it has more to do with the fact we are generating a whole bunch of template instantiations for the algorithms themselves (for the different types of property maps, filtering, etc). I already split things in independent compilation units, to avoid even larger memory spikes, but the whole thing is quite suboptimal at this point. The true culprit is of course GCC, since I don't think it makes sense to use around 2GB RAM to compile a 5MB object file.
Ccache somehow did not help here to much for some reason as I always have to reject all compilations and start from scratch. Maybe the ccache macports integration will help here though, I will try.
ccache works very reliably for me (even on macos). I only need to recompile things if the source code really changes in some meaningful way. Simple modifications, such as comments, indentations, etc, should never trigger a recompile. And yes, you should of course enable it in the macports config file...
Cheers, Tiago