PyTom: FAQ

Here you will find a collection of frequently asked questions. More up to date comments on potential pitfalls are documented in the mailling list.

Directly navigate to question types:

Installation Questions

Question
I get following error message during compilation

function pytom_volume_wrap.cpp:
swigCpp/pytom_volume_wrap.cpp:2051: error: invalid conversion from ‘const
char*’ to ‘char*’
swigCpp/pytom_volume_wrap.cpp:2774: error: cannot convert ‘int*’ to
‘Py_ssize_t*’ for argument ‘3’ to ‘int PyString_AsStringAndSize(PyObject*,char**, Py_ssize_t*)’
swigCpp/pytom_volume_wrap.cpp:11661: error: invalid conversion from ‘const char*’ to ‘char*’


Answer
This is due to a old swig version (< 1.3.40).
Please install a newer version and rerun the PyTom installation.

------------------------------------------------------------------------

Question
On OSX, compile fails with errors like:
error: use of undeclared identifier 'is_reduced'

Answer
This is because Apple decided to skip G++ compilers as a default compiler and replaced it with CLANG. Hence the errors.
Replace the c++ call in the with g++-mp-4.5 or similar that you installed from MacPorts. The best is to specify the compiler in the Makefiles and nail compilation to these.
Edit :

pytomc/Makefile
pytomc/libs/libtomc/Makefile.params


Specify CC in pytomc/Makefile to CC = /opt/local/bin/g++-mp-4.5 and CXX in pytomc/libs/libtomc/Makefile.params to CXX = /opt/local/bin/g++-mp-4.5 in the first line.
Re-run compilation. Worked for OSX 10.8.1 / 10.8.2 , XCode 4.4.1 / 4.5 and Macports 2.0.4

------------------------------------------------------------------------

Question
I have XCode 4.3 running and compiling with macports fails with:

Error: Target org.macports.configure returned: configure failure: shell command failed (see log for details)
Error: Failed to install fftw-3
Log for fftw-3 is at: /opt/local/var/macports/logs/_opt_local_var_macports_sources_rsync.macports.org_release_tarballs_ports_math_fftw-3/fftw-3/main.log
main.log:
:info:configure configure: error: C compiler cannot create executables
Answer
After installing Xcode you have to install the command line developer tools manually. Run Xcode and install command line tools through
Xcode --> Open Developer Tool --> Command Line Tools for Xcode.dmg

Follow this link XCode tutorial

------------------------------------------------------------------------

Question
I have downloaded and compiled PyTom but I get this error message during runtime:

openmpi/mca_paffinity_linux.so: undefined symbol: mca_base_param_reg_int or something like: [localhost.localdomain:29329] mca: base: component_find: unable to open /usr/lib64/openmpi/lib/openmpi/mca_shmem_sysv: perhaps a missing symbol, or compiled for a different version of Open MPI? (ignored) -------------------------------------------------------------------------- It looks like opal_init failed for some reason; your parallel process is likely to abort. There are many reasons that a parallel process can fail during opal_init; some of which are due to configuration or environment problems. This failure appears to be an internal failure; here's some additional information (which may only be relevant to an Open MPI developer): opal_shmem_base_select failed --> Returned value -1 instead of OPAL_SUCCESS -------------------------------------------------------------------------- -------------------------------------------------------------------------- It looks like orte_init failed for some reason; your parallel process is likely to abort. There are many reasons that a parallel process can fail during orte_init; some of which are due to configuration or environment problems. This failure appears to be an internal failure; here's some additional information (which may only be relevant to an Open MPI developer): opal_init failed --> Returned value Error (-1) instead of ORTE_SUCCESS -------------------------------------------------------------------------- -------------------------------------------------------------------------- It looks like MPI_INIT failed for some reason; your parallel process is likely to abort. There are many reasons that a parallel process can fail during MPI_INIT; some of which are due to configuration or environment problems. This failure appears to be an internal failure; here's some additional information (which may only be relevant to an Open MPI developer): ompi_mpi_init: ompi_rte_init failed --> Returned "Error" (-1) instead of "Success" (0) -------------------------------------------------------------------------- *** An error occurred in MPI_Init *** on a NULL communicator *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort, *** and potentially your MPI job) [localhost.localdomain:29329] Local abort before MPI_INIT completed successfully; not able to aggregate error messages, and not able to guarantee that all other processes were killed!
Answer
This is a known issue in openmpi when you dynamically link against the openmpi libraries as it is done in PyTom. Here is a link to another software that has this problem and how it is solved: link Thus, you must include the
BASH:
export LD_PRELOAD='/usr/local/openmpi/lib/libmpi.so'
CSH:
setenv LD_PRELOAD '/usr/local/openmpi/lib/libmpi.so'

to 'preload' the library. Of course, you must adjust the path to your local installation of openmpi.

------------------------------------------------------------------------

Question
I have downloaded and compiled PyTom but I get this error message during runtime:

ImportError: No module named pytom
Answer

Your pytom directory must be called pytom. Any other directory name will cause errors because the directory name will be used as the default module search path at runtime.

------------------------------------------------------------------------

Question
I am on OSX trying to compile openmpi from Macports for PyTom. I get a memory leak during compilation!?

Answer
After the release of OSX 10.7.2 and XCode 4.2, a memory leak during compilation of openmpi occurs. MacPorts ticket This can be fixed, however, by using another C++ compiler rather than XCode delivered llvm - clang for compiling gcc44 which openmpi depends on.

sudo port install apple-gcc42 sudo port install gcc44 configure.compiler=apple-gcc-4.2 sudo port install openmpi

That worked as a fix for now

------------------------------------------------------------------------



MPI and parallel processing questions

Question
How do I start a parallel PyTom job?

Answer
That actually depends on your system setup. So far, we have tested OpenMPI successfully on Linux clusters and Mac computers.
For user interactive parallel processing deployed directly from the terminal you have to first create a hostfile.

computerHostname slots=NumberCPUsYouWantToUse
secondComputerHostname slots=NumberCPUsYouWantToUse
#...


In order to run your parallel job, you have to type

mpirun --hostfile youHostFile -c numberOfCPUsUsed pytom yourScript.py


Please note, a queueing system on a large compute cluster may have system specific submission scripts that can not be covered here. Please consult this website for OpenMPI related questions and tutorials.

Question
How can I test my MPI environment using PyTom?

Answer
In order to determine if MPI and PyTom are properly installed you should first run the pytomc/check.py in the PyTom installation directory to see that all libraries are found at runtime. Next, you can run a short script that prints all MPI node ids (ranks) a PyTom job would be running on. Save it as mpiCheck.py.

import pytom_mpi

pytom_mpi.init()
print pytom_mpi.rank()
pytom_mpi.finalise()

Start the script using
mpirun -c 4 /Your/Path/To/pytom/bin/pytom mpiCheck.py
and you shold see the numbers 0 - 3 printed in an arbitrary order depending on the response time of each cpu / machine. You can and should specify the hostfile if you plan on running a PyTom job on multiple machines with the --hostfile parameter (see above).

------------------------------------------------------------------------



PyTom Questions

Question
What is the libtomc notation always refered to for binning?

Answer
Contrary to the binning notation used in TOM, PyTom and libtomc use a different notation for the binning parameter:
the parameter determines the kernel size used for binning
-> 1 means the kernel size is 1 voxel => no binning (downscaling) happens
-> 2 " " kernel size is 2 voxel => equal to tom binning == 1
-> 3 " " kernel size is 3 voxel => unsupported in tom
-> 4 " " kernel size is 4 voxel => equal to tom binning == 2
. . .

------------------------------------------------------------------------

Question
While writing code using the swig interfaced functions, I get the following error:

unsupported operand type(s) for *: 'SwigPyObject' and 'vol'
*' at 0x26e6cc0>

Or similar.

Answer
If you get this error, python for some reason does not know the class type and keeps working with a pointer object to the class. You can resolve this by importing the class before running some code.

from pytom_volume import vol


The code above imports vol before the line error occurs and resolves this bug.

------------------------------------------------------------------------