ctest failed due to segmentation fault errors

Extension of Elmer in computational glaciology
michelle
Posts: 6
Joined: 12 Mar 2013, 02:42
Antispam: Yes

ctest failed due to segmentation fault errors

Post by michelle »

Hi everyone,

We are trying to build Elmer/Ice on Mac OSX 10.10.3 and after finally getting the code to compile we are getting errors for nearly every test when running ctest. It failed at 338/340 tests. We get the same error when we try to run tests individually.

The error in the test-stderr.log files is: "Program received signal SIGSEGV: Segmentation Fault - invalid memory reference"

We would really appreciate any help to navigate this error. We are not experts in this and don't have much local support to move forward. And would really like to starting using Elmer/Ice as soon as possible.

Thanks!
tzwinger
Site Admin
Posts: 99
Joined: 24 Aug 2009, 12:20
Antispam: Yes

Re: ctest failed due to segmentation fault errors

Post by tzwinger »

Hi Michelle,
this is a little bit tricky to analyze remotely, not only (but mainly) because of OSX. I think a good start would be, if you could send the log of the seg-faulted test (should be under Testing/Temporary in your build-directory) and perhaps give us hints how you configured your environment (homebrew? what components?).

Best wishes,

Thomas
michelle
Posts: 6
Joined: 12 Mar 2013, 02:42
Antispam: Yes

Re: ctest failed due to segmentation fault errors

Post by michelle »

Hi Thomas,

We installed the dependent packages using Homebrew and then built Elmer and Elmer/Ice using cmake and the following compile script:
#!/bin/bash
CMAKE=cmake
# Installation directory (set these!)
TIMESTAMP=$(date +"%m-%d-%y")
ELMER_REV="Elmer_devel_${TIMESTAMP}"
ELMERSRC="/Users/johnchristian/Elmer/elmerice"
BUILDDIR="/Users/johnchristian/Elmer/builddir"
IDIR="/Users/johnchristian/Elmer/install"
echo "Building Elmer from within " ${BUILDDIR}
echo "installation into " ${IDIR}
cd ${BUILDDIR}
pwd
ls -ltr
echo $CMAKE $ELMERSRC
$CMAKE $ELMERSRC \
-DCMAKE_INSTALL_PREFIX=$IDIR \
-DWITH_MPI:BOOL=TRUE \
-DWITH_Mumps:BOOL=FALSE \
-DWITH_Hypre:BOOL=FALSE \
-DWITH_Trilinos:BOOL=FALSE \
-DWITH_ELMERGUI:BOOL=FALSE \
-DWITH_ElmerIce:BOOL=TRUE \
-Wno-dev
# change the -j4 to the number of available cores on your system
make -j4 && sudo make install
# this automatically links /usr/local/Elmer-devel to your new built
#sudo rm /usr/local/Elmer-devel
#sudo ln -s $IDIR /usr/local/Elmer-devel

This latest build did not include Hypre and Mumps, but we tried a separate build that did include these and the errors when running ctest were the same. Attached is a log (part of LastTest.log), but the error messages seem to be piped only to the log file in the directory for each test. For Buoyancy this is the log:
TEST CASE 1
TEST CASE 1
Program received signal SIGSEGV: Segmentation fault - invalid memory reference.
Backtrace for this error:
#0 0x107b1d4f2
#1 0x107b1dcae
#2 0x7fff8bdf8f19

This segmentation fault error seems to occur at all tests that fail.

We really appreciate any advice and let me know if more information is needed. Thanks for the help.

Michelle
Attachments
LastTest_part.txt
(1019.28 KiB) Downloaded 673 times
tzwinger
Site Admin
Posts: 99
Joined: 24 Aug 2009, 12:20
Antispam: Yes

Re: ctest failed due to segmentation fault errors

Post by tzwinger »

Hello,
checking your logs, it seems to me that the ctests want to access directories that
a) either do not exist
b) are not set for permission

Code: Select all

CMake Error at /Users/johnchristian/Elmer/elmerice/elmerice/Tests/test_macros.cmake:70 (FILE):
  file failed to open for reading (No such file or directory):

    /Users/johnchristian/Elmer/elmerice/elmerice/Tests/Buoyancy/TEST.PASSED
I got the same error when I was cross-compiling on a cluster and then was trying to run the tests on a compute node without access to the installation directory - might that apply to your situation?

Regards,

Thomas
michelle
Posts: 6
Joined: 12 Mar 2013, 02:42
Antispam: Yes

Re: ctest failed due to segmentation fault errors

Post by michelle »

Thanks for the reply. Unfortunately we still cannot get past the segmentation fault errors, and all tests still fail.

While the log does say that it crashes at test_macros.cmake this file does exist, and I tried changing permissions on this file to be executable but that did not help and it has the same permissions as all of the other .cmake files (644). Should the *.cmake files have different permissions?

There is no file TEST.PASSED but I am assuming that this is generated only if the test passes.

I tried to run the test Buoyancy on the command line in that directory and it gets through "ElmerGrid 1 2 mesh.grd" but crashes with the error I have given already when running "ElmerSolver buoyancy.sif". I added the path to the installation directory in ~/.bashrc but that did not help. If I change the path to ELMER_HOME to be something that does not direct to the xx/install directory then it fails very very quickly because it did not find BINARY_DIR, which is in xx/install.

This seems difficult to troubleshoot over the forum but any ideas are welcome. Many thanks for the help.
tzwinger
Site Admin
Posts: 99
Joined: 24 Aug 2009, 12:20
Antispam: Yes

Re: ctest failed due to segmentation fault errors

Post by tzwinger »

Michelle,
the ctest is meant for testing _before_ installation, hence falls back on the binaries inside your build-tree. Hence, all paths there are more or less hard-coded. If you change something after compiling (renaming, copying), this might cause problems. Else, I am lacking any explanation for that.
Concerning the repeatedly reported seg-fault: Checking the output of the Buoyancy test, I think it actually happens while running Elmer, as the closing "END TEST CASE 1" is not print on screen. You could gather some information where it crashes by running the test case (for instance Buoancy) through a debugger (for instance gdb) or an analysis tool like valgrind - at least the latter is available for OS X.

Cheers,

Thomas
raback
Site Admin
Posts: 4812
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

Re: ctest failed due to segmentation fault errors

Post by raback »

Hi Michelle

Looking at the output it seems that only ElmerGrid is run. I cannot see any output of ElmerSolver. Do you have the ElmerSolver binary at all?

The TEST.PASSED is written at the end of Solver. It gets value 1 in success and value 0 otherwise. But now there is no such file as ElmerSolver is not run.

-Peter
michelle
Posts: 6
Joined: 12 Mar 2013, 02:42
Antispam: Yes

Re: ctest failed due to segmentation fault errors

Post by michelle »

Thanks for the replies and input. We have had help here to try a number of things with our attempted Mac build, but unfortunately none of them worked. We are still getting segmentation faults at every test. We do have ElmerSolver in the right place and it seems to be found. We think there is an issue with the Mac OSX 10.10 (Yosemite) that is giving the problem, as previous installs on different OS X did not have this problem. The build displayed many warning errors so it seemed possible that the clang compiler was a problem, but we changed to use gcc and while we had almost no warnings we still had the segmentation faults with ctest.

We have tried another option, which is to install on CentOS7. The build of Elmer went smoothly thanks to a very helpful post about building on CentOS6.5. The Elmer/Ice build has been successful (only 4 tests failed) with Mumps and Hypre excluded, but not yet with these included because they need to be installed. Two questions:

- If we do not need both Mumps and Hypre, which one should we use?
- I was asked to check if you have a make script for installation of Mumps5.0 on CentOS7, or know of a place where one can be found? In your experience, should Mumps or Hypre be easier to install on CentOS7?

If you have other suggestions related to a build on CentOS7 we are glad for the input. Thanks again for all of the help.

Michelle
tzwinger
Site Admin
Posts: 99
Joined: 24 Aug 2009, 12:20
Antispam: Yes

Re: ctest failed due to segmentation fault errors

Post by tzwinger »

Hi Michelle,
we are working on a documented plan how to install on OSX. For your CentOS installation: you definitely want to have MUMPS and easily can forget about Hypre if you are doing glaciological problems. On RHEL I found a RPM package for MUMPS in the EPEL repository and this looks very promising (http://pkgs.org/centos-7/epel-x86_64/MU ... 4.rpm.html). Else, compiling MUMPS is not so tricky - I can send you a Makefile.inc for Debian-style Linux. You just need ScaLapack (which itself needs BLACS) compiled, but these should be available as RPM's.

Cheers,

Thomas
tzwinger
Site Admin
Posts: 99
Joined: 24 Aug 2009, 12:20
Antispam: Yes

Re: ctest failed due to segmentation fault errors

Post by tzwinger »

One more hint I got yesterday from a colleague for OSX: one frequent cause of troubles could be the Blas library provided by Apple. If I understood correctly, there might be a way to replace it by one from homebrew. Dunno if this helps

Thomas
Post Reply