Page 1 of 1

The status of ElmerIce tests

Posted: 19 Feb 2016, 16:03
by rgladstone
Hi all, I have recently tried to run all the elmer tests on my laptop computer. Some of them fail :o, including two of the elmerice tests. I repeated the elmerice tests that failed on Taito, the CSC hpc machine. In both cases I was using the latest version of elmerice from github (checked out within the last day or so) and compiled myself. On Taito one them passed and one of them failed. See below.

I'd be interested to hear from anyone else for whom these tests either pass or fail, and also if there any other tests that fail for you but pass for me. We really need to have a robust set of tests for the sake of testing our changes before committing to the repository.

My laptop is running Linux Mint VMware under a Windows 10 host. I am using gnu compliers. I can give more details if you like.

There are 438 tests altogether. Here is the ctest output listing the failed tests on my laptop:

The following tests FAILED:
20 - ThicknessSolver (Failed)
28 - Damage (Failed)
80 - PhaseChangeB (Failed)
251 - freesurf_maxd (Failed)
252 - freesurf_maxd_np2 (Failed)
253 - freesurf_maxd_np4 (Failed)
274 - CoilSolverTwoLoops (Failed)
275 - CoilSolverTwoLoops_np2 (Failed)
276 - CoilSolverTwoLoops_np4 (Failed)
381 - structmap5 (Failed)
Errors while running CTest


I had a closer look at the elmerice failures. This is how the thickness solver fails on my laptop:

FlowSolve: -------------------------------------
FlowSolve: NAVIER-STOKES ITERATION 11
FlowSolve: -------------------------------------
FlowSolve:
FlowSolve: Starting Assembly...
FlowSolve: Assembly done
FlowSolve: Dirichlet conditions done
ComputeChange: NS (ITER=11) (NRM,RELC): ( 9.3953629 0.17507641E-12 ) :: navier-stokes
FlowSolve: iter: 11 Assembly: (s) 1.73 18.57
FlowSolve: iter: 11 Solve: (s) 0.92 10.83
FlowSolve: Result Norm : 9.3953629195726087
FlowSolve: Relative Change : 1.7507641241383808E-013
StructuredProjectToPlane: ------------------------------------------
StructuredProjectToPlane: Performing projection on a structured mesh
StructuredProjectToPlane: ------------------------------------------
DetectExtrudedStructure: Top and bottom pointer init time: 1.9800000000103068E-004
DetectExtrudedStructure: Top and bottom pointer init rounds: 5
DetectExtrudedStructure: Number of nodes at the top: 121
DetectExtrudedStructure: Number of nodes at the bottom: 121
StructuredProjectToPlane: Projection time: 1.7899999999926308E-004
StructuredProjectToPlane: ------------------------------------------
ThicknessSolver (h): Memory allocations done
ThicknessSolver (h): Non-linear Iteration 1 out of max. 50
1500 NaN
1501 NaN
ERROR:: IterSolve: Failed convergence tolerances.
ERROR:: ComputeChange: Norm of solution appears to be NaN

This is certainly a failure. Does this test fail for anyone else? Does this test pass for anyone else? Who "owns" the test? Does it work for you with a standard recent checkout of the elmerice branch? I ran this test on Taito and it also failed, though I have not looked closely at the output.


The other elmerice test that fails is the Damage test. Here is the last bit of output from my laptop:
...
SaveScalars: All done
SaveScalars: -----------------------------------------
WARNING:: CompareToReferenceSolution: Solver 7 FAILED: Norm = 8.14727269E-02 RefNorm = 8.14660480E-02
CompareToReferenceSolution: Relative Error to reference norm: 8.198387E-05
WARNING:: CompareToReferenceSolution: FAILED 1 tests out of 1!
ElmerSolver: *** Elmer Solver: ALL DONE ***
ElmerSolver: The end
SOLVER TOTAL TIME(CPU,REAL): 217.46 225.42
ELMER SOLVER FINISHED AT: 2016/02/18 23:31:45

This looks like a successful simulation, with perhaps a tolerance that is too strict? The test passed on Taito. Who "owns" this test? Do you think we should increase the tolerance?

Regards,
Rupert

Re: The status of ElmerIce tests

Posted: 19 Feb 2016, 18:52
by raback
Hi

At least 6 of the failed tests are very new. Are you certain that you reconfigured with the latest source. Just remake does not guarantee that the tests are performed properly. Generally, I agree. We should strive that all tests pass because if we are even slightly sloppy some bugs may be introduced without anybody noticing.

I must confess that I seldom run the elmerice test due to the large resources that some tests require. I think that testing of elmerice when merging with devel is therefore quite essential.

-Peter

Re: The status of ElmerIce tests

Posted: 19 Feb 2016, 20:29
by rgladstone
Hi Peter, I don't really understand the cmake build system to be honest. So a git pull and running make is not sufficient? I'll empty the build directory and run the build script again from scratch... that should do it! Then I'll run the tests again and see if we get anything different. In general should running the build script (which centers around a cmake command line) be sufficient? Or does the presence of the cmake cache file mean that one should empty the build directory first?

Yes, the full elmerice tests take a while. The elmerice-fast test list is quick though.

Cheers,
Rupert

Re: The status of ElmerIce tests

Posted: 19 Feb 2016, 20:38
by raback
Hi

I'm no expert but it seems that running the script is needed, no clean build is necessary. Simple remake does not seem to consider the changed tests properly.

-Peter

Re: The status of ElmerIce tests

Posted: 19 Feb 2016, 22:45
by rgladstone
After a clean build, the following tests still fail...

The following tests FAILED:
20 - ThicknessSolver (Failed)
28 - Damage (Failed)
252 - freesurf_maxd_np2 (Failed)
253 - freesurf_maxd_np4 (Failed)
Errors while running CTest

A separate email discussion is now underway regarding the ThicknessSolver, the outcome of which will be added to this thread at some point.

Does anyone know who is responsible for the Damage test? In particular would it be acceptable to widen the tolerance slightly?

I don't have any comment on the other tests, but let me know if you are keen to see the output from them.

Regards,
Rupert

Re: The status of ElmerIce tests

Posted: 21 Feb 2016, 11:19
by tzwinger
Hi Rupert,
as the freesurf_maxd serial test runs through, I presume there is a problem with parallel set up. Could you send me output to see where it fails (is it the norm?)?

Damage test never worked for me and the Thickness problem is - as I reckon - being worked over.

Regards,

Thomas

Re: The status of ElmerIce tests

Posted: 24 Feb 2016, 18:15
by rgladstone
Hi all,
in my previous message I had 4 failed tests.

The ThicknessSolver has been corrected by Peter in the devel branch. Laure Tavard has verified this fix. Next time Thomas merges devel to elmerice it will be available to all elmerice users. Let me or Thomas know if you need it sooner than that.

Laure is looking in to adjusting the tolerance for the Damage solver.

The other two failures were due to a minor issue with my installation. I had some issues with my installation, and found that cleaning everything except the cmake cache from by build directory worked to give a clean install consistent with the old one. This is on one of the Linux Mint images provided by the CSC team for training courses.

So I think I've resolved my test failures, but if anyone has other test failures that they cannot resolve it would be interesting for them to be posted here. It is especiallly important for anyone who regularly commits their developments to the github repository.

Thanks to Thomas, Peter, Laure and Lenneke for helping sort out the tests.

Cheers,
Rupert