Saving results & restarting

Discussion about coding and new developments
raback
Site Admin
Posts: 4812
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

Saving results & restarting

Post by raback »

Hi All

I did some tuning of the Output File saving and restarting (subroutines SaveResult & LoadRestartFile). This is to inform of the changes but also to warn if something is not working on your restart as yor are accustomed to. All the current test cases do run but there could be special cases out there that do not, even though all changes should be backward compatible.

Some changes:
  • Output file version was changed from 2 to 3. There are some changes in the header section. The possible problem could be that you save with new Elmer version and would like to read with the old one.
  • You can now give list of fields to save: "Output Variable i", i=1,2,3,...
  • You can now give list of fields to read: "Restart Variable i", i=1,2,3,...
  • The global variables, such as "time", "timestep size" etc., are saved when "Output Global Variables" is set True.
  • The coordinates are saved when "Save Coordinates" is set True (they are also saved for certain free surface problems).
  • Quite a lot of error detection and warnings were added
There is a test case under tests/NaturalConvectionRestart that helps to understand the changes. Note that there is a "init.sif" that saves and "case.sif" that restarts.

-Peter
ygong
Posts: 15
Joined: 08 May 2013, 16:51
Antispam: Yes

Re: Saving results & restarting

Post by ygong »

Hi Peter,
I have got some problems with the restarting since this week which I have never had before.
In one case, there are some warnings when reading from certain time steps saying

LoadRestartFile: Reading timestep: 1
WARNING:: LoadRestartFile: Could not associate variable to solver: ***
LoadRestartFile: Creating variable: ***

But the simulation finished and it seemed the variables have been read somehow.

In another case, there are some warnings saying

LoadRestartFile: Reading timestep: 1
WARNING:: LoadRestartFile: Could not associate variable to solver: ***
LoadRestartFile: Creating variable: ***

WARNING:: LoadRestartFile: Variable is not present for reading: ***

Then the simulation got crushed with a segmentation fault : forrtl: severe (174): SIGSEGV, segmentation fault occurred
I restarted from position 0 in both cases.
Do you think this is about the restarting subroutine. I saw you did this tuning at May, but my problem occurred after 17 June.

Cheers,
Yongmei
raback
Site Admin
Posts: 4812
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

Re: Saving results & restarting

Post by raback »

Hi Yongmei

Maybe you could share the problematic case.

-Peter
ygong
Posts: 15
Joined: 08 May 2013, 16:51
Antispam: Yes

Re: Saving results & restarting

Post by ygong »

Hi Peter,

Here is the sif file, output file and error file. This case went fine before. So I don't think the problem is really about the sif file.

Cheers,
Yongmei
Attachments
ASF_ti_2_errors.txt
(5.42 KiB) Downloaded 561 times
ASF_ti_2_output.txt
(21.31 KiB) Downloaded 565 times
ASF_ti_2.sif
(13.35 KiB) Downloaded 613 times
Martina
Posts: 39
Joined: 17 Apr 2013, 14:06
Antispam: Yes

Re: Saving results & restarting

Post by Martina »

Hi Peter and Yongmei,

I have reproduced some of Yongmei's errors - once by running her simulations on my account and once by using my own simulations (very similar setup) which also have been working earlier. So I don't think the main problem is not specific to her own userfunctions or her recent changes in sif-files.

The warnings

Code: Select all

WARNING:: LoadRestartFile: Could not associate variable to solver
are in my simulations as they should limited to variables which are really not associated to a solver, however "flow solution" appears also in this list. ComputeDevStress finds the solution trivially zero (mine and Yongmei's case) even though velocities are correctly read and I'm wondering if this is because of not having associated + having recreated "flow solution"?
(freesurf(grad) also appears in that list (defined via Freesurf Name = String "FreeSurf" in another Solver) but it's without consequences that it is recreated.)

The

Code: Select all

WARNING:: LoadRestartFile: Variable is not present for reading:
preceding the seg.fault is also in mine and Yongmei's case with respect to "flow solution 1".

So I'm wondering if there is an issue related how to "flow solution" is dealt with (not being associated to a solver and than recreated) in the new release (something between r6158 and r6235, probably r6184 when you added these "not associated to a solver" tests).

I will look more in that again tomorrow, Yongmei, now I have to see to something else.

Martina
raback
Site Admin
Posts: 4812
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

Re: Saving results & restarting

Post by raback »

Hi Martina & Youngmei

I was little bit discouraged by Youngmei's large parallel case as debugging is easiest done for small cases. So I followed Martina's hints and hope that I could remove the problem.

The fact that variable is not associated to any solver is not detrimental. This will happen if you have different set of solvers. For restart each variable should find it's own solver. So you should not worry too much about this warning. In fact, it might be dropped as a warning. The previous version just put all solvers to be 'fs' which was quite wrong.

Now there really was a bug in how 'flow solution' is checked. If flow solution was not previously there, it was not allocated for as a whole but its components were allocated separately. This then results to a problem since even though 'Velocity 1' and 'Velocity 2' would be there, Velocity as a vector will not.

Thank you for reporting!

-Peter
Martina
Posts: 39
Joined: 17 Apr 2013, 14:06
Antispam: Yes

Re: Saving results & restarting

Post by Martina »

Thanks, great news Peter!

I'm trying on taito now, first runs look good, but I won't be able before tomorrow morning to see if the problem is really solved (if not, I'll post it here).

Martina
ygong
Posts: 15
Joined: 08 May 2013, 16:51
Antispam: Yes

Re: Saving results & restarting

Post by ygong »

Hi Peter and Martina
Dorothee and me had another issue with restarting a simulation that is without Navier-Stockes solver from one that has.
(We are using the inverse Adjoint method this time and it has to be mentionned that we don't have the same problem with the Inverse Robin method...)

In the sif file we restarted from we wrote:

Code: Select all

Calculate Loads = Logical true
And we got the following error message and a seg. fault :

Code: Select all

WARNING:: LoadRestartFile: Could not associate variable to solver: flow solution loads 1
LoadRestartFile: Creating variable: flow solution loads 1
WARNING:: LoadRestartFile: Could not associate variable to solver: flow solution loads 2
LoadRestartFile: Creating variable: flow solution loads 2
WARNING:: LoadRestartFile: Could not associate variable to solver: flow solution loads 3
LoadRestartFile: Creating variable: flow solution loads 3
WARNING:: LoadRestartFile: Could not associate variable to solver: flow solution loads 4
LoadRestartFile: Creating variable: flow solution loads 4
WARNING:: LoadRestartFile: Could not associate variable to solver: adjoint 1
WARNING:: LoadRestartFile: Could not associate variable to solver: adjoint 2
WARNING:: LoadRestartFile: Could not associate variable to solver: adjoint 3
WARNING:: LoadRestartFile: Could not associate variable to solver: adjoint 4
WARNING:: LoadRestartFile: Could not associate variable to solver: vsurfini 1
LoadRestartFile: Creating variable: vsurfini 1
WARNING:: LoadRestartFile: Could not associate variable to solver: vsurfini 2
LoadRestartFile: Creating variable: vsurfini 2
WARNING:: LoadRestartFile: Could not associate variable to solver: freesurf
LoadRestartFile: Creating variable: freesurf
WARNING:: LoadRestartFile: Could not associate variable to solver: freesurfgrad1
LoadRestartFile: Creating variable: freesurfgrad1
WARNING:: LoadRestartFile: Could not associate variable to solver: freesurfgrad2
LoadRestartFile: Creating variable: freesurfgrad2
LoadRestartFile: Creating variable: viscosity
LoadRestartFile: Total number of dofs to load: 27
LoadRestartFile: Reading time sequence:    1.000E+00
LoadRestartFile: Reading timestep: 1
WARNING:: LoadRestartFile: Variable is not present for reading: flow solution loads 1
WARNING:: LoadRestartFile: Variable is not present for reading: flow solution loads 1
WARNING:: LoadRestartFile: Variable is not present for reading: flow solution loads 1
What do you think? Maybe another bug with flow solution loads?

Yongmei and Dorothée
DVallot
Posts: 1
Joined: 01 Aug 2013, 11:32
Antispam: Yes

Re: Saving results & restarting

Post by DVallot »

Hej,

I've solved the problem by adding the Navier-Stokes solver with exec = never to the second sif file. Don't know if it is satisfying but it works!

Dorothée
ygong
Posts: 15
Joined: 08 May 2013, 16:51
Antispam: Yes

Re: Saving results & restarting

Post by ygong »

Hi,
Good to know! But I think we don't have navier -stocks solver in the second run. But I'll check anyway.
Cheers,
Yongmei
Post Reply