Trouble with mesh of over 1 billion elements

General discussion about Elmer
Post Reply
edge025
Posts: 28
Joined: 27 Apr 2019, 08:48
Antispam: Yes
Location: NJ, USA
Contact:

Trouble with mesh of over 1 billion elements

Post by edge025 »

Hello,

I am trying to solve a mesh with over 1 billion 3d elements (partitioned over 120 parts). In ElmerSolver_mpi I am getting load mesh errors consistent with integer types perhaps being overrun. Is there a simple way to compile with LONG INT support?

Regards,
Joe
raback
Site Admin
Posts: 4812
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

Re: Trouble with mesh of over 1 billion elements

Post by raback »

Hi Joe,

Nice!

Unfortunately current the integer type is 32 bits for all integers which means max value of 2,147,483,647. So I could see indexing hitting a wall here.

We are aware of the problem but so far there has been no reason to use long integers. It would probably make sense to only alter the global indexing since it will be some time until the local indexing will hit this limit. The global indexing is more localized in the code. Altering all INTEGERs would loose some memory/efficiency in vain.

Are you really solving this large problems. For use the limit has been hit only in benchmarking of Poisson type of equations.

-Peter
edge025
Posts: 28
Joined: 27 Apr 2019, 08:48
Antispam: Yes
Location: NJ, USA
Contact:

Re: Trouble with mesh of over 1 billion elements

Post by edge025 »

Thanks for getting back to me, Peter. I know gfortran has a commandline option to use 8-byte integers instead of 4-byte. Would this make my troubles go away? If so, where can I tell elmer to compile with this option? would anything be needed for mpi?

Are fortran integers the only culprets? Or is there C code as well (I hope not)!

Joe
raback
Site Admin
Posts: 4812
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

Re: Trouble with mesh of over 1 billion elements

Post by raback »

Hi Joe,

How do you create your mesh. I hope internally with "Mesh Levels"? Then the preprocessing with ElmerGrid should not lead to any problems.

There are a number of linear solver libraries and I don't know which of them would work out-of-box. Probably it would be easiest to start with the internal iterative solvers of Elmer.

May I ask what is such a huge case that you're hitting the integer length?

-Peter
edge025
Posts: 28
Joined: 27 Apr 2019, 08:48
Antispam: Yes
Location: NJ, USA
Contact:

Re: Trouble with mesh of over 1 billion elements

Post by edge025 »

Peter,

Thank you for your suggestions! Sorry it took so long to get back to you. I am running a coupled cfd and thermal model on two magnetics structures in a large cabinet being cooled by both water and air.

Fortunately (and unfortunately) I have given up at the present at getting such a large mesh to run. After reading your suggestions above along with that you have suggested to other people in other posts, I have been able to pursue a much better path to solution as well as learn much more about Elmer! Thank you again for such valuable help!

Regards,
Joe
Post Reply