Error: Insufficient space in Bsend buffer

Discussion about building and installing Elmer
Post Reply
vencels
Posts: 67
Joined: 20 Sep 2016, 17:05
Antispam: Yes
Location: Latvia
Contact:

Error: Insufficient space in Bsend buffer

Post by vencels »

Hi,

I get an error related to MPI on Windows 10. MPI version is one that comes with Elmer.
The case has ~1M elements, memory >20GB. When running the case in serial without MPI, it works.

Is this buffer size hardcoded (e.g., CheckBuffer @ SParIterComm.F90) or there is a way to increase it?

Code: Select all

SolveLinearSystem: Parallel linear System Solver: iterative
SParIterSolver: Solving linear in parallel with iterative methods
IterSolver: Using iterative method: bicgstabl
IterSolver: Matrix is complex valued

job aborted:
[ranks] message

[0-10] terminated

[11] fatal error
Fatal error in MPI_Bsend: Invalid buffer pointer, error stack:
MPI_Bsend(buf=0x000002037CE64B60, count=149008, MPI_DOUBLE_PRECISION, dest=9, tag=6001, comm=0x84000000) failed
Insufficient space in Bsend buffer; requested 1192064; total buffer size is 6847912

---- error analysis -----

[11] on DESKTOP-KFHJ6TB
mpi has detected a fatal error and aborted ElmerSOlver_mpi

---- error analysis -----
kevinarden
Posts: 2419
Joined: 25 Jan 2019, 01:28
Antispam: Yes

Re: Error: Insufficient space in Bsend buffer

Post by kevinarden »

There is no MPI supplied with Elmer, Elmer uses the MPI on your machine. However Elmer is programmed to use MPI on your machine, and Bsend buffer is a default size in mpi programming, it can be controlled by the programmer.

https://learn.microsoft.com/en-us/messa ... d-function

Most programmers do not do this by default.
raback
Site Admin
Posts: 4853
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

Re: Error: Insufficient space in Bsend buffer

Post by raback »

Hi Juris, Could you locate the line calling this? Perhaps adding output level you could narrow it down a bit if no proper way under Windows. -Peter
vencels
Posts: 67
Joined: 20 Sep 2016, 17:05
Antispam: Yes
Location: Latvia
Contact:

Re: Error: Insufficient space in Bsend buffer

Post by vencels »

Hi Peter,

I am back with the issue and, this time, with more info. The same problem happens on Windows and Linux, and does not matteron compilation
Recompiling in debug mode does not give the exact place where the error happens. This is the output from Elmer:

Code: Select all

List_ToCRSMatrix: Matrix format changed from List to CRS
SolveLinearSystem: Assuming parallel linear system
ScaleLinearSystem: Scaling diagonal entries to unity in parallel
ScaleLinearSystem: Computing > DiagScaling < vector
ScaleLinearSystem: Assuming complex matrix while scaling
ScaleLinearSystem: Scaling matrix values
ScaleLinearSystem: Scaling Rhs vector
SolveLinearSystem: Linear System Preconditioning: circuit
List_ToCRSMatrix: Changing matrix type with number of non-zeros: 400
List_ToCRSMatrix: Matrix format changed from List to CRS
VectorValuesRange:  [min,max,sum] for A:  -1.0000000000000000        1.0000000000000004        1743460.8964499999
VectorValuesRange:  [min,max,sum] for b:   0.0000000000000000        1.0000000000000000        1.0000000000000000
SolveLinearSystem: Parallel linear System Solver: iterative
SParIterSolver: Solving linear in parallel with iterative methods
IterSolver: Using iterative method: bicgstabl
IterSolver: Matrix is complex valued
IterSolver: Calling complex iterative solver
[DESKTOP-KFHJ6TB:01787] *** An error occurred in MPI_Bsend
[DESKTOP-KFHJ6TB:01787] *** reported by process [375783425,0]
[DESKTOP-KFHJ6TB:01787] *** on communicator MPI COMMUNICATOR 3 SPLIT FROM 0
[DESKTOP-KFHJ6TB:01787] *** MPI_ERR_BUFFER: invalid buffer pointer
[DESKTOP-KFHJ6TB:01787] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[DESKTOP-KFHJ6TB:01787] ***    and potentially your MPI job)
I tried increasing buffer size (on Linux)
export MPI_BUFFER_SIZE=2000000000
with no luck.

Any ideas?

Juris
vencels
Posts: 67
Joined: 20 Sep 2016, 17:05
Antispam: Yes
Location: Latvia
Contact:

Re: Error: Insufficient space in Bsend buffer

Post by vencels »

Found the line with a debugger.

SParIterComm.F90, Line 4628:

Code: Select all

     IF ( L(j) > 0 ) THEN
        CALL MPI_BSEND( VecL(1:L(j),j), L(j), MPI_DOUBLE_PRECISION, &
                 J-1, 6001, ELMER_COMM_WORLD, ierr )
     END IF
kevinarden
Posts: 2419
Joined: 25 Jan 2019, 01:28
Antispam: Yes

Re: Error: Insufficient space in Bsend buffer

Post by kevinarden »

This function returns an error if there is insufficient buffer space. The amount of available buffer space is controlled by the user by using the MPI_Buffer_attach function.

MPI_BUFFER_ATTACH(BUFFER, SIZE, IERROR)
<type> BUFFER(*)
INTEGER SIZE, IERROR
vencels
Posts: 67
Joined: 20 Sep 2016, 17:05
Antispam: Yes
Location: Latvia
Contact:

Re: Error: Insufficient space in Bsend buffer

Post by vencels »

Thanks for the info, Kevin.

What can we do to fix it?
raback
Site Admin
Posts: 4853
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

Re: Error: Insufficient space in Bsend buffer

Post by raback »

Hi Juris,

What if you increase the size at line 4602. E.g. to
CALL CheckBuffer( 8*TotalL + ParEnv % NumOfNeighbours*(1+MPI_BSEND_OVERHEAD) )

-Peter
vencels
Posts: 67
Joined: 20 Sep 2016, 17:05
Antispam: Yes
Location: Latvia
Contact:

Re: Error: Insufficient space in Bsend buffer

Post by vencels »

Hi Peter,

I ran the test with/without change a couple of times, your suggestion fixes the problem.
raback
Site Admin
Posts: 4853
Joined: 22 Aug 2009, 11:57
Antispam: Yes
Location: Espoo, Finland
Contact:

Re: Error: Insufficient space in Bsend buffer

Post by raback »

Hi Juris,

Great! I just committed the fix to devel.

-Peter
Post Reply