Error: Insufficient space in Bsend buffer

[vencels]

Hi,

I get an error related to MPI on Windows 10. MPI version is one that comes with Elmer.
The case has ~1M elements, memory >20GB. When running the case in serial without MPI, it works.

Is this buffer size hardcoded (e.g., CheckBuffer @ SParIterComm.F90) or there is a way to increase it?

Code: Select all

SolveLinearSystem: Parallel linear System Solver: iterative
SParIterSolver: Solving linear in parallel with iterative methods
IterSolver: Using iterative method: bicgstabl
IterSolver: Matrix is complex valued

job aborted:
[ranks] message

[0-10] terminated

[11] fatal error
Fatal error in MPI_Bsend: Invalid buffer pointer, error stack:
MPI_Bsend(buf=0x000002037CE64B60, count=149008, MPI_DOUBLE_PRECISION, dest=9, tag=6001, comm=0x84000000) failed
Insufficient space in Bsend buffer; requested 1192064; total buffer size is 6847912

---- error analysis -----

[11] on DESKTOP-KFHJ6TB
mpi has detected a fatal error and aborted ElmerSOlver_mpi

---- error analysis -----

[kevinarden]

There is no MPI supplied with Elmer, Elmer uses the MPI on your machine. However Elmer is programmed to use MPI on your machine, and Bsend buffer is a default size in mpi programming, it can be controlled by the programmer.

https://learn.microsoft.com/en-us/messa ... d-function

Most programmers do not do this by default.

[raback]

Hi Juris, Could you locate the line calling this? Perhaps adding output level you could narrow it down a bit if no proper way under Windows. -Peter

[vencels]

Hi Peter,

I am back with the issue and, this time, with more info. The same problem happens on Windows and Linux, and does not matteron compilation
Recompiling in debug mode does not give the exact place where the error happens. This is the output from Elmer:

Code: Select all

List_ToCRSMatrix: Matrix format changed from List to CRS
SolveLinearSystem: Assuming parallel linear system
ScaleLinearSystem: Scaling diagonal entries to unity in parallel
ScaleLinearSystem: Computing > DiagScaling < vector
ScaleLinearSystem: Assuming complex matrix while scaling
ScaleLinearSystem: Scaling matrix values
ScaleLinearSystem: Scaling Rhs vector
SolveLinearSystem: Linear System Preconditioning: circuit
List_ToCRSMatrix: Changing matrix type with number of non-zeros: 400
List_ToCRSMatrix: Matrix format changed from List to CRS
VectorValuesRange:  [min,max,sum] for A:  -1.0000000000000000        1.0000000000000004        1743460.8964499999
VectorValuesRange:  [min,max,sum] for b:   0.0000000000000000        1.0000000000000000        1.0000000000000000
SolveLinearSystem: Parallel linear System Solver: iterative
SParIterSolver: Solving linear in parallel with iterative methods
IterSolver: Using iterative method: bicgstabl
IterSolver: Matrix is complex valued
IterSolver: Calling complex iterative solver
[DESKTOP-KFHJ6TB:01787] *** An error occurred in MPI_Bsend
[DESKTOP-KFHJ6TB:01787] *** reported by process [375783425,0]
[DESKTOP-KFHJ6TB:01787] *** on communicator MPI COMMUNICATOR 3 SPLIT FROM 0
[DESKTOP-KFHJ6TB:01787] *** MPI_ERR_BUFFER: invalid buffer pointer
[DESKTOP-KFHJ6TB:01787] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[DESKTOP-KFHJ6TB:01787] ***    and potentially your MPI job)

I tried increasing buffer size (on Linux)
export MPI_BUFFER_SIZE=2000000000
with no luck.

Any ideas?

Juris

[vencels]

Found the line with a debugger.

SParIterComm.F90, Line 4628:

Code: Select all

     IF ( L(j) > 0 ) THEN
        CALL MPI_BSEND( VecL(1:L(j),j), L(j), MPI_DOUBLE_PRECISION, &
                 J-1, 6001, ELMER_COMM_WORLD, ierr )
     END IF

[kevinarden]

This function returns an error if there is insufficient buffer space. The amount of available buffer space is controlled by the user by using the MPI_Buffer_attach function.

MPI_BUFFER_ATTACH(BUFFER, SIZE, IERROR)
<type> BUFFER(*)
INTEGER SIZE, IERROR

[vencels]

Thanks for the info, Kevin.

What can we do to fix it?