Page 1 of 1
Error: Insufficient space in Bsend buffer
Posted: 14 Oct 2022, 23:36
by vencels
Hi,
I get an error related to MPI on Windows 10. MPI version is one that comes with Elmer.
The case has ~1M elements, memory >20GB. When running the case in serial without MPI, it works.
Is this buffer size hardcoded (e.g., CheckBuffer @ SParIterComm.F90) or there is a way to increase it?
Code: Select all
SolveLinearSystem: Parallel linear System Solver: iterative
SParIterSolver: Solving linear in parallel with iterative methods
IterSolver: Using iterative method: bicgstabl
IterSolver: Matrix is complex valued
job aborted:
[ranks] message
[0-10] terminated
[11] fatal error
Fatal error in MPI_Bsend: Invalid buffer pointer, error stack:
MPI_Bsend(buf=0x000002037CE64B60, count=149008, MPI_DOUBLE_PRECISION, dest=9, tag=6001, comm=0x84000000) failed
Insufficient space in Bsend buffer; requested 1192064; total buffer size is 6847912
---- error analysis -----
[11] on DESKTOP-KFHJ6TB
mpi has detected a fatal error and aborted ElmerSOlver_mpi
---- error analysis -----
Re: Error: Insufficient space in Bsend buffer
Posted: 17 Oct 2022, 23:57
by kevinarden
There is no MPI supplied with Elmer, Elmer uses the MPI on your machine. However Elmer is programmed to use MPI on your machine, and Bsend buffer is a default size in mpi programming, it can be controlled by the programmer.
https://learn.microsoft.com/en-us/messa ... d-function
Most programmers do not do this by default.
Re: Error: Insufficient space in Bsend buffer
Posted: 24 Oct 2022, 16:39
by raback
Hi Juris, Could you locate the line calling this? Perhaps adding output level you could narrow it down a bit if no proper way under Windows. -Peter
Re: Error: Insufficient space in Bsend buffer
Posted: 10 Apr 2024, 14:21
by vencels
Hi Peter,
I am back with the issue and, this time, with more info. The same problem happens on Windows and Linux, and does not matteron compilation
Recompiling in debug mode does not give the exact place where the error happens. This is the output from Elmer:
Code: Select all
List_ToCRSMatrix: Matrix format changed from List to CRS
SolveLinearSystem: Assuming parallel linear system
ScaleLinearSystem: Scaling diagonal entries to unity in parallel
ScaleLinearSystem: Computing > DiagScaling < vector
ScaleLinearSystem: Assuming complex matrix while scaling
ScaleLinearSystem: Scaling matrix values
ScaleLinearSystem: Scaling Rhs vector
SolveLinearSystem: Linear System Preconditioning: circuit
List_ToCRSMatrix: Changing matrix type with number of non-zeros: 400
List_ToCRSMatrix: Matrix format changed from List to CRS
VectorValuesRange: [min,max,sum] for A: -1.0000000000000000 1.0000000000000004 1743460.8964499999
VectorValuesRange: [min,max,sum] for b: 0.0000000000000000 1.0000000000000000 1.0000000000000000
SolveLinearSystem: Parallel linear System Solver: iterative
SParIterSolver: Solving linear in parallel with iterative methods
IterSolver: Using iterative method: bicgstabl
IterSolver: Matrix is complex valued
IterSolver: Calling complex iterative solver
[DESKTOP-KFHJ6TB:01787] *** An error occurred in MPI_Bsend
[DESKTOP-KFHJ6TB:01787] *** reported by process [375783425,0]
[DESKTOP-KFHJ6TB:01787] *** on communicator MPI COMMUNICATOR 3 SPLIT FROM 0
[DESKTOP-KFHJ6TB:01787] *** MPI_ERR_BUFFER: invalid buffer pointer
[DESKTOP-KFHJ6TB:01787] *** MPI_ERRORS_ARE_FATAL (processes in this communicator will now abort,
[DESKTOP-KFHJ6TB:01787] *** and potentially your MPI job)
I tried increasing buffer size (on Linux)
export MPI_BUFFER_SIZE=2000000000
with no luck.
Any ideas?
Juris
Re: Error: Insufficient space in Bsend buffer
Posted: 11 Apr 2024, 07:59
by vencels
Found the line with a debugger.
SParIterComm.F90, Line 4628:
Code: Select all
IF ( L(j) > 0 ) THEN
CALL MPI_BSEND( VecL(1:L(j),j), L(j), MPI_DOUBLE_PRECISION, &
J-1, 6001, ELMER_COMM_WORLD, ierr )
END IF
Re: Error: Insufficient space in Bsend buffer
Posted: 11 Apr 2024, 11:56
by kevinarden
This function returns an error if there is insufficient buffer space. The amount of available buffer space is controlled by the user by using the MPI_Buffer_attach function.
MPI_BUFFER_ATTACH(BUFFER, SIZE, IERROR)
<type> BUFFER(*)
INTEGER SIZE, IERROR
Re: Error: Insufficient space in Bsend buffer
Posted: 15 Apr 2024, 18:33
by vencels
Thanks for the info, Kevin.
What can we do to fix it?