GroupBasedParameter error when using parallel

Hello everyone,
I am getting a critical error while running a project with a parallel capability (Petsc) and simulation freeze after that. The output message is below:

This is the similar problem to Zhang Ning post in gitlab (Issue), where he said this could be solved after rebasing to latest development version but I am still getting this error.

Way to reproduce the problem

  1. Compile OGS (CC=mpicc CXX=mpic++ cmake -G Ninja -DOGS_USE_PETSC=ON -DOGS_USE_MFRONT=ON -DCMAKE_BUILD_TYPE=Release …/ogs)
  2. Run parallel model with GroupBasedParameter. (
    3D_Trial_Model.zip (145.2 KB))

laxman@laxman-HP-15-Notebook-PC:~/build/bin$ mpirun -np 3 --oversubscribe ogs /home/laxman/Desktop/3D_Trial_Model/Trial.prj
info: This is OpenGeoSys-6 version 6.4.0-1036-gdd7be4565.
info: This is OpenGeoSys-6 version 6.4.0-1036-gdd7be4565.
info: This is OpenGeoSys-6 version 6.4.0-1036-gdd7be4565.
info: OGS started on 2021-09-05 04:26:20+0545.
info: OGS started on 2021-09-05 04:26:20+0545.
info: OGS started on 2021-09-05 04:26:20+0545.
[0] info: Reading corresponding part of mesh data from binary file /home/laxman/Desktop/3D_Trial_Model/mesh …
[2] info: Reading corresponding part of mesh data from binary file /home/laxman/Desktop/3D_Trial_Model/mesh …
[1] info: Reading corresponding part of mesh data from binary file /home/laxman/Desktop/3D_Trial_Model/mesh …
[0] warning: Could not open file ‘/home/laxman/Desktop/3D_Trial_Model/mesh_partitioned_node_properties_cfg3.bin’.
You can ignore this warning if the mesh does not contain node-wise property data.
[0] info: [time] Reading the mesh took 0.009408 s.
[0] info: Reading corresponding part of mesh data from binary file /home/laxman/Desktop/3D_Trial_Model/bottom …
[0] info: [time] Reading the mesh took 0.002045 s.
[0] info: Reading corresponding part of mesh data from binary file /home/laxman/Desktop/3D_Trial_Model/top …
[0] info: [time] Reading the mesh took 0.004421 s.
[0] info: Reading corresponding part of mesh data from binary file /home/laxman/Desktop/3D_Trial_Model/X0 …
[2] warning: Could not open file ‘/home/laxman/Desktop/3D_Trial_Model/mesh_partitioned_node_properties_cfg3.bin’.
You can ignore this warning if the mesh does not contain node-wise property data.
[2] info: [time] Reading the mesh took 0.008598 s.
[2] info: Reading corresponding part of mesh data from binary file /home/laxman/Desktop/3D_Trial_Model/bottom …
[2] info: [time] Reading the mesh took 0.002107 s.
[2] info: Reading corresponding part of mesh data from binary file /home/laxman/Desktop/3D_Trial_Model/top …
[2] info: [time] Reading the mesh took 0.004363 s.
[2] info: Reading corresponding part of mesh data from binary file /home/laxman/Desktop/3D_Trial_Model/X0 …
[1] warning: Could not open file ‘/home/laxman/Desktop/3D_Trial_Model/mesh_partitioned_node_properties_cfg3.bin’.
You can ignore this warning if the mesh does not contain node-wise property data.
[1] info: [time] Reading the mesh took 0.008915 s.
[1] info: Reading corresponding part of mesh data from binary file /home/laxman/Desktop/3D_Trial_Model/bottom …
[1] info: [time] Reading the mesh took 0.001971 s.
[1] info: Reading corresponding part of mesh data from binary file /home/laxman/Desktop/3D_Trial_Model/top …
[1] info: [time] Reading the mesh took 0.004542 s.
[1] info: Reading corresponding part of mesh data from binary file /home/laxman/Desktop/3D_Trial_Model/X0 …
[2] info: [time] Reading the mesh took 0.002246 s.
[1] info: [time] Reading the mesh took 0.002230 s.
[0] info: [time] Reading the mesh took 0.002268 s.
[2] info: Reading corresponding part of mesh data from binary file /home/laxman/Desktop/3D_Trial_Model/X1 …
[1] info: Reading corresponding part of mesh data from binary file /home/laxman/Desktop/3D_Trial_Model/X1 …
[0] info: Reading corresponding part of mesh data from binary file /home/laxman/Desktop/3D_Trial_Model/X1 …
[0] info: [time] Reading the mesh took 0.002067 s.
[2] info: [time] Reading the mesh took 0.002204 s.
[1] info: [time] Reading the mesh took 0.002219 s.
[1] info: Reading corresponding part of mesh data from binary file /home/laxman/Desktop/3D_Trial_Model/Z0 …
[0] info: Reading corresponding part of mesh data from binary file /home/laxman/Desktop/3D_Trial_Model/Z0 …
[2] info: Reading corresponding part of mesh data from binary file /home/laxman/Desktop/3D_Trial_Model/Z0 …
[0] info: [time] Reading the mesh took 0.001971 s.
[1] info: [time] Reading the mesh took 0.002064 s.
[2] info: [time] Reading the mesh took 0.002108 s.
[0] info: Reading corresponding part of mesh data from binary file /home/laxman/Desktop/3D_Trial_Model/Z1 …
[2] info: Reading corresponding part of mesh data from binary file /home/laxman/Desktop/3D_Trial_Model/Z1 …
[1] info: Reading corresponding part of mesh data from binary file /home/laxman/Desktop/3D_Trial_Model/Z1 …
[2] info: [time] Reading the mesh took 0.002065 s.
[1] info: [time] Reading the mesh took 0.002122 s.
[0] info: [time] Reading the mesh took 0.002364 s.
[2] info: GroupBasedParameter: YoungModulus
[1] info: GroupBasedParameter: YoungModulus
[1] critical: /home/laxman/ogs/ParameterLib/GroupBasedParameter.cpp:76 createGroupBasedParameter()
[2] critical: /home/laxman/ogs/ParameterLib/GroupBasedParameter.cpp:76 createGroupBasedParameter()
[0] info: GroupBasedParameter: YoungModulus
[0] info: GroupBasedParameter: PoissonRatio
[0] info: GroupBasedParameter: Cohesion
[0] info: GroupBasedParameter: FrictionAngle
[0] info: GroupBasedParameter: DilatancyAngle
[0] info: GroupBasedParameter: TransitionAngle
[0] info: GroupBasedParameter: TensionCutOffParameter
[0] info: ConstantParameter: displacement0
[0] info: ConstantParameter: dirichlet0
[0] info: FunctionParameter: p0
[0] info: ConstantParameter: Rainfall
[0] info: No source terms for process variable ‘displacement’ found.
[0] info: No source terms for process variable ‘pressure’ found.
[0] info: ### MFRONT ########################################################
[0] info: Behaviour: MohrCoulombAbboSloan'. [0] info: Hypothesis: Tridimensional’.
[0] info: Source: MohrCoulombAbboSloan.mfront'. [0] info: TFEL version: 3.4.2-dev’.
[0] info: Behaviour type: STANDARDSTRAINBASEDBEHAVIOUR'. [0] info: Kinematic: SMALLSTRAINKINEMATIC’.
[0] info: Symmetry: ISOTROPIC'. [0] info: #Mat. props.: 7 (array size 7). [0] info: --> type SCALAR’ with name YoungModulus', size 1, offset 0. [0] info: --> type SCALAR’ with name PoissonRatio', size 1, offset 1. [0] info: --> type SCALAR’ with name Cohesion', size 1, offset 2. [0] info: --> type SCALAR’ with name FrictionAngle', size 1, offset 3. [0] info: --> type SCALAR’ with name DilatancyAngle', size 1, offset 4. [0] info: --> type SCALAR’ with name TransitionAngle', size 1, offset 5. [0] info: --> type SCALAR’ with name TensionCutOffParameter', size 1, offset 6. [0] info: #Gradients: 1 (array size 6). [0] info: --> type STENSOR’ with name Strain', size 6, offset 0. [0] info: #Thdyn. forces: 1 (array size 6). [0] info: --> type STENSOR’ with name Stress', size 6, offset 0. [0] info: #Int. StVars.: 2 (array size 7). [0] info: --> type STENSOR’ with name ElasticStrain', size 6, offset 0. [0] info: --> type SCALAR’ with name EquivalentPlasticStrain', size 1, offset 6. [0] info: #Ext. StVars.: 1 (array size 1). [0] info: --> type SCALAR’ with name Temperature', size 1, offset 0. [0] info: #Real-valued parameters: 8. [0] info: --> with name RelativeValueForTheEquivalentStressLowerBoundDefinition’.
[0] info: → with name theta'. [0] info: --> with name epsilon’.
[0] info: → with name pi'. [0] info: --> with name local_zero_tolerance’.
[0] info: → with name minimal_time_step_scaling_factor'. [0] info: --> with name maximal_time_step_scaling_factor’.
[0] info: → with name numerical_jacobian_epsilon'. [0] info: #Integer parameters: 0. [0] info: #Unsigned parameters: 1. [0] info: --> with name iterMax’.
[0] info: Using OGS parameter YoungModulus' for material property YoungModulus’.
[0] info: Using OGS parameter PoissonRatio' for material property PoissonRatio’.
[0] info: Using OGS parameter Cohesion' for material property Cohesion’.
[0] info: Using OGS parameter FrictionAngle' for material property FrictionAngle’.
[0] info: Using OGS parameter DilatancyAngle' for material property DilatancyAngle’.
[0] info: Using OGS parameter TransitionAngle' for material property TransitionAngle’.
[0] info: Using OGS parameter TensionCutOffParameter' for material property TensionCutOffParameter’.
[0] info: ### MFRONT END ####################################################

@zhangning737 @wenqing.wang : can either of you can give advice here?

Sounds related to MFRONT. Please first try without the MFRONT model but with GroupBasedParameter.

If that works, we can handle the issue with MFront.

Dear Wenqing Wang,

Thank you very much for your reply. This is not only related to MFRONT, GroupBasedParameter still have the same error with constitutive equation native to OGS.
Now the output message looks like this: Error Log.docx (5.1 KB)
Trial input files: 3D_Trial_Model.zip (145.2 KB)

I have explained parallel computing anomaly especially with MFRONT in my previous post: Use of parallel computing (PETSc) in plastic model (MFront)

@Thomas_Nagel @wenqing.wang Ahhhhhhh, I also find the same issue in recent update(maybe exist long before, not sure), GroupBasedParameter causes memory errors and then some threads are frozen. Still didn’t locate the problem in the code~Any advice to locate the bug code, parallel program seems hardly debug~

Discussion of the bug is in issue GroupBasedParameter error when using parallel (#3180) · Issues · ogs / ogs · GitLab and a very primarily fix in MR !3833.

Thanks @thomas.fischer , it works well now