Hello everyone,
I am running a simulation with OpenGeoSys-6 (built from source, latest master) using a custom UCG (underground coal gasification) process. I have partitioned my mesh into 4 subdomains and run with MPI (mpiexec -n 4). In my PRJ file, I defined a linear solver with PETSc options to use LU with MUMPS, as follows:
<nonlinear_solvers>
<nonlinear_solver>
<name>basic_newton</name>
<type>Newton</type>
<max_iter>10</max_iter>
<recompute_jacobian>1</recompute_jacobian>
<damping>0.7</damping>
<linear_solver>general_linear_solver</linear_solver>
</nonlinear_solver>
</nonlinear_solvers>
<linear_solvers>
<linear_solver>
<name>general_linear_solver</name>
<petsc>
<prefix>ucg_</prefix>
<parameters>
-ucg_mat_type aij
-ucg_pc_type lu
-ucg_pc_factor_mat_solver_type mumps
-ucg_ksp_rtol 1e-12
-ucg_ksp_atol 1e-12
</parameters>
</petsc>
</linear_solver>
</linear_solvers>
I start the simulation with:
mpiexec -n 4 ~/projects/ogs-build/bin/ogs PJ_D.prj -m partitioned_final -o mpi_test_n4_debug
However, the log shows that the linear solver used is gmres with bjacobi preconditioner, and I also get warnings about unused options. Here are some relevant lines from the log:
Linear solver gmres with (null) preconditioner using PRECONDITIONED
...
Linear solver gmres with bjacobi preconditioner using PRECONDITIONED
...
WARNING! There are options you set that were not used!
Option left: name:-ucg_ksp_rtol value: 1e-12 source: code
Option left: name:-ucg_pc_type value: lu source: code
...
It appears that the PETSc options with the ucg_ prefix are completely ignored, and the solver falls back to default settings.
What I have tried:
-
I removed the
<prefix>ucg_</prefix>tag and changed the parameters to global PETSc options (e.g.,-ksp_type preonly,-pc_type lu, etc.). After that, the linear solver did use LU with MUMPS (as confirmed by “Linear solver preonly with lu preconditioner using NONE”), but the nonlinear solver then diverged (Newton iterations blew up, time step kept reducing until failure). That is a separate issue, and I am still debugging. -
I also tried adding
-ksp_initial_guess_nonzero falseto the global options, but the nonlinear divergence persisted. -
Crucially, I tested the exact same PRJ file (with the
ucg_prefix in either original linear solver parameters or global options) in single-core (non-MPI) mode by simply runningogs PJ_D.prj -o(withoutmpiexecand without using partitioned mesh). In that case, the linear solver options were correctly recognized, and the simulation proceeded successfully without any PETSc errors. This suggests that the issue is specific to MPI + partitioned mesh runs.
My main question: Why are the options with the ucg_ prefix not recognized in MPI runs, while they work fine in serial? Is there a known problem with PETSc option handling in parallel OGS? Do I need to set a different prefix (e.g., ls_ or something else) when using MPI? Could it be that OGS creates the KSP object with a different prefix when running in parallel, or that the prefix is overridden by the partitioning process?
I would appreciate any advice on how to correctly specify PETSc solver options in the PRJ file for MPI runs so that they are actually applied. Additionally, if you have suggestions for handling the nonlinear divergence (after the linear solver is fixed), that would be a bonus, but my primary concern is the proper configuration of the linear solver in parallel.
System info:
-
OGS version: NO_VERSION (built from source, likely 6.x)
-
PETSc version: 3.24.0 (configured with MUMPS, METIS, ParMETIS)
-
OS: Ubuntu 22.04 (WSL2)
-
Compiler: GCC 11
Thank you in advance for your help!
Best regards,
Wu
