# PETSc solver options in PRJ file ignored – falls back to gmres with bjacobi

**URL:** <https://discourse.opengeosys.org/t/petsc-solver-options-in-prj-file-ignored-falls-back-to-gmres-with-bjacobi/2035>\
**Category:** Help & Support\
**Created:** [August 17, 2026, 11:38am UTC](https://discourse.opengeosys.org/t/petsc-solver-options-in-prj-file-ignored-falls-back-to-gmres-with-bjacobi/2035 "2026-08-17T11:38:34Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![wugao](https://discourse.opengeosys.org/letter_avatar_proxy/v4/letter/w/76d3ee/32.png) [@wugao](https://discourse.opengeosys.org/u/wugao)\
**Post date:** [August 17, 2026, 11:38am UTC](https://discourse.opengeosys.org/t/petsc-solver-options-in-prj-file-ignored-falls-back-to-gmres-with-bjacobi/2035/1 "2026-08-17T11:38:34Z")

</div>

Hello everyone,

I am running a simulation with OpenGeoSys-6 (built from source, latest master) using a custom UCG (underground coal gasification) process. I have partitioned my mesh into 4 subdomains and run with MPI (`mpiexec -n 4`). In my PRJ file, I defined a linear solver with PETSc options to use LU with MUMPS, as follows:

```auto
<nonlinear_solvers>
    <nonlinear_solver>
        <name>basic_newton</name>
        <type>Newton</type>
        <max_iter>10</max_iter>
        <recompute_jacobian>1</recompute_jacobian>
        <damping>0.7</damping>
        <linear_solver>general_linear_solver</linear_solver>
    </nonlinear_solver>
</nonlinear_solvers>

<linear_solvers>
    <linear_solver>
        <name>general_linear_solver</name>
        <petsc>
            <prefix>ucg_</prefix>
            <parameters>
                -ucg_mat_type aij
                -ucg_pc_type lu
                -ucg_pc_factor_mat_solver_type mumps
                -ucg_ksp_rtol 1e-12
                -ucg_ksp_atol 1e-12
            </parameters>
        </petsc>
    </linear_solver>
</linear_solvers>

```

I start the simulation with:

```auto
mpiexec -n 4 ~/projects/ogs-build/bin/ogs PJ_D.prj -m partitioned_final -o mpi_test_n4_debug

```

However, the log shows that the linear solver used is `gmres` **with** `bjacobi` preconditioner, and I also get warnings about unused options. Here are some relevant lines from the log:

```auto
Linear solver gmres with (null) preconditioner using PRECONDITIONED
...
Linear solver gmres with bjacobi preconditioner using PRECONDITIONED
...
WARNING! There are options you set that were not used!
Option left: name:-ucg_ksp_rtol value: 1e-12 source: code
Option left: name:-ucg_pc_type value: lu source: code
...

```

It appears that the PETSc options with the `ucg_` prefix are completely ignored, and the solver falls back to default settings.

**What I have tried:**

1. I removed the `<prefix>ucg_</prefix>` tag and changed the parameters to global PETSc options (e.g., `-ksp_type preonly`, `-pc_type lu`, etc.). After that, the linear solver did use LU with MUMPS (as confirmed by “Linear solver preonly with lu preconditioner using NONE”), but the nonlinear solver then diverged (Newton iterations blew up, time step kept reducing until failure). That is a separate issue, and I am still debugging.

2. I also tried adding `-ksp_initial_guess_nonzero false` to the global options, but the nonlinear divergence persisted.

3. **Crucially** , I tested the **exact same PRJ file** (with the `ucg_` prefix in either original linear solver parameters or global options) in **single-core (non-MPI) mode** by simply running `ogs PJ_D.prj -o` (without `mpiexec` and without using partitioned mesh). In that case, **the linear solver options were correctly recognized,** and the simulation proceeded successfully without any PETSc errors. This suggests that the issue is specific to **MPI + partitioned mesh** runs.

**My main question:** Why are the options with the `ucg_` prefix not recognized in MPI runs, while they work fine in serial? Is there a known problem with PETSc option handling in parallel OGS? Do I need to set a different prefix (e.g., `ls_` or something else) when using MPI? Could it be that OGS creates the KSP object with a different prefix when running in parallel, or that the prefix is overridden by the partitioning process?

I would appreciate any advice on how to correctly specify PETSc solver options in the PRJ file for MPI runs so that they are actually applied. Additionally, if you have suggestions for handling the nonlinear divergence (after the linear solver is fixed), that would be a bonus, but my primary concern is the proper configuration of the linear solver in parallel.

**System info:**

- OGS version: NO\_VERSION (built from source, likely 6.x)

- PETSc version: 3.24.0 (configured with MUMPS, METIS, ParMETIS)

- OS: Ubuntu 22.04 (WSL2)

- Compiler: GCC 11

Thank you in advance for your help!

Best regards,

Wu

---

<div class="post-metadata">

**Author:** ![thomas.fischer](https://discourse.opengeosys.org/user_avatar/discourse.opengeosys.org/thomas.fischer/32/416_2.png) [@thomas.fischer](https://discourse.opengeosys.org/u/thomas.fischer)\
**Post date:** [August 18, 2026, 5:06am UTC](https://discourse.opengeosys.org/t/petsc-solver-options-in-prj-file-ignored-falls-back-to-gmres-with-bjacobi/2035/2 "2026-08-18T05:06:19Z")

</div>

Hello Wugao,

in the benchmark file `Tests/Data/EllipticPETSc/cube_1e3.prj`(in the source dir of OGS) the prefix is without the underscore at the end, i.e., `ucg` instead of `ucg_`.

The string within `<parameters>` tag is given to PETSc as it is.

However, OGS adds an underscore to the end of the prefix before it is given to PETSc. In your case `ucg__` is finally passed to PETSc. I think this is the reason for the PETSc default behaviour.

Best regards,

Tom

---

<div class="post-metadata">

**Author:** ![wugao](https://discourse.opengeosys.org/letter_avatar_proxy/v4/letter/w/76d3ee/32.png) [@wugao](https://discourse.opengeosys.org/u/wugao)\
**Post date:** [August 18, 2026, 5:54am UTC](https://discourse.opengeosys.org/t/petsc-solver-options-in-prj-file-ignored-falls-back-to-gmres-with-bjacobi/2035/3 "2026-08-18T05:54:21Z")

</div>

Hi Thomas,

Thanks for your reply.

I have changed the prefix of “`ucg_`” to “`ucg`” as follows:

`<linear_solvers>`  
` <linear_solver>`  
` <name>general_linear_solver</name>`  
` <petsc>`  
` <prefix>ucg</prefix>`  
` <parameters>`  
` -ucg_mat_type aij`  
` -ucg_pc_type lu`  
` -ucg_pc_factor_mat_solver_type mumps`  
` -ucg_ksp_rtol 1e-12`  
` -ucg_ksp_atol 1e-12`  
` </parameters>`  
` </petsc>`  
` </linear_solver>`  
`</linear_solvers>`

But the log this time shows:

`Linear solver gmres with lu preconditioner using PRECONDITIONED`

I have attached the full log.

[mpi\_test\_n4\_debug.docx](https://discourse.opengeosys.org/uploads/short-url/wai7dyyVcTxf1N86ftElFC4ZyBI.docx) (67.7 KB)

Best regards,

Wu

---

<div class="post-metadata">

**Author:** ![thomas.fischer](https://discourse.opengeosys.org/user_avatar/discourse.opengeosys.org/thomas.fischer/32/416_2.png) [@thomas.fischer](https://discourse.opengeosys.org/u/thomas.fischer)\
**Post date:** [August 18, 2026, 8:31am UTC](https://discourse.opengeosys.org/t/petsc-solver-options-in-prj-file-ignored-falls-back-to-gmres-with-bjacobi/2035/4 "2026-08-18T08:31:43Z")

</div>

Hi Wu,

could you please send the full log as a pure text file?

Best regards,

Tom

---

<div class="post-metadata">

**Author:** ![wugao](https://discourse.opengeosys.org/letter_avatar_proxy/v4/letter/w/76d3ee/32.png) [@wugao](https://discourse.opengeosys.org/u/wugao)\
**Post date:** [August 18, 2026, 9:35am UTC](https://discourse.opengeosys.org/t/petsc-solver-options-in-prj-file-ignored-falls-back-to-gmres-with-bjacobi/2035/5 "2026-08-18T09:35:40Z")

</div>

Hi Tom,

It seems that a .txt file is not allowed to upload. Please see the screenshot below.

![image](https://discourse.opengeosys.org/uploads/default/original/2X/2/20a0bf52f2369c606cc1495da627e549477136be.png)

Best regards,

Wu

---

<div class="post-metadata">

**Author:** ![wugao](https://discourse.opengeosys.org/letter_avatar_proxy/v4/letter/w/76d3ee/32.png) [@wugao](https://discourse.opengeosys.org/u/wugao)\
**Post date:** [August 18, 2026, 9:38am UTC](https://discourse.opengeosys.org/t/petsc-solver-options-in-prj-file-ignored-falls-back-to-gmres-with-bjacobi/2035/6 "2026-08-18T09:38:03Z")

</div>

Hi Tom,

The full log is uploaded as a zip file. Thank you.

[mpi\_test\_n4\_debug.zip](https://discourse.opengeosys.org/uploads/short-url/9rlERNEb1DBbNoC4EXXHTbxRHOS.zip) (64.6 KB)

Best regards,

Wu

---

<div class="post-metadata">

**Author:** ![bilke](https://discourse.opengeosys.org/user_avatar/discourse.opengeosys.org/bilke/32/868_2.png) [@bilke](https://discourse.opengeosys.org/u/bilke)\
**Post date:** [August 18, 2026, 9:58am UTC](https://discourse.opengeosys.org/t/petsc-solver-options-in-prj-file-ignored-falls-back-to-gmres-with-bjacobi/2035/7 "2026-08-18T09:58:01Z")

</div>

Hi Wu,

now it is possible to attach txt files (and also doc, docx and pdf) directly. Please note that you txt file you attached as zip file is a not a txt file (maybe a docx file?). Please reupload a proper txt file again.

Thanks,  
Lars

---

<div class="post-metadata">

**Author:** ![wugao](https://discourse.opengeosys.org/letter_avatar_proxy/v4/letter/w/76d3ee/32.png) [@wugao](https://discourse.opengeosys.org/u/wugao)\
**Post date:** [August 18, 2026, 10:56am UTC](https://discourse.opengeosys.org/t/petsc-solver-options-in-prj-file-ignored-falls-back-to-gmres-with-bjacobi/2035/8 "2026-08-18T10:56:55Z")

</div>

Hi Lars and Tom,

Sorry for the mistake. I really appreciate your reply.

To show more information, I have modified the linear solver configuration as follows:

```auto
    <linear_solvers>

        <linear_solver>

            <name>general_linear_solver</name>                  

            <petsc>

                <prefix>ucg</prefix>

                <parameters>

                   -ucg_mat_type aij

                   -ucg_ksp_type gmres

                   -ucg_pc_type lu

                   -ucg_pc_factor_mat_solver_type mumps

                   -ucg_ksp_rtol 1e-8

                   -ucg_ksp_atol 1e-12

                   -ucg_ksp_monitor

                   -ucg_ksp_converged_reason

                </parameters>

            </petsc>

        </linear_solver>

    </linear_solvers>

```

The corresponding full log of 4-core computation as zip file is as follows (strangely, it’s still not allowed to upload the .txt file from my side):

[mpi\_test\_n4\_petsc.zip](https://discourse.opengeosys.org/uploads/short-url/6CrUpgrjyVA6S0CfjEMO5WD4Z1M.zip) (20.4 KB)

In addition, the partial log of single-core computation, which runs successfully using the above linear solver configuration, is provided below:

[serial\_petsc.zip](https://discourse.opengeosys.org/uploads/short-url/wCvlDiaFNak6sVAmYyDjLZcLHfQ.zip) (2.8 KB)

Look forward to hearing from you, and thank you!

Best

Wu

---

<div class="post-metadata">

**Author:** ![thomas.fischer](https://discourse.opengeosys.org/user_avatar/discourse.opengeosys.org/thomas.fischer/32/416_2.png) [@thomas.fischer](https://discourse.opengeosys.org/u/thomas.fischer)\
**Post date:** [August 18, 2026, 12:26pm UTC](https://discourse.opengeosys.org/t/petsc-solver-options-in-prj-file-ignored-falls-back-to-gmres-with-bjacobi/2035/9 "2026-08-18T12:26:31Z")

</div>

The serial PETSc (1 MPI process) run and the PETSc using 4 MPI processes have both the same linear solver settings:

> Linear solver gmres with lu preconditioner using PRECONDITIONED

But this are not the values you specified in the prj-file. OGS passes the PETSc options given in the `<parameter>` tag to PETSc library using function [PetscOptionsInsertString](https://petsc.org/release/manualpages/Sys/PetscOptionsInsertString/#petscoptionsinsertstring). In the linked document it is stated:

> _ **in\_str -** _ string that contains options separated by blanks

Your string contains line breaks. I’m not sure if a line break is also a blank. Please format your option string without line break and try it again.

---

<div class="post-metadata">

**Author:** ![joergbuchwald](https://discourse.opengeosys.org/user_avatar/discourse.opengeosys.org/joergbuchwald/32/449_2.png) [@joergbuchwald](https://discourse.opengeosys.org/u/joergbuchwald)\
**Post date:** [August 18, 2026, 2:38pm UTC](https://discourse.opengeosys.org/t/petsc-solver-options-in-prj-file-ignored-falls-back-to-gmres-with-bjacobi/2035/10 "2026-08-18T14:38:56Z")

</div>

This is correct and exactly what you wanted.  
The default solver is gmres. Linear solvers in petsc are always iterative solvers. With lu preconditioning however, the iterative solver does 1 iteration. The LU preconditioning is done with the mumps solver. This is how direct solvers work in petsc, they are just preconditioners. (This is also how you specified it btw.) You could also make the output more verbose to obtain more information.

---

<div class="post-metadata">

**Author:** ![joergbuchwald](https://discourse.opengeosys.org/user_avatar/discourse.opengeosys.org/joergbuchwald/32/449_2.png) [@joergbuchwald](https://discourse.opengeosys.org/u/joergbuchwald)\
**Post date:** [August 20, 2026, 9:25am UTC](https://discourse.opengeosys.org/t/petsc-solver-options-in-prj-file-ignored-falls-back-to-gmres-with-bjacobi/2035/11 "2026-08-20T09:25:31Z")

</div>

Additionally to my last comment and looking at the logfiles, the difference between np 4 and np 1 case seems to be a memory error. This could be related to the partitioning, or you just went out of enough memory for your problem. Keep in mind that direct solvers require more memory than just pure iterative solving methods.

---

<div class="post-metadata">

**Author:** ![wugao](https://discourse.opengeosys.org/letter_avatar_proxy/v4/letter/w/76d3ee/32.png) [@wugao](https://discourse.opengeosys.org/u/wugao)\
**Post date:** [August 20, 2026, 1:11pm UTC](https://discourse.opengeosys.org/t/petsc-solver-options-in-prj-file-ignored-falls-back-to-gmres-with-bjacobi/2035/12 "2026-08-20T13:11:48Z")

</div>

Hi Joerg,

Thanks for your reply!

Do you have any good suggestions for setting the parameters for the linear solver? I tried np=2, but still the same problem. There is also a memory error. The full log is attached.

[mpi\_test\_n2\_petsc.zip](https://discourse.opengeosys.org/uploads/short-url/jkUlKjSy3sVhMmtfCnfs6Fx9l1q.zip) (23.2 KB)

Just to provide more information, the custom UCG module is mainly based on the TH2M module, but with 11 process variables, including capillary pressure, temperature, density of h2o, density of o2, density of co, density of co2, density of h2, density of ch4, density of drycoal, density of char, and displacement.

I would be happy to share more information if it helps.

Best regards,

Wu

---

<div class="post-metadata">

**Author:** ![joergbuchwald](https://discourse.opengeosys.org/user_avatar/discourse.opengeosys.org/joergbuchwald/32/449_2.png) [@joergbuchwald](https://discourse.opengeosys.org/u/joergbuchwald)\
**Post date:** [August 20, 2026, 8:44pm UTC](https://discourse.opengeosys.org/t/petsc-solver-options-in-prj-file-ignored-falls-back-to-gmres-with-bjacobi/2035/13 "2026-08-20T20:44:16Z")

</div>

Maybe this in advance: Did you correctly partition the mesh? [Running OGS with MPI](https://www.opengeosys.org/stable/docs/userguide/features/parallel_computing_mpi/)  
Another thing one should check: In my experience, combining MPI and OpenMP in PETSc solvers is not a good idea and often leads to worse performance. So you should set the OMP\_NUM\_THREADS environment variable to 1.

If you already followed both suggestions and the problem persists, then it depends very much on the problem, and it is hard to give some general advice. So you should check how the RAM/swap is filled during the simulation. If it really exceeds RAM, you might do some scaling tests with a smaller problem sizes to see how memory usage scales.  
Many process variables mean there is more to compute per node, which means the computation-to-communication ratio increases, which is good because it means MPI might be beneficial, but more variables also mean more communication, so it is also about feasibility…  
Maybe a direct solver with OpenMP parallelization like pardiso might be a better choice. For problems that are really too big you might be better off with an iterative solver or a bigger machine. So it is a bit hard to judge from a far.

---

<div class="post-metadata">

**Author:** ![wugao](https://discourse.opengeosys.org/letter_avatar_proxy/v4/letter/w/76d3ee/32.png) [@wugao](https://discourse.opengeosys.org/u/wugao)\
**Post date:** [August 21, 2026, 7:29am UTC](https://discourse.opengeosys.org/t/petsc-solver-options-in-prj-file-ignored-falls-back-to-gmres-with-bjacobi/2035/14 "2026-08-21T07:29:18Z")

</div>

Hi Joerg,

Thank you for your suggestion to verify the partitioning process. I have re-run the mesh partitioning carefully and collected all relevant files for inspection. Below I describe the steps I followed, the input files, and the generated outputs. I also include the log files and VTU outputs for both the partitioning and the binary-to-VTU conversion steps.

1. **Input VTU files (original, unpartitioned)**

I am using four mesh files (all in VTU format) located in the working directory:

- `PJ_D_domain.vtu` – main domain mesh

- `PJ_D_inlet.vtu` – inlet boundary mesh

- `PJ_D_outlet.vtu` – outlet boundary mesh

- `PJ_D_surface_points.vtu` – surface points boundary mesh

**Number of points:** 1989; **Number of cells:** 1900

The problem size is relatively small. I have packed these original VTU files into a zip archive named `vtu_for_partition.zip`([vtu\_for\_partition.zip](https://discourse.opengeosys.org/uploads/short-url/2IdJ07aQdWiJNcrS1xWTYrmniKD.zip) (36.5 KB))

1. **Mesh partitioning procedure (for 2 partitions)**

I performed the partitioning using the OGS `partmesh` tool. The steps are as follows:

Step 1 - Prepare directories

```auto
cd /mnt/c/Users/gaowu/Desktop/PJ/PJ_D_files
rm -rf partitioned_n2
mkdir partitioned_n2

```

Step 2 - Generate METIS input for the main domain

```auto
~/projects/ogs-build/bin/partmesh \
    -i PJ_D_domain.vtu \
    -o partitioned_n2 \
    -s

```

Step 3 - Partition the main domain and the three auxiliary meshes together

```auto
~/projects/ogs-build/bin/partmesh \
    -i PJ_D_domain.vtu \
    PJ_D_inlet.vtu \
    PJ_D_outlet.vtu \
    PJ_D_surface_points.vtu \
    -n 2 \
    -o partitioned_n2 \
    -m

```

The resulting binary files are stored in the `partitioned_n2` directory. I have archived them as `partitioned_n2.zip`([partitioned\_n2.zip](https://discourse.opengeosys.org/uploads/short-url/qFzL9Qzpt6wEKkutGBcToYxUcSr.zip) (52.0 KB)). The full log of the partition process is included in `partmesh_n2_log.zip`([partmesh\_n2\_log.zip](https://discourse.opengeosys.org/uploads/short-url/tKbjAtCignzSefAECYGFs7fRAXq.zip) (1.8 KB)).

1. **Conversion of binary partition files back to VTU/PVTU (for visual verification)**

To verify that the partitioning is correct, I converted the binary files back to VTU/PVTU format using `binaryToPVTU` with 2 MPI processes.

Commands:

```auto
cd /mnt/c/Users/gaowu/Desktop/PJ/PJ_D_files/partitioned_n2
mkdir -p ../output_vtu_n2

# Convert main domain
mpirun -np 2 /home/gaowu/projects/ogs-build/bin/binaryToPVTU \
    -i PJ_D_domain -o ../output_vtu_n2/PJ_D_domain

# Convert inlet boundary
mpirun -np 2 /home/gaowu/projects/ogs-build/bin/binaryToPVTU \
    -i PJ_D_inlet -o ../output_vtu_n2/PJ_D_inlet

# Convert outlet boundary
mpirun -np 2 /home/gaowu/projects/ogs-build/bin/binaryToPVTU \
    -i PJ_D_outlet -o ../output_vtu_n2/PJ_D_outlet

# Convert surface points
mpirun -np 2 /home/gaowu/projects/ogs-build/bin/binaryToPVTU \
    -i PJ_D_surface_points -o ../output_vtu_n2/PJ_D_surface_points

```

The resulting VTU/PVTU files are zipped as `output_vtu_n2.zip`([output\_vtu\_n2.zip](https://discourse.opengeosys.org/uploads/short-url/Agrfxokl7ueWH83Gu5JzFnQFNSB.zip) (39.6 KB)), and the conversion log is provided in `binaryToPVTU_n2_log.zip`([binaryToPVTU\_n2\_log.zip](https://discourse.opengeosys.org/uploads/short-url/3PWqCpeN8O4z2en9O6zip7vXhC8.zip) (1.0 KB)).

I opened the generated VTU/PVTU files in ParaView and visually inspected the partitions – everything looks correct to me (though I might have missed something subtle). The domain and boundaries appear properly split and aligned.

1. **Additional observations**

- **Memory usage:** During the simulation runs with this small mesh (1989 points, 1900 cells), the RAM usage increased from an initial baseline of ~53% to about 60–61% – it never exceeded available memory. So I do not think memory pressure is causing the PETSc option problem.

Could you please have a look at all the steps above and confirm whether everything is correct? There must be something I don’t understand or have overlooked that is causing the error. Any further suggestions would be greatly appreciated.

Thank you for your time and help!

Best regards,

Wu

---

<div class="post-metadata">

**Author:** ![ethandavis](https://discourse.opengeosys.org/letter_avatar_proxy/v4/letter/e/8491ac/32.png) [@ethandavis](https://discourse.opengeosys.org/u/ethandavis)\
**Post date:** [November 12, 2026, 11:38pm UTC](https://discourse.opengeosys.org/t/petsc-solver-options-in-prj-file-ignored-falls-back-to-gmres-with-bjacobi/2035/15 "2026-11-12T23:38:43Z")

</div>

The mesh size (~1900 cells) makes a pure MUMPS memory blow-up less likely, so I’d check whether the custom UCG module keeps any full-domain arrays per rank; with 11 process variables that can multiply quickly under MPI. Also confirm the PETSc error is really an out-of-memory from the factor, not a MUMPS/partitioning error. A quick test is to run the same UCG setup with 2 ranks on a tiny uniform mesh; if it fails there too, it points to the module/MPI path rather than solver fill-in. For this size, serial direct solve is often the practical choice.
