INTEL MKL: undefined symbol

dominik-kern · January 24, 2022, 12:56pm

I installed the latest Version of INTEL MKL and compiled the latest version of OGS with OGS_USE_MKL. On execution I obtain the error

/home/dominik/OGS/release/bin/ogs: symbol lookup error: /opt/intel/oneapi/mkl/2022.0.2/lib/intel64/libmkl_intel_lp64.so: undefined symbol: mkl_blas_dgemm

Are there other users with the same problem?

Rui_Feng · January 28, 2022, 3:13pm

Dear @dominik-kern

I am learning how to compile OGS under Ubuntu 20.04 LTS (I use WSL2), and I indeed need the MKL library to enable PardisoLU liner solver. Would you mind sharing your steps or instructions about how to install Intel MKL library and how to compile OGS under Ubuntu 20.04 LTS with OGS_USE_MKL=ON?

Btw, I would like to inquire whether MKL library is free or open-source to download and install?

Thanks for your time!
Rui

joergbuchwald · January 28, 2022, 3:28pm

Hi @dominik-kern,
I guess this might be somehow related to PardisoLU fails with unspecific message - #13 by joergbuchwald
Does that also happen if you delete the MKL paths for all BLAS variables when configuring?
Did you set the path variables via setvars source /opt/intel/oneapi/setvars.sh intel64?

joergbuchwald · January 28, 2022, 3:45pm

Hi @Rui_Feng,
MKL does not cost anything for private/academic use but it is not open source.

You can download it here: Get Started with Intel® oneAPI Math Kernel Library
The installer is just a bash script with GUI so you need to make it executable befro running.
The OneAPI thing is much bigger than what you actually need. So, you can select before install what you want (which is basically the IntelMKL) and deselect the rest.
After installing you need to set the environment variables as it is usually installed under /opt.
This can be done by something like source /opt/intel/oneapi/setvars.sh intel64. You can put it in bashrc/zshrc such that you don’t need to do that every time.
To configure OGS with MKL, you can do something like cmake -DMKL_DIR="/opt/intel/oneapi/mkl/2021.4.0" -DOGS_USE_MKL="ON" -DCMAKE_BUILD_TYPE="Release" -G Ninja .. (replace MKL_DIR by the actual MKL_DIR and ... by the OGS source dir.
If you did not do the source command before, you can compile OGS directly, otherwise you need to deselect MKL for BLAS in order to not run into problems as state above. This can be done by ccmake .., then you can toggle into advanced mode by typing t. There you can delete the strings for all BLAS related variables. After that you can re-run configure by typing c once or twice until it is possible to type g for generating. After compilation Pardiso should work as Linear solver.

dominik-kern · January 28, 2022, 3:50pm

It seems not really open-source in terms of license, but it is freely available.
I downloaded from

and then followed the instrutions

to install Intel® oneAPI Base Toolkit.

Rui_Feng · January 29, 2022, 3:48pm

Dear @joergbuchwald

Thanks for your instructions very much! In step 1-4, I think I follow your instruction very exactly and correctly, and it works well in step 1-4! But in step 5, one problem occurs, which is

when I executed this command cmake -DMKL_DIR="/opt/intel/oneapi/mkl/2022.0.2" -DOGS_USE_MKL="ON" -DCMAKE_BUILD_TYPE="Release" -G Ninja /home/ubuntu_2004tls/OGS6_linux/ogs, an error occured and the configuring was incomplete:

CMake Error at scripts/cmake/CMakeSetup.cmake:13 (CPMAddPackage):
  Unknown CMake command "CPMAddPackage".
Call Stack (most recent call first):
  CMakeLists.txt:13 (include)

Do you have some ideas why did this error occur? How can I solve it?

Looking forward to hearing from you!

Best regards,
Rui

Rui_Feng · January 30, 2022, 8:56am

Dear @joergbuchwald

I tried another time to compile OGS6 using this ccmake command in ogs source code directory: ccmake -S . --preset=release. I also deleted all strings with BLAS related variables in toggle t. It showed configuration done, but when I found /build/release/bin directory, it only contained OpenGeoSysCND.xsd OpenGeoSysGLI.xsd OpenGeoSysNum.xsd OpenGeoSysProject.xsd OpenGeoSysSTN.xsd. This output really made me confusing because I think it should be like in this video at 3:13, "OpenGeoSys Workflow Tutorial: Solving - YouTube. Would you mind giving me some advice? Thank you very much!

Btw, one good news is that I can switch OGS_USE_MKL to ON during ccmake in this new try.

OS: Ubuntu 20.04LTS under WSL2

joergbuchwald · January 30, 2022, 12:58pm

If configure with cmake/ccmake is done, you need to build ogs as shown in the videoor on this webpage: Build. So with ‘make’ or ‘ninja’ depending on how configured it. I think the preset uses ninja, the command I suggested above as well.

Rui_Feng · January 30, 2022, 1:41pm

Dear @joergbuchwald

Thanks for your prompt reply! I built ogs using ninja -j 2, but it still didn’t work.

I used this command to configure ogs: ccmake -S . --preset=release
and this command to build ogs: ninja -j 2. The error shown below:

joergbuchwald · January 30, 2022, 2:57pm

which compiler version do you use? I think you need at least gcc-10 for this option. It should be part of ubuntu-20.04 (sudo apt install gcc-10 g++-10)

Rui_Feng · January 30, 2022, 3:10pm

(base) ubuntu_2004tls@DESKTOP-GDOK97I:~$ gcc --version
gcc (Ubuntu 9.3.0-17ubuntu1~20.04) 9.3.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

It is gcc 9.3.0.

Actually, I followed this guide to Set Up Prerequisites. In this guide, Set Up Prerequisites told me that “I need to have at least gcc 9.1”

joergbuchwald · January 30, 2022, 5:42pm

Sure, you can also use an older version, but then you can’t use tigerlake/native for -march.
This refers to optimizations for your micro architecture. It is not a general OGS setting.

Rui_Feng · January 31, 2022, 9:39am

Dear @joergbuchwald

Thanks for your valuable reply very much! I updated gcc version to 10.3, but it still had errors. So, I summarize my questions to a new post, CMake error during configuring OGS6.

If possible, I hope you could give me some advice in your spare time. Thanks!

Best regards,
Rui

dominik-kern · February 2, 2022, 12:56pm

I am still having the same error, can you tell me what you (or automatically) entered into the following line (-t advanced mode)?

 BLAS_guide_LIBRARY               BLAS_guide_LIBRARY-NOTFOUND

Or even better all BLAS variables, I guess they should be set to the files inside /opt/intel/oneapi/....

FZill · February 2, 2022, 1:44pm

Just to add to this. Until I fetched some recent changes today PardisoLU was working fine for me. What worked for me until now was to delete the paths in the BLAS variables in ccmake and rebuilding it.
If I don’t do this deletion on configuration I also get a undefined symbol error.
Unfortunately now my solver just fails where before the fetch it was working fine…

dominik-kern · February 2, 2022, 1:51pm

Thank you, so I will give up for today. If somebody succeeds to compile the recent version of OGS with MKL, please let me know about the lines

BLAS_guide_LIBRARY               BLAS_guide_LIBRARY-NOTFOUND
BLAS_mkl_intel_LIBRARY           BLAS_mkl_intel_LIBRARY-NOTFOUND

The remaining BLAS variables I figured out so far, at least believe so.

FZill · February 3, 2022, 9:58am

Okay so I reset my branch to the state of 2 days ago and as of the commit 6fd4aa9daa7054c779c1fde61aa8e8b3fdb8124c PardisoLU is running fine with the configuration process described above. I should mention I didn’t install MKL via the link @joergbuchwald provided but just via sudo apt-get install intel-mkl
My variables with this working state look like this:

 BLAS_iomp5_LIBRARY               /usr/lib/x86_64-linux-gnu/libiomp5.so        
 BLAS_mkl_core_LIBRARY            /usr/lib/x86_64-linux-gnu/libmkl_core.so     
 BLAS_mkl_intel_LIBRARY           BLAS_mkl_intel_LIBRARY-NOTFOUND
 BLAS_mkl_intel_lp64_LIBRARY      /usr/lib/x86_64-linux-gnu/libmkl_intel_lp64.s
 BLAS_mkl_intel_thread_LIBRARY    /usr/lib/x86_64-linux-gnu/libmkl_intel_thread
 MKL_DIR                          MKL_DIR-NOTFOUND
 MKL_INCLUDE_DIR                  /usr/include/mkl
 MKL_LIB_CORE                     /usr/lib/x86_64-linux-gnu/libmkl_core.so     
 MKL_LIB_INTEL                    /usr/lib/x86_64-linux-gnu/libmkl_intel_lp64.s
 MKL_LIB_THREAD                   /usr/lib/x86_64-linux-gnu/libmkl_gnu_thread.s

Now when going to the current master branch, rebuilding and running a sim I get:
ogs: symbol lookup error: /lib/x86_64-linux-gnu/libmkl_intel_lp64.so: undefined symbol: mkl_blas_dsyrk
But the cmake vars above haven’t changed. And after cleaning my build dir and repeating the process it seems that the vars can’t be found as my vars suggest:

 BLAS_guide_LIBRARY               BLAS_guide_LIBRARY-NOTFOUND
 BLAS_iomp5_LIBRARY
 BLAS_mkl_LIBRARY                 BLAS_mkl_LIBRARY-NOTFOUND
 BLAS_mkl_core_LIBRARY
 BLAS_mkl_em64t_LIBRARY           BLAS_mkl_em64t_LIBRARY-NOTFOUND
 BLAS_mkl_ia32_LIBRARY            BLAS_mkl_ia32_LIBRARY-NOTFOUND
 BLAS_mkl_intel_LIBRARY           BLAS_mkl_intel_LIBRARY-NOTFOUND
 BLAS_mkl_intel_lp64_LIBRARY
 BLAS_mkl_intel_thread_LIBRARY    
 BLAS_mkl_rt_LIBRARY              /usr/lib/x86_64-linux-gnu/libmkl_rt.so

Interestingly there are now more BLAS variables??
And unsurprisingly PardisoLU again doesn’t work.
Maybe someone can find out the significant difference since the commit I used previously, which caused this change in behaviour…

joergbuchwald · February 3, 2022, 1:13pm

I was also not very successfull with MKL provided by my distribution. I would recommend deleting intel-mkl and install MKL directly from intel.

joergbuchwald · February 3, 2022, 1:15pm

These are correct. My experience is that using MKL BLAS is not working in OGS.

dominik-kern · February 3, 2022, 1:16pm

Which BLAS version do you use then and how do you install it?