Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ULMAT/UGLMAT not working for indefinite matrices with latest oneAPI 2025 MKL #13784

Closed
marcosvanella opened this issue Nov 20, 2024 · 5 comments
Assignees

Comments

@marcosvanella
Copy link
Contributor

In spark load the latest compiler, mpi and mkl modules, compile the impi_intel_linux_db target and run this case:

&HEAD CHID='test_indefinite', TITLE='Test ULMAT/UGLMAT on indefinite Poisson problem.' /
# Mesh:
&MESH IJK=9,8,8, XB=0.0,9.0,-4.0,4.0,0.0,8.0 /
# Simulation Paremeters:
&TIME T_END=.02, DT=0.01/
&MISC SIMULATION_MODE='DNS', STRATIFICATION=.FALSE., NOISE_VELOCITY=0.5 /
&RADI RADIATION=.FALSE. /
&PRES SOLVER='ULMAT', CHECK_POISSON=T  /

# OBST just to trigger PARDISO solve with ULMAT:
&OBST XB=8.,9.,-4.,4.,0.,8. /
# Boundaries:
&VENT MB='XMIN', SURF_ID='MIRROR' /
&VENT MB='XMAX', SURF_ID='MIRROR' /
&VENT MB='YMIN', SURF_ID='MIRROR' /
&VENT MB='YMAX', SURF_ID='MIRROR' /
&VENT MB='ZMIN', SURF_ID='MIRROR' /
&VENT MB='ZMAX', SURF_ID='MIRROR' / 
# Output:
&SLCF PBY=0., QUANTITY='VELOCITY',      VECTOR=.TRUE.        /
&SLCF PBY=0., QUANTITY='H' /
&TAIL /

The Poisson matrix for this problem is symmetric indefinite. Both with ULMAT and UGLMAT we are getting H=0 as solution regardless of what the Poisson RHS is.
Screenshot 2024-11-20 at 3 32 03 PM

Now unload mkl/latest and load mkl/2024.2, recompile the impi_intel_linux_db target and run the case. The solution for H is correct.
Screenshot 2024-11-20 at 3 36 45 PM

Interfaces to PARDISO or SPARSE_CLUSTER_SOLVER have not changed, and VERBOSE metrics for matrix factorization are identical.
I opened a thread in the Intel MKL forum asking about any parameter changes that might be related to this (https://community.intel.com/t5/Intel-oneAPI-Math-Kernel-Library/Change-in-behavior-of-Pardiso-and-Sparse-Cluster-Solver-for/m-p/1644570#M36659).

Next I'll setup a standalone test of the solver.

@marcosvanella marcosvanella self-assigned this Nov 20, 2024
@mcgratta
Copy link
Contributor

I ran the test case with the latest ifx (2025.0.1) and MKL (2025.0.1) on Windows, and the results are the same. It still does not work.

@marcosvanella
Copy link
Contributor Author

Ok, turns out the issue is related to small matrices. If you switch the size of the grid to 18,16,16 is should run fine.

@marcosvanella
Copy link
Contributor Author

From the response from Fengrui in the Intel forum, seems a change in default parameter took place when moving from mkl 2024.2 to 2025. I changed the code accordingly in PR #13810. I also changed the startup file of firebot in spark to load mkl/latest. We'll see tonight if this issue is sorted.

@marcosvanella
Copy link
Contributor Author

closing this, the verification tests are working correctly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants