Point ids for refined grids and addLgrsUpdateLeafGridView refactorization #802

aritorto · 2024-11-21T09:54:41Z

This PR introduces a structure to manage the point global IDs of all refined child cells per parent. This structure ensures the correct assignment of global point IDs in refined grids, gathering already-assigned IDs for interior cells and distributing them for overlap cells.

Current approach avoids duplicated point ids in the following setting:
Let's say a block of cells to be refined is distributed in P_{i_0}, ..., P_{i_n}, with n+1 < grid.comm().size().
If all cells to be refined are not "seen"(neither interior nor overlap) for every process P with P != P_{i_j}, j = 0, ..., n.

When this fails, duplicated ids appear on edges or faces "shared" with unseen cells.

Cells that only share corners (not faces) with interior cells of a process are not included in its overlap layer, therefore, some point may have multiple ids.
See discussion in #726.

Keeping this potential duplication of point ids in mind, this PR allows distributing Local Grid Refinements (LGRs) across different processes.

The last commits are dedicated to improve readability splitting code in different methods, shortening addLgrsUpdateLeafGridView's definition.

Improvement/replacement for #801

Not relevant for the Reference Manual

…VE_MPI

aritorto · 2024-11-21T12:38:18Z

jenkins build this serial please

akva2 · 2024-11-21T13:13:50Z

some new warnings, please address

…th fully interior parent

aritorto · 2024-11-21T14:53:17Z

jenkins build this serial please

aritorto · 2024-11-22T10:42:27Z

jenkins build this serial please

blattms

Thanks a lot.
I have a few suggestions where you should probably change the code.

blattms · 2024-11-25T09:58:42Z

CMakeLists_files.cmake

@@ -202,6 +202,7 @@ list (APPEND PUBLIC_HEADER_FILES
  opm/grid/LookUpData.hh
  opm/grid/cpgrid/OrientedEntityTable.hpp
  opm/grid/cpgrid/ParentToChildrenCellGlobalIdHandle.hpp
+	opm/grid/cpgrid/ParentToChildCellToPointGlobalIdHandle.hpp


Please check whitespace here. Mabye you used a tab here? I would use copy and paste to get the correct whitespace.

blattms · 2024-11-25T10:00:19Z

opm/grid/CpGrid.hpp

@@ -1089,6 +1072,34 @@ namespace Dune
                                                   int elemLgr)  const;
        /// --------------- Auxiliary methods to support Adaptivity (end) ---------------

+              // @brief Check if there non neighboring connections on blocks of cells selected for refinement.


Please fix indentation. and use three slashes ///

blattms · 2024-11-25T10:43:37Z

opm/grid/CpGrid.hpp

+              // @brief Check if there non neighboring connections on blocks of cells selected for refinement.
+        bool nonNNCs( const std::vector<std::array<int,3>>& startIJK_vec, const std::vector<std::array<int,3>>& endIJK_vec) const;
+
+        void markElemAssignLevelDetectActiveLgrs(const std::vector<std::array<int,3>>& startIJK_vec,


Please document using doxygen comment. There should be a short description at least.

blattms · 2024-11-25T10:47:11Z

opm/grid/cpgrid/CpGrid.cpp

@@ -1212,6 +1213,344 @@ Dune::cpgrid::Intersection CpGrid::getParentIntersectionFromLgrBoundaryFace(cons
    OPM_THROW(std::invalid_argument, "Face is on the boundary of the grid");
 }

+bool CpGrid::nonNNCs( const std::vector<std::array<int,3>>& startIJK_vec, const std::vector<std::array<int,3>>& endIJK_vec) const


Please change the name to be more explicit. E.g. noNNCsMarkedCellsLGR.

blattms · 2024-11-25T10:49:35Z

opm/grid/cpgrid/CpGrid.cpp

+        getIJK(element.index(), ijk);
+        for (std::size_t level = 0; level < startIJK_vec.size(); ++level) {
+            bool belongsToLevel = true;
+            for (int c = 0; c < 3; ++c) {
+                belongsToLevel = belongsToLevel && ( (ijk[c] >= startIJK_vec[level][c]) && (ijk[c] < endIJK_vec[level][c]) );
+                if (!belongsToLevel)
+                    break;
+            }


Do we mark the cells of an LGR? If we do we can probably simplify this.

Code is slightly simplified in a coming-soon-commit. In nonNNCsSelectedCellsLGR(...) we only check if the selected cells for refinement (via blocks with startIJK endIJK values) have NNCs. If at least one cell of one block has NNCs, we return false.

We mark elements for refinement in CpGrid::markElemeAssignLevelDetectActiveLgrs(...)

blattms · 2024-11-25T15:44:07Z

tests/cpgrid/addLgrsOnDistributedGrid_test.cpp

+        // LGR1 dim 6x2x2 -> 7x3x3 = 63 points
+        std::vector<int> local_point_ids;
+        local_point_ids.reserve(63); // expected_point_ids in LGR1
+        for (const auto& element : elements(grid.levelGridView(1))) {


What is the reason why we do not iterate over vertices directly?

Good point!

blattms · 2024-11-25T15:45:45Z

tests/cpgrid/addLgrsOnDistributedGrid_test.cpp

+        }
+        auto [all_point_ids, displPoint ] = Opm::allGatherv(local_point_ids, grid.comm());
+        const std::set<int> all_point_ids_set(all_point_ids.begin(), all_point_ids.end());
+        // Difference with preveious test case:


blattms · 2024-11-25T15:49:17Z

tests/cpgrid/addLgrsOnDistributedGrid_test.cpp

+        // - P1 does NOT see cells 2 and 6, but sees the rest of them.
+        // - P2 does NOT see cell 13, but sees all the others.
+        // - P3 does NOT see cell 1, 5 and 17, but sees all the others.
+        // Remark: the LGR is distributed only in 2 processes which are the only two processes seeing these cells.


I don't understand this sentence. Are there more than 1 LGR (which is said to be distributed among all processes above)

I will try to improve the comment. There is only one LGR here. The block of parent cells has dimension 2x2x2, the parent cell indices are {1,2,5,6,13,14,17,18}. Each parent cell gets refined into 2x2x2 children, therefore the LGR dimension is 4x4x4.

That comment is probably a copy-paste misunderstanding from the previous case. Sorry for that!

blattms · 2024-11-25T15:50:09Z

tests/cpgrid/addLgrsOnDistributedGrid_test.cpp

+        auto [all_point_ids, displPoint ] = Opm::allGatherv(local_point_ids, grid.comm());
+        const std::set<int> all_point_ids_set(all_point_ids.begin(), all_point_ids.end());
+        // Difference with preveious test case:
+        // Global cell ids of cells to be refined = {1,2,5,6,13,14,17,18}


Please indicate that this a bug originating from the way that we add overlap cells.

blattms · 2024-11-25T15:50:31Z

opm/grid/cpgrid/ParentToChildCellToPointGlobalIdHandle.hpp

+    // Not every cell has children. When they have children, the amount might vary.
+    bool fixedSize(std::size_t, std::size_t)
+    {
+        return false;
+    }
+    // Only communicate values attached to cells.
+    bool contains(std::size_t, std::size_t codim)
+    {
+        return codim == 0;
+    }
+    // Communicate variable size: 1 (rank) + (8* amount of child cells) from an interior parent cell from level zero grid.


These comments seem more like comments about the implementation. Maybe move them inside the methods?

… void

…R1 with NNCs, LGR2 without

…constructor

aritorto · 2024-11-26T13:53:12Z

@blattms all commnets have been addressed, many thanks for your review!

aritorto · 2024-11-26T13:53:26Z

jenkins build this serial please

aritorto added 9 commits November 19, 2024 17:02

Unique point global ids for refined level grids

5b418a6

Test point ids. Room for improvement: duplicated cases in edges.

b2bde77

Delete unused addLgrUpdateLeafView() [only one LGR]

6540b2c

Replace code calling compatibleSubdivisions

ee80221

Replace code calling nonNNCS

bbdb1fb

Replace code calling markElemAssignLevelDetectActiveLgrs

d415927

Replace code calling predictCellAndPointGlobalIds

fc98071

Replace code calling collectCellIdsAndCandidatePointIds

ab94935

Replace code calling selectWinnerPointIds and remove unnecesary if HA…

e58bb18

…VE_MPI

aritorto force-pushed the splitCode branch from 0d31525 to e58bb18 Compare November 21, 2024 12:32

aritorto changed the title ~~[refactor] Improve readability by splitting code on addLgrsUpdateLeafGridView~~ Point global ids for refined level grids and addLgrsUpdateLeafGridView refactorization Nov 21, 2024

aritorto mentioned this pull request Nov 21, 2024

Unique point global ids for refined level grids #801

Closed

aritorto requested a review from blattms November 21, 2024 12:47

aritorto changed the title ~~Point global ids for refined level grids and addLgrsUpdateLeafGridView refactorization~~ Point ids for refined grids and addLgrsUpdateLeafGridView refactorization Nov 21, 2024

aritorto added 6 commits November 21, 2024 14:25

Add documentation of ParallelIndexSet, set false for refined cells wi…

8954840

…th fully interior parent

Silence warnings

baedc80

Replace code calling populateCellIndexSetRefinedGrid

659bf26

Replace code calling populateCellIndexSetLeafGridView

42b615e

Replace code calling populateLEafGlobalIdSet

5967a8a

Move new methods into private class section

46dd8fa

Silence warning

c23611c

aritorto mentioned this pull request Nov 25, 2024

Global refinement partially supported on distributed CpGrid #803

Open

blattms requested changes Nov 25, 2024

View reviewed changes

aritorto added 3 commits November 26, 2024 10:28

Documentation for new methods, small modification return-pair<int> ->…

130f7b1

… void

Correct check nonNNCs in LGRs and compatibleSubdivisions, add case LG…

ed2f90a

…R1 with NNCs, LGR2 without

Pass comm() as an argument in PArentToChildCellToPointGlobalIdHandle …

c4cbdb9

…constructor

Fix/improve documentation

39930d7

aritorto mentioned this pull request Nov 26, 2024

Overlap layer size 1 affects uniqueness of point global ids for CpGrid with LGRs #804

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Point ids for refined grids and addLgrsUpdateLeafGridView refactorization #802

Point ids for refined grids and addLgrsUpdateLeafGridView refactorization #802

aritorto commented Nov 21, 2024 •

edited

Loading

aritorto commented Nov 21, 2024

akva2 commented Nov 21, 2024

aritorto commented Nov 21, 2024

aritorto commented Nov 22, 2024

blattms left a comment

blattms Nov 25, 2024

blattms Nov 25, 2024

blattms Nov 25, 2024

blattms Nov 25, 2024

blattms Nov 25, 2024

aritorto Nov 26, 2024

blattms Nov 25, 2024

aritorto Nov 26, 2024

blattms Nov 25, 2024

blattms Nov 25, 2024

aritorto Nov 26, 2024 •

edited

Loading

blattms Nov 25, 2024

blattms Nov 25, 2024

aritorto commented Nov 26, 2024

aritorto commented Nov 26, 2024

Point ids for refined grids and addLgrsUpdateLeafGridView refactorization #802

Are you sure you want to change the base?

Point ids for refined grids and addLgrsUpdateLeafGridView refactorization #802

Conversation

aritorto commented Nov 21, 2024 • edited Loading

aritorto commented Nov 21, 2024

akva2 commented Nov 21, 2024

aritorto commented Nov 21, 2024

aritorto commented Nov 22, 2024

blattms left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aritorto Nov 26, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aritorto commented Nov 26, 2024

aritorto commented Nov 26, 2024

aritorto commented Nov 21, 2024 •

edited

Loading

aritorto Nov 26, 2024 •

edited

Loading