Parallelization of `CliffordOperations` through `numba` and `cupy` #161

BrunoLiegiBastonLiegi · 2023-12-13T12:00:08Z

This PR implements the CliffordOperations object introduced in qiboteam/qibo#1076 to be used specifically with the NumbaBackend and CupyBackend. Namely, each operation is rewritten as a loop over the row indices that is parallelized using, prange and compiled with @njit(parallel=True, cache=True) for numba, and @jit.rawkernel() with cupy.

Companion of qiboteam/qibo#1150

src/qibojit/backends/cpu.py

alecandido · 2024-02-22T14:11:25Z

Should we silence Codecov in Qibojit?

alecandido · 2024-02-22T16:38:56Z

Should we silence Codecov in Qibojit?

I had a second thought, since tests are present in Qibojit (though in an unusual location wrt the other repos), and there is a related workflow.
https://github.com/qiboteam/qibojit/tree/main/src/qibojit/tests
https://github.com/qiboteam/qibojit/actions/workflows/rules.yml

But I get that you wrote the related tests already in Qibo.
Eventually, we will disentangle the dependencies, but this is part of the qibo-core effort (and I'd like to merge these PRs before, if possible immediately).

Before silencing Codecov, or skip tests for this PR, it could be valuable to collect the opinions of the original Qibojit authors. @scarrazza @stavros11 @andrea-pasquale

andrea-pasquale · 2024-02-22T16:42:47Z

I know that in the past there were plans to test directly on gpu through a selfhosted workflow. I believe that it should be less complicated compare to qibolab tests. If we can do so, we should be able to recover coverage, right?
If it is too complicated or if we feel like that it is a bit of an overkill I am also happy to drop coverage and eventually skip tests.

src/qibojit/backends/clifford_operations_cpu.py

src/qibojit/backends/clifford_operations_gpu.py

renatomello · 2024-02-23T05:09:43Z

src/qibojit/backends/clifford_operations_gpu.py

+def _clifford_pre_execution_reshape(state):
+    return state.ravel()
+
+
+def _clifford_post_execution_reshape(state, nqubits):
+    dim = _get_dim(nqubits)
+    return state.reshape(dim, dim)
+
+
+def identity_density_matrix(nqubits, normalize: bool = True):
+    n = 1 << nqubits
+    state = cp.eye(n, dtype="complex128")
+    cp.cuda.stream.get_current_stream().synchronize()
+    if normalize:
+        state /= 2**nqubits
+    return state.reshape((n, n))


Here I'd say that using cp.ravel and cp.reshape is more robust just in case (I know, I know...)

src/qibojit/backends/clifford_operations_gpu.py

Co-authored-by: Renato Mello <[email protected]>

for more information, see https://pre-commit.ci

BrunoLiegiBastonLiegi · 2024-02-26T13:05:19Z

@renatomello @alecandido @scarrazza is this ready to be merged? I would merge this one first and then the corresponding one in qibo.

renatomello

Is the 0% coverage because it needs the PR in qibo too?

BrunoLiegiBastonLiegi · 2024-02-26T14:56:45Z

Is the 0% coverage because it needs the PR in qibo too?

It's because the changes here are tested in qibo, as the cupy and numba clifford operations are defined here but used inside of the CliffordBackend.

renatomello · 2024-02-26T15:13:06Z

Is the 0% coverage because it needs the PR in qibo too?

It's because the changes here are tested in qibo, as the cupy and numba clifford operations are defined here but used inside of the CliffordBackend.

So then maybe they shouldn't count to coverage here (maybe # pragma: no cover everywhere or some other way to exclude entire files?)

alecandido · 2024-02-26T16:32:34Z

To be completely honest: things implemented here should be tested here (as for every other project).

Having features spread over many repos is an issue on its own. But we are starting facing that issue with qiboteam/qibo#1117 and https://github.com/qiboteam/qibo-core/, so I will not insist in this PR.

BrunoLiegiBastonLiegi · 2024-02-26T16:35:31Z

To be completely honest: things implemented here should be tested here (as for every other project).

Having features spread over many repos is an issue on its own. But we are starting facing that issue with qiboteam/qibo#1117 and https://github.com/qiboteam/qibo-core/, so I will not insist in this PR.

I agree, but this would involve copy pasting the tests that are currently implemented in qibo here. If you think it's better I can do that, it was just to avoid code duplication.

alecandido · 2024-02-26T16:47:14Z

I agree, but this would involve copy pasting the tests that are currently implemented in qibo here. If you think it's better I can do that, it was just to avoid code duplication.

I believe the best would be to move all this code to Qibo, possibly to the quantum_info module, where Clifford is defined in the first place.
But let's merge this PR, and consider this an optimization.

BrunoLiegiBastonLiegi and others added 3 commits December 11, 2023 17:02

feat: implemented numba clifford operations

ce82a33

feat: rewrote the operations as numba loops

53aecc6

Delete src/qibojit/backends/clifford.py~

e112917

renatomello self-requested a review December 13, 2023 14:42

renatomello added the enhancement New feature or request label Dec 13, 2023

renatomello assigned BrunoLiegiBastonLiegi Dec 13, 2023

BrunoLiegiBastonLiegi added 3 commits December 14, 2023 12:48

feat: implemented custom cuda kernels (to be tested)

d6f28b0

feat: implemented rowsum

e39e89e

fix: fixed H, CY and rowsum

007c348

BrunoLiegiBastonLiegi changed the title ~~Parallelization of CliffordOperations through numba~~ Parallelization of CliffordOperations through numba and cupy Dec 19, 2023

BrunoLiegiBastonLiegi added 5 commits December 22, 2023 19:13

feat: made numba operations as standalone functions

0f7be24

fix: monkey-patching qibo operations

d589890

feat: updated the gpu operations as well

72e951d

fix: removed unnecessary files

032c4b0

fix: minor import fix

f82dcd4

BrunoLiegiBastonLiegi mentioned this pull request Dec 27, 2023

Customizing Clifford operations for qibojit engines qiboteam/qibo#1150

Merged

4 tasks

renatomello mentioned this pull request Jan 4, 2024

Improvements on top of the Clifford Simulator qiboteam/qibo#1128

Open

15 tasks

BrunoLiegiBastonLiegi added 8 commits January 4, 2024 13:00

fix: fix cupy implementation of gates

57399ab

fix: fixed some typos

6c0e59e

fix: fixed parallelization of _rowsum in cupy

0bcc49b

fix: small fix to _rowsum

757904e

feat: moved clifford operations monkey-patching to backends

aaf19ff

fix: fixed patching problem

bb7cc10

feat: removed include_scratch + updated random_outcome

ad5f0b8

fix: fixed some typos

d229048

renatomello mentioned this pull request Jan 10, 2024

Improving integration of quantum_info module and qibojit qiboteam/qibo#1136

Open

renatomello changed the title ~~Parallelization of CliffordOperations through numba and cupy~~ Parallelization of CliffordOperations through numba and cupy Jan 10, 2024

BrunoLiegiBastonLiegi added 3 commits January 12, 2024 16:15

feat: implemented cupy kernels in CUDA + matrix linearization

9acd886

fix: bugfix to kernel launchers

c89a625

feat: implemented rowsum in cuda

45f9c9c

alecandido reviewed Feb 19, 2024

View reviewed changes

src/qibojit/backends/cpu.py Outdated Show resolved Hide resolved

alecandido and others added 7 commits February 19, 2024 08:39

chore: Remove unused Numba import

dc23b69

build: changed qibo dependency

7b13ea3

build: updated lock

6cc064b

fix: fix conflicts

4308b03

fix: added alternative signature to rowsum for windows

2e292d4

fix: changed rowsum signature

3e0505d

fix: removed self from cast for cupy

a6b0e26

renatomello reviewed Feb 23, 2024

View reviewed changes

src/qibojit/backends/clifford_operations_cpu.py Outdated Show resolved Hide resolved

pylint suggestions

4539ab7

renatomello reviewed Feb 23, 2024

View reviewed changes

src/qibojit/backends/clifford_operations_gpu.py Outdated Show resolved Hide resolved

renatomello reviewed Feb 23, 2024

View reviewed changes

src/qibojit/backends/clifford_operations_gpu.py Outdated Show resolved Hide resolved

renatomello reviewed Feb 23, 2024

View reviewed changes

src/qibojit/backends/clifford_operations_gpu.py Show resolved Hide resolved

BrunoLiegiBastonLiegi and others added 3 commits February 23, 2024 09:24

Update src/qibojit/backends/clifford_operations_gpu.py

7c4f22b

Co-authored-by: Renato Mello <[email protected]>

[pre-commit.ci] auto fixes from pre-commit.com hooks

a1405fe

for more information, see https://pre-commit.ci

fix: revert some pylint suggestions

0ff0e50

alecandido approved these changes Feb 26, 2024

View reviewed changes

renatomello approved these changes Feb 26, 2024

View reviewed changes

cov: omitting clifford_operations from coverage

95353c3

BrunoLiegiBastonLiegi merged commit 524aea0 into main Feb 28, 2024
21 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parallelization of `CliffordOperations` through `numba` and `cupy` #161

Parallelization of `CliffordOperations` through `numba` and `cupy` #161

BrunoLiegiBastonLiegi commented Dec 13, 2023 •

edited by alecandido

Loading

alecandido commented Feb 22, 2024

alecandido commented Feb 22, 2024

andrea-pasquale commented Feb 22, 2024 •

edited

Loading

renatomello Feb 23, 2024

BrunoLiegiBastonLiegi commented Feb 26, 2024 •

edited

Loading

renatomello left a comment •

edited

Loading

BrunoLiegiBastonLiegi commented Feb 26, 2024

renatomello commented Feb 26, 2024 •

edited

Loading

alecandido commented Feb 26, 2024 •

edited

Loading

BrunoLiegiBastonLiegi commented Feb 26, 2024

alecandido commented Feb 26, 2024

Parallelization of CliffordOperations through numba and cupy #161

Parallelization of CliffordOperations through numba and cupy #161

Conversation

BrunoLiegiBastonLiegi commented Dec 13, 2023 • edited by alecandido Loading

alecandido commented Feb 22, 2024

alecandido commented Feb 22, 2024

andrea-pasquale commented Feb 22, 2024 • edited Loading

renatomello Feb 23, 2024

Choose a reason for hiding this comment

BrunoLiegiBastonLiegi commented Feb 26, 2024 • edited Loading

renatomello left a comment • edited Loading

Choose a reason for hiding this comment

BrunoLiegiBastonLiegi commented Feb 26, 2024

renatomello commented Feb 26, 2024 • edited Loading

alecandido commented Feb 26, 2024 • edited Loading

BrunoLiegiBastonLiegi commented Feb 26, 2024

alecandido commented Feb 26, 2024

Parallelization of `CliffordOperations` through `numba` and `cupy` #161

Parallelization of `CliffordOperations` through `numba` and `cupy` #161

BrunoLiegiBastonLiegi commented Dec 13, 2023 •

edited by alecandido

Loading

andrea-pasquale commented Feb 22, 2024 •

edited

Loading

BrunoLiegiBastonLiegi commented Feb 26, 2024 •

edited

Loading

renatomello left a comment •

edited

Loading

renatomello commented Feb 26, 2024 •

edited

Loading

alecandido commented Feb 26, 2024 •

edited

Loading