Migrating Pytorch backend to Qiboml #1510

MatteoRobbiati · 2024-10-30T10:01:24Z

Checklist:

Reviewers confirm new code works as expected.
Tests are passing.
Coverage does not decrease.
Documentation is updated.

Some things to be discussed and still not touched in this PR

I think we should fix the procedure based on this function, otherwise we will always need to depend on Torch;
In examples like this one we define a torch tensor which is provided to our circuit. I would like to drop any dependence on pytorch in Qibo, thus I propose to delete these kind of advanced examples, pointing to Qiboml documentation (where we will rewrite dedicated examples). Otherwise, we should build the torch tensor doing something like the following (which is very confusing and we must avoid):

backend = construct_backend(backend="qiboml", platform="pytorch")

# create a torch tensor from backend using torch which is renamed np
tensor = backend.np.ones( .... )

codecov · 2024-10-30T12:37:50Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 99.69%. Comparing base (ea6984d) to head (832d43c).
Report is 58 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #1510      +/-   ##
==========================================
- Coverage   99.72%   99.69%   -0.03%     
==========================================
  Files          80       79       -1     
  Lines       11784    11631     -153     
==========================================
- Hits        11752    11596     -156     
- Misses         32       35       +3

Flag	Coverage Δ
unittests	`99.69% <100.00%> (-0.03%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

doc/source/code-examples/advancedexamples.rst

renatomello · 2024-11-04T07:26:33Z

doc/source/getting-started/installation.rst

+------------------+------+---------+-----------+---------+
+| Operating System | qibo | qibojit |Tensorflow | Pytorch |
+==================+======+=========+===========+=========+
+| Linux x86        | Yes  | Yes     | Yes       | Yes     |
+------------------+------+---------+-----------+---------+
+| MacOS >= 10.15   | Yes  | Yes     | Yes       | Yes     |
+------------------+------+---------+-----------+---------+
+| Windows          | Yes  | Yes     | Yes       | Yes     |
+------------------+------+---------+-----------+---------+


Should this be updated to contemplate the CPU / GPU subdivision? Both qibojit and qiboml have backends that work on both.

I don't know, since this is already shown in the backends diagram.

renatomello · 2024-11-06T09:20:50Z

I think we should fix the procedure based on this function, otherwise we will always need to depend on Torch;

That function is needed to properly have some methods of gates.Unitary working for all possible input types the matrix can be (i.e. np.ndarray, tf.Tensor, torch.Tensor, cp.ndarray). I'm not sure there is a way out of this dependency without creating a gate dependency on backends. One immediate thing we could do is to rename np as platform in all backends since that's the name we give in set_backend anyway.

BrunoLiegiBastonLiegi · 2024-11-08T11:14:54Z

doc/source/getting-started/installation.rst

+preferred package is installed following `Tensorflow's <https://www.tensorflow.org/install>`_
+or `Pytorch's <https://pytorch.org/get-started/locally/>`_ installation instructions.
+
+To switch to Tensorflow or Pytorch backend please do:


Not sure whether it is worth to have details about the ml backends. I would probably place this in the qiboml docs and link them here.

I agree, I would remove these lines. Probably it is better to wait the moment the Qiboml docs are ready so that we can just link to the proper section.

BrunoLiegiBastonLiegi · 2024-11-08T11:23:57Z

tests/test_hamiltonians.py

+        if backend.platform in ["tensorflow", "pytorch"]:
+            pytest.skip(
+                "Tensorflow and Pytorch do not support operations with sparse matrices."
+            )


This is not really for this PR, but I think that both of them support sparse matrices to some extent https://pytorch.org/docs/stable/generated/torch.sparse.mm.html
https://www.tensorflow.org/api_docs/cc/class/tensorflow/ops/sparse-mat-mul

I am not sure what was causing problems here, but I guess it is more a matter of how the operation is called rather than the impossibility to perform it. Anyway, this is probably out of the scope of this PR.

I just opened an issue with this comment.

BrunoLiegiBastonLiegi · 2024-11-08T11:27:46Z

tests/test_models_variational.py

-        (backend.name != "pytorch") or (backend.platform != "tensorflow")
+        backend.platform not in ["tensorflow", "pytorch"]


This could become a property of the backend, e.g.:

@property def support_differentiation(self,): return False

to avoid checking precisely for which backend this works, but again might be out of scope here.

I think the best thing for Qibo will be to remove all these tests about variational models.

Co-authored-by: Renato Mello <[email protected]>

…_migration

MatteoRobbiati · 2024-11-12T12:51:42Z

By my side, I think we can stop modifications here for this PR. Still we need to remove a lot of examples and subsections from the docs, but I would prefer to do that after we populate the Qiboml documentation. Otherwise we will have vacations in the documentation. What you think @renatomello, @BrunoLiegiBastonLiegi?

BrunoLiegiBastonLiegi · 2024-11-18T11:19:00Z

We need to update this https://github.com/qiboteam/qiboml/blob/800e800d54f4962f4890716c4b1539014c69c1b5/src/qiboml/backends/__init__.py#L3

BTW, is the PytorchBackend of qiboml updated or we are still waiting for qiboteam/qiboml#30 ?

MatteoRobbiati · 2024-11-19T08:22:24Z

We need to update this https://github.com/qiboteam/qiboml/blob/800e800d54f4962f4890716c4b1539014c69c1b5/src/qiboml/backends/__init__.py#L3

Done in qiboml, thanks!

BTW, is the PytorchBackend of qiboml updated or we are still waiting for qiboteam/qiboml#30 ?

That was outdated. Just closed!

BrunoLiegiBastonLiegi · 2024-11-19T15:35:21Z

okay it should be just a matter of regenerating the lock here now

renatomello · 2024-11-20T05:22:12Z

qibo is importing the torch backend from qiboml and qiboml is importing the torch backend from qibo

https://github.com/qiboteam/qibo/actions/runs/11888475398/job/33209011064?pr=1510

MatteoRobbiati · 2024-11-20T08:14:45Z

qibo is importing the torch backend from qiboml and qiboml is importing the torch backend from qibo

This should have been solved merging qiboteam/qiboml#52 (comment) yesterday. Rerunning the tests to double check :)

…moved

MatteoRobbiati · 2024-11-20T11:25:44Z

I just removed a gradient test from here and opened a dedicated issue in qiboml.
Also, the test_list_available_backends should be fixed here, so after merging the master into your branch, @renatomello, the problem mentioned in #1520 should be solved.

renatomello · 2024-11-20T11:54:48Z

I just removed a gradient test from here and opened a dedicated issue in qiboml. Also, the test_list_available_backends should be fixed here, so after merging the master into your branch, @renatomello, the problem mentioned in #1520 should be solved.

You mean after this PR is merged, right?

MatteoRobbiati · 2024-11-20T11:56:45Z

You mean after this PR is merged, right?

Yes, I mean this PR should be solving that problem too.

MatteoRobbiati added 5 commits October 30, 2024 10:45

feat: migration of pytorch backend effects in qibo source code

41e0b85

feat: adapt tests to pytorch migration

c1eef23

chore: point to qiboml torch migration branch

d3255c1

fix: import torch backend from qiboml

fdaf4c8

fix: adapt vqe and qaoa optimization tests to pytorch migration

86e1574

MatteoRobbiati added 2 commits November 1, 2024 08:23

Merge branch 'master' into torch_migration

cbc6ed9

docs: update docs after moving pytorch to qiboml

ce623b1

MatteoRobbiati requested review from BrunoLiegiBastonLiegi, renatomello and alecandido November 1, 2024 09:25

MatteoRobbiati marked this pull request as ready for review November 1, 2024 09:26

renatomello reviewed Nov 4, 2024

View reviewed changes

doc/source/code-examples/advancedexamples.rst Outdated Show resolved Hide resolved

renatomello reviewed Nov 4, 2024

View reviewed changes

renatomello self-requested a review November 4, 2024 08:17

MatteoRobbiati added this to the Qibo 0.2.14 milestone Nov 6, 2024

BrunoLiegiBastonLiegi reviewed Nov 8, 2024

View reviewed changes

MatteoRobbiati mentioned this pull request Nov 12, 2024

Execute tests with Qiboml backends and sparse matrices #1522

Open

MatteoRobbiati and others added 4 commits November 12, 2024 11:38

Update doc/source/code-examples/advancedexamples.rst

77008a8

Co-authored-by: Renato Mello <[email protected]>

Merge branch 'master' into torch_migration

b744521

docs: correct backends subection in api-reference documentation section

299d78a

Merge branch 'torch_migration' of github.com:qiboteam/qibo into torch…

1c38069

…_migration

Merge branch 'master' into torch_migration

dbf8cbe

renatomello approved these changes Nov 14, 2024

View reviewed changes

fix test

9762788

MatteoRobbiati requested a review from BrunoLiegiBastonLiegi November 14, 2024 07:34

MatteoRobbiati added 2 commits November 18, 2024 09:12

Merge branch 'master' into torch_migration

70e26ac

chore: point to qiboml main branch

aa22507

MatteoRobbiati added 2 commits November 20, 2024 09:10

Merge branch 'master' into torch_migration

5208307

chore: updating lock

eed754f

MatteoRobbiati added 2 commits November 20, 2024 09:34

fix: set tensorflow to False in test_list_available_backends since re…

b9e340b

…moved

feat: rm torch and tf gradients test from qibo

832d43c

BrunoLiegiBastonLiegi approved these changes Nov 27, 2024

View reviewed changes

MatteoRobbiati added this pull request to the merge queue Nov 28, 2024

Merged via the queue into master with commit bbf0ccf Nov 28, 2024
27 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Migrating Pytorch backend to Qiboml #1510

Migrating Pytorch backend to Qiboml #1510

MatteoRobbiati commented Oct 30, 2024 •

edited

Loading

codecov bot commented Oct 30, 2024 •

edited

Loading

renatomello Nov 4, 2024

MatteoRobbiati Nov 12, 2024

renatomello commented Nov 6, 2024 •

edited by alecandido

Loading

BrunoLiegiBastonLiegi Nov 8, 2024

MatteoRobbiati Nov 12, 2024

BrunoLiegiBastonLiegi Nov 8, 2024

MatteoRobbiati Nov 12, 2024

BrunoLiegiBastonLiegi Nov 8, 2024

MatteoRobbiati Nov 12, 2024

MatteoRobbiati commented Nov 12, 2024

BrunoLiegiBastonLiegi commented Nov 18, 2024

MatteoRobbiati commented Nov 19, 2024

BrunoLiegiBastonLiegi commented Nov 19, 2024

renatomello commented Nov 20, 2024

MatteoRobbiati commented Nov 20, 2024

MatteoRobbiati commented Nov 20, 2024

renatomello commented Nov 20, 2024

MatteoRobbiati commented Nov 20, 2024

		(backend.name != "pytorch") or (backend.platform != "tensorflow")
		backend.platform not in ["tensorflow", "pytorch"]

Migrating Pytorch backend to Qiboml #1510

Migrating Pytorch backend to Qiboml #1510

Conversation

MatteoRobbiati commented Oct 30, 2024 • edited Loading

Some things to be discussed and still not touched in this PR

codecov bot commented Oct 30, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

renatomello commented Nov 6, 2024 • edited by alecandido Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MatteoRobbiati commented Nov 12, 2024

BrunoLiegiBastonLiegi commented Nov 18, 2024

MatteoRobbiati commented Nov 19, 2024

BrunoLiegiBastonLiegi commented Nov 19, 2024

renatomello commented Nov 20, 2024

MatteoRobbiati commented Nov 20, 2024

MatteoRobbiati commented Nov 20, 2024

renatomello commented Nov 20, 2024

MatteoRobbiati commented Nov 20, 2024

MatteoRobbiati commented Oct 30, 2024 •

edited

Loading

codecov bot commented Oct 30, 2024 •

edited

Loading

renatomello commented Nov 6, 2024 •

edited by alecandido

Loading