[16.0][ADD] fs_storage: Storage Addons Refactoring RFC #252

lmignon · 2023-03-28T18:42:08Z

A PR to track changes to the initial proposal of #251

In this PR you can find a document describing the proposal and the implementation of the first part trough the addon fs_storage.

The pr #260 implements the new addon fs_attachment
The pr #261 implements the new addon fs_file

florian-dacosta · 2023-05-23T08:54:12Z

@lmignon Is this ready to test/review ? (I am asking because of the draft state)

This replace storage_backend + storage_backend submodules, right ? But storage_backend is already merged.
The idea is to keep these conccurent modules for v16 and drop the storage_backend in v17 ?

lmignon · 2023-05-23T09:44:30Z

@florian-dacosta This addon is still in draft since I'm waiting the feedback from C2C (camptocamp/odoo-cloud-platform#418) to relicence to LGPL.

This addon will replace the storage_backend addon
[16.0][ADD] fs_attachment: Add new addon #260 fs_attachment will replace base_attachment_object_storage and the companion addons from https://github.com/camptocamp/odoo-cloud-platform
[16.0][ADD] fs_file: New field to store files #261 fs_file will replace the storage_file addon

I'll try to finalize #262 in the coming days but I'm busy with others priorities even if #262 is a priority. I've a more complete implementation for the fs_file addon on my machine with the new widget etc...

florian-dacosta · 2023-05-23T10:07:25Z

@lmignon Thanks for the feedback.
But this PR (fs_storage) does not really depend on the C2C module, so this one should be ready, no ?

I'll try to use this fs_storage in a project using presently storage_backend (but not storage_file) and do a review here then.

lmignon · 2023-05-23T11:24:46Z

@lmignon Thanks for the feedback. But this PR (fs_storage) does not really depend on the C2C module, so this one should be ready, no ?

@florian-dacosta You're right, I'll change the status. Before I'll update the development_status to alpha while others companion addons are not finalized (even if I'm pretty sure that the API should not change)

fs_storage/models/fs_storage.py

florian-dacosta

LGTM
I did some tests with Odoo filestore, gtp, sftp and it went quite well.
Some minor remarks but nothing important, so I approve already.
I have tested with a real implementation (module attachment_synchronize)

Thanks for the work !

florian-dacosta · 2023-05-29T11:59:41Z

fs_storage/models/fs_storage.py

+        required=True,
+        help="Technical code used to identify the storage backend into the code."
+        "This code must be unique. This code is used for example to define the "
+        "storage backend to store the attachments vi the configuration parameter "


florian-dacosta · 2023-05-29T13:47:52Z

fs_storage/__manifest__.py

    "installable": True,
-    "depends": ["base", "component", "server_environment"],


storage_backend did depend on server_environment, probably because you usually want to have this kind of configuration depending on the environment and also because you usually need to store some secrets, which is better not to leave in clear in the database.
Is this removal intentional, because we'd prefer to have a glue module for this ?

@florian-dacosta it's an open question... I tend to prefer to have glue addon than having too much dependencies.

Well, before, we'd have different fields for each protocol (sftp_login, ftp_login, etc). Including server-env was avoiding a lot of small server-env glue module.
Now, we would just need one, since every thing is in the json field and we don't have extra fields per protocol.
So, no problem for me to have a glue module.

florian-dacosta · 2023-05-29T14:02:26Z

fs_storage/models/fs_storage.py

+
+class FSStorage(models.Model):
+    _name = "fs.storage"
+    _backend_name = "fs_storage"


I did not see it used anymore, is it still usefull ?

florian-dacosta · 2023-05-29T14:10:19Z

fs_storage/models/fs_storage.py

+
+    # the next fields are used to display documentation to help the user
+    # to configure the backend
+    options_protocol = fields.Selection(


I don't get why this field is usefull as we could use protocol to display the right options_properties.
Maybe I missed something, but it would avoid some code duplication between _get_options_protocol and _get_protocols and also make sure you display the description of the chosen protocol.
It is a bit strange to have to choose a protocol and then to have to choose again to see the right description in the available option tab.

@florian-dacosta With the lib you can nest the protocol. Therefore, the same field can be used to get access to the protocol documentation for the main protocol and when configuring nested protocols into the options field.

This addon define a new model 'fs.storage' used to get access to a filesystem storage (ftp, sftp, s3, azure, ...) through an unified interface provided by the 'fsspec' python library (https://filesystem-spec.readthedocs.io/en/latest)

lmignon · 2023-08-24T08:54:33Z

/ocabot merge nobump

OCA-git-bot · 2023-08-24T08:54:39Z

Hey, thanks for contributing! Proceeding to merge this for you.
Prepared branch 16.0-ocabot-merge-pr-252-by-lmignon-bump-nobump, awaiting test results.

OCA-git-bot · 2023-08-24T08:58:21Z

Congratulations, your PR was merged at 16f607e. Thanks a lot for contributing to OCA. ❤️

lmignon force-pushed the 16.0-refactoring-rfc branch from e98a5ad to 9ac76ff Compare March 28, 2023 20:50

lmignon mentioned this pull request Mar 29, 2023

[16.0][IMP] storage_backend: New implementation #250

Closed

4 tasks

lmignon force-pushed the 16.0-refactoring-rfc branch from 9ac76ff to 2b2ff4e Compare April 6, 2023 15:39

lmignon mentioned this pull request Apr 7, 2023

base_attachment_object_storage refactored into OCA/storage camptocamp/odoo-cloud-platform#418

Closed

lmignon force-pushed the 16.0-refactoring-rfc branch from 2b2ff4e to 50be001 Compare April 7, 2023 09:28

lmignon mentioned this pull request Apr 13, 2023

[16.0][ADD] fs_attachment: Add new addon #260

Closed

lmignon changed the title ~~[16.0] Storage Addons Refactoring RFC~~ [16.0][ADD] fs_storage: Storage Addons Refactoring RFC Apr 17, 2023

lmignon mentioned this pull request Apr 27, 2023

[16.0][ADD] fs_file: New field to store files #261

Merged

6 tasks

florian-dacosta reviewed May 23, 2023

View reviewed changes

fs_storage/models/fs_storage.py Show resolved Hide resolved

lmignon marked this pull request as ready for review May 24, 2023 09:16

florian-dacosta approved these changes May 29, 2023

View reviewed changes

florian-dacosta mentioned this pull request May 29, 2023

[16.0][MIG] attachment_synchronize OCA/server-tools#2632

Merged

This was referenced Aug 21, 2023

[16.0][MIG] storage_backend_s3: Migration to 16.0 #270

Closed

[16.0][MIG] storage_backend_sftp: Migration to 16.0 #241

Merged

lmignon force-pushed the 16.0-refactoring-rfc branch 5 times, most recently from 9ffdef7 to fbc99fd Compare August 24, 2023 08:51

[ADD] fs_storage: Get access to filesyste storages

a937c9b

This addon define a new model 'fs.storage' used to get access to a filesystem storage (ftp, sftp, s3, azure, ...) through an unified interface provided by the 'fsspec' python library (https://filesystem-spec.readthedocs.io/en/latest)

lmignon force-pushed the 16.0-refactoring-rfc branch from fbc99fd to a937c9b Compare August 24, 2023 08:52

OCA-git-bot added the bot is merging ⏳ label Aug 24, 2023

OCA-git-bot merged commit b328d6e into OCA:16.0 Aug 24, 2023
6 checks passed

OCA-git-bot added merged 🎉 and removed bot is merging ⏳ labels Aug 24, 2023

lmignon deleted the 16.0-refactoring-rfc branch August 24, 2023 09:07

lmignon mentioned this pull request Aug 24, 2023

[16.0] [IMP] fs_attachment: store attachments linked to different model/fields to different FS storages #269

Merged

lmignon mentioned this pull request Oct 7, 2024

[17.0] fs_storage: Port from 16.0 to 17.0 #400

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[16.0][ADD] fs_storage: Storage Addons Refactoring RFC #252

[16.0][ADD] fs_storage: Storage Addons Refactoring RFC #252

lmignon commented Mar 28, 2023 •

edited

Loading

florian-dacosta commented May 23, 2023

lmignon commented May 23, 2023

florian-dacosta commented May 23, 2023

lmignon commented May 23, 2023

florian-dacosta left a comment

florian-dacosta May 29, 2023

florian-dacosta May 29, 2023

lmignon Jun 5, 2023

florian-dacosta Jun 5, 2023

florian-dacosta May 29, 2023

lmignon Jun 5, 2023

florian-dacosta May 29, 2023

lmignon Jun 5, 2023

lmignon commented Aug 24, 2023

OCA-git-bot commented Aug 24, 2023

OCA-git-bot commented Aug 24, 2023

		"installable": True,
		"depends": ["base", "component", "server_environment"],

[16.0][ADD] fs_storage: Storage Addons Refactoring RFC #252

[16.0][ADD] fs_storage: Storage Addons Refactoring RFC #252

Conversation

lmignon commented Mar 28, 2023 • edited Loading

florian-dacosta commented May 23, 2023

lmignon commented May 23, 2023

florian-dacosta commented May 23, 2023

lmignon commented May 23, 2023

florian-dacosta left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lmignon commented Aug 24, 2023

OCA-git-bot commented Aug 24, 2023

OCA-git-bot commented Aug 24, 2023

lmignon commented Mar 28, 2023 •

edited

Loading