obsync fail when 404 on source

[guilhem]

boto return element None when radosgw return 404.

This is a simple hack.

[mwodrich]

You would not necessarily want to simply continue without some knowledge of the error if your source returned a 404. What is causing 404s from your source?

[guilhem]

My patch is quite simple and can be improved. I do a pull request to help if someone as the same problem and to help in debugging.

This error is given by radosgw. I do a migration from a bucket to another.
This error append every time on the same object, but it can't be list in a tool like cloudberry. It must be a problem with radosgw but I don't want to wait for a patch. This tool must not crash in every error from the s3 gateway.

[mwodrich]

That patch looks like a good quick fix for your situation, but violates the guarantee that obsync currently makes that if it finished successfully, it has to the best of its knowledge successfully moved all objects listed. You're absolutely right that it should not abort from every error from the gateway, but we also want to be sure that when we get those errors, they're not ignored. I'm currently working on some improvements to obsync to make it copy objects in parallel and make it tolerate a tunable number of failures before aborting, which should address this problem, and once I've got it working, I'd love to have some help testing it out. I'll make sure to account for this issue while I'm at it.

[guilhem]

I agree with you. This improvement can be really good.

Just For information, issue #4 make me search for something else.
I find this software : https://github.com/seedifferently/boto_rsync
It work well (with some patches in pull request). But it doesn't do it in parallel :/

[mwodrich]

Also, I'd strongly encourage you to report an issue on the ceph redmine tracker concerning your problem object with as much information as you can provide. I'm sure they would also be very interested to know what's causing it.

[liewegas]

Going to close this old request out.