Stop synchronisation with an error when pcloud is detected

[laurent22]

pcloud is a popular cloud service, however their WebDAV implementation is broken and have been so for a long time. The only reply we got from them was essentially blaming Joplin, even though the apps sync fine with dozens of other standard WebDAV services.

There's two problems with this:

• Users incorrectly think that it's a problem with Joplin and come for us for support. It uses our resources and it makes it look like Joplin is unreliable, even though it's not possible on our end to fix the issue

• Although their implementation is broken, it stills works well enough that a user might rely on it, not realising that they are losing notes or that some data get corrupted.

So we should deal with it on our end: if pcloud is detected, we immediately stop sync and throw an error explaining the situation. For example:

pcloud has a faulty WebDAV implementation and as such Joplin no longer supports it. Please use a different sync method.

That way the user is immediately alerted and can switch to a different sync target without committing to pcloud.

Edit: We don't know the full story - it's possible that for certain use cases, pCloud work fine, maybe for small notes or not many notes, so some users might already be using it. Because of this, we should add an option in Advanced settings - "Allow sync with pCloud" (with a description). Off by default. If On, we allow sync but at the user's own risks. It means we remain backward compatible for users who are currently synchronising with pCloud.

Edit: In fact, we can have a migration step - if during the upgrade we detect the user is using one of the unsupported implementations, we automatically set the Advanced option to "true".

Implementation

Based on the discussion in this thread, this is what we need to implement:

1. Create a black list of WebDAV providers - at this point it will only be pcloud and jianguoyun
2. When the user tries to sync with these (including when clicking the button "Check synchronisation configuration"), throw an error message from the synchronizer: "The WebDAV implementation of PROVIDER is incompatible with Joplin, and as such is no longer supported. Please use a different sync method."
3. Add a hidden setting "sync.allowUnsupportedProviders" (defaults to -1)
4. Add a migration step - when the app starts, and sync.allowUnsupportedProviders is -1, check the current sync settings. If it's currently synchronising with one of the unsupported provider, set sync.allowUnsupportedProviders to 1. Otherwise, set it to 0. This is so we don't break sync for people who are already synchronising with pcloud or jianguoyun
5. If sync.allowUnsupportedProviders is 1, we bypass the WebDAV black list

[Daeraxa]

Could it be made generic so we can add other "unsupported" services into it? The override could then be something like "Allow sync with unsupported provider" (https://joplinapp.org/faq/#the-following-webdav-hosts-are-not-supported).

Would also be useful if we make it obvious in the log if that option has been selected so if people do paste a log without the webdav url or details then we can immediately see that the option has been intentionally selected.

[laurent22]

Yes that's a good point, let's do that. It means we can also block jianguoyun, and we'll have a built-in list of unsupported providers.

Perhaps when there's an error with an unsupported provider, we should always add a message "Please note that pCloud WebDAV is known to be faulty so we do not support it". And since it's part of the error message, it will also end up in the log

[dpoulton-github]

Seems I was thinking along the same lines about using a list of unsupported providers...

https://discourse.joplinapp.org/t/problem-of-syncronisation-on-linux-and-android/29221/5

[laurent22]

In terms of implementation we'll probably go for simply throwing an error + advanced setting, as that would automatically work in all applications. The alert box is more user friendly but would need to be manually created in each app.

We don't want to support implementations like pCloud at all, even after warning the user. Users forget what options they've enabled and when 2 months later they find that some notes are missing they won't think it's because of that. We're only providing the option for backward compatibility because we don't want to force people to suddenly migrate - if it works for them, good, but new users shouldn't use pCloud or jianguoyun at all.

[thomasboots]

Hello,
I'm not sure if my message helps, but just in case. I am a pcloud user, so I contacted pcloud support. For them the problem comes from joplin, because via a browser for example, you can use the pcloud webdav (which is true for connecting to the webdav via a browser, but no idea if that means their webdav really works).
Have a nice day, and thanks for your collective work!
Thomas

[roman-r-m]

For them the problem comes from joplin, because via a browser for example, you can use the pcloud webdav (which is true for connecting to the webdav via a browser, but no idea if that means their webdav really works)

Any chance they can confirm what exactly is causing this? From what I understand this is an intermittent issue so something must be triggering the reject.

[laurent22]

Back when we looked into it we saw issues like these:

Issue 1

Apache/2.4.10 (Debian) Server at webdav.pcloud.com Port 443
Code: 500
Error: PUT .resource/…: Unknown error 2 (500):

500 Internal Server Error

A server error is not something we can fix, it's on their end.

Issue 2

Also errors like this:

2019-11-30 19:29:17: “FetchError: response timeout at https://webdav.pcloud.com/Applications/Joplin/ 6 over limit: 120000
FetchError: response timeout at https://webdav.pcloud.com/Applications/Joplin/ 6 over limit: 120000

If they deny service over a certain limit, it's also not going to work. The correct approach, implemented by most other services, is to throttle the connections.

Issue 3

Finally there was this one that was worrying:

DecryptionWorker: error for: 1d3ef47fb59a419bae5f1f6dd8afdf74 (resources)”, “Error: json decode: this isn’t json!

It means that we downloaded an item from their service, we checked that the status code was valid, and yet we got some corrupted data. This should not happen - if there's an error (server error or otherwise), they need to set the HTTP status codes properly and not send random data. If the Joplin app knows there's an error, it won't attempt to process the data.

I assume they sent an HTML page with some error in plaintext, but they need to check the WebDAV spec because that's not how it's supposed to work.

[Geim66]

Hello, I'm not sure if my message helps, but just in case. I am a pcloud user, so I contacted pcloud support. For them the problem comes from joplin, because via a browser for example, you can use the pcloud webdav (which is true for connecting to the webdav via a browser,

As I started using Pcloud I could make backups from my Synology NAS through webdav without any problems and decided to go for a payed plan. All went fine until December 2021, when the backups failed. Contacting them got me an answer like
"Note, however, that the WebDAV protocol is mainly for connecting to other applications and transferring smaller files."
Never got it running again. But I kept on using webdav to sync my Enpass password vault, but even that stopped working and I now found another place to store it.
In my opinion Pcloud changed something with their webdav implementation and are not open about it.

Now using C2 for my backups.

[josephbortka]

Joplin 3.1.24 linux, iOS versions as well.

I came here after downloading Joplin and setting it up with my pCloud service. Here is what I did:

First I set it up on my iPad and iPhone. They worked fine though the syncing appears to take a bit long at several mins for the tiny amount of data. I'm not sure if that's a problem with webdav, joplin, or pcloud. Having never used webdav before, it took some tinkering, but I finally got it working. Pcloud does not appear to have any webdav instructions on their website as of this date, but their main directories are https://webdav.pcloud.com (US Servers) and https://ewebdav.pcloud.com (EU). this will go to your main directory, and you can put your file in a lower level directory such as https://ewebdav.pcloud.com/folder-1/folder-2.

This worked and I believe the first error with the html code was from a conflict between the directories on different devices. I first set up the iPad and I had no issues once I got the correct directory. When I set up my iPhone I had a few issues with the html errors but i deleted the default notes and it worked fine to sync afterwards. I hadn't quite put it together yet.

Then I went to set it up on my ubuntu laptop, where again I ran into the html error. after some looking around in the settings, I activated the following: (Help/Options/Synchronization/[Delete local data and re-download from sync target]. This seems to have resolved the issue for now, but I will update if I have future issues.

Thanks laurent22 for doing open source dev like this!

[josephbortka]

Follow up, this is the answer I received when I inquired about these issues:

Hello,

Thank you for contacting pCloud's Support.

We don't have our own implementation of WebDav and just use the default WebDav module from a web server (apache). This means that we can't really fix issues with third-party software as they either work or they don't and unfortunately, we can't do much about it on our end. If you can connect to https://ewebdav.pcloud.com/ with a browser and access your files, the service is up and running.

Please note that we recommend using our GUI client pCloud Drive and our native sync/backup functions, as this is the most suitable way for larger transfers at this stage. The WebDAV connection is not suitable for large backups, so we recommend using it for transfers of small amounts of files.

Please ask if you have any questions.

Regards,
Chris
pCloud Support Team

As of now the service appears to be functioning properly for me.