Remove cached images (by API/manual)

[kleisauke]

Remove cached images (by API/manual) [15774949]

Submitted by Eugene Huseynov on 23-8-2016 0:00:00
24 votes on UserVoice prior to migration

Given

Image source (url parametre) is: https://maps.googleapis.com/maps/api/staticmap?sensor=false&size=2048x350&scale=2&maptype=roadmap&zoom=13&markers=color:blue|size:small|1400+Oakton+Street,+Evanston,+IL+60202

Encoded version: maps.googleapis.com%2Fmaps%2Fapi%2Fstaticmap%3Fsensor%3Dfalse%26size%3D2048x350%26scale%3D2%26maptype%3Droadmap%26zoom%3D13%26markers%3Dcolor%3Ablue%257Csize%3Asmall%257C1400%2BOakton%2BStreet%2C%2BEvanston%2C%2BIL%2B60202

I'm using following images.weserv.nl parameters: &w=70&t=square&h=70
Final link: http://images.weserv.nl/?url=maps.googleapis.com%2Fmaps%2Fapi%2Fstaticmap%3Fsensor%3Dfalse%26size%3D2048x350%26scale%3D2%26maptype%3Droadmap%26zoom%3D13%26markers%3Dcolor%3Ablue%257Csize%3Asmall%257C1400%2BOakton%2BStreet%2C%2BEvanston%2C%2BIL%2B60202&w=70&t=square&h=70

Issue

Some of the "google map images" cached properly, some of them is not (link provided above - final link)
Looks like when caching process is broken by any internal reason, the "image url" is still remembered as a key in your collection. So there is no way to rebuild correct cache for that "broken" image.

Question

Is any api/way to expire/delete/rebuild/invalidate existing image cache?

Response

by Andries Louw Wolthuizen on 23-8-2016 0:00:00

Need to check if there is a reliable way to remove images from all caches. Will look into this!

Comments

Comment by Eugene Huseynov on 30-8-2016 22:39:00

Hi Andries.

I appreciate for your time and energy you spend for this project.

Any luck to fix such king issues or may be add an API for the project so customers could reset/update/delete cached images themself?

Thank you one more time.

Eugene

Comment by Andries Louw Wolthuizen (Webmaster, Images.weserv.nl - Image cache & resize proxy) on 1-9-2016 18:53:00

Hi Eugene,

This project didn't start with the intent of providing access to reset/update/delete images. To serve all users, there are caches at multiple levels involved (edge, processing, backplanes, requesters). The difficulty is that not all of these caches are capable of reliable removing single entries, and the freshness is also varying wildly (based on effort to recreate, the number of requests, etc, etc).

However, the feature as proposed is still on the roadmap, and will probably be implemented in the next code-refresh. When this will be finished is uncertain, as many new features and changes are to be incorporated in such an update.

In the meanwhile, you could consider using our code to set up your own solution. The GitHub-repo is sans any cache, so you should extent it with your own caching solution, or disregard it completly. For starters I would advise some form of reverse caching proxy (like nginx).

--
Andries

Original UserVoice Submission

[andrieslouw]

See #74

[andrieslouw]

This feature is planned, but it needs additional code to delete images from the different caches. Currently it isn’t exactly clear when this will be implemented. I will update this idea when I have more info.

[tomaskazatel]

Hello @andrieslouw we use your image cache service. And now we have problem with not supported removing image cache. Do you have some update for it?

[andrieslouw]

There is no update yet, implementation is quite difficult. We have some code laying around to remove images from 1 or 2 cache-levels, but not yet from every cache (we cache on ~4 different levels nowadays), and this code isn't reliable enough for production yet. You could use the source-code to host your own image proxy, caching is something that isn't included in the source (we cache by means of 2x Nginx+Memcached+Squid).

[andrieslouw]

Small update; there is some redesigning of the back-end of the public images.weserv.nl service going on, to remove some of the caching levels. When finished, this may open the road to reliable on-demand removal from cache. We will keep you updated.

[andrieslouw]

Work has started in the 4.x-crt branch. Caching has been reduced to 2 levels, API for global cache still needs to be written. Proof of Concept is live on: https://images.weserv.nl/crt/

[DaanEasyAds]

Hi guys,

I've tried to clear the cache for this URL, URL encoded and not, but no luck so far. Also tried without ssl: and with https://.

https://images.weserv.nl/?url=ssl:www.houseofchesterfield.fr%2Ffiles%2F15543%2Fproducts%2F13109365%2F0101.jpg&w=75&h=65&errorredirect=resources.easyads.eu%2Fimages%2Fdashboard%2Fplaceholder.jpg

https://images.weserv.nl/?url=ssl:www.houseofchesterfield.fr/files/15543/products/13109365/0101.jpg&w=75&h=65&errorredirect=resources.easyads.eu/images/dashboard/placeholder.jpg

Do you see any other solution or does it simply not work? ;)

Thanks!

Daan

[andrieslouw]

@DaanEasyAds : If you're still experiencing issues with your specific URL, please create a new issue, as comments on features are not monitored every day. The Proof of Concept only removes from local cache, global is still not written.

[andrieslouw]

Due to #186 it may seem that we cache longer; but there is no change in the amount of time we cache server side. Finishing this feature is still on our roadmap, and work is ongoing, but we need to write additional API's for all the different layers of caching we use.

[3cstudio]

@andrieslouw where does this topic of clearing image cache stand? I had a client using a WordPress Plugin (Podcast Feed Player Widget and Shortcode) that shows image thumbnails for a podcast preview "cover" image via Weserv, but he was uploading large PNG files (!). Thus, he quickly "outgrew" his allotted space, I'm guessing. :-o Only the Alt tag text was showing in place of the thumbnails. We have since revamped/resized all those images, but now would love to clear the cached giant PNG files that Weserv may have on file still... Love the service! Just can't continue to use it in its present state of not allowing for cache clearing. :-/ Thoughts? Ideas? Thanks for all you do. -Steve

[laontme]

We met this issue today after updating file in our s3 but using same filename. So as workaround we found adding something like &hash=<hash-of-your-file>. If file same then hash will not change but if file changed hash will change and cache will not work

Example:
https://wsrv.nl/?url=i.imgur.com/fSgnUKW.jpeg&w=300&hash=aboba228

[paul-uz]

Any update?

[erichusted]

Any update?

I'm not complaining, because it is a free service after all, but 7 years since the original request is amazing!

[andrieslouw]

There is no update, in addition to reliable clearing caches on multiple servers, and making the nginx service aware of this removal, we'd also need to use the Cloudflare API to submit cache clearing requests to Cloudflare (after we made sure it has been removed from our servers caching).

Things get even more interesting when you consider the rate limits Cloudflare has on these API's, and the potential for abuse we'd need to limit.

Any contribution on this is welcome, our servers operate using the code hosted here on GitHub. The old 4.x-crt branch does contain some code on how to calculate and process the md5 hashes nginx uses internally, but it needs to be rewritten to C++ as we don't use Lua anymore. Furthermore, nginx will need to be reloaded, and the CF-API needs to be called afterwards to purge the specific URL's, preferably in batches.