Titiler on Google Cloud Run - 502 Http errors

[AndreaGiardini]

Hello folks 👋

This is not a bug report but a return of experience running Titiler on Google Cloud Run. We have been running Titiler in our infrastructure for quite some time now, and since the early days, we have always noticed a consistent number of HTTP errors 502 being triggered by Cloud Run.

It was clear that something weird was going on... but the service was performing well. We started digging into it as this behavior created a lot of noise in our monitoring, and we didn't fully understand the cause.

What we noticed

We narrowed down the 502 errors to requests trying to retrieve tiles where the map was empty. By default, Titiler returns an HTTP 204 status for those requests, but in our deployment, those requests were 502 (bad gateway). In the beginning, we tried investigating why the upstream was not responding to those requests. Bad gateway errors can have a couple of different causes, so we increased the number of minimum instances, tuned our health checks, reduced the number of concurrent requests, etc... Unfortunately, none of those measures improved the situation.

Even more interesting was what we discovered when performing the same request from the browser or via curl:

• Opening the URL on the browser -> 502
• Querying the URL via curl -> 204

Moreover, there was no way to reproduce the problem locally. On a local docker container, both requests would return a 204.

Debugging the issue

We compared the response of the curl requests against our local Titiler vs. the curl on our deployment running in Cloud Run.

Both requests returned 204 in this case, but the returned headers differed. In particular, we noticed that our Cloud Run deployment had an extra content-type header. This is due to this Google Cloud Run bug.

While this behavior can sound harmless (and we didn't overthink about it initially), it was the cause of our 502 errors. Several web clients (QGIS, Chrome, etc.) consider those requests malformed and will trigger a 502. Other than messing up our monitoring system, clients like QGIS retry those requests (since a 502 is a retriable error) up to three times, amplifying the problem.

How we fixed the issue

We had to build our own custom version of Titiler and add extra middleware to reduce the 204 responses to 200.

class Rewrite204Middleware:
    def __init__(self, app):
        self.app = app

    async def __call__(self, scope: Scope, receive: Receive, send: Send) -> None:
        async def rewrite_status_code(message: Message):
            if message["type"] == "http.response.start" and message["status"] == 204:
                headers = MutableHeaders(raw=message["headers"])
                message["status"] = 200
                headers.update({
                    "Content-Type": "image/png",
                    "X-Rewritten-204": "true"
                })

            await send(message)

        await self.app(scope, receive, rewrite_status_code)

The result:

[vincentsarago]

Thanks for the report @AndreaGiardini 🙏 this is really interesting

In titiler we use exception_handler to catch Exceptions and return the appropriate Status Code:

titiler/src/titiler/core/titiler/core/errors.py

Lines 33 to 70 in 4937dfa

	DEFAULT_STATUS_CODES = {
	BadRequestError: status.HTTP_400_BAD_REQUEST,
	TileOutsideBounds: status.HTTP_404_NOT_FOUND,
	TileNotFoundError: status.HTTP_404_NOT_FOUND,
	RasterioIOError: status.HTTP_500_INTERNAL_SERVER_ERROR,
	MissingBands: status.HTTP_400_BAD_REQUEST,
	MissingAssets: status.HTTP_400_BAD_REQUEST,
	InvalidColorFormat: status.HTTP_400_BAD_REQUEST,
	InvalidAssetName: status.HTTP_404_NOT_FOUND,
	InvalidBandName: status.HTTP_404_NOT_FOUND,
	RasterioError: status.HTTP_500_INTERNAL_SERVER_ERROR,
	RioTilerError: status.HTTP_500_INTERNAL_SERVER_ERROR,
	Exception: status.HTTP_500_INTERNAL_SERVER_ERROR,
	}
	def exception_handler_factory(status_code: int) -> Callable:
	"""
	Create a FastAPI exception handler from a status code.
	"""
	def handler(request: Request, exc: Exception):
	if status_code == status.HTTP_204_NO_CONTENT:
	return Response(content=None, status_code=204)
	return JSONResponse(content={"detail": str(exc)}, status_code=status_code)
	return handler
	def add_exception_handlers(
	app: FastAPI, status_codes: Dict[Type[Exception], int]
	) -> None:
	"""
	Add exception handlers to the FastAPI app.
	"""
	for (exc, code) in status_codes.items():
	app.add_exception_handler(exc, exception_handler_factory(code))

in exception_handler_factory

titiler/src/titiler/core/titiler/core/errors.py

Lines 54 to 56 in 4937dfa

	def handler(request: Request, exc: Exception):
	if status_code == status.HTTP_204_NO_CONTENT:
	return Response(content=None, status_code=204)

we check for 202 and make sure to return Empty, maybe instead of adding this middleware you could just customize the add_exception_handlers and exception_handler_factory like this:

def exception_handler_factory(status_code: int) -> Callable:
    """
    Create a FastAPI exception handler from a status code.
    """

    def handler(request: Request, exc: Exception):
        if status_code == status.HTTP_204_NO_CONTENT:
            return Response(content=b"", status_code=200, headers={"Content-Type": "image/png", "X-Rewritten-204": "true"})

        return JSONResponse(content={"detail": str(exc)}, status_code=status_code)

    return handler


def add_exception_handlers(
    app: FastAPI, status_codes: Dict[Type[Exception], int]
) -> None:
    """
    Add exception handlers to the FastAPI app.
    """
    for (exc, code) in status_codes.items():
        app.add_exception_handler(exc, exception_handler_factory(code))

[AndreaGiardini]

One thing that I found challenging was extending the main application without reinventing the wheel.

I want to:

• Take the Titiler application as-is
• Rewrite 204 into 200

Originally, I tried to do that with a simple:

from titiler.application import main as titiler_app_main

app = titiler_app_main.app

And adding a FastAPI middleware to rewrite those requests.

No matter how I did it, I would always face this error: fastapi/fastapi#7187 . Even just adding an empty middleware without any rewrite would return this error.

In a FastAPI application middlewares get added in order. My guess was that this middleware gets added at the end of the chain, and that causes all sorts of weird things.

[vincentsarago]

In a FastAPI application middlewares get added in order. My guess was that this middleware gets added at the end of the chain, and that causes all sorts of weird things.

Yes, you'll need to first clean the middleware

app.user_middleware = []
app.middleware_stack = app.build_middleware_stack()

One thing that I found challenging was extending the main application without reinventing the wheel.

I can understand this, that's said the main application should be considered as a DEMO application, I always considered that users should create their own application. If we look at titiler.application.main it's only couple hundreds of lines which are just adding endpoints to a FastAPI app from endpoint factories: https://developmentseed.org/titiler/intro/#customized-minimal-app