ODB occasionally returns 503s

[toddburnside]

I noticed this during calls to the attachment REST API. It frequently happens for file uploads, and may happen more frequently for larger files. But, it happens for smaller files, too. I believe it has also happened while trying to obtain pre-signed urls.

However, looking in the logs for an example, I see instances for it happening on the ws endpoint as well. Here is an example:

Clue must be retrying it so we haven't been seeing the error in the UI.

[tpolecat]

Do you have any other information about the 503? There should be a stacktrace or something, I would think.

[toddburnside]

That's all there is in the log. I wonder if it's even getting to the ODB. The log has heroku[router] in it.

[cquiroz]

Just got another one today:

Access to fetch at 'https://lucuma-postgres-odb-dev.herokuapp.com/attachment/obs/p-100?fileName=usco6166547.jpg&attachmentType=OBS_ATTACHMENT_TYPE%28FINDER%2CFINDER%2CFINDER%20CHART%2CLIST%28JPEG%2C%20JPG%2C%20PNG%29%29' from origin 'https://local.lucuma.xyz:8080' has been blocked by CORS policy: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

explore.js:86528     POST https://lucuma-postgres-odb-dev.herokuapp.com/attachment/obs/p-100?fileName=usco6166547.jpg&attachmentType=OBS_ATTACHMENT_TYPE%28FINDER%2CFINDER%2CFINDER%20CHART%2CLIST%28JPEG%2C%20JPG%2C%20PNG%29%29 net::ERR_FAILED 503 (Service Unavailable)

[toddburnside]

According to the heroku documentation (https://devcenter.heroku.com/articles/error-codes#h18-server-request-interrupted)

The error occurs in cases where the socket was destroyed before sending a complete response, or if the server responds with data before reading the entire body of the incoming request.

In the recent incident reported by Carlos, this happened because of an invalid attachment type in the request. The server detected this error and responded without reading the request body stream. (It seems that for small enough bodies, http4s must read the entire body up front because the error doesn't happen.)

I can refactor the services a bit to prevent this error for bad requests. The downside is that if the user is trying to upload a 10M file, we'll have to stream the entire 10M before telling them they have something like a bad attachment type or file extension.

There are cases we won't be able to prevent.

• Once we start streaming the file to AWS, I think we can't finish reading the body it there is an error.
• We are using the http4s EntityLimiter middleware to limit file size. This will always result in the 503. Whether we use EntityLimiter or not, we certainly don't want to try to read an entire 10G file. We'd timeout, anyway. We can catch this in the UI, but API users will be left with a cryptic error. (We may want to add an API endpoint for retrieving metadata like the max file size so explore can keep up with any changes)