run(): Retry Mechanism

[ChrisTalman]

Transient failures can sometimes cause a query to fail at one moment, but be capable of running successfully a moment later. For instance, network instability or limited database system resources.

Could it be useful for the driver to provide a retry mechanism of some kind which can be enabled by specifying an option to run()? The same option could also be provided as part of the connection options, so that retry is conducted by default without being explicitly specified for every run().

Up to now, I have written a kind of wrapper around the driver which extends the functionality of run() with a retry mechanism. However, this seems like the sort of thing which could be useful to have built into the driver itself.

A simple exponential backoff would probably work for most use cases. If greater flexibility were required, the user could provide a callback to manage the backoff themselves.

I imagine that retry would only be conducted for errors likely to be transient, like a timeout. It would probably not make sense to retry for errors like a duplicate primary key.

[ronzeidman]

Sounds good, can you share your wrapper? For what exceptions exactly are you retrying? Do you also reconnect or just resending the query? Are you using the connection pool?

…

On Mon, Dec 31, 2018, 14:38 Chris Talman ***@***.*** wrote: Transient failures can sometimes cause a query to fail at one moment, but be capable of running successfully a moment later. For instance, network instability or limited database system resources. Could it be useful for the driver to provide a retry mechanism of some kind which can be activated by specifying an option to run()? The same option could also be provided as part of the connection options, so that retry is conducted by default without being explicitly specified for every run(). Up to now, I have written a kind of wrapper around the driver which extends the functionality of run() with a retry mechanism. However, this seems like the sort of thing which could be useful to have built into the driver itself. A simple exponential backoff would probably work for most use cases. If greater flexibility were required, the user could provide a callback to manage the backoff themselves. I imagine that retry would only be conducted for errors likely to be transient, like a timeout. It would probably not make sense to retry for errors like a duplicate primary key. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#11>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ANWmx3ajdB8I54RNV7ri5jS8XqaktEAIks5u-gVagaJpZM4ZlaAY> .

[ChrisTalman]

Sounds good, can you share your wrapper?

It was essentially just a method which would run a query within a try...catch block using await. If it ran successfully, the result would be returned. If it failed, it would be retried after a short delay, starting at 200ms, and doubling on each failure. If it failed 3 times, the last error would be thrown.

For what exceptions exactly are you retrying?

I needed the functionality quickly, and so at the time I decided to simply retry on all errors. It would be preferable to only retry on appropriate errors. I'm not familiar with the driver protocol, so I'm not aware of the full range of potential errors. I can't remember the precise form of the transient errors I experienced in the past, but their error messages were generally concerning a connection timeout or reset. Using this page as a reference, these might be reasonable candidates: ReqlTimeoutError, ReqlAvailabilityError, ReqlInternalError, and perhaps ReqlDriverError.

Do you also reconnect or just resending the query?

I just resend the query, although I suppose it's plausible that a transient error could be due to a connection failure which needs resolution in the driver.

Are you using the connection pool?

Yep.

[GaikwadPratik]

@ronzeidman @ChrisTalman

Did you get a chance to work on this further?

[ChrisTalman]

I've just been using the function I wrote for myself.

[atassis]

@ChrisTalman Hello there. I am starting to contribute widely to this repo. I would like to provide this kind of feature for the driver. Can you share your function for faster development? If not, it'll take much longer for this task being reached.

[ChrisTalman]

Hey @atassis. The code is here.

It makes use of my own delay() method, but that can be easily substituted.

I hope you find it in some way useful. It wasn't exhaustively planned out, and so it may not be the most elegant implementation. But, it has seemed to serve me very well for quite some time at this point.