Multi-threaded Limitador #37
Conversation
didierofrivia
force-pushed
the
rfc/limitador-multithread-inmemory
branch
2 times, most recently
from
42d042a to
861612d
Compare
didierofrivia
force-pushed
the
rfc/limitador-multithread-inmemory
branch
from
861612d to
dbbd516
Compare
didierofrivia
force-pushed
the
rfc/limitador-multithread-inmemory
branch
from
dbbd516 to
9ba163d
Compare
didierofrivia
force-pushed
the
rfc/limitador-multithread-inmemory
branch
from
f2ca2d9 to
af76b43
Compare
* Bonus: fixing typos
didierofrivia
force-pushed
the
rfc/limitador-multithread-inmemory
branch
from
7ffaf67 to
3d16710
Compare
There was a problem hiding this comment.
I might be wrong, but it seems to me that no one requires 100% accuracy. So --mode=accuracy would not be necessary. For me it is more about what level of the rate throughput / accuracy you want. Having valid values meaningful names, never numbers. Something like
high=> I do not care about accuracy, I want it as fast as possiblemedium=> I care about bothlow=> I care about accuracy but assuming it will not be 100% accurate
| # Motivation | ||
| [motivation]: #motivation | ||
|
|
||
| Currently, Limitador service is single threaded, regardless of its chosen storage. This means that it can only process one request at a time. |
There was a problem hiding this comment.
This is actually not true, is it? The gRPC service (tonic) is built on top of Tokio, an event-driven, non-blocking I/O platform. Or when the storage is memory, there is no I/O involved, hence only one request at a time can be processed?
There was a problem hiding this comment.
See this issues, other than something having changed in newer version, this was still true like 10 months ago.
| value given per service. | ||
| * **Throughput**: The _throughput_ of a service is defined by the number of requests that the service can process in a | ||
| given time. As a consequence of having higher throughput, we need to introduce two more concepts: | ||
| * **Overshoot**: The _overshooting_ of a limit counter is defined by the difference between the _expected_ value of the counter |
There was a problem hiding this comment.
overshooting is when the counter goes beyond the limit and the request is still admitted, right?
There was a problem hiding this comment.
Interesting, I agree with your definition, but looks like @didierofrivia looks at it the other way around.
| ## Behaviour | ||
|
|
||
| When using the multi-threading approach, we need to understand the trade-offs that we are making. The main one is that | ||
| it's not possible to have both _Accuracy_ and _Throughput_ at the same time. This means that we need to choose one of |
There was a problem hiding this comment.
it would be nice to explain why both are not achievable.
| Now imagine that we have the following requests happening at the same time in parallel: `POST`, `POST`, `POST`, `GET`, | ||
| `GET`; it could happen that the service authorizes the first two `POST` requests and updates both counters to 2, then the third | ||
| `POST` request is not authorized *but* the counter of limit 1 is updated to 3 (wrongly), and finally then only one `GET` request | ||
| that comes in would be authorized, leaving one wrongly denied. In this case, we would have an _overshoot_ of 1 in the |
There was a problem hiding this comment.
I would say this is a undershooting example, i.e. traffic is being rejected before reaching the limit.
There was a problem hiding this comment.
I may need to review my understanding about these concepts.
| 1. Retrieve the Counter object from the collection. | ||
| 2. Compare the current timestamp with the expiration time of the counter to determine if the request falls within the | ||
| sliding window | ||
| 3. If the request falls within the window, increment the hit count of the Counter. |
There was a problem hiding this comment.
shouldn't the increment happen only when it has been verified that no counters has exceeded the associated limit?
No description provided.