Skip to content

Commit

Permalink
finagle-core: ExponentialJitteredBackoff random zero bugfix
Browse files Browse the repository at this point in the history
**Problem:** `ExponentialJitteredBackoff.next` throws an exception occasionally.
``` java.lang.IllegalArgumentException:
     at java.base/java.util.concurrent.ThreadLocalRandom.nextLong(ThreadLocalRandom.java:361)
     at com.twitter.finagle.util.Rng$$anon$2.nextLong(Rng.scala:71) ERR
     at com.twitter.finagle.Backoff$ExponentialJittered.next(Backoff.scala:331) ```
**Bug:** `Rng.nextLong(n)` generates a number from 0 (inclusive) to n
(exclusive). When it generates 0 as random number. The next occurrence of
backoff.next would try fetching `Rng.nextLong(0)` which is illegal argument.

**Fix:** Add 1 to the generated random, so that `Rng.nextLong(_)` can accept the
number. Since the random generated is exclusive on the `n` provided, this
doesn't cause overflow as well.

JIRA Issues: STOR-8734

Differential Revision: https://phabricator.twitter.biz/D1178528
  • Loading branch information
Shashikanth Reddy Potu authored and jenkins committed Nov 4, 2024
1 parent df4c740 commit 05f5618
Show file tree
Hide file tree
Showing 3 changed files with 25 additions and 7 deletions.
1 change: 1 addition & 0 deletions CHANGELOG.rst
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ Bug Fixes

* finagle-memcached: Fixed support for running memcached tests with external memcached. Added README with
instructions under finagle/finagle-memcached. ``PHAB_ID=D1120240``
* finagle-core: Fixed a bug in `ExponentialJitteredBackoff` where `rng` can be 0. ``PHAB_ID=D1178528``


Breaking API Changes
Expand Down
5 changes: 3 additions & 2 deletions finagle-core/src/main/scala/com/twitter/finagle/Backoff.scala
Original file line number Diff line number Diff line change
Expand Up @@ -328,8 +328,9 @@ object Backoff {
if (maxBackoff == maximum) {
new Const(maximum)
} else {
val random = Duration.fromNanoseconds(rng.nextLong(maxBackoff.inNanoseconds))
new ExponentialJittered(random, maximum, attempt + 1, rng)
// to avoid the case of random being 0
val random = 1 + rng.nextLong(maxBackoff.inNanoseconds)
new ExponentialJittered(Duration.fromNanoseconds(random), maximum, attempt + 1, rng)
}
}

Expand Down
26 changes: 21 additions & 5 deletions finagle-core/src/test/scala/com/twitter/finagle/BackoffTest.scala
Original file line number Diff line number Diff line change
Expand Up @@ -147,23 +147,29 @@ class BackoffTest extends AnyFunSuite with ScalaCheckDrivenPropertyChecks {
forAll(exponentialGen) {
case (startMs: Long, maxMs: Long, seed: Long) =>
val rng = Rng(seed)
val backoff: Backoff =
new ExponentialJittered(startMs.millis, maxMs.millis, 1, Rng(seed))
val result: ArrayBuffer[Duration] = new ArrayBuffer[Duration]()
var start = startMs.millis
val max = maxMs.millis
val backoff: Backoff = new ExponentialJittered(start, max, 1, Rng(seed))
val result: ArrayBuffer[Duration] = new ArrayBuffer[Duration]()
for (attempt <- 1 to 7) {
result.append(start)
start = nextStart(start, maxMs.millis, rng, attempt)
start = nextStart(start, max, rng, attempt)
}
verifyBackoff(backoff, result.toSeq, exhausted = false)

// Verify case where Rng returns 0.
val zeroRng = getZeroRng(rng)
val zeroRngBackoff: Backoff = new ExponentialJittered(start, max, 1, zeroRng)
val expectedResults = Seq(start, nextStart(start, max, zeroRng, 1))
verifyBackoff(zeroRngBackoff, expectedResults, exhausted = false)
}

def nextStart(start: Duration, max: Duration, rng: Rng, attempt: Int): Duration = {
val shift = 1L << attempt
// to avoid Long overflow
val maxBackoff = if (start >= max / shift) max else start * shift
if (maxBackoff == max) max
else Duration.fromNanoseconds(rng.nextLong(maxBackoff.inNanoseconds))
else Duration.fromNanoseconds(1 + rng.nextLong(maxBackoff.inNanoseconds))
}
}

Expand Down Expand Up @@ -331,4 +337,14 @@ class BackoffTest extends AnyFunSuite with ScalaCheckDrivenPropertyChecks {
}
assert(actualBackoff.isExhausted == exhausted)
}

private[this] def getZeroRng(rng: Rng): Rng = new Rng {
override def nextDouble(): Double = rng.nextDouble()

override def nextInt(n: Int): Int = rng.nextInt(n)

override def nextInt(): Int = rng.nextInt()

override def nextLong(n: Long): Long = 0L
}
}

0 comments on commit 05f5618

Please sign in to comment.