Set a multi-thread scheduled executor for tests

[ScottDugas]

MoreAsyncUtil installs a JVM-wide single-thread ScheduledThreadPoolExecutor by default. When the suite runs N tests concurrently and they each schedule deadline timers on that single shared thread, the timers fire late and produce spurious DeadlineExceededException from AsyncLoadingCache -- most prominently FDBDatabase.resolverStateCache during KeySpacePath.toTuple cleanup.

Override the FDBDatabaseFactory default scheduled executor in FDBDatabaseExtension and other tests with a multi-threaded pool (max(availableProcessors, 4)) shared across all tests.

Closes #4333

[alecgrieser]

Having a multi-threaded scheduled executor doesn't seem like a bad idea, but I do have some questions

[alecgrieser]

Hm, I do have a few questions about this.

For one, how backed up are we talking about here? I haven't looked too deeply here, so my intuition could be wrong here, but it seems like the tasks on the single threaded scheduled executor service really shouldn't be that much, at least under reasonable parallelism. However, looking at delayedFuture, the way the code is currently written, callbacks get executed on the scheduled executor service thread. So, it may be the case that we should modify delayedFuture so that we call completeAsync(null, executor) here:

fdb-record-layer/fdb-extensions/src/main/java/com/apple/foundationdb/async/MoreAsyncUtil.java

Line 1016 in 4e3d42b

scheduledExecutor.schedule(() -> future.complete(null), delay, unit);

That will result in the callbacks running off of the scheduled thread, which should improve the scheduling accuracy.

But actually, more to the point, it kind of seems like if the scheduling thread is backed up, then we'd get fewer DeadlineExceededExceptions, not more. Looking at getWithDeadline:

fdb-record-layer/fdb-extensions/src/main/java/com/apple/foundationdb/async/MoreAsyncUtil.java

Lines 1040 to 1047 in 4e3d42b

	return CompletableFuture.anyOf(MoreAsyncUtil.delayedFuture(deadlineTimeMillis, TimeUnit.MILLISECONDS, scheduledExecutor), valueFuture)
	.thenCompose(ignore -> {
	if (!valueFuture.isDone()) {
	// if the future is not ready then we exceeded the timeout
	valueFuture.completeExceptionally(new DeadlineExceededException(deadlineTimeMillis));
	}
	return valueFuture;
	});

We should only get a DeadlineExceededException if the delayed future completes before the valueFuture. Unless the scheduled executor responds to too much work by completing tasks prematurely, then a busy scheduling thread means that the deadline is lengthened.

[ScottDugas]

I think the issue is that the logic that is resulting in DeadlineExceededException is also doing retries with exponential backoff. The overwhelming majority in the tests, IIUC is resolving directory layers.
So I think what is happening is that it:

1. schedules the exception
2. starts the resolution
3. fails, and schedules a delayed future to retry

The ScheduledExecutor is backed up, which, since it was unable to execute either at the time it was supposed to executes the DeadlineExceededException first, because it was submitted first.

This could be indicative that there is something running on the ScheduledThreadPool that is not supposed to.
When I get a chance I will try to install a ScheduledThreadPool that tracks the time it takes for its runnables to run and throws an error if anything takes longer than some time, hopefully allowing us to find a task that is running there, but should be on a different executor. If that doesn't prove too fruitful, I think it's probably worthwhile to just change our executor to the modestly sized pool used in this PR.

[alecgrieser]

Yeah, that makes sense. Doing the experiment sounds like a good idea, and if it comes up inconclusive (or is harder to execute than we'd hope), then merging this PR as it is sounds okay

[ScottDugas]

I did some more investigation, and it appears to hint that the issue is not due to overwhelming the executor.
I suspect it is one of the other executors getting overwhelmed. I'm going to hold onto this PR as a draft until more of the work to allow parallelism gets in, and we can see if this problem happens again.