Add mixed mode BWC testing support for test-specific plugins

[ChrisHegarty]

This issues been filed to examine how best to support the inference-service-test plugin in ES|QL mixed version testing.

The ES|QL CSV and REST tests run with a variety of modes (see x-pack/plugin/esql/qa/server):

1. single-node
2. multi-node
3. mixed-cluster
4. multi-clusters

No.1 and no.2 run the same ES version on all nodes. No.3 and no.4 run with mixed ES versions, either within the same cluster or across clusters.

The issue occurs only with no.3 and no.4 where the BWC infrastructure is not able to install the interference service test-specific plugin into the older nodes. When we try it, we get the following error:

> Task :x-pack:plugin:esql:qa:server:mixed-cluster:v8.17.0#javaRestTest FAILED

MixedClusterEsqlSpecIT > classMethod FAILED
    java.lang.RuntimeException: An error occurred orchestrating test cluster.
        at __randomizedtesting.SeedInfo.seed([27B9DA09AA60FCA0]:0)
        at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.execute(DefaultLocalClusterHandle.java:270)
        at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.execute(DefaultLocalClusterHandle.java:275)
        at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.start(DefaultLocalClusterHandle.java:78)
        at org.elasticsearch.test.cluster.local.DefaultLocalElasticsearchCluster$1.evaluate(DefaultLocalElasticsearchCluster.java:46)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleStoreClassName$1.evaluate(TestRuleStoreClassName.java:38)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.NoShadowingOrOverridesOnMethodsRule$1.evaluate(NoShadowingOrOverridesOnMethodsRule.java:40)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at org.apache.lucene.tests.util.TestRuleAssertionsRequired$1.evaluate(TestRuleAssertionsRequired.java:53)
        at org.apache.lucene.tests.util.AbstractBeforeAfterRule$1.evaluate(AbstractBeforeAfterRule.java:43)
        at org.apache.lucene.tests.util.TestRuleMarkFailure$1.evaluate(TestRuleMarkFailure.java:44)
        at org.apache.lucene.tests.util.TestRuleIgnoreAfterMaxFailures$1.evaluate(TestRuleIgnoreAfterMaxFailures.java:60)
        at org.apache.lucene.tests.util.TestRuleIgnoreTestSuites$1.evaluate(TestRuleIgnoreTestSuites.java:47)
        at org.junit.rules.RunRules.evaluate(RunRules.java:20)
        at com.carrotsearch.randomizedtesting.rules.StatementAdapter.evaluate(StatementAdapter.java:36)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl$StatementRunner.run(ThreadLeakControl.java:390)
        at com.carrotsearch.randomizedtesting.ThreadLeakControl.lambda$forkTimeoutingTask$0(ThreadLeakControl.java:850)
        at java.base/java.lang.Thread.run(Thread.java:1570)

        Caused by:
        java.lang.IllegalArgumentException: Can't run executable: `/Users/chegar/git/elasticsearch-lucene_snapshot/x-pack/plugin/esql/qa/server/mixed-cluster/build/testrun/v8.17.0_javaRestTest/temp/test-cluster8788876539987453686/test-cluster-2/distro/bin/elasticsearch-plugin` does not exist.
            at org.elasticsearch.test.cluster.util.ProcessUtils.exec(ProcessUtils.java:54)
            at org.elasticsearch.test.cluster.local.AbstractLocalClusterFactory$Node.runToolScript(AbstractLocalClusterFactory.java:784)
            at org.elasticsearch.test.cluster.local.AbstractLocalClusterFactory$Node.installPlugins(AbstractLocalClusterFactory.java:633)
            at org.elasticsearch.test.cluster.local.AbstractLocalClusterFactory$Node.start(AbstractLocalClusterFactory.java:159)
            at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.lambda$start$0(DefaultLocalClusterHandle.java:78)
            at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
            at java.base/java.util.AbstractList$RandomAccessSpliterator.forEachRemaining(AbstractList.java:722)
            at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:556)
            at java.base/java.util.stream.ForEachOps$ForEachTask.compute(ForEachOps.java:291)
            at java.base/java.util.concurrent.CountedCompleter.exec(CountedCompleter.java:759)
            at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)
            at java.base/java.util.concurrent.ForkJoinTask.invoke(ForkJoinTask.java:676)
            at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateParallel(ForEachOps.java:160)
            at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateParallel(ForEachOps.java:174)
            at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:264)
            at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:611)
            at java.base/java.util.stream.ReferencePipeline$Head.forEach(ReferencePipeline.java:785)
            at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.lambda$start$1(DefaultLocalClusterHandle.java:78)
            at org.elasticsearch.test.cluster.local.DefaultLocalClusterHandle.lambda$execute$16(DefaultLocalClusterHandle.java:276)
            at java.base/java.util.concurrent.ForkJoinTask$AdaptedInterruptibleCallable.compute(ForkJoinTask.java:1689)
            at java.base/java.util.concurrent.ForkJoinTask$InterruptibleTask.exec(ForkJoinTask.java:1641)
            at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:507)
            at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1491)
            at java.base/java.util.concurrent.ForkJoinPool.scan(ForkJoinPool.java:2073)
            at java.base/java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:2035)
            at java.base/java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:187)

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[elasticsearchmachine]

Pinging @elastic/es-core-infra (Team:Core/Infra)