Add lookup index mode #115143
Add lookup index mode #115143
Conversation
dnhatn
force-pushed
the
lookup-mode
branch
8 times, most recently
from
5570065
to
88f9c68
Compare
elasticsearchmachine
added
the
serverless-linked
dnhatn
added
v8.17.0
:Analytics/ES|QL
elasticsearchmachine
added
the
Team:Analytics
|
Pinging @elastic/es-analytical-engine (Team:Analytics) |
| void validateWithOtherSettings(Map<Setting<?>, Object> settings) { | ||
| Integer defaultNumberOfShards = IndexMetadata.INDEX_NUMBER_OF_SHARDS_SETTING.get(Settings.EMPTY); | ||
| Object providedNumberOfShards = settings.getOrDefault(IndexMetadata.INDEX_NUMBER_OF_SHARDS_SETTING, defaultNumberOfShards); | ||
| if (Objects.equals(providedNumberOfShards, defaultNumberOfShards) == false) { |
There was a problem hiding this comment.
Maybe just check whether providedNumberOfShards is 1?
| public Settings defaultIndexSettings() { | ||
| return Settings.builder() | ||
| .put(IndexMetadata.SETTING_NUMBER_OF_SHARDS, 1) | ||
| .put(IndexMetadata.SETTING_AUTO_EXPAND_REPLICAS, "0-all") |
There was a problem hiding this comment.
Should we also validate whether auto expand replicas is really enabled? This just defines the defaults, but doesn't enforce that it is actually used?
There was a problem hiding this comment.
I may be missing something, but the default settings are copied to the index settings when creating an index: https://github.com/elastic/elasticsearch/pull/115143/files#diff-c02a1326ed501a89ce49b7e7a27df4bb9a2256785d00d7278bccb3609334b939R1112
There was a problem hiding this comment.
I see, the default settings are added after template and create index request settings have been applied. Maybe default settings is the wrong name? Since the settings from index mode don't act as default, but are enforced?
Additionally I think we need to mimic this logic in two other places too for consistency:
MetadataIndexTemplateService#validateIndexTemplateV2(...)to ensure that index mode default settings are applied during template validation.TransportSimulateIndexTemplateAction#resolveTemplate(...)to ensure that index mode default settings are applied during simulate template api.
This change introduces a new index mode,
lookup, for indices intended for lookup operations in ES|QL. Lookup indices must have a single shard and be replicated to all data nodes by default. Aside from these requirements, they function as standard indices. Documentation will be added later when the lookup operator in ES|QL is implemented.