Migrate SampleOperator Status transport version to standard getMinimalSupportedVersion pattern#120
Open
augment-app-staging[bot] wants to merge 504 commits intomainfrom
Open
Migrate SampleOperator Status transport version to standard getMinimalSupportedVersion pattern#120augment-app-staging[bot] wants to merge 504 commits intomainfrom
augment-app-staging[bot] wants to merge 504 commits intomainfrom
Conversation
…131429) (elastic#132073) * Prevent auto-sharding for data streams in LOOKUP index mode * Update docs/changelog/131429.yaml * Reduce test duplication
) Previously, entitlement checks got disabled when resetting the policy manager (which defaults to inactive). This change makes sure entitlements are correctly enabled during tests. Due to the lack of entitlement delegation (and usage of server's FileSystemUtils and similar in test code), there's a few remaining issues: - various tests have to run without entitlements - node base dirs cannot be removed immediately when shutting down the node due to pending cleanups (wipePendingDataDirectories) Due to Netty dependency issues (ES-12435), azure and inference tests have to run without entitlements.
…2157) (elastic#132233) This is the same failure as observed in elastic#129644 for which the original fix elastic#129680 did not really work. It did not work because the the ordering of checks. The shutdown marker is removed after the cluster passes ready check so that new shards can be allocated. But the cluster cannot pass the ready check before the shards are allocated. Hence the circular dependency. In hindsight, there is no need to put shutdown record for all nodes. It is only needed on the node that upgrades the last to prevent snapshot from completion during the upgrade process. This PR does that which ensures there are always 2 nodes for hosting new shards. Resolves: elastic#132135 Resolves: elastic#132136 Resolves: elastic#132137 (cherry picked from commit f39ccb5) # Conflicts: # muted-tests.yml
…lastic#132244) The listener is expected to be called on failure case. This PR does that. Resolves: elastic#131979
…tBlocksNewMergeTasksFromStartingExecution (elastic#132020) (elastic#132267) Avoid mocking merge tasks as backlogged (which are then re-enqueued) because it makes the merge task's queue available budget value unsteady, which breaks the test when it later schedules merge tasks that are under/over budget. Closes elastic#130205 elastic#131982
(cherry picked from commit 067a812)
(cherry picked from commit 438f627)
…ic#132170) (elastic#132270) A follow-up to elastic#131990. This PR ensures that only assigned allocations and not current allocations are used in the memory requirements calculation in AssignmentPlan. This change led to the simplification of the code in ZoneAwareAssignmentPlanner and TrainedModelRebalancer. This PR also improves readability by adding comments, code documentation, renaming variables, and making the flow of if statements more straightforward. Marking is a non-issue since the bug was already documented in elastic#131990.
If we combined two `BulkResponse`s with `ingest_took` set to `NO_INGEST_TOOK` we'd get an `ingest_took` of `-2`. Which doesn't make any sense. This fixes it to be set to `NO_INGEST_TOOK` properly.
This change is needed after release of 8.17.9 as 8.17.10 is the next version for the 8.17 branch. It has been already added previously, but reverted in https://github .com/elastic/elasticsearch/commit /f8d97f36e37fe18530e2f3858ded147f11634ebe probably by a bug in automation. (cherry picked from commit 42f9368) # Conflicts: # .buildkite/pipelines/intake.yml # .buildkite/pipelines/periodic.yml
Test was removed in elastic#132063, so we can remove the mutes and close the test failure issue. Closes elastic#131803
…lastic#132171) This commit fixes an issue whereby indices created prior to 7.14 that have an index sort on a date_nanos field can no longer be opened (with versions >= 7.14). When opening an index the configured index sort, derived from the index settings, must match exactly that of the sort encoded in the index itself. A change to fix a bug back in 7.14 changed this for date_nanos fields whose index.sort.missing value is absent, see elastic#74760. Specifically, the default minimum value changed from Long.MIN_VALUE to 0L. The change in this commit restores the default minimum value for indices prior to 7.14.
In some contexts ESQL's HeapAttack tests were failing because the TranslogWriter keeps a map with the contents of the documents in it around. But only when assertions are enabled. Fun. This flushes the map away, preventing the odd OOM we were seeing in the HeapAttack tests.
…ate (elastic#128667) (elastic#132160) (cherry picked from commit 989f72b)
…ic#132260) (elastic#132333) `DriverStatus` is an immutable record created by the Driver. However, its components are not inherently immutable. This PR fixes a live collection used by the Driver, that was being directly put into the status, leading to `ConcurrentModificationException`s when reading it (Through the Task list API, for example). Some example errors: ``` java.util.ConcurrentModificationException at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1096) at java.base/java.util.ArrayList$Itr.next(ArrayList.java:1050) at org.elasticsearch.compute.operator.DriverStatus.toXContent(DriverStatus.java:134) at org.elasticsearch.xcontent@9.2.0/org.elasticsearch.xcontent.XContentBuilder.value(XContentBuilder.java:993) at org.elasticsearch.xcontent@9.2.0/org.elasticsearch.xcontent.XContentBuilder.field(XContentBuilder.java:978) at org.elasticsearch.server@9.2.0/org.elasticsearch.tasks.TaskInfo.toXContent(TaskInfo.java:113) at org.elasticsearch.server@9.2.0/org.elasticsearch.action.admin.cluster.node.tasks.list.TaskGroup.toXContent(TaskGroup.java:63) at org.elasticsearch.server@9.2.0/org.elasticsearch.action.admin.cluster.node.tasks.list.TaskGroup.toXContent(TaskGroup.java:67) at org.elasticsearch.server@9.2.0/org.elasticsearch.action.admin.cluster.node.tasks.list.ListTasksResponse.lambda$groupedByParent$10(ListTasksResponse.java:183) ``` And: ``` java.util.ConcurrentModificationException at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1096) at java.base/java.util.ArrayList$Itr.next(ArrayList.java:1050) at org.elasticsearch.compute.operator.DriverStatus.documentsFound(DriverStatus.java:157) at org.elasticsearch.compute.operator.DriverStatus.toXContent(DriverStatus.java:129) at org.elasticsearch.xcontent@9.2.0/org.elasticsearch.xcontent.XContentBuilder.value(XContentBuilder.java:993) at org.elasticsearch.xcontent@9.2.0/org.elasticsearch.xcontent.XContentBuilder.field(XContentBuilder.java:978) at org.elasticsearch.server@9.2.0/org.elasticsearch.tasks.TaskInfo.toXContent(TaskInfo.java:113) at org.elasticsearch.server@9.2.0org.elasticsearch.action.admin.cluster.node.tasks.list.TaskGroup.toXContent(TaskGroup.java:63) at org.elasticsearch.server@9.2.0/org.elasticsearch.action.admin.cluster.node.tasks.list.TaskGroup.toXContent(TaskGroup.java:67) at org.elasticsearch.server@9.2.0/org.elasticsearch.action.admin.cluster.node.tasks.list.ListTasksResponse.lambda$groupedByParent$10(ListTasksResponse.java:183) ``` Also, this looks like the source of this issue, with another similar case: Fixes elastic#131564
…lastic#132342) (cherry picked from commit 419eeb9)
…lastic#131753)" (elastic#132363) Reverts elastic#131753 Changes in release-manager are not ready yet.
This PR contains the following updates: | Package | Type | Update | Change | |---|---|---|---| | docker.elastic.co/wolfi/chainguard-base | | digest | `2a601e3` -> `442a566` | | docker.elastic.co/wolfi/chainguard-base | final | digest | `2a601e3` -> `442a566` | | docker.elastic.co/wolfi/chainguard-base | stage | digest | `2a601e3` -> `442a566` | | docker.elastic.co/wolfi/chainguard-base-fips | | digest | `93befc0` -> `4a8fe3e` | | docker.elastic.co/wolfi/chainguard-base-fips | final | digest | `93befc0` -> `4a8fe3e` | | docker.elastic.co/wolfi/chainguard-base-fips | stage | digest | `93befc0` -> `4a8fe3e` | --- ### Configuration 📅 **Schedule**: Branch creation - "after 1pm on tuesday" (UTC), Automerge - At any time (no schedule defined). 🚦 **Automerge**: Disabled by config. Please merge this manually once you are satisfied. ♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox. 👻 **Immortal**: This PR will be recreated if closed unmerged. Get [config help](https://elastic.slack.com/archives/C07AMD4CNUR) if that's undesired. --- - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box --- This PR has been generated by [Renovate Bot](https://redirect.github.com/renovatebot/renovate). <!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzOS4xMDcuMCIsInVwZGF0ZWRJblZlciI6IjM5LjEwNy4wIiwidGFyZ2V0QnJhbmNoIjoiOS4xIiwibGFiZWxzIjpbIjpEZWxpdmVyeS9QYWNrYWdpbmciLCI+bm9uLWlzc3VlIiwiVGVhbTpEZWxpdmVyeSIsImF1dG8tbWVyZ2Utd2l0aG91dC1hcHByb3ZhbCJdfQ==-->
…etting (elastic#134099) (elastic#134108) Ensure green after rollover to avoid unexpected license state flipping. Resolves: elastic#133455
…33793) (elastic#134111) This PR focuses on the short term solution which add the logs-sentinel_one.application-* and logs-sentinel_one.application_risk-* indices under the kibana_system role with deletion privileges to prevent a failed deletion error when the index enters the deletion phase for the ILM lifecycle, in upcoming PR. As it ships transform pipeline too hence read, write permissions are also required. Current behavior: It shows permission issue while deleting the index. (cherry picked from commit bfde47a)
…T testWriteLoadForecastGetsPopulatedDuringRollovers elastic#134123
…elastic#134139) This ensures we provide a new test seed per build invocation when running with configuration cache enabled while still ensuring the configuration cache can be reused. basically test seed beeing ignored as cc input.
…lastic#134084) (elastic#134142) This reverts our other bootstrapping of file-based transport versions to make migration simpler moving forward and adds the initial version one prior to the last initial version for each of 9.1, 9.0, 8.19, and 8.18.
…4158) (elastic#134159) This leaves the latest initial elasticsearch transport version for 8.18 as a marker. ES-12334
…#134176) This leaves the latest initial elasticsearch transport version for 8.19 as a marker. ES-12334
…tic#134191) * Mark migrate_reindex APIs as public * Add indices.create_from too * Mark migrate_reindex APIs as stable * Fix indices.get_migrate_reindex_status stability
These tests would sometimes create 1-replica indices in a 1-node cluster which will never reach `green` health. This commit ensures there are enough nodes to allocate all the shards. Closes elastic#133455 Closes elastic#134124 Closes elastic#134123 Backport of elastic#134132 to `9.1`
They are used in Kibana, and documented as Kibana-only in the documentation.
…34226) This leaves the latest initial elasticsearch transport version for 9.0 as a marker. ES-12334
…artialResultsSetToFalse (elastic#134236) (elastic#134247) * Block remote query search too (cherry picked from commit 9f4262b) # Conflicts: # muted-tests.yml
(cherry picked from commit 500c4ff)
…stic#134291) We already fixed an issue with this in elastic#128362 but apparently another instance of the unnecessary determinization was hiding elsewhere and in its current state throws exceptions starting with Lucene 10 on complext patterns. This change adds the same fix as elastic#128362 and adds a test that would have triggered this. Closes elastic#133652
elastic#134302) This reverts commit 76c2ab5.
) (elastic#134313) When new cluster settings are applied, consumers watchings are notified. The consumer should see the latest value at the time of initializing a watch. However, only the node settings were ever used in finding the value to initialize the consumer. This commit fixes the consumer to use the value from latest applied if it exists. closes elastic#133701
…134311) This leaves the latest initial elasticsearch transport version for 9.1 as a marker. ES-12334
Tracks the more memory that's involved in topn. Lucene doesn't track memory usage for TopN and can use a fair bit of it. Try this query: ``` FROM big_table | SORT a, b, c, d, e | LIMIT 1000000 | STATS MAX(a) ``` We attempt to return all million documents from lucene. Is we did this with the compute engine we're track all of the memory usage. With lucene we have to reserve it. In the case of the query above the sort keys weight 8 bytes each. 40 bytes total. Plus another 72 for Lucene's `FieldDoc`. And another 40 at least for copying to the values to `FieldDoc`. That totals something like 152 bytes a piece. That's 145mb. Worth tracking! ## Esql Engine TopN Esql *does* track memory for topn, but it doesn't track the memory used by the min heap itself. It's just a big array of pointers. But it can get very big!
…lastic#134336) Comments may be added to the beginning of transport version files by starting each comment line with the hash # character.
…stic#134351) Reverts workaround for minio/minio#21456 introduced in elastic#131815.
elastic#134377 is an automatically generated backport PR that was merged without CI actually having run. It turns out that there are changes in my newly added tests there that aren't compatible with the 9.1 branch. That would have shaken out if CI had actually run, but it didn't, so I'm fixing it in post. This is tangentially related to elastic#134319.
…pattern Replace the custom supportsVersion override with the standard getMinimalSupportedVersion pattern. The default implementation in VersionedNamedWriteable already does version.supports(getMinimalSupportedVersion()) which produces the same behavior.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
20 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Migrate the
SampleOperator.Statusinner record's transport version handling from the customsupportsVersionoverride pattern to the standardgetMinimalSupportedVersionpattern.Changes
getMinimalSupportedVersion()to returnESQL_SAMPLE_OPERATOR_STATUS_9_1instead of throwingUnsupportedOperationExceptionsupportsVersion(TransportVersion)override, since the default implementation inVersionedNamedWriteablealready doesversion.supports(getMinimalSupportedVersion())which produces the same behaviorTriggered by: Slack-Bot-Evals
Slack discussion: https://augment-wic8570.slack.com/archives/C0AG2MXKEQJ/p1772675735191709