--- name: alibabacloud-elasticsearch-instance-diagnose description: | Alibaba Cloud Elasticsearch instance diagnosis skill. Use for cluster health checks, troubleshooting, and performance analysis on Elasticsearch instances. Triggers (English): Elasticsearch diagnosis, ES instance issues, slow search, write failures, cluster Red/Yellow, unassigned shards, node disconnected, load imbalance, thread pool 429, JVM/OOM/circuit breaker, disk watermark / read-only index, instance activating / change stuck, service avalanche / all shards failed. 触发词（中文）: ES诊断、阿里云ES、Elasticsearch诊断、ES集群/实例故障排查、ES健康检查、集群红灯/变红/黄灯/变黄、集群异常、分片未分配、主分片未分配、节点掉线/离线、负载不均衡、搜索/查询变慢、慢查询、写入失败/变慢/拒绝、线程池打满、HTTP 429、内存过高、OOM、断路器、磁盘满/水位、索引只读、实例激活中/activating、变更卡住/未完成、雪崩、服务不可用、all shards failed。 --- # Alibaba Cloud Elasticsearch Instance Diagnosis Collect signals from **Alibaba Cloud OpenAPI (control plane)** and the **Elasticsearch REST API (data plane)**, combine them with the SOP knowledge base under `references/`, and produce **root-cause analysis**, an **evidence chain**, **prioritized remediation guidance**, and—when multiple dimensions fire—a **recency-ordered incident timeline** (severity vs time in window; see **Timeline and recency (MUST)** in §5 Step 4). **Architecture**: Alibaba Cloud Elasticsearch OpenAPI + Alibaba CloudMonitor (CMS) + Elasticsearch REST API + diagnostic SOPs **Closure**: If MUST applies and `ES_*` is set, finish authenticated ES API evidence before the final report (see **Feasibility order** in §5). --- ## 1. Prerequisites ### 1.1 Aliyun CLI > **Pre-check: Aliyun CLI >= 3.3.1 required** (for RAM permission checks and OpenAPI CLI fallback) > Run `aliyun version` to verify the version is >= 3.3.1. If the CLI is missing or too old, see `references/cli-installation-guide.md`. > After installation, run `aliyun configure set --auto-plugin-install true` to enable automatic plugin installation (**do not** pass plaintext AccessKey pairs on this command line; see §1.2). ### 1.2 Alibaba Cloud account authentication and security (MUST) > **Security rules (mandatory):** > - **NEVER** read, echo, or print AccessKey ID or AccessKey Secret values. > - **NEVER** prompt or ask the user to paste plaintext AccessKeys in the conversation. > - **NEVER** embed AccessKeys in scripts, CLI arguments, or `curl` URLs. > - **NEVER** use `aliyun configure set` (or similar) to pass **literal** AccessKey ID/Secret on the command line. > - **NEVER** accept AccessKeys that the user pastes into the chat, even if offered voluntarily. > - **ONLY** use configured CLI profiles (`aliyun configure`) or environment variables such as `ALIBABA_CLOUD_ACCESS_KEY_ID` / `ALIBABA_CLOUD_ACCESS_KEY_SECRET` that the user has set **in their local shell** (the agent **must not** echo those values in the session). > **⚠️ If the user provides AccessKeys in the chat (e.g. “my AK is xxx”)** > > 1. **Stop immediately**: do not run any Alibaba Cloud command that requires credentials. > 2. **Decline politely** and give **only** the names of approved configuration methods (**do not** repeat any secret the user may have leaked): > - Recommended: run `aliyun configure` in a local terminal and enter credentials when prompted; credentials are stored in the local profile file. > - Alternatively: set `ALIBABA_CLOUD_ACCESS_KEY_ID` / `ALIBABA_CLOUD_ACCESS_KEY_SECRET` in the local shell (the user types values **only in the terminal**, not in chat). > 3. Resume the diagnosis request only after credentials are configured correctly. > **Verify credentials without exposing secrets:** > > ```bash > aliyun configure list > aliyun --profile sts get-caller-identity > ``` > > **Credential policy:** > 1. Prefer an `aliyun configure` profile (default or `--profile`). > 2. If there is no valid identity (`configure list` / `get-caller-identity` fails), **STOP** and guide the user to configure locally; **do not** guess or fabricate credentials. > 3. Never pass plaintext AccessKeys through the conversation. ### 1.3 Elasticsearch direct-connect credential boundary > - **NEVER** ask the user to paste `ES_PASSWORD` in chat; **NEVER** echo, print, or log the password; **NEVER** copy a password from chat into commands, hooks, or repo files. > - Shell expansion for `curl -u "$ES_USERNAME:$ES_PASSWORD"` (or equivalent) is **allowed** when vars are **pre-exported in the user’s local shell**; **NEVER** put the secret as a literal in chat, scripts checked into repos, or command output. > - If the user tries to send a password in chat: **STOP** as well and ask them to set `ES_PASSWORD` only locally via `export` (see §2.2). --- ## 2. Environment setup ### 2.1 Control plane OpenAPI (via Aliyun CLI) All control-plane and CMS data collection for this skill uses the **Aliyun CLI**. **User-Agent (required)**: set a User-Agent for Alibaba Cloud API calls: ```bash export ALIBABA_CLOUD_USER_AGENT="AlibabaCloud-Agent-Skills" ``` **CLI hardening (recommended)**: when authoring raw `aliyun` commands, add **`--connect-timeout 3 --read-timeout 10`** (increase `read-timeout` for large responses or CMS), consistent with the instance-management skill examples, to avoid indefinite hangs on network faults. If the global User-Agent is not set, add **`--user-agent AlibabaCloud-Agent-Skills`** per invocation. For **optional Elasticsearch probes** inside `check_es_instance_health.py` (when `ES_*` is set), the same knobs exist as **`--connect-timeout`** / **`--read-timeout`** on that script — they map to `curl` for engine calls only, not to the Aliyun OpenAPI client. Run before diagnosis: ```bash aliyun version aliyun configure list aliyun --profile sts get-caller-identity ``` ### 2.2 Elasticsearch API direct access (`curl`) Have the user set connection variables in a **local terminal** after you confirm the Elasticsearch endpoint (VPC or public) and admin credentials—**do not** hardcode user-specific values in chat: ```bash export ES_ENDPOINT="http://:9200" export ES_USERNAME="elastic" export ES_PASSWORD="" ``` > **Public access and `http` vs `https`:** From **`DescribeInstance`**, use **`publicDomain`** / **`domain`** and the reported **`protocol`**. When **`protocol` is `HTTP`** (typical public listener), set **`ES_ENDPOINT` to `http://:9200`**. Using **`https://`** against an **HTTP-only** endpoint causes **TLS** errors (e.g. **`WRONG_VERSION_NUMBER`**). Use **`https://`** only when **`protocol` is `HTTPS`** (or TLS is actually enabled on the port you use), and supply CA / fingerprint options as in **HTTPS options** below. > > **If `http://` “does not work” — when to try `https://`:** Treat **`DescribeInstance` `protocol`** as the **source of truth** for the **REST listener**. **`000`**, **timeouts**, or **connection refused** on **`http://`** usually mean **network path / allowlist / security group / wrong host or port** — **not** “try HTTPS next” when **`protocol` is still `HTTP`**. **Do** switch to **`https://`** when **`protocol` is `HTTPS`** (or the console / product doc states TLS on that endpoint) and the failure on `http://` is a **TLS or scheme** symptom (e.g. **`WRONG_VERSION_NUMBER`**, **`error:0A00010B`**, immediate SSL alert while probing with the wrong scheme). If **`protocol` is `HTTP`** and only **plain TCP** is advertised, **HTTPS is not a fallback** for reachability. > **Credential safety** > - **NEVER** echo, print, or log `ES_PASSWORD`; **NEVER** copy credentials from chat into shell history or saved files. > - **NEVER** ask the user to paste the password in plaintext in chat. > - **ONLY** use the following checks to verify that variables are set: > ```bash > [[ -n "$ES_ENDPOINT" ]] && echo "ES_ENDPOINT: $ES_ENDPOINT" || echo "ES_ENDPOINT: NOT SET" > [[ -n "$ES_PASSWORD" ]] && echo "ES_PASSWORD: SET" || echo "ES_PASSWORD: NOT SET" > ``` > **Network connectivity and access control** > > | Issue | How to check | Mitigation | > |------|--------------|------------| > | Public network access disabled | Elasticsearch console → **Network** | Enable public access or use the VPC endpoint | > | Public access allowlist | Console → **Security** → **Public access allowlist** | Add the agent host’s public IP | > | VPC isolation | e.g. `telnet 9200` | VPC peering, Express Connect, or equivalent | > | Security group | Inbound rules on the ECS/security group hosting Elasticsearch | Allow TCP **9200** (or the configured port) | > **Connectivity probe**: `curl -sS -o /dev/null -w "%{http_code}" --connect-timeout 5 "${ES_ENDPOINT}"` — HTTP code `000` usually means the path is unreachable. **`401` without `-u` is normal** (auth required); if `ES_PASSWORD` is SET, proceed to **authenticated** `GET /_cluster/health` (§7). **`401` with `-u`** → wrong credentials. **`000` / refused / timeout** → network, allowlist, or TLS/scheme mismatch. > **HTTPS — prerequisites (what must be true)** > 1. **Listener:** The Elasticsearch HTTP port you call (**9200** unless changed) must actually speak **TLS** — align with **`DescribeInstance` `protocol`** (**`HTTPS`**) or console/network documentation. > 2. **URL:** **`https://:`** with the same **host** (e.g. **`publicDomain`**) you would use for HTTP. > 3. **Client trust of the server certificate:** Your client must trust the cluster’s certificate chain (cluster / cloud **CA PEM**, or corporate proxy CA if TLS is intercepted). **`curl`**: prefer **`curl --cacert /path/to/ca.crt ...`**; **`-k` / `--insecure`** only for **short, non-production** diagnosis. > 4. **Auth:** Same **`ES_USERNAME` / `ES_PASSWORD`** as for HTTP (Basic auth over TLS). > > **HTTPS — how this skill documents it** > - **Manual `curl` (§7 and [es-api-call-failures.md](references/es-api-call-failures.md)):** Add **`--cacert`** (or **`-k`** for testing) to every **`curl`** when using **`https://`** if the default trust store does not include your cluster CA. > - **`check_es_instance_health.py` optional ES probes:** They invoke **`curl`** with **`-u`** only; they **do not** read **`ES_CA_CERTS`** / **`ES_SSL_FINGERPRINT`** / **`ES_VERIFY_CERTS`** (those names are common for **Python Elasticsearch** clients). For **HTTPS** instances, use **§7 `curl`** with **`--cacert`** for deep checks, or extend the script later to pass **`--cacert`** from an env var. > - **Python-style env vars (reference for other tooling):** `ES_CA_CERTS`, `ES_SSL_FINGERPRINT`, `ES_VERIFY_CERTS=false` (testing only) — **not** wired into this repo’s optional **`curl`** path today. --- ## 3. RAM permission check > **[MUST] RAM permission pre-check** > > Before running this skill, verify the principal has the required RAM permissions. > See `references/ram-policies.md` for the full list. > If the user reports insufficient permissions, direct them to attach the corresponding policies in the RAM console. --- ## 4. Parameter confirmation > **IMPORTANT: Parameter confirmation** > Confirm the following with the user **before** any command or API call. > Do not assume undeclared defaults or hardcode user-specific parameters. > **Boundary controls (MUST)** > - **`region-id` and `instance-id` must not be guessed** or taken from unverified defaults; if they disagree with `DescribeInstance` or the user’s explicit statement, reconfirm. > - **Do not** apply metrics, logs, or `DescribeInstance` conclusions from **instance A** to **instance B**; `ES_ENDPOINT` must match the instance under diagnosis (see **Pre-flight validation for Elasticsearch API** below). > - This skill is **read-only diagnosis**: **do not** invoke mutating control-plane APIs (create, resize, restart, delete instance, etc.). If the user requests a change, provide recommendations only; execution belongs in the console or an approved change workflow. | Parameter | Required | Description | Default | |-----------|----------|-------------|---------| | `instance-id` | Yes | Elasticsearch instance ID, e.g. `es-cn-xxxxx` | - | | `region-id` | Yes | Region ID, e.g. `cn-hangzhou` | - | | `profile` | No | Aliyun CLI profile (explicit `--profile` recommended) | `default` | | `ES_ENDPOINT` | No | Elasticsearch endpoint (direct API access only) | - | | `ES_PASSWORD` | No | Elasticsearch admin password (direct API access only) | - | | `--window` | No | `check_es_instance_health.py`: analysis window in minutes (default **60**) | 60 | | `--connect-timeout`, `--read-timeout` | No | `check_es_instance_health.py`: `curl` timeouts for optional **ES engine** probes when `ES_*` is set (`--connect-timeout` → `curl --connect-timeout`; **`--read-timeout`** contributes to **`curl -m`** together with connect). Defaults **5** / **10** seconds. | 5 / 10 | --- ## 5. End-to-end diagnostic workflow ### Agent hard rules (non-negotiable) > **OpenAPI/CMS cannot replace MUST engine APIs.** For any **§5 MUST** table row or **`check_es_instance_health.py` rule-engine MUST**, Alibaba Cloud OpenAPI and CloudMonitor do **not** replace the listed Elasticsearch REST calls for engine-level root cause—when **feasibility** holds, run those `curl` endpoints (see §7); they are complementary layers, not interchangeable. > > **Feasibility is decided only by checks, not by assumption.** Whether the agent may call Elasticsearch **must** be determined by actually running the **Feasibility order** (§5): at minimum verify `ES_ENDPOINT` / `ES_PASSWORD` per §2.2, align `ES_ENDPOINT` with `DescribeInstance`, then authenticated `GET /_cluster/health`. **Do not** assume `ES_*` is unset or the path is unreachable without performing these steps in the session. For Elasticsearch incidents, follow these **four steps**; each has a distinct role. ### Execution strategy (root-cause driven) > Full policy: [es-api-diagnosis-strategy.md](references/es-api-diagnosis-strategy.md) Data-plane `curl` collection requires **both**: 1. **Feasibility**: `ES_ENDPOINT` and `ES_PASSWORD` are set and the network path works. 2. **Necessity**: root-cause analysis needs data-plane evidence that the control plane or CMS cannot establish alone. > For endpoints **listed** under a fired **MUST** table row **or** **rule-engine MUST**, **necessity** for those calls is **already satisfied** by the trigger—still require **feasibility** (**Feasibility order**). For **optional** engine `curl` **not** in those lists, apply **feasibility** and **necessity** per [es-api-diagnosis-strategy.md](references/es-api-diagnosis-strategy.md). **MUST triggers** (if **any** CMS condition below holds, collect the listed Elasticsearch evidence): | Trigger | Scenario | Required Elasticsearch evidence | |---------|----------|----------------------------------| | `ClusterStatus` max ≥ Yellow / Red | Cluster health | `allocation/explain`, `_cat/shards` | | `NodeCPUUtilization` max > 80% | CPU overload | `_nodes/hot_threads`, `_tasks` | | `NodeHeapMemoryUtilization` max > 85% | Memory pressure | `_nodes/stats/breaker`, `GET /_cluster/settings?include_defaults=true` ( **`indices.breaker.*`** in transient / persistent ) | | Thread pool `rejected` > 0 | Performance | `_nodes/hot_threads`, `_nodes/stats/thread_pool` | | Inter-node resource CV > 0.3 | Load imbalance | `_cat/shards`, `_cat/allocation` | | Write failures or index read-only | Disk / watermark / blocks | `_cluster/settings`, `_all/_settings?filter_path=*.settings.index.blocks`, `_cat/allocation` | | Intermittent Elasticsearch API timeouts + CMS CPU > 80% | Possible cascading failure | `_nodes/hot_threads`, `_nodes/stats/thread_pool`, `_tasks` | > **Thread-pool row:** interpret **search** vs **write** / **bulk** using [sop-query-thread-pool.md](references/sop-query-thread-pool.md) vs [sop-write-performance.md](references/sop-write-performance.md) (see also **Write-path / bulk saturation** below). > **Rule-engine MUST:** If `check_es_instance_health.py` prints a **§5 MUST / §5–§7** callout for this run, treat it like a row above—collect that listed ES evidence when feasibility holds. > **Binding rule (MUST triggers):** If **any** MUST-trigger row **or** the **rule-engine MUST** line above applies, **necessity is satisfied** for that evidence set—OpenAPI/CMS cannot replace those calls for engine-level root cause (cluster-health: `allocation/explain` + `_cat/shards` for Yellow/Red). Confirm **feasibility** per **Feasibility order** below. If reachable with auth, **run the MUST-listed endpoints in Step 2** in parallel with control-plane collection. If still blocked **after authenticated** `GET /_cluster/health`, lead with **blocking reason**: unset `ES_*`; transport failure (`000`, refused, timeout); **401 with `-u`**; scheme/TLS mismatch—not **401 on an unauthenticated probe** when `ES_PASSWORD` is SET. ### Write-path / bulk saturation > If **`ThreadPool.WriteRejected`** or **`write`** pool stress matches **high-QPS bulk** indexing, read and follow **`references/sop-write-performance.md` — §2**, subsection **“Evidence interpretation: bulk QPS → write pool”** for the evidence chain, **`rejected`** semantics (cumulative since node start), **report ordering vs Old GC / heap** (causal chain or dual P0 — write path before JVM-only headline), **per-node `rejected`/`completed` numbers** (reject share), per-node asymmetry, and write-only vs search. **Do not** lead with a JVM-only narrative when that subsection applies. For **write-queue–style** acceptance prompts, the **opening conclusion** should read as **write-capacity** (data-plane counters + optional CMS rule names), not **only** a GC/heap headline. ### Search-primary vs write (both pools show cumulative `rejected`) > When **`_nodes/stats/thread_pool`** shows **`search.rejected` ≫ `write.rejected`** on the same node(s) and **`ThreadPool.SearchRejected`** / **query-driven** overload applies, **lead** the **executive summary** and **P0 ordering** with **`search`** (high concurrent query / terms / slow query; hot index when verified) — **not** **`write`** first. **`write.rejected`** may remain **P0/P1** as **parallel** or **secondary** (bulk, catch-up); **Old GC / CPU / node disconnect** stay **co-stress or cascade**. Checker **listing order** is not proof of narrative order — see [acceptance-criteria.md](references/acceptance-criteria.md) **§6.5** and [sop-query-thread-pool.md](references/sop-query-thread-pool.md) *Report narrative*. > > **Recency overrides this magnitude default** when **time-resolved** evidence exists: **do not** rank the opening story by **`search.rejected` vs `write.rejected` alone** — cumulative counters lack timestamps. Full rubric: [acceptance-criteria.md](references/acceptance-criteria.md) **§6.5** (*P0 / executive order vs `search` ≫ `write`*: **unless** write dominated **by time**) and **§6.6** (*Executive order*, *No false recency from counters*). **Binding:** **Timeline and recency (MUST)** below (same skill). ### `activating` / change workflow stuck (cross-layer root cause) > When an instance stays in **`activating`**, a change is unfinished, and **Red** or unassigned shards coexist, follow **`references/sop-activating-change-stuck.md`** end-to-end (**MUST** includes `ListActionRecords`, `DescribeInstance` before/after remediation, collection order **section 3.1**, reporting **section 4**). ### Pre-flight validation for Elasticsearch API > **[IMPORTANT] `ES_ENDPOINT` must match the diagnosed instance** > > Compare `publicDomain` / `domain` and **`protocol`** from `DescribeInstance` with `ES_ENDPOINT`. > If they differ, warn: `⚠️ ES_ENDPOINT does not match the current instance; run export ES_ENDPOINT="http://{publicDomain}:9200"` when **`protocol` is `HTTP`**, or `https://…` only when **`protocol` is `HTTPS`** (adjust host/port to match the deployment). ### When Elasticsearch credentials are missing or connections fail > **[CRITICAL]** Guide the user to fix connectivity explicitly; **classify** failure modes (do not default persistent timeouts to “allowlist only”). **Do not imply the agent “forgot” Elasticsearch** — if the first answer is CMS/OpenAPI-heavy, give the **blocking reason** per **Feasibility order** below: unset `ES_*`; transport errors; **401 with valid `-u`**; TLS/scheme—not **401** on a probe **without** `-u` when `ES_PASSWORD` is SET (use authenticated `curl` first). **Progressive playbook (read in order):** [references/es-api-call-failures.md](references/es-api-call-failures.md) (sections **1 → 4**). **MUST / strategy context:** [references/es-api-diagnosis-strategy.md](references/es-api-diagnosis-strategy.md) (sections 1–3 and **3.5** summary table). ### Mandatory warning when MUST applies but Elasticsearch is not configured > **[CRITICAL] If a MUST trigger fires but data-plane evidence is missing, put a warning at the top of the report:** follow **section 4** of [references/es-api-call-failures.md](references/es-api-call-failures.md) (blocking reason first, then MUST list, missing evidence; if `ES_*` unset, pointer to **section 2.2** of this SKILL; if vars are set, use es-api-call-failures **sections 1–2** for auth vs transport). ### Step 1: Quick health scan (initial signals) Run the lightweight rules engine (17 metric rules) to list P0 / P1 / P2 findings and steer deeper collection: ```bash python3 scripts/check_es_instance_health.py -i -r [--window ] [--profile ] ``` ### Feasibility order (agent) 1. **Run** §2.2 `ES_*` checks (password = SET only)—**do not skip**; never infer feasibility without this step. 2. `ES_ENDPOINT` matches `DescribeInstance` `domain` / `publicDomain` (scheme/port). 3. **Authenticated** `GET /_cluster/health`—do not stop at **401** on an unauthenticated probe if `ES_PASSWORD` is SET. 4. MUST scope: **table rows** and/or **rule-engine MUST** line in §5. ### Step 2: Collect evidence in parallel Based on Step 1, run collection in parallel (prioritize dimensions with signals). If a **MUST-trigger** row **or rule-engine MUST** applies: run **Feasibility order**, then **run that Required Elasticsearch evidence** via `curl` in the **same** round (see §7). If **no** MUST applies, add optional data-plane `curl` only when **feasibility** and **necessity** both hold per the strategy doc. Re-run **`check_es_instance_health.py`** with the same invocation pattern as Step 1; for this parallel round, **`--window 120`** and explicit **`--profile `** are common. To backfill control-plane evidence (`DescribeInstance`, `ListSearchLog`, CMS-style calls), use **`aliyun`** patterns in [references/verification-method.md](references/verification-method.md) (epoch times, profiles, namespaces). > Note: data-plane access still requires `ES_ENDPOINT` / `ES_PASSWORD`; the Aliyun CLI **cannot** replace `curl` to the cluster. > > For **MUST-trigger** rows, necessity for the **listed** endpoints is **already** established—do **not** skip them when feasibility including reachability holds. **Outside** those rows, avoid unrelated bulk `curl` solely because `ES_*` is set; use the strategy doc’s feasibility + necessity test instead. ### Step 3: Read SOPs by signal Map signals to SOPs and read for deeper reasoning. With multiple signals, process **P0 → P1 → P2** for **severity**, then apply **Timeline and recency (MUST)** in Step 4 so the **narrative order** matches **when** signals mattered in the window—not only static rule-engine print order. | Observed signal | Read | |-----------------|------| | Cluster Red/Yellow, node loss, pending tasks | `references/sop-cluster-health.md` | | Long `activating`, unfinished change records, Red / unassigned shards | `references/sop-cluster-health.md` + `references/sop-activating-change-stuck.md` | | High CPU, load, imbalance | `references/sop-cpu-load.md` | | Per-node load imbalance (CPU/memory/disk/shard count) | `references/sop-node-load-imbalance.md` | | JVM pressure, GC, circuit breaker, OOM | `references/sop-memory-gc.md` | | Disk watermark, IO, write failures (read-only) | `references/sop-disk-storage.md` | | Watermark misconfiguration, index blocks, “normal” disk % but write failures | `references/sop-disk-storage.md` (Section 3 — watermark misconfiguration) | | Write timeouts / rejections / latency / QPS drop | `references/sop-write-performance.md` | | Query timeouts / rejections / slow queries | `references/sop-query-thread-pool.md` | | Nodes look down but CPU still reported; `all shards failed` | `references/sop-service-avalanche.md` | | Intermittent Elasticsearch timeouts + CMS CPU > 80% | `references/sop-service-avalanche.md` | | Risky settings, Ngram issues, API anomalies | `references/sop-configuration.md` | | Event code definitions | `references/health-events-catalog.md` | ### Step 4: Synthesize and write the structured report > **Acceptance-style optional checklists:** [references/acceptance-criteria.md](references/acceptance-criteria.md) **§6.1**–**§6.6** — Red/Yellow; read-heavy CPU + `search` pool (+ CMS alignment); JVM / breakers / fielddata; write-queue vs GC + **`rejected`/`completed`**; read-heavy **search pool vs GC-only** headline (expand in [sop-query-thread-pool.md](references/sop-query-thread-pool.md) *Report narrative: search pool vs GC / CPU headlines*); timeline/recency. **Bulk/write:** [references/sop-write-performance.md](references/sop-write-performance.md) §2. **Shard `reroute`:** [references/sop-node-load-imbalance.md](references/sop-node-load-imbalance.md) §1.3 (allocator / change control only). > **[CRITICAL] Remediation must match the diagnosed root cause** — avoid generic templates. Wrong breaker or concurrency fixes (e.g. `in_flight_requests` vs `request`, “split query” when concurrency is the issue) → see **`sop-memory-gc.md`** and the fired signal’s SOP. > **`activating` + data-plane anomaly**: include the **one-line cross-layer root cause**; see `references/sop-activating-change-stuck.md` **section 4**. **Report skeleton (copy/fill):** [references/report-template.md](references/report-template.md). ### Timeline and recency (MUST for synthesized reports) > **Problem:** `check_es_instance_health.py` and P0/P1/P2 bands express **severity**, not **when** a signal mattered most within the analysis window. **Cumulative** engine counters (`search.rejected`, `write.rejected`) do **not** encode recency—**write** and **search** issues can both be “real” while **only one path dominated the recent past** (e.g. search pressure **closer to window end** than write pressure). **Binding rules for the agent:** 1. **Two axes** — Treat **severity** (P0/P1/P2) and **temporal relevance** (proximity to window end / “now”) as **orthogonal**. Do **not** infer recency from priority alone (e.g. “write is P0 so it must be the current headline”) when time-resolved evidence says otherwise. 2. **Mandatory human-facing section** — When more than one major finding fires (e.g. **write pool + search pool + GC/CPU**), the synthesized report **must** include an **`### Incident timeline (recency-ordered)`** (or equivalent) block **before or immediately after** the executive summary, unless the user explicitly asks for a minimal report. In that block: - Order **bullets or rows by time** (earlier → later), or state **which signal cluster peaked / persisted in the **latter** portion of `{begin} ~ {end}`**. - Call out **divergence**: e.g. “write-path stress **earlier** in window; search-path / CPU **more recent**” when CMS or logs support it. 3. **Evidence for recency** (use what exists; do not invent timestamps): - **CloudMonitor**: per-metric time series — note **peak timestamp** or **sustained-high interval** for `NodeCPUUtilization`, `NodeHeapMemoryUtilization`, GC-related metrics, `ThreadPool.*` **if** exposed as rates or non-cumulative series in the collected JSON. - **Slow logs / `ListSearchLog`**: correlate **query vs index** slow entries to minutes. - **Engine (optional):** two `_nodes/stats/thread_pool` samples at **known times** to show **delta** on `rejected` / `completed`; or **`_tasks`** / **`hot_threads`** for **current** skew vs historical cumulative counters. 4. **Executive summary ordering** — The **opening 2–4 sentences** should reflect **recency-weighted** user impact: if **search** pressure is **closer to current** than **write** pressure, **lead** with search/query concurrency and co-stress (GC/CPU) **as appropriate**, and place **historical write saturation** as **context** or **second wave**—**without** dropping P0 write findings if they remain valid for remediation backlog. 5. **Explicit uncertainty** — If only cumulative counters exist and **no** time series differentiates paths, state **one line**: recency is **undifferentiated**; recommend **narrower window**, **slow logs**, or **delta sampling** for the next run. --- ## 6. Data collection details (CLI OpenAPI + injected input) ### One-shot entry Use the same **`check_es_instance_health.py`** command as **§5 Step 1** (optional **`--window`** / **`--profile`**; default window **60** minutes if omitted). ### Injected input mode (paired with CLI) `check_es_instance_health.py` accepts external JSON to avoid duplicate calls: ```bash python3 scripts/check_es_instance_health.py \ -i -r \ --data-source input \ --input-json-file /path/to/diag-input.json ``` Input JSON shape: ```json { "status_info": {}, "metrics": {}, "events": [], "logs": [] } ``` `--data-source` modes: - `auto`: prefer injected fields; backfill gaps via Aliyun CLI. - `cli`: ignore injection; fetch everything via CLI. - `input`: injection only; no OpenAPI calls. ### Manual control-plane CLI backfill For additional OpenAPI examples, see `references/verification-method.md`. --- ## 7. Elasticsearch direct API access (data-plane deep dive) When **feasibility** holds (including reachability), execute the REST calls **required by any MUST-trigger row** (§5). For endpoints **not** listed in a fired MUST row, call them only when **feasibility** and **necessity** both hold per the strategy doc. > `ES_ENDPOINT` may be `host:port` or a full URL. For the samples below, normalize to `http://${ES_ENDPOINT#http://}` (use `https://` consistently when the cluster serves TLS). > > **Timeouts**: every `curl` must use `--connect-timeout 10 --max-time 30`. ### Red / Yellow (MUST) — recommended set **Scope:** The cluster-health MUST row uses `ClusterStatus` max ≥ **Yellow** (includes **Red**). Use this set for **unassigned / misallocated shard** root cause on the engine. ```bash curl -sS --connect-timeout 10 --max-time 30 -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ "http://${ES_ENDPOINT#http://}/_cluster/health?pretty" curl -sS --connect-timeout 10 --max-time 30 -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ -H "Content-Type: application/json" \ -X POST "http://${ES_ENDPOINT#http://}/_cluster/allocation/explain?pretty" \ -d '{}' curl -sS --connect-timeout 10 --max-time 30 -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ "http://${ES_ENDPOINT#http://}/_cat/shards?v&h=index,shard,prirep,state,node,unassigned.reason&s=state" curl -sS --connect-timeout 10 --max-time 30 -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ "http://${ES_ENDPOINT#http://}/_cluster/pending_tasks?pretty" curl -sS --connect-timeout 10 --max-time 30 -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ "http://${ES_ENDPOINT#http://}/_nodes/stats/thread_pool?pretty" ``` ### Query / write performance (MUST) — recommended set > Include **`_cluster/settings`** when **heap / GC / breaker** rules fired in Step 1 or **`_nodes/stats/breaker`** shows concern — read **transient** and **persistent** `indices.breaker.*` / `network.breaker.*`. ```bash curl -sS --connect-timeout 10 --max-time 30 -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ "http://${ES_ENDPOINT#http://}/_nodes/hot_threads?threads=3" curl -sS --connect-timeout 10 --max-time 30 -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ "http://${ES_ENDPOINT#http://}/_nodes/stats/breaker?pretty" curl -sS --connect-timeout 10 --max-time 30 -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ "http://${ES_ENDPOINT#http://}/_cluster/settings?include_defaults=true&pretty" ``` > **`/_cluster/pending_tasks`** and **`GET /_nodes/stats/thread_pool`** are also listed under **Red / Yellow (MUST)** above—one call each per session when both sections apply. If you run **only** this performance block, add those two `curl` lines from that block. ### Resource anomalies without a closed loop (SHOULD) — recommended set ```bash curl -sS --connect-timeout 10 --max-time 30 -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ "http://${ES_ENDPOINT#http://}/_cat/nodes?v&s=cpu:desc&h=name,ip,cpu,heap.percent,ram.percent,load_1m" curl -sS --connect-timeout 10 --max-time 30 -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ "http://${ES_ENDPOINT#http://}/_nodes/stats/jvm?pretty" curl -sS --connect-timeout 10 --max-time 30 -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ "http://${ES_ENDPOINT#http://}/_cat/allocation?v&bytes=gb" ``` > **`GET /_cluster/settings?include_defaults=true`** also appears under **Query / write performance (MUST)** above—reuse one response when both blocks apply. If you run **only** this SHOULD block, add the same `curl` line from that block. **Protocol sanity** (avoid `WRONG_VERSION_NUMBER`): usually **http/https scheme mismatch** on `ES_ENDPOINT` — fix scheme/port and retry. **Scenario → endpoint** index: [references/es-api-catalog.md](references/es-api-catalog.md). --- ## 8. Diagnostic coverage The knowledge base covers **48+** health-event-style rules and chained scenarios (e.g. disk pressure → allocation → Red). **Per-category counts, P0/P1/P2 mix, and event codes:** [references/health-events-catalog.md](references/health-events-catalog.md) — scenario runbooks: `references/sop-*.md` (index: [references/README.md](references/README.md)). --- ## 9. Best practices **Read-only:** no mutating control-plane APIs; no teardown. 1. **Layered + evidence-bound**: scan → SOP depth; every conclusion cites metrics/logs/events; if ES is unreachable, state limits ([es-api-call-failures.md](references/es-api-call-failures.md)). 2. **Priority vs narrative**: **P0→P2** for urgency; **Incident timeline** when multiple dimensions differ in time (Step 4). **Credentials / TLS / parameters:** §1–2 and §4. 3. **Green is not “all clear”** — watermarks, blocks, mis-set limits still matter; **MUST + reachable ES:** do not skip §5/§7 evidence because the cluster is Green or OpenAPI “explains” symptoms. 4. **Thread-pool `rejected`:** **cumulative** unless you show a delta — [sop-query-thread-pool.md](references/sop-query-thread-pool.md) §1–2; write/bulk: [sop-write-performance.md](references/sop-write-performance.md) §2. --- ## 10. Reference links - `references/verification-method.md` — Verification (how to validate diagnosis; metrics, APIs, workflows) - `references/report-template.md` — Structured diagnosis report skeleton - `references/README.md` — **Language map** (reference assets and `sop-*.md` runbooks; English in this repo) - `references/ram-policies.md` — RAM policy list - `references/acceptance-criteria.md` — Correct/incorrect patterns and acceptance (includes credential and safety anti-patterns) - `references/cli-installation-guide.md` — Aliyun CLI installation - `references/es-api-catalog.md` — Elasticsearch REST API catalog - `references/health-events-catalog.md` — Health event catalog - `references/sop-*.md` — Scenario SOPs (e.g. `sop-activating-change-stuck.md` for `activating` / change stuck, cross-layer root cause) - `references/es-api-diagnosis-strategy.md` — Elasticsearch API diagnosis strategy FILE:references/README.md # References (`references/`) Operational knowledge for the **Alibaba Cloud Elasticsearch instance diagnosis** skill: APIs, verification, event catalog, acceptance rules, and scenario SOPs (`sop-*.md`). **Discoverability:** skill **trigger keywords** (including **中文** for Chinese-speaking users) live in repo root **`SKILL.md`** frontmatter and **`metadata.yaml`** (`triggers`). Markdown here stays **English**; triggers bridge user language to the skill. --- ## Index ### APIs and access | File | Purpose | |------|---------| | [related-apis.md](related-apis.md) | Related Alibaba Cloud OpenAPIs for diagnosis | | [es-api-catalog.md](es-api-catalog.md) | Elasticsearch REST endpoints used in workflows | | [ram-policies.md](ram-policies.md) | RAM policy snippets for CLI / OpenAPI | ### Verification, strategy, and quality bar | File | Purpose | |------|---------| | [verification-method.md](verification-method.md) | How to verify diagnosis (metrics, logs, APIs) | | [report-template.md](report-template.md) | Structured diagnosis report skeleton (Markdown) | | [es-api-diagnosis-strategy.md](es-api-diagnosis-strategy.md) | When to call which ES API; CMS vs engine; MUST summary | | [es-api-call-failures.md](es-api-call-failures.md) | **Progressive:** `curl` failures (401, timeouts, refused), evidence boundary, report checklist | | [acceptance-criteria.md](acceptance-criteria.md) | PASS / PARTIAL / expectations; **§6.1** Red/Yellow + **ClusterStatus** one voice (CMS window vs engine snapshot) + **CMS `ClusterShardCount` swings** (cross-check engine + ops, avoid “half shards lost”); **§6.2** read-heavy CPU + search pool + CPU table vs log hotspot + **slow-log node vs search-pool node** (routing / phase / time); **§6.3** JVM / breakers / fielddata; **§6.4** write-queue narrative (dual P0 or causal lead + `rejected`/`completed` quant) / GC / bulk / `tripped` vs current heap; **§6.5** search vs GC headline; **`search.rejected` ≫ `write.rejected`** → search-first P0/executive order; **§6.6** timeline / recency-weighted executive lead (search vs write by time) | | [health-events-catalog.md](health-events-catalog.md) | CMS-style events ↔ skill findings | ### Tooling | File | Purpose | |------|---------| | [cli-installation-guide.md](cli-installation-guide.md) | Aliyun CLI install and configure | ### Scenario SOPs (`sop-*.md`) | File | Typical signals | |------|-----------------| | [sop-activating-change-stuck.md](sop-activating-change-stuck.md) | Long `activating`, change records + engine Red | | [sop-cluster-health.md](sop-cluster-health.md) | Red/Yellow, node loss, pending tasks, master election | | [sop-configuration.md](sop-configuration.md) | Risky cluster/index settings, API error rate, slow recovery | | [sop-cpu-load.md](sop-cpu-load.md) | Sustained or peak CPU, load imbalance | | [sop-disk-storage.md](sop-disk-storage.md) | Disk watermarks, IO bottleneck, read-only / flood | | [sop-memory-gc.md](sop-memory-gc.md) | Heap pressure, GC, circuit breakers, OOM | | [sop-node-load-imbalance.md](sop-node-load-imbalance.md) | CPU / traffic / data skew (CV-style imbalance) | | [sop-query-thread-pool.md](sop-query-thread-pool.md) | Search rejects, queue, slow queries | | [sop-service-avalanche.md](sop-service-avalanche.md) | “Node down” but process up; `all shards failed` + high CPU | | [sop-write-performance.md](sop-write-performance.md) | Write rejects, ingest latency, indexing dropped | --- ## For agents 1. Follow the workflow in the repo root **[`SKILL.md`](../SKILL.md)** (especially the signal → SOP routing table). 2. Prefer **`SKILL.md`** over ad-hoc reading order; open SOPs only when the observed signal matches. 3. Report text and evidence keys in **`scripts/check_es_instance_health.py`** are **English** — keep doc examples aligned (e.g. `cluster_status_latest`, `affected_nodes`, `configured_watermark_low`). --- ## Contributing Edits to references should stay consistent with **`SKILL.md`** and the checker script’s field names. Prefer small, accurate updates over duplicating long procedures that already live in a SOP. FILE:references/acceptance-criteria.md # Acceptance criteria: alibabacloud-elasticsearch-instance-diagnosis **Scenario**: Elasticsearch instance diagnosis **Purpose**: Skill test / acceptance checklist --- > **Version notes (2026-03)** > - Control-plane OpenAPI collection uses the `aliyun` CLI only. > - Health diagnosis entry point: `python3 scripts/check_es_instance_health.py ...`. > - Engine-level deep collection uses `curl` against ES REST APIs (no `invoke_es_api.py`). ## 1. Environment dependencies ### 1.1 CLI dependency #### ✅ CORRECT ```bash aliyun version ``` #### ❌ INCORRECT ```bash # Error: aliyun CLI not available aliyun: command not found ``` --- ## 2. Credential configuration ### 2.1 OpenAPI credentials (CLI profile) #### ✅ CORRECT ```bash aliyun configure list aliyun --profile sts get-caller-identity ``` #### ❌ INCORRECT ```bash # Error: profile does not exist aliyun --profile not-exist sts get-caller-identity ``` ### 2.2 Direct ES credentials #### ✅ CORRECT ```bash [[ -n "$ES_ENDPOINT" ]] && echo "ES_ENDPOINT: SET" || echo "ES_ENDPOINT: NOT SET" [[ -n "$ES_PASSWORD" ]] && echo "ES_PASSWORD: SET" || echo "ES_PASSWORD: NOT SET" ``` #### ❌ INCORRECT ```bash # Error: ES password not configured unset ES_PASSWORD ``` --- ## 3. Running the diagnosis ### 3.1 Main health-check entry #### ✅ CORRECT ```bash python3 scripts/check_es_instance_health.py \ -i es-cn-xxxxx -r cn-hangzhou \ --data-source cli \ --profile ``` **Expected output**: - Structured report (P0/P1/P2, evidence, remediation) - When multiple major dimensions fire, an **incident timeline (recency-ordered)** section aligning narrative with **when** signals peaked in the window (see §6.6) - Summary of key monitoring metrics - No dependency on deprecated scripts ### 3.2 Injected-input + auto backfill #### ✅ CORRECT ```bash python3 scripts/check_es_instance_health.py \ -i es-cn-xxxxx -r cn-hangzhou \ --data-source auto \ --input-json-file /path/to/diag-input.json \ --profile ``` **Expected output**: - Input JSON fields take precedence when present - Missing fields are backfilled via CLI OpenAPI --- ## 4. OpenAPI coverage ### 4.1 Elasticsearch OpenAPI #### ✅ CORRECT ```bash aliyun --profile elasticsearch DescribeInstance \ --region cn-hangzhou \ --InstanceId es-cn-xxxxx aliyun --profile elasticsearch ListSearchLog \ --region cn-hangzhou \ --InstanceId es-cn-xxxxx \ --type INSTANCELOG \ --query "*" \ --beginTime \ --endTime aliyun --profile elasticsearch ListActionRecords \ --region cn-hangzhou \ --InstanceId es-cn-xxxxx aliyun --profile elasticsearch ListAllNode \ --region cn-hangzhou \ --InstanceId es-cn-xxxxx ``` ### 4.2 CMS OpenAPI #### ✅ CORRECT ```bash aliyun --profile cms DescribeMetricList \ --region cn-hangzhou \ --Namespace acs_elasticsearch \ --MetricName ClusterStatus \ --Dimensions '[{"clusterId":"es-cn-xxxxx"}]' \ --StartTime \ --EndTime \ --Period 300 aliyun --profile cms DescribeSystemEventAttribute \ --region cn-hangzhou \ --Product elasticsearch \ --SearchKeywords es-cn-xxxxx \ --StartTime \ --EndTime aliyun --profile cms DescribeMetricMetaList \ --region cn-hangzhou \ --Namespace acs_elasticsearch ``` **Expected output**: - Success shape (`Code=200` or `Success=true`) - Key fields present (instance status, logs, metrics, events, nodes, change records) --- ## 5. Engine-level ES API checks (`curl`) If calls fail or return **401** / timeouts, classify the failure using **[es-api-call-failures.md](es-api-call-failures.md)** before judging PASS / PARTIAL. ### 5.1 Calls #### ✅ CORRECT ```bash curl -sS -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ "http://${ES_ENDPOINT#http://}/_cluster/health?pretty" curl -sS -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" -H "Content-Type: application/json" \ -X POST "http://${ES_ENDPOINT#http://}/_cluster/allocation/explain?pretty" -d '{}' curl -sS -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ "http://${ES_ENDPOINT#http://}/_nodes/stats/thread_pool?pretty" ``` #### ❌ INCORRECT ```bash # Error: wrong URL path (missing leading /) curl -sS -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ "http://${ES_ENDPOINT#http://}_cluster/allocation/explain" ``` --- ## 6. Report format ### ✅ CORRECT report shape ```markdown ## Diagnosis summary **Instance**: es-cn-xxxxx (cn-hangzhou) **Window**: 2026-03-24 10:00 ~ 2026-03-24 12:00 ### Incident timeline (recency-ordered) - Optional in minimal reports; **required** when search vs write vs GC show **different** emphasis over time (SKILL Step 4). ### Findings (by priority) #### P0 - Critical (immediate action) - [HealthCheck.ClusterUnhealthy] Cluster status Red - Evidence: ClusterStatus=2, UNASSIGNED primary shards - Likely cause: shard allocation failure - Immediate action: run `POST /_cluster/allocation/explain` ``` ### 6.1 Cluster Red / Yellow — optional “full checklist” (acceptance-style) Use when you want the report to align with **structured acceptance rubrics** (control plane vs data plane already correct). These are **additive**, not a substitute for `allocation/explain` + `sop-cluster-health.md`. | Item | What to add (one line each when data exists) | |------|-----------------------------------------------| | **Red vs Yellow (B5)** | State **`unassigned_primary_shards` > 0** (or equivalent: “at least one **primary** shard is `UNASSIGNED`”) for **Red**; for **Yellow**, primaries assigned but replicas unassigned. | | **Shard arithmetic (B6)** | e.g. `number_of_shards=3`, `number_of_replicas=1` → **3 primary shard copies + 3 replica shard copies** (one **replica shard** per primary), **6** logical copies → all may show `UNASSIGNED` when allocation cannot place any copy. Avoid wording that sounds like “`number_of_replicas` = 3”. | | **`unassigned.reason` (C4)** | From `GET /_cat/shards` / explain: cite reason when present (`ALLOCATION_FAILED`, filter / require deciders, etc.). **Optional (stricter scoring):** mention **`INDEX_CREATED`** when it appears — often consistent with “shard never allocated since creation” under a failing filter (e.g. `require._name` to a non-existent node). | | **Blast radius (B8)** | Name affected indices (from `_cat/shards`); if only one hot index, say other indices were not failing (if `_cat/shards` supports that claim). | | **Scale-out vs `total_shards_per_node` (B9)** | Do not promise Green from “add a few nodes” when a **per-index per-node cap** stays tight — verify **(data nodes × cap) ≥** total shard copies for that index (see `sop-cluster-health.md` §2 Yellow; e.g. **cap=1** + **2p/1r** ⇒ **4** copies ⇒ **3** nodes often still Yellow). | | **Post-fix verification (D3)** | After remediation, suggest re-check: `GET /_cluster/health`, `GET /_cat/shards/{index}?v` (and `allocation/explain` if any shard still unassigned). | | **Ruling out disk / node / CPU** | Prefer: “**In this report’s metrics window and collected evidence**, disk / node loss / CPU are **not** the dominant driver” — not an absolute “unrelated everywhere” claim (other clusters or later windows may differ). | | **ClusterStatus single voice** | If one paragraph cites **CMS `ClusterStatus` max = Yellow** (or Red) for **`{begin} ~ {end}`** and another cites **`GET /_cluster/health` = green** at a **single** probe time, **reconcile in one sentence**: e.g. **worst status in the monitoring window** vs **snapshot after recovery** vs **instantaneous engine API** — do not leave **Green** and **Yellow** side by side without **time / aggregation** qualifier (parallel sections should not contradict). | | **CMS `ClusterShardCount` (or total-shard) swings** | If **CMS** shows **large step changes** in **total shard count** (e.g. **76 → 38 → 76**) within or across windows, **do not** imply **“half the shards were lost”** unless **`GET /_cat/shards`** / engine health **confirms** loss — **reconcile** with **Alibaba Cloud console** change / ops events, **metric definition** (scope: data nodes vs cluster, replication, aggregation lag), and **instance lifecycle**. Prefer wording such as **“pending cross-check with ops records and engine shard view”** when the drop is **not** explained by allocation state. | ### 6.2 Read-heavy CPU + search pool (optional checklist) Optional lines when **query-driven** overload is present (e.g. `_tasks` shows `*search*` on a hot index with `match_all`-style bodies): | Item | What to add | |------|-------------| | **CPU ↔ same workload** | One explicit sentence: **NodeCPU / `hot_threads` and `search` pool queue or `rejected` stem from the same concurrent search load** (name index + task shape when known). | | **Shard counts / layout in prose** | Once **`GET {index}/_settings`** (and templates) and **`_cat/shards`** **confirm** counts and placement, **use those values consistently** in the report — they override assumptions from any other environment. Optionally one line on **provenance** (template, console, legacy index). | | **Hot node vs skew + `rejected`** | Summarize the overloaded side as a **hot node** when **`rejected` / heap / GC** align there. Cite **primary/replica skew** as **one factor** in **local** work and pressure, **plus** **parallel** read-path explanations (coordinator / single client path / `preference=_primary` / routing) — not “primaries necessarily answer most default searches.” See [sop-query-thread-pool.md](sop-query-thread-pool.md) §2. | | **CMS vs thread_pool time alignment** | State whether **CMS CPU** overlaps the **search queue / reject** episode — including **CPU no longer high** while **`rejected` is cumulative** and **`queue` already zero** (post-spike phenomenology). | | **CMS CPU metric hygiene** | If CMS **NodeCPU** looks modest vs strong **`search.rejected`**, add **one reconciliation block**: burst **early in window** vs CMS **whole-window** stat; metric **name + `Dimensions`** (this instance’s **data nodes**, not wrong rollup); optional **`_nodes/stats/os`** vs sample period. See [sop-cpu-load.md](sop-cpu-load.md) decision-tree notes. | | **Per-node CPU % vs search “hot” node** | If a **per-node CPU table** (e.g. **5-minute mean** or **window aggregate**) shows **lower** CPU on the node that **logs / `thread_pool.search` / `EsRejectedExecutionException`** identify as the **search hotspot**, add **one sentence** so readers are not stuck: **spike moment ≠ coarse mean**; **CMS bucket / dimension** may not align with **sub-minute** pool saturation; **engine or INSTANCELOG lines** anchor **which node** ran the saturated **`search` pool** — the table is **imbalance context**, not a disproof of pool rejections on that node. | | **Shard placement phrasing** | Use **consistent** qualifiers — **all** vs **N / N** vs **almost all** — title and tables must match; do not imply exceptions without a shard row to cite. | | **Short / ambiguous index names** | For generic names (e.g. **`stats`**), add **purpose** (built-in / monitoring vs business-owned) or **full index pattern** so readers do not conflate unrelated indices. | | **`rejected` counter** | Call out **cumulative since process start** (unless you show a **delta**); do not imply the full counter equals rejects “only in the report window.” | | **`hot_threads` / logs** | Say whether **`hot_threads` ran successfully** vs **timeout / empty** after spike; for OpenAPI logs, **success vs failure** (e.g. metrics API unavailable). Treat failures as **evidence gaps**, not silent omissions. | | **Slow-log node vs search-pool node in other logs** | **SEARCHSLOW** / **fetch**-phase lines may name **one data node** while **INSTANCELOG** / **thread_pool** lines at **another minute** name a **different node** as **`search` saturated** — both can be **true**: **query routing**, **primary/replica**, **coordinator vs data role**, and **time skew**. Add **one sentence**: each line reflects **that phase / time / shard copy**; ground **routing** with **`GET /_cat/shards/{index}`** (and **which shard** the slow query hit) so the report does **not** read as **self-contradictory**. | | **Wording** | Prefer **co-occurring signals** over a strict single-arrow causal chain unless ordering is evidenced. | | **P0 / P1 band** | Match the **strongest fired rule** in the window (often **P0** CPU and/or **P0** `ThreadPool.Search*` per [health-events-catalog.md](health-events-catalog.md)); avoid a **P1-only** headline when **P0** rules already fired. | ### 6.3 JVM / fielddata / circuit breakers (optional checklist) Use when **heap**, **GC**, **breaker**, or **fielddata** signals appear — cluster may still be **Green**. | Item | What to add | |------|-------------| | **`_cluster/settings` first-class** | **`GET _cluster/settings?include_defaults=true`** — read **transient** and **persistent** for **`indices.breaker.fielddata.limit`**, **`indices.breaker.request.limit`**, and related keys; they are often changed **together**. | | **Headline vs Old GC** | If settings show **very low breaker limits** and/or **`_nodes/stats/breaker`** shows **`tripped` > 0** / logs show **`CircuitBreakingException`**, lead with **breaker + settings + query/mapping** — **Old GC rate** as **parallel** or **secondary**, not the only story. See [sop-memory-gc.md](sop-memory-gc.md) §5. | | **`tripped` vs P2-only wording** | Low **`fielddata.limit`** without trips is **config risk**; **`tripped` > 0** or matching exceptions → **incident-grade** narrative (usually alongside **P0**-class breaker handling in the catalog). | | **Index-level closure** | Name **indices** and **mapping** culprits (`text` + **`fielddata`**, large **`terms` / `cardinality`**, deep paging) when engine evidence exists. | | **Heap skew across nodes** | Correlate uneven **heap** with **`_nodes/stats/breaker`**, **`_nodes/stats/indices/fielddata`**, **`_cat/shards`** — not **only** shard counts. | ### 6.4 Write-path saturation + CPU + JVM (optional checklist) Use when **ingest / bulk**, **`ThreadPool.WriteRejected`**, **CPU peaks**, and/or **Old GC / heap** signals **co-occur** — for example **sustained high-QPS bulk** indexing that stresses the **write** pool. **Write-queue / bulk prompts (rubric alignment):** Graders often treat **`ThreadPool.WriteRejected`** + **`_nodes/stats/thread_pool`** as the **primary** storyline. A conclusion that **only** headlines **Old GC / heap** while relegating **write-pool saturation** to “P1 / participating factor” can read as a **JVM/GC report**, not a **write-capacity** diagnosis. Prefer one of: - **Dual P0 (parallel):** **Opening** treats **quantified write-pool stress** (**`write` `rejected`** **and** **`completed`**, or **reject share** per node or cluster) **and** **Old GC wall-clock / `GCTimeRatioTooHigh`** as **co-equal** P0-class signals (see [health-events-catalog.md](health-events-catalog.md) **`ThreadPool.WriteRejected`** / **`HealthCheck.ThreadPoolSaturation`**). - **Causal chain (upstream first):** **First sentence** states **bulk / ingest saturates the `write` pool** → **merges / heap** → **Old GC + CPU** — so **`write.rejected`** is in the **lead**, not buried below a GC headline. Add **at least one line of numbers**: e.g. **`rejected` / `completed`** on the **busiest node(s)**, optional **reject share** (approximately `rejected / (rejected + completed)`) for the same pool, and **rule / event names** when citing CMS (e.g. **`ThreadPool.WriteRejected`**). Align structured reports with **[health-events-catalog.md](health-events-catalog.md)**: CMS **P0** for write rejects usually requires **sustained reject rate + traffic**, not cumulative counters alone — **human-written** reports should still **quote cumulative `rejected`/`completed` with context** so the **data-plane** story is concrete. **Engine snapshot (`check_es_instance_health.py`):** **`rejected` > 0** is emitted as **P1** when **`JVMMemory.GCTimeRatioTooHigh`** is **not** in the same finding set. When **`GCTimeRatioTooHigh` (P0)** **does** fire, the script **promotes `ThreadPool.WriteRejected` from P1 → P0** so severity bands match a **dual-P0** or **causal-chain** narrative — see table below. **Co-occurrence is not proof of causation**; use **`hot_threads`** / **`_tasks`** to confirm an **ingest-heavy** path before asserting **write overload → GC** as the only story. | Item | What to add | |------|-------------| | **Headline priority (dual P0)** | When **write-path evidence** and **`JVMMemory.GCTimeRatioTooHigh` / Old GC** are **both** material, prefer **two P0 headlines** (or **one causal chain**, not JVM-only): e.g. **write pool saturation / rejects** **and** **Old GC wall-clock share** — not “GC only.” Optional chain: **write overload → merge / heap → Old GC → CPU spikes** (only if evidence supports ordering). | | **GC collector wording** | Do **not** assume **G1** vs **CMS** (or other) from ES/JDK version alone. Prefer **“old-generation / Old GC”** or cite **`GET _nodes/stats/jvm`** (`gc.collectors.*`) / node GC logs — avoids technical challenges from a wrong collector name. | | **Script vs narrative** | If the checker still prints **WriteRejected as P1** (no `GCTimeRatioTooHigh` in the window) but CMS/catalog would grade **P0**, say so in one line — **do not** silently downgrade the user-facing conclusion. | | **`rejected` + `completed` + `queue` / `active`** | Include **`completed`** (same pool, per node or total) next to **`rejected`** to size **reject share**; state **cumulative since node start** for **`rejected`** / **`completed`**; add **current** `queue` / `active` so readers do not confuse **history** with **ongoing** overload. | | **Per-node skew** | When **`rejections_by_node`** is uneven, tie to **hot shards / routing / transient skew** — same direction as **CPU imbalance** checks. | | **Shard count vs `write.rejected` skew** | If **shards per data node** are **even** (e.g. similar counts from **`_cat/shards`** / routing) but **`write.rejected`** is **much higher on one node**, state explicitly: imbalance is **unlikely** from **shard count alone** — favor **hot shards / routing / client targets**; cite **numbers** for both **shards/node** and **per-node `rejected`** when available. | | **Parent breaker `tripped`** | **`tripped`** is **cumulative** (breaker semantics: trips **since JVM start**). A **non-zero** **`parent.tripped`** (e.g. **183**) is **aligned with historical peaks** / past pressure and need **not** contradict **current** `_cat/nodes` heap **~50%** — add **half a sentence** that **`tripped` reflects cumulative history**, not instantaneous heap %. Relate to **historical** heap pressure alongside **`rejected`**; reconcile with **current** heap **in one sentence**. | | **Bulk / client guidance** | Under **reject / capacity-tight** conditions, prefer: **lower parallel bulk streams**, **smaller per-request bytes/doc count**, **backoff / fewer in-flight bulks**; align total throughput to SLO. If mentioning **larger batches**, **immediately** caveat **timeouts / memory spikes / OOM** — avoid implying “bigger single requests” as the fix. See [sop-write-performance.md](sop-write-performance.md) §2 (*bulk QPS → write pool*). | ### 6.5 Read-heavy scenarios (search pool vs GC-only headline) Use when the **incident** is **query-driven** (high concurrent search, terms / QPS patterns, “search reject → cascade” timelines). **Section 6.4** already guards **write-path** vs **GC-only** headlines; **section 6.2** covers **search pool + CPU** alignment — this subsection fixes the **remaining gap**: **Old GC / CPU as the only lead** when **`search` pool saturation** is the **primary** plausible storyline. **Long-form narrative, Chinese templates, and evidence closure:** [sop-query-thread-pool.md](sop-query-thread-pool.md) **Report narrative: search pool vs GC / CPU headlines** (keep this section as a **short rubric**; avoid duplicating those paragraphs here). **Search-pool-primary vs write (both pools have cumulative `rejected`):** For **query-saturation** cases where the catalog targets **`ThreadPool.SearchRejected`** and **`_nodes/stats/thread_pool`** shows **`search.rejected` ≫ `write.rejected`** on the same data nodes, the **executive lead** and **first P0 bullets** must **not** read like a **write-primary** report with search as a footnote — put **`search` pool / query concurrency** **first** (or **before** write in the same band). **`write.rejected`** may remain **P0/P1** as **parallel** or **secondary** (bulk, catch-up, historical ingest); **say so explicitly** so readers do not infer “mainly an ingest/write incident.” Tie the **search** lead to **high concurrent query / terms / slow query** and **hot index name** when verified. | Item | What to add | |------|-------------| | **Headline order** | Do **not** make **“Old GC (P0) + CPU spike (P1)”** the **sole** executive story when **`ThreadPool.SearchRejected`** / **`search` pool capacity** is still plausible — treat **query concurrency > `search` pool** as **first or co-equal**; **GC/CPU** as **co-stress, cascade, or second wave** unless engine evidence proves otherwise. **Parallel to §6.4:** **dual P0** (search capacity + Old GC) or **causal chain** with **`search.rejected` / queue in the first sentence** when the prompt is read-heavy — mirror [sop-write-performance.md](sop-write-performance.md) §2 ordering, swap **`write` → `search`**. | | **P0 / executive order vs `search` ≫ `write` (cumulative)** | If **`search.rejected` ≫ `write.rejected`** per node (or cluster totals) and **`ThreadPool.SearchRejected`** applies, **do not** place **`ThreadPool.WriteRejected`** **above** search in the **opening summary** or **first P0 line** unless **time-resolved** evidence shows **write** dominated the window. **Script print order** (e.g. checker listing write before search) is **not** narrative order — override for human-facing text when data-plane magnitudes and scenario type say **search-first**. | | **Quantification** | Same spirit as **§6.4**: cite **`search.rejected`** with **`search.completed`** (optional reject share, cumulative semantics) when stating capacity — not only “queue was high.” | | **Rules + summary** | If **both** **GC P0-class** and **`ThreadPool.SearchRejected`** (or catalog equivalent) appear in the **same window**, **list both** in the opening summary — do not imply **only GC** is P0. | | **Engine API incomplete** | Use the **self-limiting** templates in [sop-query-thread-pool.md](sop-query-thread-pool.md) (**Report narrative: search pool vs GC / CPU headlines**): state **`GET /_nodes/stats/thread_pool` (`search`)** as **pending verification** when timeouts prevented collection. | | **Customer-facing wording** | Prefer **“引擎层必查清单（SKILL 文档第 5 节）”** over **§**-prefixed section labels in external deliverables. | ### 6.6 Timeline vs severity (recency-ordered narrative) Use when **multiple** of: **`ThreadPool.WriteRejected`**, **`ThreadPool.SearchRejected`**, **GC / heap / CPU** appear together. **Severity bands (P0/P1)** are **not** a substitute for **when** each path stressed the cluster inside `{begin} ~ {end}`. | Item | What to add | |------|-------------| | **Section present** | An **`### Incident timeline (recency-ordered)`** (or equivalent) with **time-ordered** or **“latter-window emphasis”** bullets — unless the report is explicitly minimal **and** recency cannot be distinguished (then one **uncertainty** line). | | **No false recency from counters** | **`search.rejected` / `write.rejected`** are **cumulative since node start** — do **not** say “write happened first” from **counter magnitude alone**; use **CMS series**, **slow logs**, or **paired deltas** on `rejected`/`completed`. | | **CMS peaks** | Where available, cite **which metric** peaked **when** (e.g. CPU vs GC duration vs pool-related CMS names) relative to **window start vs end**. | | **Executive order** | **Whichever path peaked or persisted closer to window end** should **lead** the **opening summary** (recency-weighted): e.g. if **search-path** pressure is **more recent** than **write-path**, **lead** with **search / query concurrency** (and co-stress) **as appropriate**; if **write-path** is **more recent**, **lead** with **write / bulk**. Same **time-over-magnitude** rule as **§6.5** row **P0 / executive order vs `search` ≫ `write`** (override when **time-resolved** evidence shows the **other** path dominated). Do **not** drop **P0** findings for the **other** path from **Findings (by priority)**. | | **Script print order** | **`check_es_instance_health.py` listing order** is **not** proof of temporal order — override for narrative when **time-resolved** evidence supports it. | --- ## 7. Common mistakes | Issue | Bad example | Correct approach | |-------|-------------|------------------| | Hard-coded secrets | AK/SK/password in command line | CLI profile + environment variables | | Wrong scheme | `http://` vs `https://` mismatch | Match the instance endpoint | | Unconfirmed params | Guessing region / instance id | Confirm with the user first | | Skipping RAM | Ignoring permission errors | Validate RAM Actions first | | P0 order = time order | Listing write-path P0 before search P1 because “P0 first” when CMS shows **search stress later** in the window | Add **Incident timeline**; **lead** the executive summary with **recency-weighted** impact; keep full **Findings by priority** | FILE:references/cli-installation-guide.md # Aliyun CLI Installation & Configuration Guide Complete guide for installing and configuring Aliyun CLI. > **Aliyun CLI 3.3.1+**: Supports installing and using all published Alibaba Cloud product plugins. Make sure to upgrade to 3.3.1 or later for full plugin ecosystem coverage. ## Installation ### macOS **Using Homebrew (Recommended)** ```bash brew install aliyun-cli # Upgrade to latest brew upgrade aliyun-cli # Verify version (>= 3.3.1) aliyun version ``` **Using Binary** ```bash # Download wget https://aliyuncli.alicdn.com/aliyun-cli-macosx-latest-amd64.tgz # Extract tar -xzf aliyun-cli-macosx-latest-amd64.tgz # Move to PATH sudo mv aliyun /usr/local/bin/ # Verify aliyun version ``` ### Linux **Debian/Ubuntu** ```bash # Download wget https://aliyuncli.alicdn.com/aliyun-cli-linux-latest-amd64.tgz # Extract and install tar -xzf aliyun-cli-linux-latest-amd64.tgz sudo mv aliyun /usr/local/bin/ # Verify aliyun version ``` **ARM64 Architecture** ```bash # Download ARM64 version wget https://aliyuncli.alicdn.com/aliyun-cli-linux-latest-arm64.tgz # Extract and install tar -xzf aliyun-cli-linux-latest-arm64.tgz sudo mv aliyun /usr/local/bin/ ``` ### Windows 1. Download from: https://aliyuncli.alicdn.com/aliyun-cli-windows-latest-amd64.zip 2. Extract the ZIP file 3. Add the directory to your PATH environment variable 4. Verify: `aliyun version` ## Configuration ### Quick Start ```bash aliyun configure set \ --mode AK \ --access-key-id \ --access-key-secret \ --region cn-hangzhou ``` ### Environment Variables ```bash export ALIBABA_CLOUD_ACCESS_KEY_ID=your_access_key_id export ALIBABA_CLOUD_ACCESS_KEY_SECRET=your_access_key_secret export ALIBABA_CLOUD_REGION_ID=cn-hangzhou ``` ## Verification ```bash # Check configuration aliyun configure list # Test authentication aliyun ecs describe-regions ``` ## References - Official Documentation: https://help.aliyun.com/zh/cli/ - Access Key Management: https://ram.console.aliyun.com/manage/ak FILE:references/es-api-call-failures.md # Elasticsearch REST API call failures (progressive guide) Use this document **in order** when `curl` to the data plane fails or returns no useful JSON. For **when** engine APIs are mandatory vs optional, see [es-api-diagnosis-strategy.md](es-api-diagnosis-strategy.md) (sections 1–3: feasibility, necessity, MUST triggers). **Security:** never print `ES_PASSWORD` or ask for it in chat — only [SKILL.md](../SKILL.md) section 2.2 patterns (local `export` in the same shell the agent uses). --- ## 0. Mandatory order (agents and operators) Do **not** infer “this environment has no Elasticsearch credentials” from a lone **unauthenticated** `curl` (for example a probe that omits `-u`), even if the response is **401**. **401 without `Authorization` is expected** on secured clusters. **Always run §1.1 first** in the **same shell** that will execute the diagnosis `curl` chain (for Cursor: the **integrated terminal** session the agent uses). Then: | Step | Action | |------|--------| | 1 | §1.1 presence checks for `ES_ENDPOINT` / `ES_PASSWORD` (and `ES_USERNAME` if you use non-default). | | 2 | If **both** `ES_ENDPOINT` and `ES_PASSWORD` are **SET** → run authenticated minimal APIs immediately (§1.3). Do not ask the user to export secrets until a **authenticated** attempt fails. | | 3 | If **either** is **NOT SET** → blocking reason is **missing engine credentials in this shell**; point to [SKILL.md](../SKILL.md) §2.2 and have the user `export` in **that same terminal**, then re-run §1.1 → §1.3. | | 4 | If authenticated calls return **401** → §1.2 (wrong user/password/realm or URL scheme). | | 5 | If authenticated calls return **000** / refused / **28** → §2 (transport). | This order matches **actual runs**: many failures are “agent skipped §1.1” or “wrong `http`/`https`,” not missing cloud APIs. --- ## 1. Preconditions — why the first answer might skip ES JSON Engine `curl` needs **both**: | Gate | Meaning | |------|---------| | **Executable** | `ES_ENDPOINT` + `ES_PASSWORD` set; TCP/TLS path to Elasticsearch works; authenticated calls return JSON. | | **Necessary** | The question needs engine proof that CMS / OpenAPI cannot supply (see strategy doc). | If **Executable** fails, the agent is **not** “forgetting” Elasticsearch: calls either were not possible (no env), were sent **without** auth and got **401**, used the **wrong** auth or **URL scheme**, or hit transport errors — so `_cluster/health`, `/_cluster/allocation/explain`, etc. do **not** yield business JSON. ### 1.1 Default habit (same shell as the agent) Run **presence only** (never echo secrets): ```bash [[ -n "$ES_ENDPOINT" ]] && echo "ES_ENDPOINT: SET" || echo "ES_ENDPOINT: NOT SET" [[ -n "$ES_PASSWORD" ]] && echo "ES_PASSWORD: SET" || echo "ES_PASSWORD: NOT SET" [[ -n "$ES_USERNAME" ]] && echo "ES_USERNAME: SET" || echo "ES_USERNAME: NOT SET" ``` Interpretation: - **`ES_ENDPOINT` + `ES_PASSWORD` both SET** → proceed to **§1.3** immediately (minimal authenticated checks). Optional: print **only** the host/scheme for sanity (`echo "$ES_ENDPOINT"` is OK if it contains no secrets; do **not** echo password). - **Either NOT SET** → state clearly: **this shell has no engine credentials**, not “Elasticsearch is unreachable.” Point to [SKILL.md](../SKILL.md) **section 2.2**; user `export` in the **integrated terminal** (same shell), then you **re-run §1.1 then §1.3** — do not treat an earlier unauthenticated 401 as proof that credentials are absent. ### 1.2 HTTP **401 Unauthorized** (after you know whether auth was sent) Classify **401**: | Situation | Meaning | What to do | |-----------|---------|------------| | **401** on a request **without** `-u` / `Authorization` | Normal for secured ES; **not** a credential verdict | Run §1.1; if vars SET, retry **with** auth (§1.3). | | **401** on a request **with** `ES_USERNAME` / `ES_PASSWORD` (or `-u`) | Auth rejected | Fix **local env only**: password, username (often `elastic`), **`http://` vs `https://`** on `ES_ENDPOINT` (must match `DescribeInstance` `protocol` / actual listener), port, or trailing path typos. See [SKILL.md](../SKILL.md) §2.2. | **Principle:** “We saw 401” is not actionable until the table above is satisfied. ### 1.3 Minimal authenticated checks (run as soon as §1.1 passes) Use timeouts on every call (see [SKILL.md](../SKILL.md) §7). Normalize base URL once: - If `ES_ENDPOINT` is `host:port` with no scheme, prepend **`http://`** or **`https://`** consistently with the cluster (Alibaba Cloud `DescribeInstance` reports `protocol` for the ES endpoint). - Strip trailing `/` before appending paths. Example pattern (do not log password): ```bash BASE="${ES_ENDPOINT}" case "$BASE" in http://*|https://*) ;; *) BASE="http://${BASE}" ;; esac BASE="${BASE%/}" curl -sS --connect-timeout 10 --max-time 30 -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" "${BASE}/_cluster/health?pretty" ``` For **`https://`**, add **`--cacert /path/to/ca.pem`** (or **`-k`** for short, non-production tests) if the system trust store does not trust the cluster certificate. **Do not** switch from **`http://` to `https://`** as a blind fix for **`000`/timeout** when **`DescribeInstance` `protocol` is `HTTP`** — prefer allowlist / SG / path checks; see [SKILL.md](../SKILL.md) §2.2 (*If `http://` does not work — when to try `https://`* and *HTTPS — prerequisites*). When **MUST** triggers require allocation evidence and the cluster is **Red**, prefer **targeted** explain after a quick shard listing — see **§1.4** (empty `explain` body can pick a **replica** while the business-critical failure is an **unassigned primary**). ### 1.4 `POST /_cluster/allocation/explain` — practical pitfall `POST /_cluster/allocation/explain` with an **empty body** `{}` returns **one** explanatory shard, which may be a **replica** (`"primary" : false`). For **Red** clusters, the P0 signal is usually an **unassigned primary**. **Runbook:** 1. `GET _cat/shards?v&h=index,shard,prirep,state,node,unassigned.reason&s=state` — find `UNASSIGNED` rows with **`p`** (primary). 2. `POST /_cluster/allocation/explain` with body, for example: ```json { "index": "", "shard": 0, "primary": true } ``` Then read `allocate_explanation`, `unassigned_info`, and `node_allocation_decisions[].deciders[]` (see [sop-cluster-health.md](sop-cluster-health.md)). --- ## 2. Transport and TLS outcomes (classify before blaming “network”) | Pattern | Typical symptom | First direction | See also | |---------|-----------------|-----------------|------| | **Connection refused** | `curl` exit **7**, `Connection refused` | Process down, public access off, wrong port | [sop-cluster-health.md](sop-cluster-health.md) | | **Persistent timeout** | exit **28**, **all** calls time out | **Context-dependent** — section 2.1 below | Allowlist + CMS triage | | **Intermittent timeout** | **some** curls OK, **some** timeout | Often **overload** / meltdown | [sop-service-avalanche.md](sop-service-avalanche.md) | | **Auth failure** | HTTP **401** **with** auth headers sent | Wrong/missing password, user, or scheme | §1.2 | | **TLS mismatch** | e.g. `WRONG_VERSION_NUMBER` | `http://` vs `https://` on `ES_ENDPOINT` | Match **`DescribeInstance` `protocol`**: public **HTTP** listener → **`http://:9200`**; **`https://`** on HTTP-only causes TLS errors. | ### 2.1 Persistent timeout (exit 28) — context-aware > Do **not** equate exit 28 with “allowlist only.” Exit 28 means the TCP connection did not complete in time — triangulate with control plane + CMS. Suggested order: 1. **Allowlist:** `DescribeInstance` `publicIpWhitelist` / `networkConfig.whiteIpGroupList` (`whiteIpType=PUBLIC_ES`) vs client egress IP. 2. **If allowlist looks OK but still timing out:** | If… | Lean toward… | Action | |-----|---------------|--------| | CMS `NodeCPUUtilization` > **80%** or `NodeHeapMemoryUtilization` > **85%** | **Server overload** — ES too busy to finish TLS/TCP quickly | Treat as degradation; [sop-service-avalanche.md](sop-service-avalanche.md) | | CMS OK, security group missing inbound **9200** | Security group | Open path from client | | CMS OK, allowlist + SG OK | Path / middlebox | `telnet` / `nc` hop check | **Principle:** exit 28 is a transport-timeout signal, not proof of a firewall drop. If CMS shows resource pressure, prefer **overload** before “pure network.” ### 2.2 Intermittent timeouts When some `curl` calls succeed and others time out in the same session: 1. Do **not** label as “preconditions not met” if `ES_*` is set. 2. **Must** cross-check CMS `NodeCPUUtilization`. 3. If CPU > **80%**, treat as suspected meltdown — [sop-service-avalanche.md](sop-service-avalanche.md). 4. Use successful responses as **partial** evidence; state coverage limits in the report. **Principle:** flaky connectivity is itself a signal — often overload with a **live** process, not a simple misconfig. ### 2.3 Flaky API collection tactics When the cluster is hot: longer `--connect-timeout` / `--max-time`, start with light APIs (`_cat/nodes`, `_cluster/health`), retry `_nodes/hot_threads` a few times — see [sop-service-avalanche.md](sop-service-avalanche.md) (intermittent timeout entry). --- ## 3. Evidence boundary (one screen) - With **only** control plane + CMS, or with **no successful authenticated engine JSON**, you may still state **high-confidence** facts such as: CMS **`ClusterStatus` = Red ⇒ at least one primary shard is unassigned**. - You **cannot** uniquely pin **allocation-explain-class** causes (e.g. bad `routing.allocation.require._name`, exact decider text) **without** at least one **authenticated successful** `/_cluster/allocation/explain` targeting the relevant shard (or equivalent) — see **§1.4**. - Say explicitly: the gap is **missing authenticated data-plane evidence** or **wrong shard in explain output**, not “Elasticsearch cannot be diagnosed.” --- ## 4. Report checklist (when MUST data is missing) Put at the **top** of the report: 1. **One line blocking reason** — only **after** §1.1: e.g. `ES_PASSWORD NOT SET in this shell`, **`401` with authenticated curl** (then scheme/user/password), **connection refused**, **persistent vs intermittent timeout** — **before** a long generic “evidence boundary” paragraph. 2. Which **MUST** conditions fired (from CMS / script). 3. Which engine evidence types **would** be collected after auth/path works (`allocation/explain` with **primary** shard if Red, `hot_threads`, …). 4. Pointer to [SKILL.md](../SKILL.md) **section 2.2** only when §1.1 shows vars **not** set; if vars are set but calls fail, point to **§1.2 / §2** instead. --- ## 5. Related references | Doc | Role | |-----|------| | [es-api-diagnosis-strategy.md](es-api-diagnosis-strategy.md) | MUST / SHOULD / executable vs necessary | | [verification-method.md](verification-method.md) | `curl` examples and checklists | | [acceptance-criteria.md](acceptance-criteria.md) | PASS / partial when engine checks blocked | | [sop-service-avalanche.md](sop-service-avalanche.md) | CPU + intermittent timeouts + `all shards failed` | | [sop-cluster-health.md](sop-cluster-health.md) | Real node loss vs cluster issues | FILE:references/es-api-catalog.md # Elasticsearch REST API catalog (`curl`) Use `curl` directly against the Elasticsearch data plane. ```bash export ES_ENDPOINT="" export ES_USERNAME="elastic" export ES_PASSWORD="" # Generic template (HTTP) curl -sS -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ "http://${ES_ENDPOINT#http://}/?pretty" ``` > If the cluster only serves **HTTPS**, switch to `https://` and add CA options or `-k` for testing only. --- ## Cluster layer | Endpoint | Purpose | |----------|---------| | `GET /_cluster/health` | Cluster health (green/yellow/red) | | `GET /_cluster/stats` | Cluster statistics (nodes/shards/disk/JVM) | | `GET /_cluster/settings` | Cluster dynamic settings | | `GET /_cluster/pending_tasks` | Master pending tasks | | `POST /_cluster/allocation/explain` | Explain unassigned shards (requires unassigned shards) | --- ## Node layer | Endpoint | Purpose | |----------|---------| | `GET /_cat/nodes?v` | Node list (IP, CPU, heap, load) | | `GET /_cat/nodes?v&s=cpu:desc` | Nodes sorted by CPU | | `GET /_nodes/hot_threads?threads=3` | Hot threads (when CPU is high) | | `GET /_nodes/stats/thread_pool` | Thread pool queue / rejected | | `GET /_nodes/stats/breaker` | Circuit breaker trips | | `GET /_nodes/stats/jvm` | JVM / GC stats | > **Version note** > - The `?local` flag on `GET /_cat/nodes` was deprecated in 7.x and removed in 8.0 — **do not use it**. > - Other node APIs are compatible across 6.x / 7.x / 8.x. --- ## Index and shard layer | Endpoint | Purpose | |----------|---------| | `GET /_cat/indices?v&s=pri.store.size:desc` | Indices by primary store size | | `GET /_cat/indices?v&s=pri:desc` | Indices by primary shard count | | `GET /_cat/shards?v&s=state` | Shards including unassigned reasons | | `GET /_cat/allocation?v&bytes=gb` | Per-node allocation and disk | | `GET /_cat/fielddata?v&s=size:desc` | Fielddata memory (6.x may need extra settings) | | `GET /_cat/recovery?v&active_only=true` | Active shard recovery | --- ## Task layer | Endpoint | Purpose | |----------|---------| | `GET /_cat/tasks?v&detailed=true` | Running tasks with descriptions | | `GET /_tasks?detailed=true` | Task details (search/write timeouts) | --- ## Snapshots and ILM | Endpoint | Purpose | |----------|---------| | `GET /_snapshot/_status` | In-flight snapshot operations | | `GET /_cat/snapshots/?v` | Snapshot list (repository required) | | `GET /_ilm/status` | ILM status (**Elasticsearch 7.0+** only) | --- ## Version compatibility | ES version | Compatible endpoints | Not supported | Notes | |------------|----------------------|---------------|--------| | **7.x – 8.x** | 18 / 18 (100%) | — | Full | | **6.x** | 17 / 18 (94%) | `_ilm/status` | ILM introduced in 7.0 | **Notes** - `_ilm/status`: ILM is not available on 6.x. - Do not use `?local` on `_cat/nodes` on 8.x. - `_cat/fielddata` on 6.x may require additional settings to return data. --- ## Scenario quick map | Scenario class | Primary endpoints | |----------------|-------------------| | Cluster Red/Yellow, unassigned shards | `POST /_cluster/allocation/explain` + `GET /_cat/shards` | | Sustained high CPU, hot threads | `GET /_nodes/hot_threads` + `GET /_cat/nodes` | | JVM pressure, GC | `GET /_nodes/stats/jvm` + `GET /_nodes/stats/breaker` | | Thread pool saturation / rejections | `GET /_nodes/stats/thread_pool` | | Disk pressure, large indices | `GET /_cat/allocation` + `GET /_cat/indices` | | Slow search/write, timeouts | `GET /_cat/tasks` + `GET /_tasks?detailed=true` | | Snapshot failures | `GET /_snapshot/_status` + `GET /_cat/snapshots/` | | Master backlog | `GET /_cluster/pending_tasks` | | ILM issues | `GET /_ilm/status` | FILE:references/es-api-diagnosis-strategy.md # Elasticsearch REST API diagnosis strategy (MUST rules) This note captures Q&A from skill usage: when `SKILL.md` says ES APIs are **MUST**, why some paths still rely on control plane + CMS only, and how skill text relates to tooling. **When `curl` fails** (401, timeouts, refused): use the progressive guide [es-api-call-failures.md](es-api-call-failures.md) first, then return here for MUST / SHOULD rules. **SKILL alignment:** MUST-trigger workflow (**reachability** including `ES_*` and endpoint match, then **required `curl` per fired row**) is **normative** in `SKILL.md` §5 (*Binding rule (MUST triggers)*), Step 2, and §7. The CMS tables in this file are supplementary; if workflow nuance differs, follow **`SKILL.md`** for agent execution. --- ## 1. Decision rules in the skill (evidence-driven) Calling engine-layer `curl` (REST) needs **both**: | Condition | Meaning | |-----------|---------| | **Executable** | `ES_ENDPOINT` and `ES_PASSWORD` are set, path to port `9200` works (scheme, allowlists, security group, etc.). | | **Necessary** | Root-cause reasoning needs engine-layer proof that control plane / CMS cannot supply or refute. | ### Summary - **MUST (run ES APIs)** - Cluster Red/Yellow: unassigned shard reasons, allocation filters. - Search/write performance: thread pools, rejections, hot threads, breakers, task backlog. - Cases where control plane looks fine but the engine is not. - **SHOULD (prefer ES APIs)** - CPU/memory/load anomalies that disagree with control-plane signals — cross-check. - Post-change incidents — confirm stuck recovery or long-running tasks on the engine. - **CAN SKIP** - Root cause is already closed on the control plane and engine data would not change the answer (e.g. clear RAM denial, instance in a controlled change window, billing/resource state making the instance unavailable). --- ## 2. MUST trigger signals (CMS) **Treat as MUST when any of the following CMS signals appear:** | CMS signal | MUST theme | Typical ES API evidence | |------------|------------|-------------------------| | `ClusterStatus` max ≥ 1 (Yellow) or ≥ 2 (Red) | Cluster health | `/_cluster/allocation/explain`, `/_cat/shards` | | `ClusterDisconnectedNodeCount` max > 0 | Node loss | `/_cat/nodes`, `/_cluster/health` | | `NodeCPUUtilization` max > 80% | CPU overload | `/_nodes/hot_threads`, `/_tasks` | | `NodeHeapMemoryUtilization` max > 85% | Memory pressure | `/_nodes/stats/jvm`, `/_nodes/stats/breaker` | | `NodeDiskUtilization` max > 85% | Disk pressure | `/_cat/allocation`, `/_cat/shards` | | Thread pool `rejected` > 0 on any node | Performance | `/_nodes/hot_threads`, `/_nodes/stats/thread_pool` | | CPU / memory / disk CV > 0.3 across nodes | Imbalance | `/_cat/shards`, `/_cat/allocation` | | **Every diagnosis (ALWAYS)** | **Disk watermarks / index read-only** | `/_cluster/settings`, `/_all/_settings?filter_path=*.settings.index.blocks`, `/_cat/allocation` | | Intermittent ES API timeouts + `NodeCPUUtilization` > 80% | Possible meltdown | `/_nodes/hot_threads`, `/_nodes/stats/thread_pool`, `/_tasks` | **Mandatory hint:** If a MUST situation is detected and `ES_ENDPOINT` is **not** set, the report **must** start with a warning listing the missing engine-layer evidence. --- ## 2.5 ALWAYS checks (when ES is reachable) **These do not wait on CMS thresholds; run whenever `ES_ENDPOINT` (and password) is available:** | Check | ES API | What it catches | Why | |-------|--------|-----------------|-----| | Watermark settings | `GET _cluster/settings?include_defaults=true&filter_path=**.watermark` | Absolute-byte or bad % watermarks | CMS disk % can look fine while transient watermarks already force read-only | | Index read-only blocks | `GET _all/_settings?filter_path=*.settings.index.blocks` | `read_only_allow_delete: true` | Cluster can be Green with “normal” disk % while indices are already read-only from flood_stage | | Free space vs absolute watermark | `GET _cat/allocation?format=json&bytes=b` | Free bytes below absolute `flood_stage` | Only needed when watermarks use absolute bytes; % breach is compared to CMS in the script | **Rationale (watermark edge case):** Low disk utilization and cluster Green can still coexist with transient absolute-byte watermarks that block writes. CMS-only views miss that class of misconfiguration. --- ## 3. Why sometimes only control plane + CMS? **MUST addresses necessity:** in the situations above, methodology says you **should** use engine APIs for deep proof. But the skill **also** states **executable** preconditions. If `ES_ENDPOINT` / `ES_PASSWORD` are missing, or TCP to `9200` fails, `curl` **cannot** run. Then you can only: - Use `aliyun` CLI + CMS + logs; - State the **evidence boundary** in the report: no ES REST path — no engine-level root cause. So: **not** “MUST can be skipped on purpose”, but **“MUST cannot run until preconditions hold.”** --- ## 3.5 Connection failures — differentiate patterns > **Full playbooks** (credential probe, HTTP 401 meaning, exit 28 triage, intermittent timeouts, evidence boundary, report checklist): [es-api-call-failures.md](es-api-call-failures.md) **Summary table** (read the linked doc for steps): | Pattern | Typical symptom | First pointer | |---------|-----------------|---------------| | **Refused** | curl exit 7 | Process / port / public access — `sop-cluster-health.md` | | **Persistent timeout** | exit 28, all calls | Allowlist + CMS triage — **not** “network only by default” | | **Intermittent timeout** | mixed success | Overload / meltdown — `sop-service-avalanche.md` | | **Auth failure** | HTTP 401 | `ES_PASSWORD` / `ES_USERNAME` / scheme | | **TLS mismatch** | wrong version errors | `http://` vs `https://` on `ES_ENDPOINT` | --- ## 4. Documentation vs `check_es_instance_health.py` today When `ES_ENDPOINT` and `ES_PASSWORD` are set, `_check_cluster_config_optional` **automatically** calls: | Area | API | Role | |------|-----|------| | Liveness | `GET /_cluster/health` | Fail fast if ES is unreachable | | Settings | `GET /_cluster/settings?include_defaults=true` | Fielddata breaker + disk **watermark** rules (incl. absolute-byte branch with `/_cat/allocation?format=json&bytes=b` when needed) | | Replicas | `GET /_cat/indices?h=index,rep&format=json` | Business indices with `number_of_replicas=0` | | Read-only | `GET /_all/_settings?filter_path=*.settings.index.blocks` | `read_only_allow_delete` blocks | | Thread pools | `GET /_nodes/stats/thread_pool` | Non-zero `rejected` counters | Finding **remediation** text may recommend `/_cluster/allocation/explain`, `/_cat/shards`, `/_nodes/hot_threads`, etc. Those are **operator / agent next steps** — the script does **not** HTTP-fetch **hot_threads** or **allocation/explain** bodies by itself. When rule findings imply a **§5 MUST** row, `check_es_instance_health.py` prints a short **§5 MUST — engine APIs** footer (deduped paths); the agent must still **execute** those calls after §2.2 reachability. **Skill / agent workflow:** for Red/Yellow, thread-pool saturation, and other MUST rows in `SKILL.md`, the agent should still run the listed `curl` (or Console) steps to collect engine evidence beyond what the script auto-pulls. ### Planned vs implemented Older drafts described stderr banners such as `[MUST采集] … auto allocation/explain`. **That auto-fetch is not in the current script** — allocation/explain and hot_threads bodies remain **manual MUST**. The script may print a **§5 MUST — engine APIs** footer (stdout) when findings map to `SKILL.md` §5; that is a checklist, not an HTTP substitute. --- ## 5. Summary | Question | Answer | |----------|--------| | MUST scenarios — must ES APIs be used? | **Methodologically yes**, for evidence control plane/CMS cannot replace. | | Why sometimes not? | Usually **executable** preconditions fail (no creds, no route, wrong endpoint). | | What does the health script auto-call? | Health + cluster settings (breakers/watermarks) + zero-replica cat + read-only settings + thread-pool stats — **not** allocation/explain or hot_threads bodies. | | Practical advice | When `9200` is reachable, set env vars; run additional MUST curls from `SKILL.md` / SOPs where the script stops at recommendations. | --- ## 6. Related references - Main skill: `../SKILL.md` (workflow, credentials, direct ES access) - **REST call failures (progressive):** [es-api-call-failures.md](es-api-call-failures.md) - Cluster health SOP: `sop-cluster-health.md` - Performance SOPs: `sop-write-performance.md`, `sop-query-thread-pool.md` FILE:references/health-events-catalog.md # Elasticsearch health event catalog > **18 event codes / 49 reason codes** across **8** categories. Each row: **reason code** under its **event code**. --- ## Quick nav: TOP 15 by case frequency | Rank | Reason code | Event code | Priority | Threshold (summary) | SOP | | --- | --- | --- | --- | --- | --- | | 1 | `CPU.PeakUsageHigh` | `HealthCheck.CPULoadHigh` | P1 | P99 > 95% for 3m | [sop-cpu-load.md](sop-cpu-load.md) | | 2 | `JVMMemory.OldGenUsageCritical` | `HealthCheck.JVMMemoryPressure` | P0 | avg > **85%** for 2m | [sop-memory-gc.md](sop-memory-gc.md) | | 3 | `Node.Disconnected` | `HealthCheck.ClusterUnhealthy` | P0 | disconnected nodes > 0 for 10m | [sop-cluster-health.md](sop-cluster-health.md) | | 4 | `CPU.PersistUsageHigh` | `HealthCheck.CPULoadHigh` | P0/P1 | avg > **70%** (P0) / > **60%** (P1) for 10m | [sop-cpu-load.md](sop-cpu-load.md) | | 5 | `JVMMemory.GCRateTooHigh` | `HealthCheck.JVMMemoryPressure` | P1 | Old GC > **1/min** for 10m | [sop-memory-gc.md](sop-memory-gc.md) | | 6 | `Disk.UsageCritical` | `HealthCheck.DiskUsageHigh` | P0 | max > **85%** for 5m | [sop-disk-storage.md](sop-disk-storage.md) | | 7 | `JVMMemory.OldGenUsageHigh` | `HealthCheck.JVMMemoryPressure` | P1 | avg > **75%** for 5m | [sop-memory-gc.md](sop-memory-gc.md) | | 8 | `ThreadPool.WriteRejected` | `HealthCheck.ThreadPoolSaturation` | P0 | write rejected > 0.1/s and TPS > 1/s | [sop-write-performance.md](sop-write-performance.md) | | 9 | `JVMMemory.GCTimeRatioTooHigh` | `HealthCheck.JVMMemoryPressure` | P0 | GC time / wall clock > **10%** for 5m | [sop-memory-gc.md](sop-memory-gc.md) | | 10 | `ThreadPool.WriteQueueHigh` | `HealthCheck.ThreadPoolSaturation` | P0 | write queue > threads×80% for 5m | [sop-write-performance.md](sop-write-performance.md) | | 11 | `ThreadPool.SearchRejected` | `HealthCheck.ThreadPoolSaturation` | P0 | search rejected > 0.1/s and QPS > 1/s for 5m | [sop-query-thread-pool.md](sop-query-thread-pool.md) | | 12 | `Balancing.NodeCPUUnbalanced` | `HealthCheck.LoadUnbalanced` | P1 | CPU CV > **0.3** for 10m | [sop-cpu-load.md](sop-cpu-load.md) | | 13 | `Disk.UsageHigh` | `HealthCheck.DiskUsageHigh` | P1 | max > **75%** for 10m | [sop-disk-storage.md](sop-disk-storage.md) | | 14 | `ThreadPool.SearchQueueHigh` | `HealthCheck.ThreadPoolSaturation` | P0 | search queue > threads×80% for 5m | [sop-query-thread-pool.md](sop-query-thread-pool.md) | | 15 | `Cluster.StatusRed` | `HealthCheck.ClusterUnhealthy` | P0 | ClusterStatus == 2 for 2m | [sop-cluster-health.md](sop-cluster-health.md) | --- ## 1. Cluster health (3 reason codes) **Event code:** `HealthCheck.ClusterUnhealthy` | Reason code | Priority | Trigger | Rule # | Key metrics | Typical root cause | | ----------- | -------- | ------- | ------ | ----------- | ------------------ | | `Cluster.StatusRed` | **P0** | `ClusterStatus == 2` for **2m** | 1 | `ClusterStatus` | Node loss / disk full / allocation misconfiguration | | `Cluster.StatusYellow` | P1 | `ClusterStatus == 1` for **30m** | 2 | `ClusterStatus` | Too few nodes / disk pressure / change in progress | | `Node.Disconnected` | **P0** | `ClusterDisconnectedNodeCount > 0` for **10m** | 3 | `ClusterDisconnectedNodeCount` | OOM / CPU pegged / disk full / network | > **Log-based (no CMS rule):** `Cluster.UnavailableShards` — `UnavailableShardsException` in logs, primary inactive; same **event code** `HealthCheck.ClusterUnhealthy`. **Metric source:** `ClusterStatus`, `ClusterDisconnectedNodeCount` from CMS (`fetch_metrics_batch`). --- ## 2. Master stability (3 reason codes) **Event code:** `HealthCheck.MasterStabilityRisk` | Reason code | Priority | Trigger | Rule # | Key metrics | Typical root cause | | ----------- | -------- | ------- | ------ | ----------- | ------------------ | | `Master.TasksPendingCritically` | **P0** | `pending_tasks > node_count×50` for **3m** | 4 | metric TBD | Small master / huge shard count / ILM backlog | | `Master.TasksPendingHigh` | P1 | `pending_tasks > 50` for **5m** | 5 | same | Same, early warning | | `Master.ElectionTooMany` | **P0** | Master elections > 0 in 1h | 6 | same | Master OOM / network / CPU pegged / long GC pauses | > **Metric status:** `ClusterPendingTasksCount`, `MasterElectionCount` not in CMS yet — collect via ES API (`GET _cluster/pending_tasks`, `GET _cat/master`). --- ## 3. Service availability (1 reason code) **Event code:** `HealthCheck.ClusterRequestError` | Reason code | Priority | Trigger | Rule # | Key metrics | Typical root cause | | ----------- | -------- | ------- | ------ | ----------- | ------------------ | | `Cluster.RequestErrorTooHigh` | P1 | HTTP 5xx rate > **5%** and QPS > 1/s for **3m** | 7 | ClusterRequest5xxQPS | Version bug / resource exhaustion / metadata issues | > Historical typo `Cluster.RequstErrorTooHigh` may appear in older exports; canonical name: **`Cluster.RequestErrorTooHigh`**. --- ## 4. Resource anomalies (17 reason codes) ### 4.1 JVM memory pressure **Event code:** `HealthCheck.JVMMemoryPressure` | Reason code | Priority | Trigger | Rule # | Key metrics | Typical root cause | | ----------- | -------- | ------- | ------ | ----------- | ------------------ | | `JVMMemory.OldGenUsageHigh` | P1 | heap avg > **75%** for **5m** | 8 | `NodeHeapMemoryUtilization` | Small heap / heavy queries / fielddata / leak | | `JVMMemory.OldGenUsageCritical` | **P0** | heap avg > **85%** for **2m** | 9 | `NodeHeapMemoryUtilization` | Same, more urgent | | `JVMMemory.HeapGrowthRateFast` | P1 | +9% in 3m and +15% in 15m | 10 | `NodeHeapMemoryUtilization` | Bulk queries / leak / traffic spike | | `JVMMemory.GCRateTooHigh` | P1 | Old GC > **1/min** for **10m** | 17 | `JVMGCOldCollectionCount` | Heap pressure / fielddata / large aggs | | `JVMMemory.GCTimeRatioTooHigh` | **P0** | GC time / wall clock > **10%** for **5m** | 18 | `JVMGCOldCollectionDuration` | Old Gen pressure / frequent Full GC | | `JVMMemory.GCDurationTooLong` | **P0** | avg GC duration > 5000ms | 19 | `JVMGCOldCollectionDuration` | Bad G1 tuning / large heap objects | | `JVMMemory.FielddataCacheTooLarge` | P1 | Fielddata > 30% heap | 20 | metric TBD | text field aggs / unbounded fielddata | | `JVMMemory.BreakerTripped` | **P0** | breaker trips ≥ 5 per 5m | 35 | metric TBD | Huge result sets / fielddata blow-up / heavy aggs | | `JVMMemory.BreakerLimitConfigLow` | P2 | `indices.breaker.fielddata.limit` < 40% (settings check, ES API) | — | `GET /_cluster/settings` | Limit below recommendation — small aggs can trip | | `JVMMemory.OOM` | **P0** | `OutOfMemoryError` in logs | — | instance logs | OldGen full / fielddata explosion | ### 4.2 High CPU **Event code:** `HealthCheck.CPULoadHigh` | Reason code | Priority | Trigger | Rule # | Key metrics | Typical root cause | | ----------- | -------- | ------- | ------ | ----------- | ------------------ | | `CPU.PeakUsageHigh` | P1 | CPU P99 > **95%** for **3m** | 11 | `NodeCPUUtilization` | Heavy queries / ingest / GC / imbalance | | `CPU.PersistUsageHigh` (P0) | **P0** | CPU avg > **70%** for **10m** | 12 | `NodeCPUUtilization` | Sustained load — node loss risk | | `CPU.PersistUsageHigh` (P1) | P1 | CPU avg > **60%** for **10m** | 12 | `NodeCPUUtilization` | Same, warning band | | `CPU.UsageGrowthRateFast` | P0 | +9% in 3m and +15% in 15m | 15 | `NodeCPUUtilization` | Slow requests / burst traffic | > `CPU.PersistUsageHigh` uses dual thresholds: avg > 70% → P0, 60–70% → P1 (same **reason_code**, different **priority**). ### 4.3 Disk utilization **Event code:** `HealthCheck.DiskUsageHigh` | Reason code | Priority | Trigger | Rule # | Key metrics | Typical root cause | | ----------- | -------- | ------- | ------ | ----------- | ------------------ | | `Disk.UsageHigh` | P1 | max > **75%** for **10m** | 13 | `NodeDiskUtilization` | Data growth / snapshots not cleaned | | `Disk.UsageCritical` | **P0** | max > **85%** for **5m** | 14 | `NodeDiskUtilization` | Near ES flood-stage (~95%) read-only | | `Disk.UsagePredictiveRisk` | P1 | `predict_linear` shows < 5% free in 24h | 22 | `NodeDiskUtilization` | Upward trend | | `Disk.IndexReadOnly` | **P0** | block count > 0 | 36 | ClusterIndexWritesBlocked | Disk ≥95% auto read-only | | `Disk.WatermarkAbsoluteFloodBreached` | **P0** | free bytes < absolute `flood_stage` | — | ES API: `_cluster/settings` + `_cat/allocation` | Absolute watermark tripped | | `Disk.WatermarkAbsoluteFloodMarginLow` | P1 | free space barely above flood_stage (margin < 500MB) | — | ES API: same | Absolute watermark, tiny margin | | `Disk.WatermarkAbsoluteValue` | P1 | watermarks configured as absolute bytes (not %) | — | ES API: `_cluster/settings` | Non-default — does not scale with disk growth | ### 4.4 Disk IO **Event code:** `HealthCheck.DiskIOBottleneck` | Reason code | Priority | Trigger | Rule # | Key metrics | Typical root cause | | ----------- | -------- | ------- | ------ | ----------- | ------------------ | | `Disk.IOPerformancePoor` | **P0** | IO util > **90%** for **5m** | 19 | `NodeStatsDataDiskUtil` | Slow disk / bad disk / heavy write / merges | | `Disk.IOBandwidthThrottling` | **P0** | disk bandwidth util > **90%** for **5m** | 16 | NodeStatsDataDiskIoCloudDiskBaseBandwidthRate | Cloud disk cap / concentrated write | > IO rule priority was P1 → updated to **P0** (rule #19). ### 4.5 File descriptors **Event code:** `HealthCheck.SystemFileDescriptorsHigh` | Reason code | Priority | Trigger | Rule # | Key metrics | Typical root cause | | ----------- | -------- | ------- | ------ | ----------- | ------------------ | | `System.FileDescriptorsUsageHigh` | **P0** | open_fd / max_fd > **90%** for **10m** | 21 | metric TBD | Too many connections / shards / leaks | --- ## 5. Performance bottlenecks (10 reason codes) ### 5.1 Thread pool saturation **Event code:** `HealthCheck.ThreadPoolSaturation` | Reason code | Priority | Trigger | Rule # | Key metrics | Typical root cause | | ----------- | -------- | ------- | ------ | ----------- | ------------------ | | `ThreadPool.SearchQueueHigh` | **P0** | search queue > threads×80% (or > 100) for **5m** | 23 | SearchThreadpoolQueue | Slow queries / high QPS / low CPU headroom | | `ThreadPool.SearchRejected` | **P0** | search rejected > 0.1/s and QPS > 1/s for **5m** | 24 | SearchThreadpoolRejected | Queue full — new searches rejected | | `ThreadPool.WriteQueueHigh` | **P0** | write queue > threads×80% (or > 100) for **5m** | 25 | WriteThreadpoolQueue | Heavy ingest / slow disk / low CPU | | `ThreadPool.WriteRejected` | **P0** | write rejected > 0.1/s and TPS > 1/s for **5m** | 26 | WriteThreadpoolRejected | Queue full — writes rejected (HTTP 429) | | `ThreadPool.GenericQueueHigh` | P1 | generic queue > threads×80% for **10m** | 27 | metric TBD | High recovery concurrency / `node_concurrent_recoveries` too high | ### 5.2 High latency **Event code:** `HealthCheck.LatencyHigh` | Reason code | Priority | Trigger | Rule # | Key metrics | Typical root cause | | ----------- | -------- | ------- | ------ | ----------- | ------------------ | | `Latency.IndexingSlow` | **P0** | indexing latency > 1000ms and QPS > 5/s for **1m** | 28 | ClusterIndexingLatency | Slow disk / CPU / aggressive refresh | | `Latency.SearchSlow` | **P0** | search latency > 2000ms and QPS > 5/s for **1m** | 29 | ClusterSearchLatency | Slow queries / CPU / memory / cold data | | `Latency.SearchTaskRunningLong` | P1 | search task runtime > 5 minutes | 30 | metric TBD | delete_by_query / reindex / heavy aggs | | `Latency.RefreshSlow` | P1 | refresh > 1000ms for **5m** | 31 | metric TBD | Too many segments / many fields / complex mappings | ### 5.3 Slow recovery **Event code:** `HealthCheck.RecoverySlow` | Reason code | Priority | Trigger | Rule # | Key metrics | Typical root cause | | ----------- | -------- | ------- | ------ | ----------- | ------------------ | | `Recovery.SlowWarning` | Info | recovery rate < 50% of configured cap or ETA > 4h | 45 | metric TBD | Low recovery throttle / slow disk / limited bandwidth | --- ## 6. Capacity planning (6 reason codes) ### 6.1 Shard misconfiguration **Event code:** `HealthCheck.ShardMisconfigured` | Reason code | Priority | Trigger | Rule # | Key metrics | Typical root cause | | ----------- | -------- | ------- | ------ | ----------- | ------------------ | | `Shard.SegmentCountTooMany` | P1 | segments > 100 per node for **30m** | 32 | metric TBD | Rare forcemerge / short refresh interval | | `Shard.TotalCountTooMany` | P1 | total shards > nodes×CPU×20 for **1h** | 41 | `ClusterShardCount` | Too many indices / shards per index | | `Shard.SizeUnreasonable` | P2 | avg shard < 10GB or > 50GB for **1h** | 42 | metric TBD | Wrong shard count / bad sizing | | `Shard.DocumentNearLimit` | **P0** | docs per shard > 2B (Lucene ~2.1B cap) for **30m** | 43 | metric TBD | No rollover / ILM | | `Shard.NodeCountTooHigh` | P1 | shards per node > 1000 for **30m** | 44 | metric TBD | Too many shards / too few nodes | ### 6.2 Load imbalance **Event code:** `HealthCheck.LoadUnbalanced` > Imbalance spans **traffic**, **data placement**, and **resource levels**. See [sop-node-load-imbalance.md](sop-node-load-imbalance.md). | Reason code | Priority | Trigger | Rule # | Key metrics | Typical root cause | | ----------- | -------- | ------- | ------ | ----------- | ------------------ | | `Balancing.NodeCPUUnbalanced` | P1 | CPU CV > **0.3** for **10m** | 40 | `NodeCPUUtilization` | Hot shards / no coordinating nodes / skew | | `Balancing.NodeTrafficUnbalanced` | P1 | QPS/TPS CV > **0.3** for **10m** | — | thread pool active/queue | No coordinating nodes / uneven clients / routing skew | | `Balancing.NodeDataUnbalanced` | P2 | shard count or store CV > **0.3** for **30m** | — | `_cat/allocation` | New nodes / large index concentration / routing skew | | `Balancing.NodeDiskUnbalanced` | P1/P0 | disk util CV > **0.3**, hot node >75% (P1) / >85% (P0) | — | `NodeDiskUtilization` | Large shards on few nodes / old data | | `Balancing.NodeMemoryUnbalanced` | P1 | heap util CV > **0.3** and max node > **75%** for **10m** | — | `NodeHeapMemoryUtilization` | Fielddata skew / hot query caches | --- ## 7. High-availability risk (4 reason codes) ### 7.1 Cluster scale **Event code:** `HealthCheck.ClusterScaleLow` | Reason code | Priority | Trigger | Rule # | Key metrics | Typical root cause | | ----------- | -------- | ------- | ------ | ----------- | ------------------ | | `Scale.NodesInsufficient` | P1 | total nodes < 3 for **10m** | 33 | `ClusterNodeCount` | Nodes lost / under-provisioned | | `Scale.MasterEligibleNodesInsufficient` | P1 | master-eligible < 3 for **5m** | 39 | metric TBD | No dedicated masters / master nodes down | ### 7.2 Missing replication **Event code:** `HealthCheck.ReplicationMissing` | Reason code | Priority | Trigger | Rule # | Key metrics | Typical root cause | | ----------- | -------- | ------- | ------ | ----------- | ------------------ | | `Replication.MissingReplicas` | P1 | index with 0 replicas and no snapshot | 37 | metric TBD | Manual setting / leftover during resize | ### 7.3 Backup failure **Event code:** `HealthCheck.BackupFailure` | Reason code | Priority | Trigger | Rule # | Key metrics | Typical root cause | | ----------- | -------- | ------- | ------ | ----------- | ------------------ | | `Backup.SnapshotFailed` | **P0** | auto snapshot status == 2 (failed) | 34 | `ClusterAutoSnapshotLatestStatus` | Space / permissions / cluster load | | `Backup.SnapshotOutdated` | P2 | no successful snapshot for **24h** | 38 | metric TBD | Policy missing / repeated failures | --- ## 8. Configuration risk (3 reason codes) **Event code:** `HealthCheck.ConfigurationRisk` | Reason code | Priority | Trigger | Rule # | Key metrics | Typical root cause | | ----------- | -------- | ------- | ------ | ----------- | ------------------ | | `Config.JVMGCStrategyNotOptimal` | Info | Old GC rate > 1/min for **30m** | 46 | `JVMGCOldCollectionCount` | CMS instead of G1 / wrong heap sizing | | `Config.RiskClusterSettings` | P1 | `concurrent_rebalance > 16` or `concurrent_recoveries > 8` or `recovery > 200MB/s` | 47 | metric TBD | Ops tuned migration too hot for nodes | | `Config.RiskndexSettings` | P1 | Ngram `max_gram > 100`, or too many shards on small index, or 0 replicas without snapshot | 48 | metric TBD | Bad Ngram / shard planning | --- ## Appendix A: Threshold cheat sheet (health script subset) Thresholds implemented in `scripts/check_es_instance_health.py` (baseline 20260318) may differ slightly from CMS-only rule numbers below — use the script `THRESHOLDS` for ground truth. | Metric | Warning (P1) | Critical (P0) | reason_code | | ----------- | ------------- | ------------- | ----------- | | Heap utilization | avg > **75%** | avg > **85%** | `JVMMemory.OldGenUsageHigh` / `Critical` | | CPU sustained | avg > **60%** | avg > **70%** | `CPU.PersistUsageHigh` | | CPU peak | — | max ≥ **95%** (with sustained avg ≤ 60%) | `CPU.PeakUsageHigh` (P0); 80–94% band → P1 | | Disk utilization | max > **75%** | max > **85%** | `Disk.UsageHigh` / `Critical` | | Disk IO util | — | max > **90%** | `Disk.IOPerformancePoor` | | Old GC rate | > **1/min** (max in window) | — | `JVMMemory.GCRateTooHigh` | | GC time ratio | — | > **10%** of wall time | `JVMMemory.GCTimeRatioTooHigh` | | CPU imbalance CV | CV > **0.3** (with CPU floor) | — | `Balancing.NodeCPUUnbalanced` | --- ## Appendix B: How metrics are collected ### Available via CMS `fetch_metrics_batch` > References: [Basic metrics (Alibaba Cloud Help)](https://help.aliyun.com/zh/es/user-guide/basic-metrics) | > [Cluster metrics guide](https://help.aliyun.com/zh/es/user-guide/metrics-and-exception-handling-suggestions) ``` ClusterStatus Cluster health (0=Green, 1=Yellow, 2=Red) ClusterAutoSnapshotLatestStatus Snapshot status (-1=none/0=ok/1=running/2=failed) ClusterDisconnectedNodeCount Disconnected node count ClusterNodeCount Node count ClusterShardCount Total shard count ClusterQueryQPS Cluster search QPS ClusterIndexQPS Cluster indexing QPS ClusterSearchLatency Avg search latency (ms) ClusterIndexingLatency Avg indexing latency (ms) ClusterSlowSearchingCount Slow query count NodeCPUUtilization Node CPU (%) NodeHeapMemoryUtilization Node heap (%) NodeDiskUtilization Node disk (%) NodeFreeStorageSpace Free storage (MiB) NodeLoad_1m 1m load NodeStatsDataDiskUtil Disk IO utilization (%) NodeStatsDataDiskRm Disk read bandwidth (MiB/s) NodeStatsDataDiskWm Disk write bandwidth (MiB/s) NodeStatsDataDiskR Read IOPS NodeStatsDataDiskW Write IOPS JVMGCOldCollectionCount Old GC count (per sample bucket) JVMGCOldCollectionDuration Old GC duration ms (per sample bucket) NodeStatsFullGcCollectionCount Full GC count NodeStatsExceptionLogCount Exception log count ... ``` ### Requires additional ES API (for deeper diagnosis) ```bash # Thread pools (ThreadPool.*) GET _nodes/stats/thread_pool?filter_path=nodes.*.thread_pool # Breakers (JVMMemory.BreakerTripped) GET _nodes/stats/breaker # Heap / mem (JVMMemory.FielddataCacheTooLarge) GET _nodes/stats/jvm?filter_path=nodes.*.jvm.mem # Running tasks (Latency.SearchTaskRunningLong) GET _tasks?detailed=true # Pending tasks (Master.TasksPending*) GET _cluster/pending_tasks # Shard states (Cluster.StatusRed/Yellow) GET _cat/shards?v&s=state&h=index,shard,prirep,state,node,unassigned.reason # Unassigned reason GET _cluster/allocation/explain # Recovery (Recovery.SlowWarning) GET _cat/recovery?v&active_only=true # Snapshots (Backup.*) GET _cat/snapshots?v # ILM (often tied to Master.TasksPending*) GET _ilm/status # Read-only blocks (Disk.IndexReadOnly / watermarks) GET _all/_settings?filter_path=*.settings.index.blocks # or GET _cat/indices?v&h=index,status,health # Watermarks (Disk.WatermarkAbsolute*) GET _cluster/settings?include_defaults=true&filter_path=**.watermark,**.flood_stage GET _cat/allocation?format=json&bytes=b ``` FILE:references/ram-policies.md # RAM permission list Minimum read-only RAM permissions required for this skill (no write actions): ## Alibaba Cloud Elasticsearch OpenAPI - `elasticsearch:DescribeInstance` — Instance details (including clusterTasks) - `elasticsearch:ListInstance` — List instances - `elasticsearch:ListSearchLog` — Instance runtime logs - `elasticsearch:ListActionRecords` — Instance change / action records - `elasticsearch:ListAllNode` — Cluster node information ## Cloud Monitor (CMS) - `cms:DescribeMetricList` — Time-series metrics (CPU, memory, disk, load, cluster health) - `cms:DescribeSystemEventAttribute` — System events (control-plane changes, restarts, etc.) - `cms:DescribeMetricMetaList` — Metric metadata (available metric catalog) ## Optional but recommended - `sts:GetCallerIdentity` — Validate the CLI profile (`aliyun sts get-caller-identity`) ## Runtime dependencies ``` aliyun CLI >= 3.3.1 curl ``` FILE:references/related-apis.md # Related APIs Complete list of Alibaba Cloud OpenAPIs and Elasticsearch REST APIs used by this skill. --- ## Control plane (OpenAPI) ### Elasticsearch OpenAPI | Product | API action | Description | Entry point | |---------|------------|-------------|-------------| | elasticsearch | DescribeInstance | Instance details (status, version, cluster-related fields) | `check_es_instance_health.py` / `openapi_cli_collect.py` | | elasticsearch | ListInstance | List instances | `aliyun elasticsearch ListInstance` | | elasticsearch | ListSearchLog | Instance logs (instance, slow, GC, etc.) | `check_es_instance_health.py` / `openapi_cli_collect.py` | | elasticsearch | ListActionRecords | Change / action records | `aliyun elasticsearch ListActionRecords` | | elasticsearch | ListAllNode | Cluster node information | `aliyun elasticsearch ListAllNode` | ### Alibaba Cloud Monitor (CMS) OpenAPI | Product | API action | Description | Entry point | |---------|------------|-------------|-------------| | cms | DescribeMetricList | Time-series metrics | `check_es_instance_health.py` / `openapi_cli_collect.py` | | cms | DescribeSystemEventAttribute | System events | `check_es_instance_health.py` / `openapi_cli_collect.py` | | cms | DescribeMetricMetaList | Metric metadata (available metric catalog) | `aliyun cms DescribeMetricMetaList` | --- ## Data plane (Elasticsearch REST API) ### Cluster health and state | API | Endpoint | Description | Invocation | |-----|----------|-------------|--------------| | Cluster health | `GET /_cluster/health` | Cluster health (green/yellow/red) | `curl` | | Cluster stats | `GET /_cluster/stats` | Cluster statistics | `curl` | | Pending tasks | `GET /_cluster/pending_tasks` | Master pending tasks | `curl` | | Allocation explain | `POST /_cluster/allocation/explain` | Unassigned shard reasons | `curl` | ### Nodes | API | Endpoint | Description | Invocation | |-----|----------|-------------|--------------| | Nodes stats | `GET /_nodes/stats` | Node stats (CPU/memory/disk) | `curl` | | Hot threads | `GET /_nodes/hot_threads` | Hot thread stacks | `curl` | | Nodes JVM | `GET /_nodes/stats/jvm` | JVM statistics | `curl` | | Thread pools | `GET /_nodes/stats/thread_pool` | Thread pool stats | `curl` | | Circuit breakers | `GET /_nodes/stats/breaker` | Breaker trips | `curl` | | Cat nodes | `GET /_cat/nodes` | Node overview | `curl` | | Cat nodes CPU | `GET /_cat/nodes?h=name,cpu,load_1m` | CPU-oriented view | `curl` | ### Indices and shards | API | Endpoint | Description | Invocation | |-----|----------|-------------|--------------| | Cat indices | `GET /_cat/indices` | Index list | `curl` | | Cat indices by size | `GET /_cat/indices?s=store.size:desc` | Indices sorted by size | `curl` | | Cat shards | `GET /_cat/shards` | Shard layout | `curl` | | Cat allocation | `GET /_cat/allocation` | Disk / shard allocation | `curl` | ### Tasks and recovery | API | Endpoint | Description | Invocation | |-----|----------|-------------|--------------| | Tasks | `GET /_tasks` | Running tasks | `curl` | | Tasks detailed | `GET /_tasks?detailed=true` | Detailed tasks | `curl` | | Cat tasks | `GET /_cat/tasks` | Task overview | `curl` | | Cat recovery | `GET /_cat/recovery` | Shard recovery | `curl` | ### Snapshots and ILM | API | Endpoint | Description | Invocation | |-----|----------|-------------|--------------| | Snapshot status | `GET /_snapshot/_status` | In-flight snapshots | `curl` | | ILM status | `GET /_ilm/status` | ILM status (Elasticsearch 7.0+) | `curl` | --- ## Dependencies ``` aliyun CLI >= 3.3.1 elasticsearch>=7,<9 # client library if used by tooling (optional for curl-only path) ``` --- ## Permissions See [ram-policies.md](ram-policies.md). FILE:references/report-template.md # Structured diagnosis report (skeleton) Use with **[acceptance-criteria.md](acceptance-criteria.md)** (§6.x) and repo root **[SKILL.md](../SKILL.md)** §5 Step 4. Copy the block below and fill placeholders. **Before publish:** Reconcile **ClusterStatus** wording if CMS **window max** (e.g. Yellow) and a **single** `/_cluster/health` snapshot (e.g. green) both appear — qualify **time vs aggregation** ([acceptance-criteria.md](acceptance-criteria.md) **§6.1**). If **per-node CPU %** seems to contradict **which node** was **`search` pool**–bound, add **one sentence** on **sampling / window mean vs spike** ([acceptance-criteria.md](acceptance-criteria.md) **§6.2**). If **slow-log node name** ≠ **node named in pool-rejection / INSTANCELOG** lines, add **one sentence** on **routing / phase / time** ([acceptance-criteria.md](acceptance-criteria.md) **§6.2**). If **CMS `ClusterShardCount`** (or similar) **jumps** (e.g. half then back), **do not** imply shard loss without **`_cat/shards`** + **ops record** cross-check ([acceptance-criteria.md](acceptance-criteria.md) **§6.1**). ```text ## Diagnosis summary **Instance**: {instance_id} ({region_id}) **Analysis window**: {begin} ~ {end} ### Cross-layer root cause (required when `activating` coexists with Red / unassigned) **One-line root cause**: {Chain per `sop-activating-change-stuck.md` section 4: change waiting for recovery ← Red ← allocation explain} (Omit or mark “N/A” if there is no `activating` or the cluster is Green.) ### Incident timeline (recency-ordered) {Earlier → later, or “latter-window emphasis”: which dimensions (search / write / GC / CPU / disk) peaked or persisted when; cite CMS peak times or log alignment} ### Findings (by priority) #### P0 - Critical (immediate) - [Event code] Description - Evidence: {metrics / log keywords / events} - Root-cause reasoning: {analysis} - Immediate actions: {commands or steps} #### P1 - Warning (within 30 minutes) ... ### Root-cause chain diagram {Propagation path, e.g. disk full → shards cannot allocate → cluster Red} ### Open questions / follow-ups {Uncertainties and next checks} ``` FILE:references/sop-activating-change-stuck.md # SOP: Instance `activating` / change stuck **Covers:** `ManagementPlane.ActivatingStuck`; often appears together with `HealthCheck.ClusterUnhealthy` (cluster Red) and unassigned shards on the engine. **Related:** For engine-side Red / allocation, follow [sop-cluster-health.md](sop-cluster-health.md) (Section 1 — cluster Red). **This** SOP explains the **cross-layer causality** between control-plane lifecycle `activating` and engine health — so the report does not only list “one control-plane line + one engine line” without a closed loop. --- ## 1. What `activating` means `activating` is **not** a random standalone fault. It is the **control-plane lifecycle state** while a change has **not finished** (console / OpenAPI). Typical causes: - **Rolling orchestration** after `RestartInstance` and similar; - Tasks still in `updating`, e.g. plugin install/remove or config changes. User experience: “The instance has been changing / activating for hours.” --- ## 2. Cross-layer causality (closed-loop root cause) ### 2.1 Control-plane facts - `DescribeInstance`: `status == activating` (or equivalent), often with `updatedAt` to detect “stuck too long” (aligned with rule engine `ManagementPlane.ActivatingStuck`). - `ListActionRecords`: change type (e.g. rolling restart, plugin op), phase, timeline. - `ListAllNode`: node roles, whether under rolling, abnormal nodes. ### 2.2 Orchestration constraint (why “change never finishes” shows as `activating`) Many changes require the **cluster to reach an acceptable health state** (e.g. rolling waits for shard assignment, waits until the cluster is servicable) before the next step. If that condition **cannot be met for a long time**, the orchestration task stays in updating and the instance stays **`activating`**. ### 2.3 Engine-side closure (why “the cluster never recovers”) **Do not duplicate** `/_cluster/allocation/explain`, `_cat/shards`, `unassigned.reason` playbooks here — use [sop-cluster-health.md](sop-cluster-health.md) (Section 1 — cluster Red). In the “long `activating`” context, remember: **orchestration is waiting for “cluster can recover / shards can allocate”**; the **engine** Red root cause still comes from **allocation explain** (e.g. allocation filter to a non-existent node, disk, allocation disabled). Until that is fixed, **“wait for cluster recovery” may never succeed** → instance stays `activating`. > **Takeaway:** On the control plane, `activating` means “change not finished”. The **technical closure** that explains **why** it cannot finish is still **`allocation/explain`**, not a paraphrase of `activating` text instead of engine evidence. ### 2.4 Remediation order 1. **When `activating` is confirmed, complete control-plane evidence before touching the engine:** at least **`DescribeInstance` + `ListActionRecords` (MUST)**, and **`ListAllNode`** is recommended. `ListActionRecords` gives change type (e.g. `RestartInstance`), phase, rolling progress, stuck node / `pendingOperation`, etc. **Reporting only “activating + RestartInstance” without this API is an incomplete control-plane chain** (a common gap when reviews expect full change-task evidence). 2. **Then** follow [sop-cluster-health.md](sop-cluster-health.md) (Red / allocation section) for engine explain / shard root cause and fixes (delete index, change settings, etc.). 3. **Before destructive or recovery actions** (e.g. `DELETE` index, `PUT` allocation-related settings): keep a **pre-action snapshot** — same-moment `DescribeInstance` + `ListActionRecords` summary (or explicit timestamps and key fields) so the report can state “control-plane state while the change was stuck”. 4. **After** engine blockers are cleared, expect **intermediate** Yellow, throttling, `same_shard`, CMS lag, etc. while moving toward Green (see cluster-health SOP) — that is normal. **Re-check control plane after remediation:** **`DescribeInstance`** again (and **`ListActionRecords`** if needed) to see whether `activating` ended and to validate **“engine recovered → change completed → lifecycle normalized”**; if still `activating`, continue from the new records. --- ## 3. Evidence checklist (MUST / SHOULD) | Dimension | Source | Requirement | |-----------|--------|-------------| | Lifecycle | `DescribeInstance` | **MUST:** `status`, `updatedAt`, etc.; at least once before and after remediation | | Change task detail | `elasticsearch ListActionRecords` | **MUST** whenever `activating` / change-stuck: type, progress, stuck or pending nodes; **do not** skip this call with only Describe text | | Nodes | `ListAllNode` | **SHOULD:** rolling and abnormal nodes | | Cluster health (control/CMS) | `DescribeMetricList` (ClusterStatus, etc.) | **SHOULD:** cross-check with engine | | Engine root cause | `allocation/explain`, `/_cat/shards` | **MUST** when Red / unassigned: steps in **sop-cluster-health** Section 1 | ### 3.1 Recommended collection order 1. `DescribeInstance` → confirm `activating` and instance fields. 2. **`ListActionRecords`** → task detail (distinguish from “we guessed RestartInstance”: **must come from API**). 3. `ListAllNode` (recommended) → corroborate with change records. 4. Engine: `allocation/explain`, `_cat/shards`, etc. (cluster-health Red section). 5. **Before risky remediation:** another snapshot of `DescribeInstance` + `ListActionRecords` if time has passed since steps 1–2 or you are about to delete indices. 6. Execute remediation → watch `_cluster/health` / CMS for intermediate states. 7. **After remediation:** `DescribeInstance` (+ optional `ListActionRecords`) → confirm `activating` cleared. --- ## 4. Report expectations (skill template) When both **`activating` (or `ManagementPlane.ActivatingStuck`)** and **engine Red / unassigned primary** exist: - **Do not** write only “control plane and engine in parallel” with no causal link. - **Evidence** must include a **`ListActionRecords` summary** (change type, phase/progress, abnormal or pending nodes), cross-checked with **`DescribeInstance`**; if remediation ran, add **before/after** control-plane comparison (Section 3.1 recommended collection order). - **Must** add a **one-line cross-layer root cause** that states explicitly: **Change waiting for cluster recovery ← cluster stays Red ← concrete reason from allocation explain (e.g. require points to a non-existent node)** Example sentence (replace index name and explain conclusion with real values): > The rolling change on the control plane waits for cluster health; on the engine, index `{index}` has `index.routing.allocation.require._name` pointing at a non-existent node, so primaries cannot allocate and the cluster stays Red — orchestration cannot proceed → instance remains `activating`. If there is no `activating` or the engine is already Green, write “N/A” or omit this subsection. FILE:references/sop-cluster-health.md # SOP: Cluster health diagnosis **Covers:** `HealthCheck.ClusterStatusRed`, `HealthCheck.ClusterStatusYellow`, `HealthCheck.NodeDisconnected`, `HealthCheck.PendingTasksCritical/Warning`, `HealthCheck.MasterReelection` > **Related:** If the instance stays `activating` for a long time, the change never finishes, and **cluster Red / unassigned shards** persist together, read [sop-activating-change-stuck.md](sop-activating-change-stuck.md) for **cross-layer causality** and reporting requirements in addition to engine steps in this SOP. > **Report polish (optional):** [acceptance-criteria.md](acceptance-criteria.md) **§6.1** — Red vs Yellow wording (`unassigned_primary_shards`), primary vs **replica shard** count arithmetic, `unassigned.reason` (incl. optional `INDEX_CREATED` for scoring), affected-index scope, post-remediation `curl` checks, and **scoped** wording when excluding disk/node/CPU. **§6.3** — JVM / breaker / fielddata headline vs GC-only framing. --- ## Diagnosis entry: cluster state decision tree ``` ClusterStatus metric ├── 2 (Red) → [P0 immediate] primary unassigned — partial data unavailable ├── 1 (Yellow) → [P1 within ~30m] replica unassigned — available but data-loss risk └── 0 (Green) → OK ClusterDisconnectedNodeCount metric └── > 0 → [P0 immediate] node disconnected ``` --- ## 1. Cluster status Red (`HealthCheck.ClusterStatusRed`) ### Trigger conditions - `ClusterStatus == 2` for 2 minutes, or - Logs contain `UnavailableShardsException` ### Diagnosis steps **Step 1 — Find Red indices** ```bash GET _cat/indices?v&health=red ``` **Step 2 — Why shards are unassigned** ```bash GET _cat/shards?v&h=index,shard,prirep,state,node,unassigned.reason&s=state GET _cluster/allocation/explain # full reason for one representative unassigned shard # With body — specific shard: GET _cluster/allocation/explain { "index": "my_index", "shard": 0, "primary": true } ``` **Step 3 — Interpret `unassigned.reason`** | unassigned.reason | Meaning | Mitigation | |-------------------|---------|------------| | `NODE_LEFT` | Node left cluster | → see **Section 3 — Node disconnected** | | `ALLOCATION_FAILED` | Allocation failed more than 5 times | `POST _cluster/reroute?retry_failed=true` | | `ALLOCATION_DISABLED` | Allocation disabled cluster-wide | `PUT _cluster/settings {"persistent":{"cluster.routing.allocation.enable":"all"}}` | | `NO_VALID_SHARD_COPY` | No valid shard copy | Data likely lost — restore from snapshot | | `DISK_THRESHOLD_EXCEEDED` | Disk watermark | → [sop-disk-storage.md](sop-disk-storage.md) | | `DECIDERS_THROTTLED` | Recovery throttled | Wait / tune throttling / `reroute` as appropriate | | `DECIDERS_NO` + `box_type` | Hot/warm `box_type` mismatch | Remove `routing.allocation.require.box_type` on the index or add matching nodes | | `AWAITING_INFO` | Waiting on snapshot | Wait for snapshot or cancel the snapshot job | **Step 4 — Special cases** **A. Hot/warm (`box_type`) mismatch** ```bash GET _cat/nodeattrs?v&h=host,attr,value PUT /index_name/_settings { "index.routing.allocation.require.box_type": null } ``` **B. Synonym configuration blocking allocation** Example errors: ```text IOException while reading synonyms_path_path: /path/to/file.txt IllegalArgumentException: term: xxx analyzed to a token with position increment != 1 (got: 2) ``` **Scenario 1 — file path dependency** ```bash POST /index_name/_close # Fix synonym file path or remove the dependency POST /index_name/_open ``` **Scenario 2 — stopword vs synonym conflict** (real case) - **Root cause:** The synonym file contains terms that are removed by a stop filter (e.g. `"perche à selfie"` where `à` is dropped), producing `position increment != 1`. - **Remediation:** 1. Inspect the synonym file for stopword overlap. 2. Remove or change synonym rules that conflict with stopwords. 3. Or reorder analysis so the stop filter runs **after** the synonym filter (product-dependent). **C. Blocked by snapshot** ```bash GET _snapshot/_status DELETE _snapshot/repo_name/snapshot_name # cancel if appropriate ``` **Step 5 — If data cannot be recovered (destructive — confirm with user)** ```bash POST _cluster/reroute { "commands": [{ "allocate_empty_primary": { "index": "index_name", "shard": 0, "node": "node_name", "accept_data_loss": true } }] } ``` ### Common causal chains ```text Node OOM → node leaves → primary missing → Red Disk full → read-only / allocation blocked → Red Resize / rolling change in progress → migration not done → temporary Red (often expected; wait) ``` > **vs. change stuck / `activating`:** The last line above is often **expected temporary Red** while migration or rolling progresses. If **`DescribeInstance` stays `activating` far longer than normal**, change records show no progress, and the cluster stays **persistently** Red, treat it as **control-plane orchestration crossed with engine allocation** — use engine steps here **and** [sop-activating-change-stuck.md](sop-activating-change-stuck.md) for cross-layer narrative and evidence. --- ## 2. Cluster status Yellow (`HealthCheck.ClusterStatusYellow`) ### Trigger conditions - `ClusterStatus == 1` for ~30 minutes, often with **replica** shards `UNASSIGNED` ### Diagnosis steps **Step 1 — Unassigned replica shards** ```bash GET _cat/shards?v&h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED ``` **Step 2 — Allocation explain** ```bash GET _cluster/allocation/explain ``` ### Common causes | Cause | Clues | Mitigation | |-------|-------|------------| | Too few nodes (`replicas >= node count`) | explain: `not enough nodes` | Add nodes or lower `number_of_replicas` | | Per-index `total_shards_per_node` too low for primaries + replicas | explain often shows **`shards_limit`** and/or **`same_shard`** (order can vary by version); Yellow with **replica** `UNASSIGNED`, primaries started | **Prefer** relax/remove or raise `index.routing.allocation.total_shards_per_node`. **If the cap stays 1**, scaling to **3** data nodes often **still** Yellow — need **(nodes × cap) ≥** shard copies for that index (2p + 1r ⇒ **4** copies) **or** a higher cap / fewer replicas / reindex. **≥4** data nodes is the usual minimum **only when** cap stays **1** for that pattern | | Change / scale-out in progress | events / actions in `Executing` | Wait (often normal); if **`activating` persists** with no progress → [sop-activating-change-stuck.md](sop-activating-change-stuck.md) | | Disk watermarks | disk-related explain | Free disk / adjust thresholds — [sop-disk-storage.md](sop-disk-storage.md) | | Routing / awareness filters | explain shows filter predicates | Review `routing.allocation.*` settings | | `same_shard` (replica on same node as primary) | explain | Temporarily set replicas to 0 then back to 1 (use with care) | **Temporary mitigation (reduces redundancy)** ```bash PUT /index_name/_settings { "index.number_of_replicas": 0 } # After disk/node issues are fixed, raise replicas again (e.g. 1) ``` ### Normal Yellow vs abnormal Yellow - **Normal Yellow:** during change / scaling; usually clears when the task completes. - **Abnormal Yellow:** persists **> ~30 minutes** — investigate actively. - **Long `activating` + Yellow/Red with no progress:** add cross-layer analysis per [sop-activating-change-stuck.md](sop-activating-change-stuck.md); do not only say “wait for the change to finish”. --- ## 3. Node disconnected (`HealthCheck.NodeDisconnected`) ### Trigger conditions - `ClusterDisconnectedNodeCount > 0` for ~1 minute ### Diagnosis steps **Step 1 — Which node** ```bash GET _cat/nodes?v&h=name,ip,heapPercent,cpu,load_1m,diskAvailInBytes,node.role ``` **Step 2 — Logs: when and why** On the master / node logs, search for: - `removed` — node removed from cluster - `node-left` — identify the node - `ERROR` / `WARN` + disconnected node IP — errors just before leave **Step 3 — Log keywords → cause** | Log pattern | Likely cause | Next step | |-------------|--------------|-----------| | `OutOfMemoryError` / `java.lang.OOM` | JVM OOM | [sop-memory-gc.md](sop-memory-gc.md) | | `Data too large` / `CircuitBreakingException` | Circuit breaker | [sop-memory-gc.md](sop-memory-gc.md) | | `rejected` + `write` / `search` | Thread pool saturated | [sop-write-performance.md](sop-write-performance.md) / [sop-query-thread-pool.md](sop-query-thread-pool.md) | | `watermark` + disk | Disk pressure | [sop-disk-storage.md](sop-disk-storage.md) | | `Connection refused` / `network` | Network partition | Check host / VPC connectivity | | `IOException` + `disk` | Disk I/O errors | Check disk health | | *(no clear error)* | Host / heartbeat timeout | Check host metrics; restart node if needed | **Step 4 — Restart the node (or process) if required, then watch recovery** After the underlying issue is addressed, restart the failed node per your change process, then monitor shard recovery: ```bash GET _cat/recovery?v&active_only=true&h=index,shard,time,type,stage,source_node,target_node ``` ### Frequent root causes (from 150+ cases) 1. **Node OOM** (most common): heap pressure → long GC pauses → heartbeat timeout → node leaves 2. **Slow disk I/O:** writes + merges saturate I/O → heartbeat timeout 3. **CPU pegged:** no CPU for heartbeat handling → timeout 4. **Too many shards:** e.g. **> ~50k** shards → slow cluster-state application → node appears offline (details below) 5. **ILM backlog:** ILM work saturates generic thread pool → instability 6. **Recovery concurrency too high:** small nodes + aggressive recovery settings → generic pool full → node leaves ### Special case: recovery concurrency too high **Typical pattern:** - During resize / change, cold nodes drop and cluster goes Red - Logs: `rejected execution of TransportReplicationAction` - Generic thread pool queue full **Chain:** ```text Manually raised recovery concurrency (e.g. concurrent_recoveries = 128) → small node size (e.g. 4c16g cold) → generic queue full, high CPU → heartbeat lag → master marks node disconnected ``` **Mitigation:** 1. Reset recovery-related settings to defaults (Elasticsearch generally discourages changing these): ```bash PUT _cluster/settings { "persistent": { "cluster.routing.allocation.node_concurrent_recoveries": null, "cluster.routing.allocation.node_initial_primaries_recoveries": null, "cluster.routing.allocation.node_concurrent_incoming_recoveries": null, "cluster.routing.allocation.node_concurrent_outgoing_recoveries": null } } ``` 2. Restart affected nodes. 3. **Note:** Raising these rarely improves migration speed proportionally and can destabilize the cluster. ### Special case: too many shards → node disconnect **Typical pattern:** - **50k+** shards; nodes flap offline - Master log: `cluster state applier task [...] took [57.9s] which is above the warn threshold of [30s]` - `no master connection` style messages **Chain:** ```text Very high shard count → long IndicesClusterStateService / applier latency (tens of seconds) → heartbeat lag → master marks node disconnected ``` **Checks:** ```bash GET _cluster/health?filter_path=active_shards,unassigned_shards,number_of_pending_tasks GET _cat/pending_tasks?v # Search master logs for "cluster state applier task" duration warnings ``` **Guidelines:** - Roughly **≤ 20 shards per GB heap** on a data node (e.g. 16 GB heap → ~320 shards per node guideline) - **≤ ~1000 shards per node** as a practical ceiling - **≤ ~50k total shards** per cluster as a soft target **Mitigation:** - Drop expired indices (especially many small log indices) - Merge / shrink where possible — see [sop-configuration.md](sop-configuration.md) for Shrink-related guidance - Split clusters if the workload allows ### After a node leaves ```text Node leaves → shard rerouting (Yellow; Red if primaries affected) → higher I/O / network (recovery) → risk of cascading failures if the cluster was already hot ``` --- ## 4. Pending tasks backlog (`HealthCheck.PendingTasksCritical/Warning`) ### Trigger conditions - **Critical:** `pending_tasks > node_count × 50` for ~5 minutes - **Warning:** `pending_tasks > node_count × 20` for ~10 minutes ### Diagnosis steps **Step 1 — Inspect pending tasks** ```bash GET _cluster/pending_tasks GET _cat/pending_tasks?v ``` **Step 2 — Task flavor** | Dominant task type | Likely cause | Mitigation | |--------------------|--------------|------------| | Many `put-mapping` | Mapping explosion / too many fields | Cap dynamic fields / redesign mappings | | Many `create-index` | Runaway index creation | Reduce dynamic index churn | | Many ILM-related | ILM backlog | Temporarily `POST _ilm/stop`, reduce shards / fix ILM, then `POST _ilm/start` | | Many `shard-started` | Nodes joining/leaving often | Fix node stability (Section 3) | | Master CPU high | Undersized dedicated master | Scale dedicated master tier | **Step 3 — Temporary relief (ILM pile-up)** ```bash POST _ilm/stop GET _ilm/status # After remediation POST _ilm/start ``` --- ## 5. Master re-election (`HealthCheck.MasterReelection`) ### Trigger conditions - Monitoring shows the **elected master node id** changed ### Diagnosis steps **Step 1 — Current master** ```bash GET _cat/master?v GET _cat/nodes?v&h=name,ip,node.role,master ``` **Step 2 — Election in logs** Search for: `elected-as-master`, `master node changed`, `new_master` **Step 3 — Typical drivers** 1. Previous master **OOM** — memory metrics + OOM logs on old master 2. Previous master **CPU saturated** 3. **Long GC pauses** — no heartbeat during GC 4. **Network partition** — subset of nodes cannot reach master → new election **Step 4 — Dedicated master topology** ```bash GET _cat/nodes?v&h=name,ip,node.role # Role includes `m` → master-eligible # Only one `m` in production → single point of failure risk ``` > **Practice:** Use **dedicated master nodes** in production (often **3** master-eligible nodes that do not hold data) instead of data nodes also being master-eligible. --- ## Appendix: Quick command reference (cluster health) ```bash GET _cluster/health?pretty GET _cat/health?v GET _cat/nodes?v&h=name,ip,heapPercent,cpu,load_1m,disk.used_percent,node.role GET _cat/shards?v&s=state&h=index,shard,prirep,state,node,unassigned.reason GET _cluster/allocation/explain POST _cluster/reroute?retry_failed=true PUT _cluster/settings { "persistent": { "cluster.routing.allocation.enable": "all" } } GET _cat/recovery?v&active_only=true ``` FILE:references/sop-configuration.md # SOP: Configuration safety and API health **Covers:** `ConfigSafety.DangerousClusterConfig` (P1), `ConfigSafety.DangerousIndexConfig` (P1), `ApiHealth.HighErrorRate` (P1), `CapacityPlanning.SlowRecovery` (P2) *(Event IDs aligned with catalog **V4.5+**.)* --- ## 1. Dangerous cluster settings (`ConfigSafety.DangerousClusterConfig`, P1) ### Trigger conditions (any one) - `cluster.routing.allocation.cluster_concurrent_rebalance > 16` - `cluster.routing.allocation.node_concurrent_recoveries > 8` - `indices.recovery.max_bytes_per_sec > 200MB` ### Why it is risky These are often raised during data migration to speed it up. **Too high** values tend to: 1. Saturate the **generic** thread pool → pending tasks pile up → nodes drop 2. Saturate node **CPU / I/O** → heartbeat timeouts → nodes drop 3. Trigger **cascading failures** → cluster Red ### Typical cases **Case 1 — cold tier resize, `rebalance=128`** - Before the change, `cluster_concurrent_rebalance=128` was set to speed migration - Many shards move at once → CPU / I/O saturated → node leaves → Red **Case 2 — node loss with `recovery=200mb`** - `max_bytes_per_sec=200mb` to speed recovery - Network + I/O saturated together → heartbeat timeout → node leaves ### Diagnosis steps **Step 1 — Inspect current settings** ```bash GET _cluster/settings?include_defaults=true&filter_path=*.cluster.routing.allocation*,*.indices.recovery* ``` **Step 2 — Compare to safe thresholds** | Setting | Risky if | Suggested default | Off-peak ceiling | |---------|----------|-------------------|------------------| | `cluster_concurrent_rebalance` | > 16 | 2 | 8 | | `node_concurrent_recoveries` | > 8 | 2 | 4 | | `recovery.max_bytes_per_sec` | > 200MB | 40mb | 100mb | **Step 3 — Restore safer settings** ```bash PUT _cluster/settings { "persistent": { "cluster.routing.allocation.cluster_concurrent_rebalance": "2", "cluster.routing.allocation.node_concurrent_recoveries": "2", "indices.recovery.max_bytes_per_sec": "40mb" } } ``` ### Tuning guidance - **Peak hours:** keep defaults (rebalance `2`, recovery concurrency `2`, recovery rate `40mb`). - **Off-peak migration:** you may raise modestly, but stay **below** the ceilings in the table. - **Before peak returns:** reset to defaults. --- ## 2. Dangerous index settings (`ConfigSafety.DangerousIndexConfig`, P1) ### Trigger conditions (any one) - Ngram tokenizer: `max_gram > 100` or `min_gram = 0` - Small index (stored size under ~10 GB) with **primary shard count > 10** - **Zero replicas** and **no usable snapshot** (data-loss risk) ### Case A — bad Ngram settings **Example:** write timeouts with `min_gram=0`, `max_gram=1024`. - `min_gram=0` can produce empty tokens → `ArrayIndexOutOfBoundsException` - `max_gram=1024` explodes tokens per term → high CPU on indexing, timeouts **Inspect tokenizer settings** ```bash GET /index_name/_settings?filter_path=*.analysis.tokenizer ``` **Safer Ngram example** (adjust names to your mapping; `min_gram` must be **≥ 1**; `max_gram` often **≤ ~20**, never **> 100**): ```json { "analysis": { "tokenizer": { "ngram_tokenizer": { "type": "ngram", "min_gram": 2, "max_gram": 10 } } } } ``` **Fix (requires reindex)** ```bash # 1. Create new index with corrected analysis PUT /new_index { ... } # 2. Reindex POST _reindex { "source": {"index": "old_index"}, "dest": {"index": "new_index"} } # 3. Swap alias POST _aliases { "actions": [ {"remove": {"index": "old_index", "alias": "alias_name"}}, {"add": {"index": "new_index", "alias": "alias_name"}} ] } # 4. Delete old index DELETE /old_index ``` ### Case B — too many primaries on a small index **Find candidates** ```bash GET _cat/indices?v&s=store.size:asc&h=health,index,pri,rep,docs.count,store.size # Look for store.size < ~10GB but pri > 10 ``` **Why it hurts** - Each shard is a Lucene index (~**50–100MB** heap per shard is a common rule-of-thumb band) - Many tiny shards slow the master and increase GC pressure **Mitigation — reduce primary count (Shrink)** Shrink requires the index to be **read-only** and (for a single target shard layout) primaries typically co-located per Elasticsearch shrink rules — follow product docs for your version. ```bash # Step 1: Route to one node and block writes PUT /old_index/_settings { "index.routing.allocation.require._name": "target_node", "index.blocks.write": true } # Step 2: Shrink POST /old_index/_shrink/new_index { "settings": { "index.number_of_shards": 2, "index.number_of_replicas": 1 } } # Step 3: After verification, delete old index and point aliases as needed ``` ### Case C — zero replicas and no snapshot **Risk:** if a node holding the only copy goes away, data can be **lost permanently**. **Check** ```bash GET _cat/indices?v&h=index,rep&s=rep:asc # indices with rep=0 GET _cat/snapshots?v # confirm backup coverage ``` **Mitigation** ```bash PUT /index_name/_settings { "index.number_of_replicas": 1 } # and/or schedule snapshot policies ``` --- ## 3. High Elasticsearch API error rate (`ApiHealth.HighErrorRate`, P1) ### Trigger conditions - HTTP **5xx** rate **> ~5%** for **~3 minutes** - APIs such as `_cat/indices`, `_cat/nodes` **repeatedly** fail ### Diagnosis steps **Step 1 — Probe key APIs** ```bash GET _cat/health?v GET _cat/nodes?v GET _cat/indices?v GET _cluster/stats ``` **Step 2 — Elasticsearch logs** Filter **ERROR** on master / data nodes. Watch for `NullPointerException`, `NumberFormatException`, `IllegalStateException`, etc. **Step 3 — Error pattern → cause** | Exception / pattern | Likely cause | Action | |---------------------|--------------|--------| | `NullPointerException` | Metadata glitch / version bug | Restart affected nodes; escalate if cluster-wide | | `NumberFormatException` | Counter overflow (known-class bugs in some versions) | Upgrade or restart | | `IllegalStateException` | Bad index lifecycle state | `POST /index/_close` then `POST /index/_open` | | `CircuitBreakingException` | Memory pressure | [sop-memory-gc.md](sop-memory-gc.md) | **Examples** - `_cat/indices` NPE on a specific build → often **node restart** after vendor guidance - Brand-new cluster: `_cat` empty/errors until cluster formation finishes → **wait** **Step 4 — Emergency mitigation** ```bash # Node-level issues: restart the affected node (console / automation) # Index state issues: POST /problem_index/_close POST /problem_index/_open ``` --- ## 4. Slow data migration / recovery (`CapacityPlanning.SlowRecovery`, P2) ### Trigger conditions - Recovery throughput stays **below roughly half** of the configured cap **and** estimated completion **exceeds ~4 hours** (tune thresholds to your SLOs) ### Diagnosis steps **Step 1 — Active recovery** ```bash GET _cat/recovery?v&active_only=true&h=index,shard,time,type,stage,source_node,target_node,bytes_total,bytes_percent GET _cat/recovery?v&active_only=true&h=index,shard,time,stage,translog_ops,translog_ops_percent ``` **Step 2 — Rough ETA** ```text ETA ≈ remaining_bytes / observed_throughput Example: 500 GB left at 40 MB/s → on the order of hours (validate live) ``` **Step 3 — Tune migration speed to the window** **Speed up off-peak** (only if the cluster has headroom — stay under **Section 1** safety caps): ```bash PUT _cluster/settings { "transient": { "indices.recovery.max_bytes_per_sec": "100mb" } } ``` Use `200mb` only with extreme care and monitoring; the **danger** threshold in Section 1 is **> 200MB**. **Slow down during peak** (protect latency) ```bash PUT _cluster/settings { "transient": { "indices.recovery.max_bytes_per_sec": "40mb" } } ``` **Pause rebalance** (if deferrable) ```bash PUT _cluster/settings { "transient": { "cluster.routing.rebalance.enable": "none" } } # Re-enable off-peak PUT _cluster/settings { "transient": { "cluster.routing.rebalance.enable": "all" } } ``` ### Examples - **~1.5 TB** at **~100 MB/s** → many hours can still be **normal**; no action if within SLO - Migration “stuck” with cap **10 MB/s** → raising toward **40–100 MB/s** during a safe window often helps --- ## Appendix: Quick commands (configuration and API health) ```bash GET _cluster/settings?include_defaults=true PUT _cluster/settings { "persistent": { "cluster.routing.allocation.cluster_concurrent_rebalance": "2", "cluster.routing.allocation.node_concurrent_recoveries": "2", "indices.recovery.max_bytes_per_sec": "40mb" } } GET /index_name/_settings?filter_path=*.analysis GET _cat/indices?v&s=store.size:asc&h=health,index,pri,rep,docs.count,store.size GET _cat/snapshots?v GET _cluster/health GET _cat/health?v GET _cat/indices?v GET _cat/recovery?v&active_only=true ``` FILE:references/sop-cpu-load.md # SOP: High CPU load and uneven node load **Covers:** `ResourceMonitor.CPUSustainedHigh` (P0), `ResourceMonitor.CPUPeakHigh` (P1), `CapacityPlanning.LoadImbalance` (P1) CMS metric ↔ finding names may differ by catalog version — cross-check [health-events-catalog.md](health-events-catalog.md). --- ## Diagnosis decision tree ``` NodeCPUUtilization (example CMS family) ├── Any node avg > ~80% sustained → CPUSustainedHigh (P0), treat as urgent ├── Any node max > ~95% (spike) → CPUPeakHigh (P1), respond within ~30m └── Coefficient of variation (CV) of per-node avg CPU > ~0.3 → LoadImbalance (P1) NodeLoad_1m └── High load but CPU not high → often IO wait; correlate with disk / queue metrics ``` > **Read-heavy overload:** When **`_tasks?detailed=true&actions=*search*`** shows sustained heavy search on specific indices and **`search` thread_pool** queue/rejected rises together with **NodeCPU**, say so in one line — **CPU spike and search rejects share the same query load**, not two unrelated incidents. Co‑stress wording: [sop-query-thread-pool.md](sop-query-thread-pool.md) (note after the decision tree). Use the **highest priority** already fired by metrics/rules (often **P0** CPU and/or **P0** search pool per catalog) for the summary band. > **CMS CPU looks “low” vs heavy `search.rejected`:** Before finalizing the story, add **one short reconciliation** (especially for external readers): **(1)** whether the **load burst** sat mostly **early** in the report window while CMS shows a **whole-window** max/mean; **(2)** that **`NodeCPUUtilization` (or the catalog’s CPU metric)** is scoped to **this instance’s data nodes** — not a cluster rollup or wrong `Dimensions`; **(3)** optional **`GET _nodes/stats/os`** (process CPU) near the burst — **sub-minute** spikes can sit **between** coarse CMS samples. If **(1–3)** hold, it is **legitimate** to conclude **thread-pool / reject saturation without prolonged high CMS CPU** — but if a check fails, **fix the metric or window wording** instead of forcing the claim. > **Per-node CPU % “lower” on the node that logs show as `search` hotspot:** Table **means** (e.g. **5 min** buckets) can **rank nodes differently** from **minute-of-peak** behavior or **thread-pool saturation** (queue/reject, `EsRejectedExecutionException` in logs). Prefer **time-aligned** evidence (**slow logs**, **thread_pool**, **hot_threads** at spike) for **which node** was search-bound; **CPU imbalance** rows are **supporting**, not a contradiction when **granularity** differs. See [acceptance-criteria.md](acceptance-criteria.md) **§6.2** (*Per-node CPU % vs search “hot” node*). --- ## 1. Three-step triage (any high-CPU scenario) **Step 1 — Hot threads (capture while the issue is live)** ```bash GET _nodes/hot_threads # Thread name; stack frames: search / index / merge / GC ``` Prefer **`hot_threads` while CMS CPU is elevated** or immediately after a spike is reported. If the call **times out**, returns **thin stacks**, or the spike already ended while **`_nodes/stats/thread_pool`** / **`_tasks`** still show search overload, state **time-skewed or degraded hot_threads evidence** — do not claim “no search CPU work” for the whole incident window. **Step 2 — Running tasks** ```bash GET _tasks?detailed=true GET _cat/tasks?v # action, running_time_in_nanos (very long tasks) ``` **Step 3 — Slow logs** Filter logs for `took_millis`, `query_time_in_millis`, or slow-log channels (`SEARCHSLOW` / `INDEXINGSLOW`). --- ## 2. Sustained high CPU (`CPUSustainedHigh`, P0) ### Criteria (example) - Average CPU **> ~80%** for **> ~10 minutes** (tune to your alerts) ### Root-cause paths **Path A — Traffic spike** - **Signals:** `ClusterQueryQPS` / `ClusterIndexQPS` (or access logs) step up - **Check:** large `body_size` requests - **Mitigate:** throttle / rate-limit, scale out nodes urgently **Path B — Slow queries blocking** - **Signals:** hot threads dominated by **search** stacks; `_tasks` shows long-running search - **Common anti-patterns** (see [sop-query-thread-pool.md](sop-query-thread-pool.md) for tuning detail): - `wildcard` / `prefix` / `regexp` (leading wildcards are worst) - Heavy `aggregations` (high cardinality) - Deep `from + size` paging (e.g. > 10k) - `nested` / `parent-child` - Oversized `highlight` `fragment_size` - Numeric fields queried as full-text; prefer `keyword` where appropriate - **Mitigate:** `POST _tasks/{task_id}/_cancel` for runaway tasks; fix queries **Path C — Heavy indexing** - **Signals:** hot threads show **index** / **bulk**; high ingest QPS - **Mitigate:** lower **parallel** bulk pressure and client retries; tune **per-bulk payload** (avoid oversized single bulks — see [sop-write-performance.md](sop-write-performance.md) **§2**); relax `refresh_interval` where safe - **If CPU / GC spike with `write.rejected`:** align the **report narrative** with [sop-write-performance.md](sop-write-performance.md) **§2** (write-path first or dual P0 with ingest before JVM-only headline) **Path D — Frequent merges** - **Signals:** hot threads show **merge** - **Mitigate (temporary):** cap merge scheduler threads (impacts indexing latency): ```bash PUT _cluster/settings { "transient": { "indices.merge.scheduler.max_thread_count": 1 } } ``` ### Emergency order of operations ```text 1. Throttle / shed load (often fastest) 2. Cancel huge tasks (GET _tasks → POST _tasks/{id}/_cancel) 3. Scale out if needed 4. Deeper root-cause and query/index design work ``` --- ## 3. Very high CPU spikes (`CPUPeakHigh`, P1) ### Criteria (example) - CPU **max > ~95%** even if the average stays lower ### vs `CPUSustainedHigh` | | CPUPeakHigh | CPUSustainedHigh | |---|-------------|------------------| | Average | often < ~80% | > ~80% | | Peak | > ~95% | often also > ~95% | | Duration | short burst | ~10m+ | | Risk | heartbeat / latency spikes | higher chance of node loss / overload cascade | ### Focus - Spikes are often **one huge query** or **merge** - **Hot threads** must be captured **during** the spike; evidence disappears after - Also review: BKD range queries, large aggs, heavy `script` queries --- ## 4. Uneven load across nodes (`LoadImbalance`, P1) ### Criteria (example) - Per-node CPU **CV = stddev / mean > ~0.3** for **~10+ minutes** ### Typical pattern - A few nodes at **80%+** CPU, others **below ~30%** - Hot traffic pinned to specific nodes ### Causes and mitigations **Cause A — Shard skew** - **Verify:** ```bash GET _cat/shards?v&s=node&h=index,shard,prirep,state,docs,store,node GET _cat/nodes?v&h=name,ip,cpu,heapPercent,diskAvailInBytes,shards ``` - If shards of **hot indices** pile on the hot CPUs → rebalance - **Mitigate — move a shard:** ```bash POST _cluster/reroute { "commands": [{ "move": { "index": "hot_index", "shard": 0, "from_node": "node_hot", "to_node": "node_cool" } }] } ``` **Cause B — Skew *inside* shards (segment / doc locality)** - **Pattern:** shard **count** looks balanced, but one node still burns CPU; hot threads point at one shard - **Mitigation:** `forcemerge` can compact segments and sometimes smooth read patterns — **use off-peak**; understand IO cost and [Elasticsearch guidance](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-forcemerge.html) for your version ```bash POST hot_index/_forcemerge?max_num_segments=1 ``` **Cause C — No ingest-only / coordinating tier** - **Pattern:** large QPS differences by node for request handling - **Mitigate:** add **coordinating** (client) nodes so data nodes focus on data **Cause D — Custom `_routing` skew** - **Pattern:** custom routing concentrates docs in a few shards - **Mitigate:** review routing keys and shard/key design with the application team --- ## 5. Single-node CPU hot spots (field experience; not a separate V4.5 catalog line) ### Pattern A — JVM `ThreadLocal` / stale-entry churn **Typical signals** - Only **1–2** nodes at **80%+** CPU; peers ~**40%** - Hot threads: **`management`** thread ~**100%** CPU - Stack traces mention **`ThreadLocal`** / **`expungeStaleEntry`** **Mechanism (simplified)** - Many stale `ThreadLocal` entries; **G1** may not clean aggressively → **`ThreadLocalMap.expungeStaleEntry`** dominates CPU. **Confirm** ```bash GET _nodes/hot_threads # e.g.: # 100% cpu usage by thread 'elasticsearch[node][management][T#3]' # at java.lang.ThreadLocal$ThreadLocalMap.expungeStaleEntry(...) ``` **Mitigation** - **Short term:** coordinated **Full GC** or node restart **only** with vendor / SRE guidance - **Long term:** upgrade to a **7.10+** line where your distribution documents ThreadLocal / JVM fixes (major upgrades need a full migration plan) ### Pattern B — Huge search; coordinating node melts **Typical signals** - Hot node is often the **coordinating** node for the request - Heavy **query reduce** / coordination cost - Not explained by shard placement alone **Check** ```bash GET _tasks?detailed=true&actions=*search* # action, node, running_time ``` --- ## Appendix: Quick commands (CPU) ```bash GET _nodes/hot_threads GET _tasks?detailed=true GET _cat/tasks?v POST _tasks/{task_id}/_cancel GET _cat/nodes?v&h=name,ip,cpu,load_1m,heapPercent,disk.used_percent GET _cat/shards?v&s=node&h=index,shard,prirep,docs,store,node PUT _cluster/settings { "transient": { "indices.merge.scheduler.max_thread_count": 1 } } POST _cluster/reroute { "commands": [{ "move": { "index": "index_name", "shard": 0, "from_node": "node_hot", "to_node": "node_cool" } }] } ``` FILE:references/sop-disk-storage.md # SOP: Disk watermarks and IO bottlenecks **Covers:** `ResourceMonitor.DiskUsageWarning` (P1), `ResourceMonitor.DiskUsageCritical` (P0), `ResourceMonitor.DiskIOBottleneck` (P1), and **`DiskWatermarkConfigAnomaly`**-class cases (P0/P1; see **Section 3**) CMS metric names are examples — align thresholds with [health-events-catalog.md](health-events-catalog.md) and your deployment. --- ## Diagnosis decision tree ``` Disk space (e.g. NodeDiskUtilization) ├── max > ~95% → DiskUsageCritical (P0); flood-stage / read-only imminent or active ├── max > ~85% → DiskUsageWarning (P1); plan cleanup / expansion ├── Logs contain "watermark" → watermark path └── Utilization looks fine but writes fail → MUST check watermark settings (absolute-byte injection) Cluster watermark settings (_cluster/settings) ├── Watermarks use absolute values (e.g. "19605mb") → misconfiguration risk (P1); compare to free space ├── Watermark percentages extremely low (e.g. < ~5%) → misconfiguration (P1); easy write block └── _all/_settings has read_only_allow_delete: true → flood protection active (P0) Disk performance (use multiple signals) ├── IO utilization (NodeStatsDataDiskUtil) near 100% → device saturated ├── Bandwidth rate (NodeStatsDataDiskIoCloudDiskBaseBandwidthRate) near 100% → throughput cap └── IOPS rate (NodeStatsDataDiskIoCloudDiskBaseIopsRate) near 100% → random-IO cap → Any one near 100% supports an IO bottleneck diagnosis (combine with workload context) Write failures ├── Logs: "index read-only" / blocks.read_only_allow_delete → flood-stage protection └── Disk % normal + bad watermarks → config-driven block (disk not “actually full”) ``` --- ## 1. Disk usage warning (`DiskUsageWarning`, ~85%+) ### Response time Within **~2 hours** by policy — treat as **urgent** in practice to avoid escalation to Critical. ### Steps **Step 1 — Per-node disk** ```bash GET _cat/allocation?v GET _cat/nodes?v&h=name,ip,disk.used_percent,disk.avail,disk.total ``` **Step 2 — Largest indices** ```bash GET _cat/indices?v&s=store.size:desc&h=health,status,index,docs.count,store.size,pri.store.size ``` **Step 3 — Cleanup options** | Action | How | Notes | |--------|-----|-------| | Drop expired indices | `DELETE /old_index_name` | Confirm with the app owner | | Drop `.monitoring-*` | `DELETE /.monitoring-*` | Loses built-in monitoring history in-cluster | | Lower replicas | `PUT /index/_settings` → `number_of_replicas: 0` | Temporary; reduces redundancy | | Prune old snapshots | `GET _cat/snapshots` then `DELETE /_snapshot/repo/snap` | Confirm backups are not needed | | Expand disk | Console / provider API | Brief impact during resize | **Step 4 — Alerting** so the next incident is not a surprise. --- ## 2. Critical disk usage (`DiskUsageCritical`, ~95%+) ### Treat as immediate (~5 minutes) **Elasticsearch disk-based shard routing / flood behavior (typical defaults):** - Above **~85%** free space low: stop **allocating new shards** to the node (low watermark). - Above **~90%**: try to **relocate shards away** (high watermark). - Above **~95%** (flood stage): indices on that node may get **`read_only_allow_delete`**. **Step 1 — Confirm read-only blocks** ```bash GET _all/_settings?filter_path=*.settings.index.blocks # or per index GET /your_index/_settings ``` **Step 2 — Free space urgently** ```bash DELETE /old_index_name DELETE /.monitoring-* PUT /big_index/_settings { "index.number_of_replicas": 0 } ``` **Step 3 — Clear read-only after free space is healthy (often below ~85% on that path)** ```bash PUT _all/_settings { "index.blocks.read_only_allow_delete": null } ``` > **Important:** After disk pressure drops, **`read_only_allow_delete` is not always cleared automatically** — you usually must run the command above (Elasticsearch 7.x behavior). **Step 4 — Temporary watermark relief (buys time only)** ```bash PUT _cluster/settings { "transient": { "cluster.routing.allocation.disk.watermark.low": "95%", "cluster.routing.allocation.disk.watermark.high": "98%", "cluster.routing.allocation.disk.watermark.flood_stage": "99%" } } ``` > Raising watermarks can **unblock writes briefly** — you must **still reclaim disk** and **revert** to sane defaults afterward. --- ## 3. Disk watermark misconfiguration (`DiskWatermarkConfigAnomaly`, P0/P1) > **When:** Disk **utilization % looks fine** but watermarks or flood-stage still block writes. ### vs “disk actually full” | | Disk actually full | Watermark misconfiguration | |---|-------------------|----------------------------| | Utilization | High (~85–95%+) | Can be **~1–5%** | | Root cause | Data growth / retention | `transient` / `persistent` **absolute-byte** watermarks (or absurd %) | | Typical source | Logs / indices | Scripts, drills, bad copy-paste | | Trap | Low | **High** if you only glance at `% used` | ### Example chain (absolute-byte flood) ```text Script sets transient watermarks to absolute free-space requirements → flood_stage requires > 19605 MB free → only ~19632 MB free (~27 MB headroom) → small writes trip flood_stage → read_only_allow_delete on indices → writes fail while CMS “disk %” still looks ~4% ``` ### Extended write-failure tree ```text Write failures ├── High disk % (> ~85%) → Sections 1–2 (real capacity) └── Normal disk % → MUST also: ├── GET _cluster/settings?include_defaults=true&filter_path=**.watermark │ ├── Absolute-byte watermarks → misconfiguration │ └── Absurdly low % watermarks → misconfiguration ├── GET _all/_settings?filter_path=*.settings.index.blocks │ └── read_only_allow_delete: true → flood already applied └── transient vs persistent ├── transient → may reset on full-cluster restart; often script residue └── persistent → must be cleared explicitly ``` ### Steps **Step 1 — Inspect watermarks** ```bash GET _cluster/settings?include_defaults=true&filter_path=**.watermark,**.flood_stage ``` Distinguish **absolute** values (`19605mb`, `500gb`) from **percentages** (`85%`). Absolute-byte watermarks on small volumes are a common foot-gun. **Step 2 — Index blocks** ```bash GET _all/_settings?filter_path=*.settings.index.blocks ``` If you see `read_only_allow_delete: true`, flood-stage protection is in effect. **Step 3 — Compare free space to thresholds** ```bash GET _cat/allocation?v&bytes=mb ``` Compare `disk.avail` to the effective watermark / flood thresholds. **Step 4 — Remediate** | Finding | Order | Action | |---------|-------|--------| | Bad transient watermarks | 1 | Reset: `PUT _cluster/settings` with `transient` keys set to `null` for `cluster.routing.allocation.disk.watermark.{low,high,flood_stage}` | | Indices read-only | 2 | `PUT _all/_settings` → `index.blocks.read_only_allow_delete`: `null` (after space / config fixed) | | Find who set it | 3 | Audit change tickets, automation, load tests | | Verify | 4 | Test index/write paths | > **Elasticsearch 7.x:** `read_only_allow_delete` from flood stage often **persists** until cleared manually even after disk recovers. --- ## 4. Disk IO bottleneck (`DiskIOBottleneck`, P1) ### Criteria (example) - `NodeStatsDataDiskUtil` **> ~80%** for **~5+ minutes**, or - Logs show growing write latency / IO wait ### vs disk space - **High watermark / flood:** capacity (free space). - **IO bottleneck:** performance; disk can be half empty and still saturated. ### Three dimensions (recommended) > Do not rely on **IO Util alone** — combine **utilization**, **bandwidth %**, and **IOPS %**. | Dimension | Meaning | CMS utilization metric | Raw / rate metrics | Bottleneck hint | |-----------|---------|------------------------|--------------------|-----------------| | **IO Util** | Device busy % | `NodeStatsDataDiskUtil` | — | Near **100%** | | **Throughput** | MB/s vs cap | `NodeStatsDataDiskIoCloudDiskBaseBandwidthRate` | `NodeStatsDataDiskIoMbPerS` (MiB/s) | Near **100%** | | **IOPS** | Ops/s vs cap | `NodeStatsDataDiskIoCloudDiskBaseIopsRate` | `NodeStatsDataDiskIo` (count) | Near **100%** | **If any of the three nears 100%,** narrow the shape of load: - **IO Util high**, bandwidth/IOPS % not maxed → weak device / heavy queueing / mixed workload. - **Bandwidth % maxed**, IO util not maxed → large sequential reads/writes (merge, snapshot, recovery). - **IOPS % maxed** → many small random IOs (many shards, chatty queries, small writes). > Effective caps depend on **disk SKU + size + instance type** (often **min(disk limit, instance limit)**). CMS **%** metrics encode that — you usually do not hand-compute ceilings. #### CMS metric catalog (namespace `acs_elasticsearch`) > Discover fields with `DescribeMetricMetaList --Namespace acs_elasticsearch` (Alibaba Cloud Monitor). | Category | Metric | Unit | Notes | |----------|--------|------|-------| | **IO Util** | `NodeStatsDataDiskUtil` | % | Busy time | | **Throughput — absolute** | `NodeStatsDataDiskIoMbPerS` | MiB/s | Total | | | `NodeStatsDataDiskRm` / `NodeStatsDataDiskWm` | MiB/s | Read / write | | **Throughput — %** | `NodeStatsDataDiskIoCloudDiskBaseBandwidthRate` | % | Node-level vs baseline | | | `NodeStatsDataDiskIoSingleDiskMaxThroughputRate` | % | Per-disk | | | `NodeStatsDataDiskRmCloudDiskBaseBandwidthRate` | % | Read vs baseline | | | `NodeStatsDataDiskWmCloudDiskBaseBandwidthRate` | % | Write vs baseline | | | `NodeStatsDataDiskIoNetworkBaseBandwidthRate` | % | IO network baseline % | | **IOPS — absolute** | `NodeStatsDataDiskIo` | count | Total IOPS | | | `NodeStatsDataDiskR` / `NodeStatsDataDiskW` | count | Read / write IOPS | | **IOPS — %** | `NodeStatsDataDiskIoCloudDiskBaseIopsRate` | % | Node-level | | | `NodeStatsDataDiskIoSingleDiskMaxIopsRate` | % | Per-disk | | | `NodeStatsDataDiskRSingleDiskMaxIopsRate` | % | Read | | | `NodeStatsDataDiskWSingleDiskMaxIopsRate` | % | Write | | **Queue** | `DiskAverageQueueSize` | count | Avg queue depth | > **Practical triage:** start with **`NodeStatsDataDiskUtil`**, **`NodeStatsDataDiskIoCloudDiskBaseBandwidthRate`**, **`NodeStatsDataDiskIoCloudDiskBaseIopsRate`** — whichever pegs first drives the story; then use split read/write metrics. ### Causes and mitigations **Cause A — Heavy ingest** - Throttle at the source; larger bulk batches (fewer round-trips); relax `refresh_interval` when safe: ```bash PUT /index_name/_settings { "refresh_interval": "30s" } ``` **Cause B — Frequent merges (common)** - Hot threads show **merge** stacks. Short-term: ```bash PUT _cluster/settings { "transient": { "indices.merge.scheduler.max_thread_count": 1 } } ``` - Longer term: review `index.merge.policy.*` (e.g. `segments_per_tier`) with version docs — trade merge cost vs segment count. **Cause C — Hitting disk / instance ceilings** - **Ceiling ≈ min(disk SKU limit, instance type disk limit).** - Example pitfall: upgraded ESSD tier but **no gain** because the **instance** throughput cap dominates. - **Order:** confirm which side is saturated from CMS vs quotas → scale **instance** and/or **disk** → if both maxed, **add nodes** to spread IO. **Cause D — Bad hardware / media errors** - Logs: `IOException`, `EIO` → replace disk / escalate to infrastructure support. ### Example chain (slow IO → node leaves) ```text Sustained high IO → slow translog / fsync → slow heartbeat handling → node marked offline Mitigation: cap merge concurrency + reduce ingest pressure (+ fix disk tier if capped) ``` --- ## 5. Data disk vs system disk (field note; not a separate catalog line) ### Data volume full - **Signals:** watermark logs + high data-disk metrics. - **Mitigate:** Sections 1–2. ### Root / system volume full (rare, severe) - **Signals:** data disk OK but JVM/host misbehaves (cannot write temp files, logs). - **Common causes:** core dumps under `/var/crash` or `/tmp`, container image sprawl, huge system logs. - **Mitigate:** host cleanup by **SRE / infrastructure**; monitor root mount (e.g. `/` or `/dev/vda1`) in host monitoring, not only Elasticsearch data paths. --- ## 6. Disk issues during changes ### 6.1 High disk + change stuck (`activating`) Typical story: **disk pressure → user triggers restart → change hangs in `activating`**. 1. Relieve disk (delete / resize / temporary watermark relief per policy). 2. Let nodes come back cleanly. 3. If the change is still stuck, use [sop-activating-change-stuck.md](sop-activating-change-stuck.md) and escalate per your support model. ### 6.2 Recovery bandwidth too high → node / FS hang **Pattern** (cold-tier resize case): new nodes cannot join; shell `cd` into `logs` hangs; cluster stays Yellow/Red. ```text indices.recovery.max_bytes_per_sec set very high (e.g. 300MB/s) → exceeds disk SKU sustained throughput (e.g. ~250MB/s) → block layer stalls → filesystem appears hung → Elasticsearch cannot flush logs / data → bootstrap fails ``` **Mitigation:** 1. Roll back the change / restore service on surviving nodes if required. 2. Lower recovery bandwidth to a safe value for the disk class: ```bash PUT _cluster/settings { "persistent": { "indices.recovery.max_bytes_per_sec": "100mb" } } ``` 3. Retry the change in a **low-traffic** window. 4. If on a low baseline disk SKU, plan upgrade to a higher baseline (e.g. ESSD PL1+) per product guidance. --- ## Appendix: Quick commands (disk) ```bash GET _cat/allocation?v GET _cat/nodes?v&h=name,ip,disk.used_percent,disk.avail,disk.total GET _cat/indices?v&s=store.size:desc DELETE /index_name GET _all/_settings?filter_path=*.settings.index.blocks PUT _all/_settings { "index.blocks.read_only_allow_delete": null } PUT _cluster/settings { "transient": { "cluster.routing.allocation.disk.watermark.low": "95%", "cluster.routing.allocation.disk.watermark.high": "98%" } } PUT _cluster/settings { "transient": { "cluster.routing.allocation.disk.watermark.low": null, "cluster.routing.allocation.disk.watermark.high": null, "cluster.routing.allocation.disk.watermark.flood_stage": null } } PUT /large_index/_settings { "index.number_of_replicas": 0 } PUT _cluster/settings { "transient": { "indices.merge.scheduler.max_thread_count": 1 } } ``` FILE:references/sop-memory-gc.md # SOP: JVM memory, GC, and circuit breakers **Covers:** `ResourceMonitor.OldGenMemoryWarning` (P0), `ResourceMonitor.OldGenMemoryCritical` (P0), `ResourceMonitor.MemoryRapidGrowth` (P1), `ResourceMonitor.OldGCFrequent` (P1), `ResourceMonitor.GCTimeRatioHigh` (P0), `HighAvailability.CircuitBreakerTriggered` (P0), `ResourceMonitor.OOM` (P0) CMS / catalog names may differ — align thresholds with [health-events-catalog.md](health-events-catalog.md). --- ## Diagnosis decision tree ``` NodeHeapMemoryUtilization (example) ├── avg > ~95% → OldGenMemoryCritical (P0); OOM imminent ├── avg > ~85% → OldGenMemoryWarning (P0); elevated OOM risk └── rises > ~30% within ~30m → MemoryRapidGrowth (P1); trend alert JVMGCOldCollectionCount (per minute) └── max > ~5 / min → OldGCFrequent (P1) JVMGCOldCollectionDuration └── GC time / wall time > ~30% → GCTimeRatioHigh (P0) Master log keywords ├── OutOfMemoryError / OOM → ResourceMonitor.OOM (P0) ├── Data too large / CircuitBreakingException → circuit breaker (P0) └── Killed / OOM killer → OS killed the process ``` > **Headline priority (heap / GC vs breakers):** Cluster **Green** does not rule out **JVM pressure**. When rules or logs point to **fielddata**, **`CircuitBreakingException`**, or **artificially low `indices.breaker.*` limits** in **`_cluster/settings`** (**transient** and **persistent**), treat **settings + `_nodes/stats/breaker` (`tripped`) + query/mapping** as the **primary** story — not **Old GC frequency alone** (GC is often a **parallel** or **downstream** signal unless isolated as the driver). > **Per-node heap skew:** Uneven **heap percent** across data nodes should be read with **`_nodes/stats/breaker`**, **`_nodes/stats/indices/fielddata`**, **`_cat/shards`** / hot indices, and **query shape** — not **only** “bad shard counts” without breaker and workload context. > **Co-occurrence with `ThreadPool.WriteRejected` / bulk saturation:** If **`_nodes/stats/thread_pool`** shows **`write` / `bulk` `rejected`** and **`hot_threads`** or traffic points at **ingest**, do **not** headline **Old GC alone** ahead of the write path — use the **causal chain or dual-P0 ordering** in [sop-write-performance.md](sop-write-performance.md) **§2** (*Evidence interpretation: bulk QPS → write pool*). Old GC / `GCTimeRatioTooHigh` remains **P0-class** but is often **downstream** of indexing + merges + retries. > **Collector names in prose:** Alibaba Cloud ES 7.x often ships **JDK 11** with **G1**, but the **actual** collector is **JVM / image dependent**. Do **not** assert **G1** vs **CMS** from version alone — confirm via **`GET _nodes/stats/jvm`** (`gc.collectors.*`) or node **GC logs**; otherwise write **“old-generation / Old GC”** or **“old GC (collector per JVM stats)”**. --- ## 1. Old-gen memory warning (`OldGenMemoryWarning`, ~85%+) ### Act first (before deep analysis) **1. Clear caches (often fastest)** ```bash POST _cache/clear POST _cache/clear?fielddata=true # Or per index POST /index_name/_cache/clear?fielddata=true&query_cache=true&request_cache=true ``` **2. Heap snapshot from the API** ```bash GET _nodes/stats/jvm?filter_path=nodes.*.jvm.mem,nodes.*.name # heap_used_percent, old_gen_used_bytes (names vary by version) ``` **3. Fielddata footprint** ```bash GET _nodes/stats/indices/fielddata?fields=*&filter_path=nodes.*.indices.fielddata ``` ### Root causes **A — Fielddata too large** - **Signals:** slow logs show **aggregations on `text`** fields - **Effect:** fielddata stays on-heap until evicted / cleared - **Mitigate:** ```bash POST /index_name/_cache/clear?fielddata=true PUT _cluster/settings { "persistent": { "indices.fielddata.cache.size": "20%" } } ``` (`indices.fielddata.cache.size` behavior depends on version — confirm in your distribution.) - **Proper fix:** avoid **`text`** for aggs; use a **`keyword`** sub-field or `doc_values` where appropriate. **B — Huge aggregations** - **Signals:** hot threads in agg paths; long-running `_tasks` with search/agg - **Mitigate:** cancel runaway tasks; reduce cardinality, bucket count, `size`; consider `execution_hint: map` where supported **C — Oversized result sets** - **Signals:** very large `size`, deep `from + size` (e.g. > 10k) - **Mitigate:** cap `size`; use **scroll** or **search_after** **D — Very large `terms` queries** - **Signals:** `terms` with huge ID lists (tens of thousands); hot threads with `TermsQuery` / `TermInSetQuery` - **Effect:** large bitsets on heap (default `index.max_terms_count` is **65536**) - **Mitigate:** ```bash GET _tasks?detailed=true&actions=*search* POST _tasks/{task_id}/_cancel GET {index}/_settings?filter_path=**.max_terms_count PUT {index}/_settings { "index.max_terms_count": 10000 } ``` - **Proper fix:** **batch** terms lists (e.g. **< ~1000** per request) or use **`ids`** where applicable. > **Example:** tens of thousands of IDs in one `terms` query → high heap churn and GC → slow cluster. Fix: smaller batches + app-side chunking. **E — Undersized heap for the workload** - **Signals:** “normal” traffic but heap pegged - **Mitigate:** scale node RAM / heap per vendor guidance --- ## 2. Old-gen memory critical (`OldGenMemoryCritical`, ~95%+) ### Emergency sequence ```text 1. POST _cache/clear (all tiers you can afford to drop) 2. Cancel long searches: GET _tasks → POST _tasks/{id}/_cancel 3. Restart affected nodes if still unstable (plan brief outage) 4. After stabilization, find the recurring driver (queries, mappings, breakers) ``` **Cancel large searches** ```bash GET _tasks?detailed=true&actions=*search* POST _tasks/{task_id}/_cancel ``` **Restart** - Use console / automation for the specific node. - After restart, confirm `heapPercent` trends flat, not climbing endlessly. --- ## 3. Frequent old GC (`OldGCFrequent`, P1) ### Criteria (example) - `JVMGCOldCollectionCount` **max > ~5 / minute** ### Causes | Cause | Clues | Mitigation | |-------|-------|------------| | Old gen pegged (> ~75%) | heap trend | Sections 1–2 | | Large fielddata | node stats | clear + mapping fix | | Heavy aggs | slow logs | optimize / throttle | | G1 never “full” enough to reclaim certain structures | rare JVM / TL patterns | upgrade path or vendor-guided Full GC | **G1 / `ThreadLocal` stale-entry churn** G1 may not run a “classic” full GC often; some `ThreadLocal` / weak-reference cleanup paths need a full collection cycle. See **Pattern A** in [sop-cpu-load.md](sop-cpu-load.md) (single-node management thread hot + `expungeStaleEntry`). --- ## 4. High GC time ratio (`GCTimeRatioHigh`, P0) ### Criteria (example) - **Old GC duration / wall time > ~30%** - Users see slow responses; long STW pauses ### Chain ```text Old-gen pressure → frequent old-gen collections → long pauses → heartbeat lag → node leaves cluster → client timeouts / connection errors ``` ### Quick check ```bash GET _nodes/stats/jvm?filter_path=nodes.*.jvm.gc,nodes.*.name # collection_count, collection_time_in_millis ``` --- ## 5. Circuit breaker tripped (`CircuitBreakerTriggered`, P0) ### Symptoms - Logs like: ```text CircuitBreakingException[[parent] Data too large, data for [...] would be [XXX], which is larger than the limit of [YYY]] ``` - Requests fail with **HTTP 429** or **503** ### Breaker types | Breaker | Role | Setting | Typical default (JVM %) | |---------|------|---------|------------------------| | `parent` | Ceiling for breaker accounting | `indices.breaker.total.limit` | ~95% | | `fielddata` | Fielddata heap | `indices.breaker.fielddata.limit` | **~40%** (lowering below ~40% is risky) | | `request` | Single in-flight request structure | `indices.breaker.request.limit` | ~60% | | `in_flight_requests` | **All** in-flight HTTP request bytes | `network.breaker.inflight_requests.limit` | **~100%** | | `accounting` | Long-lived accounting | `indices.breaker.accounting.limit` | — | > **`request` vs `in_flight_requests` (important)** > - **`request`:** one heavy query (huge agg, large `size`, wide fields). Fix: **smaller query**, fewer buckets, better mapping. > - **`in_flight_requests`:** **aggregate** traffic from **many concurrent** requests. Fix: **lower client concurrency**, queueing, **more nodes** — **not** “split one big query into many small concurrent queries” (that usually **raises** concurrency and makes this worse). > - If the log shows **`[in_flight_requests]`**, do **not** default to “shard the query into more parallel clients.” > **Breaker limits in `_cluster/settings` (MUST when JVM/breaker rules fire):** Always pull **both** **transient** and **persistent** (and defaults when using `include_defaults=true`) — e.g. `GET _cluster/settings?include_defaults=true` and read **`indices.breaker.*`** / **`network.breaker.*`** (use `filter_path` if you need a smaller payload). Mis-tuning often appears as **`indices.breaker.fielddata.limit`** set to a **very small JVM fraction** (e.g. **~1–5%**); **`indices.breaker.request.limit`** is **frequently lowered at the same time** (e.g. **~5%**) — cite **both** when present. > **Severity:** A low **`fielddata.limit` alone** is a **configuration risk**. If **`_nodes/stats/breaker`** shows **`tripped` > 0** for **`fielddata`** / **`request`** / **`parent`**, or logs show **`CircuitBreakingException`**, elevate the narrative to **incident-grade** (typically **P0**-class alongside query relief) — **not** only a **P2 “potential risk”** footnote. > **`tripped` vs current heap:** **`tripped`** is a **cumulative** counter (trips **since JVM start**). A **large** **`parent.tripped`** (e.g. **183**) reflects **historical** breaker events and **past** heap pressure peaks — it can **coexist** with **moderate current heap** from **`_cat/nodes`**. When both appear in one report, add **one sentence** so readers do not read “**tripped** high” and “heap **~50%** now” as contradictory. > **Index-level closure:** Tie symptoms to **concrete indices** (`_cat/shards`, **`GET {index}/_mapping`**) — especially **`text` fields with `fielddata: true`**, **large `terms` / `cardinality` aggs**, or deep paging — so remediation is query- and mapping-specific. ```bash GET _cluster/settings?include_defaults=true # Inspect persistent, transient, and defaults for indices.breaker.* and network.breaker.* ``` If `fielddata` limit is **below ~40%** of JVM **without** a deliberate vendor exception, treat as misconfiguration and restore recommended values after change approval. ### Response steps **Step 1 — Breaker stats** ```bash GET _nodes/stats/breaker ``` **Step 2 — Map `name` to action** - **`fielddata`** → text-field aggs / huge fielddata → `POST _cache/clear?fielddata=true` - **`parent`** → overall pressure → `POST _cache/clear` + load relief - **`request`** → single huge request → rewrite query (size, aggs, fields) - **`in_flight_requests`** → too much **parallel** traffic: 1. **Throttle clients** / shrink pools / add queue depth server-side 2. **Scale out** nodes 3. Only if a **single** request is proven huge, tune that query — avoid fan-out that increases concurrency 4. Raising `network.breaker.inflight_requests.limit` is rarely the right first move (often already at JVM %) **Step 3 — Corroborate** ```bash GET _tasks?detailed=true # plus slow logs around the trip time ``` ### Example (e-commerce, coordinating node) - Traffic spike with heavy **agg** at ~15:00 - Heap already ~80% - **`parent`** trips - Mitigation: throttle + clear fielddata + larger nodes --- ## 6. OOM (`ResourceMonitor.OOM`, P0) ### Symptoms - `java.lang.OutOfMemoryError: Java heap space` in logs - Node may have restarted on its own - Host: `/var/log/messages` (or journal) for **OOM killer** lines ### Steps **Step 1 — Is the node back?** ```bash GET _cat/nodes?v&h=name,ip,heapPercent ``` **Step 2 — If not healthy, restart** the process / node per runbook. **Step 3 — After restart,** if heap climbs again immediately → sustained leak or abusive query pattern. **Step 4 — Heap histogram (when permitted on host)** ```bash jmap -histo:live {pid} | head -50 ``` ### Prevention - Enable **heap dump on OOM:** `-XX:+HeapDumpOnOutOfMemoryError` - Alert at **~85%** heap (`OldGenMemoryWarning` class) - **Very small** flavors (e.g. **2 vCPU / 4 GB**) are a poor fit for production Elasticsearch — overhead dominates. --- ## Appendix: Quick commands (memory / GC) ```bash GET _nodes/stats/jvm?filter_path=nodes.*.jvm.mem,nodes.*.name GET _nodes/stats/jvm?filter_path=nodes.*.jvm.gc,nodes.*.name GET _nodes/stats/indices/fielddata?fields=* GET _nodes/stats/breaker POST _cache/clear POST /index_name/_cache/clear?fielddata=true GET _tasks?detailed=true&actions=*search* POST _tasks/{task_id}/_cancel GET _cat/nodes?v&h=name,ip,heapPercent,heapCurrent,heapMax,ram.percent ``` FILE:references/sop-node-load-imbalance.md # SOP: Per-node load imbalance **Covers:** `HealthCheck.LoadUnbalanced` **Reason codes:** - `Balancing.NodeCPUUnbalanced` — uneven CPU - `Balancing.NodeTrafficUnbalanced` — uneven query / index traffic - `Balancing.NodeDataUnbalanced` — uneven data placement - `Balancing.NodeDiskUnbalanced` — uneven disk utilization - `Balancing.NodeMemoryUnbalanced` — uneven heap / memory pressure --- ## Diagnosis entry: imbalance decision tree ``` Three dimensions (CV = coefficient of variation = stddev / mean) │ ├── Traffic imbalance │ ├── Per-node NodeQPS CV > ~0.3 → uneven search load │ └── Per-node NodeIndexQPS CV > ~0.3 → uneven write load │ ├── Data imbalance │ ├── Per-node shard count CV > ~0.3 → shard skew (very common) │ ├── Per-node document count CV > ~0.5 → doc skew │ └── Per-node stored size CV > ~0.3 → storage skew │ └── Resource imbalance ├── Per-node NodeCPUUtilization CV > ~0.3 → CPU skew ├── Per-node NodeHeapMemoryUtilization CV > ~0.3 → heap skew └── Per-node NodeDiskUtilization CV > ~0.3 → disk skew ``` **Severity heuristic** - Resource skew **and** any hot node **> ~80%** → treat as **P0** (overload / failure risk) - Resource skew but hot nodes **< ~80%** → **P1** (latent risk) - Traffic / data skew only, resources healthy → **P2** (optimize when convenient) --- ## Time-series CV and peak windows > The diagnosis engine may compute several CV flavors so short spikes are not washed out by daily averages: | CV flavor | Meaning | Use | |-----------|---------|-----| | **CV (avg)** | CV of per-node **means** | Overall trend; can hide short spikes | | **CV (max)** | CV of per-node **maxes** | Catches peak-window skew | | **Peak-window CV** | CV across nodes at each timestamp; take **max** | Pinpoints the worst clock time | **Effective CV** ≈ `max(CV_avg, CV_max, peak_window_CV)`. If the report shows **peak-window CV ≫ average CV**, the skew is **time-localized** — inspect traffic / tasks in that window. --- ## 1. Shard-count skew (most common driver) > **Shard skew is the single most common upstream cause of uneven CPU / heap / disk — check it first.** ### 1.1 Quick commands ```bash GET _cat/nodes?v&h=name,ip,cpu,heap.percent,disk.used_percent,shards&s=shards:desc GET _cat/allocation?v # If one node owns far more shards than peers → placement skew ``` ### 1.2 Typical causes | Cause | How you know | Mitigation | |-------|--------------|------------| | New nodes not drained/rebalanced | new nodes have far fewer shards | enable rebalance; manual `reroute` | | Allocation rules | review `cluster.routing.allocation.*` | relax / fix filters | | Index created when cluster was tiny | old indices lopsided | reindex / shrink / reroute | | Hot indices packed on few nodes | `_cat/shards` shows concentration | move hot shards | ### 1.3 Rebalance / move ```bash PUT _cluster/settings { "transient": { "cluster.routing.rebalance.enable": "all" } } POST _cluster/reroute { "commands": [{ "move": { "index": "hot_index", "shard": 0, "from_node": "node_many_shards", "to_node": "node_few_shards" } }] } ``` > **`reroute` scope:** `POST _cluster/reroute` issues **allocator-respecting moves** — it is **not** a supported pattern to “flip primary vs replica by hand” just to **even out primary counts** for load. Durable fixes for shard / primary skew are usually **new index settings + reindex**, **shrink** (when applicable), **node add/remove + rebalance**, and fixing **allocation / awareness** rules. Use **`move`** only under **change control**, in a **maintenance window**, after **`_cat/shards`** / explain confirm the plan — **do not** treat ad-hoc primary/replica swaps as a routine performance knob. --- ## 2. Traffic imbalance ### 2.1 Trigger (example) - Per-node QPS / index QPS **CV > ~0.3** for **~10+ minutes** ### 2.2 Steps **Step 1 — Node-level traffic proxies** ```bash GET _cat/nodes?v&h=name,ip,cpu,load_1m,search,search.total,indexing GET _cat/thread_pool?v&h=node_name,name,active,queue,rejected&s=name ``` **Step 2 — Index-level hot spots** ```bash GET _cat/indices?v&h=index,pri,rep,docs.count,store.size,search.query_current,indexing.index_current&s=search.query_current:desc ``` **Step 3 — Shard placement for hot indices** ```bash GET _cat/shards/{hot_index}?v&h=index,shard,prirep,state,docs,store,node ``` ### 2.3 Causes and fixes | Cause | Clues | Mitigation | |-------|-------|------------| | No coordinating tier | data nodes take client traffic | add **coordinating** / client nodes | | Sticky clients | few nodes get all connections | load-balance / shuffle client endpoints | | Hot shards | shards of hot index on few nodes | reroute or increase shards (plan carefully) | | Custom `_routing` | doc skew + query skew | redesign routing / keys | | Multi-zone + awareness | coordinators per zone see uneven ingress | see **Section 2.5** | ### 2.4 Commands (illustrative) ```bash # After adding coordinators, point clients at the full coordinator pool POST _cluster/reroute { "commands": [{ "move": { "index": "hot_index", "shard": 0, "from_node": "node_hot", "to_node": "node_cool" } }] } ``` ### 2.5 Multi–availability-zone: uneven **coordinator** load **Background:** With `cluster.routing.allocation.awareness.attributes: zone`, primaries and replicas spread across zones. **How skew appears:** ```text Clients reach Elasticsearch via a VPC endpoint / proxy → the proxy prefers coordinators in the **same zone** as the client → if **clients** cluster in zone K, zone-K coordinators absorb most traffic ``` **Checks** ```bash GET _cluster/settings?filter_path=**.awareness* # e.g. awareness.attributes = zone GET _cat/nodeattrs?v&h=node,attr,value&s=attr # inspect zone attrs GET _cat/nodes?v&h=name,ip,cpu,load_1m,node.role&s=node.role # focus on nodes without `d` (no data role) if those are your coordinators ``` **Mitigations** | Option | Notes | When | |--------|-------|------| | Spread clients across zones | best if you control deploy topology | preferred | | Private connectivity to **all** coordinators | bypasses zone-sticky proxy | requires network design | | More coordinators in hot zones | when traffic mix cannot change | capacity fix | > **Example:** many app VMs in zone **K**; coordinators in **K / H / G**. Proxy sends K clients to K-only coordinators → K coordinators run hot. Fix: rebalance **clients** across zones or widen coordinator access. --- ## 3. Data imbalance ### 3.1 Trigger (example) - Shard count, document count, or stored-size **CV > ~0.3** (tune to your catalog) ### 3.2 Steps ```bash GET _cat/nodes?v&h=name,ip,disk.total,disk.used,disk.avail,disk.used_percent,shards GET _cat/allocation?v GET _cat/shards?v&s=node&h=index,shard,prirep,state,docs,store,node GET _cat/nodes?v&h=name,shards GET _cat/indices?v&h=index,pri,rep,docs.count,store.size&s=store.size:desc GET {index}/_settings?include_defaults=true&filter_path=**.number_of_shards,**.number_of_replicas ``` ### 3.3 Patterns #### Type A — Shard count skew **Signs:** one node has many more shards than others. **Mitigate:** rebalance + `reroute` as in **Section 1**. #### Type B — Doc count skew **inside one index** **Signs:** same index, shard doc counts differ a lot (`_cat/shards/{index}&s=docs:desc`). **Causes:** custom routing, uneven IDs / hashing. **Mitigate:** better routing key; `index.routing_partition_size` where appropriate; reindex if needed. ```bash GET _cat/shards/{index}?v&h=shard,prirep,docs,store,node&s=docs:desc ``` #### Type C — Stored size skew **Signs:** large gap in **disk used** per node. **Causes:** huge shards on few nodes; retention; snapshot footprint. ```bash GET _cat/allocation?v GET _cat/indices?v&h=index,store.size&s=store.size:desc ``` **Mitigate:** ```bash POST _cluster/reroute { "commands": [{ "move": { "index": "large_index", "shard": 0, "from_node": "node_full", "to_node": "node_empty" } }] } DELETE old_index_* ``` Optional: nudge disk routing — align with [sop-disk-storage.md](sop-disk-storage.md) before changing watermarks: ```bash PUT _cluster/settings { "transient": { "cluster.routing.allocation.disk.watermark.low": "85%", "cluster.routing.allocation.disk.watermark.high": "90%" } } ``` --- ## 4. Resource-level imbalance Cross-links: [sop-cpu-load.md](sop-cpu-load.md) (CPU / hot threads), [sop-memory-gc.md](sop-memory-gc.md) (fielddata / breakers), [sop-disk-storage.md](sop-disk-storage.md) (disk / flood). ### 4.1 CPU skew **Trigger (example):** per-node CPU **CV > ~0.3** and max node **> ~10%** (sanity floor). ```bash GET _cat/nodes?v&h=name,ip,cpu,load_1m,load_5m,load_15m&s=cpu:desc GET _nodes/hot_threads GET _tasks?detailed=true GET _cat/tasks?v ``` | Cause | Clues | Mitigation | |-------|-------|------------| | Shard skew | hot node has more shards | reroute | | Hot segments inside shards | even shard counts but skewed CPU | `forcemerge` off-peak (see ES docs); see [sop-cpu-load.md](sop-cpu-load.md) | | Missing coordinators | one node coordinates most queries | add coordinators | | Monster queries | hot threads = search | optimize / cancel tasks | ```bash POST {hot_index}/_forcemerge?max_num_segments=1 ``` Use **forcemerge** only with maintenance windows — IO heavy. ### 4.2 Heap skew **Trigger (example):** heap **CV > ~0.3**. ```bash GET _cat/nodes?v&h=name,ip,heap.percent,heap.current,heap.max,ram.percent&s=heap.percent:desc GET _nodes/stats/indices?human&filter_path=nodes.*.indices.fielddata,nodes.*.indices.query_cache,nodes.*.indices.segments GET _cat/fielddata?v&s=size:desc ``` | Cause | Clues | Mitigation | |-------|-------|------------| | Fielddata skew | `_cat/fielddata` dominated by one node | fix aggs; clear caches — [sop-memory-gc.md](sop-memory-gc.md) | | Query cache | large `query_cache` on some nodes | tune cache limits / queries | | Segment memory | `segments` stats skewed | merge policy / forcemerge | ### 4.3 Disk skew **Trigger (example):** disk **CV > ~0.3**. ```bash GET _cat/nodes?v&h=name,ip,disk.total,disk.used,disk.avail,disk.used_percent&s=disk.used_percent:desc GET _cat/allocation?v GET _cat/indices?v&h=index,store.size&s=store.size:desc GET _cat/shards/{large_index}?v&h=shard,prirep,store,node ``` **Mitigate:** reroute large shards first. Watermark tuning **only** with governance — defaults and danger cases are in [sop-disk-storage.md](sop-disk-storage.md). ```bash PUT _cluster/settings { "persistent": { "cluster.routing.allocation.disk.watermark.low": "80%", "cluster.routing.allocation.disk.watermark.high": "85%", "cluster.routing.allocation.disk.watermark.flood_stage": "90%" } } POST _cluster/reroute { "commands": [{ "move": { "index": "large_index", "shard": 0, "from_node": "node_disk_full", "to_node": "node_disk_free" } }] } ``` --- ## 5. End-to-end workflow ### 5.1 One-pass command bundle ```bash GET _cat/nodes?v&h=name,ip,cpu,heap.percent,disk.used_percent,load_1m,shards&s=cpu:desc GET _cat/allocation?v GET _cat/thread_pool?v&h=node_name,name,active,queue,rejected&s=queue:desc GET _cat/indices?v&h=index,pri,rep,docs.count,store.size&s=store.size:desc ``` ### 5.2 Priority ```text 1. [Urgent] Resource skew + any node > ~85% → reroute / relieve hot nodes now 2. [High] Traffic skew + any node CPU > ~70% → coordinators / hot shard moves 3. [Normal] Data skew only, resources OK → plan rebalance; not always urgent 4. [Watch] Mild skew (CV < ~0.5, resources < ~60%) → document and monitor ``` ### 5.3 Causal sketch ```text Traffic skew ├── no coordinators → data nodes take client work → CPU skew ├── sticky clients → few nodes get connections → CPU skew └── hot shards → queries hit same nodes → CPU / IO skew Data skew ├── allocation rules → new nodes underfilled → disk skew ├── custom routing → doc hotspots → shard hotspots └── huge indices localized → disk / IO skew Resource skew ├── shard count skew → uneven work → CPU / heap skew ├── hot segments → query concentration → CPU skew └── uneven fielddata → heap skew ``` --- ## Appendix: Quick commands (imbalance) ```bash GET _cat/nodes?v&h=name,ip,cpu,heap.percent,disk.used_percent,load_1m,shards&s=cpu:desc GET _cat/allocation?v GET _cat/shards?v&s=node&h=index,shard,prirep,docs,store,node GET _cat/thread_pool?v&h=node_name,name,active,queue,rejected GET _cat/nodes?v&h=name,search,indexing GET _cat/fielddata?v&s=size:desc GET _nodes/stats/indices?human&filter_path=nodes.*.indices.fielddata GET _nodes/hot_threads PUT _cluster/settings { "transient": { "cluster.routing.rebalance.enable": "all" } } POST _cluster/reroute { "commands": [{ "move": { "index": "index_name", "shard": 0, "from_node": "node_from", "to_node": "node_to" } }] } PUT _cluster/settings { "persistent": { "cluster.routing.allocation.disk.watermark.low": "80%", "cluster.routing.allocation.disk.watermark.high": "85%" } } ``` FILE:references/sop-query-thread-pool.md # SOP: Query performance and thread pools **Covers:** `ThreadPool.SearchRejected` (P0), `ThreadPool.SearchQueueHigh` (P0), `Performance.SearchLatencyHigh` (P1), `Performance.LongRunningTask` (P1) *(Frequency notes in parentheses refer to internal case statistics — tune to your catalog.)* --- ## Diagnosis decision tree ``` Query symptoms ├── HTTP 429 on search → SearchRejected (pool reject) ├── High search queue → SearchQueueHigh (backlog) ├── Latency above SLO → SearchLatencyHigh └── Task running > ~10m → LongRunningTask Key signals ├── GET _nodes/stats/thread_pool → search.queue / search.rejected ├── GET _tasks?detailed=true → long-running search tasks └── SEARCHSLOW logs → slow query bodies / timings ``` > **Co‑stress vs strict causality:** Wide / high-QPS searches (e.g. `match_all` on a large index) typically **raise CPU, search queue/rejections, and GC pressure in the same window**. Describe these as **co-occurring or mutually reinforcing** under one workload — not a fixed chain like “rejected → therefore GC rose first,” unless timestamps prove ordering. > **CPU rubric alignment:** When **`_tasks`** / slow logs show a hot index + heavy `*search*`, state explicitly that **NodeCPU / `hot_threads` and `search` pool saturation are the same overload story** (see [sop-cpu-load.md](sop-cpu-load.md)). If CMS or the rule catalog already fires **CPU at P0** and/or **`ThreadPool.Search*` at P0** ([health-events-catalog.md](health-events-catalog.md)), keep the **headline severity** aligned — e.g. **“resource overload (CPU + search path)”** — rather than narrating only secondary signals as **P1-only**. > **CMS time alignment:** Add **one sentence** on whether **`NodeCPUUtilization` (or equivalent CMS window)** overlaps the period of **`search` queue / reject** pressure — including the case where **CPU mean is no longer high** but **CPU peaked in the same window** as rejects (common after the queue drains). --- ## Report narrative: search pool vs GC / CPU headlines **Audience:** Authors of **human-written** diagnosis summaries (and agent prose). This section does **not** change CMS thresholds or rule engines; it fixes **headline ordering** and **explicit evidence gaps** when **read-path** overload is plausible. Aligns with [acceptance-criteria.md](acceptance-criteria.md) section **6.2** (read-heavy CPU + search pool) and section **6.5** (search pool vs GC-only headline). For write-path vs GC ordering, see [sop-write-performance.md](sop-write-performance.md) and acceptance-criteria section **6.4**. ### Query-bound overload: what to lead with - When the workload is **read-heavy** (high concurrent search, terms / high-QPS patterns, **`ThreadPool.SearchRejected`** or queue stress), the **primary** storyline should be **`search` pool saturation / query concurrency exceeding pool capacity** — **not** a **GC-only** headline. **`ThreadPool.SearchRejected`** (or catalog-equivalent) should appear **first or co-equal** with resource signals. - **Old GC + CPU spikes** often **co-occur** or form a **second wave** with concurrent search and segment merge load — they are **not** wrong as evidence, but **misleading as the sole lead** when the incident is **query-bound** and **`thread_pool.search` has not been ruled out**. - **Production caveat:** The same CMS signals can still support a **legitimately GC-primary** or **breaker-primary** story if engine APIs show **heap / breaker / mapping** issues **without** search-pool saturation. Separate **what CMS proved** from **what still needs** `GET /_nodes/stats/thread_pool` (or `_cat/thread_pool`). ### When engine REST evidence is incomplete If **`GET /_nodes/stats/thread_pool`** or **`_cat/thread_pool`** was **not** successfully collected (timeouts, unstable public path, firewall), but **GC / CPU** findings exist from **CMS or the rule engine**: **Suggested wording (Chinese — adapt to facts):** > 在引擎 API 未完整采集前：若规则引擎或监控已出现 **`ThreadPool.SearchRejected`** 或 **线程池饱和类事件**，应优先按 **「查询并发打满 `search` 池」** 排查；当前因超时/网络未能用 **`/_nodes/stats/thread_pool`** 证实 **`search.rejected` / queue**，以下 **GC / CPU** 为 **监控侧风险画像**，需在 **VPC 内或白名单稳定路径** 上补 **`thread_pool.search`**、**`/_tasks`**、**`/_nodes/hot_threads`** 后再定主次因果。 If **`ThreadPool.SearchRejected`** (or equivalent) **did not** fire in the **same window**, still add **one explicit line** that **`search` pool reject/queue remains unverified** (do not invent rejects): > **待补证：** **`search` 池是否拒绝或排队**（读路径过载时通常为首要排查点之一），需引擎层 **`/_nodes/stats/thread_pool`** 或 **`_cat/thread_pool`**。 Self-limiting discipline for failures: [es-api-call-failures.md](es-api-call-failures.md). ### Executive summary: parallel P0 / P1 lines - If the rule engine reports **`JVMMemory.GCTimeRatioTooHigh` (P0-class)** **and** **`ThreadPool.SearchRejected`** (exact priority depends on catalog; some snapshots emit P1), the **human-facing** summary should still **mention both** when the catalog treats search rejects as incident-class. - Opening bullets must **not** read as **“the only P0 is Old GC”** when **search-path** rules also fired. Order: **search pool / reject / query concurrency** **first or co-equal**; GC/CPU as **co-stress, cascade, or pending confirmation** after `thread_pool` + `_tasks`. **Quantification (mirrors [sop-write-performance.md](sop-write-performance.md) §2):** For rubric-style reports, add **one line of numbers** from **`GET _nodes/stats/thread_pool`**: per-node **`search.rejected`** and **`search.completed`**, optional **reject share** (roughly `rejected / (rejected + completed)`). **`rejected` is cumulative** since node start (see §1 **Inspect**). When aligning to CMS, cite rule/event names (e.g. **`ThreadPool.SearchRejected`**) alongside API stats. **Both `search` and `write` show `rejected` (cumulative):** Compare **magnitudes** on the **same node(s)**. When **`search.rejected` ≫ `write.rejected`** and the case is **query-saturation** (high QPS search, **`ThreadPool.SearchRejected`** in scope), the **first storyline** is **`search` pool / query concurrency** — link to **terms / wide query / slow query**, **`_tasks`** / **`hot_threads`**, and **hot index name** when confirmed. **`write.rejected`** stays **valid** as **secondary or parallel** (bulk backfill, catch-up indexing); **do not** open with **write** ahead of **search** in the **executive summary** or **first P0 bullet** unless **timelines** or **paired deltas** show **write** was the dominant stressor in the window. See [acceptance-criteria.md](acceptance-criteria.md) **§6.5** (*P0 / executive order vs `search` ≫ `write`*). **Different nodes in slow logs vs pool-rejection logs:** A **SEARCHSLOW** / **fetch** entry may attribute latency to **node A** while an **INSTANCELOG** line at **another timestamp** shows **`search` pool** pressure on **node B** — not necessarily a mistake: **fetch** runs where the **shard copy** lives; **pool saturation** can concentrate on the **busiest data node** for that window; **routing** and **replica** choice shift over time. **Reconcile** with **`GET /_cat/shards/{index}`** and **time ordering**; add **one explicit sentence** in the report so reviewers do not treat the two as contradictory. See [acceptance-criteria.md](acceptance-criteria.md) **§6.2** (*Slow-log node vs search-pool node*). ### Customer-facing references to SKILL / engine APIs | Audience | Preferred phrasing | |----------|---------------------| | **External / 对客** | **引擎层必查清单（见 SKILL 文档第 5 节）**、**第 5–7 节** — avoid **§**-style section markers in PDFs or customer email. | | **Internal** | **“SKILL.md section 5”** in prose is fine without the **§** symbol. | The health-check script’s footer uses the same **no-§** convention. ### Evidence closure checklist 1. **Rules / CMS:** Did **`ThreadPool.SearchRejected`** (or a metric/event with the same meaning) fire in the report window? State **yes / no** explicitly. 2. **Engine (stable path):** **`GET /_nodes/stats/thread_pool`** with **`filter_path`** for **`search`**; **`GET /_tasks?detailed=true&actions=*search*`** when CPU or reject signals exist. 3. **Narrative:** If (1) is yes or engine shows **`search.rejected` / queue**, do **not** deliver a **GC-only** conclusion for a **read-heavy** incident — tie **GC/CPU** to **co-occurring search load** when evidence supports it (see **Co‑stress** note at the top of this document). --- ## 1. Search thread-pool queue buildup (`SearchQueueHigh`, P0) ### Inspect ```bash GET _nodes/stats/thread_pool?filter_path=nodes.*.thread_pool.search,nodes.*.name # queue = current depth; rejected = cumulative rejects (since node process start unless reset) ``` **Counter semantics:** `search.rejected` is **cumulative** for the node process (until restart). Do **not** treat the current value as “rejects only inside the diagnosis window” unless you have a **delta** between two samples or a known reset. **`queue == 0`** can still follow a burst where **`rejected` increased** — the pool drained after the spike. ### Root causes **A — Slow queries holding workers** - **Signals:** few tasks tie up `search` workers; queue grows behind them - **Check:** `GET _tasks?detailed=true&actions=*search*` - **Mitigate:** `POST _tasks/{task_id}/_cancel`, then fix the query **B — QPS exceeds capacity** - **Signals:** queue grows roughly linearly with traffic - **Mitigate:** throttle / shed load at the client or gateway; **scale out** nodes **C — CPU starved** - **Signals:** high CPU **and** high search queue - **Mitigate:** [sop-cpu-load.md](sop-cpu-load.md) ### Note: search queue size is not a tuning knob On typical Elasticsearch builds the **search** queue has a **fixed** upper bound (often **1000**). Unlike some write-side pools, you usually **cannot** “fix” saturation by raising the search queue — you must **reduce work per query**, **reduce concurrency**, or **add capacity**. --- ## 2. Search thread-pool rejections (`SearchRejected`, P0) ### Symptoms - Clients: **`HTTP 429 Too Many Requests`** on search paths - Logs: `rejected execution of search` (wording varies by version) ### `_cat/thread_pool` vs `_nodes/stats/thread_pool` Either **`GET _cat/thread_pool`** or **`GET _nodes/stats/thread_pool?filter_path=...search...`** is valid engine evidence for the **`search`** pool. For **`rejected`**, both surface the **same node-level cumulative semantics** (see §1 counter note); use one or both if reviewers ask for **source parity**. ### Immediate response **1. Capture what is running (while the incident is live)** ```bash GET _nodes/hot_threads GET _tasks?detailed=true&actions=*search* ``` **2. Cancel runaway searches** ```bash POST _tasks/{task_id}/_cancel ``` **3. Check gateway / vendor QoS (if applicable)** ```bash # Alibaba Cloud Elasticsearch — QoS limiter metrics (if exposed in your build) GET _qos/limiter/metric ``` ### Node skew (`rejected` higher on one node) Uneven **`search.rejected`** can come from **one client entry path**, shard / routing skew, or **query coordination** landing more often on certain nodes. The elected master participates in **coordination / routing** (and may receive client searches depending on topology) — it does **not** imply “the master executes all shard searches.” Treat one hot node as a **lead**, then confirm with **`_cat/shards`**, **`_tasks`**, and node roles. **Hot node + layout (wording):** When **`_cat/shards`** shows **extreme primary/replica skew** and **`search.rejected`**, **heap**, or **GC** concentrate on the **same node**, treat that node as the **hot node** in the narrative — the chain can be **internally consistent**. Describe **skew** as **one amplifier of local engine work** on that side (**merge / refresh / indexing** when those paths are active), **together with** read-side concentration (**coordinator / single client URL / `preference=_primary` / routing**, time-skewed peaks). **Coordinator + ingress can still bias index-level work** toward one node even when replicas exist — this is **not** the same as “all reads hit primaries only.” **Do not** imply that **default read traffic** is routed **only** to primaries; **replicas serve queries** unless clients opt out. **`_cat/shards` + per-node `rejected`** stay **facts**; the bullets above are **hypotheses to narrow** with `_tasks`, client settings, and query parameters. **Placement wording:** Use **one convention** everywhere — e.g. **“all primaries on {node}”**, **“6 / 6 primaries on {node}”**, or **“every `p` row on {node}”** — avoid mixing **“almost all”** in the title with **“all”** in tables unless you document a real exception shard. **CPU vs `rejected` on different nodes:** Per-node **CPU max** and **rejected** need not align at the **same instant** (peaks can occur at **different minutes**), and one node may host **other hot indices or system work**. When CMS shows **higher CPU on node X** but **higher `search.rejected` on node Y**, say so explicitly — avoid implying “the hottest CPU node must always be the highest-reject node” for the whole window. --- ## 3. Slow queries (`Performance.SearchLatencyHigh`, P1) ### Three-step triage **Step 1 — Search Profiler** Use Kibana Search Profiler, or add `"profile": true` to the search body: ```bash GET /index_name/_search { "profile": true, "query": { "match": { "field": "value" } } } ``` Inspect shard query breakdown (`shards.*.query`, etc.) for dominant query types. **Step 2 — Slow logs** Filter `SEARCHSLOW` (or equivalent) for high `took` / `search_time_ms` (e.g. **> 1000 ms** as a starting filter — align to your SLO). **Step 3 — Baseline** - Versus yesterday / last week same window? - All queries slow, or only certain indices / patterns? --- ## 4. Common slow query patterns and mitigations ### 4.1 Wildcard-style queries (often worst) - **Issue:** `wildcard`, `prefix`, `regexp` can scan huge posting lists. - **Worst:** leading wildcards (e.g. `*foo`) — effectively full-term scans. - **Mitigations:** - `prefix` → **completion** suggester or **edge n-gram** indexing where appropriate - `wildcard` → structured filters / `match` where possible - `regexp` → narrower `term` / `terms` / keyword patterns where possible ### 4.2 Deep paging (`from` + `size`) - **Issue:** each shard must fetch **`from + size`** ordered hits; cost scales with shards. - **Guard:** default **`index.max_result_window` is 10000** (`from + size` cap). - **Mitigations:** - Live paging → **`search_after`** - Export / scan → **`scroll`** or **PIT + search_after** (modern ES) - Avoid raising `max_result_window` as a “fix” — it hides the real cost ### 4.3 Aggregations - **Issue:** high-cardinality `terms` aggs (e.g. raw `user_id`, `ip`) burn CPU / heap. - **Mitigations:** - Lower **`size`** (return fewer buckets) - **`execution_hint: map`** where it fits **low-cardinality** workloads (check version docs) - Never aggregate on **`text`** — use **`keyword`** (or `doc_values`) - Prefer indexed fields over heavy **`script`** aggs - **Never aggregate on `_id` or `text` for production traffic** — fielddata / heap explosions: [sop-memory-gc.md](sop-memory-gc.md) ### 4.4 Script queries - **Issue:** scripts bypass most index structures. - **Mitigations:** precompute at index time; **Painless** (not legacy Groovy); avoid hot loops in scripts. ### 4.5 Numeric `term` queries - **Issue:** building doc-id sets for numeric `term` can be expensive at scale. - **When:** you only need exact match, not range / sort on that numeric. - **Mitigation:** add a **`keyword`** (or string) parallel field for equality filters. ### 4.6 `nested` / `parent-child` - **Issue:** nested docs multiply work; parent-child is often **several×** heavier than nested. - **Mitigation:** model as **`object`** / flattened structures when business rules allow. ### 4.7 Highlighting - **Issue:** huge **`fragment_size`** → CPU / memory on highlight. - **Mitigation:** smaller fragments or highlight off-cluster. ### 4.8 Huge time ranges - **Issue:** scanning years of data every request. - **Mitigation:** bound time filters; batch by window. ### 4.9 Unbounded `range` queries - **Issue:** e.g. `{"range": {"date": {"lte": "2024-01-01"}}}` with **no lower bound** — cost grows forever with data. - **Mitigation:** always set **both** bounds when possible. ### 4.10 Massive hit counts - **Issue:** very broad query → **`hits.total`** in the millions; even `size: 10` can be expensive if every shard scores huge sets. - **Mitigations:** tighten filters; **`track_total_hits: false`** or a capped integer; existence checks → **`size: 0`** + **`terminate_after`** where appropriate. ### 4.11 `refresh_interval` vs query cost - **Issue:** very low **`refresh_interval`** (e.g. **1s**) → many small segments → more segment files to query / more merge pressure on small nodes. - **Inspect:** ```bash GET {index}/_settings?filter_path=**.refresh_interval GET _cat/segments/{index}?v&h=index,shard,segment,size ``` Many small segments per shard (rule-of-thumb **> ~50** per shard) suggests excessive refresh / merge pressure — validate against your workload. | Scenario | Suggested `refresh_interval` | Notes | |----------|------------------------------|-------| | Logs / high ingest, relaxed search freshness | **30s–60s** | Fewer segments, better bulk throughput | | Near-real-time dashboards | **5s–10s** | Balance freshness vs cost | | Bulk load window | **`-1`** (disabled) | After bulk: `POST {index}/_refresh`, then restore interval | | Small nodes (e.g. 2c4g / 4c8g) | **≥ 30s** typical | Avoid 1s refresh on tiny flavors | ```bash PUT {index}/_settings { "index.refresh_interval": "30s" } PUT {index}/_settings { "index.refresh_interval": "-1" } POST {index}/_refresh PUT {index}/_settings { "index.refresh_interval": "30s" } ``` --- ## 5. Long-running tasks (`LongRunningTask`, P1) | Task kind | Duration | Notes | |-----------|----------|-------| | `delete_by_query` | can exceed **30m** on large data | prefer off-peak | | `reindex` | hours possible | tune `scroll_size` / slices per docs | | `_update_by_query` | same class as delete_by_query | same cautions | | Snapshot / restore | depends on data + network | usually expected; watch progress | | Scroll | leaks if clients omit `clear_scroll` | monitor open contexts | ### Inspect / cancel / scroll hygiene ```bash GET _tasks?detailed=true&nodes=* GET _tasks/{task_id} POST _tasks/{task_id}/_cancel GET /_nodes/stats/indices/search?filter_path=nodes.*.indices.search.scroll_* DELETE /_search/scroll { "scroll_id": ["scroll_id_1", "scroll_id_2"] } ``` --- ## Appendix: Quick commands (query) ```bash GET _cat/thread_pool?v&h=name,node_name,active,queue,rejected GET _nodes/stats/thread_pool?filter_path=nodes.*.thread_pool.search,nodes.*.name GET _nodes/hot_threads GET _tasks?detailed=true&actions=*search* POST _tasks/{task_id}/_cancel # Profiling: use a JSON body (not inline on the GET line) GET /index_name/_search { "profile": true, "query": { "match_all": {} } } GET _nodes/stats/indices/search?filter_path=nodes.*.indices.search # query_total, query_time_in_millis, fetch_total, fetch_time_in_millis ``` FILE:references/sop-service-avalanche.md # SOP: Node “unavailable” vs search-pool meltdown (service avalanche) **Covers:** `ServiceAvalanche.SearchServiceDown` (P0), `ServiceAvalanche.AllShardsFailed` (P0), `ServiceAvalanche.CPUInducedUnavailability` (P0) > **Why this SOP exists:** It targets **“the node looks dead but the process is still up.”** CMS may **not** increment `ClusterDisconnectedNodeCount`, and you may **not** see `NODE_LEFT` / `removed` in `INSTANCELOG` — the usual **node-offline** playbook fails. You need **cross-signals** (CMS CPU still reporting + search pool rejects + `all shards failed`) to pin **CPU overload → search thread-pool exhaustion → cascading search failures**. --- ## Diagnosis entry: “node offline” vs meltdown ``` User: "node offline / ES unavailable" OR intermittent ES API timeouts ├── Control plane DescribeInstance.status │ ├── active → control plane OK → continue to engine │ ├── activating → change in flight (scale/restart/upgrade); wait or see sop-activating-change-stuck.md │ └── inactive / invalid → platform issue → open a ticket with your cloud provider │ ├── CMS NodeCPUUtilization still reporting? │ ├── Yes → 【process likely alive】 at OS level │ │ ├── CPU > ~80% sustained → ★ ServiceAvalanche (core scenario in this SOP) │ │ └── CPU normal but API bad → network / security group / allowlist │ └── Gaps / no points → 【process may be dead】 │ ├── INSTANCELOG "removed" / "left" → real node departure → [sop-cluster-health.md](sop-cluster-health.md) │ └── CMS SystemEvent node events → [sop-cluster-health.md](sop-cluster-health.md) │ ├── ES API call pattern (during diagnosis) │ ├── All calls time out consistently → often network / allowlist (not avalanche) │ ├── Intermittent success + timeout + CPU > ~80% → ★ suspected avalanche │ └── Connection refused → process/port down → [sop-cluster-health.md](sop-cluster-health.md) │ └── INSTANCELOG contains "all shards failed"? ├── Yes + high CPU → ★ avalanche (strong) ├── Yes + normal CPU → often allocation / shard issues → [sop-cluster-health.md](sop-cluster-health.md) └── No → other paths (query, gateway, client bugs) ``` --- ## 1. Core scenario: CPU overload → search pool meltdown ### What “service avalanche” means here **Definition:** OS process is up, CMS still shows points, but **search is effectively unusable** — searches fail with `SearchPhaseExecutionException: all shards failed` (wording varies). To the app this feels like **node down**. **Mechanism:** Not “node left cluster,” but **search worker pool saturation + queue overflow + timeouts + cascade**. ### When to call it confirmed (example) | Signal | Example threshold | CMS / logs | |--------|-------------------|------------| | Sustained CPU | avg **> ~80%** for **≥ ~10 min** | `NodeCPUUtilization` | | Search rejects | `rejected` rising | `SearchThreadpoolRejected` | | Cascade in logs | burst of **all shards failed** | `INSTANCELOG` | ### vs true node loss | | Real node offline | Service avalanche (this SOP) | |---|-------------------|------------------------------| | Control-plane status | may go `activating` | often stays **active** | | CMS CPU series | **stops** or is missing | **continues** (often **80%+**) | | CMS `ClusterStatus` | may go Red | may stay Green / sparse | | INSTANCELOG | `node left` / `removed` | **no** clean leave | | INSTANCELOG | no mass `all shards failed` | **many** `all shards failed` | | `ClusterDisconnectedNodeCount` | **> 0** | often **0** | | REST `_cluster/health` | sometimes **refused** | often **timeouts** (intermittent) | | Call pattern | all fail same way | **some succeed, some time out** | > **Key discriminator:** **CPU metrics still arriving** from the node, at high values, **plus** `all shards failed` → treat as avalanche, not “host gone.” **Intermittent** timeouts + high CPU support overload; **uniform** timeouts lean network/ACL; **refused** leans dead process/port. --- ## 2. Entry signal: intermittent Elasticsearch API timeouts > **When:** During diagnosis, REST calls **sometimes** return, **sometimes** time out, while CMS shows **CPU > ~80%**. ### Why intermittent timeouts fit overload - TCP can connect; process listens. - Internal **`search` pool** is saturated → work queues → latency blows up. - Some requests finish before the client timeout; others do not → **jittery** success/failure. - That pattern differs from **“every call times out the same way”** (often path/network). ### Branching checklist ``` Intermittent timeouts ├── CMS NodeCPUUtilization (last ~10m) │ ├── Sustained > ~80% → highly suspicious → gather evidence in Section 3 Step 2 │ ├── < ~50% → deprioritize avalanche; look at network instability │ └── sparse points → tighten CMS window / resolution; re-sample ├── DescribeInstance.status │ ├── active → continue engine analysis │ └── not active → not this avalanche path └── INSTANCELOG "rejected" / "all shards failed" ├── rejected lines → pool overflow (strong) └── none yet → capture hot_threads in brief windows when API responds ``` ### Collection tactics when the API is flaky 1. **Longer client timeouts:** e.g. `--connect-timeout 20 --max-time 60`. 2. **Light calls first:** `_cat/nodes`, `_cluster/health`. 3. **Retry heavy calls** (`_nodes/hot_threads`, `_nodes/stats/thread_pool`) up to **~3** times. 4. **Log success ratio** as a severity side-channel. --- ## 3. Four-step diagnosis ### Step 1 — Is the process actually gone? **Goal:** separate **real** departure from **meltdown**. ```bash # 1) Control plane aliyun elasticsearch DescribeInstance --region --InstanceId # status active → usually OK at CP layer # 2) CMS CPU last ~5 minutes (points exist?) aliyun cms DescribeMetricList \ --Namespace "acs_elasticsearch" \ --MetricName "NodeCPUUtilization" \ --Dimensions '[{"instanceId":""}]' \ --StartTime "<5_min_ago_ms>" \ --EndTime "" \ --Period 60 # Points on both data nodes → likely alive at process level # No points on a node → suspect real loss / agent gap # 3) Leave events in INSTANCELOG aliyun elasticsearch ListSearchLog \ --region --InstanceId \ --type INSTANCELOG \ --query "removed OR left OR disconnect" \ --beginTime "" --endTime "" # empty → no classic "node left" story ``` **Rule:** CMS CPU **present** + **no** `removed/left` → **not** classic offline → **Step 2**. CMS CPU **absent** + leave logs → **real offline** → [sop-cluster-health.md](sop-cluster-health.md). ### Step 2 — Prove the avalanche chain (CPU → pool → search failure) ```bash # 1) CPU trend (~30m) — when did it cross ~80%? aliyun cms DescribeMetricList \ --Namespace "acs_elasticsearch" \ --MetricName "NodeCPUUtilization" \ --Dimensions '[{"instanceId":""}]' \ --StartTime "<30_min_ago_ms>" --EndTime "" --Period 60 # 2) Search pool rejects aliyun cms DescribeMetricList \ --Namespace "acs_elasticsearch" \ --MetricName "SearchThreadpoolRejected" \ --Dimensions '[{"instanceId":""}]' \ --StartTime "<30_min_ago_ms>" --EndTime "" --Period 60 # 3) Write pool rejects (ingest also melting?) aliyun cms DescribeMetricList \ --Namespace "acs_elasticsearch" \ --MetricName "WriteThreadpoolRejected" \ --Dimensions '[{"instanceId":""}]' \ --StartTime "<30_min_ago_ms>" --EndTime "" --Period 60 # 4) "all shards failed" in INSTANCELOG aliyun elasticsearch ListSearchLog \ --region --InstanceId \ --type INSTANCELOG \ --query "shards" \ --beginTime "" --endTime "" ``` **Example “confirmed” triad:** 1. CPU avg **> ~80%** for **≥ ~10 min** 2. `SearchThreadpoolRejected` **> 0** (rising) 3. `INSTANCELOG` shows **`all shards failed`** burst ### Step 3 — Find the trigger (index / traffic / query) **3.1 Index from rejected / slow paths** ```bash aliyun elasticsearch ListSearchLog \ --region --InstanceId \ --type INSTANCELOG \ --query "rejected" \ --beginTime "" --endTime "" # path like "/my-index/_search" → hot index ``` **3.2 Thread-pool excerpt (`EsRejectedExecutionException`)** Example block (shape varies by version): ```text QueueResizingEsThreadPoolExecutor[ name = /search, pool size = 4, active threads = 4, queued tasks = 1005, queue capacity = 1000, task execution EWMA = 371μs ] ``` | Field | Healthy-ish | Meltdown-ish | Meaning | |-------|-------------|--------------|---------| | active | < pool | **= pool** | all workers busy | | queued | low | **> capacity** | queue overflow | | EWMA | low μs | high μs | slower per-task work | | completed | rising | stalls | throughput collapse | **3.3 `Caused by` under `all shards failed`** | `Caused by` | Meaning | Next | |-------------|---------|------| | `EsRejectedExecutionException` | pool saturated | CPU / QPS / capacity | | `TaskCancelledException` (parent cancelled) | timeout cascade | parent timeouts / load | | `CircuitBreakingException` | breaker | [sop-memory-gc.md](sop-memory-gc.md) | | `NodeNotConnectedException` | shard/node channel dead | real disconnect path → [sop-cluster-health.md](sop-cluster-health.md) | **3.4 Elasticsearch APIs (when REST is reachable)** ```bash curl -sS --connect-timeout 10 --max-time 30 \ -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ "http://${ES_ENDPOINT#http://}/_nodes/hot_threads?threads=3" curl -sS --connect-timeout 10 --max-time 30 \ -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ "http://${ES_ENDPOINT#http://}/_tasks?detailed=true&actions=*search*" curl -sS --connect-timeout 10 --max-time 30 \ -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ "http://${ES_ENDPOINT#http://}/_nodes/stats/thread_pool?filter_path=nodes.*.thread_pool.search,nodes.*.name" ``` Deeper query tuning: [sop-query-thread-pool.md](sop-query-thread-pool.md), [sop-cpu-load.md](sop-cpu-load.md). ### Step 4 — Report template ```markdown ## Conclusion: service avalanche (not physical node loss) **Instance:** {instance_id} ({region}) **Topology:** {node_count} × {spec} (search workers ≈ {pool_size} — verify with _cat/thread_pool) **Window:** {begin} ~ {end} ### Differentiation - Control plane: active - CMS CPU: continuous points (process likely alive) - Leave logs: none - Verdict: **meltdown**, not host-down ### Evidence chain 1. CPU inflection at {time}: {before}% → {after}% 2. Search rejects on {node}: {count} 3. `all shards failed` burst at {time}; indices: {index} 4. Root `Caused by`: {text} ### Chain sketch {trigger} → CPU ~{cpu}% for {duration}m → search pool full (active {active}/{pool}, queue {queue}/{cap}) → rejects ({n}) → parent cancels → shard failures cascade → `all shards failed` → user-visible outage ### Trigger hypothesis - Hot index: {index} - Driver: {QPS spike / slow query / load test} - EWMA: {a}μs → {b}μs ### Actions {concrete steps} ``` --- ## 4. Remediation ### Emergency (P0) **1) Cut read traffic to the hot index / cluster** (fastest) - Pause or slash client QPS; stop abusive load tests; route reads to a standby if you have one; enable gateway throttling. **2) Cancel huge searches** (when `_tasks` works) ```bash curl -sS -u "${ES_USERNAME}:${ES_PASSWORD}" \ "http://${ES_ENDPOINT#http://}/_tasks?detailed=true&actions=*search*" curl -sS -u "${ES_USERNAME}:${ES_PASSWORD}" \ -X POST "http://${ES_ENDPOINT#http://}/_tasks/{task_id}/_cancel" ``` **3) Watch recovery** - CPU should fall within **~1–2 min** after load drops. - When `search` queue drains, new searches succeed; `all shards failed` stops. ### Short term (days) - **Scale vCPU/RAM** — search worker count tracks allocated processors (exact formula is **version/vendor-specific**; on small 7.x-style clusters a **2 vCPU** node often maps to **~4** `search` workers — easy to saturate). - **Add data nodes** to spread QPS. - Add **coordinating-only** nodes if reduce-phase / fan-out is the bottleneck. ### Long term - Query hygiene: wildcards, huge aggs, deep paging, scripts — see [sop-query-thread-pool.md](sop-query-thread-pool.md). - **Capacity:** rough stress thinking: sustainable QPS scales with **workers / p95 latency**; keep **steady CPU < ~60%**, spikes **< ~80%** if possible. --- ## 5. Load-test checklist ### Before the test | Check | Guidance | How | |-------|------------|-----| | SKU | prefer **≥ 4c16g** class nodes | `DescribeInstance` | | Count | **≥ 3** nodes incl. coordinators if used | `DescribeInstance` | | Search workers | know baseline (see `_cat/thread_pool`) | `GET _cat/thread_pool?v&h=node_name,name,active,queue,rejected` | | Replicas | **≥ 1** for hot indices | `_cat/indices` | | `refresh_interval` | consider **30s** during soak | `GET /{index}/_settings` | ### During the test | Metric | Safe band | Danger | CMS | |--------|-----------|--------|-----| | CPU avg | < ~60% | > ~80% | `NodeCPUUtilization` | | Search rejected | 0 | > 0 | `SearchThreadpoolRejected` | | Heap | < ~75% | > ~85% | `NodeHeapMemoryUtilization` | | GC overhead | low | high | GC logs | | Latency | < ~2× baseline | > ~5× | EWMA / APM | ### Ramp strategy ```text Baseline → 50% → 75% → 100% → 120% → target QPS Hold each step ≥ ~5m; require CPU < ~60% and rejected = 0 before advancing If CPU > ~80% OR rejected > 0 → drop back one step immediately ``` --- ## 6. Case study: 2000 QPS load test → meltdown **Cluster:** 2 × **2c8g** (search workers ≈ **4** in observed build) **Timeline:** ```text T+0: 2000 QPS to index simple-ppt T+4m: CPU ~5% → 80%+ T+10m: CPU ~85%+, GC overhead visible (~277ms/s class) T+21m: search queue 1005 > 1000 → EsRejectedExecutionException (~385 rejects) T+26m: burst of "all shards failed", EWMA ~67μs → ~371μs T+26m: users report "node offline" ``` **Differentiation:** `DescribeInstance` active; CMS CPU still reporting; no `removed/left`; `rejected` + `all shards failed` present. **Conclusion:** 2c nodes with **4** search workers cannot carry **2000 QPS** for that workload → CPU peg → pool meltdown. **Mitigation:** stop test → recovery → retest on **4c16g** (or more nodes) with ramp rules above. --- ## Appendix: quick commands ```bash aliyun elasticsearch DescribeInstance --region --InstanceId aliyun cms DescribeMetricList --Namespace "acs_elasticsearch" \ --MetricName "NodeCPUUtilization" \ --Dimensions '[{"instanceId":""}]' \ --StartTime "<5_min_ago_ms>" --EndTime "" --Period 60 aliyun elasticsearch ListSearchLog --region --InstanceId \ --type INSTANCELOG --query "removed OR left OR disconnect" \ --beginTime "" --endTime "" aliyun cms DescribeMetricList --Namespace "acs_elasticsearch" \ --MetricName "SearchThreadpoolRejected" \ --Dimensions '[{"instanceId":""}]' \ --StartTime "" --endTime "" --Period 60 aliyun elasticsearch ListSearchLog --region --InstanceId \ --type INSTANCELOG --query "shards" \ --beginTime "" --endTime "" aliyun elasticsearch ListSearchLog --region --InstanceId \ --type INSTANCELOG --query "rejected" \ --beginTime "" --endTime "" curl -sS --connect-timeout 10 --max-time 30 \ -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ "http://${ES_ENDPOINT#http://}/_nodes/hot_threads?threads=3" curl -sS --connect-timeout 10 --max-time 30 \ -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ "http://${ES_ENDPOINT#http://}/_tasks?detailed=true&actions=*search*" curl -sS --connect-timeout 10 --max-time 30 \ -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ "http://${ES_ENDPOINT#http://}/_nodes/stats/thread_pool?filter_path=nodes.*.thread_pool.search,nodes.*.name" ``` FILE:references/sop-write-performance.md # SOP: Ingest performance and write thread pools **Covers:** `ThreadPool.WriteRejected` (P0), `ThreadPool.WriteQueueHigh` (P0), `Performance.IndexLatencyHigh` (P0), `Performance.IndexingDropped` (P0) *(Frequency notes refer to internal case statistics — align names with your rule catalog.)* --- ## Diagnosis decision tree ``` Ingest symptoms ├── HTTP 429 "Too Many Requests" → write pool rejection (WriteRejected) ├── Ingest QPS drops > ~50% → IndexingDropped ├── Ingest latency above SLO → IndexLatencyHigh └── Writes fail with index read-only → disk flood / blocks → [sop-disk-storage.md](sop-disk-storage.md) Thread pools └── GET _nodes/stats/thread_pool → write / bulk queue + rejected ``` --- ## 1. Write thread-pool queue buildup (`WriteQueueHigh`, P0) ### Inspect ```bash GET _nodes/stats/thread_pool?filter_path=nodes.*.thread_pool.write,nodes.*.name # queue = depth now; rejected = cumulative rejects ``` ### Root causes **A — Ingest QPS exceeds capacity** - **Signals:** `write` queue grows with traffic. - **Mitigate:** short term — lower client QPS; **fewer round-trips** via larger bulks **only when** each bulk stays within a safe payload (see **§2 Bulk guidelines** — avoid **oversized** single requests that time out); long term — **scale out**. **B — Slow disk IO backs up writes** - **Signals:** high `write` queue **and** high disk utilization (e.g. `NodeStatsDataDiskUtil`). - **Mitigate:** [sop-disk-storage.md](sop-disk-storage.md) (IO + watermarks). **C — Merges steal CPU / IO** - **Signals:** `hot_threads` shows merge; queue high without extreme QPS. - **Mitigate:** cap merge scheduler threads (see [sop-disk-storage.md](sop-disk-storage.md) / [sop-configuration.md](sop-configuration.md)); reduce segment pressure. **D — `refresh_interval` too aggressive** - **Signals:** `refresh_interval` at **1s** or **100ms** on heavy ingest. - **Mitigate:** ```bash PUT /index_name/_settings { "index.refresh_interval": "30s" } ``` Tune to the freshness your product actually needs. **E — Too few nodes for concurrent writers** - **Signals:** (ingest QPS × payload) / node count exceeds per-node sustainable rate. - **Mitigate:** add nodes. ### Raising the write queue (usually a poor long-term fix) Parameter names and supported pools **vary by version** — confirm in your distribution before applying. ```bash PUT _cluster/settings { "transient": { "thread_pool.write.queue_size": 1000 } } ``` Prefer **fewer, larger bulks** and **more capacity** over permanently huge queues. --- ## 2. Write thread-pool rejections (`WriteRejected`, P0) ### Symptoms - Clients: **`HTTP 429 Too Many Requests`** - Logs: `rejected execution of ...` (wording varies) ### Immediate response **1) Thread pool snapshot** ```bash GET _cat/thread_pool?v&h=name,node_name,active,queue,rejected # inspect write / bulk rows ``` **2) Pattern** - **Sustained** buildup → cluster **over capacity** → scale or cut traffic. - **Spiky** rejects → tune client: **fewer parallel** bulk streams first; then adjust **docs / payload per bulk** within safe upper bounds (not “always bigger bulks”). **3) Client-side (most direct)** - **First:** cap **parallel** bulk workers / connections (reject storms often come from concurrency × retries). - Target **~5–15 MB** per bulk request (depends on doc size); **oversized** bulks risk **single-request timeouts** and long write stalls — balance **fewer requests** vs **per-request cap**. - Retries with **exponential backoff** on 429. ### Bulk guidelines ```text Payload per bulk: often ~10–30 MB (adjust for doc size) Docs per bulk: often ~500–2000 (if each doc is small) Parallel bulks: often ≤ ~2× node count (validate under load) Heuristic: - Low CPU but many rejects → too much parallelism → reduce parallel bulks / client connections - High CPU but slow ingest → bulks may be too small → increase docs per bulk only if each request stays below a safe max size (avoid monster single bulks) ``` ### Evidence interpretation: bulk QPS → write pool **When this applies:** `ThreadPool.WriteRejected` (or equivalent alerts), `_nodes/stats/thread_pool` shows **`write`** queue / **`rejected`**, and/or **`hot_threads`** shows **`TransportShardBulkAction`**, **`IndexShard`**, **`DocumentParser`**. Typical chain: **high bulk QPS → write-pool saturation** (queue buildup, **`rejected`** counters, bulk/index stacks in **`hot_threads`**, often concurrent merges). **Primary evidence (prefer over a generic “JVM-only” story):** | Layer | What to cite | |-------|----------------| | Thread pools | `GET _nodes/stats/thread_pool` — `write` **queue**, **active**, **`rejected`** | | Hot threads | `GET _nodes/hot_threads` — **`TransportShardBulkAction`**, indexing, **`DocumentParser`** | | Merges | Often concurrently: **`ConcurrentMergeScheduler`** / **`IndexWriter`** / Lucene merge stacks after heavy writes | **One-line causal chain for reports:** **bulk → write-thread saturation → segment merges → CPU**. **Old GC / heap pressure** from CMS or `JVMMemory.*`-style rules may appear as **concurrent or downstream** stress from indexing+merges—treat as **additive**, not a substitute headline when **`write` / bulk** evidence is clear. **Report ordering when `ThreadPool.WriteRejected` and JVM P0 both fire (e.g. acceptance / high-QPS-bulk scenarios):** Do **not** imply “GC first, ingest second.” Use one of: - **Causal chain (recommended for write-queue prompts):** **write overload / `rejected` → client retries → merge + heap pressure → Old GC STW + CPU / IO spikes** — `write` path is the **first** mechanism link; put **`write.rejected` / bulk saturation in the first sentence** of the conclusion when the prompt centers on **queue full / write reject**. - **Dual P0 headlines:** **`ThreadPool.WriteRejected` (P0)** and **`JVMMemory.GCTimeRatioTooHigh` / Old-gen pressure (P0)** as **co-equal** critical findings; order **write / bulk** before **GC** in the body so the summary matches the one-line conclusion. **Quantification (rubric-friendly):** Beside **“cumulative large”**, add **concrete counters** from **`_nodes/stats/thread_pool`**: per-node **`write.rejected`** and **`write.completed`** (same pool), optional **reject share** (roughly `rejected / (rejected + completed)`) on the hottest node(s). When aligning to control plane / CMS, cite **rule or event names** (e.g. **`ThreadPool.WriteRejected`**, **`HealthCheck.ThreadPoolSaturation`**) in addition to ES API stats. **Rule-engine alignment:** `scripts/check_es_instance_health.py` **promotes `ThreadPool.WriteRejected` from P1 → P0** when **`JVMMemory.GCTimeRatioTooHigh`** is present in the same run, and includes **`completed_total` / `completed_by_node`** beside **`rejected`** for reject-share context — keep **human-written** conclusions in the same order as this subsection. Treat promotion as a **label heuristic for co-occurring P0 signals**; confirm **写入 workload** (ingest-heavy vs mixed) with **`hot_threads`** / **`_tasks`** before a strict write→GC causal chain. **`rejected` semantics:** `thread_pool.write.rejected` (and `bulk` where present) is **cumulative since each data node JVM process started** — **not** “last 120 minutes” unless you show a **delta** between two samples. Say **“cumulative since node start”** in customer-facing summaries to avoid window misread (see also **§1 Inspect** above). **Per-node asymmetry:** The node with the busiest **`write`** activity in **`hot_threads`** may **not** be the node with the highest **`rejected`** count—**primary shard placement, coordinating/ingest paths, and per-node stats need not align.** Add **one sentence** when both are cited to avoid false contradictions. **Search-path hypotheses:** Under **write-only** overload, **deprioritize** fielddata / heavy-aggregation root causes. Open **`_nodes/stats/breaker`** / fielddata analysis only if **search**-path signals exist (`search` pool saturation, query rejections, documented read-heavy workload). For mixed CPU/GC symptoms, see [sop-memory-gc.md](sop-memory-gc.md). --- ## 3. High indexing latency (`IndexLatencyHigh`, P0) ### Steps **Step 1 — Indexing stats** ```bash GET _nodes/stats/indices/indexing?filter_path=nodes.*.indices.indexing # index_time_in_millis / index_total ≈ avg indexing time per op (rough) ``` **Step 2 — Common drivers** | Cause | Clues | Mitigation | |-------|-------|------------| | Disk IO | disk metrics high | disk tier / merge limits — [sop-disk-storage.md](sop-disk-storage.md) | | CPU | CPU high | cheaper mappings, scale out — [sop-cpu-load.md](sop-cpu-load.md) | | Refresh | very low `refresh_interval` | raise interval for bulk windows | | Analysis cost | hot threads in analyzer | simplify analyzers / reduce field count | | Translog `request` durability | fsync every op | `async` + `sync_interval` for bulk (risk tradeoff) | | Huge documents | multi-MB docs | split fields / shrink payloads | **Bulk-oriented index settings (example — validate for your ES version)** ```bash PUT /index_name/_settings { "index": { "refresh_interval": "30s", "translog": { "durability": "async", "sync_interval": "30s" }, "merge": { "policy": { "segments_per_tier": 30, "max_merged_segment": "256mb" } } } } ``` `async` translog is **weaker** durability than `request` — use only when the business accepts the risk. --- ## 4. Field notes (from production) ### A — `refresh_interval` and write timeouts (counter-intuitive band) **Pattern:** very rich analysis → **large refresh batches**; on some topologies **larger nodes** use **longer effective refresh windows**, so **each refresh flushes more data** and can spike latency → **write timeouts**. **Mitigation (example from a real case):** shorten the window so each refresh does less work: ```bash PUT /index/_settings { "index.refresh_interval": "100ms" } ``` This **conflicts** with the usual “raise `refresh_interval` for ingest” advice — pick based on **measurement** (indexing rate, segment count, p99 write latency). See also [sop-query-thread-pool.md](sop-query-thread-pool.md) Section 4.11 for the **opposite** problem (refresh **too** fast → segment churn). ### B — ILM cold phase blocks writes **Pattern:** log indices on ILM enter **cold** with **`read_only`** or move to **searchable snapshot / cold storage**; writes after the transition fail. **Check** ```bash GET /index_name/_settings # look for index.blocks.write ``` **Mitigation** ```bash PUT /index_name/_settings { "index.blocks.write": null } ``` Use **temporarily** to recover; fix **ILM phase timing** so active writers stay on **hot** tiers, or route writes to the **new** active index. ### C — Queue full but CPU / disk look “fine” **Pattern:** **many** connections sending **tiny** bulks (e.g. **1–10** docs each). **Check** ```bash GET _cat/thread_pool?v&h=name,node_name,active,queue,rejected GET _nodes/stats?filter_path=nodes.*.process.open_file_descriptors ``` **Mitigate:** fewer connections, **much larger** bulks (e.g. toward **500+** small docs per request when safe). **1000 bulks × 1 doc** is vastly more expensive than **1 bulk × 1000 docs**. --- ## 5. Sudden ingest collapse (`Performance.IndexingDropped`, P0) ### Branching ```text Ingest drops ├── Same-time drop across many instances → platform / upstream (e.g. shared control plane, network path) └── Single instance ├── Disk full / flood read-only → sop-disk-storage.md ├── Node left / shards relocating → sop-cluster-health.md ├── Planned change → often temporary; see sop-activating-change-stuck.md if stuck └── Client throttle / feature flag → check app logs ``` --- ## Appendix: quick commands (ingest) ```bash GET _cat/thread_pool?v&h=name,node_name,active,queue,rejected GET _nodes/stats/thread_pool?filter_path=nodes.*.thread_pool.write,nodes.*.name GET _nodes/stats/indices/indexing GET _cat/indices?v&h=index,docs.count,indexing.index_total,indexing.index_time,indexing.index_failed PUT /index_name/_settings { "index.refresh_interval": "30s" } PUT /index_name/_settings { "index": { "translog": { "durability": "async", "sync_interval": "30s" } } } PUT _cluster/settings { "transient": { "indices.merge.scheduler.max_thread_count": 1 } } ``` FILE:references/verification-method.md # Verification method How to validate the Elasticsearch instance diagnosis skill against the **current architecture**: - Control plane + CMS: `aliyun` CLI OpenAPI - Health rule engine: `python3 scripts/check_es_instance_health.py` - Engine-level collection: `curl` to ES REST APIs When `curl` fails (**401**, **timeouts**, **connection refused**, TLS mismatch), use the progressive guide **[es-api-call-failures.md](es-api-call-failures.md)** before re-running checks blindly. --- ## 1. Environment and credentials ### 1.1 CLI and profile ```bash aliyun version aliyun configure list aliyun --profile sts get-caller-identity ``` **Checklist**: - [ ] CLI works (recommend >= 3.3.1) - [ ] `aliyun configure list` shows at least one `Valid` profile - [ ] The chosen profile returns caller identity successfully ### 1.2 Direct ES credentials ```bash [[ -n "$ES_ENDPOINT" ]] && echo "ES_ENDPOINT: SET" || echo "ES_ENDPOINT: NOT SET" [[ -n "$ES_PASSWORD" ]] && echo "ES_PASSWORD: SET" || echo "ES_PASSWORD: NOT SET" ``` **Checklist**: - [ ] `ES_ENDPOINT` and `ES_PASSWORD` are set when engine checks are required - [ ] Plain-text password is not echoed --- ## 2. Control-plane OpenAPI (`aliyun` CLI) These steps can be run standalone or as manual backfill in a runbook. ### 2.1 Elasticsearch OpenAPI ```bash aliyun --profile elasticsearch DescribeInstance \ --region \ --InstanceId aliyun --profile elasticsearch ListSearchLog \ --region \ --InstanceId \ --type INSTANCELOG \ --query "*" \ --beginTime \ --endTime aliyun --profile elasticsearch ListActionRecords \ --region \ --InstanceId aliyun --profile elasticsearch ListAllNode \ --region \ --InstanceId ``` ### 2.2 CMS OpenAPI ```bash aliyun --profile cms DescribeMetricList \ --region \ --Namespace acs_elasticsearch \ --MetricName ClusterStatus \ --Dimensions '[{"clusterId":""}]' \ --StartTime \ --EndTime \ --Period 300 aliyun --profile cms DescribeSystemEventAttribute \ --region \ --Product elasticsearch \ --SearchKeywords \ --StartTime \ --EndTime aliyun --profile cms DescribeMetricMetaList \ --region \ --Namespace acs_elasticsearch ``` **Checklist**: - [ ] Each call returns `Code=200` or `Success=true` - [ ] `DescribeMetricList` returns datapoints (or empty series without error) - [ ] `ListSearchLog` returns `Result` or an empty array (not a hard failure) - [ ] `ListActionRecords`, `ListAllNode`, `DescribeMetricMetaList` return structured JSON --- ## 3. Health-check script ### 3.1 CLI data source ```bash python3 scripts/check_es_instance_health.py \ -i -r \ --window 60 \ --data-source cli \ --profile ``` **Checklist**: - [ ] Script completes without traceback - [ ] Structured report (P0/P1/P2, evidence, remediation) - [ ] Metric summary matches the live instance state ### 3.2 Injected input + auto ```bash python3 scripts/check_es_instance_health.py \ -i -r \ --data-source auto \ --input-json-file /path/to/diag-input.json \ --profile ``` **Checklist**: - [ ] Fields from input JSON are preferred when present - [ ] Missing fields fall back to CLI collection - [ ] With `--data-source input`, no OpenAPI calls are made --- ## 4. Engine-level ES APIs (`curl`) ### 4.1 Connectivity ```bash curl -sS -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ "http://${ES_ENDPOINT#http://}/_cluster/health?pretty" ``` ### 4.2 Key endpoints ```bash curl -sS -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ -H "Content-Type: application/json" \ -X POST "http://${ES_ENDPOINT#http://}/_cluster/allocation/explain?pretty" \ -d '{}' curl -sS -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ "http://${ES_ENDPOINT#http://}/_cat/shards?v&h=index,shard,prirep,state,node,unassigned.reason&s=state" curl -sS -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ "http://${ES_ENDPOINT#http://}/_nodes/stats/thread_pool?pretty" ``` **Checklist**: - [ ] JSON or cat text is returned - [ ] No auth or connection errors - [ ] Fields useful for root-cause analysis appear (`unassigned.reason`, `thread_pool.rejected`, etc.) --- ## 5. Negative / edge cases ### 5.1 Invalid CLI profile ```bash python3 scripts/check_es_instance_health.py \ -i -r \ --data-source cli \ --profile not-exist ``` **Expected**: Clear profile / authentication error (no silent success). ### 5.2 Non-existent instance ```bash python3 scripts/check_es_instance_health.py \ -i es-cn-invalid -r cn-hangzhou \ --data-source cli ``` **Expected**: Instance-not-found or invalid resource error; script does not crash. ### 5.3 Wrong HTTP scheme for endpoint ```bash curl -sS -u "${ES_USERNAME:-elastic}:${ES_PASSWORD}" \ "https://${ES_ENDPOINT#http://}/_cluster/health?pretty" ``` **Expected**: May see `WRONG_VERSION_NUMBER` if TLS mismatch; fix by switching between `http://` and `https://` to match the endpoint. FILE:scripts/_common.py """ _common.py — shared time utilities for the health-check rule engine. """ import re import sys from datetime import datetime, timedelta, timezone from typing import Optional # --------------------------------------------------------------------------- # Time helpers # --------------------------------------------------------------------------- def to_ms_timestamp(time_str: str) -> int: """ Parse several time formats into epoch milliseconds. Supported: -60 → now minus 60 minutes 2024-01-01 00:00:00 → full datetime 2024-01-01T00:00:00 → ISO-style 2024-01-01 → midnight that day """ try: minutes = int(time_str) dt = datetime.now() + timedelta(minutes=minutes) return int(dt.timestamp() * 1000) except (ValueError, TypeError): pass for fmt in ("%Y-%m-%d %H:%M:%S", "%Y-%m-%dT%H:%M:%S", "%Y-%m-%d"): try: return int(datetime.strptime(time_str, fmt).timestamp() * 1000) except ValueError: continue print(f"Error: cannot parse time string '{time_str}'") print("Supported: '2024-01-01 00:00:00' / '2024-01-01T00:00:00' / '2024-01-01' / '-60'") sys.exit(1) def format_timestamp(ts_ms: int) -> str: """Epoch ms → local time string YYYY-MM-DD HH:MM:SS.""" try: return datetime.fromtimestamp(int(ts_ms) / 1000).strftime("%Y-%m-%d %H:%M:%S") except Exception: return str(ts_ms) def now_ms() -> int: """Current time in epoch milliseconds.""" return int(datetime.now().timestamp() * 1000) def ago_ms(minutes: int) -> int: """Epoch ms for *minutes* ago.""" return int((datetime.now() - timedelta(minutes=minutes)).timestamp() * 1000) def ms_to_str(ms: int) -> str: """Alias of format_timestamp.""" return format_timestamp(ms) def parse_utc(s: str) -> Optional[datetime]: """ Parse ISO UTC strings into timezone-aware datetime (UTC). Supports: ...Z / +00:00 / [UTC] suffix / fractional seconds stripped. """ if not s: return None clean = re.sub(r"\.\d+", "", s.replace("[UTC]", "").strip()) for fmt in ("%Y-%m-%dT%H:%M:%SZ", "%Y-%m-%dT%H:%M:%S+00:00", "%Y-%m-%dT%H:%M:%S"): try: return datetime.strptime(clean, fmt).replace(tzinfo=timezone.utc) except ValueError: continue return None FILE:scripts/check_es_instance_health.py #!/usr/bin/env python3 """ Alibaba Cloud Elasticsearch instance health check. Concurrently collects instance status, CloudMonitor metrics, system events, and key logs, applies Elasticsearch health-event rules (20260318 baseline), and prints a structured report (findings + evidence + remediation). Each Finding carries: event_code — e.g. HealthCheck.ClusterUnhealthy reason_code — e.g. Cluster.StatusRed """ import argparse import json import re import sys from dataclasses import dataclass, field from datetime import datetime, timezone from typing import Any, Dict, List, Optional # Shared helpers (time utilities) from _common from _common import now_ms, ago_ms, ms_to_str, parse_utc # Control-plane + CMS data collection via aliyun CLI from openapi_cli_collect import ( METRIC_DEFINITIONS, normalize_datapoints, fetch_instance_status_info, fetch_metrics_batch, fetch_events, fetch_log_items, ) # --------------------------------------------------------------------------- # Thresholds (ES health-event rules baseline 20260318) # --------------------------------------------------------------------------- THRESHOLDS = { # Cluster health (rules #1–3) "cluster_red": 2, # ClusterStatus == 2 → Cluster.StatusRed (P0) "cluster_yellow": 1, # ClusterStatus == 1 → Cluster.StatusYellow (P1) # CPU (rules #12–13: P0=70%, P1=60%; rule #11 peak: P0=95%, P1=80%) "cpu_sustained_critical": 70.0, # avg CPU > 70% → CPU.PersistUsageHigh (P0) "cpu_sustained_warning": 60.0, # avg CPU > 60% → CPU.PersistUsageHigh (P1) "cpu_peak_critical": 95.0, # max CPU ≥ 95% → CPU.PeakUsageHigh (P0) "cpu_peak_warning": 80.0, # max CPU ≥ 80% → CPU.PeakUsageHigh (P1) # Heap (rules #8–9: warning 75% P1, critical 85% P0) "heap_critical_avg": 85.0, # avg heap > 85% → JVMMemory.OldGenUsageCritical (P0) "heap_warning_avg": 75.0, # avg heap > 75% → JVMMemory.OldGenUsageHigh (P1) # Disk utilization (rules #13–14) "disk_critical_max": 85.0, # max disk > 85% → Disk.UsageCritical (P0) "disk_warning_max": 75.0, # max disk > 75% → Disk.UsageHigh (P1) # Disk IO (rule #19: IO util > 90% P0) "disk_io_max": 90.0, # max IO util > 90% → Disk.IOPerformancePoor (P0) # Old GC rate (rule #22: >1/min → P1) "old_gc_rate_per_min": 1.0, # max GC count per sample > 1 → JVMMemory.GCRateTooHigh (P1) # GC time ratio (rule #18: >10% wall time → P0) "gc_time_ratio": 0.10, # avg GC duration / 60000ms > 10% → JVMMemory.GCTimeRatioTooHigh (P0) # CPU imbalance CV (rule #40) + absolute baseline (Bug-New-01) "load_imbalance_cv": 0.3, # CV > 0.3 → Balancing.NodeCPUUnbalanced (P1) "load_imbalance_min_cpu": 10.0, # require max node avg CPU > 10% before CV check (idle-cluster guard) # Disk / memory imbalance "disk_imbalance_cv": 0.3, # CV > 0.3 → Balancing.NodeDiskUnbalanced "disk_imbalance_warning": 75.0, # max node > 75% → P1 "disk_imbalance_critical": 85.0, # max node > 85% → P0 "memory_imbalance_cv": 0.3, # CV > 0.3 → Balancing.NodeMemoryUnbalanced "memory_imbalance_min": 75.0, # fire only if max heap > 75% # Management-plane duration "activating_stuck_min": 30, # activating longer than N min → stuck (P0) "event_stuck_min": 60, # Executing event longer than N min → stuck (P1) } PRIORITY_ICON = {"P0": "🔴", "P1": "🟡", "P2": "🔵"} PRIORITY_LABEL = {"P0": "Critical", "P1": "Warning", "P2": "Info"} # --------------------------------------------------------------------------- # Data structures # --------------------------------------------------------------------------- @dataclass class Finding: """One diagnostic finding.""" event_code: str # e.g. HealthCheck.ClusterUnhealthy reason_code: str # e.g. Cluster.StatusRed name: str priority: str # P0 / P1 / P2 category: str description: str evidence: Dict[str, Any] = field(default_factory=dict) remediation: List[str] = field(default_factory=list) @property def code(self) -> str: """Backward compatible alias for event_code.""" return self.event_code # --------------------------------------------------------------------------- # Statistics helpers # --------------------------------------------------------------------------- def _cv(values: List[float]) -> float: """Coefficient of variation std/mean.""" if len(values) < 2: return 0.0 mean = sum(values) / len(values) if mean == 0: return 0.0 std = (sum((v - mean) ** 2 for v in values) / len(values)) ** 0.5 return std / mean def _summarize(datapoints: List[Dict], metric_name: str) -> Dict[str, Dict]: """ Group by node/cluster; compute avg / max / min / latest. Uses normalize_datapoints + METRIC_DEFINITIONS. Returns {group_key: {"avg", "max", "min", "latest", "points"}}. """ meta = METRIC_DEFINITIONS.get(metric_name, { "group_field": "nodeIP", "value_field": "Average", }) grouped = normalize_datapoints(datapoints, meta) result: Dict[str, Dict] = {} for key, points in grouped.items(): valid_pts = [p for p in points if p["value"] is not None] if not valid_pts: continue vals = [p["value"] for p in valid_pts] # latest: newest sample by timestamp (current state), not window max latest_pt = max(valid_pts, key=lambda p: p.get("timestamp", 0)) result[key] = { "avg": sum(vals) / len(vals), "max": max(vals), "min": min(vals), "latest": latest_pt["value"], "points": valid_pts, # raw points for time-series CV } return result def _compute_timeseries_cv(summary: Dict[str, Dict]) -> Dict[str, Any]: """ Time-series load imbalance: CV across nodes per timestamp, track peak window. Returns: cv_avg, cv_max, peak_cv, peak_time, peak_values """ if len(summary) < 2: return {"cv_avg": 0.0, "cv_max": 0.0, "peak_cv": 0.0, "peak_time": 0, "peak_values": {}} # CV from per-node avg and per-node max series avg_vals = [s["avg"] for s in summary.values()] max_vals = [s["max"] for s in summary.values()] cv_avg = _cv(avg_vals) cv_max = _cv(max_vals) # Per-timestamp cross-node CV; track worst imbalance bucket all_timestamps = set() for s in summary.values(): for pt in s.get("points", []): all_timestamps.add(pt.get("timestamp", 0)) peak_cv = cv_avg peak_time = 0 peak_values = {} for ts in sorted(all_timestamps): ts_values = {} for node, s in summary.items(): for pt in s.get("points", []): if pt.get("timestamp") == ts and pt["value"] is not None: ts_values[node] = pt["value"] break if len(ts_values) >= 2: vals = list(ts_values.values()) ts_cv = _cv(vals) if ts_cv > peak_cv: peak_cv = ts_cv peak_time = ts peak_values = ts_values return { "cv_avg": cv_avg, "cv_max": cv_max, "peak_cv": peak_cv, "peak_time": peak_time, "peak_values": peak_values, } # Metrics subset for diagnosis (from METRIC_DEFINITIONS) _DIAG_METRICS = [ "ClusterStatus", "ClusterDisconnectedNodeCount", "ClusterNodeCount", "ClusterShardCount", "ClusterQueryQPS", "ClusterIndexQPS", "NodeCPUUtilization", "NodeHeapMemoryUtilization", "NodeDiskUtilization", "NodeFreeStorageSpace", "NodeLoad_1m", "NodeStatsDataDiskUtil", "JVMGCOldCollectionCount", "JVMGCOldCollectionDuration", ] def _cms_period_for_window(window_min: int) -> str: """ CMS sampling period from window length: - window ≤ 30 min → period 60s (finer; ~30 min retention for 60s series) - window > 30 min → period 300s (31-day retention; avoids empty CMS at 60s boundary) Bug-New-02: window=60 could align begin_ms outside 60s retention → empty datapoints. """ return "60" if window_min <= 30 else "300" def _get_cluster_setting(raw_settings: Dict, *keys: str) -> Optional[str]: """Walk transient → persistent → defaults; return first string leaf at *keys.""" for section in ("transient", "persistent", "defaults"): node = raw_settings.get(section, {}) for k in keys: node = node.get(k, {}) if isinstance(node, dict) else None if node is None: break if node and isinstance(node, str): return node return None def _compute_security_score(findings: List[Finding]): """ Overall health grade from finding severities. Returns (grade, icon, description): A — Healthy: no issues B — Stable: P2 hints only C — Warning: P1 present, respond within ~30 minutes D — Critical: P0 present, immediate action Rule: any P0 → D; P1 without P0 → C; P2 only → B; none → A. """ p0 = [f for f in findings if f.priority == "P0"] p1 = [f for f in findings if f.priority == "P1"] p2 = [f for f in findings if f.priority == "P2"] if p0: return "D", "🔴", f"Critical — {len(p0)} P0 finding(s) require immediate action" if p1: return "C", "🟡", f"Warning — {len(p1)} P1 finding(s); respond within ~30 minutes" if p2: return "B", "🔵", f"Stable — {len(p2)} P2 informational finding(s) (no runtime risk)" return "A", "✅", "Healthy — no findings; instance appears normal" # --------------------------------------------------------------------------- # Validation # --------------------------------------------------------------------------- # Characters rejected in ES user/password (command-injection hardening for curl args) _DANGEROUS_CHARS = set('`$(){}[]|;&<>\n\r') def _validate_endpoint_consistency( endpoint: str, instance_id: Optional[str], ) -> Optional[str]: """ Endpoint vs instance_id consistency (Bug-07). For *.aliyuncs.com public endpoints, require instance_id substring in URL to avoid optional ES checks hitting the wrong cluster. Returns: None — proceed "skip" — mismatch; warning printed; caller should skip optional checks "info" — private endpoint; informational message only """ if not instance_id: return None is_aliyun_public = "aliyuncs.com" in endpoint instance_in_endpoint = instance_id.lower() in endpoint.lower() if is_aliyun_public and not instance_in_endpoint: print( f"⚠️ [ES config check] Endpoint mismatch (Bug-07):\n" f" ES_ENDPOINT={endpoint}\n" f" Does not contain diagnosed instance_id={instance_id}; endpoint may belong to another instance.\n" f" Skipping optional config checks (circuit breakers / watermarks / thread pools) to avoid cross-instance false positives.\n" f" Fix: export ES_ENDPOINT=\"http://<{instance_id}-endpoint>:9200\"", file=sys.stderr, ) return "skip" elif not is_aliyun_public and not instance_in_endpoint: # Private IP/hostname — cannot infer ownership automatically print( f"ℹ️ [ES config check] Using private endpoint {endpoint} (not *.aliyuncs.com).\n" f" Confirm this endpoint belongs to instance {instance_id}; otherwise results may reflect another cluster.", file=sys.stderr, ) return "info" return None def _validate_credentials_safe(username: str, password: str) -> bool: """ Reject username/password if they contain characters unsafe for curl argv. Returns: True if safe to pass to curl -u; False if unsafe (message printed). """ if any(c in _DANGEROUS_CHARS for c in password) or any(c in _DANGEROUS_CHARS for c in username): print( f"⚠️ [ES config check] Username/password contain unsafe characters; skipping Elasticsearch API collection.\n" f" Do not use these in credentials: ` $ ( ) {{ }} [ ] | ; & < > newlines", file=sys.stderr, ) return False return True # --------------------------------------------------------------------------- # Data fetch / transform # --------------------------------------------------------------------------- def _fetch_error_logs( instance_id: str, region_id: str, begin_ms: int, end_ms: int, profile: Optional[str] = None, ) -> List[Dict]: """Fetch error/warning logs; normalize fields for rule engine.""" try: raw_items = fetch_log_items( instance_id, region_id, log_type="INSTANCELOG", begin_ms=begin_ms, end_ms=end_ms, query="Exception OR OutOfMemory OR OOM OR ERROR", profile=profile, ) logs = [] for d in raw_items: cc = d.get("contentCollection") or {} if not isinstance(cc, dict): cc = {} logs.append({ "level": cc.get("level", ""), "host": cc.get("host", d.get("host", "")), "time": cc.get("time", ""), "content": (cc.get("content", "") or "")[:300], }) return logs except Exception: return [] def _load_json_bundle(inline_json: Optional[str], json_file: Optional[str]) -> Dict[str, Any]: """Load injected diagnostic bundle from CLI args.""" if inline_json: obj = json.loads(inline_json) return obj if isinstance(obj, dict) else {} if json_file: with open(json_file, "r", encoding="utf-8") as f: obj = json.load(f) return obj if isinstance(obj, dict) else {} return {} # --------------------------------------------------------------------------- # Rules — CMS / management metrics # --------------------------------------------------------------------------- def _check_instance_status(info: Dict) -> List[Finding]: findings = [] if not info or "_error" in info: return findings status = info.get("status", "") updated_at_str = info.get("updated_at", "") if status == "activating" and updated_at_str: updated_at = parse_utc(updated_at_str) if updated_at: elapsed_min = (datetime.now(timezone.utc) - updated_at).total_seconds() / 60 if elapsed_min > THRESHOLDS["activating_stuck_min"]: findings.append(Finding( event_code="ManagementPlane.ActivatingStuck", reason_code="ManagementPlane.ActivatingStuck", name="Activating change stuck too long", priority="P0", category="Management plane", description=( f"Instance has stayed in activating for {int(elapsed_min)} minute(s) " f"(last updated: {updated_at_str}); control-plane workflow may be stuck." ), evidence={ "current_status": status, "last_updated_at": updated_at_str, "elapsed_minutes": int(elapsed_min), }, remediation=[ "In the console or CMS, check for system events stuck in Executing", "If no legitimate in-flight change, trigger a harmless update (e.g. description) to refresh state", "Contact Alibaba Cloud support if the workflow remains stuck", ], )) elif status == "inactive": findings.append(Finding( event_code="ManagementPlane.InstanceInactive", reason_code="ManagementPlane.InstanceInactive", name="Instance frozen (inactive)", priority="P0", category="Management plane", description="Instance is inactive (frozen), often due to billing or manual freeze; service unavailable.", evidence={"current_status": status}, remediation=[ "Verify account balance and payment status", "After payment, contact support to unfreeze if needed", ], )) elif status == "invalid": findings.append(Finding( event_code="ManagementPlane.InstanceInvalid", reason_code="ManagementPlane.InstanceInvalid", name="Instance invalid", priority="P0", category="Management plane", description="Instance is invalid; service unavailable.", evidence={"current_status": status}, remediation=["Contact Alibaba Cloud support to determine cause and remediation"], )) return findings def _check_cluster_health(metrics: Dict[str, List[Dict]]) -> List[Finding]: findings = [] # --- Cluster health (CMS ClusterStatus) --- cluster_status_dps = metrics.get("ClusterStatus", []) if cluster_status_dps: summary = _summarize(cluster_status_dps, "ClusterStatus") for cid, stat in summary.items(): # Use latest sample, not max, so stale Red in the window does not override current state current_status = stat["latest"] if current_status >= THRESHOLDS["cluster_red"]: findings.append(Finding( event_code="HealthCheck.ClusterUnhealthy", reason_code="Cluster.StatusRed", name="Cluster status Red", priority="P0", category="Cluster health", description="Cluster is Red: at least one primary shard is unassigned; some data is unavailable.", evidence={ "cluster_status_latest": f"{int(current_status)} (Red)", "cluster_status_window_max": f"{int(stat['max'])}", "sample_point_count": len(cluster_status_dps), }, remediation=[ "GET _cat/indices?v&health=red # list red indices", "GET _cat/shards?v&h=index,shard,prirep,state,node,unassigned.reason # shard state", "GET _cluster/allocation/explain # unassigned reason", "Fix root cause: offline node → recover; disk full → free space / expand; bad settings → correct", ], )) elif current_status >= THRESHOLDS["cluster_yellow"]: findings.append(Finding( event_code="HealthCheck.ClusterUnhealthy", reason_code="Cluster.StatusYellow", name="Cluster status Yellow", priority="P1", category="Cluster health", description="Cluster is Yellow: all primaries assigned but at least one replica is unassigned.", evidence={ "cluster_status_latest": f"{int(current_status)} (Yellow)", "cluster_status_window_max": f"{int(stat['max'])}", }, remediation=[ "GET _cat/shards?v&h=index,shard,prirep,state,unassigned.reason | grep UNASSIGNED", "GET _cluster/allocation/explain # replica allocation reason", "If disk pressure: delete cold data or expand capacity", "If relocation in progress: wait for completion", ], )) # --- Disconnected nodes --- disconn_dps = metrics.get("ClusterDisconnectedNodeCount", []) if disconn_dps: summary = _summarize(disconn_dps, "ClusterDisconnectedNodeCount") for _, stat in summary.items(): if stat["max"] > 0: findings.append(Finding( event_code="HealthCheck.ClusterUnhealthy", reason_code="Node.Disconnected", name="Node disconnected or failed", priority="P0", category="Cluster health", description=f"Up to {int(stat['max'])} node(s) disconnected from the cluster; data on those nodes may be unavailable.", evidence={ "max_disconnected_nodes": int(stat["max"]), "avg_disconnected_nodes": round(stat["avg"], 2), }, remediation=[ "GET _cat/nodes?v # which nodes are missing", "Check ES logs on offline nodes: OutOfMemoryError / JVM crash / Connection refused", "OOM → more heap or query tuning; network → fix connectivity; high CPU → reduce load", "After restart: GET _cat/recovery?v&active_only=true", ], )) return findings def _check_cpu(metrics: Dict[str, List[Dict]]) -> List[Finding]: findings = [] dps = metrics.get("NodeCPUUtilization", []) if not dps: return findings summary = _summarize(dps, "NodeCPUUtilization") # P0: avg > 70%; P1: 60% < avg <= 70% (rules #12–13) critical = [(n, s) for n, s in summary.items() if s["avg"] > THRESHOLDS["cpu_sustained_critical"]] warning = [(n, s) for n, s in summary.items() if THRESHOLDS["cpu_sustained_warning"] < s["avg"] <= THRESHOLDS["cpu_sustained_critical"]] # Peak: sustained avg not high (<= 60%) but spike high — >=95% P0, >=80% P1 (rule #11 extension) peak_critical = [(n, s) for n, s in summary.items() if s["max"] >= THRESHOLDS["cpu_peak_critical"] and s["avg"] <= THRESHOLDS["cpu_sustained_warning"]] peak_warning = [(n, s) for n, s in summary.items() if THRESHOLDS["cpu_peak_warning"] <= s["max"] < THRESHOLDS["cpu_peak_critical"] and s["avg"] <= THRESHOLDS["cpu_sustained_warning"]] if critical: findings.append(Finding( event_code="HealthCheck.CPULoadHigh", reason_code="CPU.PersistUsageHigh", name="Sustained high CPU (critical)", priority="P0", category="Resource metrics", description=( f"Node CPU average exceeds {THRESHOLDS['cpu_sustained_critical']}%; " f"severe bottleneck — high risk of node loss or cascading failure." ), evidence={ "affected_nodes": [ f"{n} avg={s['avg']:.1f}% max={s['max']:.1f}%" for n, s in critical ] }, remediation=[ "GET _nodes/hot_threads # hot threads", "GET _tasks?detailed=true&actions=*search* # find/cancel heavy searches", "Throttle clients: lower QPS or pause bulk ingest temporarily", "Tune slow queries: filters, avoid deep pagination, reduce wildcard/fuzzy", "Scale out: add data nodes or upgrade instance class if urgent", ], )) elif warning: findings.append(Finding( event_code="HealthCheck.CPULoadHigh", reason_code="CPU.PersistUsageHigh", name="Sustained high CPU (warning)", priority="P1", category="Resource metrics", description=( f"Node CPU average exceeds {THRESHOLDS['cpu_sustained_warning']}%; " f"elevated load — watch for slow queries or write backlog." ), evidence={ "affected_nodes": [ f"{n} avg={s['avg']:.1f}% max={s['max']:.1f}%" for n, s in warning ] }, remediation=[ "GET _nodes/hot_threads # CPU consumers", "GET _tasks?detailed=true # in-flight tasks", "Tune slow queries: deep pagination, wildcard, large aggs", "Consider more nodes or larger nodes if sustained", ], )) elif peak_critical: findings.append(Finding( event_code="HealthCheck.CPULoadHigh", reason_code="CPU.PeakUsageHigh", name="CPU spike critical", priority="P0", category="Resource metrics", description=( f"Node CPU peak ≥ {THRESHOLDS['cpu_peak_critical']}%; " f"very high risk of heartbeat timeouts and nodes leaving the cluster." ), evidence={ "affected_nodes": [ f"{n} max={s['max']:.1f}% avg={s['avg']:.1f}%" for n, s in peak_critical ] }, remediation=[ "Cancel heavy tasks: GET _tasks?detailed=true&actions=*search*", "GET _nodes/hot_threads # identify CPU consumers", "Temporarily reduce client concurrency", "Tune heavy queries: BKD ranges, large aggs, complex scripts", "Scale out or upgrade nodes if spikes recur", ], )) elif peak_warning: findings.append(Finding( event_code="HealthCheck.CPULoadHigh", reason_code="CPU.PeakUsageHigh", name="CPU spike warning", priority="P1", category="Resource metrics", description=( f"Node CPU peak in {THRESHOLDS['cpu_peak_warning']}–{THRESHOLDS['cpu_peak_critical']}%; " f"not sustained high load, but spikes indicate heavy queries or ingest bursts " f"that may become sustained if ignored." ), evidence={ "affected_nodes": [ f"{n} max={s['max']:.1f}% avg={s['avg']:.1f}%" for n, s in peak_warning ] }, remediation=[ "GET _nodes/hot_threads # CPU consumers", "GET _tasks?detailed=true # in-flight tasks", "Tune heavy queries: large aggs, deep pagination, BKD-heavy patterns", "Consider more nodes or larger nodes if spikes recur", ], )) # Load imbalance (rule #40) — time-series CV cv_stats = _compute_timeseries_cv(summary) cv_avg = cv_stats["cv_avg"] cv_max = cv_stats["cv_max"] peak_cv = cv_stats["peak_cv"] peak_time = cv_stats["peak_time"] peak_values = cv_stats["peak_values"] avg_vals = [s["avg"] for s in summary.values()] max_vals = [s["max"] for s in summary.values()] max_avg_cpu = max(avg_vals) if avg_vals else 0.0 # Use max of peak / window CV so a short spike is not averaged away effective_cv = max(cv_avg, cv_max, peak_cv) # Absolute CPU floor: skip CV when cluster is idle (idle-cluster CV false positive guard) if (effective_cv > THRESHOLDS["load_imbalance_cv"] and len(avg_vals) >= 2 and max_avg_cpu > THRESHOLDS["load_imbalance_min_cpu"]): evidence = { "cv_avg_full_window": round(cv_avg, 3), "cv_max_full_window": round(cv_max, 3), "cv_peak_window": round(peak_cv, 3), "max_node_cpu_avg_pct": f"{max(avg_vals):.1f}%", "min_node_cpu_avg_pct": f"{min(avg_vals):.1f}%", "max_node_cpu_max_pct": f"{max(max_vals):.1f}%", "avg_spread_pct": f"{max(avg_vals) - min(avg_vals):.1f}%", "node_count": len(avg_vals), } if peak_cv > cv_avg and peak_time and peak_values: evidence["peak_time_local"] = ms_to_str(peak_time) evidence["peak_window_cpu_by_node"] = [ f"{node}: {val:.1f}%" for node, val in sorted(peak_values.items(), key=lambda x: -x[1]) ] desc_suffix = "" if peak_cv > cv_avg * 1.2: desc_suffix = ( f" Note: peak-window CV={peak_cv:.2f} is much higher than average — inspect that time range." ) findings.append(Finding( event_code="HealthCheck.LoadUnbalanced", reason_code="Balancing.NodeCPUUnbalanced", name="Uneven CPU load across nodes", priority="P1", category="Capacity planning", description=( f"Cross-node CPU coefficient of variation CV={effective_cv:.2f} " f"(threshold {THRESHOLDS['load_imbalance_cv']}); " f"max avg {max(avg_vals):.1f}%, min avg {min(avg_vals):.1f}%. " f"Common causes: uneven shard counts, hot shards, missing coordinating nodes.{desc_suffix}" ), evidence=evidence, remediation=[ "# Step 1 — shard balance (common root cause)", "GET _cat/nodes?v&h=name,ip,cpu,heap.percent,disk.used_percent,shards", "GET _cat/allocation?v", "", "# Step 2 — hot indices / large shards", "GET _cat/shards?v&s=store:desc&h=index,shard,prirep,store,node", "GET _cat/indices?v&s=store.size:desc&h=index,pri,rep,store.size", "", "# Step 3 — rebalance", "POST _cluster/reroute { \"commands\": [{\"move\": {\"index\": \"hot_index\", \"shard\": 0, \"from_node\": \"node_busy\", \"to_node\": \"node_idle\"}}] }", "", "# Longer term", "Add coordinating-only nodes, ILM for tiering, tune routing where applicable", ], )) return findings def _check_memory(metrics: Dict[str, List[Dict]]) -> List[Finding]: findings = [] dps = metrics.get("NodeHeapMemoryUtilization", []) if not dps: return findings summary = _summarize(dps, "NodeHeapMemoryUtilization") # rule #9: P0 critical=85%; rule #8: P1 warning=75% critical = [(n, s) for n, s in summary.items() if s["avg"] > THRESHOLDS["heap_critical_avg"]] warning = [(n, s) for n, s in summary.items() if THRESHOLDS["heap_warning_avg"] < s["avg"] <= THRESHOLDS["heap_critical_avg"]] if critical: findings.append(Finding( event_code="HealthCheck.JVMMemoryPressure", reason_code="JVMMemory.OldGenUsageCritical", name="Heap usage critical (OOM risk)", priority="P0", category="Resource metrics", description=( f"Node heap utilization avg exceeds {THRESHOLDS['heap_critical_avg']}%; " f"OutOfMemoryError risk is high." ), evidence={ "affected_nodes": [ f"{n} avg={s['avg']:.1f}% max={s['max']:.1f}%" for n, s in critical ] }, remediation=[ "POST _cache/clear # clear caches", "GET _tasks?detailed=true&actions=*search* # find heavy tasks", "POST _tasks//_cancel # cancel large tasks", "Restart affected nodes off-peak if required (brief disruption)", "Longer term: larger heap / node class; avoid huge result sets", ], )) elif warning: findings.append(Finding( event_code="HealthCheck.JVMMemoryPressure", reason_code="JVMMemory.OldGenUsageHigh", name="Heap usage warning", priority="P1", category="Resource metrics", description=( f"Node heap utilization avg exceeds {THRESHOLDS['heap_warning_avg']}%; " f"OOM risk and frequent Full GC are possible." ), evidence={ "affected_nodes": [ f"{n} avg={s['avg']:.1f}% max={s['max']:.1f}%" for n, s in warning ] }, remediation=[ "GET _nodes/stats/jvm?filter_path=nodes.*.jvm.mem", "POST _cache/clear", "POST /index_name/_cache/clear?fielddata=true # fielddata cache", "Tune queries: smaller pages, less fielddata, shallower aggs", "Consider larger nodes if sustained", ], )) # GC time ratio (rule #18: P0 when GC time > 10% of wall clock per sample window) gc_dur_dps = metrics.get("JVMGCOldCollectionDuration", []) if gc_dur_dps: dur_summary = _summarize(gc_dur_dps, "JVMGCOldCollectionDuration") # JVMGCOldCollectionDuration: GC ms per sample bucket; ratio = avg_ms / 60000 gc_ratio_nodes = [ (n, s, s["avg"] / 60000.0) for n, s in dur_summary.items() if s["avg"] / 60000.0 > THRESHOLDS["gc_time_ratio"] ] if gc_ratio_nodes: findings.append(Finding( event_code="HealthCheck.JVMMemoryPressure", reason_code="JVMMemory.GCTimeRatioTooHigh", name="GC time ratio too high", priority="P0", category="Resource metrics", description=( f"Old GC time exceeds {THRESHOLDS['gc_time_ratio'] * 100:.0f}% of wall time; " f"pauses hurt search and indexing latency." ), evidence={ "affected_nodes": [ f"{n} avg_gc_ms={s['avg']:.0f}/min ratio={ratio * 100:.1f}%" for n, s, ratio in gc_ratio_nodes ] }, remediation=[ "GET _nodes/stats/jvm", "POST _cache/clear # relieve heap pressure", "Inspect large fielddata / request cache usage", "Tune GC / heap only after confirming collector + pause profile " "(see jvm.gc.collectors in _nodes/stats/jvm or node GC logs; do not assume G1 vs CMS from version alone)", "Reduce heavy aggs / large allocations", ], )) # Old GC frequency (rule #22: P1, >1 per minute) gc_cnt_dps = metrics.get("JVMGCOldCollectionCount", []) if gc_cnt_dps: gc_summary = _summarize(gc_cnt_dps, "JVMGCOldCollectionCount") gc_nodes = [(n, s) for n, s in gc_summary.items() if s["max"] > THRESHOLDS["old_gc_rate_per_min"]] if gc_nodes: findings.append(Finding( event_code="HealthCheck.JVMMemoryPressure", reason_code="JVMMemory.GCRateTooHigh", name="Old GC too frequent", priority="P1", category="Resource metrics", description=( f"Old GC rate exceeds {THRESHOLDS['old_gc_rate_per_min']:.0f}/minute; " f"heap pressure may increase search/index latency." ), evidence={ "affected_nodes": [ f"{n} max_gc={s['max']:.1f}/min avg_gc={s['avg']:.1f}/min" for n, s in gc_nodes ] }, remediation=[ "GET _nodes/stats/jvm", "Check fielddata and deep aggregations on heap", "Tune queries to reduce memory churn", "Tune GC / heap or upgrade nodes if sustained (confirm collector via jvm stats or GC logs, not version guesswork)", ], )) return findings def _check_disk(metrics: Dict[str, List[Dict]]) -> List[Finding]: findings = [] dps = metrics.get("NodeDiskUtilization", []) if not dps: return findings summary = _summarize(dps, "NodeDiskUtilization") # rule #14: P0 critical=85%; rule #13: P1 warning=75% critical = [(n, s) for n, s in summary.items() if s["max"] > THRESHOLDS["disk_critical_max"]] warning = [(n, s) for n, s in summary.items() if THRESHOLDS["disk_warning_max"] < s["max"] <= THRESHOLDS["disk_critical_max"]] if critical: findings.append(Finding( event_code="HealthCheck.DiskUsageHigh", reason_code="Disk.UsageCritical", name="Disk usage critical", priority="P0", category="Resource metrics", description=( f"Node disk usage exceeds {THRESHOLDS['disk_critical_max']}%; " f"near ES flood-stage (~95%) writes may fail." ), evidence={ "affected_nodes": [ f"{n} max={s['max']:.1f}% avg={s['avg']:.1f}%" for n, s in critical ] }, remediation=[ "GET _cat/allocation?v", "GET _cat/indices?v&s=store.size:desc # largest indices", "DELETE /old_index # drop cold indices if safe", "PUT _all/_settings {\"index.blocks.read_only_allow_delete\": null} # clear read-only if set", "Expand disk capacity in console if needed", ], )) elif warning: findings.append(Finding( event_code="HealthCheck.DiskUsageHigh", reason_code="Disk.UsageHigh", name="Disk usage warning", priority="P1", category="Resource metrics", description=( f"Node disk usage exceeds {THRESHOLDS['disk_warning_max']}%; " f"free space before flood-stage read-only (~95%)." ), evidence={ "affected_nodes": [ f"{n} max={s['max']:.1f}%" for n, s in warning ] }, remediation=[ "GET _cat/indices?v&s=store.size:desc", "Delete or archive cold data (e.g. OSS)", "Use ILM to expire old indices", "Plan disk expansion if growth continues", ], )) # Disk IO utilization (rule #19: P0 when IO util > 90%) io_dps = metrics.get("NodeStatsDataDiskUtil", []) if io_dps: io_summary = _summarize(io_dps, "NodeStatsDataDiskUtil") io_nodes = [(n, s) for n, s in io_summary.items() if s["max"] > THRESHOLDS["disk_io_max"]] if io_nodes: findings.append(Finding( event_code="HealthCheck.DiskIOBottleneck", reason_code="Disk.IOPerformancePoor", name="Disk IO utilization too high", priority="P0", category="Resource metrics", description=( f"Disk IO utilization exceeds {THRESHOLDS['disk_io_max']}%; " f"higher IO latency can cause heartbeat loss or node drop-out." ), evidence={ "affected_nodes": [ f"{n} max_io={s['max']:.1f}% avg_io={s['avg']:.1f}%" for n, s in io_nodes ] }, remediation=[ "Lower ingest pressure: fewer concurrent bulks, larger bulk batches", "Raise refresh_interval (e.g. 30s) to cut refresh cost", "PUT _cluster/settings {\"indices.merge.scheduler.max_thread_count\": 1}", "Longer term: faster disk tier (ESSD PL1/PL2)", ], )) return findings def _check_resource_imbalance(metrics: Dict[str, List[Dict]]) -> List[Finding]: """ Detect uneven disk and heap utilization across nodes (CV-based). Complements CPU imbalance in _check_cpu. See references/sop-node-load-imbalance.md. """ findings = [] # --- Disk utilization imbalance --- disk_dps = metrics.get("NodeDiskUtilization", []) if disk_dps: disk_summary = _summarize(disk_dps, "NodeDiskUtilization") if len(disk_summary) >= 2: cv_stats = _compute_timeseries_cv(disk_summary) cv_avg = cv_stats["cv_avg"] cv_max = cv_stats["cv_max"] peak_cv = cv_stats["peak_cv"] effective_cv = max(cv_avg, cv_max, peak_cv) avg_vals = [s["avg"] for s in disk_summary.values()] max_vals = [s["max"] for s in disk_summary.values()] max_disk = max(max_vals) # use max utilization for severity min_disk = min(avg_vals) if effective_cv > THRESHOLDS["disk_imbalance_cv"]: if max_disk > THRESHOLDS["disk_imbalance_critical"]: priority = "P0" desc_suffix = "Hot node above critical watermark — read-only / flood risk." elif max_disk > THRESHOLDS["disk_imbalance_warning"]: priority = "P1" desc_suffix = "Hot node disk high — rebalance shards soon." else: priority = "P2" desc_suffix = "CV high but absolute disk OK — plan rebalance." findings.append(Finding( event_code="HealthCheck.LoadUnbalanced", reason_code="Balancing.NodeDiskUnbalanced", name="Uneven disk usage across nodes", priority=priority, category="Capacity planning", description=( f"Cross-node disk utilization CV={effective_cv:.2f} " f"(threshold {THRESHOLDS['disk_imbalance_cv']}); " f"max {max_disk:.1f}%, min {min_disk:.1f}%. " f"Common causes: uneven shard counts / large shards on few nodes. {desc_suffix}" ), evidence={ "cv_avg_full_window": round(cv_avg, 3), "cv_max_full_window": round(cv_max, 3), "cv_peak_window": round(peak_cv, 3), "max_node_disk_pct": f"{max_disk:.1f}%", "min_node_disk_avg_pct": f"{min_disk:.1f}%", "spread_pct": f"{max_disk - min_disk:.1f}%", "node_count": len(avg_vals), "top_nodes": [ f"{n} avg={s['avg']:.1f}% max={s['max']:.1f}%" for n, s in sorted(disk_summary.items(), key=lambda x: -x[1]["max"]) ][:5], }, remediation=[ "# Step 1 — shard balance", "GET _cat/nodes?v&h=name,ip,disk.used_percent,shards", "GET _cat/allocation?v", "", "# Step 2 — move large shards", "GET _cat/shards?v&s=store:desc&h=index,shard,prirep,store,node", "POST _cluster/reroute { \"commands\": [{\"move\": {...}}] }", "", "# Longer term", "Tune disk watermarks for earlier relocation; ILM for retention", ], )) # --- Heap utilization imbalance --- heap_dps = metrics.get("NodeHeapMemoryUtilization", []) if heap_dps: heap_summary = _summarize(heap_dps, "NodeHeapMemoryUtilization") if len(heap_summary) >= 2: cv_stats = _compute_timeseries_cv(heap_summary) cv_avg = cv_stats["cv_avg"] cv_max = cv_stats["cv_max"] peak_cv = cv_stats["peak_cv"] effective_cv = max(cv_avg, cv_max, peak_cv) avg_vals = [s["avg"] for s in heap_summary.values()] max_vals = [s["max"] for s in heap_summary.values()] max_heap = max(max_vals) min_heap = min(avg_vals) if (effective_cv > THRESHOLDS["memory_imbalance_cv"] and max_heap > THRESHOLDS["memory_imbalance_min"]): findings.append(Finding( event_code="HealthCheck.LoadUnbalanced", reason_code="Balancing.NodeMemoryUnbalanced", name="Uneven heap usage across nodes", priority="P1", category="Capacity planning", description=( f"Cross-node heap utilization CV={effective_cv:.2f} " f"(threshold {THRESHOLDS['memory_imbalance_cv']}); " f"max {max_heap:.1f}%, min {min_heap:.1f}%. " f"Common causes: uneven shards, fielddata skew, hot caches." ), evidence={ "cv_avg_full_window": round(cv_avg, 3), "cv_max_full_window": round(cv_max, 3), "cv_peak_window": round(peak_cv, 3), "max_node_heap_pct": f"{max_heap:.1f}%", "min_node_heap_avg_pct": f"{min_heap:.1f}%", "spread_pct": f"{max_heap - min_heap:.1f}%", "node_count": len(avg_vals), "top_nodes": [ f"{n} avg={s['avg']:.1f}% max={s['max']:.1f}%" for n, s in sorted(heap_summary.items(), key=lambda x: -x[1]["max"]) ][:5], }, remediation=[ "# Step 1 — shard balance", "GET _cat/nodes?v&h=name,ip,heap.percent,shards", "", "# Step 2 — fielddata", "GET _cat/fielddata?v&s=size:desc", "GET _nodes/stats/indices?filter_path=nodes.*.indices.fielddata", "", "# Step 3 — rebalance / query tuning", "Avoid huge text field aggs; reroute shards if needed", "POST _cluster/reroute { \"commands\": [{\"move\": {...}}] }", ], )) return findings def _check_metrics_sparse(metrics: Dict[str, List[Dict]], window_min: int, period: str) -> None: """ Warn on stderr when CMS datapoints are much sparser than expected. Sparse if ClusterStatus / NodeCPUUtilization point count < 20% of expected minimum. """ key_metrics = ["ClusterStatus", "NodeCPUUtilization"] expected_min = max(1, window_min // (int(period) // 60)) threshold = max(1, int(expected_min * 0.2)) sparse = [ m for m in key_metrics if len(metrics.get(m, [])) < threshold ] if sparse: print( f"\n⚠️ [CMS] Sparse datapoints for key metrics: {sparse}\n" f" Possible causes:\n" f" 1. Retention / granularity window shorter than {window_min} minutes (try --window 30)\n" f" 2. Fault injection just started; wait ≥5 minutes for CMS to populate\n" f" 3. Wrong instance ID or region — CMS returned no data\n" f" Conclusions may be incomplete; interpret with care.", file=sys.stderr, ) # --------------------------------------------------------------------------- # Rules — events / logs # --------------------------------------------------------------------------- def _check_events(events: List[Dict]) -> List[Finding]: findings = [] now_utc = datetime.now(timezone.utc) stuck = [] for e in events: if e.get("event_status") == "Executing": start_dt = parse_utc(e.get("start_time", "")) if start_dt: elapsed = (now_utc - start_dt).total_seconds() / 60 if elapsed > THRESHOLDS["event_stuck_min"]: stuck.append({ "event_name": e.get("name", ""), "reason": e.get("reason", ""), "start_time": e.get("start_time", ""), "elapsed_minutes": int(elapsed), }) if stuck: findings.append(Finding( event_code="ManagementPlane.EventStuck", reason_code="ManagementPlane.EventStuck", name="Change event stuck in Executing", priority="P1", category="Management plane", description=( f"{len(stuck)} system event(s) stayed Executing longer than " f"{THRESHOLDS['event_stuck_min']} minutes; change may be stuck." ), evidence={"stuck_events": stuck}, remediation=[ "Check change progress in the Alibaba Cloud console", "If truly stuck, contact support with event IDs", "Also verify data-node health — node faults often block changes", ], )) return findings def _check_logs(logs: List[Dict]) -> List[Finding]: findings = [] oom = [l for l in logs if "OutOfMemoryError" in l.get("content", "") or ("OOM" in l.get("content", "") and "ERROR" in l.get("content", "").upper())] if oom: sample = oom[0] findings.append(Finding( event_code="HealthCheck.JVMMemoryPressure", reason_code="JVMMemory.OOM", name="Node OutOfMemoryError", priority="P0", category="Resource metrics", description="Logs show OutOfMemoryError; node likely ran out of heap and may have restarted.", evidence={ "oom_log_lines": len(oom), "affected_host": sample.get("host", "-"), "log_time": sample.get("time", "-"), "log_excerpt": sample.get("content", "")[:150], }, remediation=[ "Restart affected nodes after confirming recovery path", "GET _nodes/stats/jvm", "Analyze heap dumps if captured", "Increase heap / node size; reduce query memory footprint", ], )) # UnavailableShardsException in logs (often alongside Red) shard_errs = [l for l in logs if "UnavailableShardsException" in l.get("content", "")] if shard_errs and not oom: sample = shard_errs[0] # Extract index name from log line (supports [[idx][0]] style) m = re.search(r"\[+([^\]\[]+)\]\[0\] primary shard is not active", sample.get("content", "")) idx_name = m.group(1) if m else "unknown" # System indices (.monitoring-*, .security-*, .kibana*, …) → P2 informational: # usually a side-effect of resource pressure, not primary business data loss. _SYS_PREFIXES = (".monitoring-", ".security-", ".kibana", # .kibana-* / .kibana_ ".apm-", ".fleet-", ".async-search", ".ds-ilm", ".watches") is_system_idx = any(idx_name.startswith(p) for p in _SYS_PREFIXES) findings.append(Finding( event_code="HealthCheck.ClusterUnhealthy", reason_code="Cluster.UnavailableShards", name=( "System index shard unavailable (informational)" if is_system_idx else "Primary shard unavailable" ), priority="P2" if is_system_idx else "P0", category="Informational" if is_system_idx else "Cluster health", description=( f"UnavailableShardsException in logs (index: {idx_name})." + ( " System / stack index — unassigned shards often follow CPU or heap pressure; " "fix resource pressure first." if is_system_idx else " Primary shard inactive — reads/writes to this index will time out." ) ), evidence={ "matching_log_lines": len(shard_errs), "sample_index": idx_name, "is_system_index": "yes" if is_system_idx else "no", "affected_host": sample.get("host", "-"), "log_excerpt": sample.get("content", "")[:150], }, remediation=( [ "Usually no direct fix on the system index — relieve CPU/heap pressure first", "Optional cleanup: DELETE .monitoring-es-7-* (expired monitoring)", "To disable: xpack.monitoring in kibana.yml (product-specific)", ] if is_system_idx else [ "GET _cat/shards?v&h=index,shard,prirep,state,unassigned.reason", "GET _cluster/allocation/explain", "Apply allocation explain guidance (disk, settings, node availability)", ] ), )) return findings # --------------------------------------------------------------------------- # Rules — optional ES API (cluster settings / _cat / thread pool) # --------------------------------------------------------------------------- def _check_fielddata_breaker(raw_settings: Dict) -> List[Finding]: """Fielddata circuit breaker limit vs recommended 40%.""" findings: List[Finding] = [] fd_limit = _get_cluster_setting(raw_settings, "indices", "breaker", "fielddata", "limit") if not fd_limit or "%" not in fd_limit: return findings try: pct = float(fd_limit.rstrip("%")) except ValueError: return findings if pct < 40.0: findings.append(Finding( event_code="HealthCheck.JVMMemoryPressure", reason_code="JVMMemory.BreakerLimitConfigLow", name="Fielddata breaker limit too low", priority="P2", category="Configuration", description=( f"indices.breaker.fielddata.limit = {fd_limit}, below the 40% recommendation. " f"Even when tripped=0, text-field aggregations can easily hit CircuitBreakingException." ), evidence={ "configured_value": fd_limit, "recommended": "40% (Elasticsearch default)", "risk": "Low traffic hides the issue until a spike trips the breaker", }, remediation=[ 'PUT _cluster/settings { "persistent": { "indices.breaker.fielddata.limit": "40%" } }', "Before changing: GET _nodes/stats/indices/fielddata?fields=*", ], )) return findings def _parse_watermark_to_bytes(value: str) -> Optional[int]: """Parse disk watermark to bytes. Percentages return None; supports e.g. 500gb, 19605mb, 1000000b.""" if not value: return None value = value.strip().lower() if "%" in value: return None # percentage watermarks are handled separately multipliers = {"b": 1, "kb": 1024, "mb": 1024**2, "gb": 1024**3, "tb": 1024**4, "pb": 1024**5} for suffix, mult in sorted(multipliers.items(), key=lambda x: -len(x[0])): if value.endswith(suffix): try: return int(float(value[:-len(suffix)]) * mult) except ValueError: return None # Bare integer → bytes try: return int(value) except ValueError: return None def _check_disk_watermark_config( raw_settings: Dict, metrics: Optional[Dict[str, List[Dict]]] = None, curl_fn=None, ) -> List[Finding]: """Disk watermark settings: absolute-byte mode, breach vs CMS, too-low / too-high %.""" findings: List[Finding] = [] wm_low = _get_cluster_setting(raw_settings, "cluster", "routing", "allocation", "disk", "watermark", "low") wm_flood = _get_cluster_setting(raw_settings, "cluster", "routing", "allocation", "disk", "watermark", "flood_stage") if not wm_low: return findings # --- Branch A: absolute-byte watermarks (e.g. transient "19605mb") --- flood_bytes = _parse_watermark_to_bytes(wm_flood) if wm_flood else None low_bytes = _parse_watermark_to_bytes(wm_low) if low_bytes is not None: disk_avail_bytes_list: list = [] # [(node_name, avail_bytes)] if curl_fn: try: alloc_data = curl_fn("/_cat/allocation?format=json&bytes=b") for row in (alloc_data or []): node = row.get("node", "unknown") avail = int(row.get("disk.avail", 0)) disk_avail_bytes_list.append((node, avail)) except Exception: pass if flood_bytes is not None and disk_avail_bytes_list: flood_breached = [(n, avail) for n, avail in disk_avail_bytes_list if avail < flood_bytes] flood_margin = [(n, avail) for n, avail in disk_avail_bytes_list if avail >= flood_bytes and (avail - flood_bytes) < 500 * 1024 * 1024] if flood_breached: findings.append(Finding( event_code="HealthCheck.DiskUsageHigh", reason_code="Disk.WatermarkAbsoluteFloodBreached", name="Free disk below absolute flood_stage (indices read-only)", priority="P0", category="Configuration", description=( f"flood_stage is an absolute value {wm_flood} (~{flood_bytes / 1024**2:.0f} MB); " f"these nodes are below it — indices should be read_only_allow_delete." ), evidence={ "flood_stage_setting": wm_flood, "flood_stage_bytes": flood_bytes, "breached_nodes": [ f"{n}: free {avail / 1024**2:.0f}MB < threshold {flood_bytes / 1024**2:.0f}MB" for n, avail in flood_breached ], "setting_layer": ( "transient" if raw_settings.get("transient", {}).get("cluster", {}).get("routing", {}).get("allocation", {}).get("disk", {}).get("watermark") else "persistent" ), }, remediation=[ 'Reset watermarks: PUT _cluster/settings { "transient": { "cluster.routing.allocation.disk.watermark.low": null, ' '"cluster.routing.allocation.disk.watermark.high": null, ' '"cluster.routing.allocation.disk.watermark.flood_stage": null } }', 'Clear read-only: PUT _all/_settings { "index.blocks.read_only_allow_delete": null }', "Find who set transient watermarks (scripts / runbooks / drills)", ], )) elif flood_margin: findings.append(Finding( event_code="HealthCheck.DiskUsageHigh", reason_code="Disk.WatermarkAbsoluteFloodMarginLow", name="Free disk margin above absolute flood_stage is tiny", priority="P1", category="Configuration", description=( f"flood_stage absolute {wm_flood} (~{flood_bytes / 1024**2:.0f} MB); " f"these nodes are barely above it — read-only may return quickly." ), evidence={ "flood_stage_setting": wm_flood, "at_risk_nodes": [ f"{n}: free {avail / 1024**2:.0f}MB, margin {(avail - flood_bytes) / 1024**2:.0f}MB" for n, avail in flood_margin ], }, remediation=[ "GET _cluster/settings?include_defaults=true&filter_path=**.watermark", 'If mis-set: PUT _cluster/settings { "transient": { "cluster.routing.allocation.disk.watermark.low": null, ' '"cluster.routing.allocation.disk.watermark.high": null, ' '"cluster.routing.allocation.disk.watermark.flood_stage": null } }', ], )) is_transient = bool(raw_settings.get("transient", {}).get("cluster", {}).get("routing", {}).get("allocation", {}).get("disk", {}).get("watermark")) findings.append(Finding( event_code="HealthCheck.ConfigurationRisk", reason_code="Disk.WatermarkAbsoluteValue", name="Disk watermarks use absolute bytes (non-default)", priority="P1", category="Configuration", description=( f"Disk watermarks are absolute bytes (low={wm_low}, flood_stage={wm_flood or 'not set'}), " f"not percentage mode. Absolute values do not scale when disks grow and are easy to misconfigure." ), evidence={ "low": wm_low, "flood_stage": wm_flood or "n/a", "setting_layer": "transient" if is_transient else "persistent_or_defaults", "risk": "Does not auto-adjust after disk expansion; can block writes while disk looks healthy", }, remediation=[ 'Revert to percentage defaults: PUT _cluster/settings { "transient": { "cluster.routing.allocation.disk.watermark.low": null, ' '"cluster.routing.allocation.disk.watermark.high": null, ' '"cluster.routing.allocation.disk.watermark.flood_stage": null } }', "Trace the source of absolute-byte watermarks (scripts / ops / fault injection)", ], )) return findings # --- Branch B: percentage watermarks --- if "%" not in wm_low: return findings try: pct = float(wm_low.rstrip("%")) except ValueError: return findings if metrics: disk_summary = _summarize(metrics.get("NodeDiskUtilization", []), "NodeDiskUtilization") breached = [(node, stat["max"]) for node, stat in disk_summary.items() if stat["max"] > pct] if breached: findings.append(Finding( event_code="HealthCheck.DiskUsageHigh", reason_code="Disk.WatermarkBreached", name="Disk usage above configured watermark.low", priority="P1", category="Configuration", description=( f"cluster.routing.allocation.disk.watermark.low = {wm_low}, " f"but node disk utilization max exceeds it — relocations should be active or imminent." ), evidence={ "configured_watermark_low": wm_low, "breached_nodes_max_pct": [f"{n} {v:.1f}%" for n, v in breached], }, remediation=[ "Free disk: delete old indices/snapshots or expand capacity", "GET _cat/shards?v # relocations in flight?", "GET _cluster/settings", 'If low was mis-tuned: PUT _cluster/settings { "persistent": { "cluster.routing.allocation.disk.watermark.low": "85%" } }', ], )) if pct < 5.0: findings.append(Finding( event_code="HealthCheck.DiskUsageHigh", reason_code="Disk.WatermarkConfigLow", name="Disk watermark.low extremely low (write-block risk)", priority="P1", category="Configuration", description=( f"cluster.routing.allocation.disk.watermark.low = {wm_low}, far below a sane floor (~5%). " f"Normal ingest will cross it quickly and hit flood-stage write blocks." ), evidence={ "configured_low": wm_low, "recommended_low": "85% (Elasticsearch default)", "risk": "Tiny headroom before read-only / flood-stage", }, remediation=[ 'PUT _cluster/settings { "persistent": { "cluster.routing.allocation.disk.watermark.low": "85%", ' '"cluster.routing.allocation.disk.watermark.high": "90%", ' '"cluster.routing.allocation.disk.watermark.flood_stage": "95%" } }', 'If writes blocked: PUT index_name/_settings { "index.blocks.read_only_allow_delete": null }', ], )) if pct > 90.0: findings.append(Finding( event_code="HealthCheck.DiskUsageHigh", reason_code="Disk.WatermarkConfigHigh", name="Disk watermark.low too high (relocations start late)", priority="P2", category="Configuration", description=( f"cluster.routing.allocation.disk.watermark.low = {wm_low}, above the ~85% recommendation. " f"Shards relocate only at very high utilization — little time to react." ), evidence={ "configured_low": wm_low, "recommended_low": "85% (Elasticsearch default)", "risk": "With high=95%, only ~5% between low and high triggers", }, remediation=[ 'PUT _cluster/settings { "persistent": { "cluster.routing.allocation.disk.watermark.low": "85%" } }', "Also review watermark.high (~90%) and flood_stage (~95%)", ], )) return findings def _check_index_readonly_blocks(curl_fn) -> List[Finding]: """Detect indices blocked with read_only_allow_delete.""" findings: List[Finding] = [] try: settings_raw = curl_fn("/_all/_settings?filter_path=*.settings.index.blocks") if not settings_raw: return findings readonly_indices = [] for idx_name, idx_settings in settings_raw.items(): blocks = idx_settings.get("settings", {}).get("index", {}).get("blocks", {}) if str(blocks.get("read_only_allow_delete", "")).lower() == "true": readonly_indices.append(idx_name) if readonly_indices: biz_indices = [i for i in readonly_indices if not i.startswith(".")] sys_indices = [i for i in readonly_indices if i.startswith(".")] findings.append(Finding( event_code="HealthCheck.DiskUsageHigh", reason_code="Disk.IndexReadOnly", name="Indices read-only (read_only_allow_delete)", priority="P0", category="Configuration", description=( f"{len(readonly_indices)} index(es) have read_only_allow_delete; writes are rejected. " f"Usually from disk flood_stage; on ES 7.x blocks do not auto-clear after disk recovers." ), evidence={ "readonly_index_count": len(readonly_indices), "business_indices_sample": biz_indices[:10] if biz_indices else "(none)", "system_indices_sample": sys_indices[:10] if sys_indices else "(none)", }, remediation=[ 'PUT _all/_settings { "index.blocks.read_only_allow_delete": null }', "GET _cat/allocation?v # confirm free disk", "GET _cluster/settings?include_defaults=true&filter_path=**.watermark", "On ES 7.x you must clear read-only manually after fixing disk", ], )) except Exception as e: print(f"⚠️ [ES config check] Index read-only block check failed: {e}", file=__import__('sys').stderr) return findings def _check_zero_replicas(curl_fn) -> List[Finding]: """Detect business indices with number_of_replicas=0.""" findings: List[Finding] = [] try: indices_raw = curl_fn("/_cat/indices?h=index,rep&format=json") zero_rep = [ i["index"] for i in (indices_raw or []) if str(i.get("rep", "1")) == "0" and not i.get("index", "").startswith(".") ] if zero_rep: findings.append(Finding( event_code="HealthCheck.ClusterUnhealthy", reason_code="Index.ZeroReplicas", name="Business indices have zero replicas", priority="P2", category="Configuration", description=( f"{len(zero_rep)} non-system index(es) have number_of_replicas=0; " f"a single node loss makes that index unavailable." ), evidence={"zero_replica_index_count": len(zero_rep), "sample_indices": zero_rep[:5]}, remediation=[ 'PUT /index_name/_settings { "number_of_replicas": 1 }', "Repeat per index (or use index patterns carefully)", "Ignore if intentionally cost-optimized", ], )) except Exception as e: print(f"⚠️ [ES config check] Failed to read index replica settings: {e}", file=sys.stderr) return findings def _check_thread_pool_rejected(curl_fn) -> List[Finding]: """Detect thread_pool rejected counters > 0.""" findings: List[Finding] = [] _MONITORED_POOLS = { "search": ("ThreadPool.SearchRejected", "search"), "write": ("ThreadPool.WriteRejected", "write"), "bulk": ("ThreadPool.WriteRejected", "bulk"), } try: raw_tp = curl_fn("/_nodes/stats/thread_pool") nodes_tp = raw_tp.get("nodes", {}) pool_rejected: Dict[str, Dict[str, int]] = {} pool_completed: Dict[str, Dict[str, int]] = {} for _nid, _ninfo in nodes_tp.items(): _nname = _ninfo.get("name", _nid) for _pool, (_rcode, _label) in _MONITORED_POOLS.items(): _tp = _ninfo.get("thread_pool", {}).get(_pool, {}) _rej = _tp.get("rejected", 0) _comp = _tp.get("completed", 0) if _rej > 0: pool_rejected.setdefault(_pool, {})[_nname] = _rej pool_completed.setdefault(_pool, {})[_nname] = _comp merged: Dict[str, Dict] = {} for _pool, _by_node in pool_rejected.items(): _rcode, _label = _MONITORED_POOLS[_pool] entry = merged.setdefault( _rcode, {"label": _label, "by_node": {}, "total": 0, "completed_by_node": {}, "completed_total": 0}, ) for _n, _c in _by_node.items(): entry["by_node"][_n] = entry["by_node"].get(_n, 0) + _c entry["total"] += _c _c_done = pool_completed.get(_pool, {}).get(_n, 0) entry["completed_by_node"][_n] = entry["completed_by_node"].get(_n, 0) + _c_done entry["completed_total"] += _c_done for _rcode, _info in merged.items(): findings.append(Finding( event_code="HealthCheck.ThreadPoolSaturation", reason_code=_rcode, name=f"{_info['label']} thread pool rejected requests", priority="P1", category="Resource metrics", description=( f"The {_info['label']} thread pool rejected {_info['total']} request(s) " f"since last restart (cumulative); backlog exceeded pool capacity. " f"CMS/catalog P0 for this class usually requires sustained reject rate plus traffic " f"(see health-events-catalog); use engine evidence with CMS to grade severity." ), evidence={ "total_rejected": _info["total"], "completed_total": _info["completed_total"], "completed_by_node": _info["completed_by_node"], "rejections_by_node": _info["by_node"], "note": "rejected counters are cumulative since node restart; correlate with logs and traffic", }, remediation=[ f"Reduce concurrency or QPS for {_info['label']} workloads", "Inspect hot threads: GET /_nodes/hot_threads", "Inspect queue depth: GET /_nodes/stats/thread_pool (queue field)", "Longer term: scale out or add data nodes", ], )) except Exception as e: msg = str(e) if "-u" in msg or "elastic:" in msg: msg = "(error details omitted to avoid leaking credentials in logs)" print(f"⚠️ [ES config check] Failed to read thread pool stats: {msg}", file=sys.stderr) return findings def _check_cluster_config_optional( metrics: Optional[Dict[str, List[Dict]]] = None, instance_id: Optional[str] = None, *, connect_timeout: float = 5.0, read_timeout: float = 10.0, ) -> List[Finding]: """ Optional Elasticsearch API checks when ES_ENDPOINT and ES_PASSWORD are set. Preconditions (skip all checks, return []): - Endpoint vs instance_id consistency (Bug-07) - Safe credentials (no shell metacharacters in user/password) Checks (REST probes, not only CMS alerts): 1. JVMMemory.BreakerLimitConfigLow — fielddata breaker < 40% 2. Disk.WatermarkBreached — disk % above configured watermark.low vs CMS 3. Disk.WatermarkConfigLow — watermark.low < 5% 4. Disk.WatermarkConfigHigh — watermark.low > 90% 5. Index.ZeroReplicas — business indices with replicas=0 6. ThreadPool.SearchRejected — pool rejected > 0 (P1 snapshot; catalog P0 is rate+sustained). ThreadPool.WriteRejected — P1 unless JVMMemory.GCTimeRatioTooHigh also fires → promoted to P0 (ingest+GC co-headline). Args: metrics: CMS bundle including NodeDiskUtilization for % watermark breach; None skips breach. instance_id: Instance under diagnosis for endpoint consistency. connect_timeout: Passed to curl ``--connect-timeout`` (TCP connect phase). read_timeout: Transfer budget after connect; curl ``-m`` is set to ``connect_timeout + read_timeout`` (total operation ceiling, same spirit as prior fixed 15s default when defaults 5+10). Returns: List of Finding objects for configuration issues. """ import os endpoint = os.environ.get("ES_ENDPOINT", "").strip() password = os.environ.get("ES_PASSWORD", "").strip() if not endpoint or not password: return [] endpoint_check = _validate_endpoint_consistency(endpoint, instance_id) if endpoint_check == "skip": return [] username = os.environ.get("ES_USERNAME", "elastic").strip() or "elastic" endpoint_url = endpoint if endpoint.startswith(("http://", "https://")) else f"http://{endpoint}" if not _validate_credentials_safe(username, password): return [] ct = float(connect_timeout) rt = float(read_timeout) max_time = ct + rt subprocess_timeout = max_time + 5.0 _ct_s, _mt_s = f"{ct:g}", f"{max_time:g}" def _curl_json(path: str, method: str = "GET", body: Optional[str] = None) -> Dict[str, Any]: """ Call Elasticsearch HTTP API via curl argv list (not shell=True). Credentials are pre-validated for unsafe characters; argv list avoids shell injection. """ import json as _json import subprocess as _subprocess # argv list — no shell interpretation cmd = [ "curl", "-sS", "--connect-timeout", _ct_s, "-m", _mt_s, "-u", f"{username}:{password}", "-X", method.upper(), f"{endpoint_url}{path}", ] if body is not None: cmd += ["-H", "Content-Type: application/json", "-d", body] try: proc = _subprocess.run( cmd, capture_output=True, text=True, check=False, timeout=subprocess_timeout ) except _subprocess.TimeoutExpired: # Do not propagate TimeoutExpired — str(e) includes argv and leaks -u credentials. raise RuntimeError( f"request timed out (curl budget connect+read={max_time}s, subprocess cap={subprocess_timeout}s)" ) from None if proc.returncode != 0: raise RuntimeError((proc.stderr or proc.stdout or "").strip()) txt = (proc.stdout or "").strip() if not txt: return {} return _json.loads(txt) cluster_status: Optional[str] = None # green/yellow/red thread_pool_has_rejected = False # reserved for future use try: health_resp = _curl_json("/_cluster/health") cluster_status = health_resp.get("status", "").lower() except Exception as e: msg = str(e) if "-u" in msg or "elastic:" in msg: msg = "(request failed; details omitted to avoid leaking credentials in logs)" print( f"⚠️ [ES config check] Cannot reach Elasticsearch at {endpoint_url}: {msg}\n" f" Skipping optional config checks (circuit breakers / watermarks / zero-replica).", file=sys.stderr, ) return [] findings: List[Finding] = [] try: raw_settings = _curl_json("/_cluster/settings?include_defaults=true") findings.extend(_check_fielddata_breaker(raw_settings)) findings.extend(_check_disk_watermark_config(raw_settings, metrics, curl_fn=_curl_json)) except Exception as e: print(f"⚠️ [ES config check] Failed to read circuit breaker / watermark settings: {e}", file=sys.stderr) findings.extend(_check_zero_replicas(_curl_json)) findings.extend(_check_index_readonly_blocks(_curl_json)) findings.extend(_check_thread_pool_rejected(_curl_json)) return findings # --------------------------------------------------------------------------- # Rule engine — merge & order # --------------------------------------------------------------------------- def _promote_write_rejected_if_old_gc_wallclock_p0(findings: List[Finding]) -> None: """ When Old GC wall-clock share fires as P0 (JVMMemory.GCTimeRatioTooHigh), promote engine ThreadPool.WriteRejected from P1 → P0 so severity bands can match ingest+GC dual-P0 narratives. Co-occurrence is not causation — confirm with hot_threads / _tasks. """ codes = {f.reason_code for f in findings} if "JVMMemory.GCTimeRatioTooHigh" not in codes: return extra = ( " Co-occurs with JVMMemory.GCTimeRatioTooHigh (Old GC wall-clock share, P0): " "use dual P0 headlines or chain write/bulk overload → merges / heap → Old GC → CPU spikes — " "not a GC-tuning-only narrative. " "Co-occurrence does not prove ingest caused GC — confirm with hot_threads / _tasks." ) for i, f in enumerate(findings): if f.reason_code != "ThreadPool.WriteRejected" or f.priority != "P1": continue findings[i] = Finding( event_code=f.event_code, reason_code=f.reason_code, name=f.name, priority="P0", category=f.category, description=f.description + extra, evidence=f.evidence, remediation=f.remediation, ) def _finding_sort_key(f: Finding) -> tuple: """Priority first; within the same priority, thread-pool saturation before JVM memory (write/search narrative before GC-only).""" pri = {"P0": 0, "P1": 1, "P2": 2}.get(f.priority, 9) if f.reason_code in ("ThreadPool.WriteRejected", "ThreadPool.SearchRejected"): band = 0 elif f.reason_code.startswith("JVMMemory."): band = 1 else: band = 2 return (pri, band, f.reason_code) def _apply_rules( status_info: Dict, metrics: Dict[str, List[Dict]], events: List[Dict], logs: List[Dict], instance_id: Optional[str] = None, *, es_curl_connect_timeout: float = 5.0, es_curl_read_timeout: float = 10.0, ) -> List[Finding]: findings: List[Finding] = [] findings += _check_instance_status(status_info) findings += _check_cluster_health(metrics) findings += _check_cpu(metrics) findings += _check_memory(metrics) findings += _check_disk(metrics) findings += _check_resource_imbalance(metrics) findings += _check_events(events) findings += _check_logs(logs) findings += _check_cluster_config_optional( metrics, instance_id=instance_id, connect_timeout=es_curl_connect_timeout, read_timeout=es_curl_read_timeout, ) seen: set = set() deduped = [] for f in findings: if f.reason_code not in seen: seen.add(f.reason_code) deduped.append(f) _promote_write_rejected_if_old_gc_wallclock_p0(deduped) deduped.sort(key=_finding_sort_key) return deduped def _must_engine_footer_lines(findings: List[Finding]) -> List[str]: """ Deduped engine REST paths for SKILL.md section 5 (MUST table) rows implied by findings. Does not replace running curl — reminds the operator/agent after environment reachability checks. """ codes = {f.reason_code for f in findings} out: List[str] = [] def add(path: str) -> None: if path not in out: out.append(path) if codes & {"Cluster.StatusRed", "Cluster.StatusYellow"}: add("POST /_cluster/allocation/explain {}") add("GET /_cat/shards?v&h=index,shard,prirep,state,node,unassigned.reason") if codes & {"ThreadPool.SearchRejected", "ThreadPool.WriteRejected"}: add("GET /_nodes/hot_threads?threads=3") if codes & {"CPU.PersistUsageHigh", "CPU.PeakUsageHigh"}: add("GET /_nodes/hot_threads?threads=3") add("GET /_tasks?detailed=true&actions=*search*") if codes & { "JVMMemory.OldGenUsageCritical", "JVMMemory.OldGenUsageHigh", "JVMMemory.GCTimeRatioTooHigh", "JVMMemory.GCRateTooHigh", "JVMMemory.OOM", }: add("GET /_nodes/stats/breaker?pretty") if "JVMMemory.BreakerLimitConfigLow" in codes: add("GET /_cluster/settings?include_defaults=true # transient + persistent: indices.breaker.*") add("GET /_nodes/stats/breaker?pretty") if codes & { "Balancing.NodeCPUUnbalanced", "Balancing.NodeDiskUnbalanced", "Balancing.NodeMemoryUnbalanced", }: add("GET /_cat/shards?v&h=index,shard,prirep,state,node,docs,store") add("GET /_cat/allocation?v") if codes & { "Disk.IndexReadOnly", "Disk.WatermarkBreached", "Disk.WatermarkAbsoluteFloodBreached", "Disk.WatermarkAbsoluteFloodMarginLow", "Disk.WatermarkAbsoluteValue", }: add("GET /_cluster/settings?include_defaults=true&filter_path=**.watermark") add("GET /_all/_settings?filter_path=*.settings.index.blocks") return out # --------------------------------------------------------------------------- # Report # --------------------------------------------------------------------------- def _print_report( instance_id: str, status_info: Dict, findings: List[Finding], metrics: Dict[str, List[Dict]], begin_ms: int, end_ms: int, ): W = 82 print(f"\n{'═' * W}") print(f" Elasticsearch instance health report") print(f"{'═' * W}") print(f" Instance ID : {instance_id}") if status_info and "_error" not in status_info: status = status_info.get("status", "-") from_v = { "active": "active (running)", "activating": "activating (change in progress)", "inactive": "inactive (frozen)", "invalid": "invalid (unavailable)", } print(f" Instance name : {status_info.get('name', '-')}") print(f" Control status : {from_v.get(status, status)}") print(f" ES version : {status_info.get('es_version', '-')}") print(f" Node count : {status_info.get('node_count', '-')}") print(f" Last updated : {status_info.get('updated_at', '-')}") print(f" Analysis window : {ms_to_str(begin_ms)} ~ {ms_to_str(end_ms)}") print(f" Report time : {datetime.now().strftime('%Y-%m-%d %H:%M:%S')}") # ── Summary ── print(f"\n{'─' * W}") if not findings: print(f" ✅ No findings — instance appears healthy.") else: p0 = [f for f in findings if f.priority == "P0"] p1 = [f for f in findings if f.priority == "P1"] p2 = [f for f in findings if f.priority == "P2"] print(f" 📊 {len(findings)} finding(s): " f"🔴 P0×{len(p0)} 🟡 P1×{len(p1)} 🔵 P2×{len(p2)}") # ── Findings ── for i, f in enumerate(findings, 1): icon = PRIORITY_ICON.get(f.priority, "⚪") print(f"\n{'─' * W}") print(f" [{i}] {icon} {f.priority} | {f.category} | {f.name}") print(f" Event code : {f.event_code}") print(f" Reason code : {f.reason_code}") print(f" Description : {f.description}") if f.evidence: print(f" Evidence :") for k, v in f.evidence.items(): if isinstance(v, list): for item in v: print(f" - {item}") else: print(f" {k}: {v}") if f.remediation: print(f" Remediation :") for r in f.remediation: print(f" • {r}") must_paths = _must_engine_footer_lines(findings) if must_paths: print(f"\n{'─' * W}") print( " Engine API checklist (SKILL.md sections 5–7); run after ES endpoint is reachable:" ) for p in must_paths: print(f" • {p}") print(f"{'─' * W}") # Read-heavy path: do not let GC/CPU findings imply search pool was ruled out. # Skip when write-path rejection is present — bulk/write-primary diagnosis should follow # sop-write-performance §2, not a search-pool headline. # See references/sop-query-thread-pool.md (Report narrative: search pool vs GC / CPU headlines). rc = {f.reason_code for f in findings} if ( "ThreadPool.SearchRejected" not in rc and "ThreadPool.WriteRejected" not in rc and ( "JVMMemory.GCTimeRatioTooHigh" in rc or "JVMMemory.GCRateTooHigh" in rc or "CPU.PeakUsageHigh" in rc or "CPU.PersistUsageHigh" in rc ) ): print(f"\n{'─' * W}") print(" Narrative note (read-heavy / search-pool overload):") print(" ThreadPool.SearchRejected was not in this run's findings. If query load may") print(" exceed search pool capacity, verify GET /_nodes/stats/thread_pool") print(" (search.rejected / queue) on a stable path; GC/CPU may be co-stress.") print(" See references/sop-query-thread-pool.md (Report narrative: search pool vs GC / CPU headlines).") print(f"{'─' * W}") # ── Health grade ── print(f"\n{'─' * W}") grade, grade_icon, grade_desc = _compute_security_score(findings) print(f" Health grade {grade_icon} {grade} — {grade_desc}") print(f"{'─' * W}") # ── Metric summary ── print(f"\n{'═' * W}") print(f" Key metrics (avg / max / min in window)") print(f"{'─' * W}") DISPLAY_METRICS = [ ("ClusterStatus", "Cluster health (CMS) "), ("ClusterDisconnectedNodeCount", "Disconnected nodes "), ("NodeCPUUtilization", "Node CPU (%) "), ("NodeHeapMemoryUtilization", "Node heap (%) "), ("NodeDiskUtilization", "Node disk (%) "), ("NodeStatsDataDiskUtil", "Node disk IO util (%) "), ("NodeLoad_1m", "Node load_1m "), ("JVMGCOldCollectionCount", "Old GC count "), ("JVMGCOldCollectionDuration", "Old GC duration (ms) "), ] STATUS_MAP = {0: "Green", 1: "Yellow", 2: "Red"} for metric_name, label in DISPLAY_METRICS: dps = metrics.get(metric_name, []) if not dps: continue summary = _summarize(dps, metric_name) print(f"\n {label}") for node, stat in summary.items(): avg_v = stat["avg"] if metric_name == "ClusterStatus": val_str = f"avg={STATUS_MAP.get(int(round(avg_v)), str(int(round(avg_v))))} max={STATUS_MAP.get(int(round(stat['max'])), str(int(round(stat['max']))))}" else: val_str = f"avg={avg_v:.2f} max={stat['max']:.2f} min={stat['min']:.2f}" print(f" {node:<50} {val_str}") print(f"\n{'═' * W}") print(f" Check complete") print(f"{'═' * W}\n") # --------------------------------------------------------------------------- # CLI entry # --------------------------------------------------------------------------- def check( instance_id: str, region_id: str, window_min: int = 60, profile: Optional[str] = None, data_source: str = "auto", input_bundle: Optional[Dict[str, Any]] = None, *, connect_timeout: float = 5.0, read_timeout: float = 10.0, ): end_ms = now_ms() begin_ms = ago_ms(window_min) period = _cms_period_for_window(window_min) print(f"\n🔍 Health check: {instance_id} (region: {region_id})") print(f" Window: {ms_to_str(begin_ms)} ~ {ms_to_str(end_ms)} (last {window_min} minutes)") print(f" CMS period: {period}s ({int(period)//60} minute(s) bucket)") print(f" Data source: {data_source}") if profile: print(f" CLI profile: {profile}") print(f" Collecting status / metrics / events / logs...\n") bundle = input_bundle or {} injected_status = bundle.get("status_info") injected_metrics = bundle.get("metrics") injected_events = bundle.get("events") injected_logs = bundle.get("logs") use_input_only = data_source == "input" use_cli_only = data_source == "cli" if isinstance(injected_status, dict) and not use_cli_only: status_info = injected_status elif use_input_only: status_info = {} else: status_info = fetch_instance_status_info(instance_id, region_id, profile=profile) if isinstance(injected_metrics, dict) and not use_cli_only: metrics = injected_metrics elif use_input_only: metrics = {} else: metrics = fetch_metrics_batch( instance_id, region_id, _DIAG_METRICS, begin_ms, end_ms, period, profile=profile ) if isinstance(injected_events, list) and not use_cli_only: events = injected_events elif use_input_only: events = [] else: events = fetch_events(instance_id, region_id, begin_ms, end_ms, profile=profile) if isinstance(injected_logs, list) and not use_cli_only: logs = injected_logs elif use_input_only: logs = [] else: logs = _fetch_error_logs(instance_id, region_id, begin_ms, end_ms, profile=profile) if data_source != "input": _check_metrics_sparse(metrics, window_min, period) findings = _apply_rules( status_info, metrics, events, logs, instance_id=instance_id, es_curl_connect_timeout=connect_timeout, es_curl_read_timeout=read_timeout, ) _print_report(instance_id, status_info, findings, metrics, begin_ms, end_ms) def main(): parser = argparse.ArgumentParser( description="Alibaba Cloud Elasticsearch instance health check (rules baseline 20260318).", formatter_class=argparse.RawDescriptionHelpFormatter, epilog=""" Coverage (event / reason codes): Control plane ManagementPlane.ActivatingStuck Long-running activating ManagementPlane.EventStuck System event stuck Executing ManagementPlane.InstanceInactive Instance frozen ManagementPlane.InstanceInvalid Instance invalid Cluster HealthCheck.ClusterUnhealthy Cluster.StatusRed Cluster Red (P0) Cluster.StatusYellow Cluster Yellow (P1) Node.Disconnected Node disconnected (P0) Cluster.UnavailableShards Primary unavailable [logs] (P0) CPU / load HealthCheck.CPULoadHigh CPU.PersistUsageHigh Sustained CPU high >70% (P0) / >60% (P1) CPU.PeakUsageHigh CPU peak >95% (P0) / 80–94% (P1) HealthCheck.LoadUnbalanced Balancing.NodeCPUUnbalanced CPU imbalance CV>0.3 (P1) Balancing.NodeDiskUnbalanced Disk imbalance CV>0.3 (P0/P1/P2) Balancing.NodeMemoryUnbalanced Memory imbalance CV>0.3 (P1) JVM / GC HealthCheck.JVMMemoryPressure JVMMemory.OldGenUsageCritical Heap >85% (P0) JVMMemory.OldGenUsageHigh Heap >75% (P1) JVMMemory.GCTimeRatioTooHigh GC time ratio >10% (P0) JVMMemory.GCRateTooHigh Old GC >1/min (P1) JVMMemory.OOM OutOfMemoryError in logs (P0) Disk HealthCheck.DiskUsageHigh Disk.UsageCritical Disk >85% (P0) Disk.UsageHigh Disk >75% (P1) HealthCheck.DiskIOBottleneck Disk.IOPerformancePoor Disk IO util >90% (P0) Examples: python3 check_es_instance_health.py -i es-cn-xxx -r cn-hangzhou python3 check_es_instance_health.py -i es-cn-xxx -r cn-hangzhou --window 120 python3 check_es_instance_health.py -i es-cn-xxx -r cn-hangzhou --profile prod python3 check_es_instance_health.py -i es-cn-xxx -r cn-hangzhou --connect-timeout 3 --read-timeout 10 python3 check_es_instance_health.py -i es-cn-xxx -r cn-hangzhou --data-source input --input-json-file /tmp/diag.json Credentials: Prefer `aliyun configure` profiles; pass --profile when needed. """, ) parser.add_argument("--instance-id", "-i", required=True, help="Elasticsearch instance ID") parser.add_argument("--region-id", "-r", required=True, help="Region ID, e.g. cn-hangzhou") parser.add_argument( "--window", "-w", type=int, default=60, help=( "Analysis window in minutes (default 60). " "For window ≤30 minutes, CMS period=60s; for >30 minutes, period=300s " "(avoids empty CMS responses at 60s retention boundaries)." ), ) parser.add_argument("--profile", default=None, help="Aliyun CLI profile name (default profile if omitted)") parser.add_argument( "--data-source", choices=["auto", "cli", "input"], default="auto", help="auto: use injected bundle first, then CLI; cli: CLI only; input: injected bundle only", ) parser.add_argument("--input-json", default=None, help="Injected diagnostic JSON string") parser.add_argument("--input-json-file", default=None, help="Path to injected diagnostic JSON file") parser.add_argument( "--connect-timeout", type=float, default=5.0, metavar="SEC", help=( "Elasticsearch engine probes (optional curl to ES_*): max seconds for TCP connect " "(curl --connect-timeout). Default 5." ), ) parser.add_argument( "--read-timeout", type=float, default=10.0, metavar="SEC", help=( "Elasticsearch engine probes: transfer time budget in seconds; curl -m uses " "connect-timeout + read-timeout (total operation ceiling). Default 10 (15s total with defaults)." ), ) args = parser.parse_args() if args.connect_timeout <= 0 or args.read_timeout <= 0: parser.error("--connect-timeout and --read-timeout must be positive") input_bundle = _load_json_bundle(args.input_json, args.input_json_file) check( args.instance_id, args.region_id, args.window, profile=args.profile, data_source=args.data_source, input_bundle=input_bundle, connect_timeout=args.connect_timeout, read_timeout=args.read_timeout, ) if __name__ == "__main__": main() FILE:scripts/openapi_cli_collect.py #!/usr/bin/env python3 """ OpenAPI data collection via Aliyun CLI (shared by check_es_instance_health.py). Design goals: - Do not rely on OPENAPI_* env vars; prefer `aliyun configure` profiles. - Preserve response shapes compatible with legacy get_es_instance_*.py scripts so rules do not regress. """ import json import os import subprocess from concurrent.futures import ThreadPoolExecutor, as_completed from datetime import datetime, timedelta from typing import Any, Dict, List, Optional NAMESPACE = "acs_elasticsearch" # Metric definitions required for health diagnosis (field names aligned with legacy collectors) METRIC_DEFINITIONS: Dict[str, Dict[str, str]] = { "ClusterStatus": {"group_field": "clusterId", "value_field": "Value"}, "ClusterDisconnectedNodeCount": {"group_field": "clusterId", "value_field": "Maximum"}, "ClusterNodeCount": {"group_field": "clusterId", "value_field": "Maximum"}, "ClusterShardCount": {"group_field": "clusterId", "value_field": "Maximum"}, "ClusterQueryQPS": {"group_field": "clusterId", "value_field": "Average"}, "ClusterIndexQPS": {"group_field": "clusterId", "value_field": "Average"}, "NodeCPUUtilization": {"group_field": "nodeIP", "value_field": "Average"}, "NodeHeapMemoryUtilization": {"group_field": "nodeIP", "value_field": "Average"}, "NodeDiskUtilization": {"group_field": "nodeIP", "value_field": "Average"}, "NodeFreeStorageSpace": {"group_field": "nodeIP", "value_field": "Minimum"}, "NodeLoad_1m": {"group_field": "nodeIP", "value_field": "Average"}, "NodeStatsDataDiskUtil": {"group_field": "nodeIP", "value_field": "Maximum"}, "JVMGCOldCollectionCount": {"group_field": "nodeIP", "value_field": "Maximum"}, "JVMGCOldCollectionDuration": {"group_field": "nodeIP", "value_field": "Maximum"}, } def _run_aliyun_json( product: str, action: str, params: Dict[str, Any], region_id: Optional[str] = None, profile: Optional[str] = None, timeout_sec: int = 30, ) -> Dict[str, Any]: """ Invoke Aliyun CLI and parse JSON. Raises RuntimeError on failure; callers may degrade. Security: subprocess.run uses argv list mode (no shell=True). Arguments are passed as discrete list elements and are not interpreted by a shell. Values are validated to reject characters that could be abused for injection. """ env = os.environ.copy() if not env.get("ALIBABA_CLOUD_USER_AGENT"): env["ALIBABA_CLOUD_USER_AGENT"] = "AlibabaCloud-Agent-Skills" _DANGEROUS_CHARS = set('`$()|;&<>\n\r') _JSON_LITERAL_PARAM_KEYS = frozenset({"Dimensions"}) def _is_safe(val: str) -> bool: return not any(c in _DANGEROUS_CHARS for c in str(val)) for k, v in params.items(): if v is None: continue if k in _JSON_LITERAL_PARAM_KEYS: continue if not _is_safe(str(v)): raise RuntimeError( f"aliyun {product} {action}: parameter {k!r} contains unsafe characters; refused" ) cmd: List[str] = ["aliyun"] if profile: if not _is_safe(profile): raise RuntimeError("profile name contains unsafe characters") cmd += ["--profile", profile] cmd += [product, action] if region_id: if not _is_safe(region_id): raise RuntimeError("region_id contains unsafe characters") cmd += ["--region", region_id] for k, v in params.items(): if v is None: continue cmd += [f"--{k}", str(v)] proc = subprocess.run( cmd, capture_output=True, text=True, check=False, timeout=timeout_sec, env=env, ) if proc.returncode != 0: stderr = (proc.stderr or "").strip() raise RuntimeError(f"aliyun {product} {action} failed: {stderr or proc.stdout.strip()}") try: return json.loads(proc.stdout or "{}") except json.JSONDecodeError as e: raise RuntimeError(f"aliyun {product} {action} returned non-JSON: {e}") def normalize_datapoints(datapoints: List[Dict], metric_def: Dict[str, str]) -> Dict[str, List[Dict]]: """Group CMS datapoints by group_field and normalize value/timestamp fields.""" group_field = metric_def.get("group_field", "nodeIP") value_field = metric_def.get("value_field", "Average") grouped: Dict[str, List[Dict]] = {} for dp in datapoints or []: key = dp.get(group_field) or dp.get("nodeIP") or dp.get("clusterId") or "UNKNOWN" value = dp.get(value_field) if value is None: value = dp.get("Value", dp.get("Average", dp.get("Maximum", dp.get("Minimum")))) ts = dp.get("timestamp", dp.get("Timestamp", 0)) try: value = float(value) if value is not None else None except (TypeError, ValueError): value = None grouped.setdefault(str(key), []).append({"timestamp": int(ts or 0), "value": value, "raw": dp}) return grouped def fetch_instance_status_info(instance_id: str, region_id: str, profile: Optional[str] = None) -> Dict[str, Any]: """Replacement for get_es_instance_status.fetch_instance_status_info.""" try: data = _run_aliyun_json( "elasticsearch", "DescribeInstance", {"InstanceId": instance_id}, region_id=region_id, profile=profile, timeout_sec=30, ) r = data.get("Result", {}) return { "name": r.get("description", "-"), "status": r.get("status", "-"), "es_version": r.get("esVersion", "-"), "updated_at": r.get("updatedAt", "") or "", "created_at": r.get("createdAt", "") or "", "node_count": r.get("nodeAmount", 0) or 0, "protocol": r.get("protocol", "-"), } except Exception as e: return {"_error": str(e)} def fetch_instance_detail(instance_id: str, region_id: str, profile: Optional[str] = None) -> Dict[str, Any]: """Full DescribeInstance payload (clusterTasks, etc.) — replaces get_es_instance_detail.py.""" return _run_aliyun_json( "elasticsearch", "DescribeInstance", {"InstanceId": instance_id}, region_id=region_id, profile=profile, timeout_sec=30, ).get("Result", {}) def _fetch_metric_points( instance_id: str, region_id: str, metric_name: str, begin_ms: int, end_ms: int, period: str, profile: Optional[str] = None, ) -> List[Dict]: """Fetch one metric with DescribeMetricList pagination (NextToken).""" points: List[Dict] = [] next_token: Optional[str] = None max_pages = 20 # CMS DescribeMetricList expects millisecond timestamps start_ms = str(int(begin_ms)) end_ms_str = str(int(end_ms)) for _ in range(max_pages): params = { "Namespace": NAMESPACE, "MetricName": metric_name, "Dimensions": json.dumps([{"clusterId": instance_id}], ensure_ascii=False), "StartTime": start_ms, "EndTime": end_ms_str, "Period": period, "Length": 1440, "NextToken": next_token, } resp = _run_aliyun_json("cms", "DescribeMetricList", params, region_id=region_id, profile=profile) raw = resp.get("Datapoints", "[]") batch = json.loads(raw) if isinstance(raw, str) else (raw or []) if isinstance(batch, list): points.extend(batch) next_token = resp.get("NextToken") if not next_token: break return points def fetch_metrics_batch( instance_id: str, region_id: str, metric_names: List[str], begin_ms: int, end_ms: int, period: str = "300", profile: Optional[str] = None, ) -> Dict[str, List[Dict]]: """Replacement for get_es_instance_metrics.fetch_metrics_batch.""" result: Dict[str, List[Dict]] = {} with ThreadPoolExecutor(max_workers=min(8, max(1, len(metric_names)))) as ex: fut_map = { ex.submit( _fetch_metric_points, instance_id, region_id, m, begin_ms, end_ms, period, profile, ): m for m in metric_names } for fut in as_completed(fut_map): m = fut_map[fut] try: result[m] = fut.result() except Exception: result[m] = [] return result def fetch_events( instance_id: str, region_id: str, begin_ms: int, end_ms: int, profile: Optional[str] = None, ) -> List[Dict]: """Replacement for get_es_instance_events.fetch_events — list of normalized dicts.""" events: List[Dict] = [] page = 1 page_size = 100 start_ms = str(int(begin_ms)) end_ms_str = str(int(end_ms)) while True: resp = _run_aliyun_json( "cms", "DescribeSystemEventAttribute", { "Product": "elasticsearch", "SearchKeywords": instance_id, "StartTime": start_ms, "EndTime": end_ms_str, "PageNumber": page, "PageSize": page_size, }, region_id=region_id, profile=profile, ) wrapper = (resp.get("SystemEvents") or {}).get("SystemEvent") or [] if not wrapper: break for e in wrapper: content_raw = e.get("Content") content = {} if isinstance(content_raw, str): try: content = json.loads(content_raw) except json.JSONDecodeError: content = {} events.append( { "name": e.get("Name", ""), "time_ms": int(e.get("Time", 0) or 0), "level": e.get("Level", ""), "event_status": content.get("eventStatus", ""), "reason": content.get("reasonCode", content.get("reason", "")), "start_time": content.get("executeStartTime", ""), "finish_time": content.get("executeFinishTime", ""), } ) if len(wrapper) < page_size: break page += 1 events.sort(key=lambda x: x.get("time_ms", 0), reverse=True) return events def fetch_log_items( instance_id: str, region_id: str, log_type: str = "INSTANCELOG", begin_ms: Optional[int] = None, end_ms: Optional[int] = None, query: str = "*", size: int = 50, profile: Optional[str] = None, ) -> List[Dict]: """Replacement for get_es_instance_log.fetch_log_items.""" now = datetime.now() if begin_ms is None: begin_ms = int((now - timedelta(minutes=10)).timestamp() * 1000) if end_ms is None: end_ms = int(now.timestamp() * 1000) resp = _run_aliyun_json( "elasticsearch", "ListSearchLog", { "InstanceId": instance_id, "type": log_type, "query": query or "*", "beginTime": int(begin_ms), "endTime": int(end_ms), "page": 1, "size": max(1, min(int(size), 50)), }, region_id=region_id, profile=profile, ) return resp.get("Result") or []

Alibabacloud Elasticsearch Instance Diagnose

Comments (0)