Step 1 — Verify which indices are blocked

# List indices and health
curl -sS "$OS_URL/_cat/indices?v"

# Show block-related settings across all indices
curl -sS "$OS_URL/_all/_settings?flat_settings=true&filter_path=**.blocks.*" | jq

# Inspect a specific index
curl -sS "$OS_URL/<index>/_settings?flat_settings=true" | jq

Step 2 — Check common root causes

• Disk flood-stage triggered (very common).
• Manual/automation block (admin set block flags).
• Index closed (cannot receive writes).
• Security/privilege issues (less common if it worked before).

Quick disk checks:

# Watermark configuration
curl -sS "$OS_URL/_cluster/settings?include_defaults=true"

# Node disk usage
curl -sS "$OS_URL/_cat/allocation?v"
curl -sS "$OS_URL/_cat/nodes?v&h=name,ip,diskAvail,diskUsedPct"

Step 3 — Unblock the affected indices

# Clear write-block flags on all indices
curl -sS -X PUT "$OS_URL/_all/_settings" -H 'Content-Type: application/json' -d '{
 "index.blocks.read_only_allow_delete": false,
 "index.blocks.read_only": false,
 "index.blocks.write": false
}'

# Reopen a closed index
curl -sS -X POST "$OS_URL/<index>/_open"

Optional: tune watermarks (persistent):

curl -sS -X PUT "$OS_URL/_cluster/settings" -H 'Content-Type: application/json' -d '{
 "persistent": {
 "cluster.routing.allocation.disk.watermark.low": "85%",
 "cluster.routing.allocation.disk.watermark.high": "90%",
 "cluster.routing.allocation.disk.watermark.flood_stage": "95%",
 "cluster.info.update.interval": "1m"
 }
}'

Step 4 — Manually rotate the Graylog write index

Via Graylog UI

1. Go to System → Indices → Index Sets.
2. Open the affected Index Set.
3. Click Rotate active write index.

Step 5 — Post-rotation checks

• Graylog inputs show a non-zero ingest rate again.
• /_cat/aliases?v shows the deflector alias pointing to the newest index.
• /_cat/indices?v shows the new index increasing in docs.count.
• No new read-only / allow delete errors in Graylog logs.

Check alias and index activity:

curl -sS "$OS_URL/_cat/aliases?v"
curl -sS "$OS_URL/_cat/indices?v&s=pri.store.size:desc"