Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Error connecting to API]Something went wrong #23522

Closed
jesse-zhangh opened this issue May 20, 2024 · 3 comments
Closed

[Error connecting to API]Something went wrong #23522

jesse-zhangh opened this issue May 20, 2024 · 3 comments

Comments

@jesse-zhangh
Copy link

jesse-zhangh commented May 20, 2024 •
edited

Wazuh version Component Install type Install method Platform
4.7.2 Wazuh component Manager/Agent helm k8s 1.28

Web access:
image
image

curl -k -u 'admin:***' https://xxx:xxx/_cluster/health?pretty

{
  "cluster_name" : "wazuh",
  "status" : "red",
  "timed_out" : false,
  "number_of_nodes" : 2,
  "number_of_data_nodes" : 2,
  "discovered_master" : true,
  "discovered_cluster_manager" : true,
  "active_primary_shards" : 242,
  "active_shards" : 373,
  "relocating_shards" : 0,
  "initializing_shards" : 0,
  "unassigned_shards" : 175,
  "delayed_unassigned_shards" : 0,
  "number_of_pending_tasks" : 0,
  "number_of_in_flight_fetch" : 0,
  "task_max_waiting_in_queue_millis" : 0,
  "active_shards_percent_as_number" : 68.06569343065694
}

curl -k -X GET "https://xxxx:xxxx/" -H "Authorization: Bearer $(curl -u 'wazuh-wui:***' -k -X GET 'https://xxxx:xxxx/security/user/authenticate?raw=true')"

 % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   404  100   404    0     0    907      0 --:--:-- --:--:-- --:--:--   907
{"data": {"title": "Wazuh API REST", "api_version": "4.7.2", "revision": 40711, "license_name": "GPL 2.0", "license_url": "https://github.com/wazuh/wazuh/blob/v4.7.2/LICENSE", "hostname": "wazuh-manager-master-0", "timestamp": "2024-05-20T10:32:53Z"}, "error": 0}

Wazuh indexer logs

[2024-05-20T10:33:31,232][WARN ][o.o.c.c.ClusterFormationFailureHelper] [wazuh-indexer-0] cluster-manager not discovered or elected yet, an election requires at least 2 nodes with ids from [cmKpdlIeQdmcSCkD74n-_Q, tedR2ga_TQqLsJhy1wyekw, uBwEvV4HSEiv4qDCMi6w9A], have discovered [{wazuh-indexer-0}{cmKpdlIeQdmcSCkD74n-_Q}{jThzC5zoQQGHitPfUPyhDg}{10.244.1.105}{10.244.1.105:9300}{dimr}{shard_indexing_pressure_enabled=true}] which is not a quorum; discovery will continue using [] from hosts providers and [{wazuh-indexer-0}{cmKpdlIeQdmcSCkD74n-_Q}{jThzC5zoQQGHitPfUPyhDg}{10.244.1.105}{10.244.1.105:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 74, last-accepted version 8100 in term 74
[2024-05-20T10:33:41,232][WARN ][o.o.c.c.ClusterFormationFailureHelper] [wazuh-indexer-0] cluster-manager not discovered or elected yet, an election requires at least 2 nodes with ids from [cmKpdlIeQdmcSCkD74n-_Q, tedR2ga_TQqLsJhy1wyekw, uBwEvV4HSEiv4qDCMi6w9A], have discovered [{wazuh-indexer-0}{cmKpdlIeQdmcSCkD74n-_Q}{jThzC5zoQQGHitPfUPyhDg}{10.244.1.105}{10.244.1.105:9300}{dimr}{shard_indexing_pressure_enabled=true}] which is not a quorum; discovery will continue using [] from hosts providers and [{wazuh-indexer-0}{cmKpdlIeQdmcSCkD74n-_Q}{jThzC5zoQQGHitPfUPyhDg}{10.244.1.105}{10.244.1.105:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 74, last-accepted version 8100 in term 74
[2024-05-20T10:33:50,449][ERROR][o.o.s.a.BackendRegistry  ] [wazuh-indexer-0] Not yet initialized (you may need to run securityadmin)
[2024-05-20T10:33:51,232][WARN ][o.o.c.c.ClusterFormationFailureHelper] [wazuh-indexer-0] cluster-manager not discovered or elected yet, an election requires at least 2 nodes with ids from [cmKpdlIeQdmcSCkD74n-_Q, tedR2ga_TQqLsJhy1wyekw, uBwEvV4HSEiv4qDCMi6w9A], have discovered [{wazuh-indexer-0}{cmKpdlIeQdmcSCkD74n-_Q}{jThzC5zoQQGHitPfUPyhDg}{10.244.1.105}{10.244.1.105:9300}{dimr}{shard_indexing_pressure_enabled=true}] which is not a quorum; discovery will continue using [] from hosts providers and [{wazuh-indexer-0}{cmKpdlIeQdmcSCkD74n-_Q}{jThzC5zoQQGHitPfUPyhDg}{10.244.1.105}{10.244.1.105:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 74, last-accepted version 8100 in term 74
[2024-05-20T10:33:51,390][ERROR][o.o.s.a.BackendRegistry  ] [wazuh-indexer-0] Not yet initialized (you may need to run securityadmin)
[2024-05-20T10:34:01,233][WARN ][o.o.c.c.ClusterFormationFailureHelper] [wazuh-indexer-0] cluster-manager not discovered or elected yet, an election requires at least 2 nodes with ids from [cmKpdlIeQdmcSCkD74n-_Q, tedR2ga_TQqLsJhy1wyekw, uBwEvV4HSEiv4qDCMi6w9A], have discovered [{wazuh-indexer-0}{cmKpdlIeQdmcSCkD74n-_Q}{jThzC5zoQQGHitPfUPyhDg}{10.244.1.105}{10.244.1.105:9300}{dimr}{shard_indexing_pressure_enabled=true}] which is not a quorum; discovery will continue using [] from hosts providers and [{wazuh-indexer-0}{cmKpdlIeQdmcSCkD74n-_Q}{jThzC5zoQQGHitPfUPyhDg}{10.244.1.105}{10.244.1.105:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 74, last-accepted version 8100 in term 74
[2024-05-20T10:34:11,233][WARN ][o.o.c.c.ClusterFormationFailureHelper] [wazuh-indexer-0] cluster-manager not discovered or elected yet, an election requires at least 2 nodes with ids from [cmKpdlIeQdmcSCkD74n-_Q, tedR2ga_TQqLsJhy1wyekw, uBwEvV4HSEiv4qDCMi6w9A], have discovered [{wazuh-indexer-0}{cmKpdlIeQdmcSCkD74n-_Q}{jThzC5zoQQGHitPfUPyhDg}{10.244.1.105}{10.244.1.105:9300}{dimr}{shard_indexing_pressure_enabled=true}] which is not a quorum; discovery will continue using [] from hosts providers and [{wazuh-indexer-0}{cmKpdlIeQdmcSCkD74n-_Q}{jThzC5zoQQGHitPfUPyhDg}{10.244.1.105}{10.244.1.105:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 74, last-accepted version 8100 in term 74
[2024-05-20T10:34:21,234][WARN ][o.o.c.c.ClusterFormationFailureHelper] [wazuh-indexer-0] cluster-manager not discovered or elected yet, an election requires at least 2 nodes with ids from [cmKpdlIeQdmcSCkD74n-_Q, tedR2ga_TQqLsJhy1wyekw, uBwEvV4HSEiv4qDCMi6w9A], have discovered [{wazuh-indexer-0}{cmKpdlIeQdmcSCkD74n-_Q}{jThzC5zoQQGHitPfUPyhDg}{10.244.1.105}{10.244.1.105:9300}{dimr}{shard_indexing_pressure_enabled=true}] which is not a quorum; discovery will continue using [] from hosts providers and [{wazuh-indexer-0}{cmKpdlIeQdmcSCkD74n-_Q}{jThzC5zoQQGHitPfUPyhDg}{10.244.1.105}{10.244.1.105:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 74, last-accepted version 8100 in term 74
[2024-05-20T10:34:26,372][ERROR][o.o.s.a.BackendRegistry  ] [wazuh-indexer-0] Not yet initialized (you may need to run securityadmin)
[2024-05-20T10:34:31,234][WARN ][o.o.c.c.ClusterFormationFailureHelper] [wazuh-indexer-0] cluster-manager not discovered or elected yet, an election requires at least 2 nodes with ids from [cmKpdlIeQdmcSCkD74n-_Q, tedR2ga_TQqLsJhy1wyekw, uBwEvV4HSEiv4qDCMi6w9A], have discovered [{wazuh-indexer-0}{cmKpdlIeQdmcSCkD74n-_Q}{jThzC5zoQQGHitPfUPyhDg}{10.244.1.105}{10.244.1.105:9300}{dimr}{shard_indexing_pressure_enabled=true}] which is not a quorum; discovery will continue using [] from hosts providers and [{wazuh-indexer-0}{cmKpdlIeQdmcSCkD74n-_Q}{jThzC5zoQQGHitPfUPyhDg}{10.244.1.105}{10.244.1.105:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 74, last-accepted version 8100 in term 74
[2024-05-20T10:34:38,944][ERROR][o.o.s.a.BackendRegistry  ] [wazuh-indexer-0] Not yet initialized (you may need to run securityadmin)
[2024-05-20T10:34:41,234][WARN ][o.o.c.c.ClusterFormationFailureHelper] [wazuh-indexer-0] cluster-manager not discovered or elected yet, an election requires at least 2 nodes with ids from [cmKpdlIeQdmcSCkD74n-_Q, tedR2ga_TQqLsJhy1wyekw, uBwEvV4HSEiv4qDCMi6w9A], have discovered [{wazuh-indexer-0}{cmKpdlIeQdmcSCkD74n-_Q}{jThzC5zoQQGHitPfUPyhDg}{10.244.1.105}{10.244.1.105:9300}{dimr}{shard_indexing_pressure_enabled=true}] which is not a quorum; discovery will continue using [] from hosts providers and [{wazuh-indexer-0}{cmKpdlIeQdmcSCkD74n-_Q}{jThzC5zoQQGHitPfUPyhDg}{10.244.1.105}{10.244.1.105:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 74, last-accepted version 8100 in term 74
[2024-05-20T10:34:51,235][WARN ][o.o.c.c.ClusterFormationFailureHelper] [wazuh-indexer-0] cluster-manager not discovered or elected yet, an election requires at least 2 nodes with ids from [cmKpdlIeQdmcSCkD74n-_Q, tedR2ga_TQqLsJhy1wyekw, uBwEvV4HSEiv4qDCMi6w9A], have discovered [{wazuh-indexer-0}{cmKpdlIeQdmcSCkD74n-_Q}{jThzC5zoQQGHitPfUPyhDg}{10.244.1.105}{10.244.1.105:9300}{dimr}{shard_indexing_pressure_enabled=true}] which is not a quorum; discovery will continue using [] from hosts providers and [{wazuh-indexer-0}{cmKpdlIeQdmcSCkD74n-_Q}{jThzC5zoQQGHitPfUPyhDg}{10.244.1.105}{10.244.1.105:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 74, last-accepted version 8100 in term 74
[2024-05-20T10:35:01,235][WARN ][o.o.c.c.ClusterFormationFailureHelper] [wazuh-indexer-0] cluster-manager not discovered or elected yet, an election requires at least 2 nodes with ids from [cmKpdlIeQdmcSCkD74n-_Q, tedR2ga_TQqLsJhy1wyekw, uBwEvV4HSEiv4qDCMi6w9A], have discovered [{wazuh-indexer-0}{cmKpdlIeQdmcSCkD74n-_Q}{jThzC5zoQQGHitPfUPyhDg}{10.244.1.105}{10.244.1.105:9300}{dimr}{shard_indexing_pressure_enabled=true}] which is not a quorum; discovery will continue using [] from hosts providers and [{wazuh-indexer-0}{cmKpdlIeQdmcSCkD74n-_Q}{jThzC5zoQQGHitPfUPyhDg}{10.244.1.105}{10.244.1.105:9300}{dimr}{shard_indexing_pressure_enabled=true}] from last-known cluster state; node term 74, last-accepted version 8100 in term 74

Wazuh manager logs

2024/05/20 03:55:13 wazuh-modulesd: WARNING: 'update_from_year' option cannot be used for 'nvd' provider.
2024/05/20 03:55:20 wazuh-syscheckd: WARNING: The check_unixaudit option is deprecated in favor of the SCA module.
2024/05/20 03:55:22 wazuh-logcollector: ERROR: (1103): Could not open file '/var/log/syslog' due to [(2)-(No such file or directory)].
2024/05/20 03:55:24 wazuh-modulesd: WARNING: 'update_from_year' option cannot be used for 'nvd' provider.
2024/05/20 06:36:35 wazuh-modulesd: WARNING: 'update_from_year' option cannot be used for 'nvd' provider.
2024/05/20 06:36:43 wazuh-syscheckd: WARNING: The check_unixaudit option is deprecated in favor of the SCA module.
2024/05/20 06:36:45 wazuh-logcollector: ERROR: (1103): Could not open file '/var/log/syslog' due to [(2)-(No such file or directory)].
2024/05/20 06:36:46 wazuh-modulesd: WARNING: 'update_from_year' option cannot be used for 'nvd' provider.

Wazuh dashboard logs

{"date":"2024-05-20T10:35:57.226Z","level":"error","location":"update-registry:updateClusterInfo","message":"ENOENT: no such file or directory, open '/usr/share/wazuh-dashboard/data/wazuh/config/wazuh-registry.json'"}
{"date":"2024-05-20T10:35:57.228Z","level":"error","location":"update-registry:readContent","message":"ENOENT: no such file or directory, open '/usr/share/wazuh-dashboard/data/wazuh/config/wazuh-registry.json'"}
{"date":"2024-05-20T10:35:57.228Z","level":"error","location":"update-registry:updateClusterInfo","message":"ENOENT: no such file or directory, open '/usr/share/wazuh-dashboard/data/wazuh/config/wazuh-registry.json'"}
{"date":"2024-05-20T10:35:57.230Z","level":"error","location":"update-registry:readContent","message":"ENOENT: no such file or directory, open '/usr/share/wazuh-dashboard/data/wazuh/config/wazuh-registry.json'"}
{"date":"2024-05-20T10:35:57.230Z","level":"error","location":"update-registry:updateClusterInfo","message":"ENOENT: no such file or directory, open '/usr/share/wazuh-dashboard/data/wazuh/config/wazuh-registry.json'"}
{"date":"2024-05-20T10:35:57.231Z","level":"error","location":"wazuh-api:checkStoredAPI","message":"ENOENT: no such file or directory, open '/usr/share/wazuh-dashboard/data/wazuh/config/wazuh-registry.json'"}
{"date":"2024-05-20T10:35:57.237Z","level":"error","location":"wazuh-api:checkStoredAPI","message":"ENOENT: no such file or directory, open '/usr/share/wazuh-dashboard/data/wazuh/config/wazuh-registry.json'"}
{"date":"2024-05-20T10:35:57.238Z","level":"error","location":"wazuh-api:checkStoredAPI","message":"ENOENT: no such file or directory, open '/usr/share/wazuh-dashboard/data/wazuh/config/wazuh-registry.json'"}
{"date":"2024-05-20T10:35:57.241Z","level":"error","location":"wazuh-api:checkStoredAPI","message":"ENOENT: no such file or directory, open '/usr/share/wazuh-dashboard/data/wazuh/config/wazuh-registry.json'"}
{"date":"2024-05-20T10:35:57.243Z","level":"error","location":"wazuh-api:checkStoredAPI","message":"ENOENT: no such file or directory, open '/usr/share/wazuh-dashboard/data/wazuh/config/wazuh-registry.json'"}
{"date":"2024-05-20T10:35:57.244Z","level":"error","location":"wazuh-api:checkStoredAPI","message":"Request failed with status code 429"}
{"date":"2024-05-20T10:35:57.246Z","level":"error","location":"wazuh-api:checkStoredAPI","message":"ENOENT: no such file or directory, open '/usr/share/wazuh-dashboard/data/wazuh/config/wazuh-registry.json'"}
{"date":"2024-05-20T10:35:57.247Z","level":"error","location":"wazuh-api:checkStoredAPI","message":"ENOENT: no such file or directory, open '/usr/share/wazuh-dashboard/data/wazuh/config/wazuh-registry.json'"}
{"date":"2024-05-20T10:35:57.248Z","level":"error","location":"wazuh-api:checkStoredAPI","message":"ENOENT: no such file or directory, open '/usr/share/wazuh-dashboard/data/wazuh/config/wazuh-registry.json'"}
{"date":"2024-05-20T10:35:57.249Z","level":"error","location":"wazuh-api:checkStoredAPI","message":"Request failed with status code 429"}
{"date":"2024-05-20T10:35:57.250Z","level":"error","location":"wazuh-api:checkStoredAPI","message":"Request failed with status code 429"}
{"date":"2024-05-20T10:35:57.352Z","level":"error","location":"wazuh-api:checkStoredAPI","message":"Request failed with status code 429"}

Filebeat status:

root@wazuh-manager-master-0:/# filebeat test output
elasticsearch: https://wazuh-indexer-0.wazuh-indexer:9200...
  parse url... OK
  connection...
    parse host... OK
    dns lookup... OK
    addresses: 10.244.1.105
    dial up... OK
  TLS...
    security: server's certificate chain verification is enabled
    handshake... OK
    TLS version: TLSv1.3
    dial up... OK
  talk to server... ERROR 503 Service Unavailable: OpenSearch Security not initialized.
root@wazuh-manager-master-0:/#

The '/usr/share/wazuh-dashboard/data' directory:

wazuh-dashboard@wazuh-dashboard-5f95f76f99-pr5wg:~/data$ pwd
/usr/share/wazuh-dashboard/data
wazuh-dashboard@wazuh-dashboard-5f95f76f99-pr5wg:~/data$ ls -al
total 28
drwxr-x--- 1 wazuh-dashboard wazuh-dashboard 4096 May 20 03:11 .
drwxr-xr-x 1 wazuh-dashboard wazuh-dashboard 4096 May 20 07:11 ..
-rw-r--r-- 1 wazuh-dashboard wazuh-dashboard   36 May 20 03:11 uuid
drwxrwxr-x 1 wazuh-dashboard wazuh-dashboard 4096 Jan 10 14:36 wazuh
wazuh-dashboard@wazuh-dashboard-5f95f76f99-pr5wg:~/data$

pod status：

NAME                                   READY   STATUS    RESTARTS   AGE
pod/wazuh-dashboard-5f95f76f99-pr5wg   1/1     Running   0          7h48m
pod/wazuh-indexer-0                    1/1     Running   0          4h25m
pod/wazuh-indexer-1                    1/1     Running   0          8h
pod/wazuh-indexer-2                    1/1     Running   0          8h
pod/wazuh-manager-master-0             1/1     Running   0          4h25m
pod/wazuh-manager-worker-0             1/1     Running   0          7h6m
pod/wazuh-manager-worker-1             1/1     Running   0          7h5m

NAME                    TYPE           CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE
service/dashboard       LoadBalancer   10.105xx    <pending>     443:31046/TCP                    88d
service/indexer         LoadBalancer   10.98.xx   <pending>     9200:30319/TCP                   88d
service/wazuh           NodePort       10.97.xxx  <none>        1515:30015/TCP,55000:30055/TCP   88d
service/wazuh-cluster   ClusterIP      None            <none>        1516/TCP                         88d
service/wazuh-indexer   ClusterIP      None            <none>        9300/TCP                         88d
service/wazuh-workers   NodePort       10.106.xx    <none>        1514:30014/TCP                   88d

NAME                              READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/wazuh-dashboard   1/1     1            1           88d

NAME                                         DESIRED   CURRENT   READY   AGE
replicaset.apps/wazuh-dashboard-5f95f76f99   1         1         1       88d

NAME                                    READY   AGE
statefulset.apps/wazuh-indexer          3/3     88d
statefulset.apps/wazuh-manager-master   1/1     88d
statefulset.apps/wazuh-manager-worker   2/2     88d

How can I solve this issue?
Thanks.
@jesse-zhangh jesse-zhangh changed the title Something went wrong [Error connecting to API]Something went wrong May 21, 2024
@jesse-zhangh
Copy link
Author

for log:
{"date":"2024-05-21T01:50:00.179Z","level":"error","location":"update-registry:readContent","message":"ENOENT: no such file or directory, open '/usr/share/wazuh-dashboard/data/wazuh/config/wazuh-registry.json'"}

I check the dir and don't find the file wazuh-registry.json

wazuh-dashboard@wazuh-dashboard-5f95f76f99-pr5wg:~/data/wazuh/config$ pwd
/usr/share/wazuh-dashboard/data/wazuh/config
wazuh-dashboard@wazuh-dashboard-5f95f76f99-pr5wg:~/data/wazuh/config$ ls -al
total 24
drwxrwxr-x 1 wazuh-dashboard wazuh-dashboard 4096 Jan 10 14:36 .
drwxrwxr-x 1 wazuh-dashboard wazuh-dashboard 4096 Jan 10 14:36 ..
-rwxrwxr-x 1 wazuh-dashboard wazuh-dashboard 5480 May 20 03:11 wazuh.yml
wazuh-dashboard@wazuh-dashboard-5f95f76f99-pr5wg:~/data/wazuh/config$

@jesse-zhangh
Copy link
Author

I think this perhaps is the root cause:

// 20240521104951
// https://wazuh.xxx.xxx/app/wazuh

{
  "message": "[parent] Data too large, data for [indices:data/read/get[s]] would be [1043665234/995.3mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1043665088/995.3mb], new bytes reserved: [146/146b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=1083622/1mb]: circuit_breaking_exception: [circuit_breaking_exception] Reason: [parent] Data too large, data for [indices:data/read/get[s]] would be [1043665234/995.3mb], which is larger than the limit of [1020054732/972.7mb], real usage: [1043665088/995.3mb], new bytes reserved: [146/146b], usages [request=0/0b, fielddata=0/0b, in_flight_requests=1083622/1mb]",
  "statusCode": 429,
  "error": "Too Many Requests"
}

@jesse-zhangh
Copy link
Author

I solved this issue by Increasing the overall JVM heap size from 1G to 1280m.

Change the statefulset of indexer config:
kubectl edit statefulset.apps/wazuh-indexer -n wazuh

change value from
value: '-Xms1g -Xmx1g -Dlog4j2.formatMsgNoLookups=true'
to
value: '-Xms1280m -Xmx1280m -Dlog4j2.formatMsgNoLookups=true'

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jesse-zhangh