Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Could not run with docker-compose #12986

Closed
taoqf opened this issue May 19, 2024 · 16 comments
Closed

Could not run with docker-compose #12986

taoqf opened this issue May 19, 2024 · 16 comments
Assignees
Labels
docker Pull requests that update Docker code type/bug Somehing is not working as expected

Comments

@taoqf
Copy link

taoqf commented May 19, 2024

Describe the bug
A clear and concise description of what the bug is.

I could not run compose in production/docker

To Reproduce
Steps to reproduce the behavior:

  1. clone this repo
  2. cd production/docker
  3. docker compose up -d
  4. container could not run healthy.

Expected behavior
A clear and concise description of what you expected to happen.

Environment:

  • Infrastructure: docker
  • Deployment tool: docker

Screenshots, Promtail config, or terminal output
If applicable, add any output to help explain your problem.

@JStickler JStickler added type/bug Somehing is not working as expected docker Pull requests that update Docker code labels May 22, 2024
@paul1r
Copy link
Contributor

paul1r commented May 23, 2024

Please describe what error(s) you are seeing, along with what version of docker, and what operating system you are running on.

@rea1shane
Copy link

Same issue, alloy started and then exit, error log:

ts=2024-05-29T06:44:56.887806714Z level=warn msg="could not transfer logs" component_path=/ component_id=loki.source.docker.flog_scrape target=docker/39b515f95b20876a207d1a6b773340322a7be345d5f1d726ded6e6765473b459 written=0 container=39b515f95b20876a207d1a6b773340322a7be345d5f1d726ded6e6765473b459 err="context canceled"
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x39e1a70]

goroutine 425 [running]:
github.com/grafana/alloy/internal/component/loki/source/docker/internal/dockertarget.(*Target).Stop(0x4002478480)
	/src/alloy/internal/component/loki/source/docker/internal/dockertarget/target.go:238 +0x20
github.com/grafana/alloy/internal/component/loki/source/docker.(*tailer).Run(0x40024d5460, {0x8717cd0, 0x40024ae640})
	/src/alloy/internal/component/loki/source/docker/runner.go:116 +0x260
github.com/grafana/alloy/internal/runner.(*Runner[...]).ApplyTasks.func2()
	/src/alloy/internal/runner/runner.go:166 +0x98
created by github.com/grafana/alloy/internal/runner.(*Runner[...]).ApplyTasks in goroutine 188
	/src/alloy/internal/runner/runner.go:163 +0x490

env:

Chip: M1
ProductName:		macOS
ProductVersion:		14.5
BuildVersion:		23F79
Docker version 25.0.5, build 5dc9bcc

@paul1r
Copy link
Contributor

paul1r commented May 29, 2024

@rea1shane I have opened #13065 to track that issue, as it appears you are running a different docker compose than the one this report is on. If I am mistaken, please let me know.

@taoqf
Copy link
Author

taoqf commented Jun 1, 2024

I've tried Debian and Fedora. There behaviour is same.
without change anything, i tried the compose file in production , production/docker, even in examples/getting-started. only production/docker-compose could be started.

Please describe what error(s) you are seeing, along with what version of docker, and what operating system you are running on.

@taoqf
Copy link
Author

taoqf commented Jun 1, 2024

I checked the logs in loki-read, first error is level=error ts=2024-06-01T07:02:19.1157467Z caller=loki.go:517 msg="module failed" module=query-scheduler-ring error="failed to start query-scheduler-ring, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Stopping, expected: Running".
the logs in loki-write, and loki-backend shows

level=error ts=2024-06-01T07:07:25.792414429Z caller=loki.go:517 msg="module failed" module=distributor error="failed to start distributor, because it depends on module analytics, which has failed: context canceled"
level=error ts=2024-06-01T07:07:25.792426167Z caller=loki.go:517 msg="module failed" module=ring error="failed to start ring, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0005eb900 { true 10000000000 4 30000000000 2000000000 3 15000000000 30000000000 true  7946  false [loki-read loki-write loki-backend] 1000000000 60000000000 10 false 0 30000000000 20000000000 0 {[0.0.0.0] 7946 2000000000 5000000000 false loki false {    false   <nil>}} loki [{ringDesc 0xea4480} {}]} 0xc0006920a0 0x43908c0 0xc000b9bab0 {[] {[] 0}} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xea4480} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc0006d0900 0xc0004562a0 0xc000456360 0xc000456420 0xc0004564e0 0xc000456720 0xc0004565a0 0xc0004567e0 0xc000456660 0xc000706d40 0xc000706d80 0xc000456960 0xc000456a20 0xc000456ba0 0xc000456ae0 0xc0009862c0 0xc000f14850 0xc0009862b0 0xc0009862b8 0xc000706dc0 0xc000706e00 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
level=error ts=2024-06-01T07:07:25.792435237Z caller=loki.go:517 msg="module failed" module=store error="failed to start store, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0005eb900 { true 10000000000 4 30000000000 2000000000 3 15000000000 30000000000 true  7946  false [loki-read loki-write loki-backend] 1000000000 60000000000 10 false 0 30000000000 20000000000 0 {[0.0.0.0] 7946 2000000000 5000000000 false loki false {    false   <nil>}} loki [{ringDesc 0xea4480} {}]} 0xc0006920a0 0x43908c0 0xc000b9bab0 {[] {[] 0}} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xea4480} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc0006d0900 0xc0004562a0 0xc000456360 0xc000456420 0xc0004564e0 0xc000456720 0xc0004565a0 0xc0004567e0 0xc000456660 0xc000706d40 0xc000706d80 0xc000456960 0xc000456a20 0xc000456ba0 0xc000456ae0 0xc0009862c0 0xc000f14850 0xc0009862b0 0xc0009862b8 0xc000706dc0 0xc000706e00 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
level=error ts=2024-06-01T07:07:25.792439299Z caller=loki.go:517 msg="module failed" module=analytics error="failed to start analytics, because it depends on module memberlist-kv, which has failed: invalid service state: Failed, expected: Running, failure: invalid service state: Failed, expected: Running, failure: service &{0xc0005eb900 { true 10000000000 4 30000000000 2000000000 3 15000000000 30000000000 true  7946  false [loki-read loki-write loki-backend] 1000000000 60000000000 10 false 0 30000000000 20000000000 0 {[0.0.0.0] 7946 2000000000 5000000000 false loki false {    false   <nil>}} loki [{ringDesc 0xea4480} {}]} 0xc0006920a0 0x43908c0 0xc000b9bab0 {[] {[] 0}} <nil> <nil> {0 0} map[] map[ringDesc:{ringDesc 0xea4480} usagestats.jsonCodec:{}] {0 0} map[] map[] {0 0} [] 0 [] 0 0 {0 0} map[] 0xc0006d0900 0xc0004562a0 0xc000456360 0xc000456420 0xc0004564e0 0xc000456720 0xc0004565a0 0xc0004567e0 0xc000456660 0xc000706d40 0xc000706d80 0xc000456960 0xc000456a20 0xc000456ba0 0xc000456ae0 0xc0009862c0 0xc000f14850 0xc0009862b0 0xc0009862b8 0xc000706dc0 0xc000706e00 10} failed: failed to create memberlist: Failed to get final advertise address: no private IP address found, and explicit IP not provided"
level=error ts=2024-06-01T07:07:25.792446405Z caller=loki.go:517 msg="module failed" module=ingester error="failed to start ingester, because it depends on module store, which has failed: context canceled"
level=info ts=2024-06-01T07:07:25.792566169Z caller=modules.go:1484 msg="server stopped"
level=info ts=2024-06-01T07:07:25.792573837Z caller=module_service.go:114 msg="module stopped" module=server
level=info ts=2024-06-01T07:07:25.792582401Z caller=loki.go:506 msg="Loki stopped"
level=error ts=2024-06-01T07:07:25.792615783Z caller=log.go:230 msg="error running loki" err="failed services\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:559\nmain.main\n\t/src/loki/cmd/loki/main.go:110\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"

alert-manager logs

evel=error ts=2024-06-01T07:00:20.322Z caller=main.go:250 msg="unable to initialize gossip mesh" err="create memberlist: Failed to get final advertise address: No private IP address found, and explicit IP not provided"

@paul1r
Copy link
Contributor

paul1r commented Jun 3, 2024

I'm sorry to hear you are running into an issue with this. I just tried a fresh install of the docker-compose setup in the production/docker directory on Fedora 34, and did not have any problems.

I suspect there's an error earlier on in the logs that may clue us in as to the issue. Can you capture the full logs for all of the docker containers and attach them here?

@taoqf
Copy link
Author

taoqf commented Jun 4, 2024

production.log
I stopped it manually because the container keep restarting.

@paul1r
Copy link
Contributor

paul1r commented Jun 4, 2024

Thank you. What does your networking look like? What IPs are you using, what are your interface names?

I wonder if some of the information in this issue would help?

@taoqf
Copy link
Author

taoqf commented Jun 5, 2024

ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host noprefixroute
valid_lft forever preferred_lft forever
2: enp2s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 74:56:3c:a7:d2:ad brd ff:ff:ff:ff:ff:ff
inet 172.16.164.43/24 brd 172.16.164.255 scope global noprefixroute enp2s0
valid_lft forever preferred_lft forever
inet6 fe80::7656:3cff:fea7:d2ad/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: br-f7837756fd18: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:e8:6e:5c:44 brd ff:ff:ff:ff:ff:ff
inet 172.0.7.1/24 brd 172.0.7.255 scope global br-f7837756fd18
valid_lft forever preferred_lft forever
4: br-1728647c68c0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:d1:a3:18:ad brd ff:ff:ff:ff:ff:ff
inet 172.0.14.1/24 brd 172.0.14.255 scope global br-1728647c68c0
valid_lft forever preferred_lft forever
5: br-2a59656a95fe: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:b1:e6:74:ee brd ff:ff:ff:ff:ff:ff
inet 172.0.12.1/24 brd 172.0.12.255 scope global br-2a59656a95fe
valid_lft forever preferred_lft forever
inet6 fe80::42:b1ff:fee6:74ee/64 scope link
valid_lft forever preferred_lft forever
6: br-82336f0d9578: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:d8:ce:e3:b4 brd ff:ff:ff:ff:ff:ff
inet 172.0.2.1/24 brd 172.0.2.255 scope global br-82336f0d9578
valid_lft forever preferred_lft forever
8: br-ecd9d8f89caa: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:8b:19:9a:41 brd ff:ff:ff:ff:ff:ff
inet 172.0.3.1/24 brd 172.0.3.255 scope global br-ecd9d8f89caa
valid_lft forever preferred_lft forever
9: br-17165532b731: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:25:fa:94:b4 brd ff:ff:ff:ff:ff:ff
inet 172.0.73.1/24 brd 172.0.73.255 scope global br-17165532b731
valid_lft forever preferred_lft forever
10: br-5a851ebd9345: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:39:42:3f:45 brd ff:ff:ff:ff:ff:ff
inet 172.0.13.1/24 brd 172.0.13.255 scope global br-5a851ebd9345
valid_lft forever preferred_lft forever
11: br-b71f53b36f98: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:39:76:30:f0 brd ff:ff:ff:ff:ff:ff
inet 172.0.42.1/24 brd 172.0.42.255 scope global br-b71f53b36f98
valid_lft forever preferred_lft forever
12: br-7ad0b0e0af48: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:d9:61:fd:bd brd ff:ff:ff:ff:ff:ff
inet 172.0.10.1/24 brd 172.0.10.255 scope global br-7ad0b0e0af48
valid_lft forever preferred_lft forever
14: br-a30fe3aa1ff9: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:90:2a:a7:c7 brd ff:ff:ff:ff:ff:ff
inet 172.0.6.1/24 brd 172.0.6.255 scope global br-a30fe3aa1ff9
valid_lft forever preferred_lft forever
15: br-ef91845e7628: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:c9:c8:2d:ed brd ff:ff:ff:ff:ff:ff
inet 172.0.11.1/24 brd 172.0.11.255 scope global br-ef91845e7628
valid_lft forever preferred_lft forever
16: br-cbcd2a857f95: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:64:ba:30:d4 brd ff:ff:ff:ff:ff:ff
inet 172.0.4.1/24 brd 172.0.4.255 scope global br-cbcd2a857f95
valid_lft forever preferred_lft forever
17: br-df0182b5f1c0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:16:b0:dd:ac brd ff:ff:ff:ff:ff:ff
inet 172.0.72.1/24 brd 172.0.72.255 scope global br-df0182b5f1c0
valid_lft forever preferred_lft forever
18: br-e58ca235adba: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:33:48:d1:6e brd ff:ff:ff:ff:ff:ff
inet 172.0.33.1/24 brd 172.0.33.255 scope global br-e58ca235adba
valid_lft forever preferred_lft forever
inet6 fe80::42:33ff:fe48:d16e/64 scope link
valid_lft forever preferred_lft forever
19: br-014cef3412f4: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:87:b0:55:12 brd ff:ff:ff:ff:ff:ff
inet 172.0.9.1/24 brd 172.0.9.255 scope global br-014cef3412f4
valid_lft forever preferred_lft forever
20: br-2bd59f3dd8de: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:01:24:f5:7e brd ff:ff:ff:ff:ff:ff
inet 172.0.8.1/24 brd 172.0.8.255 scope global br-2bd59f3dd8de
valid_lft forever preferred_lft forever
21: br-589f84ac3592: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:3f:2f:4c:e9 brd ff:ff:ff:ff:ff:ff
inet 172.0.34.1/24 brd 172.0.34.255 scope global br-589f84ac3592
valid_lft forever preferred_lft forever
inet6 fe80::42:3fff:fe2f:4ce9/64 scope link
valid_lft forever preferred_lft forever
22: br-9735ec5642ea: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:38:20:4b:4c brd ff:ff:ff:ff:ff:ff
inet 172.0.1.1/24 brd 172.0.1.255 scope global br-9735ec5642ea
valid_lft forever preferred_lft forever
23: br-a5fdcd2ed483: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:8b:30:50:94 brd ff:ff:ff:ff:ff:ff
inet 172.0.5.1/24 brd 172.0.5.255 scope global br-a5fdcd2ed483
valid_lft forever preferred_lft forever
25: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:60:de:61:03 brd ff:ff:ff:ff:ff:ff
inet 172.0.0.1/24 brd 172.0.0.255 scope global docker0
valid_lft forever preferred_lft forever
inet6 fe80::42:60ff:fede:6103/64 scope link
valid_lft forever preferred_lft forever
2181: veth43a11ca@if2180: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-589f84ac3592 state UP group default
link/ether 3e:cd:ea:c9:80:b3 brd ff:ff:ff:ff:ff:ff link-netnsid 4
inet6 fe80::3ccd:eaff:fec9:80b3/64 scope link
valid_lft forever preferred_lft forever
1427: vethe1b611f@if1426: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-589f84ac3592 state UP group default
link/ether 86:28:3e:fe:1c:7b brd ff:ff:ff:ff:ff:ff link-netnsid 2
inet6 fe80::8428:3eff:fefe:1c7b/64 scope link
valid_lft forever preferred_lft forever
1429: veth2099e62@if1428: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-589f84ac3592 state UP group default
link/ether ee:6f:8f:95:38:af brd ff:ff:ff:ff:ff:ff link-netnsid 0
inet6 fe80::ec6f:8fff:fe95:38af/64 scope link
valid_lft forever preferred_lft forever
1431: veth0028feb@if1430: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-589f84ac3592 state UP group default
link/ether 8e:36:a4:1c:91:29 brd ff:ff:ff:ff:ff:ff link-netnsid 1
inet6 fe80::8c36:a4ff:fe1c:9129/64 scope link
valid_lft forever preferred_lft forever
1433: vethc3e9a58@if1432: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-589f84ac3592 state UP group default
link/ether 1a:47:3f:eb:02:f6 brd ff:ff:ff:ff:ff:ff link-netnsid 3
inet6 fe80::1847:3fff:feeb:2f6/64 scope link
valid_lft forever preferred_lft forever
1437: vethd85fe36@if1436: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-589f84ac3592 state UP group default
link/ether 22:ff:6b:7a:c0:87 brd ff:ff:ff:ff:ff:ff link-netnsid 5
inet6 fe80::20ff:6bff:fe7a:c087/64 scope link
valid_lft forever preferred_lft forever
2004: br-262fbcbe84f2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:1a:86:4b:88 brd ff:ff:ff:ff:ff:ff
inet 172.0.21.1/24 brd 172.0.21.255 scope global br-262fbcbe84f2
valid_lft forever preferred_lft forever
inet6 fe80::42:1aff:fe86:4b88/64 scope link
valid_lft forever preferred_lft forever
2005: br-0798f19c15a2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
link/ether 02:42:44:09:b8:b9 brd ff:ff:ff:ff:ff:ff
inet 172.0.22.1/24 brd 172.0.22.255 scope global br-0798f19c15a2
valid_lft forever preferred_lft forever
inet6 fe80::42:44ff:fe09:b8b9/64 scope link
valid_lft forever preferred_lft forever

@paul1r
Copy link
Contributor

paul1r commented Jun 5, 2024

By utilizing the production/docker/docker-compose.yaml file, and modifying the production/docker/config/loki.yaml file to point to an invalid interface on my system, I was able to recreate what you are seeing.

I modified the loki.yaml config, and put in interface names for all of the components that mapped to what is available in docker on my system, and Loki was able to start. The contents of this are below (only changes are related to the eth0 entries)

auth_enabled: true

server:
  http_listen_address: 0.0.0.0
  grpc_listen_address: 0.0.0.0
  http_listen_port: 3100
  grpc_listen_port: 9095
  log_level: info

common:
  path_prefix: /loki
  compactor_address: http://loki-backend:3100
  replication_factor: 3
  instance_interface_names:
    - eth0
  ring:
    instance_interface_names:
      - eth0

storage_config:
  aws:
    endpoint: minio:9000
    insecure: true
    bucketnames: loki-data
    access_key_id: loki
    secret_access_key: supersecret
    s3forcepathstyle: true

memberlist:
  join_members: ["loki-read", "loki-write", "loki-backend"]
  dead_node_reclaim_time: 30s
  gossip_to_dead_nodes_time: 15s
  left_ingesters_timeout: 30s
  bind_addr: ['0.0.0.0']
  bind_port: 7946
  gossip_interval: 2s

ingester:
  lifecycler:
    join_after: 10s
    observe_period: 5s
    interface_names:
      - eth0
    ring:
      replication_factor: 3
      kvstore:
        store: memberlist
    final_sleep: 0s
  chunk_idle_period: 1m
  wal:
    enabled: true
    dir: /loki/wal
  max_chunk_age: 1m
  chunk_retain_period: 30s
  chunk_encoding: snappy
  chunk_target_size: 1.572864e+06
  chunk_block_size: 262144
  flush_op_timeout: 10s

ruler:
  enable_api: true
  enable_sharding: true  
  wal:
    dir: /loki/ruler-wal
  evaluation:
    mode: remote
    query_frontend:
      address: dns:///loki-read:9095
  storage:
    type: local
    local:
      directory: /loki/rules
  ring:
    instance_interface_names:
      - eth0
  rule_path: /loki/prom-rules
  remote_write:
    enabled: true
    clients:
      local:
         url: http://prometheus:9090/api/v1/write
         queue_config:
           # send immediately as soon as a sample is generated
           capacity: 1
           batch_send_deadline: 0s

schema_config:
  configs:
  - from: 2020-08-01
    store: boltdb-shipper
    object_store: s3
    schema: v11
    index:
      prefix: index_
      period: 24h
  - from: 2023-07-11
    store: tsdb
    object_store: s3
    schema: v12
    index:
      prefix: index_
      period: 24h
  - from: 2024-01-10
    store: tsdb
    object_store: s3
    schema: v12
    index:
      prefix: index_
      period: 24h
  - from: 2024-03-29
    store: tsdb
    object_store: s3
    schema: v13
    index:
      prefix: index_
      period: 24h


limits_config:
  max_cache_freshness_per_query: '10m'
  reject_old_samples: true
  reject_old_samples_max_age: 30m
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20
  # parallelize queries in 15min intervals
  split_queries_by_interval: 15m
  volume_enabled: true

table_manager:
  retention_deletes_enabled: true
  retention_period: 336h

query_range:
  # make queries more cache-able by aligning them with their step intervals
  align_queries_with_step: true
  max_retries: 5
  parallelise_shardable_queries: true
  cache_results: true

frontend:
  log_queries_longer_than: 5s
  compress_responses: true
  max_outstanding_per_tenant: 2048
  instance_interface_names:
    - eth0

query_scheduler:
  max_outstanding_requests_per_tenant: 1024
  scheduler_ring:
    instance_interface_names:
      - eth0

querier:
  query_ingesters_within: 2h

compactor:
  working_directory: /tmp/compactor
  compactor_ring:
    instance_interface_names:
      - eth0

@taoqf
Copy link
Author

taoqf commented Jun 6, 2024

There must something else blocking me. after replace config file with yours,and replace eth0 with enp2s0, I've got this:

loki-read-1      | level=error ts=2024-06-06T07:28:31.989245249Z caller=log.go:230 msg="error running loki" err="no address found for [enp2s0]\nfailed to get frontend address\ngithub.com/grafana/loki/pkg/lokifrontend/frontend.InitFrontend\n\t/src/loki/pkg/lokifrontend/frontend/config.go:52\ngithub.com/grafana/loki/pkg/loki.(*Loki).initQueryFrontend\n\t/src/loki/pkg/loki/modules.go:797\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:136\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:461\nmain.main\n\t/src/loki/cmd/loki/main.go:110\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650\nerror initialising module: query-frontend\ngithub.com/grafana/dskit/modules.(*Manager).initModule\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:138\ngithub.com/grafana/dskit/modules.(*Manager).InitModuleServices\n\t/src/loki/vendor/github.com/grafana/dskit/modules/modules.go:108\ngithub.com/grafana/loki/pkg/loki.(*Loki).Run\n\t/src/loki/pkg/loki/loki.go:461\nmain.main\n\t/src/loki/cmd/loki/main.go:110\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:267\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1650"

this is my loki.yaml

auth_enabled: true

server:
  http_listen_address: 0.0.0.0
  grpc_listen_address: 0.0.0.0
  http_listen_port: 3100
  grpc_listen_port: 9095
  log_level: info

common:
  path_prefix: /loki
  compactor_address: http://loki-backend:3100
  replication_factor: 3
  instance_interface_names:
    - enp2s0
  ring:
    instance_interface_names:
      - enp2s0

storage_config:
  aws:
    endpoint: minio:9000
    insecure: true
    bucketnames: loki-data
    access_key_id: loki
    secret_access_key: supersecret
    s3forcepathstyle: true

memberlist:
  join_members: ["loki-read", "loki-write", "loki-backend"]
  dead_node_reclaim_time: 30s
  gossip_to_dead_nodes_time: 15s
  left_ingesters_timeout: 30s
  bind_addr: ['0.0.0.0']
  bind_port: 7946
  gossip_interval: 2s

ingester:
  lifecycler:
    join_after: 10s
    observe_period: 5s
    interface_names:
      - enp2s0
    ring:
      replication_factor: 3
      kvstore:
        store: memberlist
    final_sleep: 0s
  chunk_idle_period: 1m
  wal:
    enabled: true
    dir: /loki/wal
  max_chunk_age: 1m
  chunk_retain_period: 30s
  chunk_encoding: snappy
  chunk_target_size: 1.572864e+06
  chunk_block_size: 262144
  flush_op_timeout: 10s

ruler:
  enable_api: true
  enable_sharding: true  
  wal:
    dir: /loki/ruler-wal
  evaluation:
    mode: remote
    query_frontend:
      address: dns:///loki-read:9095
  storage:
    type: local
    local:
      directory: /loki/rules
  ring:
    instance_interface_names:
      - enp2s0
  rule_path: /loki/prom-rules
  remote_write:
    enabled: true
    clients:
      local:
         url: http://prometheus:9090/api/v1/write
         queue_config:
           # send immediately as soon as a sample is generated
           capacity: 1
           batch_send_deadline: 0s

schema_config:
  configs:
  - from: 2020-08-01
    store: boltdb-shipper
    object_store: s3
    schema: v11
    index:
      prefix: index_
      period: 24h
  - from: 2023-07-11
    store: tsdb
    object_store: s3
    schema: v12
    index:
      prefix: index_
      period: 24h
  - from: 2024-01-10
    store: tsdb
    object_store: s3
    schema: v12
    index:
      prefix: index_
      period: 24h
  - from: 2024-03-29
    store: tsdb
    object_store: s3
    schema: v13
    index:
      prefix: index_
      period: 24h


limits_config:
  max_cache_freshness_per_query: '10m'
  reject_old_samples: true
  reject_old_samples_max_age: 30m
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20
  # parallelize queries in 15min intervals
  split_queries_by_interval: 15m
  volume_enabled: true

table_manager:
  retention_deletes_enabled: true
  retention_period: 336h

query_range:
  # make queries more cache-able by aligning them with their step intervals
  align_queries_with_step: true
  max_retries: 5
  parallelise_shardable_queries: true
  cache_results: true

frontend:
  log_queries_longer_than: 5s
  compress_responses: true
  max_outstanding_per_tenant: 2048
  instance_interface_names:
    - enp2s0

query_scheduler:
  max_outstanding_requests_per_tenant: 1024
  scheduler_ring:
    instance_interface_names:
      - enp2s0

querier:
  query_ingesters_within: 2h

compactor:
  working_directory: /tmp/compactor
  compactor_ring:
    instance_interface_names:
      - enp2s0

@taoqf
Copy link
Author

taoqf commented Jun 6, 2024

loki.log
This is whole logs

@paul1r
Copy link
Contributor

paul1r commented Jun 6, 2024

If you were to not make any Loki configuration changes, for the pods that stay up, can you shell into them to see what interfaces are available? This log appears to say that enp2s0 isn't valid for your Docker networking.
For instance, my local docker setup offers eth0 and lo, and neither of those are options within my normal non-Docker environment.

Ultimately, this feels like something specific with your Docker setup, and I'm hopeful that once the correct interface is found, you can move forward.

@taoqf
Copy link
Author

taoqf commented Jun 7, 2024

Thank you. After I checked my config, default-address-pools is the reason.
/etc/docker/daemon.json

{
	"data-root":"/home/taoqf/docker/",
 	"default-address-pools": [{
		"base": "172.0.0.1/16",
		"size": 24
	}]
}

If I remove default-address-pools, with default loki.yml, I finally run the compose up.

Then, how can I fix this without change my configure?

@paul1r
Copy link
Contributor

paul1r commented Jun 7, 2024

OK, my suspicion is that the configuration you have for default address pools (thank you for finding this!) is
allocating subnets from 172.0.0.1 to 172.0.255.255, and each of those subnets is 256 IP address.

The ring code (by default) is doing some magic to find private network interfaces. The official set of 172.x.x.x private addresses is 172.16.0.0 to 172.31.255.255, which does not overlap with what you have specified.

As such, my belief is that, in order to not change your docker configuration, you'll need to modify your loki.yaml config file, like I did above, to explicitly call out eth0 for each of the components. (You should also verify that that particular interface name exists in your docker containers for Loki, and if not, change it to whatever you have configured).

@taoqf
Copy link
Author

taoqf commented Jun 11, 2024

OK, Thank you for your patient. I will change my docker configure.
btw, modify loki.yml by adding interface name won't help.
I can't figure out why loki in a container have the ability to operate local host's network.
anyway, I can run loki now.

@taoqf taoqf closed this as completed Jun 11, 2024
@paul1r paul1r self-assigned this Jun 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docker Pull requests that update Docker code type/bug Somehing is not working as expected
Projects
None yet
Development

No branches or pull requests

4 participants