no rule groups found when using S3 and configMap to load rules

[yelhouti]

Describe the bug
I'm trying to load rules using configMap while using s3 and ssd . I can see that the sidecar copying rule files into /rules (I even tried changing config to have them copied to /loki/rules).
Querying the rules always returns: no rule groups found

To Reproduce
Steps to reproduce the behavior:

1. Started Loki 3.0.0
2. Started Promtail 2.9.3

values:

loki:
  auth_enabled: false 
  storage:
    bucketNames:
      chunks: loki-chunks
      ruler: loki-ruler
      admin: loki-admin
    type: s3
    s3:
      endpoint: minio.minio.svc.cluster.local:9000
      accessKeyId: loki
      s3ForcePathStyle: true
      insecure: true
  schemaConfig:
    configs:
      - from: 2024-04-01
        store: tsdb
        object_store: s3
        schema: v13
        index:
          prefix: index_
          period: 24h

curl http://loki-backend:3100/api/prom/rules
# or
curl http://loki-gateway/api/prom/rules

Expected behavior
To see the rules.

Environment:

• Infrastructure: Kubernetes
• Deployment tool: helm

Screenshots, Promtail config, or terminal output
If applicable, add any output to help explain your problem.

[JStickler]

Questions have a better chance of being answered if you ask them on the community forums.

[edwlarkey]

I'm having a similar problem to this using the helm chart. I am not able to provision recording rules in the ruler section.

loki:
  storage:
    bucketNames:
      chunks: REPLACEME
      ruler: REPLACEME
      admin: REPLACEME
    type: s3
    s3:
      region: REPLACEME
      s3ForcePathStyle: true
  rulerConfig:
    wal:
      dir: /var/loki/ruler-wal
    rule_path: /etc/loki/rules
    remote_write:
      enabled: true
      clients:
        mimir:
          url: http://mimir-nginx.monitoring.svc.cluster.local/api/v1/push
ruler:
  replicas: 1
  nodeSelector:
    monitoring-worker: "true"
  persistence:
    enabled: true
  directories:
    fake:
      rules.yaml: |
        groups:
        - name: api
          rules:
            - record: api:loglevel:sum_rate
              expr: sum by(level) (rate({container="api"} [10m]))

The ruler logs:

level=info ts=2024-05-22T16:22:21.599510548Z caller=api.go:509 org_id=fake msg="no rule groups found" userID=fake
level=info ts=2024-05-22T16:22:23.445419254Z caller=api.go:509 org_id=fake msg="no rule groups found" userID=fake
level=info ts=2024-05-22T16:22:33.81113425Z caller=api.go:509 org_id=fake msg="no rule groups found" userID=fake
level=info ts=2024-05-22T16:22:41.711968209Z caller=api.go:509 org_id=fake msg="no rule groups found" userID=fake
level=info ts=2024-05-22T16:22:43.424880297Z caller=api.go:509 org_id=fake msg="no rule groups found" userID=fake
level=info ts=2024-05-22T16:22:53.69105127Z caller=api.go:509 org_id=fake msg="no rule groups found" userID=fake
level=info ts=2024-05-22T16:23:02.140777628Z caller=api.go:509 org_id=fake msg="no rule groups found" userID=fake
level=info ts=2024-05-22T16:23:03.763753726Z caller=api.go:509 org_id=fake msg="no rule groups found" userID=fake
level=info ts=2024-05-22T16:23:13.713900816Z caller=api.go:509 org_id=fake msg="no rule groups found" userID=fake
level=info ts=2024-05-22T16:23:22.022359636Z caller=api.go:509 org_id=fake msg="no rule groups found" userID=fake

[yelhouti]

You can not use both configMaps and s3. I recommand configMaps for gitops...
to use configMaps instead of s3:

  rulerConfig:
    storage:
      type: local
      local:
        directory: /var/loki/rules
    rule_path: /var/loki/rules-temp
    alertmanager_url: http://kube-prometheus-stack-alertmanager.prometheus.svc:9093
    enable_alertmanager_v2: true
    wal:
      dir: /var/loki/ruler-wal

If you are deploying on default mode (SSD).

IMO this should be better documented. Thanks for the person who helped me figure it out.

[edwlarkey]

I suspected that, but I couldn't figure out how to turn off S3 just for the ruler. Using your config to override the storage config for ruler worked. Thank you!