Unable to enable loopback feature on IBM Cloud Openshift

[ocofaigh]

Environment: IBM Cloud Openshift cluster 4.13

Steps:

1. git clone git@github.com:logdna/logdna-agent-v2.git
2. cd logdna-agent-v2
3. git checkout 3.9.1
4. oc new-project logdna-agent
5. oc create serviceaccount logdna-agent
6. oc create secret generic logdna-agent-key --from-literal=logdna-agent-key=XXXXXX
7. oc adm policy add-scc-to-user privileged system:serviceaccount:logdna-agent:logdna-agent
8. Modified the yaml k8s/agent-resources-openshift.yaml with the following changes:
   • Run as Non-Root (using these steps)
   • added volume-mount-permissions-fix initContainer (using these steps)

Problem:
The initContainer fails with:

% oc logs logdna-agent-6gkdq -c volume-mount-permissions-fix
chmod: /var/lib/logdna: Permission denied
chmod: /var/lib/logdna: Permission denied

Why would the initContainer running as root with the privileged SCC not be able to set permissions on /var/lib/logdna?

I even exec into the initContiner, and can see this:

/ # whoami
root
/ # id
uid=0(root) gid=0(root) groups=0(root),10(wheel)
/ # ls -la /var/lib/logdna/
total 12
drwxr-xr-x    2 root     root          4096 Jan 13 20:25 .
drwxr-xr-x    3 root     root          4096 Jan 13 20:38 ..

Here is the final yaml I used:

---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: logdna-agent-ds-priority
  namespace: logdna-agent
value: 1000000
preemptionPolicy: PreemptLowerPriority
globalDefault: false
description: "Logdna Agent"
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: logdna-agent
  name: logdna-agent
  labels:
    app.kubernetes.io/name: logdna-agent
    app.kubernetes.io/instance: logdna-agent
    app.kubernetes.io/version: 3.9.1
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get","list", "create", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: logdna-agent
  namespace: logdna-agent
  labels:
    app.kubernetes.io/name: logdna-agent
    app.kubernetes.io/instance: logdna-agent
    app.kubernetes.io/version: 3.9.1
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: logdna-agent
subjects:
  - kind: ServiceAccount
    name: logdna-agent
    namespace: logdna-agent
---
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: logdna-agent
  labels:
    app.kubernetes.io/name: logdna-agent
    app.kubernetes.io/instance: logdna-agent
    app.kubernetes.io/version: 3.9.1
rules:
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["get","list", "create", "watch"]
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["get","list", "watch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: logdna-agent
  labels:
    app.kubernetes.io/name: logdna-agent
    app.kubernetes.io/instance: logdna-agent
    app.kubernetes.io/version: 3.9.1
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: logdna-agent
subjects:
  - kind: ServiceAccount
    name: logdna-agent
    namespace: logdna-agent
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: logdna-agent
  namespace: logdna-agent
  labels:
    app.kubernetes.io/name: logdna-agent
    app.kubernetes.io/instance: logdna-agent
    app.kubernetes.io/version: 3.9.1
spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      maxUnavailable: 100%
  selector:
    matchLabels:
      app: logdna-agent
  template:
    metadata:
      labels:
        app: logdna-agent
        app.kubernetes.io/name: logdna-agent
        app.kubernetes.io/instance: logdna-agent
        app.kubernetes.io/version: 3.9.1
    spec:
      serviceAccountName: logdna-agent
      priorityClassName: logdna-agent-ds-priority
      initContainers:
        - name: volume-mount-permissions-fix
          image: busybox
          command: ["sh", "-c", "chmod -R 775 /var/lib/logdna && chown -R 5000:5000 /var/lib/logdna"]
          volumeMounts:
          - name: varliblogdna
            mountPath: /var/lib/logdna
      containers:
        - name: logdna-agent
          image: logdna/logdna-agent:3.9.1
          imagePullPolicy: Always
          securityContext:
            runAsUser: 5000
            runAsGroup: 5000
            privileged: true
            capabilities:
              add:
                - DAC_READ_SEARCH
              drop:
                - all
          env:
            - name: LOGDNA_INGESTION_KEY
              valueFrom:
                secretKeyRef:
                  name: logdna-agent-key
                  key: logdna-agent-key
            - name: LOGDNA_LOOKBACK
              value: smallfiles
            - name: LOGDNA_DB_PATH
              value: /var/lib/logdna
            - name: POD_APP_LABEL
              valueFrom:
                fieldRef:
                  fieldPath: metadata.labels['app.kubernetes.io/name']
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: NODE_NAME
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          resources:
            requests:
              cpu: 20m
            limits:
              memory: 500Mi
          volumeMounts:
            - name: varlog
              mountPath: /var/log
            - name: vardata
              mountPath: /var/data
            - name: varliblogdna
              mountPath: /var/lib/logdna
            - name: varlibdockercontainers
              mountPath: /var/lib/docker/containers
              readOnly: true
            - name: mnt
              mountPath: /mnt
              readOnly: true
            - name: osrelease
              mountPath: /etc/os-release
            - name: logdnahostname
              mountPath: /etc/logdna-hostname
      volumes:
        - name: varlog
          hostPath:
            path: /var/log
        - name: vardata
          hostPath:
            path: /var/data
        - name: varliblogdna
          hostPath:
            path: /var/lib/logdna
        - name: varlibdockercontainers
          hostPath:
            path: /var/lib/docker/containers
        - name: mnt
          hostPath:
            path: /mnt
        - name: osrelease
          hostPath:
            path: /etc/os-release
        - name: logdnahostname
          hostPath:
            path: /etc/hostname

[ocofaigh]

Oh, I found the issue. I had to explicitly add this to the initContainer:

securityContext:
  privileged: true
  runAsUser: 0

This is missing from the doc here -> https://github.com/logdna/logdna-agent-v2/blob/master/docs/KUBERNETES.md#enabling-file-offset-tracking-across-restarts