multus: add host checking to validation tool

[BlaineEXE]

In order to help users check that they have implemented the newly-added Multus host configuration prerequisites, add a check to the validation tool to verify connectivity.

Because users who are already running clusters with Multus enabled, add a flag that allows users to only check for host configuration prerequisites. This mode will not start the large number of clients that would normally be started because those clients could disrupt a running Rook cluster negatively.

Host checking pods require host network access. Many Kubernetes
distributions have pod security features enabled. In order to allow
non-Vanilla distros to run this tool, allow specifying a service account
that pods will run as, which can be configured by the admin to allow
test pods.

Manual validation tests:

• flakiness check still works
• expect to fail when NAD doesn't have route
• cleanup cleans up host checkers
• expect to fail when host doesn't have route
• expect to succeed when NAD and host routing is correct
• --host-check-only results in tool stopping after host check succeeds
• --host-check-only exits with early success when public-net is unset
• tool enters host check only mode when --host-check-only flag AND/OR config file config is set
• test on openshift - works, but needs SA+role+binding to allow with SCCs
  • add example resources needed for openshift?
• update documentation to mention using the validation tool
• fix KinD-based CI tests

Checklist:

• Commit Message Formatting: Commit titles and messages follow guidelines in the developer guide.
• Reviewed the developer guide on Submitting a Pull Request
• Pending release notes updated with breaking and/or notable changes for the next minor release.
• Documentation has been updated, if necessary.
• Unit tests have been added, if necessary.
• Integration tests have been added, if necessary.

[mergify]

This pull request has merge conflicts that must be resolved before it can be merged. @BlaineEXE please rebase it. https://rook.io/docs/rook/latest/Contributing/development-flow/#updating-your-fork

[BlaineEXE]

I was able to get this working on openshift, but I wasn't able to define my own custom SCC. The pod was perpetually saying that it wasn't allowed by any SCC, and the custom SCC was never in the list. @subhamkrai or @Madhu-1 do you remember if you saw this issue when testing other things and how you might've resolved that?

kind: SecurityContextConstraints
apiVersion: security.openshift.io/v1
metadata:
  name: multus-validation-test
# host checker daemonset runs on host network
allowHostNetwork: true
allowedCapabilities: ["NET_BIND_SERVICE"]
runAsUser:
  type: MustRunAsNonRoot
seLinuxContext:
  type: RunAsAny
# disallow everything noncritical
allowPrivilegedContainer: false
allowHostDirVolumePlugin: false
allowHostPID: false
allowHostIPC: false
allowHostPorts: false
readOnlyRootFilesystem: true
users:
  - system:serviceaccount:openshift-storage:multus-validation-test
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: multus-validation-test
  namespace: openshift-storage

In the end, I was able to get this working by instead specifying a Role and RoleBinding that allowed the SA to use an existing SCC (hostnetwork-v2). I think this solution is a safe go-forward strategy, but I'm still curious why my first attempt didn't work.

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: multus-validation-test
  namespace: openshift-storage
rules:
  - apiGroups:
      - security.openshift.io
    resourceNames:
      - hostnetwork-v2
    resources:
      - securitycontextconstraints
    verbs:
      - use
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: multus-validation-test
  namespace: openshift-storage
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: multus-validation-test
subjects:
  - kind: ServiceAccount
    name: multus-validation-test
    namespace: openshift-storage