when source-range is added then deleted, service stays unavailable

[christiancadieux]

Is there an existing issue for this?

• I have searched the existing issues

What happened?

when a valid or invalid source-range is added to a service, then removed, the service stays unavailable, as if the source-range was corrupted.
the only way to get the service available again is to include a valid source-range again or re-create it.

not a lot of tickets related to source-range, I did look at #30073 - maybe related ( ipv4 and v6 source range cidrs being mixed-up )

Cilium Version

cilium version
cilium-cli: v0.15.21 compiled with go1.21.6 on linux/amd64
cilium image (default): v1.14.6
cilium image (stable): v1.15.5
cilium image (running): 1.15.4

Kernel Version

Linux caas-bglab-comp010--10-112-182-136 5.15.119-flatcar #1 SMP Fri Jul 14 17:48:03 -00 2023 x86_64 Intel(R) Xeon(R) Gold 6354 CPU @ 3.00GHz GenuineIntel GNU/Linux

Kubernetes Version

Server Version: version.Info{Major:"1", Minor:"25", GitVersion:"v1.25.15", GitCommit:"da6089da4974a0a180c226c9353e1921fa3c248a", GitTreeState:"clean", BuildDate:"2023-10-18T13:29:23Z", GoVersion:"go1.20.10", Compiler:"gc", Platform:"linux/amd64"}

Regression

No response

Sysdump

No response

Relevant log output

the only source-range found in any of the nodes is a different namespace.

for i in `k get pod -n kube-system |grep cilium | awk '{print $1}'`; do   k exec -ti $i -n kube-system -- cilium bpf lb list --source-ranges; done

SOURCE RANGE (REVNAT_ID)   
10.XX.189.0/24 (1346)          

SOURCE RANGE (REVNAT_ID)   
10.XX.189.0/24 (1345)          
...

kubectl get services -A  -o jsonpath='{range .items[*]}{.metadata.namespace}{"\t"}{.spec.loadBalancerSourceRanges}{"\n"}{end}' | grep '\['
ccadie883-2	["10.XX.189.8/24"]
tenant20-2	["6001:XX:104c:10a::9:b83b/64"]

service:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: nginx-test-public
  name: nginx-public
spec:
  allocateLoadBalancerNodePorts: true
  clusterIP: 2001:XX:104c:10a:1::297b
  clusterIPs:
  - 2001:XX:104c:10a:1::297b
  externalTrafficPolicy: Cluster
  internalTrafficPolicy: Cluster
  loadBalancerSourceRanges:
    - 2001:558:104c:10a::7:6be2/16 
  ipFamilies:
  - IPv6
  ipFamilyPolicy: SingleStack
  ports:
  - name: http
    nodePort: 9063
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: nginx-public
  sessionAffinity: None
  type: LoadBalancer
~                    

Anything else?

as a general comment, it would be nice if sourceRange was integrated with NetworkPolicies and cidrGroups, because in cases where sourceIP is lost or not available (envoy, gateway, ingress), sourceRange is the only defense against ingress from external CIDR.

Cilium Users Document

• Are you a user of Cilium? Please add yourself to the Users doc

Code of Conduct

• I agree to follow this project's Code of Conduct

[lmb]

Does valid -> valid source range update work? Can you include the valid and invalid manifest? Your example doesn't contain loadBalancerSourceRanges.

[christiancadieux]

I added a sourceRange property.
yes - source-range works in general, it's only when it's created and then removed that the service stays unavailable, which is what the bug title says.

[lmb]

Thanks. I'm guessing that we treat invalid source range as no source range somewhere, and then "fixing" the manifest ends up looking like no change was made. Leaving this for the experts.