-
Notifications
You must be signed in to change notification settings - Fork 2.7k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
object: virtulhostnames is not required in the endpoint for rgw #14034
Conversation
The virtualhostnames for rgw is added to possible endpoint list, this endpoint is mainly used by rook operator for communicating with RGW server. If enable the virtualhostname feature for rgw, by default the service endpoint is added to the list. So the service endpoint can accessed even if the feature is enabled. Signed-off-by: Jiffin Tony Thottan <thottanjiffin@gmail.com>
@@ -348,9 +348,6 @@ func getDomainName(s *cephv1.CephObjectStore, returnRandomDomainIfMultiple bool) | |||
for _, e := range s.Spec.Gateway.ExternalRgwEndpoints { | |||
endpoints = append(endpoints, e.String()) | |||
} | |||
} else if s.Spec.Hosting != nil && len(s.Spec.Hosting.DNSNames) > 0 { | |||
// if the store is internal and has DNS names, pick a random DNS name to use | |||
endpoints = s.Spec.Hosting.DNSNames | |||
} else { | |||
return domainNameOfService(s) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Are you saying the dns host name is already added to this service name that would be returned here?
What was the bug? Were we returning an invalid domain name with line 353?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm also somewhat confused. Please update the commit description and PR description text to be clear about what problem exists and how this resolves it.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@travis and @BlaineEXE it is not because invalid DNS name.
The endpoint
of cephobjectstore
is populated with help of BuildDNSEndpoint
which combines domain name from getDomainName
and port in the objectstore.spec.gateway
. This approach is used in most of the Rook code base and even for adminsops api for Rook to communicate with RGW.
In case of host feature is enabled it returns the domain from rgw DNS name list, but the domain picked may not have the same port as the rgw internal port which is failing for adminsops use case. So the populated endpoint in this case is wrong.
Even if the feature is enabled by default rgw service endpoint will be part of the rgw DNS names so we can pick RGW service endpoint for all the internal communications. As I mentioned earlier by default rgw service endpoint is part of there is no specific need to pick the domain name from the rgw DNS list
I have added this change in the first version based on the assumption that by default rgw service endpoint
would not be added to rgw DNS names, when vhost feature is enabled. But later we decided to include the rook-service-endpoint
to rgw DNS names if not it will break the existing clusters access
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm still struggling to understand this.
However, I do understand this statement, and I don't think the logic makes sense to me:
rgw service endpoint will be part of the rgw DNS names so we can pick RGW service endpoint for all the internal communications. As I mentioned earlier by default rgw service endpoint is part of there is no specific need to pick the domain name from the rgw DNS list
If the RGW service endpoint exists, I don't think that necessarily means that we should pick it when reporting/selecting the endpoint. From what I understand, the point of adding rgw dns names
is to allow users to use wildcard addressing, which is not possible for default service endpoints. Only the endpoints added by the user in .spec.hosting.dnsNames
are wildcard-addressable, and those endpoints should be preferred selections for OBCs and CephObjectStoreUsers. This is especially true because the wildcard-addressability is actually supposed to be the S3 default (the older path-style addressing is deprecated).
@BlaineEXE : At least according to https://access.redhat.com/solutions/7032743 the k8s endpoint can be wildcarded with the help of DNS service(doc was using Noobaa k8s service endpoint of s3). Secondly, even if we pick a domain from list of |
I agree with this.
I disagree with this for the reasons I explained here: #14034 (comment) To expand on this: Because vhost-style bucket addressing is the only non-deprecated S3 access method, it is only natural that Rook should treat it as the preferred method. It is against the ethos of Kubernetes and of Rook to foist onto end users the responsibility of configuring their CR-based (OBC-based) clients based on the input parameters of the dependent resource (CephObjectStore) that those end users likely don't have read permissions to. Rook must figure out how to use vhost-style However, if a user mis-configures wildcarding for an endpoint they add to |
@BlaineEXE @travisn is another issue which I found while testing this feature with QA in ODF. If we enable both SSL and normal ports in then the DNS list contains domains which list to |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Requesting changes to await discussion and clarification about what are the current problems with current design.
I am not clear on what exactly the problems are that are leading to this patch, and without a clear understanding, I don't want to merge code that attempts to band-aid patch something that is a bigger design issue. I want a clear breakdown of how to repro workflows that are bugged, specific error output, what the root cause of the failure is, as well as what attempts were made to resolve the issue.
It also seems like #14303 may be intended to be a replacement for this one, taking a different approach. Should this one be closed in favor of the other?
Personally I favours this PR over other one, Once we decide which approach should take then we can close the PR accordingly |
For the reasons I mentioned in this post, Rook won't take this change because it won't treat the wildcard-enabled endpoints as preferred outputs to OBCs. Closing this. |
The endpoint of cephobjectstore is populated with help of
BuildDNSEndpoint
which combines domain name fromgetDomainName
andport
in theobjectstore.spec.gateway
. This approach is used in most of the Rook code base and even for adminsops api for Rook to communicate with RGW.In case vhost feature is enabled it returns the domain from rgw DNS name list, but the domain picked may not have the same port as the rgw internal port which is failing for adminsops use case. So the populated endpoint in this case is wrong.
Even if the feature is enabled by default rgw service endpoint will be part of the rgw DNS names so we can pick RGW service endpoint for all the internal communications
Checklist: