Replies: 20 comments 68 replies
-
|
Sorry but that violates RFC 7686 #10705 |
Beta Was this translation helpful? Give feedback.
-
Yeah but you can't even force it with torify or proxychains... Isn't that kind of silly especially when you can specify a socks host/port? Like it wouldn't be hard to run a check to see if the socks proxy used the same default port as tor and if it is allow it or if it is able to resolve the DNS fine with a preloaded library like tsocks/torify/proxychains then just allow the request through? |
Beta Was this translation helpful? Give feedback.
-
That's a bit surprising since 0ae0abb was merged after 8.0.1. We could consider adding an option that allows a user to change this behavior. |
Beta Was this translation helpful? Give feedback.
-
When you connect to tor in a normal setup, you do not let curl resolve the host name because it can't resolve If you would instead ask curl to resolve So this is why I ask for actual use cases. Because I think you're asking for features you don't quite understand. |
Beta Was this translation helpful? Give feedback.
-
|
Ignore my comment, curl does resolve the onion sites if given the --socks5-hostname option |
Beta Was this translation helpful? Give feedback.
This comment was marked as disruptive content.
This comment was marked as disruptive content.
-
|
Looks completely broken for me now. proxychains curl https://duckduckgogg42xjoc72x3sjasowoarfbgcmvfimaftt6twagswzczad.onion/ |
Beta Was this translation helpful? Give feedback.
-
|
@klikevil you're just bringing back old topics that add nothing. You're using the wrong proxy option. If you want the proxy to resolve the host name, which you typically do for tor, you'd use |
Beta Was this translation helpful? Give feedback.
This comment was marked as disruptive content.
This comment was marked as disruptive content.
-
|
curl implemented this to follow RFC 7686 which states
Applications that do not implement the Tor protocol SHOULD generate an error upon the use of .onion and SHOULD NOT perform a DNS lookup.
curl & libcurl do not implement the Tor protocol, so "Please be reasonable" to me means to follow the RFCs.
|
Beta Was this translation helpful? Give feedback.
-
I think "it breaks my weirdo use case" is a weak argument when there is a clear RFC saying how things should work. |
Beta Was this translation helpful? Give feedback.
This comment was marked as disruptive content.
This comment was marked as disruptive content.
-
|
Please speak to the Tor people about the best way to achieve this, as they don't seem to agree with an environment variable either, per the discussion in #11236 -> https://gitlab.torproject.org/tpo/core/torspec/-/issues/202#note_2909909. |
Beta Was this translation helpful? Give feedback.
-
|
bro, --resolve-onion is too complex? Please don't bikeshed open source software like this. We've been using flags for stuff like this since eons ago. If want you want is mess with how tor and proxies work please refrain from doing so. Curl should just have the flag. |
Beta Was this translation helpful? Give feedback.
-
|
Completely agree, wget works just fine, but curl now can't work with onion urls at all even when libraries are preloaded? It's bonkers! |
Beta Was this translation helpful? Give feedback.
-
|
@klikevil curl works fine with Tor and SOCKS and has done so for decades. You are not helping. |
Beta Was this translation helpful? Give feedback.
-
|
IIRC we backported @Kangie's patch in Gentoo once it'd been reviewed, but OP was already having issues because we switched to enabling adns by default. We don't do anything else to do with proxies and it's purely their setup. |
Beta Was this translation helpful? Give feedback.
This comment was marked as disruptive content.
This comment was marked as disruptive content.
-
|
I don't think that has much to do with what I said. I was just giving more context as to what happened. |
Beta Was this translation helpful? Give feedback.
-
|
Hi all, I thought this might come up (and alluded to it in my PR). I was alerted to this being an "issue" due to Gentoo bug 887287. TL;DR: depending on how cURL was compiled before (using internal or c-ares DNS resolution) you would get different results when attempting to resolve The PR both resolved a longstanding known issue with cURL where we were not respecting RFC7686 and ensured consistency between DNS resolution backends. @kaizushi: You're a Gentoo user. Please feel free to use portages localpatches function to patch out the RFC7686 checks; the PR has been linked here and you only need to worry about the 8 lines in @bagder Sorry about the 8.0.x confusion, the onion resolution stuff got patched in Gentoo a little early as I knew I'd be away for a bit around the 8.0.0 release and wanted to ensure that our users weren't vulnerable. |
Beta Was this translation helpful? Give feedback.
This comment was marked as disruptive content.
This comment was marked as disruptive content.
-
|
@kaizushi, we've discussed this in IRC. I originated this patch and I maintain the For 8.0.1 in Gentoo, the patch was backported and is applied by portage. You can find it here. For 8.0.1 remove that patch (preferably in a non ::gentoo repo to save you some effort). For 8.1.0+ revert that patch in localpatches. That is all that there is to it. I accept that your particular use case for cURL + .onion resolution is valid, however the RFC we're adhering to came from the TOR developers and protects the greatest possible number of users from accidentally leaking information via DNS lookups while seemingly inconveniencing only you. Anyone who is sufficiently technically competent to run a system as complex as yours (I'm thinking maybe whonix, as a potential example) is competent enough to build their own cURL binary with this functionality patched out. |
Beta Was this translation helpful? Give feedback.
-
|
It costs 0 dollars to a flag, or a Env variable that simply covers the use case. What is the point if the software does not try to cover valid use cases? What is curl doing except serving the user and that is done covering valid setups. I find you attitude very petty considering your alternative for not adding a flag to this old behaviour being each user brings their own build of curl everywhere and to all devices on their network (if it is transparent proxied). Just add a flag and stop having a power trip. |
Beta Was this translation helpful? Give feedback.
-
|
@Pachin0 you are not helping by being hostile. Please remain friendly and constructive. This topic has partially continued in #11236 |
Beta Was this translation helpful? Give feedback.
-
|
Fair enough, I apologize for the tone. |
Beta Was this translation helpful? Give feedback.
-
What we're doing is being compliant with the RFC originated by the Tor developers. If you check the PR you'll see that I considered it and decided against it. |
Beta Was this translation helpful? Give feedback.
-
|
Don't use proxychains with curl, use its flag --socks5-hostname and it should resolve everything properly. @kaizushi in my setup |
Beta Was this translation helpful? Give feedback.
This comment was marked as disruptive content.
This comment was marked as disruptive content.
-
|
Hi, I have implemented a centralized tor outbound in my internal network. As well I implemented a dns resolver for the # curl -vI https://dns4torpnlfs2ifuz2s2yf3fc7rdmsbhm6rw75euj35pac6ap25zgqad.onion
* Trying *.*.*.*:443...
* Connected to dns4torpnlfs2ifuz2s2yf3fc7rdmsbhm6rw75euj35pac6ap25zgqad.onion (*.*.*.*) port 443 (#0)
* ALPN: offers h2,http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* CAfile: /etc/ssl/certs/ca-certificates.crt
* CApath: /etc/ssl/certs
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted h2
* Server certificate:
* subject: jurisdictionC=US; jurisdictionST=Delaware; businessCategory=Private Organization; serialNumber=4710875; C=US; ST=California; L=San Francisco; O=Cloudflare, Inc.; CN=tor.cloudflare-dns.com
* start date: Dec 14 00:00:00 2022 GMT
* expire date: Dec 13 23:59:59 2023 GMT
* subjectAltName: host "dns4torpnlfs2ifuz2s2yf3fc7rdmsbhm6rw75euj35pac6ap25zgqad.onion" matched cert's "dns4torpnlfs2ifuz2s2yf3fc7rdmsbhm6rw75euj35pac6ap25zgqad.onion"
* issuer: C=US; O=DigiCert Inc; CN=DigiCert TLS Hybrid ECC SHA384 2020 CA1
* SSL certificate verify ok.
* using HTTP/2
* h2h3 [:method: HEAD]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: dns4torpnlfs2ifuz2s2yf3fc7rdmsbhm6rw75euj35pac6ap25zgqad.onion]
* h2h3 [user-agent: curl/7.88.1]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x557b914e0c70)
> HEAD / HTTP/2
> Host: dns4torpnlfs2ifuz2s2yf3fc7rdmsbhm6rw75euj35pac6ap25zgqad.onion
> user-agent: curl/7.88.1
> accept: */*
>
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
< HTTP/2 200
HTTP/2 200
< date: Tue, 20 Jun 2023 **:**:** GMT
date: Tue, 20 Jun 2023 **:**:** GMT
< content-type: text/html
content-type: text/html
< nel: {"report_to":"cf-nel","max_age":604800}
nel: {"report_to":"cf-nel","max_age":604800}
< last-modified: Thu, 04 Aug 2022 19:10:01 GMT
last-modified: Thu, 04 Aug 2022 19:10:01 GMT
< strict-transport-security: max-age=31536000
strict-transport-security: max-age=31536000
< served-in-seconds: 0.002
served-in-seconds: 0.002
< cache-control: max-age=600
cache-control: max-age=600
< cf-cache-status: DYNAMIC
cf-cache-status: DYNAMIC
< server: cloudflare
server: cloudflare
<
* Connection #0 to host dns4torpnlfs2ifuz2s2yf3fc7rdmsbhm6rw75euj35pac6ap25zgqad.onion left intactBut with curl 8.1 this behavior was broken (even with the $ curl -vi https://dns4torpnlfs2ifuz2s2yf3fc7rdmsbhm6rw75euj35pac6ap25zgqad.onion --resolve "*:443:*.*.*.*"
* Added *:443:*.*.*.* to DNS cache
* RESOLVE *:443 using wildcard
* Not resolving .onion address (RFC 7686)
* Could not resolve host: dns4torpnlfs2ifuz2s2yf3fc7rdmsbhm6rw75euj35pac6ap25zgqad.onion
* Closing connection 0
curl: (6) Not resolving .onion address (RFC 7686)Is it possible to add a compatibility flag to make the old program work as well? |
Beta Was this translation helpful? Give feedback.
-
|
see #11236 |
Beta Was this translation helpful? Give feedback.
-
|
OK, I'm going to do a drive-by revival of this discussion and then safely ignore it for a while. Current status (July '23):
My personal thoughts are that adding an envvar for this is a short-sighted band-aid solution at best; we (that is, you, the people who care about this) can do better. To quote my last update upstream:
Unfortunately, this isn't something that cURL can resolve. I would encourage those that are interested in this topic to continue the discussion along these lines upstream. The above question (and implied RFC compliance) does bring me to another question that I have: Is transproxying the recommended / sane approach for what you want to do? I see a lot of unsupported claims of better security from individuals with these "interesting" network configurations, but no real-world evidence that it's the better / preferred approach (and I'm not digging into it myself right now, nor will I engage in flame wars on the topic here). Please consider that those using custom transproxy (or SOCKS) configurations make up a very small proportion of total Tor users. My primary concern in this matter is the far greater number of non-technically-inclined users who are at risk of accidentally leaking that they're using Tor to an adversary and ending up imprisoned (or worse...). I don't think that anyone is necessarily wrong for wanting their existing solution to work, but I do think that anyone complaining about this (and not working to find a real solution) is failing to see the bigger picture. If you have the expertise to run a complex environment please revert the cURL RFC 7686 patches and build your own binary (on my laptop that takes about 90 seconds [including running tests]); groups such as Tails or Whonix who provide turnkey environments will probably consider doing this anyway. |
Beta Was this translation helpful? Give feedback.
-
|
a workaround is to explicitly use a tor socks5h proxy usually this is - torsocks curl http://gg6zxtreajiijztyy5g6bt5o6l3qu32nrg7eulyemlhxwwl6enk6ghad.onion
+ curl --proxy socks5h://127.0.0.1:9050 http://gg6zxtreajiijztyy5g6bt5o6l3qu32nrg7eulyemlhxwwl6enk6ghad.onion- torsocks git clone http://gg6zxtreajiijztyy5g6bt5o6l3qu32nrg7eulyemlhxwwl6enk6ghad.onion/RightToPrivacy/Gitea-Onion
+ git -c remote.origin.proxy=socks5h://127.0.0.1:9050 clone http://gg6zxtreajiijztyy5g6bt5o6l3qu32nrg7eulyemlhxwwl6enk6ghad.onion/RightToPrivacy/Gitea-Onion
+ # TODO ideally avoid this:
+ git -C Gitea-Onion config --add remote.origin.proxy socks5h://127.0.0.1:9050see also https://gitlab.torproject.org/tpo/core/torsocks/-/issues/40016 - torsocks fails to set socks5h proxy for curl, git, ... - Not resolving .onion address (RFC 7686) |
Beta Was this translation helpful? Give feedback.
-
|
I have no idea of and no opinion on the curl C code or how maintainable (or not) it is with and without the proposed change. However I'd like to make a general commentary on the philosophical idea of code. Specifically, difficult to maintain code is difficult to maintain code regardless of how many lines of code it is. The number of lines of code is almost totally irrelevant to this; what matters is architectural design, whether code causes side effects from the perspective of other code, how "clever" a particular line of code is in terms of abusing strange language features, whether the code has unittests, whether the code constitutes a subsystem in need of a dedicated maintainer, etc. Please discuss the maintainability of the proposed code based on the merits of the code, not based on incorrect metrics such as how many lines it takes up in a textual source file. (Yes, if you contribute changes to a project I maintain, I will absolutely treat "just 20 lines" as seriously as a 15k line change in terms of reviewability and maintainability. Both are serious topics. How many lines the code is, is one of many factors that influences the final decision.) |
Beta Was this translation helpful? Give feedback.
-
|
brainstorm mode enabled libcurltransfer options are set in libcurl with curl_easy_setopt(), and I would imagine this could be a new option there. It could possibly be made as a generic If done proper, we should be able to create a unit test to verify it. curlAdd a new option |
Beta Was this translation helpful? Give feedback.
-
|
Sounds sane to me. |
Beta Was this translation helpful? Give feedback.
-
|
IMO this is overboard. If we have to do something about this, just treat RFC compliance as a default-enabled feature so that users can toggle it at compile time, and so that if this ever comes up a quick Edit: To be clear, I don't agree with making this change in the first place, but if we have to make it, let's do it in a way that keeps DNS resolution behaviour consistent with the way that the binary was compiled. I've asked around on the upstream development mailing list and have had no response; we can probably wait a while before taking action either way - at this point any downstreams that care to are already maintaining their own patchset while Tails and Whonix just use a socks proxy. That's actually a pretty good argument for maintaining the status quo, now that I think about it. |
Beta Was this translation helpful? Give feedback.
-
moving this issue upstream to glibc would be nice... edit 2024-02-24: i left a comment on glibc bug 24255 then to allow resolving .onion domains /etc/nsswitch.conf feels like the wrong place for this the issue also affects other DNS clients like drill, kdig, unbound-host, dig, host, nslookup, resolvectl, ...
|
Beta Was this translation helpful? Give feedback.
-
While this is incredibly admirable open source effort, do be sure this is the right thing to do/concession to make, considering TOR people wrote the RFC that sparked this whole thing. AIUI TOR is intended to be a completely anonymised practically sandboxed browsing experience, libcurl is practically ubiquitous across a huge surface of open source as well as private projects. "App leaks TOR user information due to improper use of libcurl" becomes a possibility with this decision. I'm not implying misuse is the responsibility of you the maintainer, I am simply pointing out that you're opening a door for it that's previously been closed due to adherence to an RFC that intended to close it. |
Beta Was this translation helpful? Give feedback.
-
thats like saying "we should prohibit all drugs, because 1% of all people are too stupid to use them, and they will hurt themselves." oh wait, most people actually are that stupid, and support this policy... curl's default config would still be "never resolve onion domains", and if users disable this protection, its "at own risk". if an attacker has access to this config, again, it's the user's fault, not curl's fault. https://gitlab.torproject.org/tpo/core/torspec/-/issues/202
|
Beta Was this translation helpful? Give feedback.
-
You need to calm down and consider this from a higher level, broader strokes perspective. It's nothing like potential drug abuse; it does however add rather wide reaching considerations for developers, system administrators, sec ops and code reviewers as soon as the option exists. I'd actually suggest asking TOR to modify the RFC to sanction these kinds of overrides if they're going to happen, so people looking up the TOR spec are informed adequately. Going off spec historically causes problems and this isn't an exception no matter how much you want your own personal little Hackers cosplay to work the way you say it should. |
Beta Was this translation helpful? Give feedback.
-
|
I agree with you on that the scenario described by the RFC is highly imaginary. We cannot decide what's best for the users, and they need to make their own choices with the most degrees of freedom we can offer. |
Beta Was this translation helpful? Give feedback.
-
|
Hey all, I have started on a pull request to provide the plumbing needed to make the prohibition on |
Beta Was this translation helpful? Give feedback.
-
|
Can somebody point me to the reason why .onion is not supposed to be resolved by non-tor software? How do we define "Applications that do not implement the Tor protocol"? Say there's a browser that has nothing to do with the Tor protocol, but can be configured to use a socks5 proxy. Is it considered "Applications that do not implement the Tor protocol"? My browser gives me the option to select whether to use proxy for DNS resolution, which has been working sufficiently for me when using Tor circuit. While the DNS problem can be perfectly solved in a very general way aforementioned, why is it necessary to force all non-tor software to give error when resolving .onion domain names? |
Beta Was this translation helpful? Give feedback.
-
|
https://www.rfc-editor.org/rfc/rfc7686#section-2
|
Beta Was this translation helpful? Give feedback.
-
|
It's all in RFC 7686.
|
Beta Was this translation helpful? Give feedback.
This comment was marked as disruptive content.
This comment was marked as disruptive content.
-
|
The RFC in question is dated October 2015. Why is it only implemented recently? |
Beta Was this translation helpful? Give feedback.
-
Developer interest, time, and necessity. There was an abortive attempt a while ago (#5159) but it never got through to being merged. I encountered a bug report about different |
Beta Was this translation helpful? Give feedback.
-
That's why we have an RFC to comply with. To paraphrase it says "(for applications) do not attempt to resolve onion addresses unless you are tor aware."
If there was a sane, standards compliant, mechanism for an application to discover that it is in a Tor-enabled environment without inadvertently putting users at risk due to DNS leakage we'd implement it here. As you've identified though, deviation from the standard might result in behaviour that puts users at risk. Remember that this "issue" only impacts the transparent proxy scenario where all traffic is funneled into Tor without the applications being aware that there behind a proxy - we're not blanket preventing access to Tor / onion addresses. If I've misunderstood your point I apologise, but from where I'm sitting it looks like you've misunderstood where this is coming from. |
Beta Was this translation helpful? Give feedback.
-
I don't think it was a polite way to address others and it also ignores the substantive discussion which has taken place. "You people" and a "mess" are not going to move a discussion forward. (also, the technical reasons to not implement that suggestion have been well covered here) |
Beta Was this translation helpful? Give feedback.
-
|
@Kangie I do not think we are disagreeing. |
Beta Was this translation helpful? Give feedback.
-
only the user knows, if his LAN gateway can safely resolve .onion domains solutions: #11125 (reply in thread) |
Beta Was this translation helpful? Give feedback.
-
|
POP-3 support was added 21 years after the POP-3 RFC came out. That's just when it happened.
|
Beta Was this translation helpful? Give feedback.
-
|
https://lists.torproject.org/pipermail/tor-dev/2023-November/014862.html provides new insights by Silvio Rhatto from torproject.org. Quote:
So this should be reverted from There was no opposition against options to change these settings either.
The only missing thing is a standardized recommendation how to disable this. Meanwhile I suggest environment variable |
Beta Was this translation helpful? Give feedback.
-
I would say that the decision logic being required to be in the application is actually an argument against reverting it. The only valid option is, of course, to remove all API and ABI support for existing programs and require them to use brand new APIs that include this decision as a parameter. Every application must become tor-aware in order to use c-ares, because c-ares isn't allowed to make decisions about whether or not its own API is, and must belligerently challenge all its API consumes. A decision such as, for example, "if you need to support right now, include some decision logic that calls into an alternative library".
I don't think the curl project is worried about causing offended feelings. Just about https://xkcd.com/927/ |
Beta Was this translation helpful? Give feedback.
-
|
I ducked out for a bit as I was busy with real life. c-ares is a DNS resolution library. They are REQUIRED not to resolve onion addresses. I am still not in support of having curl resolve onion differently based on DNS backend. I am also not in favour of dropping the asynch dns resolver to remove this block. Do you have any productive ideas? |
Beta Was this translation helpful? Give feedback.
-
I did this
Curl will not resolve onion addresses anymore on my system. This is because of RFC 7686, and the change was made because of DNS leakage. I have a good handle on any possible DNS leakage and do not need this feature, and it in fact breaks my system. I use transparent proxying, and I am certain DNS leakage is impossible here.
curl/libcurl version
curl 8.0.1 (x86_64-pc-linux-gnu) libcurl/8.0.1 OpenSSL/1.1.1t zlib/1.2.13 libidn2/2.3.4 nghttp2/1.51.0
Release-Date: 2023-03-20
Protocols: dict file ftp ftps http https imap imaps mqtt pop3 pop3s rtsp smtp smtps tftp
Features: AsynchDNS HTTP2 HTTPS-proxy IDN IPv6 Largefile libz NTLM SSL threadsafe TLS-SRP UnixSockets
operating system
Gentoo Linux (default/linux/amd64/17.1/hardened/selinux)
Beta Was this translation helpful? Give feedback.
All reactions