Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tlsdump can decrypt SSL/TLS traffic? #1437

Open
federicofantini opened this issue Mar 15, 2023 · 19 comments
Open

tlsdump can decrypt SSL/TLS traffic? #1437

federicofantini opened this issue Mar 15, 2023 · 19 comments

Comments

@federicofantini
Copy link
Contributor

Hi, today I was looking at this really interesting feature in the file: analyzer/windows/modules/auxiliary/tlsdump.py.
At the end of the analysis I can download the pcap and log files containing respectively the whole network traffic and the dumped keys. CAPEv2 is able to decrypt the traffic? In this way the suricata signatures will be more effective!

I also tried to import these two files to wireshark in order to decrypt the traffic. This feature of wireshark is supported only for TLS <=1.2 versions but also in this legacy communication version I can’t see clear traffic. Maybe not all keys are collected?
I did a test by analyzing my exe file that contacts (HTTP GET) to this URL: https://tls-v1-2.badssl.com:1012/

P.S. I know I'm not using templates but I don't know if these issues I'm opening are "bug" or "feature request" or something else... I hope it's not a problem.

Thanks in advance!

@github-actions
Copy link

@federicofantini: hello! 👋

This issue is being automatically closed because it does not follow the issue template.

This is open source project!
So please apreciate our time that we sacrify from other thing that we could enjoy, instead of asking boring things over and over.

@kevoreilly
Copy link
Owner

Yes indeed the tlsdump feature should mean that the encrypted traffic in the pcap gets decrypted and thus seen by suricata - you should therefore see the results of this in the network tab when you run your test exe - is this the case?

@federicofantini
Copy link
Contributor Author

Yes, that would be my use case!
So I intend to both see the decrypted traffic in the pcap in the network tab and apply suricata to the decrypted traffic to get a better vision.
I thought it was already like this but actually reading the sources I can't find the point where the decryption is done, I only see the tlsdump.log file is saved and proposed in the gui.

Also, trying to decrypt the traffic manually from wireshark (need to convert the tlsdump.log file to the CLIENT_RANDOM supported format by wireshark), I can't see the decrypted traffic for TLSv1.2 connections.

@kevoreilly
Copy link
Owner

No I didn't mean "is this your use case?" - I meant "is it the case that your exe produces the expected output in the network tab?"!

@federicofantini
Copy link
Contributor Author

Nope, the pcap still encrypted and I'm unable to decrypt with wireshark. The tabs in the network tab are:

  • Hosts (7)
  • DNS (8)
  • TCP (18)
  • UDP (33)
  • HTTP (0)
  • SMTP (0)
  • IRC (0)
  • ICMP (1)
  • Suricata Alerts (0)
  • Suricata TLS (0)
  • Suricata HTTP (0)
  • Suricata Files (0)

And my test exe is just a py2exe conversion of this script:

import requests
import time

urls = [
    "https://www.google.com",
    "https://www.apple.com",
    "https://www.aruba.it",
    "https://www.repubblica.it",
    "https://tls-v1-2.badssl.com:1012/",
    "https://tls-v1-1.badssl.com:1011/",
    "https://tls-v1-0.badssl.com:1010/"
]

for u in urls:
    r = requests.get(u)
    print(r.status_code)
    time.sleep(5)

I was looking at the public instance and I saw that this feature is already present and that it works on win7: https://capesandbox.com/analysis/374610/ (sorry for the misunderstanding).
However I have a question: why some traffic is not decrypted on that analysis? I see some TCP communications on port 443.

@federicofantini
Copy link
Contributor Author

federicofantini commented Mar 17, 2023

Hi, I went a little bit deeper and I want to update you about this issue.

Thanks to this script https://github.com/lbirchler/tls-decryption I was able to see the decrypted version of the dump.pcap file by passing in input client_random/server_random bytes and the master_secret key.
The decrypted traffic contains stuff about windows noisy telemetry and updates but nothing about my forged requests. I don't understand why this decrypted but noisy traffic, which contains some HTTP requests, is not visible on the Network HTTP(s) tab.

Futhermore in the journalctl of cape-processor I can see this skipped errors:

[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9d[g?\xee(\xb3\x92\xbc\xd9\x9a\xc7*\xb1\x81KM\xb6*\xf1\xb1\xe1\xd0F\xb0\x93\x9f4+\x>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9d\\\x83=Vw\xb4\x91_\x89@L\xe8\xb2e\x9dCU\x96\x83\xe3\xcb\x8f\xb4\xc5\x04+;\xdaR', >
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9d]\xb1\xef0\t\xe7K\xf2,3\xa1\xf15g\xa1\xd5\xa5AQ\xb6]0\xccm;W\x82|\xed', skipping >
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9d`H\x9aT\x8609\\\x12\x0c0%?\x9e\xa3\xf6\xe5\xeb\xaeB{\xd8ua]\x0b\x81Q\xa1', skippi>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b"d\x14\x9d`\x8a\xcb\x0b\xbfzL\x82}\xe4\x91N\n\x15\xda\xd9V\x98\x98zaCr\x85\xe2\xb1'\x16\xc8>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b"d\x14\x9d`\xaf~]`\xd2\xd9b\x13z\x98\xdfc\\\x9ay\xd0\x90\x17m\x8b6\xb5\r'\xa8\xbb\xfd\x93",>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9dhXE\xd7\x85\x83k1\xb2a~\x0b\xf5I\x94\xcc\x89Y\xce\x84}\xde3e\x9b\xc5\xf9\x8fn', s>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9di\x85\xf6,f?\xd7\xfe\xb5\\Z\xc3\x96\x1e:\xabW\xe1\xe7\xd1\xa5?\xc5/\x85\x049\xb7\>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9diO\x87\xe8I \xc5\xe1\xb4\xe2N8\xf3^`Vu\x9a_A\xfb\xde\x1bU9<\xf9l\xe6', skipping it
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9dii\xb8G\x99\xa2S%\x84\xacA\x88\xbe\xaa\x99\x8f\\\x84\xc9\xa2\x8a\x8f\xec\xae\x7f;>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9d\x917T(1\xa1P\xab\xe3W\x88_ZoCOb\xee\xe5\x97\x0c8\xc4\xe8\\=\xee\xb0\xdd', skippi>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9d\x92=\xb3\xe9*6\xa3\x15\xa9\x13\x8ea\xf01a\xd2\x84\xf4\xff\x82x`G\xe5\x00Wr\xac\x>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9d\x92[\xba\xb5\x87\x1e\x17\xe5\xe0\xb7#=\x18\x05\x96L\xd0)z\xbf\x85m\xe0\x94\x97\x>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9d\x95\xd6\xa0g\xd3U\x8aM\xe8\xa1\xd3\xf6\x94\xear\x8fJ\x1e\x01\x9f\xe0[bF&>t\xd3\x>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9d\x9e\xd6^\x91\x1e\xf8\x92\x8a\x91\x19\xc2x\x92 \xc8M3(B\x9a\x1b\xf2\x7f\xb9\xbc>w>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9d\x9fwq\xa8U\xa3\x05v\x13\x9b6H\xdaW\xc7\xf5L\xda\xdf\xafx\x8fY\xf0\xbc\xcb\x82l\x>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b'd\x14\x9d\xa2&\xd2\xb9i\xbe\x97\x8b\xe0\xa0\xb7\x92\xce\xc1"\xecV\xb0J\xb9192<\xbc\x16\x80>
[Task 104] [modules.processing.dumptls] DEBUG: Was unable to extract TLS master secret for server random b"d\x14\x9d\xa2\x9dt\x1c\xb3\xad\xf8\xfc\xdc\xd6\xcb,\xb8'\x8d]\xae|\xb8\xec\xb5 \xc4\xef*=v>

I think this analysis https://capesandbox.com/analysis/374610/ may have something like mine for the TCP 443 requests.

@daothinh
Copy link

Hi @federicofantini, Can you show me the configuration of INETSIM for CAPE?, i can't find any instructions :(

@federicofantini
Copy link
Contributor Author

@daothinh you need to install inetsim on a dedicated VM inside the libvirt network: https://www.techanarchy.net/installing-and-configuring-inetsim/
When the installation is completed insert the IP address of the machine here: https://capev2.readthedocs.io/en/latest/installation/host/routing.html?highlight=inetsim#inetsim-routing

@mvasilescu
Copy link

Yes indeed the tlsdump feature should mean that the encrypted traffic in the pcap gets decrypted and thus seen by suricata

@kevoreilly i'm not sure how this feature works - are there any specific CAPE configuration requirements for this to work?
Does suricata have to work in socket mode ? Does it work in cli mode as well?
I've looked through the documentation and haven't found anything related to this.

In my scenario, i have some HTTPs traffic (that i can see in the CAPE networking tab under HTTP(s) - but does not appear under Suricata HTTP tab or Suricata alerts - regardless of the suricata rules that i wrote)

@kevoreilly
Copy link
Owner

The HTTP(s) field is straight from the pcap whereas the suricata fields depend on suricata processing the pcap. So it sounds like the capture to pcap and decryption is fine, suricata side is not working.

@mvasilescu
Copy link

Hi @kevoreilly - appreciate you taking the time to respond.

I've also added some "test" rules for suricata for that pcap - and i see them in the report.
I was not able however to get a suricata signature to trigger on HTTP(s) decrypted traffic.
I saw your reply - and was wondering how is this actually done?

Yes indeed the tlsdump feature should mean that the encrypted traffic in the pcap gets decrypted and thus seen by suricata
My understanding is that for a HTTPs request that is normally encrypted in a PCAP - it actually gets decrypted and passed to suricata. i don't understand how to get this working.

e.g.
alert http $HOME_NET any -> $EXTERNAL_NET any (msg:"test rule"; flow:established,from_client; http.method; content:"GET"; http.uri; content:"prof"; sid:99999997;)
this will trigger on a HTTP request but not on a HTTPs decrypted request.

Is there anything i'm missing?
Thanks!

@doomedraven
Copy link
Collaborator

we don't decrypt pcap. we decrypt them for us to show in cape in network data processing, if you need that to be working with suricata, you can write extension for cape to decrypt pcap and save updated pcap, as far as i know you just need to supply certificate, maybe even you can do that with suricata if they support that

@mvasilescu
Copy link

Hi @doomedraven - thanks for the update.
That sounds more like what i thought it would be.
Suricata does not do any decryption - if you want to run signatures on that traffic - you need to decrypt it beforehand.
So yeah, probably the decrypt pcap extension sounds the best path for now.

@doomedraven
Copy link
Collaborator

i know that people has code for that, but they don't share code, i don't need this feature so i don't do it by myself. but if you want to contribute you are more than welcome to implement it

@kevoreilly
Copy link
Owner

I will be making a PR for this soon - watch this space.

@CarsonHrusovsky
Copy link
Contributor

Nope, the pcap still encrypted and I'm unable to decrypt with wireshark. The tabs in the network tab are:

  • Hosts (7)
  • DNS (8)
  • TCP (18)
  • UDP (33)
  • HTTP (0)
  • SMTP (0)
  • IRC (0)
  • ICMP (1)
  • Suricata Alerts (0)
  • Suricata TLS (0)
  • Suricata HTTP (0)
  • Suricata Files (0)

And my test exe is just a py2exe conversion of this script:

import requests
import time

urls = [
    "https://www.google.com",
    "https://www.apple.com",
    "https://www.aruba.it",
    "https://www.repubblica.it",
    "https://tls-v1-2.badssl.com:1012/",
    "https://tls-v1-1.badssl.com:1011/",
    "https://tls-v1-0.badssl.com:1010/"
]

for u in urls:
    r = requests.get(u)
    print(r.status_code)
    time.sleep(5)

I was looking at the public instance and I saw that this feature is already present and that it works on win7: https://capesandbox.com/analysis/374610/ (sorry for the misunderstanding). However I have a question: why some traffic is not decrypted on that analysis? I see some TCP communications on port 443.

Hello did you ever get this to work with your setup? I see now that official CAPE doesn't contain any http or https traffic anymore - I hope this is still possible. Is it as simple as taking the tlsdump.log and using that to decrypt the pcap - then just feed that pcap to the mechanism responsible for populating the Network Analysis tab on the UI?

@doomedraven
Copy link
Collaborator

doomedraven commented Sep 16, 2023 via email

@kevoreilly
Copy link
Owner

This is still very much a feature, nothing has been removed from 'official' cape!

@doomedraven
Copy link
Collaborator

Kev is due that VPN is down

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

6 participants