Unable to enable specific ciphers

[ivanvaccari]

I need to enable a specific ciphers for Tls v1.2 wich is used by an old system to connect to our service via nginx as reverse proxy.

The deault ssl configuration don't let the service connect and give us a generic "ssl handshake failed". After some research i found out i need to enable the AES128-GCM-SHA256 cipher, but despite adding it to the nginx.conf file, which the gets mounted in the image, nginx won't use it.

Using ssllabs.com tool the cipher suite i get is the following one:

which does not include the needed cipher.

• I'm using the debian 5.01 image.
• openssl ciphers -s shows it in list.

I'm getting a bit lost online searching for various infos, it looks like that if:

• openssl supports it (which it does, checked with "openssl ciphers")
• nginx.conf defines it in "ssl_ciphers" string

Then it sould appear and be available. But it's not.

An older setup (nginx directly installed in the host, not via docker) from which the config was copy-pasted does allow the usage, as confirmed via ssllabs tool:

There's something about it i need to know?

[JonasAlfredsson]

Hi ivanvaccari,

Can you post your nginx.conf file and explain how you mount it into the container.

On a server level you can define ciphers like in this example: https://github.com/JonasAlfredsson/docker-nginx-certbot/blob/master/examples/example_server_multicert.conf

[ivanvaccari]

Hi, this is is the nginx.config in use:

worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 768;
        # multi_accept on;
}

http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        # server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;
        client_max_body_size 16M;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings. See https://ssl-config.mozilla.org/#server=nginx
        ##
        ssl_protocols TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;
        ssl_session_cache shared:le_nginx_SSL:10m;
        ssl_session_timeout 1440m;
        ssl_session_tickets off;
        # Updated list: https://ssl-config.mozilla.org/#server=nginx
        ssl_ciphers TLS_RSA_WITH_AES_128_GCM_SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305;

        ##
        # Logging Settings
        ##
        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;

        ##
        # Gzip Settings
        ##
        gzip on;

        # gzip_vary on;
        # gzip_proxied any;
        # gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
        # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

        ##
        # Virtual Host Configs
        ##
        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}

The needed cipher is TLS_RSA_WITH_AES_128_GCM_SHA256 (https://ciphersuite.info/cs/TLS_RSA_WITH_AES_128_GCM_SHA256/)

The compose configuration used to launch the instance:

version: '3.7'
services:
  nginx:
    image: jonasal/nginx-certbot:5.0.1
    volumes:
    - ./nginx/maintenance:/var/www/maintenance
    - ./nginx/maintenancemode:/etc/nginx/maintenancemode
    - ./nginx/nginx.conf:/etc/nginx/nginx.conf     # volume per config nginx
    - ./nginx/templates:/etc/nginx/templates
    - ./letsencrypt:/etc/letsencrypt # volume per certbot. Necessario solo se si usa immagine jonasal/nginx-certbot
    ports:
      - target: 80
        published: 80
        protocol: tcp
        mode: host
      - target: 443
        published: 443
        protocol: tcp
        mode: host 
    environment:
    - NGINX_HOST=***************
    - CERTBOT_EMAIL=*****************
    deploy:
      replicas: 1
      update_config:
        parallelism: 1
        delay: 10s
        order: stop-first

[JonasAlfredsson]

Well, if you look at the at the list of ciphers you listed:

ssl_ciphers TLS_RSA_WITH_AES_128_GCM_SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305;

You see that none of the other alternatives have underscores in them. If you follow your link you also see the "openssl name" (AES128-GCM-SHA256) listed just below the IANA name, so I think if you just replace TLS_RSA_WITH_AES_128_GCM_SHA256 with that it should work.

[ivanvaccari]

Used all the names in various tentatives.

Currently the ssl_ciphers i'm using is

ssl_ciphers AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305;

And the value is correctly set in the container:

docker exec -it `docker ps | grep nginx-certbot | cut -d' ' -f1` cat /etc/nginx/nginx.conf | grep ssl_ciphers

shows:

ssl_ciphers AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-CHACHA20-POLY1305;

However it's not active (testing from a local cli):

$ openssl s_client -cipher AES128-GCM-SHA256 -connect HIDDEN:443 -tls1_2
CONNECTED(000001A8)
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 7 bytes and written 178 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : 0000
    Session-ID:
    Session-ID-ctx:
    Master-Key:
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    Start Time: 1715843593
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
    Extended master secret: no
---
18260:error:14094410:SSL routines:ssl3_read_bytes:sslv3 alert handshake failure:../openssl-1.1.1q/ssl/record/rec_layer_s3.c:1543:SSL alert number 40

Neither ssllabs shows it.

Does the fact that the container uses openssl 3 have something that influences it?

[JonasAlfredsson]

Did you restart or just reload the container after changing the config?

And does nginx say anything in the logs about any problems?

[JonasAlfredsson]

Questions regarding how nginx interacts with openssl is probably better to ask in the parent container's GitHub since we just add some nice to have she'll scripts at startup and don't touch nginx at all.

[ivanvaccari]

Did you restart or just reload the container after changing the config?

And does nginx say anything in the logs about any problems?

i'm using docker swarm to manage images, the command i usually use to force the reload is this:

docker service update default_stack_nginx --force

Actually i'm not 100% sure what it does, but it should restart the container because usually it creates/remove containers before creating new ones.
Palying around with other ciphers are reflected in the container, both ssllabs and the openssl-cli confirm the presence or absence of the changed ciphers

[JonasAlfredsson]

If you see the config changed inside the container then I think it does a full restart of it, as is necessary with file mounts, so that is good.

The TLS_RSA_WITH_AES_128_GCM_SHA256 (AES128-GCM-SHA256) cipher suite is part of TLS1.2, and appears to be supported by OpenSSL v3, so I don't think that is the problem.

You don't happen to override something through the other configuration files here?

        ##
        # Virtual Host Configs
        ##
        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;

Then I would suggest you restart the container with the environment variable DEBUG=1 and look through the logs to see if Nginx says something about invalid ciphers.

[ivanvaccari]

You don't happen to override something through the other configuration files here?

No overrides apparently. /etc/nginx/sites-enabled/ does not exists, and the content of /etc/nginx/conf.d/ is the following:

redirector is the default of the image, default.cnf is the following:

# maintenance mode
geo $geo_host {
    # our vpn gateway
    HIDDEN allowed;
    # dynamic ips
    include /etc/nginx/maintenancemode/geo_dyn.conf;
}



server {

    server_name HIDDEN;

    # maintenance mode
    error_page 503 /var/www/maintenance/error503.html;
    location /var/www/maintenance/error503.html {
        alias /var/www/maintenance/error503.html;
    }

    # SocketIO. By default socketIo is managed by standalone mit-ray.
    location /socket.io/ {

        # maintenance mode
        if (-f /etc/nginx/maintenancemode/maintmode$geo_host) {
            return 503;
        }


        proxy_pass http://mit-ray:3100/socket.io/;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;

        # dont cache anything
        proxy_cache off;

        #websocket
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection 'upgrade';
        proxy_cache_bypass $http_upgrade;

    }

    # web server
    location / {

        # maintenance mode
        if (-f /etc/nginx/maintenancemode/maintmode$geo_host) {
            return 503;
        }

        proxy_pass http://prodocu:3000/;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        # dont cache anything
        proxy_cache off;
    }


    listen [::]:443 ssl ipv6only=on;
    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/HIDDEN/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/HIDDEN/privkey.pem;
    ssl_dhparam /etc/letsencrypt/dhparams/ssl-dhparams.pem;

}

I'll try DEBUG=1 (not now, i have some other work to do)

[JonasAlfredsson]

Unless there is something in the log you should probably reach out to the Nginx community to potentially get answers from those who knows this program in more detail than me :)

[ivanvaccari]

Then I would suggest you restart the container with the environment variable DEBUG=1 and look through the logs to see if Nginx says something about invalid ciphers.

I feel a little bit stupid to ask this question: i have enabled the DEBUG=1 flag, also by adding the "debug" directive to the "error_log" (docs here https://nginx.org/en/docs/debugging_log.html) but... where i'm supposed to read this log?

The /var/log/nginx/error.log pipes data to /dev/stderr, and when i cat it, it's initially empty.
Setting it to a standard file also does not log boot data, only requests errors.

Some ideas?

[JonasAlfredsson]

The parent image forwards (by default) the logs to stdout and stderr so it can be handled by the Docker log collector.
I have not used Docker swarm, so I don't know how to get logs from such a setup, but docker logs <container> should net you all output from a container running on your system.

[ivanvaccari]

Docker swarm is just an alternative mode to orchestrate containers, but all the other options of docker ar still available.

Definitely, docker logs with DEBUG=1 does not show anything about ciphers.

I'ts horrible to say it, but to make it work i've proxied the set of specific REST api used by that old system via a unencrypted http endpoint. The sysadmin of the remote system don't want to enable more modern ciphers on their system.

[JonasAlfredsson]

This seems like a little bit unexpected behavior since OpenSSL defines it as being part of TLS1.2, but using the Mozilla generator your linked the cipher mentioned only show up on the OLD setting which defines all the other TLS versions as well.

ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; 

a last try could perhaps be to add these as well to see if it works?

While I think the remote system is in the wrong (and I sure hope they don't handle any sensitive data), you could point a "legacy" domain name to your old Nginx deployment and then just let it establish a connection with your new Nginx for this particular system.

[JonasAlfredsson]

Also, I don't think this ciphser needs it, but you could also try to add the ssl_dhparam and let my container generate it at boot and see if that makes Nginx happy.

[ivanvaccari]

ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; 

a last try could perhaps be to add these as well to see if it works?

Already tried (was one of the first things i did), but it still only enable TLS>=1.2, probably due to this: openssl/openssl#13299 (comment)
Also tried to lower SECLEVEL as that message suggests, with no luck.

I'm stopping following this problem for now and maturate some other ideas/research

[JonasAlfredsson]

Alright, you could also post this issue for the original Nginx container to see if someone have similar experiences: https://github.com/nginxinc/docker-nginx

[ivanvaccari]

Fyi: this solved the issue: nginxinc/docker-nginx#893

Quick solution: AES128-GMC-SHA256 requires a rsa key-based ssl certificate to be enabled:

From openssl ciphers -v:
AES128-GCM-SHA256 TLSv1.2 Kx=RSA Au=RSA Enc=AESGCM(128) Mac=AEAD

but the certificate generated by docker.nginx-certbot is ECDSA-based. I have deleted the previous certificate and set USE_ECDSA=0 to let the image regenerate a certificate with RSA key and now nginx also load the needed cipher.

[JonasAlfredsson]

Thank you for returning with the answer here, today I learned something as well!

In the linked issue they mention dual certificates, and we actually have an example here if you are interested: https://github.com/JonasAlfredsson/docker-nginx-certbot/blob/master/examples/example_server_multicert.conf