this change moves stack of compression methods, now global variable #24414

Sashan · 2024-05-16T04:39:02Z

in libssl into crypto context. This change is rquired by atexit PR.

mattcaswell

When I build this I get linker errors:

/usr/bin/ld: ./libssl.so: undefined reference to `ossl_lib_ctx_get_data'
/usr/bin/ld: ./libssl.so: undefined reference to `ossl_find_compression'
/usr/bin/ld: ./libssl.so: undefined reference to `ossl_lib_ctx_set0_compression_methods'

This is because these symbols are not exported from libcrypto. It seems to me the only symbol that you actually need to export from libcrypto is `ossl_lib_ctx_get_data'. The other two could probably just be in libssl.

We have as yet not had the concept of internal symbols that are exported from libcrypto and are privately used by libssl. All symbols exported from libcrypto are public and form part of the API. With that in mind I would actually create a new public libcrypto function OSSL_LIB_CTX_get_data as a simple wrapper around ossl_lib_ctx_get_data. The public documentation would say that, although this is a public function, all the indexes that you can pass are reserved for internal use and so applications should not call this function. We might at some point in the future extend it with the idea to support application supplied indexes - but we don't have to worry about that now.

crypto/comp_methods.c

crypto/context.c

include/internal/stack_of_comp_methods.h.in

ssl/ssl_init.c

Sashan · 2024-05-16T13:57:49Z

When I build this I get linker errors:

/usr/bin/ld: ./libssl.so: undefined reference to `ossl_lib_ctx_get_data'
/usr/bin/ld: ./libssl.so: undefined reference to `ossl_find_compression'
/usr/bin/ld: ./libssl.so: undefined reference to `ossl_lib_ctx_set0_compression_methods'

I've not noticed those errors on OpenBSD/clang tool chain. I can see those errors on Debian-12/gcc tool chain. I agree we can remove ossl_find_compression() which implementation can be kept in libssl. I will also need public version of ossl_lib_ctx_set0_compression_methods() because it is a workhorse for SSL_COMP_set0_compression_methods() function. Unless we want to remove SSL_COMP_set0_compression_methods() from libssl API.

mattcaswell · 2024-05-16T14:08:26Z

I will also need public version of ossl_lib_ctx_set0_compression_methods() because it is a workhorse for SSL_COMP_set0_compression_methods() function. Unless we want to remove SSL_COMP_set0_compression_methods() from libssl API.

I'm not sure we need to move this to libcrypto.

What if the libssl data we stored in the libctx was actually a STACK_OF(OSSL_COMP) **? Then SSL_COMP_set0_compression_methods() could become:

STACK_OF(SSL_COMP) *SSL_COMP_set0_compression_methods(STACK_OF(SSL_COMP)
                                                      *meths)
{
    STACK_OF(OSSL_COMP) **data = OSSL_LIB_CTX_get_data(NULL, OSSL_LIB_CTX_COMP_METHODS);
    STACK_OF(SSL_COMP) *old_meths = (STACK_OF(SSL_COMP) *)*data;

    *data = (STACK_OF(OSSL_COMP) *)meths;
    return old_meths;
}

Sashan · 2024-05-17T04:57:30Z

The updated PR addresses all suggestions from @mattcaswell

I failed to figure out how to hook newly added file crypto/comp_methods.c with FIPS module build. Therefore the PR currently disables compression methods for FIPS. With out this workaround in place the fips module fails to load due to unresolved symbols (ossl_load_builtin_compressions()). Perhaps we should dissolve crypto/comp_methods.c into existing source code files in libcrypto.

crypto/context.c

crypto/comp_methods.c

crypto/context.c

doc/man3/OSSL_LIB_CTX.pod

Sashan · 2024-05-17T08:37:17Z

I will also need public version of ossl_lib_ctx_set0_compression_methods() because it is a workhorse for SSL_COMP_set0_compression_methods() function. Unless we want to remove SSL_COMP_set0_compression_methods() from libssl API.

I'm not sure we need to move this to libcrypto.

What if the libssl data we stored in the libctx was actually a STACK_OF(OSSL_COMP) **? Then SSL_COMP_set0_compression_methods() could become:
STACK_OF(SSL_COMP) *SSL_COMP_set0_compression_methods(STACK_OF(SSL_COMP)
                                                      *meths)
{
    STACK_OF(OSSL_COMP) **data = OSSL_LIB_CTX_get_data(NULL, OSSL_LIB_CTX_COMP_METHODS);
    STACK_OF(SSL_COMP) *old_meths = (STACK_OF(SSL_COMP) *)*data;

    *data = (STACK_OF(OSSL_COMP) *)meths;
    return old_meths;
}

I'll update PR shortly to see how it will look. On the other hand I'm not sure if those changes are heading in desired direction. I feel it goes against the concept 'all data are private' to OSSL_LIB_CTX. Changing OSSL_LIB_CTX_get_data() so it returns address of OSSL_LIB_CTX member makes auditing/reasoning about the code harder. Explicit set function is better in my opinion, because it makes reasoning/auditing the code bit easier.

mattcaswell · 2024-05-17T08:45:34Z

I feel it goes against the concept 'all data are private' to OSSL_LIB_CTX.

That isn't actually the current design of OSSL_LIB_CTX. Currently the context.c code mostly has a bunch of internal members which are "black boxes" as far as context.c is concerned. The internals of the black box are private to the module that uses it. E.g. consider ctx->drbg. The context.c code knows nothing about what that data member is - rather this is managed by the crypto/rand/rand_lib.c code where we discover that ctx->drbg is actually a RAND_GLOBAL structure.

Sashan · 2024-05-17T10:06:41Z

New version makes OSSL_LIB_CTX_get_data() to return pointer-to-pointer for OSSL_LIB_CTX_COMP_METHODS index.

crypto/comp_methods.c

doc/man3/OSSL_LIB_CTX.pod

include/internal/stack_of_comp_methods.h.in

mattcaswell

Looks good. Just a few minor nits remain.

doc/man3/OSSL_LIB_CTX.pod

include/internal/cryptlib.h

mattcaswell · 2024-05-22T14:54:42Z

CI failures seem to be due to #24463

t8m · 2024-05-22T15:26:33Z

include/openssl/ssl.h.in

-typedef struct ssl_comp_st SSL_COMP;
+typedef struct ossl_comp_st SSL_COMP;


I would not change the typedef from ssl_comp_st to ossl_comp_st. Just move it to openssl/comp.h and keep using SSL_COMP everywhere instead of OSSL_COMP.

Those casts from OSSL_COMP to SSL_COMP are awkward and unnecessary if we just do not introduce the new type.

t8m

Just a few final tweaks needed

t8m · 2024-05-22T18:30:15Z

include/openssl/comp.h

+typedef struct ssl_comp_st {
+    int id;
+    const char *name;
+    COMP_METHOD *method;
+} SSL_COMP;


The structure content should be kept internal -> into internal/comp.h

t8m · 2024-05-22T18:31:32Z

include/openssl/comp.h

+
+STACK_OF(SSL_COMP);
+
+DEFINE_STACK_OF(SSL_COMP)


This must be generated through the generate_stack_macros("SSL_COMP") perl generator by moving that from ssl.h.in -> i.e., it requires generating comp.h from comp.h.in

I'm wondering why we are bothering to move it? Why not just keep it in ssl.h.in?

You need to be able to release the stack from libcrypto, so that means the definition needs to be in a header that code linked into libcrypto can include. IMO ssl.h should not be included.

IMO ssl.h should not be included.

Why not if we're only using it for some typedefs etc? Moving things to comp.h is problematic because it is excluded in the case of a no-comp build - whereas SSL_COMP should not be.

Alternatively you just replicate the DEFINE_STACK_OF(SSL_COMP) line in context.c and not include ssl.h at all.

the updated PR includes all suggestions from t8m. the code in comp.h.in got reshuffled so only relevant portion in that file is protected by OPENSSL_NO_COMP guard. I did test and build it:
- perl ./Configure --strict-warnings enable-zlib enable-comp
- perl ./Configure --strict-warnings no-zlib no-brotli no-zstd no-comp
both versions did pass build and make test

ssl/ssl_ciph.c

t8m · 2024-05-23T13:51:06Z

include/openssl/ssl.h.in

@@ -976,7 +974,6 @@ extern "C" {
 */
 {-
    generate_const_stack_macros("SSL_CIPHER")


isn't this missing ; ?

t8m · 2024-05-23T13:51:43Z

include/openssl/comp.h.in

 *
 * Licensed under the Apache License 2.0 (the "License").  You may not use
 * this file except in compliance with the License.  You can obtain a copy
 * in the file LICENSE in the source distribution or at
 * https://www.openssl.org/source/license.html
 */

+{-
+use OpenSSL::stackhash qw(generate_stack_macros generate_const_stack_macros);


generate_const_stack_macros in't needed

t8m · 2024-05-23T13:52:38Z

include/openssl/comp.h.in

+ *
+ * Copyright 4 The OpenSSL Project Authors. All Rights Reserved.
+ * Copyright (c) 2002, Oracle and/or its affiliates. All rights reserved
+ * Copyright 2005 Nokia. All rights reserved.


I wouldn't add this. None of the things moved were IMO added by these copyright holders.

t8m · 2024-05-23T13:53:00Z

include/openssl/comp.h.in

@@ -1,12 +1,20 @@
 /*
- * Copyright 2015-2018 The OpenSSL Project Authors. All Rights Reserved.
+ * Copyright 2024 The OpenSSL Project Authors. All Rights Reserved.


The original copyright should be kept 2015-2024

t8m · 2024-05-23T13:54:31Z

ssl/ssl_ciph.c

-    return ssl_comp_methods;
+    STACK_OF(SSL_COMP) **rv;
+
+    rv = (STACK_OF(SSL_COMP) **) OSSL_LIB_CTX_get_data(NULL,


Nit: no space after **)

in libssl into crypto context. This change is rquired by atexit PR.

t8m

@mattcaswell IMO this is OK. Maybe the SSL_COMP stack declarations could have been handled differently but I see this as cleaner way.

mattcaswell · 2024-05-24T07:18:34Z

include/openssl/comp.h.in

+typedef struct ssl_comp_st SSL_COMP;
+
+{-
+    generate_stack_macros("SSL_COMP");


I would have preferred to keep this in ssl.h.in. But ok.

openssl-machine · 2024-05-25T08:00:16Z

This pull request is ready to merge

t8m · 2024-05-28T06:57:35Z

Merged to the master branch with reworded commit message. Thank you.

The compression methods are now a global variable in libssl. This change moves it into OSSL library context. It is necessary to eliminate atexit call from libssl. Reviewed-by: Matt Caswell <matt@openssl.org> Reviewed-by: Tomas Mraz <tomas@openssl.org> (Merged from #24414)

The compression methods are now a global variable in libssl. This change moves it into OSSL library context. It is necessary to eliminate atexit call from libssl. Reviewed-by: Matt Caswell <matt@openssl.org> Reviewed-by: Tomas Mraz <tomas@openssl.org> (Merged from openssl#24414)

Sashan requested a review from mattcaswell May 16, 2024 04:39

Sashan force-pushed the compression branch 2 times, most recently from 6e47851 to d4a5fb3 Compare May 16, 2024 08:02

mattcaswell requested changes May 16, 2024

View reviewed changes

nhorman assigned Sashan May 16, 2024

Sashan force-pushed the compression branch from d4a5fb3 to 7a7c1ea Compare May 17, 2024 04:51

Sashan force-pushed the compression branch from 7a7c1ea to ad4e0e4 Compare May 17, 2024 05:24

github-actions bot added severity: fips change The pull request changes FIPS provider sources severity: ABI change This pull request contains ABI changes labels May 17, 2024

mattcaswell requested changes May 17, 2024

View reviewed changes

Sashan force-pushed the compression branch from ad4e0e4 to bfa5854 Compare May 17, 2024 08:10

Sashan force-pushed the compression branch from bfa5854 to 38da98a Compare May 17, 2024 10:03

Sashan requested a review from mattcaswell May 17, 2024 12:31

Sashan force-pushed the compression branch from 38da98a to ebb778b Compare May 21, 2024 07:36

mattcaswell requested changes May 21, 2024

View reviewed changes

Sashan force-pushed the compression branch from ebb778b to b1a1113 Compare May 21, 2024 15:19

Sashan marked this pull request as ready for review May 21, 2024 15:23

Sashan requested a review from mattcaswell May 22, 2024 07:55

mattcaswell requested changes May 22, 2024

View reviewed changes

doc/man3/OSSL_LIB_CTX.pod Outdated Show resolved Hide resolved

doc/man3/OSSL_LIB_CTX.pod Outdated Show resolved Hide resolved

doc/man3/OSSL_LIB_CTX.pod Outdated Show resolved Hide resolved

include/internal/cryptlib.h Outdated Show resolved Hide resolved

Sashan force-pushed the compression branch from b1a1113 to 0ea6f8f Compare May 22, 2024 14:47

mattcaswell approved these changes May 22, 2024

View reviewed changes

mattcaswell added branch: master Merge to master branch approval: review pending This pull request needs review by a committer labels May 22, 2024

mattcaswell added the tests: exempted The PR is exempt from requirements for testing label May 22, 2024

t8m closed this May 22, 2024

t8m reopened this May 22, 2024

t8m added the triaged: refactor The issue/pr requests/implements refactoring label May 22, 2024

t8m requested changes May 22, 2024

View reviewed changes

Sashan force-pushed the compression branch from 0ea6f8f to 4b65968 Compare May 22, 2024 17:22

t8m requested changes May 22, 2024

View reviewed changes

Sashan force-pushed the compression branch from 4b65968 to a131287 Compare May 23, 2024 13:41

Sashan requested a review from t8m May 23, 2024 13:45

t8m requested changes May 23, 2024

View reviewed changes

this change moves stack of compression methods, now global variable

76bf3f6

in libssl into crypto context. This change is rquired by atexit PR.

Sashan force-pushed the compression branch from a131287 to 76bf3f6 Compare May 23, 2024 15:17

Sashan requested a review from t8m May 23, 2024 15:30

t8m approved these changes May 23, 2024

View reviewed changes

mattcaswell approved these changes May 24, 2024

View reviewed changes

mattcaswell added approval: done This pull request has the required number of approvals and removed approval: review pending This pull request needs review by a committer labels May 24, 2024

mattcaswell mentioned this pull request May 24, 2024

Massive performance degradation in OpenSsl 3.0 if used in a heavy multi threaded server application #17064

Open

openssl-machine added approval: ready to merge The 24 hour grace period has passed, ready to merge and removed approval: done This pull request has the required number of approvals labels May 25, 2024

t8m closed this May 28, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

this change moves stack of compression methods, now global variable #24414

this change moves stack of compression methods, now global variable #24414

Sashan commented May 16, 2024

mattcaswell left a comment

Sashan commented May 16, 2024

mattcaswell commented May 16, 2024

Sashan commented May 17, 2024

Sashan commented May 17, 2024

mattcaswell commented May 17, 2024

Sashan commented May 17, 2024

mattcaswell left a comment

mattcaswell commented May 22, 2024

t8m May 22, 2024

t8m May 22, 2024

t8m left a comment

t8m May 22, 2024

t8m May 22, 2024

mattcaswell May 23, 2024

t8m May 23, 2024

mattcaswell May 23, 2024

mattcaswell May 23, 2024

Sashan May 23, 2024

t8m May 23, 2024

t8m May 23, 2024

t8m May 23, 2024

t8m May 23, 2024

t8m May 23, 2024

t8m left a comment

mattcaswell May 24, 2024

openssl-machine commented May 25, 2024

t8m commented May 28, 2024

		typedef struct ssl_comp_st SSL_COMP;
		typedef struct ossl_comp_st SSL_COMP;


		STACK_OF(SSL_COMP);

		DEFINE_STACK_OF(SSL_COMP)

this change moves stack of compression methods, now global variable #24414

this change moves stack of compression methods, now global variable #24414

Conversation

Sashan commented May 16, 2024

mattcaswell left a comment

Choose a reason for hiding this comment

Sashan commented May 16, 2024

mattcaswell commented May 16, 2024

Sashan commented May 17, 2024

Sashan commented May 17, 2024

mattcaswell commented May 17, 2024

Sashan commented May 17, 2024

mattcaswell left a comment

Choose a reason for hiding this comment

mattcaswell commented May 22, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

t8m left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

t8m left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

openssl-machine commented May 25, 2024

t8m commented May 28, 2024