floating point exception (divide by zero)

[Arinerron]

Hi,

Somehow my RDP server gave a response to rdesktop that caused pz_powm(y, x, exp, mod); (mod) to be 0 presumably, causing a SIGFPE arithmetic exception.

[#0] 0x7ffff7b5b55a → __gmp_exception()
[#1] 0x7ffff7b5b58e → __gmp_divide_by_zero()
[#2] 0x7ffff7b70b0d → __gmpz_powm()
[#3] 0x555555680280 → rdssl_rsa_encrypt(out=0x5555558b5440 <g_sec_crypted_random> "", in=0x5555559c60a0 <g_client_random> "", len=0x20, modulus_size=0x0, modulus=0x7fffffffd930 "", exponent=0x7fffffffd8a8 "")
[#4] 0x5555555e12ea → sec_rsa_encrypt(exponent=0x7fffffffd8a8 "", modulus=0x7fffffffd930 "", modulus_size=<optimized out>, len=0x20, in=<optimized out>, out=0x5555558b5440 <g_sec_crypted_random> "")
[#5] 0x5555555e12ea → sec_process_crypt_info(s=0x5555558b5220 <g_in>)
[#6] 0x5555555e487c → sec_process_mcs_data(s=0x5555558b5220 <g_in>)
[#7] 0x5555555dddf7 → mcs_recv_connect_response(mcs_data=0x1fd)
[#8] 0x5555555dddf7 → mcs_connect_finalize(mcs_data=0x555555c106f0)
[#9] 0x5555555e8cb3 → sec_connect(server=<optimized out>, username=<optimized out>, domain=0x7fffffffdfe0 "", password=0x5555558b23a0 <g_password> "", reconnect=0x0)

If I break at _gmpz_powm (to see what values are in the registers):

Breakpoint 1, 0x00007ffff7b70284 in __gmpz_powm () from /usr/lib/x86_64-linux-gnu/libgmp.so.10
gef➤  reg
$rax   : 0x00007fffffffd750  →  0x0000000000000004
$rbx   : 0x0               
$rcx   : 0x00007fffffffd730  →  0x0000000000000001
$rdx   : 0x00007fffffffd720  →  0x0000000000000001
$rsp   : 0x00007fffffffd6d0  →  0x00007fffffffd730  →  0x0000000000000001
$rbp   : 0x00007fffffffd6d0  →  0x00007fffffffd730  →  0x0000000000000001
$rsi   : 0x00007fffffffd750  →  0x0000000000000004
$rdi   : 0x00007fffffffd740  →  0x0000000000000001
$rip   : 0x00007ffff7b70284  →  <__gmpz_powm+4> push r15
$r8    : 0x1               
$r9    : 0x00005555559c60c0  →  0x0000000000000000
$r10   : 0x0               
$r11   : 0x00005555559c60c0  →  0x0000000000000000
$r12   : 0x00007fffffffd720  →  0x0000000000000001
$r13   : 0x00007fffffffd750  →  0x0000000000000004
$r14   : 0x00007fffffffd740  →  0x0000000000000001
$r15   : 0x00005555558b5440  →  0x0000000000000000
$eflags: [zero carry parity adjust sign trap INTERRUPT direction overflow resume virtualx86 identification]
$cs: 0x0033 $ss: 0x002b $ds: 0x0000 $es: 0x0000 $fs: 0x0000 $gs: 0x0000 

Doesn't really make sense since the 4th arg (rcx) is 1 not 0. Perhaps _gmpz_powm internally subtracts 1 first or smth?

If I break at rdssl_rsa_encrypt:

[#0] 0x555555680150 → rdssl_rsa_encrypt(out=0x5555558b5440 <g_sec_crypted_random> "", in=0x5555559c60a0 <g_client_random> "", len=0x20, modulus_size=0x0, modulus=0x7fffffffd930 "", exponent=0x7fffffffd8a8 "")

modulus is "\0"!

Is the correct solution here to add a assert(modulus) to prevent undefined behavior? Happy to submit a PR