rfbShutdownServer() in threaded mode with clients connected on Win32 crashes

[lynara]

Hi,
I hope it is okay to post here about my issue.

Describe the bug
I'm using libvncserver in a plugin for x-plane. Everything works fine but when I'm trying to shutdown the vncserver(s) with rfbShutdownServer as long as clients are connected the application crashes. If no clients are connected it works fine.

To Reproduce

1. Start X-Plane
2. Connect to vncserver
3. Disable x-plane plugin (which executes rfbShutdownServer)
4. X-Plane crashes

Expected Behavior
rfbShutdownServer should disconnect clients and stop the vnc server thread

Logs/Backtraces

Your environment (please complete the following information):

• OS and version: Windows 10 20H2
• Compiler and version: msvc 19.27.29111

Additional context
LibVNCServer-0.9.13

Server init code:

for (int i = 0; i < cockpitWindows; i++) {
		...
		server[i] = rfbGetScreen(NULL, NULL, width[i], height[i], 8, 3, 4);
		server[i]->frameBuffer = &sub_buffer[i][0];
		server[i]->desktopName = &windowName[i][0];
		server[i]->port = 5900 + i;
		server[i]->newClientHook = newclient;
		rfbInitServer(server[i]);
		rfbRunEventLoop(server[i], -1, TRUE);
	}

Shutdown code in plugin loop:

while (1) {
		if (pluginDisabled) {
			for (int i = 0; i < cockpitWindows; i++) {
				rfbShutdownServer(server[i], TRUE);
			}
			//ExitThread(0);
			return 0;
		}
...

Thank you! :)

[bk138]

Uh, looks complicated! ;-)

Some ideas:

1. Try running a single-threaded server via rfbRunEventLoop(server[i], -1, FALSE);
2. I'm not aware of any program that runs multiple servers aka rfbScreens simultaneously. You're on terra incognita there and might try with a single server/rfbScreen first.
3. You might also try with current LibVNCServer master, there have been some commits since 0.9.13.

[lynara]

Hi,
I made some tests.

1. With rfbRunEventLoop(server[i], -1, FALSE); I couldn't get it to work after a quick try.
   I changed my plugin loop to use rfbProcessEvents(server[i], 100000);

while (1) {
		if (pluginDisabled) {
			for (int i = 0; i < cockpitWindows; i++) {
				rfbShutdownServer(server[i], TRUE);
				rfbProcessEvents(server[i], 100000);
			}
			//ExitThread(0);
			return 0;
		}

		for (int i = 0; i < cockpitWindows; i++) {
			rfbProcessEvents(server[i], 100000);
		}

Here the shutdown works without a crash, but performance is miserable because of the many blocking calls of rfbProcessEvents i guess.

2. Just using a single server results in the same crash behavior.
3. Compiled with master, still crashed but now just getting this error (void rfbClientConnectionGone(rfbClientPtr cl))
   
   

Thanks!

[bk138]

After looking some more into this, it might be a bug in the WIN32THREADS implementation, which is quite new.

It would really help if you could try the blooptest example (maybe extending it to call rfbShutdownServer() on its own) and see if the crash happens there as well.

[lynara]

Thanks for your effort!

I tried with blooptest example as suggested and I'm getting the same results there.
With no client connected shutdown works, with client connected it crashes at the same line.

[bk138]

Great! Did you simply press F12 with a client connected?

[lynara]

No, I edited the while loop in example.c

    int i = 0;
    while (1) {
        if (i > 300) {
            fprintf(stderr, "shutting down...\n");
            rfbShutdownServer(rfbScreen, TRUE);
        }
        fprintf(stderr, "loop...\n");
        sleep(5); /* render(); */
        i++;
    }

Just tried with F12. Without debugger it seems to shutdown just fine but with debugger I see that an exception gets thrown.

[bk138]

Just tried with F12. Without debugger it seems to shutdown just fine but with debugger I see that an exception gets thrown.

Ha, that's why it never showed up in testing.