Segmentation fault when running local dev

[stephenlacy]

I have a rather basic app with partykit and partysocket/react which randomly exits with Segmentation fault: 11 after a few minutes when idle or sending messages.
Is there a detailed debug log mode available?

╭──────────────────────────────────────────────────────────────────────────────╮
 │ [b] open a browser, [c] clear console, [x] to exit                           │
╰──────────────────────────────────────────────────────────────────────────────╯
[pk:inf] GET /party/test 101 Switching Protocols (1ms)
Connected:
  id: 3afea696-7a48-4734-89c5-bfbbd1b95ef3
  room: test
  url: /party/test
[pk:inf] GET /party/test 101 Switching Protocols (1ms)
connection 3afea696-7a48-4734-89c5-bfbbd1b95ef3 sent message: {"type":"self:update","payload":{"username":"test","isReady":true}}
✘ [ERROR] *** Received signal #11: Segmentation fault: 11

node: v21.5.0
macos: ARM64 14.2.1
partysocket: 0.0.19
partykit: 0.0.63

[threepointone]

Ugh, that sounds bad, sorry about that. I updated the runtime at 0.0.63, can you try with 0.0.62 and tell me if the problem persists?

If you can make a small repro, I'd be happy to dig into that too.

[stephenlacy]

Super hard to reproduce, it seems to have to do with party state on 0.0.63, but cant tell exactly what triggers the segfault as I had many parties with dev data.

[threepointone]

To clarify, does this mean using 0.0.62 fixes the issue for you? If so, I'll revert the miniflare update I did, else not.

[stephenlacy]

Got a segfault on both 0.0.62 and 0.0.63. I copied the .partykit -PartyKitDurable folder to a minimal example and after 5-7 minutes got a segfault:
https://github.com/stephenlacy/partykit-repro-1

pnpm i
pnpm dev

Running pnpm dev in each folder (app, partykit) also works to isolate the tty sessions.
open http://localhost:5173/game/MUBJ click both buttons a few times, wait 5-7 minutes.

[threepointone]

This is super helpful, thank you! I'll investigate tomorrow. One last thing: can you try turning hibernation off and see if that fixes the segfaults?

[stephenlacy]

Ran it for a few hours with several manual restarts, it did not seem to trigger a segfault with hibernation off.

[threepointone]

Ok, this is great info. I'm going to talk to cloudflare/workerd about this and file an issue with them. Been trying to reproduce it locally with your repro but haven't been able to, will keep trying this weekend.

[stephenlacy]

Thanks, is there a way to turn on verbose logging?

[threepointone]

Yeah, partykit dev --verbose should output more information, though I'm not sure whether it'll help here. Let me know if it does!

[stephenlacy]

Seems I'll need to dig into why --verbose fails

🎈 PartyKit v0.0.63
--------------------
Build succeeded, starting server...
✘ [ERROR] workerd/server/server.c++:3170: info: Inspector is listening


[pk:inf] Ready on http://0.0.0.0:1999
[pk:inf] - http://127.0.0.1:1999
[pk:inf] - http://192.168.1.3:1999
✘ [ERROR] workerd/server/server.c++:1142: info: Inspector client attaching [core:user:]


Connected:
  id: 370aa7bd-6963-449e-b36f-a7e164b778ec
  room: partytest
  url: /party/partytest
[pk:inf] GET /party/partytest 101 Switching Protocols (19ms)
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ [b] open a browser, [c] clear console, [x] to exit                                                                                                                                                                                                                                                                                                                                        │
✘ [ERROR] workerd/io/worker.c++:989: info: console warning; message = A promise rejection was handled asynchronously. This warning occurs when attaching a catch handler to a promise after it rejected. (rejection #1)


✘ [ERROR] workerd/io/worker.c++:1771: info: uncaught exception; source = Uncaught (in promise); exception = Error: [onClose] Worker not initialized.


✘ [ERROR] workerd/io/worker.c++:1771: info: uncaught exception; source = Uncaught (async); exception = Error: [onClose] Worker not initialized.
[pk:inf] GET /party/partytest 101 Switching Protocols (2ms)
connection d3bb6d21-b853-4122-9fd1-1f2194d0808c sent message: {"type":"fetch","payload":"count"}
connection d3bb6d21-b853-4122-9fd1-1f2194d0808c sent message: {"type":"fetch","payload":"members"}
connection d3bb6d21-b853-4122-9fd1-1f2194d0808c sent message: {"type":"fetch","payload":"self"}
connection d3bb6d21-b853-4122-9fd1-1f2194d0808c sent message: {"type":"self","payload":{"username":"test"}}
╭───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╮
│ [b] open a browser, [c] clear console, [x] to exit                                                                                                                                                                                                                                                                                                                                        │
✘ [ERROR] *** Received signal #11: Segmentation fault: 11

  stack:

[threepointone]

Oh hmm interesting

[threepointone]

can you try with partykit@latest and let me know if you still see this problem?

[stephenlacy]

Still getting it:

connection 73ba3b03-dce6-4b7d-a35c-8bb84d3d19d2 sent message: {"type":"self","payload":{"username":"test"}}
╭──────────────────────────────────────────────────────────────────────────────╮
partykit:dev: ✘ [ERROR] *** Received signal #11: Segmentation fault: 11
partykit:dev:
partykit:dev: │ [b] open a browser, [c] clear console, [x] to exit                           │
partykit:dev: ╰──────────────────────────────────────────────────────────────────────────────╯
partykit:dev:   stack:
partykit:dev:
partykit:dev:

dependencies:
partysocket 0.0.20
partykit 0.0.71

Is there a way to dump the stack from the runtime vs an empty stack?

[krachtstefan]

I'm using partysocket 0.0.25 and partykit 0.0.91 seem to have the described issue as well. It happens after a couple of minutes with just one client connected. I have hibernate set to true as well. It did not come up with hibernate set to false.

I'm pretty sure this is hard to debug/reproduce/fix.

[abizek]

I'm getting workerd/server/server.c++:3314: error: Uncaught exception: kj/compat/http.c++:4060: disconnected: worker_do_not_log; Request failed due to internal error on client connect/disconnect with partykit 0.0.105. Seems to be randomly occurring. Is this related to this issue?