isolationcamera stopped answering and took display down with it
The contract says display must keep working when camera fails (at least 3 completed messages). camera stopped answering; display completed only 0 messages afterwards and its thread 1 is still REPLY-blocked on camera.
Evidence: trace events 58318
Show the 18 trace events behind this
| Event | Time | What happened |
|---|---|---|
| 55076 | 2234.580 ms | display[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10' |
| 55086 | 2234.605 ms | camera[tid 1] MsgReceive got the message from display[tid 1] |
| 55111 | 2235.704 ms | camera[tid 1] MsgReply to display[tid 1] reply '\x03\x10' |
| 55492 | 2292.234 ms | display[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10' |
| 55502 | 2292.256 ms | camera[tid 1] MsgReceive got the message from display[tid 1] |
| 55512 | 2292.303 ms | camera[tid 1] MsgReply to display[tid 1] reply '\x03\x10' |
| 55840 | 2348.586 ms | display[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10' |
| 55850 | 2348.644 ms | camera[tid 1] MsgReceive got the message from display[tid 1] |
| 55860 | 2348.809 ms | camera[tid 1] MsgReply to display[tid 1] reply '\x03\x10' |
| 56249 | 2406.380 ms | display[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10' |
| 56259 | 2406.451 ms | camera[tid 1] MsgReceive got the message from display[tid 1] |
| 56269 | 2406.899 ms | camera[tid 1] MsgReply to display[tid 1] reply '\x03\x10' |
| 58019 | 2467.649 ms | display[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10' |
| 58029 | 2467.705 ms | camera[tid 1] MsgReceive got the message from display[tid 1] |
| 58039 | 2467.792 ms | camera[tid 1] MsgReply to display[tid 1] reply '\x03\x10' |
| 58318 | 2521.298 ms | display[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10' |
| 58328 | 2521.347 ms | camera[tid 1] MsgReceive got the message from display[tid 1] |
| -58319 (note) | - | at trace end display[tid 1] is still REPLY-blocked on camera (no reply to event 58318) |
camera answered five frame requests, then on the sixth (camera_frames > 5, COCKPIT_FAULT=camera-hang) camera_handle wedged in pause() after MsgReceive and never called MsgReply. display's main loop calls MsgSend to camera inline with no isolation, so its only thread stays REPLY-blocked forever and never reaches nav or logger again. A send timeout would not help: the name_attach() channel has _NTO_CHF_UNBLOCK, so the client stays blocked until the wedged camera handles the unblock pulse, which it never does.
At fault: MsgReply missing in camera (camera_handle hangs); synchronous MsgSend to camera in display's main thread
Fix: Take the camera MsgSend out of display's drawing loop. A dedicated camera_reader thread does the MsgSend and stores the last frame; the loop only reads that frame under the mutex and shows 'camera stale' when it is old. Make that the unconditional behaviour, not something enabled by COCKPIT_FIXED. Separately fix camera's grabber so it always replies, or calls MsgError on failure.
Proposed patch
--- a/cockpit.c
+++ b/cockpit.c
@@ static int run_display(int ticks) {
- if (fixed) {
- pthread_t reader;
- pthread_create(&reader, NULL, camera_reader, &camera);
- }
+ pthread_t reader;
+ pthread_create(&reader, NULL, camera_reader, &camera);
+ pthread_detach(reader);
@@
- if (fixed) {
- pthread_mutex_lock(&frame_lock);
- if (++frame_age > 3) {
- snprintf(frame.text, sizeof(frame.text), "camera stale");
- } else {
- snprintf(frame.text, sizeof(frame.text), "%s", last_frame);
- }
- pthread_mutex_unlock(&frame_lock);
- } else if (ask(camera, MSG_FRAME, seq, "frame?", &frame) == -1) {
- fprintf(stderr, "display: camera failed: %s, giving up\n", strerror(errno));
- return 1;
- }
+ pthread_mutex_lock(&frame_lock);
+ if (++frame_age > 3) {
+ snprintf(frame.text, sizeof(frame.text), "camera stale");
+ } else {
+ snprintf(frame.text, sizeof(frame.text), "%s", last_frame);
+ }
+ pthread_mutex_unlock(&frame_lock);
Cites events 58019, 58029, 58039, 58318, 58328, -58319 · Docs: MsgSend MsgReply MsgReceive name_attach TimerTimeout Synchronous message passing