FAIL

cockpit architecture check

1 error, 1 warning. Trace demo/cockpit/traces/camera-hang.kev: QNX 8.0.0 on x86pc, 2 CPUs, 6.00 s, 34,857 events.

Architecture

display to camera: 6 messagesdisplay to logger: 5 messagesdisplay to nav: 6 messagesnav to gps: 6 messagesnav to logger: 6 messagesgpsnavcameraloggerdisplay
  • allowed, used
  • allowed, not seen
  • not in contract
  • deadlock loop
  • path in a finding

Findings

  1. camera stopped answering and took display down with it isolationstart here
  2. camera took a message from display and never replied stuck-reply
errorisolation

camera stopped answering and took display down with it

The contract says display must keep working when camera fails (at least 3 completed messages). camera stopped answering; display completed only 0 messages afterwards and its thread 1 is still REPLY-blocked on camera.

Evidence: trace events 58318

Show the 18 trace events behind this
EventTimeWhat happened
550762234.580 msdisplay[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10'
550862234.605 mscamera[tid 1] MsgReceive got the message from display[tid 1]
551112235.704 mscamera[tid 1] MsgReply to display[tid 1] reply '\x03\x10'
554922292.234 msdisplay[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10'
555022292.256 mscamera[tid 1] MsgReceive got the message from display[tid 1]
555122292.303 mscamera[tid 1] MsgReply to display[tid 1] reply '\x03\x10'
558402348.586 msdisplay[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10'
558502348.644 mscamera[tid 1] MsgReceive got the message from display[tid 1]
558602348.809 mscamera[tid 1] MsgReply to display[tid 1] reply '\x03\x10'
562492406.380 msdisplay[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10'
562592406.451 mscamera[tid 1] MsgReceive got the message from display[tid 1]
562692406.899 mscamera[tid 1] MsgReply to display[tid 1] reply '\x03\x10'
580192467.649 msdisplay[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10'
580292467.705 mscamera[tid 1] MsgReceive got the message from display[tid 1]
580392467.792 mscamera[tid 1] MsgReply to display[tid 1] reply '\x03\x10'
583182521.298 msdisplay[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10'
583282521.347 mscamera[tid 1] MsgReceive got the message from display[tid 1]
-58319 (note)-at trace end display[tid 1] is still REPLY-blocked on camera (no reply to event 58318)
Why, according to claude-opus-5-5 confidence high

camera answered five frame requests, then on the sixth (camera_frames > 5, COCKPIT_FAULT=camera-hang) camera_handle wedged in pause() after MsgReceive and never called MsgReply. display's main loop calls MsgSend to camera inline with no isolation, so its only thread stays REPLY-blocked forever and never reaches nav or logger again. A send timeout would not help: the name_attach() channel has _NTO_CHF_UNBLOCK, so the client stays blocked until the wedged camera handles the unblock pulse, which it never does.

At fault: MsgReply missing in camera (camera_handle hangs); synchronous MsgSend to camera in display's main thread

Fix: Take the camera MsgSend out of display's drawing loop. A dedicated camera_reader thread does the MsgSend and stores the last frame; the loop only reads that frame under the mutex and shows 'camera stale' when it is old. Make that the unconditional behaviour, not something enabled by COCKPIT_FIXED. Separately fix camera's grabber so it always replies, or calls MsgError on failure.

Proposed patch
--- a/cockpit.c
+++ b/cockpit.c
@@ static int run_display(int ticks) {
-    if (fixed) {
-        pthread_t reader;
-        pthread_create(&reader, NULL, camera_reader, &camera);
-    }
+    pthread_t reader;
+    pthread_create(&reader, NULL, camera_reader, &camera);
+    pthread_detach(reader);
@@
-        if (fixed) {
-            pthread_mutex_lock(&frame_lock);
-            if (++frame_age > 3) {
-                snprintf(frame.text, sizeof(frame.text), "camera stale");
-            } else {
-                snprintf(frame.text, sizeof(frame.text), "%s", last_frame);
-            }
-            pthread_mutex_unlock(&frame_lock);
-        } else if (ask(camera, MSG_FRAME, seq, "frame?", &frame) == -1) {
-            fprintf(stderr, "display: camera failed: %s, giving up\n", strerror(errno));
-            return 1;
-        }
+        pthread_mutex_lock(&frame_lock);
+        if (++frame_age > 3) {
+            snprintf(frame.text, sizeof(frame.text), "camera stale");
+        } else {
+            snprintf(frame.text, sizeof(frame.text), "%s", last_frame);
+        }
+        pthread_mutex_unlock(&frame_lock);

Cites events 58019, 58029, 58039, 58318, 58328, -58319 · Docs: MsgSend MsgReply MsgReceive name_attach TimerTimeout Synchronous message passing

warningstuck-reply

camera took a message from display and never replied

display thread 1 has been REPLY-blocked on camera for 3479 ms (budget 20.0 ms). Server thread 1, which holds the message, is in SIGWAITINFO and not blocked on another server. The contract plans for this server failing, see the isolation check.

Evidence: trace events 58318

Show the 18 trace events behind this
EventTimeWhat happened
550762234.580 msdisplay[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10'
550862234.605 mscamera[tid 1] MsgReceive got the message from display[tid 1]
551112235.704 mscamera[tid 1] MsgReply to display[tid 1] reply '\x03\x10'
554922292.234 msdisplay[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10'
555022292.256 mscamera[tid 1] MsgReceive got the message from display[tid 1]
555122292.303 mscamera[tid 1] MsgReply to display[tid 1] reply '\x03\x10'
558402348.586 msdisplay[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10'
558502348.644 mscamera[tid 1] MsgReceive got the message from display[tid 1]
558602348.809 mscamera[tid 1] MsgReply to display[tid 1] reply '\x03\x10'
562492406.380 msdisplay[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10'
562592406.451 mscamera[tid 1] MsgReceive got the message from display[tid 1]
562692406.899 mscamera[tid 1] MsgReply to display[tid 1] reply '\x03\x10'
580192467.649 msdisplay[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10'
580292467.705 mscamera[tid 1] MsgReceive got the message from display[tid 1]
580392467.792 mscamera[tid 1] MsgReply to display[tid 1] reply '\x03\x10'
583182521.298 msdisplay[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10'
583282521.347 mscamera[tid 1] MsgReceive got the message from display[tid 1]
-58319 (note)-at trace end display[tid 1] is still REPLY-blocked on camera (no reply to event 58318)

Message traffic

ClientServerIn contractMessagesRepliedErrorsUnansweredMedian replyWorst replyBudget
displaycamerayes65010.22 ms1.12 ms20 ms
displayloggeryes55000.10 ms0.22 ms20 ms
displaynavyes66000.87 ms2.27 ms50 ms
navgpsyes66000.19 ms1.20 ms20 ms
navloggeryes66000.10 ms0.11 ms20 ms