isolationcamera died and took display down with it
The contract says display must keep working when camera fails (at least 3 completed messages). camera died; display died 33 ms later.
Evidence: trace events 91117, 95162
Show the 38 trace events behind this
| Event | Time | What happened |
|---|---|---|
| 66641 | 2621.162 ms | display[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10' |
| 66651 | 2621.219 ms | camera[tid 1] MsgReceive got the message from display[tid 1] |
| 66676 | 2622.957 ms | camera[tid 1] MsgReply to display[tid 1] reply '\x03\x10' |
| 67114 | 2679.347 ms | display[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10' |
| 67124 | 2679.407 ms | camera[tid 1] MsgReceive got the message from display[tid 1] |
| 67134 | 2679.489 ms | camera[tid 1] MsgReply to display[tid 1] reply '\x03\x10' |
| 67400 | 2738.653 ms | display[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10' |
| 67410 | 2738.715 ms | camera[tid 1] MsgReceive got the message from display[tid 1] |
| 67420 | 2738.950 ms | camera[tid 1] MsgReply to display[tid 1] reply '\x03\x10' |
| 67935 | 2794.192 ms | display[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10' |
| 67945 | 2794.236 ms | camera[tid 1] MsgReceive got the message from display[tid 1] |
| 67955 | 2794.325 ms | camera[tid 1] MsgReply to display[tid 1] reply '\x03\x10' |
| 68251 | 2849.696 ms | display[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10' |
| 68261 | 2849.762 ms | camera[tid 1] MsgReceive got the message from display[tid 1] |
| 68271 | 2849.870 ms | camera[tid 1] MsgReply to display[tid 1] reply '\x03\x10' |
| 69633 | 2904.072 ms | display[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10' |
| 69654 | 2904.358 ms | camera[tid 1] MsgReceive got the message from display[tid 1] |
| 69664 | 2904.464 ms | camera[tid 1] MsgReply to display[tid 1] reply '\x03\x10' |
| 70002 | 2959.562 ms | display[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10' |
| 70012 | 2959.620 ms | camera[tid 1] MsgReceive got the message from display[tid 1] |
| 70022 | 2959.697 ms | camera[tid 1] MsgReply to display[tid 1] reply '\x03\x10' |
| 70410 | 3014.937 ms | display[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10' |
| 70420 | 3014.986 ms | camera[tid 1] MsgReceive got the message from display[tid 1] |
| 70430 | 3015.075 ms | camera[tid 1] MsgReply to display[tid 1] reply '\x03\x10' |
| 69484 | 3070.275 ms | display[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10' |
| 69494 | 3070.339 ms | camera[tid 1] MsgReceive got the message from display[tid 1] |
| 69504 | 3070.430 ms | camera[tid 1] MsgReply to display[tid 1] reply '\x03\x10' |
| 72919 | 3124.610 ms | display[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10' |
| 72929 | 3124.667 ms | camera[tid 1] MsgReceive got the message from display[tid 1] |
| 72939 | 3124.756 ms | camera[tid 1] MsgReply to display[tid 1] reply '\x03\x10' |
| 75347 | 3187.457 ms | display[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10' |
| 75357 | 3187.521 ms | camera[tid 1] MsgReceive got the message from display[tid 1] |
| 75367 | 3187.614 ms | camera[tid 1] MsgReply to display[tid 1] reply '\x03\x10' |
| 83151 | 3240.667 ms | display[tid 1] MsgSend to camera (coid 0x40000002, chid 1) message '\x03\x10' |
| 83161 | 3240.688 ms | camera[tid 1] MsgReceive got the message from display[tid 1] |
| 83171 | 3240.732 ms | camera[tid 1] MsgReply to display[tid 1] reply '\x03\x10' |
| 91117 | 3274.794 ms | process camera (pid 1376282) destroyed |
| 95162 | 3304.806 ms | process display (pid 1400857) destroyed |
camera served every request normally (last reply at 3240.7 ms) and was then destroyed at 3274.8 ms. On display's next tick, MsgSend() on the dead camera connection returned -1. The camera-dies regression in run_display() treats that error as fatal ('camera failed ... giving up', return 1), so display exited 30 ms after camera instead of continuing to draw nav/logger frames.
At fault: MsgSend in display (error return from dead camera treated as fatal)
Fix: Do not exit when MsgSend to camera fails. Degrade to a 'no camera' frame and keep the nav/logger loop running. Better still, build with COCKPIT_FIXED=1 so the camera is read from its own camera_reader thread, which also protects display from a hung camera (a send timeout would not, because of _NTO_CHF_UNBLOCK on name_attach channels).
Proposed patch
--- a/cockpit.c
+++ b/cockpit.c
@@ run_display
- } else if (ask(camera, MSG_FRAME, seq, "frame?", &frame) == -1) {
- fprintf(stderr, "display: camera failed: %s, giving up\n", strerror(errno));
- return 1;
- }
+ } else if (camera == -1 || ask(camera, MSG_FRAME, seq, "frame?", &frame) == -1) {
+ if (camera != -1) {
+ fprintf(stderr, "display: camera failed: %s, continuing without it\n", strerror(errno));
+ ConnectDetach(camera);
+ camera = -1;
+ }
+ snprintf(frame.text, sizeof(frame.text), "no camera");
+ }
Cites events 83151, 83161, 83171, 91117, 95162 · Docs: MsgSend Synchronous message passing name_open name_attach