FAIL

cockpit architecture check

5 errors, 0 warnings. Trace demo/cockpit/traces/prio-inversion.kev: QNX 8.0.0 on x86pc, 2 CPUs, 6.01 s, 51,196 events.

Architecture

display to camera: 31 messagesdisplay to logger: 31 messagesdisplay to nav: 31 messagesnav to gps: 31 messagesnav to logger: 31 messagesgpsnavcameraloggerdisplay
  • allowed, used
  • allowed, not seen
  • not in contract
  • deadlock loop
  • path in a finding

Findings

  1. display waits on logger behind lower priority work priority-inversionstart here
  2. nav waits on logger behind lower priority work priority-inversion
  3. logger answers display too slowly reply-budget
  4. nav answers display too slowly reply-budget
  5. logger answers nav too slowly reply-budget
errorpriority-inversion

display waits on logger behind lower priority work

17 messages from display to logger sat behind threads with a priority between the two: its channel was created with _NTO_CHF_FIXED_PRIORITY, so logger keeps priority 10 instead of taking display's 30. Lost time per message up to 46.4 ms (limit 2 ms), 491.5 ms in total. CPU time taken meanwhile: media (priority 20) 917.2 ms, io-usb-otg (priority 21) 35.8 ms, procnto-smp-instr (priority 21) 6.8 ms.

Evidence: trace events 105783, 105809, 105820, 104932, 109142

Show the 43 trace events behind this
EventTimeWhat happened
-20327001 (note)1131.024 mslogger ChannelCreate chid 1 flags 0x4b (_NTO_CHF_FIXED_PRIORITY | _NTO_CHF_UNBLOCK | _NTO_CHF_DISCONNECT | _NTO_CHF_COID_DISCONNECT)
1018524906.254 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1025944951.577 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1026154953.599 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
1030545007.965 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1030685008.008 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1031765055.623 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
1036035112.444 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1038115162.998 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1038325165.025 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
1052965221.194 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1053145221.248 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1053355223.256 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
1057835277.945 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1057965277.974 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1058095278.586 mslogger[tid 1] is READY at priority 10 on CPU 0
1058205278.613 msmedia[tid 1] is RUNNING at priority 20 on CPU 0
1049325279.577 msmedia[tid 2] is RUNNING at priority 20 on CPU 1
1091425325.043 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
1095735379.197 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1095875379.239 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1096235381.250 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
1110095482.636 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1110265482.678 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1110835484.795 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
1115495540.860 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1115635540.900 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1115845542.909 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
1126745644.575 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1126885644.613 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1127095646.622 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
1135605702.238 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1135745702.291 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1152105747.195 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
1156185800.670 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1156325800.708 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1156425802.720 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
1193095859.152 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1193235859.198 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1190775903.158 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
1202455957.719 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1202595957.762 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1202695959.770 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
Why, according to claude-opus-5-5 confidence high

logger's channel is created with _NTO_CHF_FIXED_PRIORITY (event -20327001, flags 0x4b), so when logger receives display's MsgSend it stays at priority 10 instead of inheriting display's 30. media's priority-20 busy loops on both CPUs then preempt logger mid-request (105809/105820/104932), and display (priority 30) sits REPLY-blocked for up to ~47 ms per log line, e.g. receive at 5277.974 ms, reply at 5325.043 ms, against a 2 ms limit.

At fault: ChannelCreate with _NTO_CHF_FIXED_PRIORITY in logger (run_logger); display's MsgSend to logger is REPLY-blocked behind media

Fix: Drop _NTO_CHF_FIXED_PRIORITY from the logger channel so logger's receiving thread runs at the sender's priority (display 30, nav's priority for nav). If the intent was to keep logging from competing with real work, the synchronous MsgSend contract is the wrong tool. Instead make log_line asynchronous: send a pulse, or queue to a local thread in display.

Proposed patch
--- a/cockpit.c
+++ b/cockpit.c
@@ static void run_logger(void) {
-    /* The regression: someone made the logger fixed priority so that logging
-     * never competes with real work. Clients still wait for its reply. */
-    unsigned flags = _NTO_CHF_DISCONNECT | _NTO_CHF_COID_DISCONNECT | _NTO_CHF_UNBLOCK;
-    if (!fixed) {
-        flags |= _NTO_CHF_FIXED_PRIORITY;
-    }
+    /* Clients block in MsgSend until we reply, so we must inherit their
+     * priority: no _NTO_CHF_FIXED_PRIORITY on this channel. */
+    unsigned flags = _NTO_CHF_DISCONNECT | _NTO_CHF_COID_DISCONNECT | _NTO_CHF_UNBLOCK;
     int chid = ChannelCreate(flags);

Cites events -20327001, 105783, 105796, 105809, 105820, 104932, 109142, 103068, 103176, 113574, 115210, 119323, 119077 · Docs: ChannelCreate MsgSend MsgReceive MsgReply Synchronous message passing

errorpriority-inversion

nav waits on logger behind lower priority work

17 messages from nav to logger sat behind threads with a priority between the two: its channel was created with _NTO_CHF_FIXED_PRIORITY, so logger keeps priority 10 instead of taking nav's 30. Lost time per message up to 48.5 ms (limit 2 ms), 185.0 ms in total. CPU time taken meanwhile: media (priority 20) 355.4 ms, io-sock (priority 21) 1.0 ms, devb-eide (priority 21) 0.3 ms.

Evidence: trace events 109849, 109875, 109886, 110750, 110899

Show the 44 trace events behind this
EventTimeWhat happened
-20327001 (note)1131.024 mslogger ChannelCreate chid 1 flags 0x4b (_NTO_CHF_FIXED_PRIORITY | _NTO_CHF_UNBLOCK | _NTO_CHF_DISCONNECT | _NTO_CHF_COID_DISCONNECT)
-20331001 (note)1158.944 msnav ChannelCreate chid 1 flags 0x4a (_NTO_CHF_UNBLOCK | _NTO_CHF_DISCONNECT | _NTO_CHF_COID_DISCONNECT)
1017024903.272 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1017154903.301 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1017254905.319 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1028655004.990 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1028785005.020 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1029035007.038 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1034645107.688 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1034765107.720 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1034865109.740 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1051095218.545 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1051355218.635 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1051865220.654 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1056285275.177 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1056415275.205 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1056625277.221 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1094205376.345 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1094465376.410 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1094675378.509 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1098495432.770 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1098625432.801 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1098755433.586 mslogger[tid 1] is READY at priority 10 on CPU 1
1098865433.623 msmedia[tid 2] is RUNNING at priority 20 on CPU 1
1107505437.066 msmedia[tid 1] is RUNNING at priority 20 on CPU 0
1108995482.145 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1113605537.919 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1114135538.227 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1114345540.248 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1118835593.935 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1118965593.974 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1125685643.996 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1130005698.865 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1130135698.894 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1134505701.571 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1154795798.036 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1154915798.062 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1155125800.079 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1160215854.159 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1160345854.194 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1192035858.714 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1200835954.746 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1201095954.979 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1201305957.009 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
Why, according to claude-opus-5-5 confidence high

logger's channel is created with _NTO_CHF_FIXED_PRIORITY (event -20327001, flags 0x4b), so when nav (priority 30, running at display's inherited priority inside nav_handle) does MsgSend to logger, logger's receiving thread stays at priority 10 instead of inheriting 30. media at priority 20 keeps both CPUs busy, preempts logger mid-request (109875 READY at 10, 109886/110750 media RUNNING at 20), and nav stays REPLY-blocked: e.g. 5432.8 to 5482.1 ms (109849 to 110899), and 5594.0 to 5644.0 ms (111883 to 112568), versus the normal ~2 ms.

At fault: ChannelCreate with _NTO_CHF_FIXED_PRIORITY in logger (run_logger), served to nav's MsgSend

Fix: Drop _NTO_CHF_FIXED_PRIORITY from the logger channel so logger inherits the priority of the waiting client (what COCKPIT_FIXED=1 already does), or simply use name_attach() like the other servers. If logging must really never compete with real work, change the contract instead: make log_line asynchronous (MsgSendPulse or a local queue drained by a low-priority thread) so nav never REPLY-blocks on a fixed-low-priority server.

Proposed patch
--- a/cockpit.c
+++ b/cockpit.c
@@ static void run_logger(void) {
-    /* The regression: someone made the logger fixed priority so that logging
-     * never competes with real work. Clients still wait for its reply. */
-    unsigned flags = _NTO_CHF_DISCONNECT | _NTO_CHF_COID_DISCONNECT | _NTO_CHF_UNBLOCK;
-    if (!fixed) {
-        flags |= _NTO_CHF_FIXED_PRIORITY;
-    }
+    /* No _NTO_CHF_FIXED_PRIORITY: clients block on our reply, so we must run
+     * at their priority (priority inheritance) while serving them. */
+    unsigned flags = _NTO_CHF_DISCONNECT | _NTO_CHF_COID_DISCONNECT | _NTO_CHF_UNBLOCK;
     int chid = ChannelCreate(flags);

Cites events -20327001, 109849, 109862, 109875, 109886, 110750, 110899, 111883, 111896, 112568 · Docs: ChannelCreate MsgSend MsgReceive MsgReply Synchronous message passing

errorreply-budget

logger answers display too slowly

15 of 31 replies from logger to display took longer than 20 ms; worst 52.6 ms. Times come from the trace clock; under emulation they are inflated, so budgets should be generous.

Where the slowest reply (52.6 ms) went
  • 50.4 ms server preempted · media at priority 20 50.3 ms, io-sock at priority 21 0.7 ms, procnto-smp-instr at priority 254 0.2 ms
  • 2.0 ms server running
  • 0.2 ms queued, server busy elsewhere

Evidence: trace events 103603, 85020, 92779

Show the 45 trace events behind this
EventTimeWhat happened
850203374.332 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
841853375.541 mslogger[tid 1] MsgReceive got the message from display[tid 1]
851963423.677 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
927794268.457 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
970244269.631 mslogger[tid 1] MsgReceive got the message from display[tid 1]
963024316.976 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
1018524906.254 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1025944951.577 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1026154953.599 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
1030545007.965 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1030685008.008 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1031765055.623 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
1036035112.444 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1038115162.998 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1038325165.025 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
1052965221.194 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1053145221.248 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1053355223.256 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
1057835277.945 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1057965277.974 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1091425325.043 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
1095735379.197 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1095875379.239 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1096235381.250 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
1110095482.636 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1110265482.678 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1110835484.795 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
1115495540.860 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1115635540.900 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1115845542.909 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
1126745644.575 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1126885644.613 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1127095646.622 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
1135605702.238 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1135745702.291 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1152105747.195 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
1156185800.670 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1156325800.708 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1156425802.720 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
1193095859.152 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1193235859.198 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1190775903.158 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
1202455957.719 msdisplay[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10'
1202595957.762 mslogger[tid 1] MsgReceive got the message from display[tid 1]
1202695959.770 mslogger[tid 1] MsgReply to display[tid 1] reply '\x04\x10'
Why, according to claude-opus-5-5 confidence high

The logger's channel is created with ChannelCreate(_NTO_CHF_FIXED_PRIORITY) under the prio-inversion build. Its receive thread therefore keeps its own low priority instead of inheriting display's priority 30. The priority-20 media job keeps every CPU busy for 45 ms out of each 50 ms, so it preempts logger whenever display is SEND- or REPLY-blocked on it. Logger only gets the CPU in media's 5 ms sleep window, which turns a 2 ms busy_ms() handler into a ~47-53 ms wait. The waits land either before MsgReceive (e.g. 101852→102594: 45 ms SEND-blocked) or between MsgReceive and MsgReply (e.g. 103068→103176: 47.6 ms). Fast cycles of about 2 ms happen when the request lands in media's idle gap.

At fault: ChannelCreate with _NTO_CHF_FIXED_PRIORITY in logger (run_logger), causing slow MsgReceive/MsgReply to display's MsgSend

Fix: Drop _NTO_CHF_FIXED_PRIORITY from the logger channel so logger's receiving thread inherits display's priority 30 and outranks media (priority 20) while serving it. This is what COCKPIT_FIXED=1 already does; make it unconditional, or simply serve the logger via name_attach() like the other servers. If logging truly must never compete with real work, change the contract instead: make log_line asynchronous (MsgSendPulse or a queue drained by a separate display thread) so display never waits on logger.

Proposed patch
--- a/cockpit.c
+++ b/cockpit.c
@@ static void run_logger(void) {
-    unsigned flags = _NTO_CHF_DISCONNECT | _NTO_CHF_COID_DISCONNECT | _NTO_CHF_UNBLOCK;
-    if (!fixed) {
-        flags |= _NTO_CHF_FIXED_PRIORITY;
-    }
+    /* No _NTO_CHF_FIXED_PRIORITY: logger must inherit the priority of the
+     * client waiting on it, or mid-priority work (media) starves the reply. */
+    unsigned flags = _NTO_CHF_DISCONNECT | _NTO_CHF_COID_DISCONNECT | _NTO_CHF_UNBLOCK;
     int chid = ChannelCreate(flags);

Cites events 85020, 84185, 85196, 101852, 102594, 102615, 103054, 103068, 103176, 103603, 103811, 103832, 105783, 105796, 109142, 105296, 105314, 105335 · Docs: ChannelCreate MsgReceive MsgReply MsgSend Synchronous message passing name_attach

errorreply-budget

nav answers display too slowly

2 of 31 replies from nav to display took longer than 50 ms; worst 51.6 ms. Times come from the trace clock; under emulation they are inflated, so budgets should be generous.

Where the slowest reply (51.6 ms) went
  • 51.0 ms server waiting on logger · there: server preempted 45.2 ms (media at priority 20)
  • 0.3 ms server waiting on gps · there: server running 0.3 ms
  • 0.1 ms server running
  • 0.1 ms queued, server busy elsewhere

Evidence: trace events 101163, 111825

Show the 42 trace events behind this
EventTimeWhat happened
1011634797.219 msdisplay[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10'
1011734797.291 msnav[tid 1] MsgReceive got the message from display[tid 1]
1013654848.790 msnav[tid 1] MsgReply to display[tid 1] reply '\x02\x10'
1016444902.979 msdisplay[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10'
1016544903.058 msnav[tid 1] MsgReceive got the message from display[tid 1]
1017414905.382 msnav[tid 1] MsgReply to display[tid 1] reply '\x02\x10'
1028075004.720 msdisplay[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10'
1028175004.790 msnav[tid 1] MsgReceive got the message from display[tid 1]
1029195007.103 msnav[tid 1] MsgReply to display[tid 1] reply '\x02\x10'
1034075107.409 msdisplay[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10'
1034175107.476 msnav[tid 1] MsgReceive got the message from display[tid 1]
1035125111.790 msnav[tid 1] MsgReply to display[tid 1] reply '\x02\x10'
1050525218.349 msdisplay[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10'
1050625218.387 msnav[tid 1] MsgReceive got the message from display[tid 1]
1052025220.700 msnav[tid 1] MsgReply to display[tid 1] reply '\x02\x10'
1055705274.922 msdisplay[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10'
1055805274.979 msnav[tid 1] MsgReceive got the message from display[tid 1]
1056785277.267 msnav[tid 1] MsgReply to display[tid 1] reply '\x02\x10'
1093635376.125 msdisplay[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10'
1093735376.174 msnav[tid 1] MsgReceive got the message from display[tid 1]
1094825378.621 msnav[tid 1] MsgReply to display[tid 1] reply '\x02\x10'
1097915432.489 msdisplay[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10'
1098015432.569 msnav[tid 1] MsgReceive got the message from display[tid 1]
1109145482.204 msnav[tid 1] MsgReply to display[tid 1] reply '\x02\x10'
1113035537.648 msdisplay[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10'
1113135537.725 msnav[tid 1] MsgReceive got the message from display[tid 1]
1114555540.318 msnav[tid 1] MsgReply to display[tid 1] reply '\x02\x10'
1118255593.677 msdisplay[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10'
1118355593.757 msnav[tid 1] MsgReceive got the message from display[tid 1]
1125835644.045 msnav[tid 1] MsgReply to display[tid 1] reply '\x02\x10'
1129425698.608 msdisplay[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10'
1129525698.676 msnav[tid 1] MsgReceive got the message from display[tid 1]
1134655701.636 msnav[tid 1] MsgReply to display[tid 1] reply '\x02\x10'
1154225797.800 msdisplay[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10'
1154325797.854 msnav[tid 1] MsgReceive got the message from display[tid 1]
1155275800.119 msnav[tid 1] MsgReply to display[tid 1] reply '\x02\x10'
1159635853.496 msdisplay[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10'
1159735853.544 msnav[tid 1] MsgReceive got the message from display[tid 1]
1192185858.757 msnav[tid 1] MsgReply to display[tid 1] reply '\x02\x10'
1200265954.555 msdisplay[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10'
1200365954.597 msnav[tid 1] MsgReceive got the message from display[tid 1]
1201465957.098 msnav[tid 1] MsgReply to display[tid 1] reply '\x02\x10'
Why, according to claude-opus-5-5 confidence medium

nav's reply path is not just nav: before MsgReply it makes two nested synchronous sends, gps_position() to gps and log_line() to logger, so display stays REPLY-blocked for the whole chain. Most round trips take about 2.5 ms (e.g. 101644→101741), but three take about 50 ms. Two exceed the 50 ms budget: 101163→101365 at 51.6 ms and 111825→112583 at 50.4 ms. A third, 109791→110914, takes 49.7 ms. The trace shown does not include nav's nested sends, so it cannot say whether gps or logger caused the stall. This is not the 200 ms slow-reply fault; it is occasional ~50 ms stalls inside nav's downstream calls, inflated by emulation, landing just over a tight budget.

At fault: MsgReply delayed in nav (nested MsgSend to gps and logger before replying to display)

Fix: Take the logger out of display's critical path: nav should MsgReply to display as soon as it has the gps position, then send the log line. With serve_channel, the handler signals this by returning nonzero after replying itself. If the remaining outliers come from gps or emulator jitter, this is a contract question: the budget of 50 ms sits right on the observed tail under QEMU, and the blueprint itself says budgets should be generous. Raise reply_budget_ms.nav (for example to 100) for emulated runs, or measure on hardware.

Proposed patch
--- a/cockpit.c
+++ b/cockpit.c
@@ static int nav_handle(int rcvid, cockpit_msg_t *msg, cockpit_msg_t *reply) {
-    (void)rcvid;
     if (msg->type == MSG_HINT) {
@@
     snprintf(reply->text, sizeof(reply->text), "route via %.36s", pos.text);
-    log_line(nav_logger, msg->seq, "route sent");
-    return 0;
+    /* Reply first: display must not wait for the logger. */
+    MsgReply(rcvid, EOK, reply, sizeof(*reply));
+    log_line(nav_logger, msg->seq, "route sent");
+    return 1;   /* already replied */
 }
--- a/blueprint.yaml
+++ b/blueprint.yaml
 reply_budget_ms:
-  nav: 50
+  nav: 100   # nav makes nested sends to gps; emulation inflates times

Cites events 101163, 101173, 101365, 111825, 111835, 112583, 109791, 110914, 101644, 101741 · Docs: MsgSend MsgReceive MsgReply Synchronous message passing

errorreply-budget

logger answers nav too slowly

7 of 31 replies from logger to nav took longer than 20 ms; worst 51.0 ms. Times come from the trace clock; under emulation they are inflated, so budgets should be generous.

Where the slowest reply (51.0 ms) went
  • 45.2 ms server preempted · media at priority 20 45.2 ms, io-sock at priority 21 0.4 ms, procnto-smp-instr at priority 254 0.3 ms
  • 5.8 ms server running

Evidence: trace events 101221, 111883, 109849

Show the 45 trace events behind this
EventTimeWhat happened
1006754740.952 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
997064741.187 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
997164743.215 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1012214797.740 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1012344797.768 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1013504848.746 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1017024903.272 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1017154903.301 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1017254905.319 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1028655004.990 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1028785005.020 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1029035007.038 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1034645107.688 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1034765107.720 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1034865109.740 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1051095218.545 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1051355218.635 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1051865220.654 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1056285275.177 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1056415275.205 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1056625277.221 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1094205376.345 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1094465376.410 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1094675378.509 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1098495432.770 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1098625432.801 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1108995482.145 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1113605537.919 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1114135538.227 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1114345540.248 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1118835593.935 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1118965593.974 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1125685643.996 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1130005698.865 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1130135698.894 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1134505701.571 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1154795798.036 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1154915798.062 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1155125800.079 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1160215854.159 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1160345854.194 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1192035858.714 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
1200835954.746 msnav[tid 1] MsgSend to logger (coid 0x40000002, chid 1) message '\x04\x10'
1201095954.979 mslogger[tid 1] MsgReceive got the message from nav[tid 1]
1201305957.009 mslogger[tid 1] MsgReply to nav[tid 1] reply '\x04\x10'
Why, according to claude-opus-5-5 confidence high

logger serves a channel made with ChannelCreate(_NTO_CHF_FIXED_PRIORITY) under the prio-inversion build, so its receiving thread does not inherit the priority of the caller. nav is handling display's request at display's inherited priority 30, but logger keeps its own low priority. The priority-20 media job, busy 45 ms out of every 50, preempts logger in the middle of its 2 ms busy_ms() work. Normal replies take about 2 ms; the slow ones take about 50 ms, one media burst. Example: received at 4797.768, replied at 4848.746, a 51 ms wait.

At fault: ChannelCreate with _NTO_CHF_FIXED_PRIORITY in logger (run_logger), which delays MsgReply to nav

Fix: Remove _NTO_CHF_FIXED_PRIORITY from the logger channel, so logger's receiving thread runs at the priority of the client it is serving and media cannot preempt it while nav waits. Currently this only happens with COCKPIT_FIXED=1; make it unconditional. Alternatively, go back to serve("cockpit-logger") / name_attach(), which does not set the flag. If logging really must never compete with real work, change the contract instead: make log_line asynchronous (send a pulse or queue to a logger thread) so no client waits on a low-priority reply.

Proposed patch
--- a/cockpit.c
+++ b/cockpit.c
@@ static void run_logger(void) {
     unsigned flags = _NTO_CHF_DISCONNECT | _NTO_CHF_COID_DISCONNECT | _NTO_CHF_UNBLOCK;
-    if (!fixed) {
-        flags |= _NTO_CHF_FIXED_PRIORITY;
-    }
+    /* No _NTO_CHF_FIXED_PRIORITY: logger must inherit its client's priority,
+     * otherwise media (prio 20) preempts it while nav/display wait for MsgReply. */
     int chid = ChannelCreate(flags);

Cites events 99706, 99716, 101221, 101234, 101350, 109849, 109862, 110899, 111883, 111896, 112568 · Docs: ChannelCreate MsgReceive MsgReply MsgSend Synchronous message passing

Message traffic

ClientServerIn contractMessagesRepliedErrorsUnansweredMedian replyWorst replyBudget
displaycamerayes3131000.08 ms0.87 ms20 ms
displayloggeryes3131005.96 ms52.58 ms20 ms
displaynavyes3131003.03 ms51.57 ms50 ms
navgpsyes3131000.08 ms1.35 ms20 ms
navloggeryes3131002.64 ms51.01 ms20 ms