priority-inversiondisplay waits on logger behind lower priority work
17 messages from display to logger sat behind threads with a priority between the two: its channel was created with _NTO_CHF_FIXED_PRIORITY, so logger keeps priority 10 instead of taking display's 30. Lost time per message up to 46.4 ms (limit 2 ms), 491.5 ms in total. CPU time taken meanwhile: media (priority 20) 917.2 ms, io-usb-otg (priority 21) 35.8 ms, procnto-smp-instr (priority 21) 6.8 ms.
Evidence: trace events 105783, 105809, 105820, 104932, 109142
Show the 43 trace events behind this
| Event | Time | What happened |
|---|---|---|
| -20327001 (note) | 1131.024 ms | logger ChannelCreate chid 1 flags 0x4b (_NTO_CHF_FIXED_PRIORITY | _NTO_CHF_UNBLOCK | _NTO_CHF_DISCONNECT | _NTO_CHF_COID_DISCONNECT) |
| 101852 | 4906.254 ms | display[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10' |
| 102594 | 4951.577 ms | logger[tid 1] MsgReceive got the message from display[tid 1] |
| 102615 | 4953.599 ms | logger[tid 1] MsgReply to display[tid 1] reply '\x04\x10' |
| 103054 | 5007.965 ms | display[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10' |
| 103068 | 5008.008 ms | logger[tid 1] MsgReceive got the message from display[tid 1] |
| 103176 | 5055.623 ms | logger[tid 1] MsgReply to display[tid 1] reply '\x04\x10' |
| 103603 | 5112.444 ms | display[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10' |
| 103811 | 5162.998 ms | logger[tid 1] MsgReceive got the message from display[tid 1] |
| 103832 | 5165.025 ms | logger[tid 1] MsgReply to display[tid 1] reply '\x04\x10' |
| 105296 | 5221.194 ms | display[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10' |
| 105314 | 5221.248 ms | logger[tid 1] MsgReceive got the message from display[tid 1] |
| 105335 | 5223.256 ms | logger[tid 1] MsgReply to display[tid 1] reply '\x04\x10' |
| 105783 | 5277.945 ms | display[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10' |
| 105796 | 5277.974 ms | logger[tid 1] MsgReceive got the message from display[tid 1] |
| 105809 | 5278.586 ms | logger[tid 1] is READY at priority 10 on CPU 0 |
| 105820 | 5278.613 ms | media[tid 1] is RUNNING at priority 20 on CPU 0 |
| 104932 | 5279.577 ms | media[tid 2] is RUNNING at priority 20 on CPU 1 |
| 109142 | 5325.043 ms | logger[tid 1] MsgReply to display[tid 1] reply '\x04\x10' |
| 109573 | 5379.197 ms | display[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10' |
| 109587 | 5379.239 ms | logger[tid 1] MsgReceive got the message from display[tid 1] |
| 109623 | 5381.250 ms | logger[tid 1] MsgReply to display[tid 1] reply '\x04\x10' |
| 111009 | 5482.636 ms | display[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10' |
| 111026 | 5482.678 ms | logger[tid 1] MsgReceive got the message from display[tid 1] |
| 111083 | 5484.795 ms | logger[tid 1] MsgReply to display[tid 1] reply '\x04\x10' |
| 111549 | 5540.860 ms | display[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10' |
| 111563 | 5540.900 ms | logger[tid 1] MsgReceive got the message from display[tid 1] |
| 111584 | 5542.909 ms | logger[tid 1] MsgReply to display[tid 1] reply '\x04\x10' |
| 112674 | 5644.575 ms | display[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10' |
| 112688 | 5644.613 ms | logger[tid 1] MsgReceive got the message from display[tid 1] |
| 112709 | 5646.622 ms | logger[tid 1] MsgReply to display[tid 1] reply '\x04\x10' |
| 113560 | 5702.238 ms | display[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10' |
| 113574 | 5702.291 ms | logger[tid 1] MsgReceive got the message from display[tid 1] |
| 115210 | 5747.195 ms | logger[tid 1] MsgReply to display[tid 1] reply '\x04\x10' |
| 115618 | 5800.670 ms | display[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10' |
| 115632 | 5800.708 ms | logger[tid 1] MsgReceive got the message from display[tid 1] |
| 115642 | 5802.720 ms | logger[tid 1] MsgReply to display[tid 1] reply '\x04\x10' |
| 119309 | 5859.152 ms | display[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10' |
| 119323 | 5859.198 ms | logger[tid 1] MsgReceive got the message from display[tid 1] |
| 119077 | 5903.158 ms | logger[tid 1] MsgReply to display[tid 1] reply '\x04\x10' |
| 120245 | 5957.719 ms | display[tid 1] MsgSend to logger (coid 0x40000003, chid 1) message '\x04\x10' |
| 120259 | 5957.762 ms | logger[tid 1] MsgReceive got the message from display[tid 1] |
| 120269 | 5959.770 ms | logger[tid 1] MsgReply to display[tid 1] reply '\x04\x10' |
logger's channel is created with _NTO_CHF_FIXED_PRIORITY (event -20327001, flags 0x4b), so when logger receives display's MsgSend it stays at priority 10 instead of inheriting display's 30. media's priority-20 busy loops on both CPUs then preempt logger mid-request (105809/105820/104932), and display (priority 30) sits REPLY-blocked for up to ~47 ms per log line, e.g. receive at 5277.974 ms, reply at 5325.043 ms, against a 2 ms limit.
At fault: ChannelCreate with _NTO_CHF_FIXED_PRIORITY in logger (run_logger); display's MsgSend to logger is REPLY-blocked behind media
Fix: Drop _NTO_CHF_FIXED_PRIORITY from the logger channel so logger's receiving thread runs at the sender's priority (display 30, nav's priority for nav). If the intent was to keep logging from competing with real work, the synchronous MsgSend contract is the wrong tool. Instead make log_line asynchronous: send a pulse, or queue to a local thread in display.
Proposed patch
--- a/cockpit.c
+++ b/cockpit.c
@@ static void run_logger(void) {
- /* The regression: someone made the logger fixed priority so that logging
- * never competes with real work. Clients still wait for its reply. */
- unsigned flags = _NTO_CHF_DISCONNECT | _NTO_CHF_COID_DISCONNECT | _NTO_CHF_UNBLOCK;
- if (!fixed) {
- flags |= _NTO_CHF_FIXED_PRIORITY;
- }
+ /* Clients block in MsgSend until we reply, so we must inherit their
+ * priority: no _NTO_CHF_FIXED_PRIORITY on this channel. */
+ unsigned flags = _NTO_CHF_DISCONNECT | _NTO_CHF_COID_DISCONNECT | _NTO_CHF_UNBLOCK;
int chid = ChannelCreate(flags);Cites events -20327001, 105783, 105796, 105809, 105820, 104932, 109142, 103068, 103176, 113574, 115210, 119323, 119077 · Docs: ChannelCreate MsgSend MsgReceive MsgReply Synchronous message passing