reply-budgetnav answers display too slowly
14 of 14 replies from nav to display took longer than 50 ms; worst 222.3 ms. Times come from the trace clock; under emulation they are inflated, so budgets should be generous.
- 204.9 ms server sleeping (NANOSLEEP)
- 9.0 ms server running
- 4.4 ms server waiting on logger
- 3.6 ms server preempted · io-sock at priority 21 1.5 ms, logger at priority 10 1.5 ms, procnto-smp-instr at priority 254 0.4 ms
- 0.2 ms queued, server busy elsewhere
- 0.2 ms server waiting on gps · there: queued, server busy elsewhere 0.1 ms
Evidence: trace events 64268, 70867, 56202
Show the 45 trace events behind this
| Event | Time | What happened |
|---|---|---|
| 56202 | 2216.631 ms | display[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10' |
| 56212 | 2216.662 ms | nav[tid 1] MsgReceive got the message from display[tid 1] |
| 56441 | 2427.024 ms | nav[tid 1] MsgReply to display[tid 1] reply '\x02\x10' |
| 57745 | 2489.414 ms | display[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10' |
| 57755 | 2489.570 ms | nav[tid 1] MsgReceive got the message from display[tid 1] |
| 61676 | 2692.877 ms | nav[tid 1] MsgReply to display[tid 1] reply '\x02\x10' |
| 61860 | 2748.070 ms | display[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10' |
| 61870 | 2748.364 ms | nav[tid 1] MsgReceive got the message from display[tid 1] |
| 62112 | 2951.696 ms | nav[tid 1] MsgReply to display[tid 1] reply '\x02\x10' |
| 62324 | 3005.559 ms | display[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10' |
| 62334 | 3005.708 ms | nav[tid 1] MsgReceive got the message from display[tid 1] |
| 62498 | 3212.505 ms | nav[tid 1] MsgReply to display[tid 1] reply '\x02\x10' |
| 63400 | 3268.385 ms | display[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10' |
| 63410 | 3268.537 ms | nav[tid 1] MsgReceive got the message from display[tid 1] |
| 64097 | 3472.329 ms | nav[tid 1] MsgReply to display[tid 1] reply '\x02\x10' |
| 64268 | 3528.287 ms | display[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10' |
| 64278 | 3528.528 ms | nav[tid 1] MsgReceive got the message from display[tid 1] |
| 68074 | 3750.628 ms | nav[tid 1] MsgReply to display[tid 1] reply '\x02\x10' |
| 69431 | 3805.171 ms | display[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10' |
| 69441 | 3805.437 ms | nav[tid 1] MsgReceive got the message from display[tid 1] |
| 69644 | 4011.757 ms | nav[tid 1] MsgReply to display[tid 1] reply '\x02\x10' |
| 70867 | 4069.427 ms | display[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10' |
| 70877 | 4070.092 ms | nav[tid 1] MsgReceive got the message from display[tid 1] |
| 71067 | 4280.587 ms | nav[tid 1] MsgReply to display[tid 1] reply '\x02\x10' |
| 71410 | 4338.465 ms | display[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10' |
| 71420 | 4338.661 ms | nav[tid 1] MsgReceive got the message from display[tid 1] |
| 71627 | 4541.237 ms | nav[tid 1] MsgReply to display[tid 1] reply '\x02\x10' |
| 75958 | 4597.397 ms | display[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10' |
| 75968 | 4597.552 ms | nav[tid 1] MsgReceive got the message from display[tid 1] |
| 76387 | 4799.321 ms | nav[tid 1] MsgReply to display[tid 1] reply '\x02\x10' |
| 76575 | 4858.809 ms | display[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10' |
| 76585 | 4858.950 ms | nav[tid 1] MsgReceive got the message from display[tid 1] |
| 77835 | 5063.109 ms | nav[tid 1] MsgReply to display[tid 1] reply '\x02\x10' |
| 78086 | 5121.419 ms | display[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10' |
| 78096 | 5121.570 ms | nav[tid 1] MsgReceive got the message from display[tid 1] |
| 78260 | 5323.408 ms | nav[tid 1] MsgReply to display[tid 1] reply '\x02\x10' |
| 78499 | 5375.990 ms | display[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10' |
| 78509 | 5376.158 ms | nav[tid 1] MsgReceive got the message from display[tid 1] |
| 78686 | 5578.767 ms | nav[tid 1] MsgReply to display[tid 1] reply '\x02\x10' |
| 79393 | 5632.179 ms | display[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10' |
| 79403 | 5632.321 ms | nav[tid 1] MsgReceive got the message from display[tid 1] |
| 83525 | 5838.224 ms | nav[tid 1] MsgReply to display[tid 1] reply '\x02\x10' |
| 83760 | 5891.507 ms | display[tid 1] MsgSend to nav (coid 0x40000001, chid 1) message '\x02\x10' |
| 83770 | 5891.687 ms | nav[tid 1] MsgReceive got the message from display[tid 1] |
| -83761 (note) | - | at trace end display[tid 1] is still REPLY-blocked on nav (no reply to event 83760) |
nav_handle() sleeps 200 ms (the COCKPIT_FAULT=slow-reply branch, sleep_ms(200)) between MsgReceive() and MsgReply(). Every round trip therefore takes about 203-222 ms against a 50 ms budget. Because MsgSend() is synchronous, display stays REPLY-blocked for the whole time and its frame loop stalls with it. Priority is not a factor: nav is sleeping, not being preempted.
At fault: MsgReply delayed in nav (nav_handle sleeps 200 ms before replying to display's MsgSend)
Fix: Remove the 200 ms sleep from nav's request path so nav replies as soon as it has the gps position. If nav really needs 200 ms of work per route, move that work out of the handler: compute routes in a background thread and reply with the latest one. Only raise reply_budget_ms.nav if the contract is wrong, and a 200 ms route lookup is not acceptable for a display ticking every 50 ms.
Proposed patch
--- a/cockpit.c
+++ b/cockpit.c
@@ static int nav_handle(int rcvid, cockpit_msg_t *msg, cockpit_msg_t *reply) {
cockpit_msg_t pos;
gps_position(nav_gps, msg->seq, "where?", &pos);
- if (has_fault("slow-reply")) {
- sleep_ms(200);
- }
snprintf(reply->text, sizeof(reply->text), "route via %.36s", pos.text);
log_line(nav_logger, msg->seq, "route sent");
return 0;
}Cites events 56202, 56212, 56441, 57745, 57755, 61676, 64268, 64278, 68074, 83760, 83770, -83761 · Docs: MsgSend MsgReceive MsgReply Synchronous message passing