My patch series “N_TTY input path fixes”, which fixes lockless TTY input on weakly-ordered arches, is part of the 4.0-rc kernel. So last Friday a post to the linux-serial mailing list leading with $subject “[BUG] n_tty: 4.0-rc3+ …” got my attention. The report was from one of the Ubuntu Server devs
Just found the following oops during kernel booting when I
test the latest linus tree on one arm64 VM booted from uefi
Keywords: n_tty “latest linus tree” arm64. So I scroll down to the stack trace
Call trace: [<ffffffc000362490>] n_tty_receive_buf_common+0x60/0xa28 [<ffffffc000362e64>] n_tty_receive_buf2+0xc/0x18 [<ffffffc000365b64>] flush_to_ldisc+0xec/0x140 [<ffffffc0000b5228>] process_one_work+0x138/0x338 [<ffffffc0000b5970>] worker_thread+0x148/0x420 [<ffffffc0000ba8fc>] kthread+0xd4/0xf0
Uh-oh. Definitely related to my code.
My first thought was “I’ll boot an ARM64 VM and see if I can reproduce”. Except I don’t have an ARM64 VM setup and I quickly discovered my QEMU version was too old. After spending about an hour reading out-of-date blogs on how to get ARM64 emulation running on QEMU, I shelved that approach as not productive.
Next step: static analysis.
I find it easiest to work from a mixed assembly/source listing to form a mental context for the bug analysis — sometimes it’s immediately obvious what the cause is just from looking at the listing. To generate a listing file with the cross-compiler,
make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- drivers/tty/n_tty.lst
Now, looking at the the bug report header,
Unable to handle kernel paging request at virtual address 000044d0 pgd = ffffffc0007fb000 [000044d0] *pgd=0000000079407003, *pud=0000000079407003, *pmd=0000000079408003, *pte=0060000008004707 Internal error: Oops: 94000005 [#1] PREEMPT SMP Modules linked in: CPU: 1 PID: 632 Comm: kworker/1:1 Not tainted 4.0.0-rc3+ #121 Hardware name: linux,dummy-virt (DT) Workqueue: events flush_to_ldisc task: ffffffc0395eac00 ti: ffffffc038cb0000 task.ti: ffffffc038cb0000 PC is at n_tty_receive_buf_common+0x60/0xa28 LR is at n_tty_receive_buf_common+0x4c/0xa28 pc : [<ffffffc000362490>] lr : [<ffffffc00036247c>] pstate: 20000145 sp : ffffffc038cb3c70 x29: ffffffc038cb3c70 x28: ffffffc0385ab400 x27: ffffffc03ffe3380 x26: 0000000000000008 x25: ffffffc0006a5260 x24: ffffffc0385b0e28 x23: ffffffc038557340 x22: ffffffc0385ab400 x21: ffffffc0385b0e08 x20: ffffffc0385b0e00 x19: ffffffc03854c800 x18: 0000007fda9ad810 x17: 0000007f8291ee7c x16: ffffffc000185540 x15: 0000007f8298f590 x14: 0000007472617473 x13: 2d65727000000009 x12: 0000000000000020 x11: 000000000000002c x10: 0000007fb1e23360 x9 : ffffffc038cb3d40 x8 : ffffffc00079cb10 x7 : ffffffc000362e58 x6 : ffffffc03854c820 x5 : 00000000000044d0 x4 : ffffff8000278000 x3 : 000000000000002e x2 : 0000000000000000 x1 : 0000000000000001 x0 : ffffffc00058f250
the bolded line shows the symbolic address (in
symbol+offset/length form) of the instruction after the faulting instruction. The corresponding code in the listing is determined by finding the symbol address in the listing and adding the offset from the bug report; in this case,
ffffffc0003660b4 + 0x60 = ffffffc000366114. The excerpt is shown below with the suspected instruction bolded.
ffffffc0003660b4 <n_tty_receive_buf_common>: .... * paired with store in *_copy_from_read_buf() -- guarantees * the consumer has loaded the data in read_buf up to the new * read_tail (so this producer will not overwrite unread data) */ size_t tail = smp_load_acquire(&ldata->read_tail); ffffffc000366108: d2844d00 mov x0, #0x2268 ffffffc00036610c: 8b170000 add x0, x0, x23 ffffffc000366110: c8dffc03 ldar x3, [x0]
While this pointed to the load from
[x0] as the cause and this code was added in 4.0-rc1, the result didn’t make sense.
&ldata->read_tail points into a large aggregate structure, that if invalid, would have caused crashes earlier in other code. At this point, I admit I studied the ARM64 implementation of
smp_load_acquire(), thinking maybe it was mis-coded. Nope.
Re-examining the bug report confirmed the listing did not match the report (register
x0 ≠ fault address [0x000044d0]). Next step: disassemble the code from the bug report.
scripts/decodecode from the kernel git tree does cross-binutils disassembly too!
peter@thor:~/src/kernels/mainline$ echo "Code: 91094000 f90057a0 d2844d05 8b0500a5 (c8dffca3)" | CROSS_COMPILE=aarch64-linux-gnu- scripts/decodecode Code: 91094000 f90057a0 d2844d05 8b0500a5 (c8dffca3) All code ======== 0: 91094000 add x0, x0, #0x250 4: f90057a0 str x0, [x29,#168] 8: d2844d05 mov x5, #0x2268 // #8808 c: 8b0500a5 add x5, x5, x5 10:* c8dffca3 ldar x3, [x5] <-- trapping instruction Code starting with the faulting instruction =========================================== 0: c8dffca3 ldar x3, [x5]
Hmmm. Those last three instructions look suspiciously similar to the listing — except the machine code is wrong. In the mixed listing, the effective address is
x23 + 0x2268 whereas, in the report, the effective address is
0x2268 + 0x2268 = 0x44d0!
Mystery solved: a compiler bug.