Kernel BUG report walkthrough

My patch series “N_TTY input path fixes”, which fixes lockless TTY input on weakly-ordered arches, is part of the 4.0-rc kernel. So last Friday a post to the linux-serial mailing list leading with $subject “[BUG] n_tty: 4.0-rc3+ …” got my attention. The report was from one of the Ubuntu Server devs

Hi Guys,

Just found the following oops during kernel booting when I
test the latest linus tree on one arm64 VM booted from uefi
plus grub2

Keywords: n_tty “latest linus tree” arm64. So I scroll down to the stack trace

Call trace:
[<ffffffc000362490>] n_tty_receive_buf_common+0x60/0xa28
[<ffffffc000362e64>] n_tty_receive_buf2+0xc/0x18
[<ffffffc000365b64>] flush_to_ldisc+0xec/0x140
[<ffffffc0000b5228>] process_one_work+0x138/0x338
[<ffffffc0000b5970>] worker_thread+0x148/0x420
[<ffffffc0000ba8fc>] kthread+0xd4/0xf0

Uh-oh. Definitely related to my code.

My first thought was “I’ll boot an ARM64 VM and see if I can reproduce”. Except I don’t have an ARM64 VM setup and I quickly discovered my QEMU version was too old. After spending about an hour reading out-of-date blogs on how to get ARM64 emulation running on QEMU, I shelved that approach as not productive.

Next step: static analysis.

I find it easiest to work from a mixed assembly/source listing to form a mental context for the bug analysis — sometimes it’s immediately obvious what the cause is just from looking at the listing. To generate a listing file with the cross-compiler,

make ARCH=arm64 CROSS_COMPILE=aarch64-linux-gnu- drivers/tty/n_tty.lst

Now, looking at the the bug report header,

Unable to handle kernel paging request at virtual address 000044d0
pgd = ffffffc0007fb000
[000044d0] *pgd=0000000079407003, *pud=0000000079407003,
*pmd=0000000079408003, *pte=0060000008004707
Internal error: Oops: 94000005 [#1] PREEMPT SMP
Modules linked in:
CPU: 1 PID: 632 Comm: kworker/1:1 Not tainted 4.0.0-rc3+ #121
Hardware name: linux,dummy-virt (DT)
Workqueue: events flush_to_ldisc
task: ffffffc0395eac00 ti: ffffffc038cb0000 task.ti: ffffffc038cb0000
PC is at n_tty_receive_buf_common+0x60/0xa28
LR is at n_tty_receive_buf_common+0x4c/0xa28
pc : [<ffffffc000362490>] lr : [<ffffffc00036247c>] pstate: 20000145
sp : ffffffc038cb3c70
x29: ffffffc038cb3c70 x28: ffffffc0385ab400
x27: ffffffc03ffe3380 x26: 0000000000000008
x25: ffffffc0006a5260 x24: ffffffc0385b0e28
x23: ffffffc038557340 x22: ffffffc0385ab400
x21: ffffffc0385b0e08 x20: ffffffc0385b0e00
x19: ffffffc03854c800 x18: 0000007fda9ad810
x17: 0000007f8291ee7c x16: ffffffc000185540
x15: 0000007f8298f590 x14: 0000007472617473
x13: 2d65727000000009 x12: 0000000000000020
x11: 000000000000002c x10: 0000007fb1e23360
x9 : ffffffc038cb3d40 x8 : ffffffc00079cb10
x7 : ffffffc000362e58 x6 : ffffffc03854c820
x5 : 00000000000044d0 x4 : ffffff8000278000
x3 : 000000000000002e x2 : 0000000000000000
x1 : 0000000000000001 x0 : ffffffc00058f250

the bolded line shows the symbolic address (in symbol+offset/length form) of the instruction after the faulting instruction. The corresponding code in the listing is determined by finding the symbol address in the listing and adding the offset from the bug report; in this case, ffffffc0003660b4 + 0x60 = ffffffc000366114. The excerpt is shown below with the suspected instruction bolded.

ffffffc0003660b4 <n_tty_receive_buf_common>:
....
             * paired with store in *_copy_from_read_buf() -- guarantees
             * the consumer has loaded the data in read_buf up to the new
             * read_tail (so this producer will not overwrite unread data)
             */
             size_t tail = smp_load_acquire(&ldata->read_tail);
ffffffc000366108:       d2844d00        mov     x0, #0x2268
ffffffc00036610c:       8b170000        add     x0, x0, x23
ffffffc000366110:       c8dffc03        ldar    x3, [x0]

While this pointed to the load from [x0] as the cause and this code was added in 4.0-rc1, the result didn’t make sense. &ldata->read_tail points into a large aggregate structure, that if invalid, would have caused crashes earlier in other code. At this point, I admit I studied the ARM64 implementation of smp_load_acquire(), thinking maybe it was mis-coded. Nope.

Re-examining the bug report confirmed the listing did not match the report (register x0 ≠ fault address [0x000044d0]). Next step: disassemble the code from the bug report. scripts/decodecode from the kernel git tree does cross-binutils disassembly too!

peter@thor:~/src/kernels/mainline$ echo "Code: 91094000 f90057a0 d2844d05 8b0500a5 (c8dffca3)" | CROSS_COMPILE=aarch64-linux-gnu- scripts/decodecode
Code: 91094000 f90057a0 d2844d05 8b0500a5 (c8dffca3)
All code
========
   0:	91094000 	add	x0, x0, #0x250
   4:	f90057a0 	str	x0, [x29,#168]
   8:	d2844d05 	mov	x5, #0x2268                	// #8808
   c:	8b0500a5 	add	x5, x5, x5
  10:*	c8dffca3 	ldar	x3, [x5]		<-- trapping instruction

Code starting with the faulting instruction
===========================================
   0:	c8dffca3 	ldar	x3, [x5]

Hmmm. Those last three instructions look suspiciously similar to the listing — except the machine code is wrong. In the mixed listing, the effective address is x23 + 0x2268 whereas, in the report, the effective address is 0x2268 + 0x2268 = 0x44d0!

Mystery solved: a compiler bug.

Comments