LKMP Getting Started
Table of Contents
I was selected for LKMP Fall 2025! This is where I’ll document the debugging tools and workflow I’m learning. Will update as I go.
1. Baby Steps
My goal: Find and fix a syzbot bug. Any subsystem, just want to get started. From the mentorship videos, here’s the basic process:
- Build kernel with
CONFIG_DEBUG_INFO=yCONFIG_FRAME_POINTER=y - Boot QEMU with
-s -Sandnokaslr ftrace_dump_on_oops - Attach GDB:
gdb vmlinux,target remote :1234, set breakpoints - Trace with
trace-cmd recordthentrace-cmd report - Check scheduling:
perf sched record,perf sched latency sort max - Enable watchdogs & hung task detectors
- Add
trace_printk()in suspected code paths - For crashes: use kdump + crash tool
Starting with GDB first.
2. Debugging Tools
2.1. Core Debugging
- QEMU - Run custom kernels safely
qemu-system-x86_64 -kernel bzImage -append "console=ttyS0 root=/dev/sda nokaslr" -s -S -nographic
-s -S means GDB server on port 1234, wait for attach
- GDB - Debug kernel in QEMU
Common commands: target remote :1234, bt, info threads, b panic, list
- KGDB - GDB for real hardware (serial/network)
- JTAG - For early boot problems
2.2. Tracing
- Ftrace - Built-in kernel tracer
Boot params: ftrace_dump_on_oops, traceoff_on_warning, panic_on_warn=1
- trace-cmd - Ftrace frontend
trace-cmd record -p function_graph -g kfree trace-cmd report
- perf - Scheduler analysis
perf sched record -- sleep 10 perf sched latency sort max perf script
2.3. Detectors
- Lockup Watchdogs - Catch hung CPUs
Use: nmi_watchdog=1 watchdog_thresh=2
- Hung Task Detector - Find stuck processes
Use: echo 10 > /proc/sys/kernel/hung_task_timeout_secs
- RCU Stall Detector - RCU problems
Use: rcupdate.rcu_cpu_stall_timeout=20
2.4. Sanitizers
- KASAN - memory bugs (use-after-free, out-of-bounds)
- KCSAN - data races
- LOCKDEP - locking bugs
2.5. Post-Mortem
- kdump/crash - Capture and analyze kernel crashes
- SysRq - Emergency debugging keys
2.6. Logging
-printk:
printk(KERN_DEBUG "value=%d\n", value);
-traceprintk (lower overhead):
trace_printk("value=%d\n", value);
3. Required Config
Kernel:
CONFIG_DEBUG_INFO=yCONFIG_FRAME_POINTER=yCONFIG_GDB_SCRIPTS=y
Boot params:
nokaslrpanic_on_warn=1traceoff_on_warningftrace_dump_on_oops
4. Next Steps
- Setup QEMU environment
- Build debug kernel
- Pick first syzbot bug
- Document the fix
- Submit patch