DOC HOME SITE MAP MAN PAGES GNU INFO SEARCH PRINT BOOK
 

kcrash(1M)

kcrash - examine system images

Synopsis

# crash
. . .
> kcrash

Description

The Data Center Feature Supplement provides a new primary crash command, kcrash. When you invoke kcrash as shown in the synopsis above, the following commands are then available for system dump analysis.

Commands

All kcrash commands are brief mnemonics (usually two characters) followed by zero or more arguments. In the following descriptions, optional arguments are enclosed in square brackets. Arguments are separated by spaces or commas, and each argument must be one of the following:

Output from kcrash functions may be piped to another program in the following way:

function [argument ...] ! shell_command

Example:

ps ! grep fsflush

Input Commands

The kcrash command prompts with S>. This prompt indicates that kcrash is ready to accept any of the commands described below. Input characters can be erased with BACKSPACE or DEL. An entire input line can be erased with CTRL-U or CTRL-X. In addition, kcrash supports flow control (CTRL-S, CTRL-Q) and keyboard interrupt (CTRL-C).

During any of the display, modify, examine, or write commands, you can enter one of the following:

RETURN
Move to the next item.
+n
Move to the nth next item.
-
Move to the previous item.
-n
Move to the nth previous item.
=addr
Move to the item at address addr. (This command is only valid when operating on memory, not on registers.)
n
Change the value of the item to n. This is only valid for modify or write commands, not display or examine. The mi command allows you to enter multiple numbers separated by spaces to change more than one byte.
.
Exit the command and returns to the prompt. (This can be any character other than +, -, =, or a hex number.)

If an attempt is made to access an invalid virtual address, the command and all levels of invoked macros will be aborted and kcrash will prompt for the next command.

Display Commands

These commands allow you to examine memory only. This prevents accidental modification of system memory when in kcrash.
dl addr [count]
Display memory as long integers (4 byte hex integers), 32 bytes at a time. If a count is given, memory is displayed 32 times (*) count bytes at a time.
dw addr [count]
Display memory as words (2-byte hex integers) 32 bytes at a time.
db addr [count]
Display memory as bytes (1- byte hex integers) 32 bytes at a time.
di [addr]
Display memory as disassembled instructions. The default addr is the contents of %eip.
dr [addr]
Display the CPU stored at addr. The default addr is the automatically determined register save area (see the rg command).
dR
Display the CPU "special" registers (debug, control, and table base registers).
dy addr [count]
Similar to dl, but displays the long integers in symbolic form if possible.
se
search start end pattern [mask]
Search for the given pattern in the range of addresses starting at start, up to (but not including) end. The search is performed on long integers. If a mask is given, only those bits corresponding to 1 bits in the mask are significant in the search.

Examine Commands

el addr
Examine memory as longs one at a time.
ew addr
Examine memory as words one at a time.
eb addr
Examine memory as bytes one at a time.
ei [addr]
Examine memory as disassembled instructions. (Same as di.)
er
Examine CPU general registers one at a time.
eR
Examine the CPU special registers one at a time.

Execute Commands

call addr [args]
Call a function with the specified arguments and show the return value.

Miscellaneous Commands

< file
Read and execute commands from the given file.
! shell-command
Execute the given shell-command.
q
Quit kcrash, returning to crash program.
? or ??
List the kcrash commands.
# or ## or no
Ignore input lines beginning with a pound sign (#) or the string no. Those lines are treated as comments.
@ command
Issue the command to crash.
as value
Limit the size of arrays the the structure command will display.  (Default is unlimited)
bt [addr]
Display a stack backtrace, using addr as a frame pointer. The default address is the contents of %ebp. (This works only with C language routines in protected mode.)
BT [addr][offset]
Display a kernel process stack backtrace, using addr as a frame pointer. The default address is the contents of %ebp. offset may be used to override %esp. The backtrace displays the entire stack in an easy-to-read format. This works only with C language routines in protected mode.
Bt [addr][offset]
Display a kernel and user stack backtrace, using addr as a frame pointer. The default address is the contents of %epb. offset may be used to override %esp. The backtrace displays the entire stack in an easy-to-read format. This works only with C language routines in protected mode.
c3 [addr]
Use the specified physical address as the base of the page directory for translating linear-to-physical addresses. This address is obtained from the special register CR3 if no c3 command is given. To restore the base to that original value, use an addr of zero. If addr is missing, display the current page directory base.
cd dirname
Change working directory while in kcrash.
cf
Close the file opened by the of command.
ds addr
Print the value of the address as an offset from the nearest symbol.
fill start end value
Fill memory from address start up to (but not including) address end with the byte value.
kml addr value
Set the crash internal variable at addr to value.
? or ??
he or help
List the kcrash commands.
ll addr link
Walks a linked list starting at addr with link as the offset to the next link. Uses the dl command to display data.
lw addr link
Walks a linked list starting at addr with link as the offset to the next link. Uses the dw command to display data.
lk filename
Load filename as kernel symbols, overwriting current kernel symbols.
lu filename
Load filename as user symbols, overwriting current user symbols.
ma addr
map addr
Display the page directory and page table entries used to map the given linear address to a physical address. This behaves the same whether paging is on or off.
more [lines]
Set the number of display lines to lines. If lines is greater than zero, it enables output paging. When lines or more contiguous lines of information are printed without asking the user for input, the message --press space for more-- is displayed and output is temporarily suspended until the user presses the space bar. This prevents kcrash from printing too many lines of output at once on video terminals. If lines is zero, it disables output paging. Output paging is disabled by default. If lines is missing, it reports whether output paging is enabled or disabled.
nm [regular expression]
Display all symbols that match the regular expression.
no
Ignore input lines beginning with a pound sign (#) or no. Those lines are treated as comments.
of [filename]
Open filename for output. All data sent to the screen is also sent to filename. If filename is omitted, of toggles sending output to filename.
offset strname element [addr]
This command displays the offset of a structure element given strname element. If addr is present offset will display the structure element with its current contents.
pause
Pause until the user types something.
pf string [args.. .]
printf string [args...]
Print the string. Percent signs in the string are treated as in printf(3S): %d, %u, %x, %o, %b, %s, and %c are supported. In addition, %y prints its argument in symbolic form, if possible; and %I prints its argument in disassembled instruction form.
pg [n]
Turn paging off if n is 0. If n is 1, turn paging on. If n is missing, report whether paging is on or off. If paging is off, kcrash interprets all addresses as linear (virtual) addresses. (Breakpoints are always linear addresses.)
pr addr [radix]
Print the value of addr in the specified radix, or in the current input radix if radix is not given. This is most useful if addr is an expression (see the earlier discussion of arguments).
printbits "bit-desc" word
Display the bits that are set (the 1 bits) in word symbolically according to "bit-desc," which is a colon-separated list of names associated with the corresponding bit positions, starting with bit 0 (the least significant bit). For example, the command printbits "X:Y:Z:FOO:BAR" 0x9D prints X Z FOO BAR.
printcase "case_desc" word
Display the "case_desc" symbolically based on the value in word according to case_desc. case_desc is a colon-separated list of names associated with the corresponding case value and starting with case 0 (the least significant case). For example, the command printcase "X:Y:Z:FOO:BAR" 0x3 prints FOO.
radix [n]
Set the input radix to n. If n is omitted, display the current input radix. The default radix is hexadecimal.
rd [n]
Turn reverse 286/386 disassembly off if n is 0. If n is 1, turn reverse 286/386 disassembly on. Disallows disassembly of 286 and real-mode instructions in real mode.
read variable
Read an expression from the user and set the named kcrash variable to the expression’s value.
real [n]
Turn real-mode off if n is 0. If n is 1, turn real-mode on. If n is missing, report whether real-mode is on or off. If real-mode is off, the kcrash traces in protected mode. If real-mode is on, the kcrash traces in real mode.
rg [addr]
Change the pointer to the "register save area," from which all references to CPU registers retrieve registers. Normally, the register save area is set up automatically, but you can use a different set of registers when you use rg to change the pointer. To restore the pointer to its original value, use an addr of zero. If addr is not given, the current register save area pointer is displayed.
set var value
Set the user variable var to have the given value. If the variable var has not been previously defined, it becomes defined. If the variable var was previously defined, its old value is lost.
setk var value
Set the kernel variable var to have the given value. If the variable var was not previously defined, it becomes defined. If the variable var was previously defined, its old value is lost.
size name
Displays the size of a structure.
sm [regexpr]
Displays the known structure names that match regular expression.
st strname addr
Display a structure at addr using strname as a template.
sp [addr]
Use addr as the address of a kernel proc structure. kcrash uses the context of that process to translate linear-to-physical addresses. Use of the sp command overrides the CR3 register: after an sp command, c3 commands have no effect.
ST [filename]
If filename is given, load structure definitions. If filename is not given, display known structure names.
sl strname addr[offset]
Display a linked list of structures using offset as address of next structure.
sy [n] [max]
Turn symbolic display off if n is 0. If n is 1, turn symbolic display on. If n is not given, report whether symbolic display is on or off. If max is given, it specifies the maximum offset for printing symbols. For example, if max is 1000, a symbol may be displayed in the form name+NNN, where NNN is 1 through 1000; but if NNN would be greater than 1000, the non-symbolic display format is used.
ve
Print the version number of kcrash.

Macro commands

dm
define name [argdesc] [maxsize]
Defines a macro with the given name and the specified argument description string argdesc. The optional maxsize argument specifies the maximum size (in bytes) of the macro; the default size is 4096 bytes. The macro can be invoked after its definition by simply typing its name, like any other command. The arg-desc string describes to kcrash what arguments the macro expects. Each lowercase letter specifies the type of the corresponding argument, as follows:

a or i
address or integer, the result of an arbitrary expression
s
string
?
means the following arguments are optional
*
means any number of arguments or any type
.
means don’t parse more arguments
,
is ignored.

For example, the argument description for the se command is aai?i and for pf it is s*. If arg-desc is not given, the macro will be defined as requiring no arguments. Any kcrash commands can be entered as the body of the macro, although interactive commands, such as di are not recommended (see the interact command). The expression $n, where n is a digit from 1 to 9, is replaced on invocation with the nth argument to the macro. The expression $# evaluates to the number of arguments to the macro. Entry of the macro body is terminated by a period (.) anywhere in the macro body. Include a period in the macro body by preceding the character with a backslash (\).

args n
Set the number of macro arguments to n.
delm name
em name
Erase (delete) the named macro.
do name [args. . . ]
Call the macro name repeatedly with any args specified until an exit command is executed. This is the only explicit form of iteration kcrash provides. The args are passed to the named macro; and, if setarg is not used within the macro, args is passed to every subsequent iteration of the named macro.
ec
echo [n]
Echo macros when they are invoked if n is 1. If n is 0 (the default), macros are not echoed. If n is not given, the status of the echo flag is printed. If the ec command is given within a macro body, it is in effect for that macro only.
exit
Stop iterating a repeated macro call (see do). Note that exit does not terminate the execution of the current macro; it merely prevents further iterations.
interact n
Read input from the user when interactive commands (such as di, bt, and dr ) are invoked during macro execution if n is 1. If n is 0 (the default), interactive commands inside macros will read input from the macro body. The interact command affects only the currently executing macro and has no effect outside a macro body.
lm [regexpr]
List the macros that match the regular expression.
lm [name]
List the named macro. If the macro name is omitted, all macros are listed.
nx
Repeat the call to the previously invoked macro. The arguments used are those used on the previous call that were possibly modified by any intervening sa commands.
sa n value
setarg n value
Set the nth macro argument to the given value. The value of n should be between 1 and 9. This feature works within a macro to set up the arguments for the next call through an nx command.

Conditional commands

IF expr
EL
FI
Skips all commands up to the matching EL or FI if the expression evaluates to zero. If the expression is non-zero, execution proceeds normally to the matching FI unless a matching EL is found, in which case, commands between the EL and the FI are skipped. During any of the skipping, the prompt changes from Kn> to -Kn> to indicate that the commands are being read but not executed.
if expr
elseif expr
else
Works the same as IF-EL-FI but includes the elseif construct that allows chaining of conditional statements.
ifdef name
Works like if, but the condition is true if a macro named name exists.
ifsdef name
Works like if, but the condition is true if a symbol named name exists.

Effects of crash and kcrash commands

The tables below describe the kcrash alternatives to primary crash commands. To use kcrash effectively, you must understand the results of a given command in crash. You can then select an alternative to use with kcrash. The table below lists the effects of kcrash commands used as alternatives to crash commands.

Command

Effect in crash

Effect in kcrash
!cmd Works ! [cmd] or !! [cmd]
? Works ? [cmd] or ?? [cmd]
as Works Load proc.k and dscr.k, address from as field in proc can be given to dscr macro
async Fails "aio_list not found in
symbol table"		  
b (buffer) Fails "s not found in symbol
table"		 Load buf.k
base Works pr
buf (bufhdr) Fails "header, no content" Load buf.k
c (callout) Fails "header, no content" Load stat.k, cm callout
class Works Load info.k, cm class
dbfree Not supported dl *mdbfreelist
dblock Not supported Not applicable
defproc Works Load proc.k, cm setproc
dis Works di [address]
dispq Fails Load proc.k, cm dispq
ds Works ds address
evactive Fails "ev_actqp not found
in symbol table"		 None
evmm Fails "ev_mminfo not found
in symbol table"		 None
f (file) Works dl file or *file
findaddr Works Requires manual calculation of table size and offset
findslot Fails "no match for rt_proc
 in size table"		 None
fs (vfssw) Works Load stat.k, cm vfssw
gdp Fails "gdp table not found" None
gdt Works Load dscr.k, cm gdt
help Works ? [cmd] or ?? [cmd]
hrt Fails "header, no content" dl hrtimes use structure in /usr/include/sys/hrtsys.h
i (inode) Fails "header, no content" Load inode.k, cm inode address
idt Works Load dscr.k, cm idt
kfp Works dr save_panic_regs, use value in bp
kmastat Works Load page.k, cm kmastat
l (lck) Works  
ldt Works Use dR command
linkblk Works Load stream.k, cm linkblk
map Works None
mbfree Not supported Not applicable
mblock Not supported Not applicable
mode Works pg
mount (m, vfs) Works Load stat.k, cm vfsw
nm Works nm "regular expression"
od Works dl address
p (proc) Works Load proc.k cm ps
page Works Load page.k cm page address
panic Works dl *panicstr
pcb Works Load tss.k, cm tss address
plock Works None
prnode Fails "header, no content" crash uses this to look at /proc; not needed in kcrash
ptbl Fails "core dump" Load page.k ptbl
pty Works Load sertty.k, cm pty
q (quit) Works q, qq, or quit
qrun Works Load stream.k, cm queuerun
queue Works Load stream.k, cm queues
rcvd Fails "receive descriptor table not found"  
rd (od) Works dl
rduser Fails "rcvd user table not found" Not applicable
redirect Works Use of cf
resource RFS None
rtdptbl Works Load proc.k, cm rtdptbl
rtproc Fails "header, no content"
runq Fails "runq not yet implemented" Load proc.k, cm runq
s (stack) Works bt
searc Works se
size Works None
sndd Fails "send descriptor table
not found"		 Not applicable
snode Works Load inode.k, cm snode
srmount RFS Not applicable
stat Works Load stat.k, cm stat
stream Works Load stream.k, cm streams
strstat Works Load stream.k, cm strstat
t (trace) Works Load trace.k, cm trace address
test Works Not applicable
ts Works ds address
tsdptbl Works Load proc.k, cm tsdptbl
tsproc Works Load proc.k, cm tsproc
tty Fails "tty struct for asy not found in symbol table" dl various tty structures
u (user) Works Load user.k, cm user [address]
ui (uinode) Works Load uinode.k, cm uinode address
v (var) Works Load stat.k, cm var
vnode Works Load vnode.k, cm vnode address
vtop Works None

Predefined Macros (General)

The following are general-use macros for the kcrash command. Predefined macros designed for specific purposes are provided in separate sub-sections.

To make these macros available you can execute the command
< /usr/lib/crash/macros/loadmacs

buf addr
Print selected fields of a struct buf at the given address.
buf+ addr
Run buf addr and set up kcrash to display the next adjacent buffer.
buf- addr
Run buf addr and set up kcrash to display the previous adjacent buffer.
bufv addr
Run buf addr and set up kcrash to display the buffer at addr->av_forw each time a carriage return is entered.
dscr addr
Print any 286/i386/i486 descriptor in its appropriate format.
dscr+ addr
Run dscr addr and set up kcrash to display the next adjacent descriptor.
inode addr
Print selected fields of a struct inode (in the same manner as sys/inode.h).
inode+ addr
Run inode addr and set kcrash up to print the next adjacent inode address.
proc addr
Print selected fields of a struct proc.
proc+ addr
Run proc addr and set up kcrash to display the next adjacent processor address.
ps
Simulate /bin/ps -l. (ps_loop and psl are submacros used by ps.)
pid pid
Find a process with the given pid (remember the default debugger radix is hex, not decimal) and run proc on it. (pid_search is a submacro used by pid. pidof and pgrpof are submacros used by proc.)
btproc addr
Set the kdb process context to addr and run the backtrace bt command.
strstat
Print selected STREAMS statistics. (strstat_aux is a submacro used by strstat.)
strmsg addr
Print selected fields of a struct msgb. (strmsg_aux and strmsg_type are submacros used by strmsg.)
strqueue addr
Print selected fields of a struct queue. (strqueue_aux is a submacro used by strqueue.)
strqueue_band addr
Print selected fields of a struct qband. (strfollow is a submacro used by stream.)
stream addr
Print selected fields of a struct stdata and substructures.
streams
Print all streams except muxs.
streams_muxs
Print all streams. (streams_aux is a submacro used by streams and streams_muxs.)
queues
Print all streams queues.
queues_flag
Print all streams queues with none of the flags set. (queues_aux and queues_loop are submacros used by queues and queues_flag.)
stream_find addr
Find and print a stream associated with the given queue. (stream_find1 and stream_find2 are submacros used by stream_find.)
flags addr
Print flags (in the same manner as %eflags) set in dword at addr.
tss addr
Print selected fields of a struct tss386.
tss+ addr
Run tss addr and set kcrash up to run tss on the next adjacent address.
tty addr
Print selected fields of a struct tty.
tty+ addr
Run tty addr and set kcrash up to run tty on the next adjacent address.
user addr
Print selected fields of a struct user.
vnode addr
Print selected fields of a struct vnode.
vnode+ addr
Run vnode addr and set up kcrash to run vnode on the next adjacent address.
pages addr
Display link list of page structures.
kmastat
Display the user area in the same format as crash.
ptbl
Display the user area in the same format as crash.
pagetov addr
Convert page addr to a virtual address.
vtopage addr
Convert virtual addr to a page.
phystopage addr
Convert physical addr to a page.
runq
Display the runq in the same format as crash.
dispq
Display the dispq in the same format as crash.
rtdptbl
Display the rtdptbl in the same format as crash.
tsdptbl
Display the tsdptbl in the same format as crash.
tsproc
Display the tsproc in the same format as crash.
wch
Display one line for every process to indicate on which channel it is waiting.
pty
Display the pty area in the same format as crash.
linkblk
Display linkblks in the same format as crash.
cqueue
Display only queues that have messages queued.
find_uinode i
Find UFS inode in ufs_inode.
find_ncache i
Find UFS inode in ncache.
find_name i
Display name of UFS inode i.
dinode addr
Display UFS disk inode.
dinode+ addr
Display UFS disk inode.
cg addr
Display UFS cylinder group structure.
fs addr
Display UFS super-block group structure.
Cgsblock i
Display address of super-block in cylinder group i.
Cgtod i
Display address of cylinder group in cylinder group i.
Cgimin i
Display address of inode information in cylinder group i.
Cgdmin i
Display address of data in cylinder group i.
Itoo i
Display inode i in a UFS file system.
dirent addr
Display address as a directory entry.
User addr
Display the user area in the same format as crash.
convmap addr
Display information about a mapping value.
hatconv addr
Display information about a hatmcp value.
vnode addr
Display vnode and all pages that belong to this vnode.
vtovnode addr
Convert virtual address to a vnode.
phystovnode addr
Convert physical address to a vnode.

Predefined device macros

device addr
Print selected fields of a struct device at the given address.
device+ addr
Run device addr and set up a kcrash to display the next adjacent device.
bdevsw addr
Print selected fields of a struct bdevsw at the given address.
bdevsw+ addr
Run bdevsw addr and set up kcrash to display the next adjacent bdevsw.
cdevsw addr
Print selected fields of a struct cdevsw at the given address.
cdevsw+ addr
Run cdevsw addr and set up kcrash to display the next adjacent cdevsw.
conssw ""
Print selected fields of a struct device at the given address.
fmodsw addr
Print selected fields of a struct device at the given address.
fmodsw+ addr
Run device addr and set up kcrash to display the next adjacent device.

Predefined file macros

strowner addr
Print the process that opened stream addr.
files addr
Print the file table for process addr.
cred addr
Print selected fields of a struct cred at the given address.
qinit addr
Print selected fields of a struct qinit at the given address.
moduleinfo addr
Print selected fields of a struct moduleinfo at the given address.
PID add
Print selected fields of a struct stdata at the given address.
streamtab addr
Print selected fields of a struct streamtab at the given address.
stdata addr
Print selected fields of a struct stdata at the given address.
sysinfo ""
Print the sysinfo structure.
vminfo ""
Print the vminfo structure.
minfo ""
Print the minfo structure.
fsinfo ""
Print the sinfo structure.
syswait ""
Print the syswait structure.
syserr ""
Print the syserr structure.
shlbinfo ""
Print the shlbinfo structure.
bpbinfo ""
Print the bpinfo structure.
rtminfo ""
Print the rtminfo structure.
kmeminfo ""
Print the kmeminfo structure.
tune ""
Print the tune structure.
queowner addr
Display process that owns queue addr.
file
Display the file area in the same format as crash.

Predefined sertty macros

sertty addr
Print selected fields of a struct sertty.
sertty+ addr
Run sertty addr and set kcrash up to run sertty on the next adjacent address.

Predefined stat macros

stat
Print system statistics.
lbolt i
Convert i into an ASCII string containing the date and time.
dumpinfo
Print miscellaneous information about the dump, such as the system name and the time the dump occurred.
var
Print the var structure.
callout
Print the callout structure.
putbuf
Print the putbuf in ASCII.

Predefined trace macros

trace addr
Print a trace of process addr.
dtrace addr
Print a trace of all processes starting at addr.
activeprocs ""
Print a trace for all processors on line.
activeproc# i
Print a trace for CPU i.
stact addr
Print the stack for processor i in long form, which breaks out each EBP EIP pair.
stack+ addr
Print the stack for processor i in long form, which breaks out each EBP EIP pair.
dstack addr
Print all the stacks in long form, starting at addr, which breaks out each EBP EIP pair.
cpu i ? i
Print the stack for processor i in long form, which breaks out each EBP EIP pair.
panic
Print information about a panic dump, indicating what each processor is doing. Print the process list. Calls the stat and putbuf macros.
panicregs i
Print the panic registers for processor i.
whatswrong
Analyze a dump and try to display the error.

Files

/usr/lib/crash/macros
Directory containing kcrash macro files.
/usr/lib/crash/libkcrash.so
Dynamic linked library added kcrash commands.

References

crash(1M)