1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 | The following are demonstrations of the hotkernel DTrace program.
Here hotkernel is run for a couple of seconds then Ctrl-C is hit,
# ./hotkernel
Sampling... Hit Ctrl-C to end.
^C
FUNCTION COUNT PCNT
unix`swtch 1 0.1%
pcplusmp`apic_redistribute_compute 1 0.1%
genunix`strrput 1 0.1%
unix`sys_call 1 0.1%
genunix`fsflush_do_pages 1 0.1%
TS`ts_wakeup 1 0.1%
genunix`callout_schedule_1 1 0.1%
unix`page_create_putback 1 0.1%
unix`mutex_enter 4 0.3%
unix`cpu_halt 1575 99.2%
The output summarises which kernel-level function was sampled on the
CPU the most. This report shows that unix`cpu_halt was sampled 1575
times, which was 99.2% of the kernel-level samples.
As it turns out, unix`cpu_halt is called on this x86 server as part of the
kernel idle thread - explaining why it is so often found on the CPU,
# dtrace -n 'fbt::cpu_halt:entry { @[stack()] = count(); }'
dtrace: description 'fbt::cpu_halt:entry ' matched 1 probe
^C
unix`idle+0x3b
unix`thread_start+0x3
956
This kernel stack trace indicates that cpu_halt() is called by idle().
The following is a SPARC example,
# ./hotkernel
Sampling... Hit Ctrl-C to end.
^C
FUNCTION COUNT PCNT
genunix`fop_ioctl 1 0.1%
genunix`allocb_cred 1 0.1%
genunix`poll_common 1 0.1%
genunix`cv_block 1 0.1%
genunix`strioctl 1 0.1%
genunix`disp_lock_exit 1 0.1%
genunix`crfree 1 0.1%
ufs`ufs_getpage 1 0.1%
SUNW,UltraSPARC-IIi`copyin 1 0.1%
genunix`strmakedata 1 0.1%
genunix`cv_waituntil_sig 1 0.1%
SUNW,UltraSPARC-IIi`prefetch_page_r 1 0.1%
unix`set_freemem 1 0.1%
unix`page_trylock 1 0.1%
genunix`anon_get_ptr 1 0.1%
unix`page_hashin 1 0.1%
genunix`bt_getlowbit 1 0.1%
unix`pp_load_tlb 1 0.1%
unix`_resume_from_idle 1 0.1%
unix`hat_pageunload 1 0.1%
genunix`strrput 1 0.1%
genunix`strpoll 1 0.1%
unix`page_do_hashin 1 0.1%
unix`cpu_vm_stats_ks_update 1 0.1%
genunix`sleepq_wakeone_chan 1 0.1%
unix`lock_set_spl 1 0.1%
tl`tl_wput 1 0.1%
genunix`kstrgetmsg 1 0.1%
genunix`qbackenable 1 0.1%
genunix`releasef 1 0.1%
genunix`callout_execute 1 0.1%
uata`ata_hba_start 1 0.1%
genunix`pcacheset_cmp 1 0.1%
genunix`sleepq_insert 1 0.1%
genunix`syscall_mstate 1 0.1%
sockfs`sotpi_recvmsg 1 0.1%
genunix`strput 1 0.1%
genunix`timespectohz 1 0.1%
unix`lock_clear_splx 1 0.1%
genunix`read 1 0.1%
genunix`as_segcompar 1 0.1%
unix`atomic_cas_64 1 0.1%
unix`mutex_exit 1 0.1%
genunix`cv_unsleep 1 0.1%
unix`putnext 1 0.1%
unix`intr_thread 1 0.1%
genunix`hrt2tv 1 0.1%
sockfs`socktpi_poll 1 0.1%
unix`sfmmu_mlspl_enter 1 0.1%
SUNW,UltraSPARC-IIi`get_ecache_tag 1 0.1%
SUNW,UltraSPARC-IIi`gethrestime 1 0.1%
genunix`cv_timedwait_sig 1 0.1%
genunix`getq_noenab 1 0.1%
SUNW,UltraSPARC-IIi`flushecacheline 1 0.1%
unix`utl0 1 0.1%
genunix`anon_alloc 1 0.1%
unix`page_downgrade 1 0.1%
unix`setfrontdq 1 0.1%
genunix`timeout_common 1 0.1%
unix`bzero 1 0.1%
unix`ktl0 2 0.1%
genunix`canputnext 2 0.1%
genunix`clear_active_fd 2 0.1%
unix`sfmmu_tlb_demap 2 0.1%
unix`page_vpadd 2 0.1%
SUNW,UltraSPARC-IIi`check_ecache_line 2 0.1%
genunix`cyclic_softint 2 0.1%
genunix`restore_mstate 2 0.1%
genunix`anon_map_getpages 2 0.1%
genunix`putq 2 0.1%
unix`page_lookup_create 2 0.1%
dtrace`dtrace_dynvar_clean 2 0.1%
unix`sfmmu_pageunload 2 0.1%
genunix`cpu_decay 2 0.1%
genunix`kmem_cache_alloc 3 0.2%
unix`rw_exit 3 0.2%
tl`tl_wput_data_ser 3 0.2%
unix`page_get_replacement_page 3 0.2%
unix`page_sub 3 0.2%
genunix`clock 3 0.2%
SUNW,UltraSPARC-IIi`copyout 3 0.2%
unix`mutex_enter 4 0.2%
genunix`pcache_poll 5 0.3%
SUNW,UltraSPARC-IIi`scrub_ecache_line 5 0.3%
SUNW,UltraSPARC-IIi`hwblkpagecopy 22 1.2%
SUNW,UltraSPARC-IIi`hwblkclr 39 2.1%
unix`generic_idle_cpu 506 26.8%
unix`idle 1199 63.5%
Which shows the most common function is unix`idle.
Now the hotkernel tool is demonstrated with the -m option, to only print
out samples by module,
# ./hotkernel -m
Sampling... Hit Ctrl-C to end.
^C
MODULE COUNT PCNT
usbms 1 0.0%
specfs 1 0.0%
uhci 1 0.0%
sockfs 2 0.0%
genunix 28 0.6%
unix 4539 99.3%
Here, genunix and unix (the two core parts of the kernel) were the most
common module to be executing on-CPU.
|