c# - Debug High CPU Usage -
i have managed application uses ucma (unified communications managed api) 4.0 sdk. trying debug issue application utilizes 100% of cpu , system hung. have used sos extensions try , debug root cause. stuck. have managed find thread ids taking cpu time unmanaged threads. need this.
threads 15, 18, 16, 17, 19, 20 unmanaged threads , have same call stack. threads 9, 10, 11, 12, 13, 14 unmanaged threads , have same call stack well. question threads 21 , 22 appear waiting on event why considered runaway threads consuming cpu time?
does know zwremoveiocompletionex doing? dormant ntwaitformultipleobjects or chewing cpu time? in case of application once spikes 100% never goes down until application restarted.
0:000> !loadby sos clr 0:009> .time debug session time: wed may 27 15:47:52.000 2015 (utc - 4:00) system uptime: 31 days 1:05:59.329 process uptime: 31 days 1:01:27.000 kernel time: 0 days 21:44:58.000 user time: 1 days 16:51:40.000 0:000> !runaway user mode time thread time 15:113c 0 days 3:46:30.510 18:1418 0 days 3:18:07.135 16:1404 0 days 3:08:01.009 17:140c 0 days 3:07:19.310 19:1428 0 days 3:04:56.943 20:1434 0 days 2:52:51.664 22:1450 0 days 0:47:50.153 9:11dc 0 days 0:45:02.904 21:1440 0 days 0:43:34.623 12:13cc 0 days 0:33:35.298 11:1250 0 days 0:32:50.386 14:fbc 0 days 0:31:57.018 10:1178 0 days 0:29:12.920 13:13c4 0 days 0:28:42.048 2:fa8 0 days 0:03:11.678 4:1164 0 days 0:02:45.080 0:015> kb retaddr : args child : call site 000007fe`fd36546f : 00000000`272946f0 000007fe`e5394b29 00000000`27295b18 00000000`27295b18 : ntdll!zwremoveiocompletionex+0xa 00000000`7700c089 : 00000000`1c4981e0 00000000`00000001 00000000`00000001 00000000`00000000 : kernelbase!getqueuedcompletionstatusex+0xdf 000007fe`e51b634b : 00000000`000009b0 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!getqueuedcompletionstatusexstub+0x19 000007fe`e538fc0b : 00000000`1c4981e0 00000000`1c4981e0 000007fe`e5905340 00000000`00000000 : rtmpal!rtcpaltaskqueuedequeue+0x17 000007fe`e538f960 : 00000000`1f55fcf0 00000000`00000000 00000000`1db59eb0 00000000`1f55fcf0 : microsoft_rtc_internal_media!cstreamingengineimpl::engineworkerthread+0x267 000007fe`e51b33c8 : 00000000`00000000 00000000`1c40a6a0 00000000`1c4a4f80 00000000`00000000 : microsoft_rtc_internal_media!cstreamingengineimpl::engineworkerthreadproc+0xf0 000007fe`f22a3d67 : 00000000`00000000 00000000`1c40a6a0 00000000`00000000 00000000`00000000 : rtmpal!rtcpalsetschedulerpolicy+0x194 000007fe`f22a3f0e : 000007fe`f233cdb0 00000000`00000000 00000000`00000000 00000000`00000000 : msvcr110!beginthreadex+0x107 00000000`76fd652d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : msvcr110!endthreadex+0x192 00000000`7720c541 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!basethreadinitthunk+0xd 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!rtluserthreadstart+0x1d 0:022> !clrstack os thread id: 0x1450 (22) child sp ip call site 000000001fdcda68 000000007723186a [helpermethodframe_1obj: 000000001fdcda68] system.threading.waithandle.waitmultiple(system.threading.waithandle[], int32, boolean, boolean) 000000001fdcdba0 000007fee968c64c system.threading.waithandle.waitany(system.threading.waithandle[], int32, boolean) 000000001fdcdc00 000007fe8e097a70 microsoft.rtc.internal.media.rtpeventhandlerthread.eventhandlerthreadproc() 000000001fdce8d0 000007fee973d0b5 system.threading.executioncontext.runinternal(system.threading.executioncontext, system.threading.contextcallback, system.object, boolean) 000000001fdcea30 000007fee973ce19 system.threading.executioncontext.run(system.threading.executioncontext, system.threading.contextcallback, system.object, boolean) 000000001fdcea60 000007fee973cdd7 system.threading.executioncontext.run(system.threading.executioncontext, system.threading.contextcallback, system.object) 000000001fdceab0 000007fee96b0301 system.threading.threadhelper.threadstart() 000000001fdcedc8 000007feed44ffe3 [gcframe: 000000001fdcedc8] 000000001fdcf0f8 000007feed44ffe3 [debuggeru2mcatchhandlerframe: 000000001fdcf0f8] 0:021> kb retaddr : args child : call site 000007fe`fd331430 : 00000000`00190398 00000000`771f3a92 00000000`c0000008 00000000`00000110 : ntdll!ntwaitformultipleobjects+0xa 00000000`76fd1220 : 00000000`1edefc18 00000000`1edefc00 00000000`00000000 00000000`00da7a64 : kernelbase!waitformultipleobjectsex+0xe8 000007fe`e53bc322 : 00000000`0000cae8 00816179`f67cb320 00000000`1c497eb0 00000000`1edefce0 : kernel32!waitformultipleobjects+0xb0 000007fe`e51b33c8 : 00000000`00000000 00000000`00000000 00000000`1dad4630 00000000`1c4a5160 : microsoft_rtc_internal_media!cstreamingengineimpl::timerthreadproc+0x37e 000007fe`f22a3d67 : 00000000`00000000 00000000`1dad4630 00000000`00000000 00000000`00000000 : rtmpal!rtcpalsetschedulerpolicy+0x194 000007fe`f22a3f0e : 000007fe`f233cdb0 00000000`00000000 00000000`00000000 00000000`00000000 : msvcr110!beginthreadex+0x107 00000000`76fd652d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : msvcr110!endthreadex+0x192 00000000`7720c541 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!basethreadinitthunk+0xd 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!rtluserthreadstart+0x1d 0:013> kb retaddr : args child : call site 000007fe`fd36546f : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!zwremoveiocompletionex+0xa 00000000`7700c089 : 00000000`00000000 00000000`000000b7 00000000`00000001 00000000`1c4a4a40 : kernelbase!getqueuedcompletionstatusex+0xdf 000007fe`e51c0fef : 000007fe`e5905340 000007fe`e53eb764 00000000`00000000 00000000`1dac2ab0 : kernel32!getqueuedcompletionstatusexstub+0x19 000007fe`e53eaf4b : 000007fe`e5905340 00000000`35bdd608 00000000`00000001 00000000`1f17fc20 : rtmpal!rtcpaliocp::getqueuedcompletionstatus+0x18f 000007fe`e53eac6d : 00000000`00000510 00000000`0000dddd 00000000`1dad9fe0 00000000`1f17fc80 : microsoft_rtc_internal_media!ctransportmanagerimpl::transportworkerthread+0xe7 000007fe`e51b33c8 : 00000000`00000000 00000000`1c409460 00000000`1c4a4e40 00000000`00000000 : microsoft_rtc_internal_media!ctransportmanagerimpl::transportworkerthreadproc+0x13d 000007fe`f22a3d67 : 00000000`00000000 00000000`1c409460 00000000`00000000 00000000`00000000 : rtmpal!rtcpalsetschedulerpolicy+0x194 000007fe`f22a3f0e : 000007fe`f233cdb0 00000000`00000000 00000000`00000000 00000000`00000000 : msvcr110!beginthreadex+0x107 00000000`76fd652d : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : msvcr110!endthreadex+0x192 00000000`7720c541 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : kernel32!basethreadinitthunk+0xd 00000000`00000000 : 00000000`00000000 00000000`00000000 00000000`00000000 00000000`00000000 : ntdll!rtluserthreadstart+0x1d
debugging performance issues windbg need several dumps. 1 dump snapshot in time , doesn't give complete picture.
right (at time dump taken), threads may doing nothing. sure used 100% cpu when took dump? or did recover 100% millisecond before dump taken?
the values displayed !runaway accumulated values on whole lifetime of program. tells thread has worked lot in past. doesn't tell doing or doing in future.
though has been done mark russinovich , other cracks, not thing beginners.
since need many dumps complete picture, use other tools analyze performance issues. typical tools called profilers, e.g. redgate's ants profiler or jetbrains' dottrace.
if want go hard way, @ least use procdump (sysinternals) -ma -c -n -s options collect nice high cpu dumps.
Comments
Post a Comment