LC9 - Citrix

LC9 - Windows Performance-Analyse
und der praktische Einsatz von Xperf
Karl Hirschberger, Support Escalation Engineer, Microsoft Deutschland GmbH
Frank Hammer, Escalation Engineer, Citrix Systems GmbH
Xperf introduction
• Official kit for windows performance analysis,
public avail. Microsoft Windows SDK for Windows 7
• Cross platform: Vista / Server 2008 / Windows 7 / Windows Server 2008 R2
• No extra drivers needed/ no hooks in the system to affect system performance
very low overhead: 2.5% CPU usage for sustained rate
of 10,000 events/sec on a 2GHz CPU
• provides a view of system activity at code level
Xperf introduction
• Lack of responsiveness
• Long delays in applications
• High CPU usage
• High disk usage
• Slow on/off transitions
Xperf introduction
• Data provided by Event Tracing for Windows (ETW)
• Event Tracing for Windows (ETW) is an efficient kernel-level tracing facility that
lets you log kernel or application-defined events to a log file. You can consume
the events in real time or from a log file and use them to debug an application
or to determine where performance issues are occurring in the application.
• ETW lets you enable or disable event tracing dynamically, allowing you to
perform detailed tracing in a production environment without requiring
computer or application restarts.
ETW in the GUI
• In Perfmon, we can see the Event Trace sessions that are started or stopped
Xperf architecture
• Many components in the system (providers) are told (by controllers) to bubble
data about themselve to controllers. The event tracing sessions could be
catched in log files.
Xperf installation
• Web Download Win7 SDK:
http://go.microsoft.com/fwlink/?LinkID=191420
Xperf installation
Windows Performance Analysis Tools
http://msdn.microsoft.com/en-us/performance/cc825801.aspx
Tool Name
File Name
Function
Trace Capture, Processing, and CommandLine Analysis tool
Xperf.exe
Captures traces, post-processes them for use on any machine, and supports
command-line (action-based) trace analysis.
http://msdn.microsoft.com/en-us/library/ff191077(v=VS.85).aspx
Visual Trace Analysis tool
Xperfview.exe
Presents trace content in the form of interactive graphs and summary tables.
http://msdn.microsoft.com/en-us/library/ff191075(v=VS.85).aspx
On/Off Transition Trace Capture tool
Xbootmgr.exe
Automates on/off state transitions and captures traces during these
transitions.
http://msdn.microsoft.com/en-us/library/ff190947
Xperf tracing
• Start kernel trace
• Run scenario
• Stop and merge kernel trace
• Hint: You can retrieve all known kernel flags and groups with
Xperfview
• four different types of graphs
+ Usage graph
+ Histogram graph
+ Lifetime graph
+ Event graph
Xperfview usage graph
• CPU Sampling by CPU / CPU Sampling by Process / CPU Sampling by Thread / CPU Usage by CPU /
CPU Usage by Process / CPU Usage by Thread / CPU Idle States / CPU Frequency / DPC CPU
Usage / Disk Utilization / Disk Utilization by Process / Hard Faults / Interrupt CPU Usage
Generic Events / Memory Utilization / Pool/Heap/Virtual Outstanding Allocation Size
Pool/Heap/Virtual Outstanding Allocation Count / Pool/Heap/Virtual Total Allocation Size
Pool/Heap/Virtual Total Allocation Count / Stack Counts by Type
Xperfview histogram graph
• Disk I/O
ReadyBoot I/O
Xperfview lifetime graph
• Boot Phases / Driver Delays / CPU Scheduling / Marks
PerfTrack / Process Lifetimes / Services / Winlogon
Xperfview event graph
• Stack Counts by Type
Windows Thread in Focus
Xperf Demo
• Xperf Demo
Xperf Stackwalking
• Set disablepagingexecutive to 1
=> is needed when running stackwalking option
http://blogs.msdn.com/b/pigscanfly/archive/2009/08/06/stack-walking-in-xperf.aspx
Xperf Symbols
• Xperfviewer => check “load symbols” and configure symbols path
Xperf Symbols
• symbols
• no symbols
Xperf other important switches
• To limit the filesize use circular log syntax:
-MaxFile 1024 -FileMode Circular
• circular log syntax combined with increased buffer size
-buffersize 1024 –Maxbuffers 1024 -MaxFile 1024 -FileMode Circular
Xperf On/Off Transition
• Windows On/Off Transition Performance Analysis
http://msdn.microsoft.com/de-at/windows/hardware/gg463386
Boot process
begins
BIOS passes control
to winload.exe
BIOSInitialization
Desktop reports
itself “ready”
Winload.exe passes
control to kernel
OSLoader
MainPathBoot
System
reasonably idle
PostBoot
Time
Personal Top 5 > No. 5
• Working set trimming issue => server unresponsive for 2min
Xperf tools
• XperfUI
(Tool to create Boot Trace Sessions)
http://xperfui.codeplex.com/
Xperf tools
• Xperf 1 – 2 – 3
http://xperf123.codeplex.com/
Personal Top 5 > No. 5
• Xperf trace
xperf.exe -on PROC_THREAD+LOADER+MEMINFO+CONTMEMGEN -stackwalk
ContiguousMemoryGeneration -f kernel.etl -BufferSize 1024 -MaxBuffers 1024 -MaxFile
1024 -FileMode Circular
• based on
956341 A Windows Server 2003-based computer becomes unresponsive because of a large memory allocation request
http://support.microsoft.com/default.aspx?scid=kb;en-US;956341
2001745 Working Set Trimming can negatively impact SQL, Exchange, and Operating System performance under Windows 2003
http://support.microsoft.com/default.aspx?scid=kb;en-US;2001745
Personal Top 5 > No. 5
Personal Top 5 > No. 4
• Server unresponsive – whole memory is used by some processes
Different processes allocates 20GB memory:
TrustedInstaller + Svchost + MonitoringHost + WMI
Personal Top 5 > No. 4
• xperf -on Base -start heapSession -stackwalk
HeapCreate+HeapAlloc+HeapRealloc+HeapFree+HeapDestroy -heap -pids
<process pid>
=> [repro heap leak]
=> xperf -stop "NT Kernel Logger" -stop heapSession -d heap.etl
Personal Top 5 > No. 4
Personal Top 5 > No. 4
Personal Top 5 > No. 4
• Damaged repository => rebuild repository
Personal Top 5 > No. 3
• Boot times from a new notebook is really poor
=> 40 seconds are too slow
=> Solid-State-Drive (SDD) installed
• Boot trace
xbootmgr -trace boot -traceFlags BASE+CSWITCH -resultPath C:\TEMP
Personal Top 5 > No. 3
• Windows On/Off Transition Performance Analysis
http://msdn.microsoft.com/de-at/windows/hardware/gg463386
Boot process
begins
BIOS passes control
to winload.exe
BIOSInitialization
Desktop reports
itself “ready”
Winload.exe passes
control to kernel
OSLoader
MainPathBoot
System
reasonably idle
PostBoot
Time
Personal Top 5 > No. 3
• Good
• Bad
Personal Top 5 > No. 3
Personal Top 5 > No. 2
• 80% Cpu usage on one Core after each start
Personal Top 5 > No. 2
• Xperf.exe –on Latency –stackWalk Profile
Personal Top 5 > No. 2
• USB / Interrupts / C1 Halts / Idle thread
=> C1 transitions => CPU change to C1 when idle and changes this state in
case of an Interrupt
Personal Top 5 > No. 2
• Interrupts from USB without any USB device connected
=> Solution => BIOS update
Personal Top 5 > No. 1
• Application slow response
Task within a database application runs on
Hardware 1 within 5 seconds
Hardware 2 within 25 seconds.
• Hardware Vendor has checked hardware – no issues found
• Xperf –on diageasy
Personal Top 5 > No. 1
Personal Top 5 > No. 1
• http://www.intel.com/support/processors/sb/cs-028739.htm
Personal Top 5 > No. 1
• Solution: Disable CPU C3-States
http://support.microsoft.com/kb/2207548
Degraded overall performance
on Windows Server 2008 R2
My No. 1 Terminal Server issue
• Terminal Server doesn't react (freezes) for couple of seconds,
several times each day
• Xperf – on diageasy
My No. 1 Terminal Server issue
• 100% disk utilization at this time
My No. 1 Terminal Server issue
My No. 1 Terminal Server issue
• $MFT contains not only links
=> although files with 512 bytes or less are stored in the $MFT
• Solution:
Group Policy preferences client-side extension hotfix rollup: May 2010
http://support.microsoft.com/kb/982051
Links
• Event Tracing for Windows
http://msdn.microsoft.com/en-us/library/ms751538.aspx
• About Event Tracing
http://msdn.microsoft.com/en-us/library/aa363668(v=VS.85).aspx
• Win7 SDK:
http://go.microsoft.com/fwlink/?LinkID=191420
• 2 minute drill from Askperf:
http://blogs.technet.com/b/askperf/archive/2008/06/27/an-intro-to-xperf.aspx
Links
• Windows Performance Analysis Tools
http://msdn.microsoft.com/en-us/performance/cc825801.aspx
• Windows Performance Tool Kit
http://www.winfaq.de/faq_html/Content/tip2500/onlinefaq.php?h=tip2667.htm
• Captures traces, post-processes them for use on any machine, and supports command-line (actionbased) trace analysis
http://msdn.microsoft.com/en-us/library/ff191077(v=VS.85).aspx
• Presents trace content in the form of interactive graphs and summary tables.
http://msdn.microsoft.com/en-us/library/ff191075(v=VS.85).aspx
Links
• Automates on/off state transitions and captures traces during these transitions.
http://msdn.microsoft.com/en-us/library/ff190947
• Stack Walking in Xperf
http://blogs.msdn.com/b/pigscanfly/archive/2009/08/06/stack-walking-in-xperf.aspx
• Windows On/Off Transition Performance Analysis
http://msdn.microsoft.com/de-at/windows/hardware/gg463386
• Diagnose High CPU on Windows with XPerf
http://www.youtube.com/watch?v=vyabPTXHYb8
Citrix Scout 2.2.2.0
• Download here: http://support.citrix.com/article/CTX130147
• Collection of environmental data
• Able to run CDF Traces with this tool
• Should be used in any case opened with Technical Support as a reference –
will avoid asking basic questions over and over
• Lets have a look
Citrix Scout 2.2.2.0
• Demo
Allgemeine Informationen
Besuchen Sie die Partner in der Ausstellung
Nutzen Sie unsere Zusatzangebote!
• Citrix Expert Desks: Unsere Produkt-Spezialisten beantworten Ihre individuellen
Fragen und geben Ihnen Einblick in aktuelle Projekte
• Citrix Tech Lounge: Lernen Sie die wichtigsten Funktionen von Citrix XenClient live
kennen - bei einem Hands-On-Test in unserer Tech Lounge
• Meet the Architects: Buchen Sie an der Info einen Kurz-Workshops mit CitrixConsulting und erarbeiten Sie eine Zielarchitektur für Ihr Unternehmen
• Citrix Datentankstelle: Lassen Sie sich auf Ihren mobilen Endgeräten einen
Citrix Receiver mit Demozugang einrichten
• Citrix Education Desk: Informieren Sie sich über die aktuellen Trainingsangebote
• Citrix Test Center: Die Plätze sind ausgebucht. Es besteht die Möglichkeit über die
Warteliste noch kurzfristig einen Platz zu bekommen
Feedback und Präsentationen
• Ihre Meinung ist uns wichtig! Bitte nehmen Sie sich einige Minuten Zeit,
unseren Online Feedbackbogen auszufüllen. Den Link dazu erhalten Sie einige
Tage nach der Veranstaltung
• Im Anschluss an den Fragebogen haben Sie Zugriff auf die Downloadseite der
Präsentationen
Bitte vormerken: Citrix Synergy 2012
• The premier event on cloud
computing, virtualization and
networking
• 17.- 19. Oktober 2012 im
International Convention Centre
Barcelona
• Weitere Infos:
http://www.citrixsynergy.com/barcelona
Work better. Live better.