.NET Developer Days 2017: The Performance Investigator's Field Guide – by Sasha Goldshtein

Although the choice of the best session was not easy, it has to be awarded to Sasha Goldstein with his session on performance detective work. His session was very well prepared, had a clear goal and path, was packed with insights into challenges and problems during performance investigation work, and offered hand-crafted solutions like his own realtime ETW event tracing tool etrace.

What follows is an excerpt of Sasha Goldshtein's presentation.

Structure of a Performance Investigation

1. Obtain the problem description
2. Build a system diagram
3. Run a quick performance checklist
4. Understand which component is exhibiting the problem
5. Investigate thoroughly
6. Find the root cause
7. Resolve the issue
8. Verify resolution
9. Conduct and document post-mortem

Performance Metrics, Goals, Monitoring

• Performance metrics don’t live in a vacuum!
• Derive performance metrics from business goals
• Monitor these metrics in your APM solution, home-made dashboard, or collection script, and get alerts

Investigation Anti-Methods

1. Make assumptions
2. Trust “instincts” and irrational beliefs
3. Look under the street light
4. Use random tools
5. Blame the tools

The USE Method

USE: Utilization, Saturation, Errors

1. Build a functional diagram of the system, including hardware/software resources
2. For each resource, identify utilization, saturation, and errors
3. Understand, resolve, and verify errors, excessive saturation/utilization, under-utilization

Statstics Lie Be Careful With Statistics

• Averages are meaningless
• Medians are almost meaningless
• Percentiles are OK if you know what you’re doing
• Find good visualizations for your performance data
• Beware coordinated omission

Look at histograms or sometimes even percentile plots aka cumulative distribution charts to really understand your data, e.g. your performance traces. Just look at this dinosaur to understand that very differently shaped data can lead to the same statistics values:

Conduct a Postmortem – Do It!

1. Document the steps taken to identify, diagnose, resolve, and verify the problem
2. Which tools did you use? Can they be improved?
3. Where were the bottlenecks in your investigation?
4. Can you add monitoring for sysadmins/ops?
5. Can you add instrumentation for investigators?
6. How do we triage this problem automatically next time it happens?

Resources

• etrace: a real-time, command-line frontend for ETW events –> https://github.com/goldshtn/etrace
• LiveStacks: a real-time, command-line stack collector and resolver –> https://github.com/goldshtn/LiveStacks
• BenchmarkDotNet is a powerful .NET library for benchmarking: http://benchmarkdotnet.org/
• HdrHistogram: A High Dynamic Range (HDR) Histogram –> https://github.com/HdrHistogram/HdrHistogram
• Slides: How NOT to Measure Latency: https://www.azul.com/files/HowNotToMeasureLatency_LLSummit_NYC_12Nov2013.pdf
• Blog post: Windows Process Memory Usage Demystified
• Course: Statistics for Engineers – https://github.com/HeinrichHartmann/Statistics-for-Engineers (use http://nbviewer.jupyter.org/ to view the ipynb files)

And now, happy performance hunting!