Righting a Sinking Ship: Troubleshooting Systems with Available Data
A presentation at Everything DevOps Meetup in in Mountain View, CA, USA by Laura Santamaria
Righting a Sinking Ship Troubleshooting systems with available data Laura Santamaria Developer Advocate @nimbinatus #righttheship @nimbinatus | #righttheship
http://www.defensie.nl/media/In_vuur_en_vlam_tcm46-102834.jpg https://www.flickr.com/photos/nzdefenceforce/6386334175/ @nimbinatus | #righttheship
Now What? @nimbinatus | #righttheship
Naked Gun courtesy of Giphy: https://gph.is/2kvXCEp @nimbinatus | #righttheship
Step 1: Don’t Panic @nimbinatus | #righttheship
90% Perseverance and Patience; 10% Luck Maybe some knowledge sprinkled on @nimbinatus | #righttheship
Step 2: Get Data @nimbinatus | #righttheship
If you’re lucky Logs! Alerts Monitoring and historical data @nimbinatus | #righttheship
$ journalctl -xe $ docker logs <container> $ kubectl logs <object> Commands $ minikube logs $ oc logs -f <object> Depends on system @nimbinatus @nimbinatus | #righttheship | #logging
@nimbinatus | #righttheship
If you’re not lucky Hit and observe Test envs Brute force @nimbinatus | #righttheship
Step 3: Analyze @nimbinatus | #righttheship
Evaluation tools History (Generated) Analytics Your brain @nimbinatus | #righttheship
@nimbinatus | #righttheship
Step 4: Act @nimbinatus | #righttheship
@nimbinatus | #righttheship Wait!
Try it! Run command Observe @nimbinatus | #righttheship
Step 5 (optional): Fail @nimbinatus | #righttheship
Common Reasons The rabbit hole Multiple causes Red herrings @nimbinatus | #righttheship
@nimbinatus | #righttheship You can still fix it!
Step 6: Repeat as Necessary @nimbinatus | #righttheship
Common Container Issues @nimbinatus | #righttheship
Code Releases Incompatibility Dependencies @nimbinatus | #righttheship
Configurations Errors Missing data @nimbinatus | #righttheship
Networking DNS Routing Virtual ethernet @nimbinatus | #righttheship
External Physical network OS Updates Physical changes @nimbinatus | #righttheship
http://www.photolib.noaa.gov/coastline/line0534.htm @nimbinatus | #righttheship
Thank you! @nimbinatus || @logdna #righttheship https://nimbinatus.com @nimbinatus @nimbinatus | #righttheship | #logging
Ever been stuck with a system that just can’t heal? A system that falls over? Let’s dig into where you can gather data from a broken system, how you can figure out what’s happening using that data, and common trouble spots that might be hidden in that data for you to find.
Buzz and feedback
Here’s what was said about this presentation on social media.
