Righting a Sinking Ship: Troubleshooting Systems with Available Data

A presentation at Everything DevOps Meetup in October 2019 in Mountain View, CA, USA by Laura Santamaria

Slide 1

Righting a Sinking Ship Troubleshooting systems with available data Laura Santamaria Developer Advocate @nimbinatus #righttheship @nimbinatus | #righttheship

Slide 2

http://www.defensie.nl/media/In_vuur_en_vlam_tcm46-102834.jpg https://www.flickr.com/photos/nzdefenceforce/6386334175/ @nimbinatus | #righttheship

Slide 3

Now What? @nimbinatus | #righttheship

Slide 4

Naked Gun courtesy of Giphy: https://gph.is/2kvXCEp @nimbinatus | #righttheship

Slide 5

Step 1: Don’t Panic @nimbinatus | #righttheship

Slide 6

90% Perseverance and Patience; 10% Luck Maybe some knowledge sprinkled on @nimbinatus | #righttheship

Slide 7

Step 2: Get Data @nimbinatus | #righttheship

Slide 8

If you’re lucky Logs! Alerts Monitoring and historical data @nimbinatus | #righttheship

Slide 9

$ journalctl -xe $ docker logs <container> $ kubectl logs <object> Commands $ minikube logs $ oc logs -f <object> Depends on system @nimbinatus @nimbinatus | #righttheship | #logging

Slide 10

@nimbinatus | #righttheship

Slide 11

If you’re not lucky Hit and observe Test envs Brute force @nimbinatus | #righttheship

Slide 12

Step 3: Analyze @nimbinatus | #righttheship

Slide 13

Evaluation tools History (Generated) Analytics Your brain @nimbinatus | #righttheship

Slide 14

@nimbinatus | #righttheship

Slide 15

Step 4: Act @nimbinatus | #righttheship

Slide 16

@nimbinatus | #righttheship Wait!

Slide 17

Try it! Run command Observe @nimbinatus | #righttheship

Slide 18

Step 5 (optional): Fail @nimbinatus | #righttheship

Slide 19

Common Reasons The rabbit hole Multiple causes Red herrings @nimbinatus | #righttheship

Slide 20

@nimbinatus | #righttheship You can still fix it!

Slide 21

Step 6: Repeat as Necessary @nimbinatus | #righttheship

Slide 22

Common Container Issues @nimbinatus | #righttheship

Slide 23

Code Releases Incompatibility Dependencies @nimbinatus | #righttheship

Slide 24

Configurations Errors Missing data @nimbinatus | #righttheship

Slide 25

Networking DNS Routing Virtual ethernet @nimbinatus | #righttheship

Slide 26

External Physical network OS Updates Physical changes @nimbinatus | #righttheship

Slide 27

http://www.photolib.noaa.gov/coastline/line0534.htm @nimbinatus | #righttheship

Slide 28

Thank you! @nimbinatus || @logdna #righttheship https://nimbinatus.com @nimbinatus @nimbinatus | #righttheship | #logging