Log Better Why it’s important and how to do it well Laura Santamaria Developer Advocate @nimbinatus #logbetter @nimbinatus | #logbetter ● ● Welcome Intro yourself
Slide 1
Slide 2
@nimbinatus | #logbetter Interactive! This is a workshop!
Slide 3
Obligatory “Why?” slide Legal/regulatory requirements Monitoring/tracing Action! @nimbinatus | #logbetter Forensics and attack detection Service quality Validity Auditing access: PCI, HIPAA, etc. General debugging Error traces, network traffic, timestamps You need to be able to take action and fix things!
Slide 4
Actionable logs! Enough info Enough details Enough history @nimbinatus | #logbetter No matter what, you want
Slide 5
Text Logs @nimbinatus | #logbetter
Slide 6
Past best practices Plain language strings Tokenizable strings @nimbinatus | #logbetter Plain language in the main place you’re from Simple, short, to the point Tokens: Variables, pass in for readability
Slide 7
@nimbinatus | #logbetter Example. Tiny app.
Slide 8
@nimbinatus | #logbetter Point out UI.message()
Slide 9
Plain language strings @nimbinatus | #logbetter Human readable. Simple, not much space. Quick to read.
Slide 10
Tokenized @nimbinatus | #logbetter Human readable, added in data. Why would you do this? Pass all of the data to yourself. It’s kind of standardized, but harder to parse from machine’s POV. Mainly relying on the human to take a look.
Slide 11
@nimbinatus | #logbetter Try It! https://logbetter.nimbinatus.com Go to website OR can send by terminal. Curl command on website. Send hello world.
Slide 12
Structured Logs @nimbinatus | #logbetter Things are a bit different. Structured logging, focus of today
Slide 13
Best practices now Timestamp plus object Structured data Single human-readable field @nimbinatus | #logbetter Structured logs are a timestamp and an object with structured data. Why send structured data? (1) Much more helpful when machines are working with the data. (2) Faster search. (3) Send a lot more info in compact setup. Why single field? We’re human! We need that field for quick scans. Unless you’re a seasoned sysadmin who is amazing.
Slide 14
@nimbinatus | #logbetter
Slide 15
Structured objects @nimbinatus | #logbetter Logger sending structured objects to logging aggregator.
Slide 16
Human readable field @nimbinatus | #logbetter Still using tokenized strings!
Slide 17
@nimbinatus | #logbetter This is the result. Walk through.
Slide 18
@nimbinatus | #logbetter Try It! https://logbetter.nimbinatus.com Go to website. If CURL, explain endpoint difference. Explain JSON. Cannot send XML. Basic idea of what JSON looks like. Key-value pairs.
Slide 19
Practical Notes @nimbinatus | #logbetter
Slide 20
Levels Debug Warn Trace Error Info Critical Fatal @nimbinatus | #logbetter ● ● ● 7 common ones Explain each, how they seem the same Use what’s standard for your language
Slide 21
Log types System Application Audit @nimbinatus | #logbetter System level: OS, Networking, Infra <— important for prod debug. Can’t change these necessarily. App level: Dealing with multi apps. Specific to each application. This is where you can add structure for sure. Audit: Who accesses what. Special type for security.
Slide 22
Text vs Structured @nimbinatus | #logbetter
Slide 23
“But they look the same” @nimbinatus | #logbetter
Slide 24
“But they look the same” Only if your system flattens it @nimbinatus | #logbetter
Slide 25
“But I can parse text” @nimbinatus | #logbetter
Slide 26
“But I can parse text” That will take time when your system is generating the data anyway. @nimbinatus | #logbetter
Slide 27
It’s a way of thinking. @nimbinatus | #logbetter You are using logs for something other than just reading them yourself.
Slide 28
Remember: You are no longer the primary user @nimbinatus | #logbetter You aren’t using your logs directly, so you aren’t the primary user.
Slide 29
Remember: You are no longer the primary user @nimbinatus | #logbetter https://commons.wikimedia.org/wiki/File:Comic-Con_2004_-_Terminator_statue.jpg The machines are.
Slide 30
Why @nimbinatus | #logbetter So why structured logging?
Slide 31
Time Milliseconds instead of minutes @nimbinatus | #logbetter Time to parse strings vs use object. Prod debug and people watching.
Slide 32
Cleanliness Clear sections in the output instead of one longer line @nimbinatus | #logbetter Begin able to look at it quickly and collapse things in UIs and such are helpful.
Slide 33
Consistency Multi-platform. Enough said. @nimbinatus | #logbetter Working in multiple platforms, strings will be sent differently.
Slide 34
Misconceptions @nimbinatus | #logbetter
Slide 35
Logging is just printing to stdout. No. Just no. @nimbinatus | #logbetter
Slide 36
Log everything! Probably will cause a performance hit @nimbinatus | #logbetter Debug is different. Don’t log every single thing. Only useful things for each level. Log lines take time. This is why levels are important. If not generated because log level not needed, doesn’t take up memory, garbage collection.
Slide 37
Logs are automatic. You need to make logs, and you need to generate good ones. @nimbinatus | #logbetter Generating non-actionable aren’t helpful. Logs don’t magically appear. You end up with an opaque thing. Set yourself up for success.
Slide 38
All those log levels are unnecessary. Parse all the things! @nimbinatus | #logbetter You need to be able to get only the data that you actually need!
Slide 39
Just use regular expressions. @nimbinatus | #logbetter
Slide 40
Just use regular expressions. … regex. https://xkcd.com/1171/ @nimbinatus | #logbetter Regex sucks. With love to XKCD Any time you try to use regex, you compound your problems.
Slide 41
Converting Your Logs @nimbinatus | #logbetter So you understand importance of structured logging. How do you deal with a legacy system, or a team of people?
Slide 42
Step 1: Identify What Why @nimbinatus | #logbetter This will come up during incidents. Look for opaque systems. Understand why broken. Understand what you needed to debug it. If just exercise, pretend incident, or knock over in dev.
Slide 43
Step 2: Discuss All teams How Levels @nimbinatus | #logbetter Take back to everyone. Discuss what was needed, what was missing. How to fix. How to set process. What levels are defined? Bring everyone on board. Everyone needs to maintain. If not adding as you go, then what’s the point? Everyone has an opinion. They’re the people you’ll work with in the future.
Slide 44
Step 3: Update API Libraries Levels @nimbinatus | #logbetter Update the things. Understand what libraries and picking one out. Start to use it well. Set Levels
Slide 45
Step 4: Update UI Optional Minimal surface @nimbinatus | #logbetter Redoing if need be. Nice to log to client in debug mode locally. Don’t surface your logs to anyone who comes to your site, though! Provide a minimal surface for attack.
Slide 46
Step 5: Monitor Observations Adjustments Process @nimbinatus | #logbetter Be ready to see what happened and tweak it. Watch. Process will change.
Slide 47
@nimbinatus | #logbetter Try It! https://logbetter.nimbinatus.com/scenario One last interactive moment here. Go to this site and look at the scenario. Talk with your neighbors.
Slide 48
@nimbinatus | #logbetter What did you decide? Talk about what you figured out.
Slide 49
Thank you! @nimbinatus || @logdna #logbetter https://nimbinatus.com @nimbinatus @nimbinatus | #logbetter | #logging
