Log Better Why it’s important and how to do it well Laura Santamaria Developer Advocate @nimbinatus #logbetter @nimbinatus | #logbetter ● ● Welcome Intro yourself

@nimbinatus | #logbetter Interactive! This is a workshop!

Obligatory “Why?” slide Legal/regulatory requirements Monitoring/tracing Action! @nimbinatus | #logbetter Forensics and attack detection Service quality Validity Auditing access: PCI, HIPAA, etc. General debugging Error traces, network traffic, timestamps You need to be able to take action and fix things!

Actionable logs! Enough info Enough details Enough history @nimbinatus | #logbetter No matter what, you want

Text Logs @nimbinatus | #logbetter

Past best practices Plain language strings Tokenizable strings @nimbinatus | #logbetter Plain language in the main place you’re from Simple, short, to the point Tokens: Variables, pass in for readability

@nimbinatus | #logbetter Example. Tiny app.

@nimbinatus | #logbetter Point out UI.message()

Plain language strings @nimbinatus | #logbetter Human readable. Simple, not much space. Quick to read.

Tokenized @nimbinatus | #logbetter Human readable, added in data. Why would you do this? Pass all of the data to yourself. It’s kind of standardized, but harder to parse from machine’s POV. Mainly relying on the human to take a look.

@nimbinatus | #logbetter Try It! https://logbetter.nimbinatus.com Go to website OR can send by terminal. Curl command on website. Send hello world.

Structured Logs @nimbinatus | #logbetter Things are a bit different. Structured logging, focus of today

Best practices now Timestamp plus object Structured data Single human-readable field @nimbinatus | #logbetter Structured logs are a timestamp and an object with structured data. Why send structured data? (1) Much more helpful when machines are working with the data. (2) Faster search. (3) Send a lot more info in compact setup. Why single field? We’re human! We need that field for quick scans. Unless you’re a seasoned sysadmin who is amazing.

@nimbinatus | #logbetter

Structured objects @nimbinatus | #logbetter Logger sending structured objects to logging aggregator.

Human readable field @nimbinatus | #logbetter Still using tokenized strings!

@nimbinatus | #logbetter This is the result. Walk through.

@nimbinatus | #logbetter Try It! https://logbetter.nimbinatus.com Go to website. If CURL, explain endpoint difference. Explain JSON. Cannot send XML. Basic idea of what JSON looks like. Key-value pairs.

Practical Notes @nimbinatus | #logbetter

Levels Debug Warn Trace Error Info Critical Fatal @nimbinatus | #logbetter ● ● ● 7 common ones Explain each, how they seem the same Use what’s standard for your language

Log types System Application Audit @nimbinatus | #logbetter System level: OS, Networking, Infra <— important for prod debug. Can’t change these necessarily. App level: Dealing with multi apps. Specific to each application. This is where you can add structure for sure. Audit: Who accesses what. Special type for security.

Text vs Structured @nimbinatus | #logbetter

“But they look the same” @nimbinatus | #logbetter

“But they look the same” Only if your system flattens it @nimbinatus | #logbetter

“But I can parse text” @nimbinatus | #logbetter

“But I can parse text” That will take time when your system is generating the data anyway. @nimbinatus | #logbetter

It’s a way of thinking. @nimbinatus | #logbetter You are using logs for something other than just reading them yourself.

Remember: You are no longer the primary user @nimbinatus | #logbetter You aren’t using your logs directly, so you aren’t the primary user.

Remember: You are no longer the primary user @nimbinatus | #logbetter https://commons.wikimedia.org/wiki/File:Comic-Con_2004_-_Terminator_statue.jpg The machines are.

Why @nimbinatus | #logbetter So why structured logging?

Time Milliseconds instead of minutes @nimbinatus | #logbetter Time to parse strings vs use object. Prod debug and people watching.

Cleanliness Clear sections in the output instead of one longer line @nimbinatus | #logbetter Begin able to look at it quickly and collapse things in UIs and such are helpful.

Consistency Multi-platform. Enough said. @nimbinatus | #logbetter Working in multiple platforms, strings will be sent differently.

Misconceptions @nimbinatus | #logbetter

Logging is just printing to stdout. No. Just no. @nimbinatus | #logbetter

Log everything! Probably will cause a performance hit @nimbinatus | #logbetter Debug is different. Don’t log every single thing. Only useful things for each level. Log lines take time. This is why levels are important. If not generated because log level not needed, doesn’t take up memory, garbage collection.

Logs are automatic. You need to make logs, and you need to generate good ones. @nimbinatus | #logbetter Generating non-actionable aren’t helpful. Logs don’t magically appear. You end up with an opaque thing. Set yourself up for success.

All those log levels are unnecessary. Parse all the things! @nimbinatus | #logbetter You need to be able to get only the data that you actually need!

Just use regular expressions. @nimbinatus | #logbetter

Just use regular expressions. … regex. https://xkcd.com/1171/ @nimbinatus | #logbetter Regex sucks. With love to XKCD Any time you try to use regex, you compound your problems.

Converting Your Logs @nimbinatus | #logbetter So you understand importance of structured logging. How do you deal with a legacy system, or a team of people?

Step 1: Identify What Why @nimbinatus | #logbetter This will come up during incidents. Look for opaque systems. Understand why broken. Understand what you needed to debug it. If just exercise, pretend incident, or knock over in dev.

Step 2: Discuss All teams How Levels @nimbinatus | #logbetter Take back to everyone. Discuss what was needed, what was missing. How to fix. How to set process. What levels are defined? Bring everyone on board. Everyone needs to maintain. If not adding as you go, then what’s the point? Everyone has an opinion. They’re the people you’ll work with in the future.

Step 3: Update API Libraries Levels @nimbinatus | #logbetter Update the things. Understand what libraries and picking one out. Start to use it well. Set Levels

Step 4: Update UI Optional Minimal surface @nimbinatus | #logbetter Redoing if need be. Nice to log to client in debug mode locally. Don’t surface your logs to anyone who comes to your site, though! Provide a minimal surface for attack.

Step 5: Monitor Observations Adjustments Process @nimbinatus | #logbetter Be ready to see what happened and tweak it. Watch. Process will change.

@nimbinatus | #logbetter Try It! https://logbetter.nimbinatus.com/scenario One last interactive moment here. Go to this site and look at the scenario. Talk with your neighbors.

@nimbinatus | #logbetter What did you decide? Talk about what you figured out.

Thank you! @nimbinatus || @logdna #logbetter https://nimbinatus.com @nimbinatus @nimbinatus | #logbetter | #logging