The differing ways to monitor and instrument

The Differing Ways to Monitor and Instrument
FullStack London July 15th, 2016
Jonah Kowall, VP Market Development and Insights
Twitter : @jkowall

Jonah Kowall’s Background
• 20+ years in IT
• Computer Science
• FreeBSD project
• Security - CISSP, CISA, PCI
• Started one of the first content filtering companies
• Infrastructure and Operations enterprises and startups
• Ran global monitoring at Thomson Reuters
• Ran IT Operations at mfg.com manufacturing marketplace
• Gartner Research VP 3.5 years
• Strategy AppDynamics 1.5 years

Agenda
• Introduction to Instrumentation
• Instrumentation done by people
• Challenges in instrumentation of the right processes
• Technical instrumentation of Browsers and Mobile
• Technical instrumentation of Server (php, Java, .NET…)
• How to implement the right logging
• How to correlate across technologies

Definitions
Instrumentation
“The design, construction, and provision of
instruments for measurement, control, etc; the
state of being equipped with or controlled by
such instruments collectively.”
Telemetry
“Automated communications process by which
measurements are made and other data
collected at remote or inaccessible points and
transmitted to receiving equipment for
monitoring.”

Software Instrumentation Data Types
• Metrics
– Key value pairs
– Numeric values
– Time series
• Events
– Informational
– Errors
– Critical Events

Use Cases for Data Types
• Metrics
– Average, Peak
– Percentage
– Correlation to Metrics
and Events
• Events
– Search
– Parse
– Correlate to Metrics

Software Instrumentation
• Logging
– Supplied by vendor
– Created by developers
– Not easily controlled
• Push Collection
– Attach and extract
– Software agent or network tap
• Pull Collection
– Polling APIs - HTTP, SNMP, WMI

Priorities for Instrumentation
• Business
• Application
• Services
• Infrastructure
Events and Metrics

Uplevel the Conversation
• Understand the customer
- Internally and externally
• Requirements should be gathered across
business and IT teams
• Responsibility for definition of monitoring
should be shared

Business Metrics and KPIs
• Customer Metrics
- Conversion between
products
- Loyalty and retention
(churn)
- Usage metrics (feature
and product)
• Sales and Marketing
Metrics
- Revenue
- Cost of customer
acquisition
- User flows through
applications

Technical Metrics and KPIs
• End to end performance
– User through transaction hops
– Error isolation
• End user experience
– Client side errors
– Latency per element (page or app) + 3rd party
– Client side DNS
• Application component performance
– Metrics from app server
– Metrics from code
– Queries
– Errors
• Intra application component performance

Use Cases for Business and Technical Data
• Usage
• Problem identification - MTTI
• Problem resolution - MTTR
• User satisfaction
• Usability
• Performance
• Change analysis
– A/B testing
– data center moves
– technology changes

Simulating Users
Synthetic transactions are good for
• SLAs
• Availability
• Baseline performance
• DNS
• SSL
But if you try to use it as a
barometer you will fail

Backend Instrumentation of Java and .NET

Java Instrumentation
JSR-163 (JavaTM Platform Profiling
Architecture) added in Java 1.5
Overloads the default behavior of
Java to allow hooks into code for
many use cases
Since JDK 1.6, for the Oracle
HotSpot JVM, a javaagent may be
dynamically attached to a running
JVM by specifying the process-id
(pid).

.NET Instrumentation
● Profiling API loaded into the same process as the application process that is being profiled.
● Callback interface (ICorProfilerCallback in the .NET Framework version 1.0 and 1.1,
ICorProfilerCallback2 in version 2.0 and later)
● CLR calls the methods in that interface to notify the .NET agent of events in the profiled process
● Profiler can also call into the CLR by using the methods in the ICorProfilerInfo and ICorProfilerInfo2
interfaces to obtain information about the state of the profiled application
● Callbacks are used to inject MSIL (Microsoft Intermediate Language) bytecode into existing application
code for instrumentation.

Backend Instrumentation of Interpreted
Languages

Monkey Patching
Wikipedia relevant definition:
In Ruby,[3] Python,[4] and many other dynamic programming languages...
dynamic modifications of a class or module at runtime, motivated by the intent
to patch existing third-party code as a workaround to a bug or feature which
does not act as desired
Disclaimer : Can be very dangerous, hard to maintain
• Replace methods / attributes / functions at runtime
• Apply a patch at runtime to the objects in memory, instead of the
source code on disk;

PHP Instrumentation
Zend callback methods zend_execute(…), zend_execute_internal(…)
and zend_compile_file(…) so that it can wrap the original
implementations with instrumentation code.
Handles state changes
and new web server
initialization (which are
PHP instances)

Node.js Instrumentation
● Wrap methods using before, after and around aspect
interceptors.
● Callback along with after, before and around aspect interceptor.
● Notifications when asynchronous calls are complete.

Logging Best Practices
• Easily parsed (JSON)
• Time (long), Source
• Log errors and exceptions
• Logs are not transaction records, they are not good metric
stores
• Write your own identifiers for each statement logged (or
instrument and inject)
• Think about security implications (plain text, on disk,
syslog insecure)
• Keep small (thanks Java, .NET…)
• Don’t overdo it (performance implications)

End-to-end visibility of business transactions
Unified Monitoring
Copyright © 2015 AppDynamics. All rights reserved. 28
Tag Trace
Instrument every user transaction Dynamically collect application and contextual data Learn behavior and performance patterns
Learn
Infrastructure VisibilityEnd User Monitoring Application Performance Management
NoSQL

Correlation in asynchronous calls (headache)

Correlation in Open Source
Zipkin is the most advanced, many new forks and
instrumentations (Java, JavaScript, Python, Ruby, C#, Go)
No async support :(

Future of Correlation in Open Source
• OpenTracing an open standard API
for instrumentation
– Doesn’t manage overhead, can hang
yourself
• PivotTracing runs distributed traces
on demand
– After issues detected
• Spigo visualization and simulation
lots of evolution

Correlation in Logs
• Log every transaction segment
• Persist a GUID or transaction ID
• This is very difficult in large teams
• Inefficient to analyze and pull metrics from logs
• Doesn’t work unless you own the code
[code]
PERF,2013-04-03 11:29:52.640,external,0x123456,NA,service1,MyAPP,jimmy,NA,336,NA,NA
INFO,2013-04-03 11:29:53.189,internal,789012,0x123456,service2,TheirApp,jimmy,NA,174,NA,NA
INFO,2013-04-03 11:29:52.892,internal,345678,789012,service3,TheirApp,jimmy,NA,163,NA,NA
[/code]

Transaction Correlation and Logs!
• Many integrations across APM
and Log vendors
• Can add correlation in code
and use any log tool
– ex:[%X{AD.requestGUID}]
• We auto inject and correlate
(one platform)

The differing ways to monitor and instrument

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (20)

Similar to The differing ways to monitor and instrument

Similar to The differing ways to monitor and instrument (20)

Recently uploaded

Recently uploaded (20)

The differing ways to monitor and instrument

Editor's Notes