Building Custom Big Data Integrations

Building Custom Big Data
Integrations
Pat Patterson
Community Champion
@metadaddy
pat@streamsets.com

Agenda
Ingest, Data Drift and StreamSets
Short Demo
Building a custom integration
Real-world integration: Salesforce Wave Analytics

Traditional and Big Data
Founders
Company Background
Top tier Investors
Momentum to Date
Strategic Partners
● Launched 2014; exited stealth 9/15
● ~30 employees
● Double-digit enterprise customers
● 10,000 downloads

Past ETL ETL
Emerging Ingest Analyze
Data Sources Data Stores Data Consumers
Market Trends

Problem: Data Drift
The unpredictable, unannounced and unending mutation of data characteristics caused
by the operation, maintenance and modernization of the systems that produce the data
Structure
Drift
Semantic
Drift
Infrastructure
Drift

Delayed and
False Insights
Solving Data Drift
Tools
Applications
Data Stores Data ConsumersData Sources
Poor Data QualityData Drift
Custom code
Fixed-schema

Trusted InsightsData KPIs
Solving Data Drift
Tools
Applications
Data Stores Data ConsumersData Sources
Data Drift
Intent-Driven
Drift-Handling

Demo
Let’s build a simple pipeline to answer a real
question:
What’s the biggest city lot in San Francisco?

Customizing StreamSets
Currently 25 standard StreamSets destinations,
covering a wide variety of target systems, from flat
files to S3 to Kafka
But… there’s always some system not on the list
Solution: DIY!

Create Your Own Destination
Five Step Process:
○ Create template from Maven archetype
○ Add logging
○ Create a record buffer
○ Add configuration parameters
○ Send data to external system
bit.ly/sdc-dest
Your
System
Here!

Create Template from Archetype
mvn archetype:generate
-DarchetypeGroupId=com.streamsets -DarchetypeArtifactId=streamsets-
datacollector-stage-lib-tutorial
-DarchetypeVersion=1.3.0.0
-DinteractiveMode=true

Add Logging
Not 100% necessary, but VERY helpful
StreamSets uses SLF4J
$ tail -f streamsets-datacollector-1.3.0.0/log/sdc.log

Create a Record Buffer
Leverage existing code where possible!
StreamSets includes generators for CSV, JSON, Avro,
Protocol Buffers etc

Configuration
Separate configuration and code
DON’T PUT CREDENTIALS IN CODE!!!
DON’T PUT CREDENTIALS IN CODE!!!
Make your users’ and your lives easier!

Send Data to the External System
Don’t forget security policy!
streamsets-datacollector/etc/sdc-security.policy
grant codebase "file://${sdc.dist.dir}/user-libs/sampletest/-" {
permission java.net.SocketPermission "requestb.in", "connect, resolve";
};

A Real Custom Destination
Salesforce Wave Analytics
● Adapt to batch processing model
○ Configure wait time before ‘closing’ a batch
● External Data API
○ Create new dataset
○ Write to dataset
○ Close dataset on timeout
○ Trigger dataflow execution

Conclusion
StreamSets Data Collector makes simple tasks easy,
complex tasks possible
Use ‘off the shelf’ stages for simple tasks
Leverage script processors (Jython, JavaScript, Groovy) for
more complex work
Build custom stages for ultimate performance, flexibility

Structure
Drift
Data structures and
formats evolve and
change unexpectedly
Implication:
Data Loss
Data Squandering
Delimited
Data
107.3.137.195 fe80::21b:21ff:fe83:90fa
Attribute Format
Changes
{
“first“: “jon”
“last“: “smith”
“email“: “jsmith@acme.com”
“add1“: “123 Washington”
“add2“: “”
“city“: “Tucson”
“state“: “AZ”
“zip“: “85756”
}
{
“first“: “jane”
“last“: “smith”
“email“: “jane@earth.net”
“add1“: “456 Fillmore”
“add2“: “Apt 120”
“city“: “Fairfield”
“state“: “VA”
“zip“: “24435-1001”
“phone”: “401-555-1212”
}
Data Structure Evolution
Structure Drift

Semantic
Drift
Data semantics change
with evolving
applications
Implication:
Data Corrosion
Data Loss
Semantic Drift
24122-52172 00-24122-52172
Account Number Expansion
M134: user {jsmith} read access granted {ac:24122-52172}
M134: user {jsmith} read access granted {ca.ac:24122-52172}
Namespace Qualification
……
…,3588310669797950,$91.41,jcb,K1088-W#9,…
…,6759006011936944,$155.04,switch,A6504-Y#9,…
…,6771111111151415,$37.78,laser,Q9936-T#9,…
…,3585905063294299,$164.48,jcb,S4643-H#9,…
…,5363527828638736,$117.52,mastercard,X3286-P#9,…
…,4903080150282806,$168.03,switch,I9133-W#3,…
……
Outlier / Anomaly Detection

Infrastructure
Drift
Physical and Logical
Infrastructure changes
rapidly
Implication:
Poor Agility
Operational Downtime
Data Center 1 Data Center 2 Data Center n
3rd
Party Service Provider
App a App k
App q
Cloud
Infrastructure
Infrastructure Drift

Building Custom Big Data Integrations

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (11)

Similar to Building Custom Big Data Integrations

Similar to Building Custom Big Data Integrations (20)

More from Pat Patterson

More from Pat Patterson (20)

Recently uploaded

Recently uploaded (20)

Building Custom Big Data Integrations