Data Pipeline for The Big Data/Data Science OKC

DATA PIPELINE
A set of data processing elements connected in series, where the output of one element is the
input of the next one

WHAT IS A DATA PIPELINE
The data we we most care about in one place, in the same format, consistency,
quality and real time
We don’t have to do it every time we want to build analysis or predictive
models.
It is reproducible
You will deﬁne what, where and how data are collected, transformed and loaded
This means that you’ll need to manually pick every ﬁeld, table, data source,
transformation, join, etc.The good news is that if you do it right you’ll have to
do it just once.Afterwards everything will be automated.

CANDIDATES I EVALUATED
Spark Structured Streaming
Apache NIFI
Spring Cloud Data Flow
AWS Glue

SPARK STRUCTURED STREAMING
Scalable and fault-tolerant Stream processing engine built on the
Spark SQL Engine
Express your streams computation in the same way you express
batch computation on static data.
The stream runs incrementally and continuously and updating the
ﬁnal result as streaming data continues to arrive.

APACHE NIFI
• Lends well to visual creation and management of directed graphs of processors
• Is inherently asynchronous which allows for very high throughput and natural buffering
even as processing and flow rates fluctuate
• Provides a highly concurrent model without a developer having to worry about the
typical complexities of concurrency
• Promotes the development of cohesive and loosely coupled components which can
then be reused in other contexts and promotes testable units
• The resource constrained connections make critical functions such as back-pressure
and pressure release very natural and intuitive
• Error handling becomes as natural as the happy-path rather than a coarse grained catch-
all
• The points at which data enters and exits the system as well as how it flows through
are well understood and easily tracked

SPRING CLOUD DATA FLOW
• The Data Flow server is also responsible for:
• Interpreting and executing a stream DSL that describes the logical
ﬂow of data through multiple long-lived applications.
• Launching a long-lived task application.
• Interpreting and executing a composed task DSL that describes the
logical ﬂow of data through multiple short-lived applications.
• Applying a deployment manifest that describes the mapping of
applications onto the runtime - for example, to set the initial number
of instances, memory requirements, and data partitioning.
• Providing the runtime status of deployed applications.

SPRING CLOUD DATA FLOW
• The Data Flow Server deploys applications onto the target runtime
that conform to the microservice architectural style.
• A stream represents a high-level application that consists of multiple
small microservice applications each running in their own process.
• Each microservice application can be scaled up or down
independently of the other and each has its own versioning lifecycle.
• Both Streaming and Task-based microservice applications build upon
Spring Boot as the foundational library.
• This gives all microservice applications functionality such as health
checks, security, conﬁgurable logging, monitoring, and management
functionality, as well as executable JAR packaging.

AWS GLUE: BUILDYOUR DATA
CATALOG

AWS GLUE: GENERATE AND EDIT
TRANSFORMATIONS

AWS GLUE: SCHEDULE AND RUN
YOUR JOBS

AWS GLUE: UNIFIEDVIEW OF DATA
ACROSS MULTIPLE DATA STORES

AWS GLUE: QUERY AGAINST S3
DATA LAKE

AWS GLUE: EVENT-DRIVEN ETL
PIPELINES

EVALUATION
valuation
Criteria
Spark Structured
Stream
Spring Cloud Data Flow Apache NIFI AWS Glue
Real Time PRO

the system runs in near real
time and can check point it
status
PRO

runs data from point to point
across a pre deﬁned acyclic
graph of microservices in
near real time
PRO

probably the best here
CON

Glue uses the basic spark
sql engine and each job
must be scheduled or
triggered. It could be setup
to run every x minutes.
Manages
updating data
PRO

The spark writing
infrastucture is very
sophisticated for writing to
large data systems like AWS
S3 or Hadoop HDFS. It has
been extended here for
managing the updating
nature of a realtime system.
CON

Managed by code you write
in Java or Groovy
PRO CON

Spark has to be managed
eﬀectively to over write a
dataset.
Manage Stops
and Restarts
PRO

This system has a
checkpointing system that
allows the system to stop
and restart and remember
what has been processed
and what has not. It make
sure duplicates are not
created
CON

Managed by code or
another 3rd party system
you must use
PRO CON

This system runs each job

EVALUATION
valuation
Criteria
Spark Structured
Stream
Manage
Failures and
Duplicates
PRO

Checkpointing
CON

Write it yourself or use 3rd
party system
PRO CON
Secure PRO PRO CON

Uses Kerberos. Can be
somewhat secure.
PRO

All the security any AWS
and Spark/Hadoop system
would have in AWS
Manage Large
Data Sets
PRO

All the power of Spark's
RDD and Spark Sql and ML
Lib World class system
CON

Diﬃcult to code or require
some kind of third party
system that would work in
Sping Boot and a Cluster of
user like Yarn or Cloud
Foundry
CON

While the system seems
scalable, the data must be
sent in as a descreet unit to
scale and not rely on joining
or groups across multiple
systems
PRO

All the power of Spark
Scale out using
Yarn, Mesos

and/or Cloud
Foundry
PRO

Yarn and Meso
PRO

Yarn, Meso and Cloud
Foundry
CON

This seems to manage it's
load itself you just start it up
on other computers. Their is
an open JIRA task for NIFI
to use Cloud Foundry
PRO

Yarn Meso

EVALUATION
valuation
Criteria
Spark Structured
Stream
Maturity of
Product
CON

Alpha product but has been
very reliable for the last two
months of use
PRO PRO

started as a NSA product
and then open sourced.
Horton Works is investing in
the Apache project
CON

Spark dataframes and RDDs
are mature but the
extensions of the data
catalog and glue api added
are still in Beta
Extraction PRO

Spark extraction is mature
but Structured Streaming
does not support the full
power of what SPARK is
currently supporting. It is a
pro because of it's support
of S3 and Kafka
CON

Must be programmed or 3rd
party product used
PRO

Work with Kafka not sure if
Kinesis. Not sure if it works
with S3
PRO

Uses the power of Spark
but also uses the AWS Glue
Catalog where data sources
and their structure can be
crawled and then recreated
in Spark python. Very easy
to use data sources with the
Glue API
Transformation PRO

The power of Spark
CON

Managed by coding or 3rd
party product
PRO/CON

The purpose of the product
is ingestion, Transformation
is possible but not sure how
sophisticated
PRO

GLUE is an ETL product

EVALUATION
valuation
Criteria
Spark Structured
Stream
Load CON

This part is Alpha but will
catch up with the spark
architecture that is very
strong
CON

party product
PRO

the key purpose of the
product
PRO

GLUE is an ETL product
Machine
Learning
PRO

Power of Spark
CON

party product
CON(with spark PRO)

This system can put a spark
streaming or structured
streaming job in niﬁ to
support ML jobs within a
NIFI pipeline
PRO

All the power of spark but
not part of the Glue
infrastrucure
Machine
Learning on
Large Datasets
PRO

Integrates with the SPARK
ML Lib
CON

party product
CON(with spark PRO)

This system can put a spark
streaming or structured
streaming job in niﬁ to
support ML jobs within a
NIFI pipeline
PRO

All the power of spark but
not part of the Glue
infrastrucure

EVALUATION
valuation
Criteria
Spark Structured
Stream
Data Catalog
and integration
to AWS
services
CON

would need integration with
3rd party
CON

party product
CON PRO

The AWS Glue Data catalog
can automatically update
itself, be used to create
Spark Dataframes and
integrate with AWS Athena
and AWS Redshift. This
product is still young but it's
usefulness is already
impressive. More evaluation
of Data Catalog products
should be evaluated.
Count PRO 11

CON 3
PRO 4

CON 10
PRO 8 or 10

CON 6 or 4
PRO 9

CON 5

Data Pipeline for The Big Data/Data Science OKC

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Data Pipeline for The Big Data/Data Science OKC

Similar to Data Pipeline for The Big Data/Data Science OKC (20)

More from Mark Smith

More from Mark Smith (10)

Recently uploaded

Recently uploaded (20)

Data Pipeline for The Big Data/Data Science OKC