The Missed Promise of Hadoop and New and Emerging Technologies

The First Step in Information Management
looker.com
Produced by:
MONTHLY SERIES
In partnership with:
The Missed Promise of Hadoop and
New and Emerging Technologies
August 2, 2018

Welcome to Today’s Discussion
 Evolution of Hadoop
 Current state of Hadoop: pros and cons for big data and analytics
 Role of Hadoop in enterprise architectures
 Successful use cases and lessons learned
 Hadoop alternatives
 Best practices and key takeaways
 Q&A
pg 2© 2018 First San Francisco Partners www.firstsanfranciscopartners.com

Evolution of Hadoop
 15 years – Hadoop is firmly entrenched in the landscape
 Hadoop in the cloud means it’s transparent
 Commoditization
 The use of Hadoop seems to be as the landing zone
 Not mission-critical but still relevant – and still maintains some of its long-term issues
MANAGED
CLOUD
SERVICES
SERVERLESS
MICROSERVICES
VENDOR
SOLUTIONS
PART OF
ORDINARY
ENTERPRISE
ARCHITECTURE
FABRIC

Scope of Hadoop for Analytics
 Data at rest
 Permanent fixture in enterprise architectures
 Will relational ever go away for BI? – No
 Likewise, Hadoop won’t go away for
departmental analytics of data at rest

Analyst Viewpoints on Hadoop
 Well-entrenched
 Spark, while a replacement, is still
entering the trough of
disillusionment
 Other large volume options (e.g., in
memory DBMS) are mature
 It is part of the landscape and other
tools take over its limitations
 AI may eventually overtake Hadoop
(let us explain)

Current State of Hadoop – Not a Focal Point, but Important
Proven data lake “at rest”
technology
The “only” Big Data store and file
management (HDFS)
Analyzing large data sets easily
Good availability of supporting
“wrapping” technology (Podium,
Cloudera, etc.)
Haas becoming more common
Trend to real-time and low-
latency affects relevance
Being supplanted by Spark
and Storm
Open source but still needs
expertise
Lack of governance and
security
Extended implementation
and time-to-market
Goodtoknow…..
Needtobeaware…

Role of Hadoop – Integral to Reference Architectures
Data Life
Cycles
DATA INSIGHT ARCHITECTURE
1Management
Data Usage
Vintage Area Contemporary Area
Business Strategy
Legacy BI and Reporting
Data Warehouse, ODS, Mart
Low latency (Spark)
Graph
Advanced Analytics
RDBMS, SQL, In-Memory
Appliance
Metadata Lineage Reference Data
Alignment
Data Monetization
Visualization ETL,EAI,ReplicationMobile Logical DW
Unstructured Data
Hadoop. HDFS (for lakes)
ERP, Legacy Applications

Scenarios for Hadoop as a Main Player and a Focal Point
Use Case 1 – Large Organization
• Very large organization with lots of data and skilled technologists
• Owned data centers and offered as a service
• Located where there is subpar internet back bone (e.g., Hawaii)
Use Case 2 – Direct Control
• Where economics are less important than direct control
• The safest connection is no connection (e.g., research facilities,
atomic/nuclear, military)

Lessons Learned from Hadoop (and Other Technologies)
Pizza Tastes Awful
Square slices (Culture management)
Lost recipe (DG)
Poor preparation ((DM)
Poor ingredients (DQ)
*Graphic source: Albert Barron “Pizza as a
Service,” published on LinkedIn July 30, 2014

Lessons Learned from Hadoop (and Other Technologies)
 Delivery and storage
architecture are irrelevant if
you don’t control the raw
ingredients and process
 You are still responsible for
above-the-line capabilities
 Regardless where Hadoop
lives, there are still significant
shortcomings across all
industries that affect its
usefulness, but aren’t the
fault of Hadoop
Pizza Tastes Awful
Square slices (Culture management)
Lost recipe (DG)
Poor preparation ((DM)
Poor ingredients (DQ)
*Graphic source:
Albert Barron “Pizza
as a Service,”
published on LinkedIn
July 30, 2014

Alternatives to Hadoop (not Exhaustive)
 Apache Spark
 Apache Storm
 Google BigQuery
 DataTorrent RTS
 Hydra
 Amazon S3
 Snowflake
 Amazon Redshift
 Podium
 Hortonworks
 Cloudera
 Cassandra
 MongoDB
 Apache HBase

Non-standard Uses of Hadoop
 AI and Machine Learning for
creating analytic data sets
− Use algorithms to match and
identify related data
− Create at-rest data sets (in
Hadoop) and use new tools
for access
− Data scientists vet results
and confirm usefulness
− Use as a Lake to store, access
and distribute results
pg 12
© 2018 First San Francisco Partners www.firstsanfranciscopartners.com
Vintage Area
1
Data Life
Cycles
Data Management
Data Usage
Vintage Area
Legacy BI and
Reporting
Data Warehouse, ODS,
Mart ETL,
EAI,
Msg,
Copy
Data Lake
Hadoop
RDBMS, SQL, In-
Memory Appliance
Unstructured
Data
Structured
Data

Non-standard Uses of Hadoop
 AI and Machine learning for metadata
− Use AI and machine learning to extract
metadata
− Provide insight to data stewards and
custodians
− Use at rest Hadoop as “system of
record” for metadata
− Use Graph data base for metadata
access
Vintage Area
1
Data
Life
Cycles
Data Management
Data Usage
Vintage Area
Legacy BI and
Reporting
Data Warehouse,
ODS, Mart ETL,
EAI,
Msg,
Copy
Data Lake
Hadoop
Advanced Analytics
RDBMS, SQL, In-
Memory Appliance
Metadata
Logical DW
Data Sources
Graph
BIVisualization

The Future of Hadoop
*Image source: The Globe and Mail “The Maytag Man
through the years” published June 19, 2017
Staple/mainstay of reliable technology…

Best Practices
 Stick with Hadoop and a product that
addresses shortcomings for data at rest
 Design a data architecture, whether or
not you are using cloud
 Consider if your organization has unique
characteristics that may place Hadoop
center stage

Key Takeaways
KEEP IN MIND…
 If streaming data, then you need to get serious
about other types of tech, or just place it in the cloud
 There is no avoiding data governance and data management,
regardless of your approach
 Hadoop will remain a part of data supply chains and analytics
ecosystems for data at rest and lakes
Hadoop is technology – which ALWAYS commoditizes or
disappears – it is not a decision of IF, but HOW

Please Share Your Questions and Comments
MONTHLY SERIES

Thank you for joining us today!
Our Thursday, September 6
#DIAnaltyics webinar is:
Advanced Databases and
Knowledge Management
.
John Ladley @jladley
john@firstsanfranciscopartners.com
Kelle O’Neal @kellezoneal
kelle@firstsanfranciscopartners.com

The Missed Promise of Hadoop and New and Emerging Technologies

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to The Missed Promise of Hadoop and New and Emerging Technologies

Similar to The Missed Promise of Hadoop and New and Emerging Technologies (20)

More from DATAVERSITY

More from DATAVERSITY (20)

Recently uploaded

Recently uploaded (20)

The Missed Promise of Hadoop and New and Emerging Technologies