Simplifying Big Data Integration with Syncsort DMX and DMX-h

1
Syncsort Confidential and Proprietary - do not copy or distribute

Housekeeping
Webcast Audio:
– Today’s webcast audio is streamed through your computer speakers.
– If you need technical assistance with the web interface or audio, please
reach out to us using the chat window.
Questions Welcome:
– Submit your questions at any time during the presentation using the
chat window.
– We will answer them during our Q&A session following the
presentations.
Recording and Slides:
– This webcast is being recorded. You will receive an email following the
webcast with a link to download both the recording and the slides.
2

Meet Today’s Presenters
3
Paige Roberts
Big Data Product Manager
Syncsort
Mark Muncy
Big Data Product Marketing Manager
Syncsort

4
Q&A
More Ways to Engage This Summer!
Next Webcast (6/23):Dickey’s Barbecue Heats Up Analytics with Amazon Web Services
Hadoop Summit San Jose – June 28-30
Strata + Hadoop World Beijing – August 3-6
Cloudera Sessions:
Minneapolis (6/22), NYC (6/28), Scottsdale (7/14), Phila (7/20), Baltimore (8/23), Atlanta (8/25)
Online:
www.syncsort.com/bigdata
blog.syncsort.com
@syncsort

Syncsort Big Data Products
What’s New and Coming Soon
June 2016

Agenda
Simplify Big Data Integration
• Access
– Data Funnel
– New Sources
– New mainframe distributable format
– AsiaPac support improvements
• Integrate
– Kafka and MapR Streams support
• Comply
– Cloudera Navigator metadata support
• Simplify
– Intelligent Execution with Spark
6

Syncsort DMX & DMX-h: Smarter Data Processing for Big Data
7
• GUI for developing MapReduce & Spark jobs
• Test & debug locally in Windows; deploy on Hadoop
• Use-case Accelerators to fast-track development
• Broad based connectivity with automated parallelism
• Simply the Best mainframe data migration to Hadoop
• Improved per node scalability and throughput
DMX-h
High Performance Hadoop ETL Software
• Template driven design for:
• High performance ETL
• SQL migration/DB offload
• Mainframe data movement
• Light weight footprint on commodity hardware
• High speed flat file processing
• Self tuning engine
High Performance ETL Software
DMX

SIMPLIFY BIG DATA INTEGRATION
Focus Area
8

Simplify Big Data Integration with Syncsort
9
Access - Get best in class data ingestion capabilities for Hadoop.
Mainframes, RDBMSs, MPP, JSON, Avro/Parquet, NoSQL, Kafka
and more.

Access: Populate At the Press of a Button
• Funnel hundreds of tables into your data hub
• Extract and move whole DB schemas in one
invocation
• Pull multiple data sources: DB2, Netezza, Oracle,
Teradata, …
• One-step data movement, auto-generating jobs
• Process multiple funnels in parallel
• Filter data in-flight
• Create and populate your data lake efficiently, reduce
development time from weeks to days
10

Access: Get Best in Class Data Ingestion Capabilities for Hadoop
11
Database
– RDBMS
– MPP
– NoSQL
Mainframe
– DB2
– VSAM
– Mainframe Fixed
– Mainframe Variable
– Mainframe Distributable
– FTP Binary
– All file formats…
Big Data
– JSON
– Avro
– Parquet
– ORC
Streaming
– Kafka
– MapR Streams
– HDF (NiFi)
Cloud
– Amazon S3
– Amazon Redshift, RDS
– Google Cloud Storage
… And more!

Access: Unique Mainframe Distributable Format
Mainframe file processing in Hadoop with DMX Mainframe Variable Hadoop
Distributable Format – We taught Hadoop how to speak mainframe.
• Access mainframe data from a Hadoop cluster, without modifying it from
its original format
• Make Hadoop understand EBCDIC data
• Make mainframe data distributable to process it with MapReduce &
Spark
• Record data is not changed
• Existing copybooks continue to work
• MF Data types stay as is, no conversions to justify or track
• Useful for regulatory compliance, data governance, archiving

Access: Simply the Best Access and Integration of Mainframe Data
13
Save MIPS by processing mainframe data on
Hadoop
Read and write Mainframe record formats
– Fixed record length, variable record
length, & variable record length with
block descriptor
– Handle complex array structures like
ODO’s, even nested
– Interpret complex copybooks
automatically
Write files to local or remote open systems
via FTP, SFTP, Connect:Direct or HDFS
Store an unmodified archive copy for
compliance and lineage tracking

AsiaPac Support Improvements – (Coming v9.x, around July)
Improved Fujitsu NetCOBOL support
Localization
Complete support of all ICU code pages
– Drop down list in GUI that provides most common code pages at the top
– Remembers most recent code page selection and pre-populates
14

15
and more.
Integrate – Single interface for streaming and batch processes.
Single data pipeline for all enterprise data, batch or streaming.

16
Integrate: DMX-h: Streaming Data Support
• Kafka sources for streaming (GA in 8.5)
– Streaming and batch via single
interface
– Ease of application development - no
need to write C or Java code to connect
to Kafka
– Insulate you from any changes in Kafka
across different releases
• Publish to Kafka topics (9.0 in June, working
in latest release)
• Certified for MapR Streams (in Beta)

17
Integrate: Single Interface for Streaming & Batch
• Support for Kafka, MapR Streams, Spark
• Easier application development – no need
to write C or Java code to connect
• Insulates user from changes in Kafka across
releases

Feed Business Intelligence Visualization
Integrate: Achieve the Fastest Path from Raw Data to Insight
Hadoop + DMX-h
NoSQL
Get the fastest, most efficient data joins and sorts
High-performance connectivity to Big Data & NoSQL databases such as Cassandra, HBase &
MongoDB
Fastest parallel loads to Amazon Redshift, Greenplum, Netezza, Oracle, Teradata & Vertica
Create Tableau & Qlikview files with one click
18

19
and more.
Comply – Secure data access, data governance and lineage.
Seamless integration with Kerberos, Apache Ranger, Apache
Ambari, Cloudera Manager, Cloudera Navigator and Sentry.

Comply: Manage, Monitor and Secure Your Cluster
Cloudera Manager and Apache Ambari
– Deploy across cluster
– Monitor jobs
Cloudera Sentry security certified
Apache Ranger support
Authenticated browsing and sampling in Kerberos-secured
clusters
– WebHDFS support for reading/loading HDFS
20

Comply: Get Governance, Metadata, Lineage and Search
21
• DMX-h provides metadata management and data lineage by
updating HCatalog when loading to Hive, Avro and Parquet
• DMX-h has certified integration with Cloudera Navigator
• Cloudera Navigator metadata extends HCatalog, HDFS, YARN,
Spark and other metadata, including lineage, tagging, business
metadata, and structural metadata

22
and more.
Comply – Secure data access, data governance and lineage.
Seamless integration with Kerberos, Apache Ranger, Apache
Ambari, Cloudera Manager, Cloudera Navigator and Sentry.
Simplify – Design once, deploy anywhere & insulate your
organization from rapidly changing eco-system. Future proof your
applications for new compute frameworks, on-premise or in the
cloud.

23
Simplify: Deploy on a Server, a Cluster or in the Cloud
Big Data + Cloud = Perfect Storm
23
• ETL engine on AWS Marketplace – (update coming
by end of June)
• Available on EC2 and EMR, Google Cloud
• S3 and Redshift connectivity
• Google Cloud Storage connectivity
• First & only leading ETL engine on Docker Hub

Intelligent Execution
Simplify: Design Once, Deploy Anywhere
Intelligent
ExecutionLayer
One interface to design jobs to run on:
Single Node, Cluster
MapReduce, Spark, Future Platforms
Windows, Unix, Linux
On-Premise, Cloud
Batch, Streaming
24
Insulate your people from underlying complexities of Hadoop. Use existing ETL skills.
No worries abut mappers, reducers, big side, small side, and so on.
No changes or tuning required, even if you change execution frameworks
Future-proof job designs for emerging compute frameworks, e.g. Spark

Using the Dell |
Cloudera | Syncsort
solution for Hadoop,
an entry-level
technician developed
and deployed Hadoop
ETL jobs in 53.7% less
time than a Hadoop
expert
Simplify: Reclaim days of valuable time
Fact dimension load
with type 2 SCD
Data validation and
pre-processing
Vendor
mainframe file
integration
Load Validate Int.
8.3 Days
3.8 Days
Cut Development Time in Half!

OTHER NEW FEATURES
Focus Area
26

27
DMX-h: Data Transformation Language (DTL)
• Metadata driven dynamic creation of DMX-h
jobs
• Enables partners and end users to build on and
extend DMX
• Human readable script-like interface for
developing jobs
• Legacy ETL migrations to DMX
– Ability to import to DMX GUI
– You can maintain these applications in
the visual interface

28
DMX-h Extensibility: Custom Functions Framework
• Enable data scientists to add news functions
• Ability to add custom transformation functions
– Shown in the GUI same as built-in functions
– Available via function pull-down and signature
• Existing functions – Available at bigdatakb.syncsort.com!
– Rounding Package
– Advanced Math Package
– 3 Pivot options

Experience to Do It Right, The First Time | Support and Services
29
Syncsort Professional Services delivers:
• Quicker Time to Value
• Simplified development with Best Practices
• Optimal performance and scalability
• Efficient usage of computing resources
“With the help of the Syncsort team, the migration from our previous solution to DMX
was completed in half the time versus going it alone. Their depth of product
knowledge, and general industry experience, saved us time and resources, and
gave us confidence knowing the job was done right.”
– Mike Breitenbeker, Director of Data Warehousing, Overstock

To view the webcast on-demand, please visit:
http://sync.st/1RXldBU
31

Simplifying Big Data Integration with Syncsort DMX and DMX-h

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Simplifying Big Data Integration with Syncsort DMX and DMX-h

Similar to Simplifying Big Data Integration with Syncsort DMX and DMX-h (20)

More from Precisely

More from Precisely (20)

Recently uploaded

Recently uploaded (20)

Simplifying Big Data Integration with Syncsort DMX and DMX-h