Denodo Data Virtualization Platform: Scalability (session 3 from Architect to Architect webinar series)

Five In-depth Technology and Architecture Sessions
on Data Virtualization
Session 3: Scalability

Today’s Speaker
■ Jesus Barrasa
Senior Solutions Architect, Denodo

Architect-to-Architect Series
■ Series of five webinars over 3 months
■ Deeper look into Denodo Platform
■ Architectural Overview
■ Performance
■ Scalability (today’s session)
■ Data Discovery and Governance
■ Security

Denodo Express
■ Denodo Express
■ Free to Download
■ Fully functioning Data Virtualization Platform
■ Single user, supports common data sources
■ Many of the same capabilities of Denodo
Platform
■ Performance, Data Discovery, Governance,
internal Security, Publishing, …

Scalability – Architecture Modules

Scalability – HA and Clustering
■ Denodo Cache
■ The Denodo Platform incorporates an advanced
cache system to improve performance
■ For data materialization it makes use of an
external JDBC database (physical or in-memory)
■ Denodo High-Availability
■ Set of features provided by the Denodo Platform
to support High Availability and Load Balancing
■ Denodo Cluster Management
■ Tool for monitoring and managing a cluster of
Denodo servers

Scalability – HA and Clustering
■ Singleton
■ High Availability Architecture
■ Active-Passive
■ Active-Active
■ Vertical Scalability
■ VM Support
■ Metadata Sharing
■ Horizontal Scalability
■ Clustering
■ Shared Cache or Individual Cache

• A single Data Virtualization server instance
…
Data sources
LDAP Auth
Cache Database
Client Application
ESB/BPM/Portal…
Singleton

…
Data sources
LDAP Auth
Cache Database
Client Application
ESB/BPM/Portal…
HA – Active-Passive
Load Balancer

HA – Active-Passive
■ All queries sent to Active server
■ Passive server running, but not handling
requests
■ Both servers ‘pinged’ by Load Balancer with
appliance health check or using ping script
■ When Load Balancer detects Active server
failure
■ Requests redirected to Passive server
■ Passive server becomes ‘Active’
■ When former Active server restored
■ Load Balancer determines new roles…

…
Data sources
LDAP Auth
Cache Database
Client Application
ESB/BPM/Portal…
HA – Active-Active
Load Balancer

HA – Active-Active
■ Load Balancer distributes queries between
both servers
■ Both servers running and handling requests
■ Both servers ‘pinged’ by Load Balancer with
appliance health check or using ping script
■ When Load Balancer detects server failure
■ All requests redirected to other server
■ When ‘failed’ server restored
■ Added back to load balancing pool
■ Starts receiving queries

Vertical Scaling
■ Java limitations inhibit vertical scaling for singletons
■ Virtualization
■ Run Denodo Platform instances in VM
■ VMWare, Hyper-V, Xen, …
■ Very common scenario
■ Multiple Instances
■ Install multiple instances of Denodo Platform
■ Run multiple instances of Denodo Platform from a single
Denodo Platform installation
■ Share metadata and cache

Vertical Scaling (Cont’d)
Cache DatabaseMetadata
Master
Slave-1
Slave-2
Slave-3

Vertical Scaling (Cont’d)
■ First started instance is ‘master’
■ All others are ‘slaves’
■ Only ‘master’ instance can modify metadata
■ Slaves read metadata at startup time
■ Pros
■ Only one copy of metadata
■ Cons
■ Port configurations need modifying
■ SPoF

Horizontal Scaling – Clustering
Load Balancer
…
Datasources
Virtual Server
VDP Cluster: 192.168.0.10:9999
Web Container Cluster: 192.168.0.10:9090
VDP Cluster:
• Denodo1:9999
• Denodo2:9999
• Denodo3:9999
• Denodo4:9999
• Denodo5:9999
Web Cont. Cluster:
• Denodo1:9090
• Denodo2:9090
• Denodo3:9090
• Denodo4:9090
• Denodo5:9090

Clustering – Shared Cache
Load Balancer
Virtual Server
VDP Cluster: 192.168.0.10:9999
VDP Cluster:
• Denodo1:9999
• Denodo2:9999
• Denodo3:9999
• Denodo4:9999
• Denodo5:9999
Web Cont. Cluster:
• Denodo1:9090
• Denodo2:9090
• Denodo3:9090
• Denodo4:9090
• Denodo5:9090
Cache Database

Clustering – Separate Caches
Load Balancer
Virtual Server
VDP Cluster: 192.168.0.10:9999
VDP Cluster:
• Denodo1:9999
• Denodo2:9999
• Denodo3:9999
• Denodo4:9999
• Denodo5:9999
Web Cont. Cluster:
• Denodo1:9090
• Denodo2:9090
• Denodo3:9090
• Denodo4:9090
• Denodo5:9090
Cache Database
Cache Database
Cache Database
Cache Database
Cache Database

Clustering - Caches
■ Shared Cache
■ Any server can populate cache
■ All servers can access cached data
■ Increased ‘hit ratio’
■ Cache can become performance bottleneck
■ Separate Caches
■ Each server has its own cache
■ Avoids potential performance bottleneck
■ Access only data it has cached
■ Removes SPoF

Distributed Environments
■ Local caches help to achieve performance goals when
servers are in geographically dispersed locations
New York City
Omaha
San Francisco
Houston

Distributed Environments (Cont’d)
■ A remote Denodo server can be viewed as a
data source to another server
■ Promotes multi-layered architecture
■ Queries can be delegated to remote servers
to reduce network data movement
■ Same query delegation mechanism as talked
about in last session

■ Remote query delegation…
Local Data Sources
Cache Cache
Local Data Sources
New York City HoustonClient
10 M
rows
5 M
rows
JOIN Result:
20,000 rows
JOIN Result:
2,500 rows
2 M
rows

■ Shared views…
Local Data Sources
Cache Cache
Local Data Sources
New York City HoustonClient
10 M
rows
5 M
rows
JOIN Result:
20,000 rows
JOIN Result:
2,500 rows
2 M
rows
New York City Houston

Disaster Recovery
■ Business Continuity policies usually require
that critical IT systems are backed up to a
DR center
■ Denodo supports scripted synchronization of
metadata with DR systems
■ Using Export and Import tools
■ Sync up can be scheduled (time-based) or on
an event (promotion to production)

Scalability - Summary
■ Denodo Platform scales from single
instance to geographically distributed
system
■ Grow system without reconfiguring
■ Supports HA configurations
■ Active-Passive, Active-Active
■ Vertical scalability options
■ Support for DR site synchronization

Data Virtualization – Next Steps
Move forward at your own pace
 Download Denodo Express –
The fastest way to Data Virtualization
 Denodo Community: Documents, Videos, Tutorials, and more.
 Attend Architect-to-Architect Series
 Performance
 Scalability
Move forward with one of our Data
Virtualization experts
 Phone: (+1) 877-556-2531 (NA)
 Phone: (+44) (0)20 7869 8053 (EMEA)
 Email: info@denodo.com | www.denodo.com
 Data Discovery and Governance
 Security

Five In-depth Technology and Architecture Sessions
on Data Virtualization
Thank You!
Next Session
Session 4
Denodo Platform: Data Discovery
and Governance

Denodo Data Virtualization Platform: Scalability (session 3 from Architect to Architect webinar series)

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (17)

Similar to Denodo Data Virtualization Platform: Scalability (session 3 from Architect to Architect webinar series)

Similar to Denodo Data Virtualization Platform: Scalability (session 3 from Architect to Architect webinar series) (20)

More from Denodo

More from Denodo (20)

Recently uploaded

Recently uploaded (20)

Denodo Data Virtualization Platform: Scalability (session 3 from Architect to Architect webinar series)