Hadoop summit 2010 frameworks panel elephant bird

DataWorks Summit/Hadoop Summit

Bigdata : Big picture

Zekeriya Besiroglu

introduction to data processing using Hadoop and Pig

Ricardo Varela

Hive is a data warehousing system built on Hadoop that allows users to query data using SQL. It addresses issues with using Hadoop for analytics like programmability and metadata. Hive uses a metastore to manage metadata and supports structured data types, SQL queries, and custom MapReduce scripts. At Facebook, Hive is used for analytics tasks like summarization, ad hoc analysis, and data mining on over 180TB of data processed daily across a Hadoop cluster.

Hadoop basics

Antonio Silveira

Introduction to Apache Hadoop

Christopher Pezza

This document provides an introduction and overview of Apache Hadoop. It discusses how Hadoop provides the ability to store and analyze large datasets in the petabyte range across clusters of commodity hardware. It compares Hadoop to other systems like relational databases and HPC and describes how Hadoop uses MapReduce to process data in parallel. The document outlines how companies are using Hadoop for applications like log analysis, machine learning, and powering new data-driven business features and products.

Powering a Virtual Power Station with Big Data

This document discusses how Open Energi is using big data and Hortonworks Data Platform (HDP) to power a virtual power station. Open Energi processes 25-40k messages per second containing 500TB-800TB of data from assets and markets. This is smaller than Boeing's 100PB of data from aircraft, but Open Energi's data volume is expected to grow significantly with demand-side response programs. HDP allows Open Energi to scale quickly, integrate multiple data sources using Apache Hive, reuse Python code for analytics, and gain insights through machine learning to predict demand, optimize systems, and more.

Hive and Apache Tez: Benchmarked at Yahoo! Scale

DataWorks Summit

This document discusses benchmarking Hive at Yahoo scale. Some key points: - Hive is the fastest growing product on Yahoo's Hadoop clusters which process 750k jobs per day across 32500 nodes. - Benchmarking was done using TPC-H queries on 100GB, 1TB, and 10TB datasets stored in ORC format. - Significant performance improvements were seen over earlier Hive versions, with 18x speedup over Hive 0.10 on text files for the 100GB dataset. - Average query time was reduced from 530 seconds to 28 seconds for the 100GB dataset, and from 729 seconds to 172 seconds for the 1TB dataset.

Introduction To Apache Pig at WHUG

This document provides an introduction to Apache Pig including: - What Pig is and how it offers a high-level language called PigLatin for analyzing large datasets. - How PigLatin provides common data operations and types and is more natural for analysts than MapReduce. - Examples of how WordCount looks in PigLatin versus Java MapReduce. - How Pig works by parsing, optimizing, and executing PigLatin scripts as MapReduce jobs on Hadoop. - Considerations for developing, running, and optimizing PigLatin scripts.

Pig, Making Hadoop Easy

Nick Dimiduk

This document introduces Pig, an open source platform for analyzing large datasets that sits on top of Hadoop. It provides an example of using Pig Latin to find the top 5 most visited websites by users aged 18-25 from user and website data. Key points covered include who uses Pig, how it works, performance advantages over MapReduce, and upcoming new features. The document encourages learning more about Pig through online documentation and tutorials.

Hadoop Operations Powered By ... Hadoop (Hadoop Summit 2014 Amsterdam)

At Spotify we collect huge volumes of data for many purposes. Reporting to labels, powering our product features, and analyzing user growth are some of our most common ones. Additionally, we collect many operational metrics related to the responsiveness, utilization and capacity of our servers. To store and process this data, we use scalable and fault-tolerant multi-system infrastructure, and Apache Hadoop is a key part of it. Surprisingly or not, Apache Hadoop generates large amounts of data in the form of logs and metrics that describe its behaviour and performance. To process this data in a scalable and performant manner we use … also Hadoop! During this presentation, I will talk about how we analyze various logs generated by Apache Hadoop using custom scripts (written in Pig or Java/Python MapReduce) and available open-source tools to get data-driven answers to many questions related to the behaviour of our 690-node Hadoop cluster. At Spotify we frequently leverage these tools to learn how fast we are growing, when to buy new nodes, how to calculate the empirical retention policy for each dataset, optimize the scheduler, benchmark the cluster, find its biggest offenders (both people and datasets) and more.

Hadoop and object stores: Can we do it better?

gvernik

Strata Data Conference, London, May 2017 Trent Gray-Donald and Gil Vernik explain the challenges of current Hadoop and Apache Spark integration with object stores and discuss Stocator, an open source (Apache License 2.0) object store connector for Hadoop and Apache Spark specifically designed to optimize their performance with object stores. Trent and Gil describe how Stocator works and share real-life examples and benchmarks that demonstrate how it can greatly improve performance and reduce the quantity of resources used.

Intro To Hadoop

Bill Graham

Hive sq lfor-hadoop

Pragati Singh

Hive provides a SQL interface for querying and analyzing large datasets stored in Hadoop. It allows data warehouse teams familiar with SQL to migrate their skills and applications to Hadoop, as Hive queries are compiled into MapReduce jobs that run on Hadoop. The document demonstrates ingesting log data into Hive, transforming it, and then querying it using SQL to analyze user behavior data around vampire-related search terms and purchases.

HBaseCon 2015: S2Graph - A Large-scale Graph Database with HBase

HBaseCon

As the operator of the dominant messenger application in South Korea, KakaoTalk has more than 170 million users, and our ever-growing graph has more than 10B edges and 200M vertices. This scale presents several technical challenges for storing and querying the graph data, but we have resolved them by creating a new distributed graph database with HBase. Here you'll learn the methodology and architecture we used to solve the problems, compare it another famous graph database, Titan, and explore the HBase issues we encountered.

Hadoop sqoop

Wei-Yu Chen

Apache Sqoop allows transferring data between structured data stores like relational databases and Hadoop. It uses MapReduce to import/export data in parallel. Sqoop can import data from databases into Hive and export data from HDFS to databases. The document provides examples of using Sqoop to import data from MySQL to Hive and export data from HDFS to MySQL. It also demonstrates creating and executing Sqoop jobs. References for more Sqoop tutorials and documentation are included.

Hadoop to spark-v2

Sujee Maniyam

Introduction to the Hadoop Ecosystem (codemotion Edition)

This document provides an introduction to the Hadoop ecosystem. It discusses data storage with HDFS and data processing with MapReduce. It also describes higher-level tools like Pig and Hive that provide interfaces for data analysis. Pig uses a language called Pig Latin to analyze data in Hadoop, while Hive provides an SQL-like interface. These tools sit on top of core Hadoop components to simplify big data workflows.

Pig Tutorial | Apache Pig Tutorial | What Is Pig In Hadoop? | Apache Pig Arch...

Simplilearn

The document discusses key concepts related to the Pig analytics framework. It covers topics like why Pig was developed, what Pig is, comparisons of Pig to MapReduce and Hive, Pig architecture involving Pig Latin scripts, a runtime engine, and execution via a Grunt shell or Pig server, how Pig works by loading data and executing Pig Latin scripts, Pig's data model using atoms and tuples, and features of Pig like its ability to process structured, semi-structured, and unstructured data without requiring complex coding.

Getting started with Hadoop, Hive, and Elastic MapReduce

obdit

1) The document provides an overview of tools for distributed computing including MapReduce, Hadoop, Hive, and Elastic MapReduce. 2) It discusses getting started with Elastic MapReduce using Python with mrjob or the AWS command line and challenges with getting started with Hive. 3) Potential pitfalls with EMR are also outlined such as JVM memory issues and problems with multiple small output files.

Introduction to the Hadoop Ecosystem (IT-Stammtisch Darmstadt Edition)

Measuring CDN performance and why you're doing it wrong

Fastly

Integrating content delivery networks into your application infrastructure can offer many benefits, including major performance improvements for your applications. So understanding how CDNs perform — especially for your specific use cases — is vital. However, testing for measurement is complicated and nuanced, and results in metric overload and confusion. It's becoming increasingly important to understand measurement techniques, what they're telling you, and how to apply them to your actual content. In this session, we'll examine the challenges around measuring CDN performance and focus on the different methods for measurement. We'll discuss what to measure, important metrics to focus on, and different ways that numbers may mislead you. More specifically, we'll cover: Different techniques for measuring CDN performance Differentiating between network footprint and object delivery performance Choosing the right content to test Core metrics to focus on and how each impacts real traffic Understanding cache hit ratio, why it can be misleading, and how to measure for it

Hadoop at Twitter (Hadoop Summit 2010)

What's hot

Getting Started on Hadoop

Paco Nathan

Introduction to the Hadoop Ecosystem (FrOSCon Edition)

DataWorks Summit/Hadoop Summit

HIVE: Data Warehousing & Analytics on Hadoop

Zheng Shao

Hadoop basics

Antonio Silveira

Introduction to Apache Hadoop

Christopher Pezza

Powering a Virtual Power Station with Big Data

Hive and Apache Tez: Benchmarked at Yahoo! Scale

DataWorks Summit

Introduction To Apache Pig at WHUG

Pig, Making Hadoop Easy

Nick Dimiduk

Hadoop Operations Powered By ... Hadoop (Hadoop Summit 2014 Amsterdam)

Hadoop and object stores: Can we do it better?

gvernik

Intro To Hadoop

Bill Graham

Hive sq lfor-hadoop

Pragati Singh

HBaseCon 2015: S2Graph - A Large-scale Graph Database with HBase

HBaseCon

Hadoop sqoop

Wei-Yu Chen

Hadoop to spark-v2

Sujee Maniyam

Introduction to the Hadoop Ecosystem (codemotion Edition)

Pig Tutorial | Apache Pig Tutorial | What Is Pig In Hadoop? | Apache Pig Arch...

Simplilearn

Getting started with Hadoop, Hive, and Elastic MapReduce

obdit

Introduction to the Hadoop Ecosystem (IT-Stammtisch Darmstadt Edition)

What's hot (20)

Getting Started on Hadoop

Introduction to the Hadoop Ecosystem (FrOSCon Edition)

HIVE: Data Warehousing & Analytics on Hadoop

Hadoop basics

Introduction to Apache Hadoop

Powering a Virtual Power Station with Big Data

Hive and Apache Tez: Benchmarked at Yahoo! Scale

Introduction To Apache Pig at WHUG

Pig, Making Hadoop Easy

Hadoop Operations Powered By ... Hadoop (Hadoop Summit 2014 Amsterdam)

Hadoop and object stores: Can we do it better?

Intro To Hadoop

Hive sq lfor-hadoop

HBaseCon 2015: S2Graph - A Large-scale Graph Database with HBase

Hadoop sqoop

Hadoop to spark-v2

Introduction to the Hadoop Ecosystem (codemotion Edition)

Pig Tutorial | Apache Pig Tutorial | What Is Pig In Hadoop? | Apache Pig Arch...

Getting started with Hadoop, Hive, and Elastic MapReduce

Introduction to the Hadoop Ecosystem (IT-Stammtisch Darmstadt Edition)

Viewers also liked

Measuring CDN performance and why you're doing it wrong

Fastly

Hadoop at Twitter (Hadoop Summit 2010)

Analyzing Big Data at Twitter (Web 2.0 Expo NYC Sep 2010)

Big Data at Twitter, Chirp 2010

Hadoop and pig at twitter (oscon 2010)

This document summarizes Kevin Weil's presentation on Hadoop and Pig at Twitter. Weil discusses how Twitter uses Hadoop and Pig to analyze massive amounts of user data, including tweets. He explains how Pig allows for more concise and readable analytics jobs compared to raw MapReduce. Weil also provides examples of how Twitter builds data-driven products and services using these tools, such as their People Search feature.

Hadoop Overview kdd2011

Scaling hadoopapplications

The document discusses best practices for scaling Hadoop applications. It covers causes of sublinear scalability like sequential bottlenecks, load imbalance, over-partitioning, and synchronization issues. It also provides equations for analyzing scalability and discusses techniques like reducing algorithmic overheads, increasing task granularity, and using compression. The document recommends using higher-level languages, tuning configuration parameters, and minimizing remote procedure calls to improve scalability.

Extending Hadoop for Fun & Profit

Apache Hadoop project, and the Hadoop ecosystem has been designed be extremely flexible, and extensible. HDFS, Yarn, and MapReduce combined have more that 1000 configuration parameters that allow users to tune performance of Hadoop applications, and more importantly, extend Hadoop with application-specific functionality, without having to modify any of the core Hadoop code. In this talk, I will start with simple extensions, such as writing a new InputFormat to efficiently process video files. I will provide with some extensions that boost application performance, such as optimized compression codecs, and pluggable shuffle implementations. With refactoring of MapReduce framework, and emergence of YARN, as a generic resource manager for Hadoop, one can extend Hadoop further by implementing new computation paradigms. I will discuss one such computation framework, that allows Message Passing applications to run in the Hadoop cluster alongside MapReduce. I will conclude by outlining some of our ongoing work, that extends HDFS, by removing namespace limitations of the current Namenode implementation.

Hadoop: The Default Machine Learning Platform ?

Apache Hadoop, since its humble beginning as an execution engine for web crawler and building search indexes, has matured into a general purpose distributed application platform and data store. Large Scale Machine Learning (LSML) techniques and algorithms proved to be quite tricky for Hadoop to handle, ever since we started offering Hadoop as a service at Yahoo in 2006. In this talk, I will discuss early experiments of implementing LSML algorithms on Hadoop at Yahoo. I will describe how it changed Hadoop, and led to generalization of the Hadoop platform to accommodate programming paradigms other than MapReduce. I will unveil some of our recent efforts to incorporate diverse LSML runtimes into Hadoop, evolving it to become *THE* LSML platform. I will also make a case for an industry-standard LSML benchmark, based on common deep analytics pipelines that utilize LSML workload.

Future of Data Intensive Applicaitons

"Big Data" is a much-hyped term nowadays in Business Computing. However, the core concept of collaborative environments conducting experiments over large shared data repositories has existed for decades. In this talk, I will outline how recent advances in Cloud Computing, Big Data processing frameworks, and agile application development platforms enable Data Intensive Cloud Applications. I will provide a brief history of efforts in building scalable & adaptive run-time environments, and the role these runtime systems will play in new Cloud Applications. I will present a vision for cloud platforms for science, where data-intensive frameworks such as Apache Hadoop will play a key role.

The Zoo Expands: Labrador *Loves* Elephant, Thanks to Hamster

The document summarizes Milind Bhandarkar's work developing Hamster, a system for running MPI applications on Hadoop YARN. Some key points: - Hamster allows MPI applications to run alongside Hadoop dataflow jobs on the same cluster managed by YARN. It implements an MPI runtime on top of YARN. - Hamster's design leverages OpenMPI's strengths while allowing it to integrate with YARN. It includes an application master, node service, and scheduler component. - Performance tests show Hamster has low overhead and scales well for large MPI jobs. It introduces only a small performance penalty compared to running MPI natively with OpenMPI. - Example results are shown

Modeling with Hadoop kdd2011

This document discusses modeling algorithms using the MapReduce framework. It outlines types of learning that can be done in MapReduce, including parallel training of models, ensemble methods, and distributed algorithms that fit the statistical query model (SQM). Specific algorithms that can be implemented in MapReduce are discussed, such as linear regression, naive Bayes, logistic regression, and decision trees. The document provides examples of how these algorithms can be formulated and computed in a MapReduce paradigm by distributing computations across mappers and reducers.

Spatial Analytics, Where 2.0 2010

NoSQL at Twitter (NoSQL EU 2010)

Using CDN to improve performanceGea-Suan Lin

Thrift vs Protocol Buffers vs Avro - Biased Comparison

Igor Anishchenko

Igor Anishchenko Odessa Java TechTalks Lohika - May, 2012 Let's take a step back and compare data serialization formats, of which there are plenty. What are the key differences between Apache Thrift, Google Protocol Buffers and Apache Avro. Which is "The Best"? Truth of the matter is, they are all very good and each has its own strong points. Hence, the answer is as much of a personal choice, as well as understanding of the historical context for each, and correctly identifying your own, individual requirements.

Practical Problem Solving with Apache Hadoop & Pig