This event is history, please check out the List of Upcoming Seminars, or send us an email
Check out our related in-house workshops:
Why do we organise this workshop about Apache Spark ?
Big Data gets a lot of attention these days in ICT and marketing, because big data lays the foundation for data-driven business. Since its inception in 2007, Apache Hadoop has been looked at as the de facto standard for the storage and processing of big data volumes in batch. But every technology has its limitations, and this is no different for Hadoop: it is batch-oriented and the MapReduce framework is too limited for handling all types of data analysis within the same technology stack.
Because the volume, the speed and the complexity of data generated by mobile, social and sensors (IoT) gradually increases, the need for faster data processing and analysis to answer the needs and expectations of end users also increases.
Apache Spark solves the problem of speed and versatility by offering an "open source data analytics cluster computing framework". Spark was developed in 2009 at the AMPLab (Algorithms, Machines, and People Lab) of the University of California in Berkeley, and donated to the open source community in 2010. It is faster than Hadoop, in some cases 100 times faster, and it offers a framework that supports different types of data analysis within the same technology stack: fast interactive queries, streaming analysis, graph analysis and machine learning. During this two-day hands-on workshop, we discuss the theory and practice of several data analysis applications.
Who should attend this workshop?
This workshop is mainly aimed at developers, data analysts and data scientists who want to know more about Apache Spark. This course uses a hands-on approach to teach you the basics of Spark and give you a flying start.
You get an introduction to all Spark components from the perspective of the "data developer". Some experience with programming is necessary to get the most out of this course.
The exercises are implemented on your own laptop, and vary from easy to complex, gradually adding functionality. We will explain how to do the exercises in Scala or Python, the choice is up to you.
We also offer this training as an in-house course for a minimum of 5 people from your company.
Spark was developed in Scala, a high-level programming language that combines object-oriented and functional programming. Programming Spark applications in Scala is straightforward for anyone who is familiar with a programming language. We look at the definition of variables, functions and the use of collections in Scala.
However, because a lot of data science and statistical applications are currently programmed in Python, the open source community has developed a wonderful toolkit called PySpark, to expose the Spark programming model to Python.
We look at the Spark Core API from the perspective of the "Data Developer": from prototyping in the Spark Shell to the compilation and packaging of Spark applications for a cluster, and how this application is efficiently executed on a cluster.
The following topics will be covered:
Having read-write shared variables across Spark tasks running on clusters would be inefficient. However, Spark does provide two limited types of shared variables for two common usage patterns: broadcast variables and accumulators.
We gradually increase the complexity of our exercises using RDDs and DataFrames
Besides Spark's core module, we look at a number of modules that were added to the Spark stack like Spark SQL, and do some exercises on processing structured data with SQL commands
A more extended, guided exercise in which most of the Spark modules are combined, showing the true power of Spark.
Geert Van Landeghem is a Big Data consultant with 25 years of experience working for companies across industries. He worked on his first big data project in 2011, and is still consulting companies on how to adopt big data within their organisation.
He has worked as the Head of BI for a gambling company in Belgium, where he led a team of 8 people. He is an Apache Spark Certified Developer since November 2014, and has worked as an instructor for IBM and Datacrunchers, where he teaches Hadoop and Spark-related courses.
He is currently examining how Artificial Intelligence can be used for business use cases and as such followed the first IBM Watson and O'Reilly AI conferences abroad.
Questions about this ? Interested but you can't attend ? Send us an email !