Get the slides: https://www.datacouncil.ai/talks/sparklens-understanding-the-scalability-limits-of-spark-applications?hsLang=en

ABOUT THE TALK

One of the common requests we receive from customers at Qubole is to debug a slow Spark application. Usually this process is done with trial and error, which takes time and requires running clusters beyond normal usage (read wasted resources). Moreover, it doesn’t tell us where to look for further improvements. We at Qubole are looking into making this process more self-serve. Towards this goal we have built Sparklens (https://github.com/qubole/sparklens), an OSS tool based on Spark's event listener framework.

From a single run of the application, Sparklens provides insights about scalability limits of a given Spark application. In this talk we will cover what Sparklens does and the theory behind it. We will talk about how the structure of a Spark application puts important constraints on its scalability; how can we find these structural constraints and how to use them as a guide in solving performance and scalability problems of Spark applications.

This talk will help the audience with answering the following questions about their Spark applications: 1) Will their application run faster with more executors? 2) How will cluster utilization change as the number of executors changes? 3) What is the absolute minimum time this application will take even if we give it infinite executors? 4) What is the expected wall clock time for the application when we fix the most important structural limits of these applications?

Sparklens makes the ROI of additional executors extremely obvious for a given application and needs just a single run of the application to determine how the application will behave with different executor counts. Specifically, it will help managers take the correct side of the tradeoff between spending developer time optimizing applications vs. spending money on compute bills.

Ashish is a Big Data leader and practitioner with more than 15 years of industry experience. Equipped with immense experience involving the design and development of petabyte-scale Big Data applications, he is a seasoned technology architect with variegated experiences in customer interfacing and technical leadership roles.

ABOUT THE SPEAKER

Ashish heads Qubole's Solutions Architecture team for International Markets, and works with a number of enterprise customers in the EMEA, APAC and India regions. Prior to Qubole, Ashish worked at Microsoft as an engineer in the Windows team. Later, he worked for Claraview (Teradata), while leading their Big Data practice and helped to scale some of their Fortune 500 clients in different industry verticals such as finance, healthcare, retail and multimedia.

ABOUT DATA COUNCIL:
Data Council (https://www.datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers. Make sure to subscribe to our channel for more videos, including DC_THURS, our series of live online interviews with leading data professionals from top open source projects and startups.

FOLLOW DATA COUNCIL:
Twitter: https://twitter.com/DataCouncilAI
LinkedIn: https://www.linkedin.com/company/datacouncil-ai
Facebook: https://www.facebook.com/datacouncilai
Eventbrite: https://www.eventbrite.com/o/data-council-30357384520