02Jun2026
14:00 Master's Defense IC3 Auditorium
Topic on
FaaS-Dask: Towards a scalable and financially efficient serverless scheduler for Dask
Student
Carlos Eduardo Millani
Advisor / Teacher
Edson Borin - Co-advisor: Carlos Alberto Astudillo Trujillo
Brief summary
Scientific workflows for large-scale data, such as seismic processing, present significant challenges for interactive data exploration. Traditional High-Performance Computing (HPC) clusters often subject users to long waiting times in queues, while cloud environments based on Infrastructure as a Service (IaaS) can generate prohibitive costs due to idle resources between user interactions. To address the trade-off between latency and cost, this work presents FaaS-Dask, a cloud-native workflow scheduler based on the Dask framework. FaaS-Dask is designed to utilize serverless Function as a Service (FaaS) instances, providing a scalable and financially efficient environment suitable for the sporadic nature of interactive workloads. This dissertation details the architecture and iterative evolution of FaaS-Dask. Starting from an initial implementation that replaces Dask's local workers with serverless functions, we apply a series of architectural and task graph-level optimizations. Architectural improvements included optimizing the task dispatch mechanism by migrating from asynchronous queue-based triggers (AWS SQS) to lower-latency synchronous API calls, and enhancing input/output (I/O) performance by transitioning the persistent storage layer from object storage (AWS S3) to a network file system (AWS EFS). Additionally, we leveraged Dask's optimization capabilities to modify the Directed Acyclic Graph (DAG) of tasks, applying constant inlining and sequential task merging, which significantly reduced the overhead associated with function invocations and data transfers inherent in the stateless FaaS paradigm. To validate our approach, we conducted a comprehensive experimental evaluation, comparing multiple versions of FaaS-Dask with traditional Dask clusters deployed on virtual machines. Using representative workflows of seismic data processing, we measured execution latency and total cost. The results demonstrate that successive optimizations substantially reduced both metrics. The final, optimized version of FaaS-Dask achieved latencies comparable to traditional clusters, offering a significant cost reduction, mainly through the elimination of idle time charges. This work concludes that a serverless-based scheduler is a viable and efficient alternative for the interactive exploration of scientific data, effectively balancing performance and cost.
Examination Board
Headlines:
| Edson Borin | IC / UNICAMP |
| Lúcia Maria de Assumpção Drummond | IC / UFF |
| Edmundo Roberto Mauro Madeira | IC / UNICAMP |
Substitutes:
| Hervé Cédric Yviquel | IC / UNICAMP |
| Hermes Senger | ICMC / USP |