Debugging support for Big Data processing applications

Big Data applications are becoming popular nowadays and being used to analyze big sets of data from different domains. Many frameworks exist for developers to develop and execute their applications like Hadoop MapReduce and Apache Spark. However, very few debugging support is currently provided in those frameworks. When an error happens, developers are lost in trying to understand what happened from the information provided in log files or use solutions to replay the execution which are slow and time-consuming. Current trends indicate that Big Data frameworks need to meet the requirements of data processing applications analyzing streams of data, rather than only batches of saved data. Properties of these streaming applications demand novel debugging tools that go beyond the concepts of stepping and replaying.
To support the development of Big Data applications, we aim to design and implement rich debugging support that helps developers to understand software failures and fix them. We will explore the combination of online and post-mortem debugging techniques which capture the global view of a Big Data application while avoiding to delve into logs from the large amount of software components forming part of the Big Data framework. Instead of developing an entirely new programming language platform and runtime environment, we propose to prototype the new debugging techniques and tools in an existing programming environment running a top of a mainstream Big Data technology.