Large-scale pattern recognition: Dynamic load balancing using code mobility

Abstract

Applications such as traffic monitoring, crowd management or gesture recognition systems intend to recognize patterns in streams of data captured by their sensors, and react to them accordingly. Large-scale pattern recognition is concerned with the recognition of complex patterns in these data-intensive streams. Programming these systems is hard, which is why a rule-based approach is preferable: a declarative language allows the programmer to define rules that describe patterns, and how to react to them.

As both the sensor hardware advances, e.g. more cameras are used or they support higher resolutions, and more complex patterns need to be recognized, the amount of data and the amount of rules and their complexity will increase. This poses a problem: how do we scale our system to handle this increasing complexity? Existing solutions for rule-based systems are limited to a single machine or require manual programming to scale.

The solution proposed in this thesis is to distribute the work over multiple machines. Based on PARTE (Parallel Actor-based ReTe Engine), which parallelizes the Rete algorithm using an actor-based approach, a system is created in which these actors are distributed over the available machines: DPARTE (Distributed PARTE).

The system uses dynamic load balancing to redistribute the work while the program is running. This is done by introducing code mobility into the system: the actors become "mobile actors", they will move at run time to distribute the work load evenly.

Based on an experimental evaluation, we substantiate our claims that distributing the work over multiple machines leads to a scalable and elastic system that can handle increasing amounts of work, and that dynamic load balancing provides an additional benefit in distributing the work evenly over the available machines.

To our knowledge, DPARTE is therefore the first Rete-based rule engine with a unified fact base that supports automatic distribution and dynamic load balancing.

Date of Award	Jun 2013
Original language	English
Supervisor	Wolfgang De Meuter (Promotor), Thierry Renaux (Advisor), Lode Hoste (Advisor) & Stefan Marr (Advisor)