On models and metrics: Improving stochastic model performance in manpower planning

When scientists stumble upon an intriguing scientific modelling problem, they face some crucial deci-
sions. They need to select an appropriate framework for addressing the modelling challenge, comprising
two essential aspects. Firstly, they need to determine the most suitable model type for the specific prob-
lem. Additionally, and often overlooked, they must define the criteria that will be used to judge the
performance of the models at hand – in other words, they need to set the indicators to tell apart good from
bad performing models. This PhD thesis contributes to both facets of this framework within the context
of manpower planning.
In the first chapter, we delve into existing models and their properties seeking to generalise them
to various settings. Specifically, the concept of maintainability for Markov chains is explored for and
extended to semi-Markov chains. This extension leads to the introduction of the new concept of ’state
re-union maintainability’. The second chapter is devoted to attainability for Markov and semi-Markov
chains. Based on the insights from the first chapter, the concept of ’state re-union attainability’ is de-
veloped and explored. The notion of model selection emerges in the third chapter when balancing the
importance of goodness of fit with the number of estimated parameters. Here, the focus is on the use of
semi-Markov models. Leveraging well-defined model criteria, a novel model type is developed, termed
the ’hybrid semi-Markov model’. The chapter highlights the need for the right model type selection to
effectively handle the trade-off between Markov and semi-Markov models. In the fourth chapter, the
emphasis further shifts to model evaluation metrics. Within the realm of uplift modelling, the Qini score
is considered the golden standard for evaluation purposes but lacks theoretical foundation. To address
this, a novel metric inspired by the Receiver Operating Characteristic (ROC) curve, named the ROCini
score, is developed here. The ideas behind this metric are then used in the context of ordinal dominance
graphs, leading to the creation of the pROCini score. This chapter concludes with a simulation study that
empirically validates the improved discriminative power of the (p)ROCini scores compared to the Qini
score.
By addressing these aspects, this thesis aims to enhance our understanding of modelling choices and
criteria for assessing model effectiveness in the context of manpower planning and related fields.