Methods, modes, and measurement errors: Mapping the validity of self-administered online time-diaries

Technological advances in society are typically leading to changes in research
methodology in the social sciences. One of the most influential recent technological
developments that has changed both society and research methodology in the social
sciences is digitisation. Digitisation also changed time-use research, that has experienced
a shift from telephone and paper-and-pencil time-diaries to online time-diaries. It is
frequently argued that the benefits of this shift include making time-diary research
cheaper, reducing the time investment for researchers and limiting the participation
burden for respondents.
At the same time, the shift to online time-diaries is accompanied by at least four
ramifications. First, online participation imposes additional requirements on
respondents in order to participate, such as access to a suitable device and the Internet,
and the competence to use them correctly. Second, the role of the interviewer has
changed profoundly. In telephone and paper-and-pencil time-diaries, interviewers are
present to a greater or lesser extent during the data collection, whereas they are largely
to completely absent in self-administered online time-diary methodology. Third, the
omnipresence of smartphones has increased, enabling respondents to participate in a
time-diary study anytime, anywhere, using their smartphone. Fourth, online registration
by respondents provides more extensive and more detailed paradata on participation
behaviour of respondents during time-diary studies. These ramifications affect the
internal and external validity of time-diaries as well as the approach to monitoring
validity. This raises the question as to what extent self-administered online time-diaries
induce valid time-use estimates and to what extent the validity is affected by
measurement errors. This main question was in this dissertation translated into
subquestions that were answered in four quantitative empirical chapters, previously
published in international peer-reviewed journals.
The first chapter describes how time-diary methods offer practical remedies for
cognitive problems in reporting working hours in stylised questions. Subsequently, it
compares estimates of working hours obtained from a stylised question with those from
a time-diary. Both registration methods initially show only small differences, although
large differences are found in in-depth analyses, which are consistently related to
sociodemographic background and type of work activities.
The second chapter explores the extent to which the representativeness of online time-
diary surveys on the individual level is compromised by the selectivity of nonresponse.
By employing mortality tables and event history analysis on paradata, the retention of
participants through the various phases of the study is examined. The results show that
finishing a time-diary study is selective, that in particular the elderly and less educated
participate less, and that they drop out more often during the study.
The third chapter focuses on the representativeness of online time-diaries on the level
of the day (i.e., weekdays, weekend days and holidays). We investigate which
respondents participate on the sampled (or designated) days or participate on other
days. The results show that nonparticipation on designated days is selective and affects
the validity of duration and timing parameters of working hours.
The fourth chapter examines the extent to which variations in the recall period impairs
the data quality of time-diaries due to recall bias. In contrast to offline time-diaries in
which the recall period was determined by the interviewer, in online time-diaries it is
the respondent’s responsibility. The recall period differs according to the type of
respondent and a longer recall period reduces data quality. For busy respondents, the
data quality decreases even faster as the recall period increases.
In sum, it can be concluded that the shift to self-administered online time-diaries is
mainly beneficial for researchers, whereas it has proved less positive for respondents
who still struggle with the complexity and high participation burden. To overcome this,
respondents adapt their participation behaviour, which undermines the validity of the
method. Future adaptations to the measuring instrument should prioritise the reduction
of the complexity and the burden of participation from the respondent’s perspective.
Further research should accompany such adjustments from two angles: on the one hand
from the user experience of respondents and on the other hand to monitor the effects
of these adjustments on the validity of the method.