In a recent large-scale prevalence study of interpersonal violence (IV) against child athletes in the Netherlands and Belgium we found that 9% of adult respondents who participated in organized sports before the age of 18 had experienced severe psychological violence, 8% severe physical violence, and 6% severe sexual violence in various sport settings. While the general literature has repeatedly shown that exposure to IV during childhood is associated with mental health problems in adulthood and to a lesser extent with reduced quality of life (QOL), these relationships have not been demonstrated in (former) athletes. Thus, the current study aims to assess the association of severe childhood IV in sport and adult wellbeing. Depression, anxiety, and somatization were assessed in the same general population sample (N = 4043) using the Brief Symptom Inventory (BSI-18) and QOL with the World Health Organization Quality of Life questionnaire (WHOQOL-Brèf). The association between severe IV in sport and adult wellbeing was investigated using multiple linear regression while controlling for demographics, recent life events, and relatives' psychological problems. We found severe sexual, physical, and psychological childhood IV in sport to be associated with more adult psychological distress and reduced QOL. Polyvictimization shows the strongest correlation with poorer wellbeing and QOL. Recent life events, relatives' psychological problems, marital status, and level of education were significant covariates in the psychological symptoms and QOL assessed. We hope that these new insights prompt sport administrators to implement broad spectrum child protection measures and raise the awareness of mental health professionals about the necessity to also screen for adverse childhood experiences in the sport context.