A father’s age at conception is predicted to affect not only his own fertility but also his offspring’s fitness. Offspring born to old fathers are assumed to be less fit than those of young fathers. However, under low mating rates, paternal age might be confounded with the duration for which mature sperm are stored in fathers prior to ejaculation. Studies that disentangle the confounding paternal effects of sperm storage duration from those of age on offspring are lacking. We use Drosophila melanogaster to test the separate and interactive effects of paternal age and sexual rest on offspring fitness. As expected, old fathers produce fewer offspring than young fathers; however, paternal age does not influence the survival or age-dependent reproductive success of sons or daughters. Instead, a long duration of paternal sexual rest negatively impacts the reproductive success of the conceived sons. Furthermore, daughters of low reproductive quality selectively disappear with age, but sons do not, highlighting that demographic processes can further modulate paternal age effects. Overall, we highlight that paternal age effects might not be as pervasive as previously assumed and suggest that paternal sexual rest might be more important in influencing offspring phenotypes.