Neurofilament light (NfL) protein is a marker of neuro-axonal damage and can be measured not only in cerebrospinal fluid but also in serum, which allows for repeated assessments. There is still limited knowledge regarding the association of serum NfL (sNfL) with age and subclinical morphologic brain changes and their dynamics in the normal population. We measured sNfL by a single molecule array (Simoa) assay in 335 individuals participating in a population-based cohort study and after a mean follow-up time of 5.9 years (n = 103). Detailed clinical examination, cognitive testing and 3T brain MRI were performed to assess subclinical brain damage. We show that rising and more variable sNfL in individuals >60 years indicate an acceleration of neuronal injury at higher age, which may be driven by subclinical comorbid pathologies. This is supported by a close association of sNfL with brain volume changes in a cross-sectional and especially longitudinal manner.