Background In older populations disturbed 24-h activity rhythms, poor sleep, and depressive symptoms are often lingering and co-morbid, making treatment difficult. To improve insights into these commonly co-occurring problems, we assessed the bidirectional association of sleep and 24-h activity rhythms with depressive symptoms in middle-aged and elderly persons. Methods In 1734 participants (mean age: 62.3 ± 9.3 years, 55% women) from the prospective Rotterdam Study, 24-h activity rhythms and sleep were estimated with actigraphy (mean duration: 146 ± 19.6 h), sleep quality with the Pittsburgh Sleep Quality Index, and depressive symptoms with the Center for Epidemiological Studies Depression scale. Repeated measures were available for 947 participants (54%) over a median follow-up of 6 years (interquartile range = 5.6-6.3). Linear-mixed models were used to assess temporal associations of 24-h activity rhythms and sleep with depressive symptoms in both directions. Results High 24-h activity rhythm fragmentation (IV) (B = 1.002, 95% confidence interval (CI) = 0.641-1.363), long time in bed (TIB) (B = 0.111, 95% CI = 0.053-0.169), low sleep efficiency (SE) (B = -0.015, 95% CI = -0.020 to -0.009), long sleep onset latency (SOL) (B = 0.009, 95% CI = 0.006-0.012), and low self-rated sleep quality (B = 0.112, 95% CI = 0.0992-0.124) at baseline were associated with increasing depressive symptoms over time. Conversely, more depressive symptoms at baseline were associated with an increasing 24-h activity rhythm fragmentation (B = 0.002, 95% CI = 0.001-0.003) and TIB (B = 0.009, 95% CI = 0.004-0.015), and a decreasing SE (B = -0.140, 95% CI = -0.196 to -0.084), SOL (B = 0.013, 95% CI = 0.008-0.018), and self-rated sleep quality (B = 0.193, 95% CI = 0.171-0.215) over time. Conclusion This study demonstrates a bidirectional association of 24-h activity rhythms, actigraphy-estimated sleep, and self-rated sleep quality with depressive symptoms over a time frame of multiple years in middle-aged and elderly persons.