Recent studies indicate that lifestyle factors in early life affect breast cancer risk. We therefore explored the association of high consumption of meat, milk, and whole grain products in adolescence and midlife, on breast cancer risk. We used data from the population based AGES-Reykjavik cohort (2002-2006), where 3,326 women with a mean age of 77 years (SD 6.0) participated. For food items and principal component derived dietary patterns we used Cox proportional models to calculate multivariate hazard ratios (HR) with 95% confidence intervals (95% CI). During a mean follow-up of 8.8 years, 97 women were diagnosed with breast cancer. For both adolescence and midlife, daily consumption of rye bread was positively associated with breast cancer (HR 1.7, 95% CI 1.1-2.6 and HR 1.8, 95% CI 1.1-2.9, respectively). In contrast, persistent high consumption of oatmeal was negatively associated with breast cancer (0.4, 95% CI 0.2-0.9). No association was found for other food items or dietary patterns that included rye bread. High rye bread consumption in adolescence and midlife may increase risk of late-life breast cancer whilst persistent consumption of oatmeal may reduce the risk.