Several genetic polymorphisms have been associated with Late Onset Alzheimer's Disease (LOAD), but there has been limited evidence on whether these polymorphisms predict intermediary stage outcomes such as cognitive changes in prospective community-based studies. Our aim was to evaluate whether polymorphisms previously established as predictors of LOAD also predict worse cognitive function and accelerated decline across multiple cognitive domains. We analyzed data from the 3C-Dijon study, in which 4931 respondents aged 65+ were examined up to 5 times over 10 years with a neuropsychological assessment. We evaluated the associations of polymorphisms in APOE, CR1, BIN1, CLU, PICALM, ABCA7, MS4A6A, CD33, MS4A4E and CD2AP with level and change in 5 neuropsychological tests, assuming a dominant effect model. To optimize measurement, we used a mixed regression model with a latent process for each cognitive domain: global cognition (Mini Mental State Examination); verbal fluency (Isaac's Set Test); visual memory (Benton Visual Retention Test); information processing (Trail Making Test B) and literacy (National Adult Reading Test). APOE was associated with accelerated decline in global cognition and verbal fluency. Only two non-APOE genetic polymorphisms were associated with cognitive decline: CR1 was associated with rate of change in verbal fluency and BIN1 was associated with rate of change in global cognition. In a large prospective population-based study of dementia-free individuals, only a few cognitive domains were associated with established LOAD risk alleles. The most consistent associations were for global cognition and verbal fluency.