Data on the association between cognition and testosterone levels in elderly men are inconclusive. Androgen deprivation therapy is commonly used in the treatment of prostate cancer with the aim of achieving castration levels of serum testosterone. The study group comprised 26 elderly men (mean age 65 years) with newly diagnosed prostate cancer. Cognitive testing was done at baseline and at 6 and 12 months on androgen deprivation therapy. Cognitive performances were evaluated using verbal, visuomotor, and memory tests as well as tests of processing speed and attention. Castration levels of testosterone were achieved in all patients by 6 months. Significant associations between cognitive performances and testosterone decline were documented: visuomotor slowing, slowed reaction times in some attentional domains including working memory and impaired hit rate in a vigilance test, impaired delayed recall and recognition speed of letters, but improvement in object recall. The results suggest selective associations between testosterone decline and cognition. Documentation of cognitive performance with changes in serum testosterone levels has substantial implications for informed patient support in prostate cancer.