A recent study reported an increased risk of contralateral estrogen-negative breast cancer after a first primary estrogen-negative breast cancer. Our study aims to confirm this result and to evaluate how the risk of second breast cancer occurrence is affected by family history of breast cancer and anti-estrogen treatment. We included all 4,152 women diagnosed with breast cancer between 1995 and 2007, using data from the population-based Geneva Cancer Registry. We compared the incidence of second breast cancer among patients according to estrogen receptor (ER) status with that expected in the general population by age-period Standardized Incidence Ratios (SIRs). Among the cohort, 63 women developed second breast cancer. Patients with ER-positive first tumors had a decreased risk of second breast cancer occurrence (SIR: 0.67, 95% CI: 0.48-0.90), whereas patients with ER-negative primary tumors had an increased risk (SIR: 1.98, 95% CI: 1.19-3.09) limited to ER-negative second tumors (SIR: 7.94, 95% CI: 3.81-14.60). Patients with positive family history had a tenfold (SIR: 9.74, 95% CI: 3.57-21.12) higher risk of ER-negative second tumor which increased to nearly 50-fold (SIR: 46.18, 95% CI: 12.58-118.22) when the first tumor was ER-negative. Treatment with anti-estrogen decreased the risk of second ER-positive tumors but not ER-negative tumors. The risk of second ER-negative breast cancer is very high after a first ER-negative tumor, in particular among women with strong family history. Surveillance and prevention of second cancer occurrence should consider both ER status of the first tumor and family history.