The protective efficacy of neutralizing antibodies (nAbs) elicited during natural infection with SARS-CoV-2 and by vaccination based on its spike protein has been compromised with emergence of the recent SARS-CoV-2 variants. Residues E484 and K417 in the receptor-binding site (RBS) are both mutated in lineages first described in South Africa (B.1.351) and Brazil (B.1.1.28.1). The nAbs isolated from SARS-CoV-2 patients are preferentially encoded by certain heavy-chain germline genes and the two most frequently elicited antibody families (IGHV3-53/3-66 and IGHV1-2) can each bind the RBS in two different binding modes. However, their binding and neutralization are abrogated by either the E484K or K417N mutation, whereas nAbs to the cross-reactive CR3022 and S309 sites are largely unaffected. This structural and functional analysis illustrates why mutations at E484 and K417 adversely affect major classes of nAbs to SARS-CoV-2 with consequences for next-generation COVID-19 vaccines.