Serological tests are an indispensable tool to understand the epidemiology of the SARS-CoV-2 pandemic, particularly in areas where molecular diagnostics are limited. Poor assay performance may hinder the utility of these tests, including high rates of false-positivity previously reported in sub-Saharan Africa. From 312 Malian samples collected prior to 2020, we measured antibodies to the commonly tested SARS-CoV-2 antigens and four other betacoronaviruses by ELISA, and assessed functional cross-reactivity in a subset by SARS-CoV-2 pseudovirus neutralization assay. We then evaluated the performance of an ELISA developed in the US, using two-antigen SARS-CoV-2 spike protein and receptor-binding domain. To optimize test performance, we compared single and two-antigen approaches using existing assay cutoffs and population-specific cutoffs for Malian control samples (positive and negative). Background reactivity to SARS-CoV-2 antigens was common in pre-pandemic samples compared to US controls (43.4% (135/311) for spike protein, 22.8% (71/312) for RBD, and 33.9% (79/233) for nucleocapsid protein). SARS-CoV-2 reactivity correlated weakly with other betacoronavirus reactivity, varied between Malian communities, and increased with age. No pre-pandemic samples demonstrated functional activity. Regardless of the cutoffs applied, specificity improved using a two-antigen approach. Test performance was optimal using a two-antigen assay with population-specific cutoffs derived from ROC curve analysis [Sensitivity: 73.9% (51.6-89.8), Specificity: 99.4% (97.7-99.9)]. In the setting of high background reactivity, such as sub-Saharan Africa, SARS-CoV-2 serological assays need careful qualification is to characterize the epidemiology of disease, prevent unnecessary harm, and allocate resources for targeted control measures.