Physical inactivity is an important and modifiable cardiovascular disease risk factor. Little is known about the social determinants of physical inactivity in older, urban-dwelling populations. We collected socio-demographic and medical risk factor information and physical activity questionnaires in the Northern Manhattan Study. Logistic regression models were constructed to examine whether measures of social isolation, race-ethnicity, and sex were associated with physical inactivity. Physical inactivity was present in 40.5% of the cohort. In multivariable models adjusted for medical comorbidities, Hispanic race-ethnicity (compared to non-Hispanic white) was associated with higher odds of physical inactivity (OR 2.18, 95% CI 1.78, 2.67), while women were more likely to be inactive than men (OR 1.33, 95% CI 1.15, 1.54). Having Medicaid/being uninsured (OR 1.20, 95% CI 1.02, 1.42), and having fewer than 3 friends (1.41, 95% CI 1.15, 1.72) were also associated with physical inactivity. Physical inactivity is common, particularly in Hispanics, women, and those who are socially isolated. Public health interventions aimed at increasing physical activity in these more sedentary groups are required.