This study sought to compare the prevalence of, and relationship between, age at first experience of sexual violence and HIV and other health risk behaviors in two populations of men at high risk of HIV infection. Data were drawn from two cohorts: Vanguard, a prospective study of young men who have sex with men (MSM), and VIDUS, the Vancouver Injection Drug Users Study. Controlling for fixed sociodemographics, multivariate logistic regression was used to assess the relationship between age at first sexual violence (vs. never experiencing it) and several health risk behaviors. There were 140/498 (28%) MSM from Vanguard and 173/932 (19%) injection drug users (IDU) from VIDUS who reported having experienced sexual violence. Among VIDUS men, 130/852 (15%) IDU-only and 43/80 (54%) who were both IDU and MSM reported a history of sexual violence. The prevalence of child sexual abuse was 13% in Vanguard MSM, and 11% among VIDUS IDU-only, but 26% among VIDUS MSM/IDU. The median age of onset was significantly lower among VIDUS IDU-only compared to the two other groups. Experiencing sexual violence first in childhood was strongly related to ever being in the sex trade in both IDU and MSM. MSM in Vanguard who experienced sexual violence in childhood were more likely to have attempted suicide, and have a diagnosed mood disorder. Non-MSM IDU in VIDUS who experienced sexual violence in childhood were more likely to have a diagnosed mental illness, to binge on alcohol, and to have ever accidentally overdosed. In conclusion, men who have ever had sex with men appear to have a higher lifetime prevalence of sexual violence, compared to non-MSM injection drug users. Sexual violence is differentially associated with different health risk behaviors, depending on the age at first occurrence and the primary HIV risk factor (i.e. MSM vs. IDU).