Testosterone (T) levels are decreased in obese men, but the underlying causes are incompletely understood. Our objective was to explore the relation between low (free) T levels and male obesity, by evaluating metabolic parameters, subcutaneous adipose tissue (SAT) aromatase expression, and parameters of the hypothalamic-pituitary-gonadal axis. We recruited 57 morbidly obese men [33 had type 2 diabetes (DM2)] and 25 normal-weight men undergoing abdominal surgery. Fourteen obese men also attended a follow-up, 2 years after gastric bypass surgery (GBS). Circulating T levels were quantified by LC-MS/MS, whereas free T levels were measured using serum equilibrium dialysis and sex hormone-binding globulin, luteinizing hormone, and follicle-stimulating hormone by immunoassay. SAT biopsies were used to determine adipocyte cell size and aromatase expression by real-time PCR. Total and free T levels were decreased in obese males versus controls, with a further decrease in obese men with DM2 versus obese men without DM2. There were no differences in aromatase expression among the study groups, and sex steroids did not correlate with aromatase expression. Pearson analysis revealed an inverse association between (free) T and SAT cell size, triglycerides, and HOMA-IR. Multivariate analysis confirmed the inverse association between (free) T and SAT cell size (β = -0.321, P = 0.037 and β = -0.441, P = 0.011, respectively), independent of age, triglycerides, HOMA-IR, obesity, or diabetes. T levels were normalized 2 years after GBS. These data suggest that SAT cell size rather than SAT aromatase expression or parameters of the hypothalamic-pituitary-gonadal axis is related to low T in male obesity, which points to adipose cell size-related metabolic changes as a major trigger in decreased T levels.