We search for evidence of a light scalar boson in the radiative decays of the Upsilon(2S) and Upsilon(3S) resonances: Upsilon(2S,3S)-->gammaA0, A0-->mu+mu-. Such a particle appears in extensions of the standard model, where a light CP-odd Higgs boson naturally couples strongly to b quarks. We find no evidence for such processes in the mass range 0.212 < or = mA0 < or = 9.3 GeV in the samples of 99 x 10(6) Upsilon(2S) and 122 x 10(6) Upsilon(3S) decays collected by the BABAR detector at the SLAC PEP-II B factory and set stringent upper limits on the effective coupling of the b quark to the A0. We also limit the dimuon branching fraction of the etab meson: B(etab-->mu+mu-)<0.9% at 90% confidence level.