Decoherence in nearly isolated GaAs quantum dots is investigated using the change in the average Coulomb blockade peak height when time-reversal symmetry is broken. The normalized change in the average peak height approaches the predicted universal value of 1/4 at temperatures well below the single-particle level spacing, T < Delta, but is greatly suppressed for T > Delta, suggesting that inelastic scattering or other dephasing mechanisms dominate in this regime.