We present simulation results for optical modes in micro-pillar cavities that were computed with the finite element method and that show good agreement with experimental data. By means of this viable tool various influences on the quality factor of the fundamental mode were calculated: Firstly, the light confinement depends strongly on the absorption of the semiconductor cavity material. Here we were able to determine absolute maximum quality factors achievable in a GaAs/AlAs Bragg micro-pillar cavity. Furthermore, small pillar diameters as well as the inclination of pillar sidewalls show critical features with respect to light confinement. Additional effects of the top and bottom Bragg stacks in the pillar were calculated as well.