We analyze the influence of the anisotropy of molecular and biological samples on polarization resolved nonlinear microscopy imaging. We show in particular the detrimental influence of birefringence on Second Harmonic Generation (SHG) and Two-Photon Excited Fluorescence (TPEF) polarization resolved microscopy imaging, which, if not accounted for, can lead to an erroneous determination of the sample properties and thus to a misinterpretation of the read-out information. We propose a method to measure this birefringence and account for this effect in nonlinear polarization resolved experiments.