A new method is presented for retrieval of the aerosol and cloud optical depth using a CCD camera equipped with a fish-eye lens (all-sky imager system). In a first step, the proposed method retrieves the spectral radiance from sky images acquired by the all-sky imager system using a linear pseudoinverse algorithm. Then, the aerosol or cloud optical depth at 500 nm is obtained as that which minimizes the residuals between the zenith spectral radiance retrieved from the sky images and that estimated by the radiative transfer code. The method is tested under extreme situations including the presence of nonspherical aerosol particles. The comparison of optical depths derived from the all-sky imager with those retrieved with a sunphotometer operated side by side shows differences similar to the nominal error claimed in the aerosol optical depth retrievals from sunphotometer networks.