Most van der Waals crystals present highly anisotropic optical responses due to their strong in-plane covalent bonding and weak out-of-plane interactions. However, the determination of the polarization-dependent dielectric constants of van der Waals crystals remains a nontrivial task, since the size and dimension of the samples are often below or close to the diffraction limit of the probe light. In this work, we apply an optical nano-imaging technique to determine the anisotropic dielectric constants in representative van der Waals crystals. Through the study of both ordinary and extraordinary waveguide modes in real space, we are able to quantitatively determine the full dielectric tensors of nanometer-thin molybdenum disulfide and hexagonal boron nitride microcrystals, the most-promising van der Waals semiconductor and dielectric. Unlike traditional reflection-based methods, our measurements are reliable below the length scale of the free-space wavelength and reveal a universal route for characterizing low-dimensional crystals with high anisotropies.