Single crystals of EuNiGe3, crystallizing in the non-centrosymmetric BaNiSn3-type structure, were grown using In flux, enabling us to explore the anisotropic magnetic properties, which was not possible with previously reported polycrystalline samples. The EuNiGe3 single crystalline sample is found to order antiferromagnetically at 13.2 K, as revealed from the magnetic susceptibility, heat capacity and electrical resistivity data. The low temperature magnetization M (H) is distinctly different for the field parallel to the ab-plane and c-axis; the ab-plane magnetization varies almost linearly with the field before the occurrence of an induced ferromagnetic (FM) phase (spin-flip) at 6.2 Tesla. On the other hand M (H) along the c-axis is accompanied by two metamagnetic transitions followed by a spin-flip at 4.1 T. A model including anisotropic exchange and dipole-dipole interactions reproduces the main features of magnetization plots but falls short of full representation. (H,T) phase diagrams have been constructed for the field applied along the principal directions. From the (151)Eu Mössbauer spectra, we determine that the 13.2 K transition leads to an incommensurate antiferromagnetic (AFM) intermediate phase followed by a transition near 10.5 K to a commensurate AFM configuration.