We present a study of the pressure dependence of the structure of partially hydrated hexagonal CePO4 up to 21 GPa using synchrotron powder X-ray diffraction. At a pressure of 10 GPa, a second-order structural phase transition is observed, associated with a novel polymorph. The previously unknown high-pressure phase has a monoclinic structure with a similar atomic arrangement as the low-pressure phase, but with reduced symmetry, belonging to space group C2. Group-subgroup relations hold for the space symmetry groups of both structures. There is no detectable volume discontinuity at the phase transition. Here we provide structural information on the new phase and determine the axial compressibility and bulk modulus for both phases. They are found to have an anisotropic behavior and to be much more compressible than the denser monazite-like polymorph of CePO4. In addition, the isothermal compressibility tensor for the high-pressure structure is reported at 10 GPa and the direction of maximum compressibility described. Finally, the possible role of water and the pressure medium in the high-pressure behavior is discussed. The results are compared with those from other rare-earth orthophosphates.