Compact and broadband non-volatile silicon devices are mainly absorption based. Hence, access to low-loss non-volatile phase shifters is still a challenge. Here, this problem is addressed by using a high-mobility transparent conducting oxide such as cadmium oxide as a floating gate in a flash-like structure. This structure is integrated in a Mach-Zehnder interferometer switch. Results show an active length of only 30 µm to achieve a $ \pi $π phase shift. Furthermore, an extinction ratio of 20 dB and insertion loss as low as 1 dB may be attained. The device shows an optical broadband response and can be controlled with low-power pulses in the nanosecond range. These results open a new, to the best of our knowledge, way for enabling compact silicon-based phase shifters with non-volatile performance.