We study the collective dynamics of a driven two-mode Bose-Hubbard model in the Josephson interaction regime. The classical phase space is mixed, with chaotic and regular components, which determine the dynamical nature of the fringe visibility. For a weak off-resonant drive, where the chaotic component is small, the many-body dynamics corresponds to that of a Kapitza pendulum, with the relative phase φ between the condensates playing the role of the pendulum angle. Using a master equation approach we show that the modulation of the intersite potential barrier stabilizes the φ=π "inverted pendulum" coherent state, and protects the fringe visibility.