Primary cilia are involved in a variety of physiological processes such as sensing of the environment, cell growth and development. Numerous developmental disorders and pathologies arise from defects in these organelles. Multiple proteins that promote formation and disassembly of the primary cilium have been identified, but little is known about the mechanisms that control steady-state cilium size. Here, we show that death inducer obliterator (Dido3)-dependent targeting of histone deacetylase 6 (HDAC6) is a key determinant of cilium size in growth-arrested cells. The amount of either protein negatively correlates with cilium size. Dido3 availability at the centrosome governs ciliary HDAC6 levels, and redistribution of the two proteins controls tubulin acetylation. In turn, basal body localization of Dido3 and HDAC6 depends on the actin network, previously shown to limit cilium size independent of the cell cycle. These results show that not only kinase-dependent activation of a deacetylase but also its subcellular distribution controls substrate selection.