Intracellular recordings were obtained from neurons of the laterodorsal tegmental and pedunculopontine tegmental nuclei in a brain-slice preparation. The action of exogenously applied 5-hydroxytryptamine and acetylcholine was studied on NADPH-diaphorase-labeled cells which contain nitric oxide synthase and are presumed to be cholinergic. Our results indicated that these cells were hyperpolarized by both 5-hydroxytryptamine and acetylcholine; the ionic mechanism of this inhibition was investigated using current and voltage clamp methods. Cells voltage-clamped at resting membrane potential exhibited a net outward current and an increased membrane conductance during 5-hydroxytryptamine and acetylcholine mediated inhibition. The membrane hyperpolarization and outward current generated by this paradigm reversed near the expected K equilibrium potential and was blocked by low concentrations of extracellular Ba. The 5-hydroxytryptamine- and acetylcholine-dependent currents showed inward rectification and the reversal potential shifted in the depolarizing direction by about 15 mV for a doubling of extracellular K, indicating that both 5-hydroxytryptamine and acetylcholine activate inwardly rectifying, potassium-selective conductances. The 5-hydroxytryptamine-evoked hyperpolarization was antagonized by spiperone and mimicked by (+)8-hydroxy-2-(Di-N-propylamino)-tetralin suggesting the presence of a 5-hydroxytryptamine1A receptor while the acetylcholine-evoked hyperpolarization was blocked by atropine and only high concentrations of pirenzepine, suggesting a muscarinic M2 receptor. The outward currents evoked by 5-hydroxytryptamine and acetylcholine were not additive, suggesting that both receptors are coupled to an overlapping pool of K channels as has been observed in several systems in which receptors are coupled to effectors by G-proteins. These results indicate that the dominant actions of 5-hydroxytryptamine and acetylcholine relate to the inhibition of mesopontine cholinergic neurons via activation of an overlapping pool of inwardly rectifying K channels. Cholinergic neurons of these nuclei are thought to play an instrumental role in the induction and maintenance of rapid eye movement sleep. It has been previously hypothesized that acetylcholine would be excitatory and that 5-hydroxytryptamine would be inhibitory to these cells in the context of rapid eye movement sleep. [McCarley R. and Massaquoi S. (1986) Am. J. Physiol. 251, R1011-R1029; McCarley R. W. et al. (1975) Science 189, 58-60]. Our results are consistent with the proposed inhibitory action of 5-hydroxytryptamine but indicate recurrent input to cholinergic neurons would be inhibitory. Accordingly, models of the neural substrate underlying rapid eye movement sleep production need to be changed to reflect this inhibitory action of acetylcholine on cholinergic neurons.