Simian varicella virus (SVV) infection of rhesus macaques (RMs) recapitulates the hallmarks of varicella-zoster virus (VZV) infection of humans, including the establishment of latency within the sensory ganglia. Various factors, including age and immune fitness, influence the outcome of primary VZV infection, as well as reactivation resulting in herpes zoster (HZ). To increase our understanding of the role of lymphocyte subsets in the establishment of viral latency, we analyzed the latent SVV transcriptome in juvenile RMs depleted of CD4 T, CD8 T, or CD20 B lymphocytes during acute infection. We have previously shown that SVV latency in sensory ganglia of nondepleted juvenile RMs is associated with a limited transcriptional profile. In contrast, CD4 depletion during primary infection resulted in the failure to establish a characteristic latent viral transcription profile in sensory ganglia, where we detected 68 out of 69 SVV-encoded open reading frames (ORFs). CD-depleted RMs displayed a latent transcriptional profile that included additional viral transcripts within the core region of the genome not detected in control RMs. The latent transcriptome of CD20-depleted RMs was comparable to the latent transcription in the sensory ganglia of control RMs. Lastly, we investigated the impact of age on the establishment of SVV latency. SVV gene expression was more active in ganglia from two aged RMs than in ganglia from juvenile RMs, with 25 of 69 SVV transcripts detected. Therefore, immune fitness at the time of infection modulates the establishment and/or maintenance of SVV latency.