Esophageal squamous cell carcinoma (ESCC) remains one of the most common malignancies in China and has a high metastasis rate and poor prognosis. Cancer-associated fibroblasts (CAFs), a prominent component of the tumor microenvironment, can affect tumor progression and metastasis, but the underlying mechanism remains unclear. There are no studies that explore the role of hydrogen peroxide-inducible clone 5 (HIC-5) in ESCC or compare the role of HIC-5 in CAFs and adjacent noncancerous normal fibroblasts (NFs). In this study, we isolated primary CAFs and NFs from ESCC patients. HIC-5 was highly expressed in CAFs from the tumor stroma of human ESCC patients. HIC-5 knockdown in CAFs inhibited the migration and invasion of ESCC cells in vitro. Supernatant CCL2 levels of CAFs were significantly higher after TGF-β stimulation and lower after knocking down HIC-5 expression, independent of TGF-β treatment. HIC-5 knockdown in CAFs led xenograft tumors derived from ESCC cells mixed with CAFs to present more regular morphology, express higher CDH1, and lower CCL2. Further RNA-seq data showed that HIC-5 has distinct biological functions in CAFs vs. NFs, especially in cell movement and the Rho GTPase signaling kinase pathway, which was verified by wound-healing assays and western blotting. An ESCC tissue microarray revealed that increased HIC-5 expression in the tumor stroma was associated with positive lymph node metastasis and a higher TNM stage. In summary, we identified that stromal HIC-5 was a predictive risk factor for lymph node metastasis in human ESCC and that CAF-derived HIC-5 regulated ESCC cell migration and invasion by regulating cytokines and modifying the ECM.