Primary cutaneous squamous cell carcinoma (cSCC) is responsible for ~10,000 deaths annually in the United States. Stratification of risk of poor outcome (PO) including recurrence, metastasis and disease specific death (DSD) at initial biopsy would significantly impact clinical decision-making during the initial post operative period where intervention has been shown to be most effective. In this multi-institutional study, we developed a state-of-the-art self-supervised deep-learning approach with interpretability power and demonstrated its ability to predict poor outcomes of cSCCs at the time of initial biopsy. By highlighting histomorphological phenotypes, our approach demonstrates that poor differentiation and deep invasion correlate with poor prognosis. Our approach is particularly efficient at defining poor outcome risk in Brigham and Women's Hospital (BWH) T2a and American Joint Committee on Cancer (AJCC) T2 cSCCs. This bridges a significant gap in our ability to assess risk among T2a/T2 cSCCs and may be useful in defining patients at highest risk of poor outcome at the time of diagnosis. Early identification of highest-risk patients could signal implementation of more stringent surveillance, rigorous diagnostic work up and identify patients who might best respond to early postoperative adjunctive treatment.