The presence of tumor-infiltrating lymphocytes (TIL), reflecting host immune activity, is frequently correlated with better clinical outcomes, particularly in HER2-positive and triple-negative breast cancer. Recent findings suggest that organization of immune infiltrates in tertiary lymphoid structures also has a beneficial effect on survival. This study investigated inter- and intra-observer variation in TIL assessment using conventional hematoxylin-eosin versus immunohistochemical staining to identify immune cells. Global, intratumoral, and stromal TIL, as well as tertiary lymphoid structures were scored independently by experienced pathologists on full-face tumor sections (n=124). The fidelity of scoring infiltrates in core biopsies compared to surgical specimens, and pathological assessment compared to quantitative digital analysis was also evaluated. The inter-observer concordance correlation coefficient was 0.80 for global, 0.72 for intratumoral, and 0.71 for stromal TIL, while the intra-observer concordance correlation coefficient was 0.90 for global, 0.77 for intratumoral, and 0.89 for stromal TIL using immunohistochemical stains. Correlations were lower with hematoxylin-eosin stains, particularly for intratumoral TIL, while global scores had the highest concordance correlation coefficients. Our study concluded that tertiary lymphoid structures are accurately and consistently scored using immunohistochemical but not hematoxylin-eosin stains. A strong association was observed between TIL in core biopsies and surgical samples (R2=0.74) but this did not extend to tertiary lymphoid structures (R2=0.26). TIL scored by pathologists and digital analysis were correlated but our analysis reveals a constant bias between these methods. These data challenge current criteria for TIL and tertiary lymphoid structure assessment in breast cancer and recommend that how pathologists evaluate immune infiltrates be reexamined for future studies.