Over the past decade, growing evidence indicates that the tumor microenvironment (TME) contributes with genomic/epigenomic aberrations of malignant cells to enhance cancer cells survival, invasion, and dissemination. Many factors, produced or de novo synthesized by immune, stromal, or malignant cells, acting in a paracrine and autocrine fashion, remodel TME and the adaptive immune response culminating in metastasis. Taking into account the recent accomplishments in the field of immune oncology and using metastatic colorectal cancer (mCRC) as a model, we propose that the evasion of the immune surveillance and metastatic spread can be achieved through a number of mechanisms that include (a) intrinsic plasticity and adaptability of immune and malignant cells to paracrine and autocrine stimuli or genotoxic stresses; (b) alteration of positional schemes of myeloid-lineage cells, produced by factors controlling the balance between tumour-suppressing and tumour-promoting activities; (c) acquisition by cancer cells of aberrant immune-phenotypic traits (NT5E/CD73, CD68, and CD163) that enhance the interactions among TME components through the production of immune-suppressive mediators. These properties may represent the driving force of metastatic progression and thus clinically exploitable for cancer prevention and therapy. In this review we summarize results and suggest new hypotheses that favour the growing impact of tumor-infiltrating immune cells on tumour progression, metastasis, and therapy resistance.