Accuracy in the measurement of mechanical properties is essential for precision engineering and for the interrogation of composition-property relationships. Conventional methods of mechanical testing, such as uniaxial tension, compression, and nanoindentation, provide highly repeatable and reliable results for stiff materials, for which they were originally developed. However, when applied to the characterization of soft and biological materials, the same cannot be said, and the spread of reported properties of similar materials is vast. Polydimethylsiloxane (PDMS), commonly obtained from Dow as SYLGARD 184, is a ubiquitous such material, which has been integral to the rapid development of biocompatible microfluidic devices and flexible electronics in recent decades. However, reported shear moduli of this material range over 2 orders of magnitude for similar chemical compositions. Taking advantage of the increased mechanical scrutiny afforded to SYLGARD 184 in recent years, we combine both published and new experimental data obtained using 9 mechanical test methods. A statistical analysis then elucidates the significant bias induced by the test method itself, and distinguishes this bias from the influence of curing protocols on the mechanical properties. The goal of this work is thus two-fold: (i) it provides a quantitative understanding of the different factors that influence reported properties of this particular material, and (ii) it serves as a cautionary tale. As researchers in the field of mechanics strive to quantify the properties of increasingly complex soft and biological materials, converging on a standardized measurement of PDMS is a necessary first step.