Diarrhea is a leading cause of death among children aged less than five years globally. Most studies of pediatric diarrhea rely on caregiver-reported stool consistency and frequency to define the disease. Research on the validity of caregiver-reported diarrhea is sparse. We collected stool samples from 2,398 children participating in two clinical trials in the Amhara region of Ethiopia. The consistency of each stool sample was graded by the child's caregiver and two trained laboratory technicians according to an illustrated stool consistency scale. We assessed the reliability of graded stool consistency among the technicians, and then compared the caregiver's grade with the technician's grade. We also tested if the illustrated stool consistency scale could improve the validity of caregiver's report. The weighted kappa measuring the agreement between the two laboratory technicians reached 0.90 after 500 stool samples were graded. The sensitivity of caregiver-reported loose or watery stool was 15.5% (95% confidence interval [CI]: 9.7, 24.2) and the specificity was 98.4% (95% CI 97.1, 99.1). With the illustrated scale, the sensitivity was 68.5% (95% CI: 58.5, 77.1) and the specificity was 86.1% (95% CI: 79.3, 90.9). The results indicate that caregiver-reported stool consistency using the terms "loose or watery" does not accurately describe stool consistency as graded by trained laboratory technicians. Given the predominance of using caregiver-reported stool consistency to define diarrheal disease, the low sensitivity identified in this study suggests that the burden of diarrheal disease may be underestimated and intervention effects could be biased. The illustrated scale is a potential low-lost tool to improve the validity of caregiver-reported stool consistency.