This study analyses the changes of moderate-to-vigorous physical activity (MVPA) in a cohort of boys and girls aged 11 (n = 50) and 14 (n = 50). Physical activity was assessed with Bodymedia SenseWear Pro Armband monitor for 6 days in October 2013 and October 2016, considering 90% of daily wear time (21h and 40min). The initial sample (n = 160) included the children who wore the monitors at age 11 but the final analyzed sample included only those children from the initial sample (n = 50), whose data fulfilled the inclusion criteria at age 11 and 14. Physical fitness and somatic characteristics of the final sample (n = 50) were compared to a representative sample of Slovenian schoolchildren at ages 11 (n = 385) and 14 (n = 236) to detect possible bias. Changes in MVPA were controlled for maturity using the timing of adolescent growth spurt as its indicator. The average MVPA decreased more than one quarter (34.96 min) from age 11 to age 14. Children were significantly more active at age 11 than at age 14 (p < 0.01, d = 0.39). The timing of puberty onset in girls was significantly earlier (12.01 ± 1.0 years) (p < 0.01) than in boys (13.2 ± 0.75 years) (p < 0.01, d = 1.35). There was a significant gender difference in moderate-to vigorous physical activity at age 14 (p < 0.05, η2 = 0.12) and between moderate-to vigorous physical activity at age 11 and 14 (η2 = 0.11). After controlling for the timing of adolescent growth spurt the girls at age 11 showed significantly higher level of physical activity than at age 14 (p < 0.01, η2 = 0.17). Early adolescence is crucial for the development of physical activity behaviours, which is especially pronounced in girls. The significant decline of MVPA between ages 11 and 14 in Slovenia are likely influenced by environmental changes since the timing of adolescent growth spurt did not prove as a factor underlying the decline of MVPA.