This is consistent with my experience as a parent and educator: my guess is that number sense is a cognitive module, at least somewhat distinct from general intelligence, and somewhat hardwired.
Number sense in infancy predicts mathematical abilities in childhood (PNAS)
Abstract: Human infants in the first year of life possess an intuitive sense of number. This preverbal number sense may serve as a developmental building block for the uniquely human capacity for mathematics. In support of this idea, several studies have demonstrated that nonverbal number sense is correlated with mathematical abilities in children and adults. However, there has been no direct evidence that infant numerical abilities are related to mathematical abilities later in childhood. Here, we provide evidence that preverbal number sense in infancy predicts mathematical abilities in preschool-aged children. Numerical preference scores at 6 months of age correlated with both standardized math test scores and nonsymbolic number comparison scores at 3.5 years of age, suggesting that preverbal number sense facilitates the acquisition of numerical symbols and mathematical abilities. This relationship held even after controlling for general intelligence, indicating that preverbal number sense imparts a unique contribution to mathematical ability. These results validate the many prior studies purporting to show number sense in infancy and support the hypothesis that mathematics is built upon an intuitive sense of number that predates language.