Neural Network Models of Morphological Inflection (NNMIs) have deep relevance to cognitive science stemming from the central role that they played in the Past Tense Debate of the 1980s and 1990s. Critics of the connectionist approach to the mind frequently pointed to NNMIs’ shortcomings in the area of developmental realism: they argued that regardless of their ultimate accuracy, they failed to capture patterns of child language acquisition including developmental regressions and a propensity for over-regularization rather than irregularization. However, NNMIs have seen impressive improvement in the deep learning era of the 2010s and 2020s. Have modern NNMIs solved the old problems of developmental realism? We find that they have not. The persistence of these shortcomings suggests that they reflect “innate” characteristics of NNMIs as a class of learner, and that even substantial advancement in neural architectures and subsequent performance increases do not necessarily entail increased cognitive plausibility.