Simply put, the infancy narratives in the gospels are not historical accounts, nor were they meant to be read as such. They are theological affirmations of Jesus's status as the anointed of God. The descendant of King David. The promised messiah.