Everything was going fine, until the generation of Enosh, the third generation from Adam. The Rambam (the 12th century Rabbi, Rabbi Moshe Ben Maimon, aka Maimonides) writes:
During the times of Enosh (the grandson of Adam), mankind made a great mistake, and the wise men of that generation gave thoughtless counsel. Enosh himself was one of those who erred.
Their mistake was as follows: They said God created stars and spheres with which to control the world. He placed them on high and treated them with honor, making them servants who minister before Him. Accordingly, it is fitting to praise and glorify them and to treat them with honor. [They perceived] this to be the will of God, blessed be He, that they magnify and honor those whom He magnified and honored, just as a king desires that the servants who stand before him be honored. Indeed, doing so is an expression of honor to the king.
After conceiving of this notion, they began to construct temples to the stars and offer sacrifices to them. They would praise and glorify them with words, and prostrate themselves before them, because by doing so, they would – according to their false conception – be fulfilling the will of God.
This was the essence of the worship of false gods, and this was the rationale of those who worshipped them. They would not say that there is no other god except for this star.
This message was conveyed by Jeremiah, who declared (10:7-8): "Who will not fear You, King of the nations, for to You it is fitting. Among all the wise men of the nations and in all their kingdoms, there is none like You. They have one foolish and senseless [notion. They conceive of their] empty teachings as wood;" i.e., all know that You alone are God. Their foolish error consists of conceiving of this emptiness as Your will.
After many years passed, there arose people – false prophets – who told [the people] that God had commanded them to say: Serve this star – or all the stars – sacrifice to it, offer libations to it, build a temple for it and make an image of it so that all people – including the women, the children, and the common people – could bow to it.
He would inform them of a form that he had conceived, and tell them that this is the image of the particular star, claiming that this was revealed to him in a prophetic vision. In this manner, the people began to make images in temples, under trees, and on the tops of mountains and hills.
People would gather together and bow down to them and the [false prophets] would say: This image is the source of benefit or harm. It is appropriate to serve it and fear it. Their priests would tell them: This service will enable you to multiply and be successful. Do this and this, or do not do this or this.
Subsequently, other deceivers arose and declared that a specific star, sphere, or angel had spoken to them and commanded them: Serve me in this manner. He would then relate a mode of service [telling them:] Do this, do not do this.
Thus, these practices spread throughout the world. People would serve images with strange practices – one more distorted than the other – offer sacrifices to them, and bow down to them. As the years passed, [God's] glorious and awesome name was forgotten by the entire population. [It was no longer part of] their speech or thought, and they no longer knew Him. Thus, all the common people, the women, and the children would know only the image of wood or stone and the temples of stone to which they were trained from their childhood to bow down and serve, and in whose name they swore.
The wise men among them would think that there is no God other than the stars and spheres for whose sake, and in resemblance of which, they had made these images. The Eternal Rock was not recognized or known by anyone in the world, with the exception of a [few] individuals…."
(Mishneh Torah, Hilkhoth Avodath Kokhavim 1:1-2)
In this way, the world was led astray into Idolatry. Within ten generations from the first humans, they had entirely forgotten God and the Torah which He had given them.