From the Super Mario Wiki
The endless stairs (also stylized as the "endless" stairs) are an obstacle within the Mushroom Castle, first appearing in Super Mario 64 and later reappearing in Super Mario 64 DS. They are behind the final Big Star Door, which leads to Bowser in the Sky. If the player has fewer than 70 Power Stars in Super Mario 64 or has 80 Power Stars and does not have Mario in Super Mario 64 DS, they can still enter the Big Star Door, but they will endlessly climb this flight of stairs, seemingly gaining no ground, though they can easily exit through the entrance. Once the player manages to collect 70 Power Stars (80 and Mario in the Nintendo DS remake), the endless stairs finally end, and Mario will be able to access the final level of the game when he reaches the top.
The "endless" stairs illusion is created by warping Mario back once he reaches a certain point. This can be easily seen in the DS remake via the minimap, as well as the fact that the ceiling at the top of the stairs is actually slightly visible in this version.
The audio track that plays seems to be based on the Shepard tone audio illusion.