From the Super Mario Wiki
Ads keep the MarioWiki independent and free :)
The Endless Stairs (stylized in game as the "endless" stairs) is an obstacle within Princess Peach's Castle first appearing in Super Mario 64, and later reappearing in Super Mario 64 DS. It is behind the final Big Star Door, which leads to Bowser in the Sky. If the player has less than 70 Power Stars in Super Mario 64 and 80 Power Stars or don't have Mario in Super Mario 64 DS, they can still enter the Big Star Door, but they will endlessly climb this flight of stairs. If the player turns back after climbing the stairs for some time, it's like they never went anywhere. Once the player manages to collect 70 Power Stars (80 and Mario in the Nintendo DS remake), the Endless Stairs finally ends.
The "endless" stairs illusion is created by warping Mario back once he reaches a certain point. This can be easily seen in the DS remake via the minimap, as well as the fact that the ceiling at the top of the stairs is actually slightly visible in this version.
The audio track that plays seems to be based off of the Shepard tone audio illusion.