From the Super Mario Wiki
Super Mario Bros. 3
? Balls first appear in Super Mario Bros. 3. They only appear in fortresses after the player defeats Boom Boom. Upon his defeat, Boom Boom will drop a ? Ball, and Mario must touch it to destroy the fortress and therefore complete the level. If Invincible Mario defeats Boom Boom by touching him, the ? Ball will appear upside down, in the center of the arena. They later appeared in the game's remakes, playing the same role.
Super Mario World
Only one ? Ball appears in Super Mario World, and that is in the Sunken Ghost Ship (this is most likely to give this level the feel of Super Mario Bros. 3, as it is suggested to be one of the Airships from that game). Touching it beats the level in a similar fashion of beating a boss, but it is left out in the open instead of being held by a mini-boss.
In volume 4 of Super Mario-Kun, whose arc is based on Super Mario World, Mario grabs the ? Ball to exit the level, and he and his friends are later found on the top of the entrance to the Valley of Bowser.