Ads keep the MarioWiki independent and free :)
An unwinnable state (also called a softlock) is when a level or a whole game is still technically playable but is impossible to complete because either a technique or an object needed to progress has become inaccessible to the player, the player character is trapped with no means of escape, or a script fails to trigger. Unwinnable situations can be the result of a glitch, the result of players going out of their way to trap themselves, or intentional on the part of the game designers, although due to the Mario franchise's accessible, player-friendly design, instances of intentional unwinnable scenarios are non-existent.
Note that this does not include scripted boss battles the player has to lose in Mario RPGs such as the first battles with Bowser and the Shroobs in Paper Mario and Mario & Luigi: Partners in Time respectively, even if they are literally "unwinnable."
Examples of unwinnable situations
The Minus World in most versions of the game (the Family Computer Disk System version excepted) is impossible to complete due to the pipe at the end of the level being glitched and taking the player back to the start.
If the player digs up all the sand at the left side of the splitting path in World 2-2, they will be unable to jump out.
To defeat Bowser in Bowser in the Dark World, Bowser in the Fire Sea, and Bowser in the Sky, Mario must throw him into the bombs around the arena. If Mario Long Jumps into all the bombs to detonate them, he will be left with no way to defeat Bowser unless he leaves the level.
In the Gusty Garden's Gravity Scramble mission in Gusty Garden Galaxy, it is possible to fall back onto the final Disk Planet with the Launch Star by performing a Long Jump in the right place. However, because the gravity has now changed on the ! and ? Planets, the player will be forced to restart the mission.
In The Chimp's Coin Challenge in Slimy Spring Galaxy, after winning the challenge the player is transported to the beginning of the galaxy where the Power Star is obtained. However, if the player goes down the waterslide, they will not be able to collect the Power Star and therefore must restart the mission.
Due to oversights or glitches, there are two examples of unwinnable situations, both in World 4. In World 4-2, a certain pipe lacks an assigned destination, meaning that if it is entered, Mario will never exit, and thus be trapped until the timer runs out. Similarly, in World 4-3, a Coin Heaven area's pipe cannot be entered, trapping Mario in that area until the timer runs out.
GO! GO! MARIO!! has a Super Star that Yoshi can reach by jumping off a Shy-Guy. However, unless he has any Yoshi Eggs or watermelons, he will be unable to get out of the area and must quit out of the level.
Paper Mario series
The Kamek curse battles may result in situations that make the battle impossible to win (for example, forcing the player to only use Hammer cards against a flying enemy). The only solution is to exit the game (as the Kamek curse battles disable the Flee command) or waste one’s entire supply of cards.
Luigi's Mansion series
In Treacherous Mansion, it's possible to open a door leading to the pirate ship on the side of the mansion by holding a nearby switch down with an object. If the player enters this area while a ghost knocks away the object, the door will become blocked again after they enter the room, leaving them stuck on a tiny platform and forced to restart the level.
It's also possible to be unable to win a multiplayer ScareScraper round thanks to a common glitch that stops a necessary key from appearing in the level. If this happens, the players are forced to lose the game.
Mario & Luigi series
Thanks to a certain series of glitches in Mushrise Park, it's possible to fall into the Mini Mario area as both Mario and Luigi, or get stuck in the collision for a nearby bridge. If this happens, the game is unwinnable and the file has to be started over. 
Additionally, it used to be possible to make the game unwinnable by backtracking as solo Luigi after Mario got captured, since a glitch would result in him getting stuck in a certain area of Dimble Wood and being unable to leave. However, after the first update for the game, this was patched, and any affected save files were also fixed to allow progress.
Impossible 100% Completion
A less severe variation is when a game can still be fully completed but denies the player 100% completion of their save file in some ways. Unlike true unwinnable scenarios, examples of intentional impossible 100% completion are more common in the Mario franchise.
If the player loses eight lives in a course from one of the first eight worlds before completing it, a Super Guide Block appears when the player re-enters that course, permanently preventing the player from ever having shiny stars on their save file. The only way to earn shiny stars is to never see a Super Guide Block in the first place.
Similar to New Super Mario Bros. Wii, if the player loses five lives when attempting a course from one of the first eight worlds, an Assist Block will appear on re-entry, permanently preventing the player from ever having shiny stars on their save file.
In all subsequent Mario platformers, making the Super Guide Block or Assist Block appear will not void the player's chance to get shiny stars on their save file; any player who uses the Super Guide or Invincibility Leaf can get shiny stars by completing the same level without using it.