From the Super Mario Wiki
Picture Poker is a single-player minigame in Super Mario 64 DS and New Super Mario Bros. In the former, it is classified as one of Luigi's minigames; in the latter, it is classified as a Table minigame.
It is a casino-style card game in which the player automatically bets coins that his or her five-card hand will beat Luigi's. After being dealt their hand, the player can bet more coins and/or have some or all of their cards replaced with random cards from the deck. Luigi will then, if need be, replace cards in his hand (which appear face down to the player) with random cards from the deck. The hands are then shown. If the player's hand is better, they win a set amount of coins based on both the rank of his or her hand and the amount of coins bet. If not, the player simply loses the coins bet. The player is awarded Stars with each victory and is deducted Stars (if they have any) with each loss. The minimum bet is determined by the amount of Stars in the player's possession (The bet increases by one for every five Stars). If the player runs out of coins, the game is lost and must be restarted.
If all opponents have the same hand, then these ranks determine who wins, in order from worst to best: Cloud, Super Mushroom, Fire Flower, Luigi, Mario, Star.
The player can choose between 3, 5, or 7 rounds. Which player goes first is chosen randomly. There isn't any betting; if the player wins, the hand they have determines how many coins they get. (Pair - 2 coins, Two Pair - 3 coins, etc.) All of the other players have to pay that many.
Get a better hand than Luigi to win! If you're confident that your hand's the best, increase the number of coins you bet!