From the Super Mario Wiki
The object is to catch gems such as rubies, emeralds, sapphires, and diamonds. Diamonds are special because the player haves to catch them by using a home run buddy catch, unlike the other gems. Although simple in concept, there is a catch: items are being thrown at the player while trying to catch the gem, and if the player get hit, he or she will fail the catch.
This minigame only takes place in daytime, not at night.