Hi there Kelly! Thanks for the question!
Personally, there's been times where we have just walked up to the show right when it was about to start and was able to find good seats but, there have also been times where we haven't even been able to enter. It's really dependent on if you want a premium seat or if you just want to experience the show. If you arrive at the park and it's very busy, then I would definitely wait in line at least 30 minutes. Finding two seats is usually easier than if you have a large group. The theater is bench seating and people will typically scoot over for you and a baby. Do remember that there are times when your baby might be startled by fireworks or cannon fire.
Another options to avoid a long wait is the Fantasmic! Dining Package.
The
Fantasmic! Dining Package gives you an unforgettable dining and entertainment experience. With the convenience of choosing lunch or dinner at The Hollywood Brown Derby or Mama Melrose's Ristorante Italiano, or dinner at Hollywood & Vine in Disney's Hollywood Studios theme park, you'll also enjoy designated seating for the Fantasmic! fireworks show. With 3 restaurants to choose from, you can decide between buffet service, casual dining or fine dining. Good to know tip: When there are two performances of Fantasmic! on the same night, the package is valid for the first performance only. You can also make an online reservation or you can call (407) WDW-DINE.
Be sure to check the Times Guide at Disney's Hollywood Studios theme park for schedule and show times. Since Fantasmic! is an outdoor show, it is also subject to cancellation due to inclement weather. I hope you and your baby have a Magical Time at the Walt Disney World Resort!