Hi Hazel!
Taking advantage of the Extra Magic Hours is definitely worth it!
At least one Theme Park every day normally offers Extra Magic Hours. Extra Magic Hours is generally where a theme park will either open early (generally an hour early) or stay open late (anywhere from 1-3 hours late, depending). If you walk in, say, at 8am, the Disney cast members are already available to make your time at the park magical!
The coolest thing is the Extra Magic Hours are for resort guests, so the crowds are generally smaller, allowing your 4 year old quicker (and more frequent) access to his or her favorite rides.
It was also nice being able to stick around a little later some nights, when you weren't ready for the fun to end.
I brought my 4-year old and they stuck it out to the end of Extra Hours. They'll quickly fall asleep, though, on the bus ride back to the resort (smile).
When you check in, they should provide you a packet with the theme park schedule, which includes the Extra Magic Hours.
Best wishes for a magical time!