Assuming you aren't visiting Memorial Day Weekend, the crowds shouldn't be too much of a problem so you'll be able to get a lot accomplished, even still having a chance to enjoy some of the kiddie rides. So, if you have the energy and determination, I'd go for it and try to spend the entire day making sure to stay through the evening parade and fireworks (soooo very romantic to watch together). Perhaps you'd even consider taking a break and going over to one of the Magic Kingdom resorts for lovely adult dinner, perhaps the
California Grill,
Citricos, or
Artist Point. Or you could even just grab something from Captain Cook's counter service restaurant at the Polynesian and have an evening picnic on the beach with the fireworks in the background.
If you do want to get the most "bang for buck" while at the Magic Kingdom, you may want to pick up a copy of The Unofficial Guide to Walt Disney World. It is one of the most comprehensive guide books, and it contains tours for specific audiences, such as one day for adults at the Magic Kingdom.
So how much time to allow yourself, I'd personally opt to spend the entire day there because I think you will find plenty to do. With traveling without children, you'll have a chance to explore and soak in all the Disney Magic more. But then you will be able to return with your children - truly the best of both worlds!