Hi Katy! I definitely don't think 4 years old is too soon, my little guys have been coming since they were infants and we've always had a great time with them. There may be some attractions you aren't able to take them on but you can also take advantage of the "Rider Swap" where you can wait in line with your husband and when you get up to the loading area, one of you will wait with the 4-year old and then switch places when the first rider returns. You can check out information on this and more planning for little ones at
DisneyWorld's site.
As far as time of year to go, right now through mid-February is a great time but you might have cold snaps or other issues with weather. Summer time will be the busiest but starting mid-September through early December (with exceptions for the holidays) the crowds are their lowest and the weather is usually great.
If you are staying in a Disney resort, you really don't have to drive anywhere during your entire stay!
Disney's transportation takes care of everything for you. There will always be a bus, boat, or monorail ready to bring you to whichever park you'd like to visit that day. There are different resorts situated near each park, so you can choose which one will work best for your family. One of my favorites is Disney's Beach Club Resort because we love Epcot and it's within walking distance.
I hope this helps with your planning, if you have any other questions about specific resorts or anything else, please come back and visit us!
~Suzannah