We have been lucky enough to have stayed at both resorts; and on our last trip this summer, we returned to the Polynesian. We love the gardens and Southern Seas feeling about the resort as well as its convenience. Access to the Magic Kingdom was simple by boat or monorail. To get to Epcot, we walked over to the Ticket and Transportation Center and went directly to the Epcot monorail. Another feature of the resort that you may find valuable with a 4 year old is the zero-entry pool with its view of the castle. it is so magical to swim and watch the fireworks!
The Contemporary is also a wonderful resort that has recently been renovated (the Polynesian rooms were, too), but it is still undergoing some construction within it and nearby it. No completion date has been published, so I don't know if it will interfere with your trip in April or May of 2009. On our stay there, having the flexibility to walk to the Magic Kingdom gave us an incredible feeling of freedom since we could move at our own pace and avoid bus/monorail/boat crowds. While everything was fine with the resort, for us it was more a "nice" hotel than a "Disney" hotel. However, with the newer rooms, that may have changed. It also has the advantage of being within one building, which during the rain is invaluable. I remember spending an afternoon in our rooms and skipping lunch while at the Polynesian because none of us wanted to venture out into a heavy downpour to get to the main building.
I hoped that my pros and cons of our experiences at the resorts are helpful for your decision making. They are both great resorts and you'll be at WDW, so I don't think a wrong decision is an option!