Hi Karen!
Thanks for your question. If I had to choose between the 2 weeks in which you've mentioned, I would probably go with the week of March 16th over the week of May 23rd. Although there really is no true slow period at Disney anymore, the month of March typically brings smaller crowds than the month of May. In addition, the week of May that you would be considering is near Memorial Day weekend, a peak time for Disney guests to visit the parks. March weather in Orlando typically averages mid to upper 70's, whereas the month of May is mid to upper 80's. March wins in the "milder" category :-)
In March, there is also the
Epcot International Flower and Garden Festival, a fantastic display of Disney characters created in various forms of plants, flowers, and other type greenery.
Smaller crowds and milder weather equal a friendlier experience for your young one's :-) Outside of Christmas trips, our family (children ages 8 and 5) typically travel during off-season periods for those very reasons.
I think March would be a good selection, and not one you'd need to put off for another year.
May you and family have a magical, fun-filled time at Disney!