Hi, Andrea!
We love afternoons at Downtown Disney, so I'm all for your plans.
Whether or not it's too busy for your little ones will depend on two things: do they require proper naps to be happy and will they each be in a stroller?
When my two kids were those ages, each one would fall soundly asleep in a stroller so we would plan a day such as this one for the day after a later night. After a light breakfast, we'd head to DD. A relaxing lunch and exploring the World of Disney Store, the LEGO Imagination Center, Once Upon a Toy and Goofy's Candy Company, to name a few favorites, would occupy several hours at which point we'd head over to a park of choice. The bus ride itself will be about 10 minutes.
To get to Epcot, take a Resort Bus from DD to
Disney's Beach Club Resort, which is the next door neighbor to Epcot, using the "back door" entrance, International Gateway. If you take a look at the map embedded in the link above, you'll see a large circular driveway to the North side of the resort; Resort Buses will drop you off near that spot, and you can walk directly up to the lobby, walk right across to the doors on the opposite side (lakeside) and out to the walkway which runs along Crescent Lake. Turn left and you'll be at the less-crowded back entrance to Epcot in just a few minutes. (You'll be entering in the World Showcase, right between United Kingdom and France, which is a perfect area to stop for dinner!)
Once you are finished at Epcot, head out the main exit to pick up transportation to your resort.
If you have a car, on the other hand, then your drive will be about 10 minutes from one parking lot to the other. Also so convenient, right?
Send more questions as you think of them, Andrea; we love to help!