The rules vary a bit, depending on exactly which Annual Passholder benefit you're talking about, but in general only one member of your party needs to be a Passholder in order for your entire group to receive discounts. For example, a Passholder may reserve a discounted room for an entire group if only he or she has an annual pass.
In my family of five, I am the only person with an annual pass. With my pass, I purchased the Tables in Wonderland dining discount card. This allows me to get 20% off on table service dining at WDW for my entire group (up to 10 people). Similarly, when my family took the Behind the Seeds tour, I was able to get the Passholder discount rate for the group, even though I was the only Passholder. For reasons such as these, it can often make sense for at least one member of a party to become a Passholder, just for the possible discounts. But of course, you really need to do the math for your own unique situation.
You can find out more information about Passholder discounts on the WDW website
HERE. For even more detail, I encourage you to call the Passholder office at 407-560-PASS. They can help you determine which Passholder benefits would most help your family.