There are several stages in a puppy's and young dogs life:
0-2 weeks old: Vegetative stage
3 weeks old: Transision stage
The next stage is the one that is relevant for you right now:
4-7 weeks: Impression stage:
This is the stage where the puppy learns important lessons and behaviours that will be firmly imprinted in his mind. A bad experience at this age, can lead to life-long problems.
At this stage a puppy should still be with his mother and siblings in a secure environment. He will take his cues from his mother who will teach him important lessons such as humans being nice and members of the pack. She will also teach him manners and because she is the one whom he will trust more than anyone else, her corrections will not frighten him unnecessarily. Playing with his littermates will start the bite-inhibition training and help him develop a sense of himself, through dominance play and play-fighting. These are not interactions that we humans can fully replace.
From a health point of view, the longer the mother will nurse the puppies, the better, as they get some immunization and healthy bacterias through the mother's milk.
The next stage of a puppy life is equally important:
8-12 weeks: Socialisation.
This is the point where, in the wild, the pup's father will start taking over raising him, playing hunting games and romping roughly with him. This is also the stage where the pup will start to look to the pack-leader and bond with him/her, which makes 8-12 weeks the ideal age for a puppy to go to his new home, as it will follow the puppy's natural development. The socialisation stage is also, as the name implies, the stage where the puppy needs as many new positive experiences with the world as possible. Now is the time to teach him to get along with small pets, ride a car, be in a crowd etc.
Removing a puppy from its mother at 4 weeks and taking it to a new home at 7 weeks, will not automatically result in a difficult, sickly pup, but the risk is higher. The way I see it is, why take the chance? I'm sure there are shelters that leave mom with the puppies and doesn't give them away before they are 8 weeks old.