According to Harvard University's director of the Child Development Unit, babies get upset when their mothers have a "still face." It's probably the same with dads, siblings, or any person the baby is familiar with.
Using the "Still Face" Experiment, in which a mother denies her baby attention for a short period of time, Tronick describes how prolonged lack of attention can move an infant from good socialization, to periods of bad but repairable socialization. In "ugly" situations the child does not receive any chance to return to the good, and may become stuck.
