Although there is no requirement that one leave the synagogue if he or she is not required to recite Yizkor, many people believe it would temp fate to be present.  Others feel it might be distracting to those who are required to recite Yizkor.  Those reciting Yizkor might feel envious or uncomfortable to see worshippers sit in silence while they stand, express grief, and perhaps shed a tear.