On the infant weight loss of low- to intermediate-mass star clusters