When you edit the group list, the system remove the old group and create a new one with the new list, removing and adding users. See macro save_users_and_group.
My suggestion is: Only add new users of the edit list, so that, if you want to remove someone you would have to go to his profile and do it.
What do you think?