The initial design is that Schedule.Contact contains all contacts in the schedule, while Calendar.ItemContacts contains a subset of contacts selected to group by. This way for example, you can have ten contacts in total but pick only three of them to group by. Therefore, currently adding contacts to Calendar.ItemContacts automatically adds them to Schedule.Contacts collection if they are not found there.

Regards,
Meppy