On 12/13/24, I was working on deployment images for Grouper 4.16.0.  I followed through the upgrade instructions v4 Upgrade instructions from v4 - Grouper - Internet2 Wiki to aid in this process.

I deployed the new images and after it came up, I performed a full sync.  During this full sync, Grouper started removing all grouper memberships in our AD for groups that were not directly managed by Grouper.

There were no changes to the provisioner that would have made Grouper believe it was authoritative for membership in all groups.  In fact, there were no changes made to the AD provisioner at all.  We only manage a very small number of groups in our AD via Grouper so this behavior was entirely unexpected.

It was a process, but we were able to restore Active Directory and bring Grouper back online to see if we could replicate the problem to determine why this happened.  We have been unable to do so.

We've put in place a lot of mitigations to narrow the scope of damage should something like this happen again but we'd really like to understand how it did.