So the last weekend was dedicated to IDM solution upgrade procedure – and it all went OK! After several months of continuous integration of new target systems and expressions and after weeks of integration and acceptance testing we got a green light and the upgrade was started. How does such upgrade look like? Is it a complicated? Is it a reinventing of a wheel?
The main reason to upgrade was not the new version of midPoint (3.0 vs. 3.0.1), but new target systems, new connectors, and new rules. The midPoint was deployed several months ago with the 3.0 version, with limited number of managed identities and only one target system (Microsoft Active Directory – AD) and multiple roles based on AD groups. Now it was the time to start provisioning to Microsoft Exchange 2013 instead of plain AD and to several database applications as well as prepare the system for the immediately following integration of web-service applications. The number of roles was increased ten times – this is a multi-tenant installation, so the number depends on the new systems added, not tenants. And, some changes were quite incompatible, so the data migration was expected. For example, some roles were previously assigned without “tenantRef” parameter, because we expected the parameter to be computed by other means. This was changed during the summer, so all existing role assignments had to be updated to contain the “tenantRef” parameter, which was derived from other user attributes.
So this upgrade was an evolution, not revolution.
Situation before upgrade:
- midPoint 3.0
- Active Directory as target system: accounts and tenants (OUs, groups) provisioned by midPoint
- several roles managing Active Directory group membership for different types of users
- Microsoft Exchange mailboxes and attributes provisioned manually/by scripts
Expected situation after upgrade:
- midPoint 3.0.1
- Microsoft Exchange as target system: accounts, tenants (OUs, groups, various Exchange objects) and mailboxes provisioned by midPoint
- multiple database applications as new target systems
- new custom connectors deployed to allow more target systems to be connected ASAP
- many new roles managing Active Directory group membership and other entitlements in other systems
The upgrade and migration procedure for our multi-tenant customer basically consisted of:
- preparation and planning (designing the upgrade, data migration and smooth transition, testing the upgrade procedures, documentation): 32 man-hours
- midPoint upgrade (backup, redeploying new version, connectors, restarts, documentation update): 8 man-hours
- reconciliations, data migration, new rule applications, documentation update: 37 man-hours
As you see, the planning and testing of the upgrade procedure took roughly the same time as the upgrade itself. The planning was especially important, because the upgrade had to take place during this specific weekend and on the next day, the provisioning was expected to be up and running. There will be some other upgrades outside of midPoint, which will depend on information provisioned by midPoint, so we really had not much choice.
Preparation and planning
In this phase, we documented intended changes in a text document, with exact steps that have to be executed in order of successful upgrade. Then we tested the upgrade in other environment, and fixed the upgrade procedures according to the reality. This was a real challenge, because there were only small testing “windows” during the ongoing acceptance tests in that environment. But at the end, the procedure was working. At least for the test data.
midPoint upgrade
This phase was the simplest and shortest one. First, upgrading midPoint is easy for supported versions, second, the upgrade didn’t require change in the database schema and was 100% data compatible, so this part was mainly about redeploying midPoint in multiple cluster nodes and then the connectors. At the end of the first day, midPoint and connectors were upgraded on all cluster nodes, but the behaviour of the solution was still the same as before. The expressions was not updated at this time. We have just confirmed that the systems works; we would be able to provision new users if necessary, but with the old business logic of course. This would be the “last resort” in case of upgrade problems (which wasn’t the case).
Reconciliations, data migration and new rules
This phase was the real fun. We had to upgrade most (but not all) rules in midPoint. Object templates, most of the roles, resource configuration for new target systems which we needed to be assigned during this task by default user template. We started to reconcile with the Microsoft Exchange with “almost read-only configuration of resource” to get the e-mail address provisioned manually up to this upgrade and, at the same time, update some AD attributes and provision new target systems according to the new rules. No other attributes and especially passwords were allowed to be changed though (which would mean a serious problem for the existing users). The existing data was merged with midPoint, because the Microsoft Exchange mailboxes and attributes were created manually/by scripts even if the accounts were provisioned by midPoint previously.
During the reconciliation, various data inconsistencies were corrected (as the accounts provisioned manually are seldom 100% correct) or reported (such as accounts with restricted permissions to update). After this task, the accounts in AD/Exchange were reconciled and fixed. Still, the same roles were assigned to the users to avoid problems during the reconciliations.
The last task was the assignment migration. We imported the new versions of the roles that were updated (with new subroles), switched the resource to read-write and using Bulk actions with XSLT fixes we added missing tenantRef parameter where it was required. At the same time, new accounts were created (because of the new subroles) on the new target systems.
During the procedure, the upgrade documentation was constantly updated and then presented to the customer with all the detected problems and fixes.
From the point of view of the end users, they didn’t even notice there was an upgrade. No passwords were changed, no mailbox deleted, no group membership influenced. No revolution, but evolution.
With midPoint, everything is possible.