Fixing Data with Bulk Actions and XSLT

Posted on by Pavol Mederly

An IDM deployment is not always a straightforward process. It perhaps bears a similarity to the development of a software product, which is far from a linear process as well. So, from time to time, it is necessary to modify or repair the data already present in midPoint system.

For simple actions, like setting a property on a set of objects to a constant value, bulk actions can be used. For example, changing all values of users’ costCenter property from CC001 to CC-001 could be done as follows:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
<s:search xmlns:s="http://midpoint.evolveum.com/xml/ns/public/model/scripting-3"
          xmlns:c="http://midpoint.evolveum.com/xml/ns/public/common/common-3"
          xmlns:t="http://prism.evolveum.com/xml/ns/public/types-3"
          xmlns:q="http://prism.evolveum.com/xml/ns/public/query-3"
          xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <s:type>c:UserType</s:type>
    <s:searchFilter>
        <q:equal>
            <q:path>c:costCenter</q:path>
            <q:value>CC001</q:value>
        </q:equal>
    </s:searchFilter>
    <s:action>
        <s:type>modify</s:type>
        <s:parameter>
            <s:name>delta</s:name>
            <c:value xsi:type="t:ObjectDeltaType">
                <t:itemDelta>
                     <t:modificationType>replace</t:modificationType>
                     <t:path>c:costCenter</t:path>
                     <t:value>CC-001</t:value>
                </t:itemDelta>
            </c:value>
        </s:parameter>
    </s:action>
</s:search>

A bit of description: the script consists of a single command, namely the s:search one. The s:type and s:searchFilter parts say which objects are to be touched (users with costCenter = CC001). The s:action part says what should be done with them: changing costCenter to CC-001.

However, what if we want to do some more complex operation? Like changing CCxxx to CC-xxx for all values of “xxx”? Or something even harder?

Recently, a colleague of mine asked me if it would be possible to change all assignments of a given role to all users with a given employeeType (“ABCD”). We had to add tenantRef property pointing to the organization the user is assigned to. So from something like this:

1
2
3
<assignment id="13">
    <targetRef oid="12345678-0000-0000-0000-000000000000" type="RoleType"/>
</assignment>

we had to create something like this:

1
2
3
4
<assignment id="13">
    <targetRef oid="12345678-0000-0000-0000-000000000000" type="RoleType"/>
    <tenantRef oid="87654321-9999-9999-9999-999999999999" type="OrgType"/>
</assignment>

…given there was an assignment of OrgType 87654321-9999-9999-9999-999999999999 to the user.

Currently, this is not possible to do using bulk actions alone; although we hope in not-much-distant future it would be so. However, a relatively easy way to do that is using XSLT:

  1. Export all users from the repository into an XML document.
  2. Run a specially-prepared XSLT stylesheet against this document, transforming it into a description of a bulk action to be executed.
  3. Take resulting document containing a bulk action description, copy it into “Bulk action” window and execute it.

The XSLT stylesheet that transforms a list of users into bulk action description that makes above-described modifications would look like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
<xsl:stylesheet version="2.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform" xmlns:c="http://midpoint.evolveum.com/xml/ns/public/common/common-3"
                xmlns:s="http://midpoint.evolveum.com/xml/ns/public/model/scripting-3"
                xmlns:q="http://prism.evolveum.com/xml/ns/public/query-3"
                xmlns:t="http://prism.evolveum.com/xml/ns/public/types-3"
                xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <xsl:output indent="yes" />
    <xsl:template match="/*">
        <s:sequence>
            <xsl:apply-templates />
        </s:sequence>
    </xsl:template>
    <!-- the OID of the role whose assignments are to be modified -->
    <xsl:variable name="givenRole" select="'12345678-0000-0000-0000-000000000000'"/>
    <!-- we only want to process users with a given value of employeeType attribute -->
    <xsl:template match="c:user [ c:employeeType='ABCD' ]">
        <!-- take assignments of the given role that are without tenantRef yet -->
        <xsl:variable name="assignmentsOfGivenRole"
                      select="c:assignment [ c:targetRef/@oid=$givenRole and count(c:tenantRef)=0 ]" />
        <!-- take the tenantOid - the organization the user has assigned
             (we expect there is only one such organization for any given user) -->
        <xsl:variable name="tenantOid" select="c:assignment/c:targetRef[@type='OrgType']/@oid" />
        <xsl:choose>
            <!-- if there is exactly one role assignment to be modified -->
            <xsl:when test="count($assignmentsOfGivenRole) = 1">         
                <!-- assignment ID to be modified -->
                <xsl:variable name="assignmentId" select="$assignmentsOfGivenRole/@id"/>       
                <xsl:choose>
                    <!-- we expect just one tenant assignment to be present -->
                    <xsl:when test="count($tenantOid) != 1">             
                        <xsl:comment><xsl:value-of
                                            select="concat(' !!! Unexpected number of tenant orgs for user ',
                                                     c:name, ': ', count($tenantOid))" /></xsl:comment>
                    </xsl:when>
                    <xsl:otherwise>
                        <!-- here we construct Bulk Action for the user -->
                        <s:expression xsi:type="s:SearchExpressionType">
                            <s:type>c:UserType</s:type>
                            <s:searchFilter>
                                <q:equal>
                                    <q:path>c:name</q:path>
                                    <q:value><xsl:value-of select="c:name"/></q:value>
                                </q:equal>
                            </s:searchFilter>
                            <s:action>
                                <s:type>modify</s:type>
                                <s:parameter>
                                    <s:name>delta</s:name>
                                    <c:value xsi:type="t:ObjectDeltaType">
                                        <t:changeType>modify</t:changeType>     <!-- this is the default, can be omitted -->
                                        <!-- objectType and oid are taken from the object being modified -->
                                        <t:itemDelta>
                                            <t:modificationType>add</t:modificationType>
                                            <t:path>c:assignment[<xsl:value-of select="$assignmentId"/>]/tenantRef</t:path>
                                            <t:value oid="{$tenantOid}" type="OrgType"/>
                                        </t:itemDelta>
                                    </c:value>
                                </s:parameter>
                            </s:action>
                        </s:expression>
                    </xsl:otherwise>
                </xsl:choose>
            </xsl:when>
            <!-- more than one assignment to be modified (will be treated manually) -->
            <xsl:when test="count($assignmentsOfGivenRole) > 1">         
                <xsl:comment><xsl:value-of select="concat(' !!! Unexpected number of given role assignments for user ', c:name, ': ', count($assignmentsOfGivenRole))"/></xsl:comment>
            </xsl:when>
            <xsl:otherwise>
                <xsl:comment><xsl:value-of select="concat(' User ', c:name, ' has no role assignments to be fixed ')"/></xsl:comment>
            </xsl:otherwise>
        </xsl:choose>
    </xsl:template>
    <!-- all other users will be ignored -->
    <xsl:template match="c:user">
        <xsl:comment><xsl:value-of select="concat(' User ', c:name, ' is not of required type ')"/></xsl:comment>
    </xsl:template>
</xsl:stylesheet>

This stylesheet generates bulk action script that modifies midPoint users to get the desired state. Something like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
<?xml version="1.0" encoding="UTF-8"?>
<s:sequence xmlns:s="http://midpoint.evolveum.com/xml/ns/public/model/scripting-3"
            xmlns:t="http://prism.evolveum.com/xml/ns/public/types-3"
            xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
            xmlns:q="http://prism.evolveum.com/xml/ns/public/query-3"
            xmlns:c="http://midpoint.evolveum.com/xml/ns/public/common/common-3">
    <!-- User administrator is not of required type -->
    <s:expression xsi:type="s:SearchExpressionType">
        <s:type>c:UserType</s:type>
        <s:searchFilter>
            <q:equal>
                <q:path>c:name</q:path>
                <q:value>hrasko</q:value>
            </q:equal>
        </s:searchFilter>
        <s:action>
            <s:type>modify</s:type>
            <s:parameter>
                <s:name>delta</s:name>
                <c:value xsi:type="t:ObjectDeltaType">
                    <t:changeType>modify</t:changeType>
                    <t:itemDelta>
                        <t:modificationType>add</t:modificationType>
                        <t:path>c:assignment[3]/tenantRef</t:path>
                        <t:value oid="87654321-9999-9999-9999-999999999999" type="OrgType"/>
                    </t:itemDelta>
                </c:value>
            </s:parameter>
        </s:action>
    </s:expression>
    <s:expression xsi:type="s:SearchExpressionType">
        <s:type>c:UserType</s:type>
        <s:searchFilter>
            <q:equal>
                <q:path>c:name</q:path>
                <q:value>hraskova</q:value>
            </q:equal>
        </s:searchFilter>
        <s:action>
            <s:type>modify</s:type>
            <s:parameter>
                <s:name>delta</s:name>
                <c:value xsi:type="t:ObjectDeltaType">
                    <t:changeType>modify</t:changeType>
                    <t:itemDelta>
                        <t:modificationType>add</t:modificationType>
                        <t:path>c:assignment[13]/tenantRef</t:path>
                        <t:value oid="76543210-7777-7777-7777-777777777777" type="OrgType"/>
                    </t:itemDelta>
                </c:value>
            </s:parameter>
        </s:action>
    </s:expression>
</s:sequence>

All unusual cases (e.g. more than one role assignment, or more than one org assigned to the user) are marked in the form of XML comments in the resulting file, allowing for manual inspection and correcting.

So, by combining strengths of bulk actions with XSLT, it is possible to quite easily modify or fix data in midPoint system in the desired way.

Some final notes:

1) This mechanism is definitely not to be used as part of regular midPoint operation, e.g. to provision accounts based on changes on a resource or anything like that. For such scenarios, standard midPoint features like synchronization actions, inbound/outbound mappings, or object templates should be used. What we’ve described just now is for ad-hoc modifying/fixing/migrating the data, usually during a deployment.

2) For cases where only the repository has to be changed – with no provisioning operations altogether – an alternative solution is direct modification of exported data and re-import of the modified data. But this approach is insufficient if we need to carry out some provisioning related to changes being made. Like creating accounts based on modified roles. Bulk actions are a good candidate in such cases.

3) It is possible to execute bulk actions also from the command line, using runscript tool.

4) The bulk actions have been changed a bit since midPoint 3.0 release. The code shown here executes cleanly on current development snapshot that will eventually appear as midPoint 3.1.

Leave a Reply

Your email address will not be published.