This article is a continuation of the previous article on a Zero Downtime Upgrade Strategy Using a Blue/Green Deployment for AM. Traditionally, ForgeRock Identity Management (IDM) upgrades are handled either in-place or by leveraging the migration service. As many deployments have constraints around this approach (zero downtime, immutable, etc.), a parallel deployment approach, or a blue/green strategy can be leveraged for upgrading ForgeRock IDM servers.
This article provides a high-level approach for using a blue/green methodology for updating ForgeRock IDM servers.
This corresponds to a Unit 4: IDM Upgrade in our overall ForgeRock approach to upgrading.
Determining the correct approach
IDM 6.5 is bundled with the migration service, which lets you migrate data from a blue environment to a green environment, table by table, and row by row. This leads to identical data on both sides. The constraint is that update traffic must be cut off in the blue environment to ensure consistent data in the green environment after the migration completes. It is not a zero downtime migration.
The method described here shows you how to perform a zero downtime migration.
This approach in this article should be seen as a foundation on which to build your migration strategy. As IDM dependencies are very elaborate and varied, it is very difficult to design a perfect strategy that will work with all kinds of deployments.
The table below shows the major differences between the migration service method and the scripted CREST method:
Before using the methodology described in this article, please confirm that you really need zero downtime. It may be that the amount of data is too big to fall within an overnight window, which prevents you from using the migration service. The solution to this may be overcome with the following measures:
- Verify that the blue system nodes are performance-optimized to ensure the lowest possible response time.
- Allocate more nodes in the green environment, and distribute the migration between the nodes by partitioning the mappings.
If neither of the above options are possible, then please read on.
1. This approach assumes that your infrastructure processes have the ability to install a parallel deployment for upgrade, or you are already using blue/green deployment.
2. In above diagram, a blue cluster represents an existing IDM deployment (like 4.5.x version) and a green one represents a new IDM deployment (like 6.5.x version).
3. Review the Release Notes for all IDM versions between existing and target IDM deployment for new, deprecated features, bug fixes, etc. For an IDM 4.5 to IDM 6.5 upgrade, review the Release Notes for IDM 5.0, 5.5, 6.0, and 6.5 versions.
4. Verify that the deployment upgrade fits with a blue/green deployment. It requires copying repository data over. When data transformation is not required, such as upgrading from 6.0 to 6.5, sharing the repository, which is not a pure blue/green deployment, might be the best fit.
5. When external systems are the source of truth for data, the IDM repository can be rebuilt through a full reconciliation. Therefore, the methodology exposed in this paper is not relevant for this case.
Upgrade the Process Using the CREST Scripted Connector
1. Prepare the migration:
- Clone the crestidmmigration project from Bitbucket, and read the project description so that you can decide whether this method fits the requirements. While implicit sync always occurs in the blue environment, decide which environment you will use to initiate the reconciliation.
- Decide which strategy to employ for external resources (such as a ForgeRock Directory Server). As described in the project, you can migrate the repository links table using the migration service, as the CREST migration preserves the managed object IDs. Alternatively, you may perform a full reconciliation after the migration. In all cases, turn off implicit sync for all mappings in the green environment to avoid unnecessary (and perhaps, conflicting) traffic to the external provisioning systems.
- Follow the instructions in the Installation Guide to prepare for a migration, except that you are not going to perform a migration.
- Provide all property mappings in sync.json. For encrypted values such as “password”, step 2 in the Installation Guide ensures that both environments share the same encryption keys. Provide all the necessary configuration information to optimize the reconciliation (in particular, paged reconciliation). However, do not alter the correlationQuery.
- Edit the onCreate and onUpdate scripts to add processing for all relationships. These scripts ensure that relationship properties are propagated to the target, but also filter out any relationships to resources that are not yet provisioned. This prevents duplicate entries.
- Edit provisioner.openicf-scriptedcrest.json to include all properties that will be migrated. Be careful not to change the configuration for the _id property, and examine how relationship properties are configured (See “devices” for multivalued, and “owner” for single value).
- Perform a blue/green upgrade to deploy the CREST scripted connector for the 4.x or 5.x nodes. Ensure that implicit sync is disabled for all mappings involving the CREST connectors. The resulting green environment becomes the blue environment in the next migration phase.
- Deploy the green environment (for example, 6.5) prepared for migration (including the CREST connector, if reconciliation will be launched from the green nodes).
2. Turn on implicit sync in blue, and launch the reconciliation of each scripted CREST mapping, one after the other completes (so they are not running concurrently).
3. You may use the migration service to migrate other data, such as internal roles, internal users, and links.
4. Switch over to the new deployment:
- After validating that the new deployment is working correctly, switch the load balancer from blue to green, and turn off implicit sync to external provisioning systems in blue. Then, turn on implicit sync to external provisioning systems in green, and perform a reconciliation with the external provisioning systems, or use a custom method if provisioning is not performed through reconciliation.
- If there are any issues, you can always roll back to blue deployment.
Note: Any managed object or configuration change made after switchover should be applied to both blue and green deployments so that no change is lost during rollback.
- Stop the migration service on the green deployment.
- Stop the blue IDM servers.
- De-provision the blue deployment.
Although a blue/green deployment requires a high level of deployment maturity, this approach provides an elegant way to minimize downtime for ForgeRock deployment upgrades. It is always advisable to practice an upgrade strategy in lower environments like dev, stage before moving to the production environment.
Depending on the complexity of your deployment, multiple things may need to be considered for these upgrades, such as customizations, new FR features, etc. It is always recommended to break the entire upgrade process into multiple releases like “base upgrade”, followed by “leveraging new features” and so on.
When deploying IDM for the first time, it is always advisable to incorporate the upgrade strategy early on in the project, so that any designed feature allows for seamless migration in the future. Also, syncing to the green environment will impact the blue update performance, as implicit syncs are all executed synchronously on the request’s thread (on 4.x, 5.x, and 6.0). Fortunately, this does not apply any more to 6.5 when queued sync is enabled. The impact will be sensible with a high relationship cardinality, as the process requires interrogating the target system for each relationship before propagating it. This is why planning the upgrade strategy well in advance is important.
The provided CREST scripts and configuration are a starting point, as well as a proof of concept, which you can use as the basis to build your own upgrade based on your deployment requirements. The details are described in the crestidmmigration project. Note that a second alternative solution is proposed there; one that preserves relationships and lets you run the migration service from the green environment (CREST+MIGRATION folder). However, this may contribute to a higher performance hit on the blue environment. Please note that crestidmmigration is an evolving project, and as such, some other variants could be proposed in the future, so stay tuned!