This topic has 14 replies, 4 voices, and was last updated 6 years ago by Bill Nelson.

  • Author
    Posts
  • #13178
     rhamilton828
    Participant

    I have added a connector and a number of mappings and managed objects to an instance of OpenIDM I have running. This instance is just a local test instance, and I would like to persist the mappings and connector/managed object configuration so that I can push these configs to whatever real-world OpenIDM instances we wind up creating.

    I am currently just using the embedded OrientDB backend. Really, other than the connector etc. I have not changed the configuration of this OpenIDM instance.

    I cannot figure out how to back up the changes I made. The documentation seems to suggest that you can just push your updated managed.json and sync.json for managed object / mapping configuration to a new server. This does NOT work; I get query stack traces when I go to the managed object page in OpenIDM after pushing these configs.

    Do I need to dump what is stored to OrientDB then reload it? Could I get away with zipping up the db/ and conf/ dirs in my OpenIDM instance and then unpacking these to a new instance?

    #13180
     Tom Wood
    Participant

    Hi,

    Have you looked at the cli.sh ‘configexport’ command? There is some information available from the Integrator’s Guide which may help you achieve this.

    #13183
     ssripathy
    Participant

    Can you please post the query stack trace here?

    That might help us provide you with specific ways you could fix it.

    #13184
     Bill Nelson
    Participant

    The documentation seems to suggest that you can just push your updated managed.json and sync.json for managed object / mapping configuration to a new server. This does NOT work; I get query stack traces when I go to the managed object page in OpenIDM after pushing these configs.

    Speaking from personal experience, this works quite well. There is definitely something else going on for you to get the stack trace. I agree with @ssripathy, you need to post details of your errors here for us to provide advice, but seriously, I push the same config to multiple servers all the time. That is just part of a standard deployment process from source control.

    #13185
     rhamilton828
    Participant

    The configexport command in cli.sh seems to work to at least export configs. I’ll try the import once my VM finishes rebuilding. If I get the query stack trace again I’ll post it here.

    #13212
     rhamilton828
    Participant

    Okay, here’s my progress so far.

    ./cli.sh configexport works fine, or at least it appears to, but if I run configimport I get errors. I will put these in a separate post.

    I then tried just manually replacing the out-of-the-box sync.json and mapping.json files. This seemed to work. I can’t get the connector JSON file to load properly, but the other two did load in and I can see the mappings and managed objects in the UI.

    My mappings are bidirectional; I have one going from the external OpenDJ LDAP store to the managed objects and another going in the reverse direction. Reconciliation in either direction is not working. This time, I don’t get the stack trace in the UI, but I see this in the logs:

    
    INFO: Reconciliation reported exception
    org.forgerock.openidm.sync.impl.SynchronizationException: Synchronization failed
            at org.forgerock.openidm.sync.impl.ObjectMapping.doRecon(ObjectMapping.java:1056)
            at org.forgerock.openidm.sync.impl.ObjectMapping.recon(ObjectMapping.java:920)
            at org.forgerock.openidm.sync.impl.ReconciliationService.reconcile(ReconciliationService.java:401)
            at org.forgerock.openidm.sync.impl.ReconciliationService.access$000(ReconciliationService.java:91)
            at org.forgerock.openidm.sync.impl.ReconciliationService$1.run(ReconciliationService.java:357)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
            at java.lang.Thread.run(Thread.java:745)
    Caused by: org.forgerock.openidm.sync.impl.SynchronizationException: Failed to resolve and parse the query select _openidm_id, @version from ${unquoted:_resource} SKIP ${unquoted:_pagedResultsOffset} LIMIT ${unquoted:_pageSize} with params: {_pageSize=-1, resourceName=managed/ForgeRockUser, _pagedResultsOffset=0, pageClause=, _resource=managed_ForgeRockUser}
            at org.forgerock.openidm.sync.impl.ReconTypeBase.query(ReconTypeBase.java:253)
            at org.forgerock.openidm.sync.impl.ReconTypeByQuery.queryTarget(ReconTypeByQuery.java:82)
            at org.forgerock.openidm.sync.impl.ReconciliationContext.queryTarget(ReconciliationContext.java:285)
            at org.forgerock.openidm.sync.impl.ObjectMapping.doRecon(ObjectMapping.java:964)
            ... 7 more
    Caused by: org.forgerock.json.resource.BadRequestException: Failed to resolve and parse the query select _openidm_id, @version from ${unquoted:_resource} SKIP ${unquoted:_pagedResultsOffset} LIMIT ${unquoted:_pageSize} with params: {_pageSize=-1, resourceName=managed/ForgeRockUser, _pagedResultsOffset=0, pageClause=, _resource=managed_ForgeRockUser}
            at org.forgerock.openidm.repo.orientdb.impl.query.Queries.query(Queries.java:218)
            at org.forgerock.openidm.repo.orientdb.impl.OrientDBRepoService.query(OrientDBRepoService.java:652)
            at org.forgerock.openidm.repo.orientdb.impl.OrientDBRepoService.handleQuery(OrientDBRepoService.java:565)
            at org.forgerock.json.resource.Router.handleQuery(Router.java:310)
            at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:99)
            at org.forgerock.json.resource.Filters$ConditionalFilter.filterQuery(Filters.java:92)
            at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:97)
            at org.forgerock.json.resource.Filters$ConditionalFilter.filterQuery(Filters.java:92)
            at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:97)
            at org.forgerock.json.resource.Filters$ConditionalFilter.filterQuery(Filters.java:92)
            at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:97)
            at org.forgerock.openidm.audit.filter.AuditFilter.filterQuery(AuditFilter.java:152)
            at org.forgerock.json.resource.Filters$ConditionalFilter.filterQuery(Filters.java:90)
            at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:97)
            at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$5.filterQuery(ServletConnectionFactory.java:520)
            at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:97)
            at org.forgerock.openidm.maintenance.impl.PassthroughFilter.filterQuery(PassthroughFilter.java:66)
            at org.forgerock.openidm.maintenance.impl.MaintenanceService.filterQuery(MaintenanceService.java:262)
            at org.forgerock.json.resource.Filters$ConditionalFilter.filterQuery(Filters.java:90)
            at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:97)
            at org.forgerock.json.resource.FilterChain.handleQuery(FilterChain.java:231)
            at org.forgerock.json.resource.InternalConnection.queryAsync(InternalConnection.java:78)
            at org.forgerock.json.resource.AbstractAsynchronousConnection.query(AbstractAsynchronousConnection.java:62)
            at org.forgerock.json.resource.AbstractConnectionWrapper.query(AbstractConnectionWrapper.java:169)
            at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$1$1.query(ServletConnectionFactory.java:305)
            at org.forgerock.json.resource.AbstractConnectionWrapper.query(AbstractConnectionWrapper.java:169)
            at org.forgerock.openidm.managed.ManagedObjectSet.queryCollection(ManagedObjectSet.java:1051)
            at org.forgerock.json.resource.InterfaceCollectionHandler.handleQuery(InterfaceCollectionHandler.java:62)
            at org.forgerock.json.resource.Router.handleQuery(Router.java:310)
            at org.forgerock.openidm.managed.ManagedObjectService$ManagedObjectSetRequestHandler.handleQuery(ManagedObjectService.java:195)
            at org.forgerock.json.resource.Router.handleQuery(Router.java:310)
            at org.forgerock.openidm.managed.ManagedObjectService.handleQuery(ManagedObjectService.java:300)
            at org.forgerock.json.resource.Router.handleQuery(Router.java:310)
            at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:99)
            at org.forgerock.json.resource.Filters$ConditionalFilter.filterQuery(Filters.java:92)
            at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:97)
            at org.forgerock.json.resource.Filters$ConditionalFilter.filterQuery(Filters.java:92)
            at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:97)
            at org.forgerock.json.resource.Filters$ConditionalFilter.filterQuery(Filters.java:92)
            at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:97)
            at org.forgerock.openidm.audit.filter.AuditFilter.filterQuery(AuditFilter.java:152)
            at org.forgerock.json.resource.Filters$ConditionalFilter.filterQuery(Filters.java:90)
            at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:97)
            at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$5.filterQuery(ServletConnectionFactory.java:520)
            at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:97)
            at org.forgerock.openidm.maintenance.impl.PassthroughFilter.filterQuery(PassthroughFilter.java:66)
            at org.forgerock.openidm.maintenance.impl.MaintenanceService.filterQuery(MaintenanceService.java:262)
            at org.forgerock.json.resource.Filters$ConditionalFilter.filterQuery(Filters.java:90)
            at org.forgerock.json.resource.FilterChain$Cursor.handleQuery(FilterChain.java:97)
            at org.forgerock.json.resource.FilterChain.handleQuery(FilterChain.java:231)
            at org.forgerock.json.resource.InternalConnection.queryAsync(InternalConnection.java:78)
            at org.forgerock.json.resource.AbstractAsynchronousConnection.query(AbstractAsynchronousConnection.java:62)
            at org.forgerock.json.resource.AbstractConnectionWrapper.query(AbstractConnectionWrapper.java:169)
            at org.forgerock.openidm.servlet.internal.ServletConnectionFactory$1$1.query(ServletConnectionFactory.java:305)
            at org.forgerock.json.resource.AbstractConnectionWrapper.query(AbstractConnectionWrapper.java:169)
            at org.forgerock.openidm.sync.impl.ReconTypeBase.query(ReconTypeBase.java:220)
            ... 10 more
    Caused by: com.orientechnologies.orient.core.exception.OQueryParsingException: Error on parsing query at position #22: Error on parsing query
    Query:  managed_ForgeRockUser SKIP 0 LIMIT -1
    ---------------------------^
            at com.orientechnologies.orient.core.sql.filter.OSQLTarget.<init>(OSQLTarget.java:76)
            at com.orientechnologies.orient.core.sql.OSQLEngine.parseTarget(OSQLEngine.java:444)
            at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.parse(OCommandExecutorSQLSelect.java:215)
            at com.orientechnologies.orient.core.sql.OCommandExecutorSQLSelect.parse(OCommandExecutorSQLSelect.java:112)
            at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.parse(OCommandExecutorSQLDelegate.java:52)
            at com.orientechnologies.orient.core.sql.OCommandExecutorSQLDelegate.parse(OCommandExecutorSQLDelegate.java:33)
            at com.orientechnologies.orient.core.storage.OStorageEmbedded.command(OStorageEmbedded.java:101)
            at com.orientechnologies.orient.core.sql.query.OSQLQuery.run(OSQLQuery.java:69)
            at com.orientechnologies.orient.core.sql.query.OSQLSynchQuery.run(OSQLSynchQuery.java:80)
            at com.orientechnologies.orient.core.query.OQueryAbstract.execute(OQueryAbstract.java:29)
            at org.forgerock.openidm.repo.orientdb.impl.query.ConfiguredQueries.doTokenSubsitutionQuery(ConfiguredQueries.java:216)
            ... 65 more
    Caused by: com.orientechnologies.orient.core.exception.OCommandExecutionException: Class 'MANAGED_FORGEROCKUSER' was not found in current database
            at com.orientechnologies.orient.core.sql.filter.OSQLTarget.extractTargets(OSQLTarget.java:247)
            at com.orientechnologies.orient.core.sql.filter.OSQLTarget.<init>(OSQLTarget.java:67)
            ... 76 more
    

    What is causing this to fail? I can see the ForgeRockUser managed object in the UI, and in the Data(account) section in my connector I can see the data it should be pulling in.

    #13216
     rhamilton828
    Participant

    The configimport part was my own mistake; I was unclear what the URL should actually be so it was not finding the OpenIDM REST endpoints.

    Here’s a followup question as well.

    Let’s say I provision a new OpenIDM instance (currently this is running in a Vagrant box I can destroy and reload at will), then I copy my connector json file (provisioner.openicf-(my connector name).json) to the OpenIDM conf directory immediately after the instance comes up. I restart the instance and then log in to OpenIDM.

    Shouldn’t I see the connector in the UI? The connector JSON file was taken directly from configexport so it appears to be valid (no errors in the logs either). When I first get into OpenIDM, I see the following error pop up briefly in red after importing the connector JSON file:

    “Cannot update the record because the revision is not the latest.”

    What revision? Of what part? Nothing is logged regarding this error AFAICT. I have my connector set as an LDAP connector, version 1.4.10, which is exactly what I see in (openidm base)/connectors, so I am confused as to what is going on here.

    #13217
     rhamilton828
    Participant

    Changing the bundleVersion element from:

    
    "bundleVersion": "1.4.1.0",
    

    to:

    
    "bundleVersion" : "[1.4.0.0,2.0.0.0)",
    

    at least gets rid of the “Cannot update the record” error, though I don’t understand why. I still don’t see the connector in OpenIDM after copying the JSON file to conf.

    #13229
     Bill Nelson
    Participant

    The revision it is referring to is the revision of the configuration object (in this case the provisioner definition) contained in the repo. OpenIDM implements version control using the _rev for that object. If you try to patch something that does not match the current _rev, then you see that error.

    I copy my connector json file (provisioner.openicf-(my connector name).json) to the OpenIDM conf directory immediately after the instance comes up. I restart the instance and then log in to OpenIDM.

    You don’t need to restart. The connector config file is hot-swappable. If you look in the openidm0.log.0 file, you should see that the configuration was updated when you placed the file in the folder structure. The log file should also tell you if there are any issues with the connector config file as well.

    The only time I have ever seen OpenIDM not recognize the connector change is if you rename the “name” parameter in the connector file, but leave the filename the same name.

    But as I said before, part of our standard process in moving files from DEV to TEST to PROD is simply to create the file in DEV, make it work and store the file in Stash. Then go to TEST box and pull the file from Stash to TEST. Then repeat process for PROD. The only differences between the files between the various environments are things like host, credentials, etc. And these are handled by variables as described in my blog here: http://www.identityfusion.com/openidm-property-value-substitution/.

    Note: I rarely use the config export as everything is handled through source control.

    #13233
     rhamilton828
    Participant

    Yeah, I did figure out the issue with the connector, at least. It makes sense but I didn’t realize this was the case.

    Basically the issue was that I had exported the connector through configexport, but every time a new instance is built via our Vagrant/Ansible definitions, the credential secret is different. If I reencrypt the password to the external OpenDJ instance using “cli.sh encrypt” and replace the “credentials” section in the connector JSON file with the output of the cli.sh command, the connector gets loaded in OpenIDM. I don’t even have to change the bundleVersion element in the JSON.

    However I am still getting the error I posted a few posts above when I try to run my mappings to sync data from the OpenDJ instance into OpenIDM.

    Also, I promised I would post the error I was getting in the OpenIDM UI (it shows up in a popup briefly when I go to the UI pages for my managed objects):

    
    Failed to resolve and parse the query SELECT * FROM ${unquoted:_resource} WHERE 1 = 1 SKIP 0 LIMIT 50 ORDER BY cn ASC with params: {_pageSize=50, _pagedResultsOffset=0, page=1, sort_by=cn, pageClause=SKIP 0 LIMIT 50 ORDER BY cn ASC, _resource=managed_ForgeRockGroup}
    

    Is this familiar to anyone?

    Thanks to all involved in this thread for all the help so far, btw.

    #13235
     rhamilton828
    Participant

    Judging from the giant stack trace from the logs (not the error I just posted), it would almost seem that the managed objects are not being created in the database. What sorts of things would cause this?

    If I find anything on my own in the meantime I’ll post it here.

    #13236
     Bill Nelson
    Participant

    The error you are seeing is because (it appears) that your OpenIDM instance does not recognize or cannot parse that query.

    A select statement such as that would most likely be defined in the repo configuration file (repo.orientdb.json or repo.jdbc.json depending on what DB you are using).

    I did a quick search in the OrientDB repo file and looked in those defined in the db folder for various DBs. I don’t see that pattern defined anywhere so I am not sure if your team customized the repo searches or if this is originating from another file (I suspect it is the former) but I would start looking there.

    #13239
     rhamilton828
    Participant

    Hmm, that’s interesting. As far as I know the repo searches were never customized. I certainly didn’t do so, at least not intentionally. The VM provisioning shouldn’t be doing anything of that nature either to my knowledge. I’m at least fairly sure the instance that gets created is a bog-standard, out-of-the-box OpenIDM.

    I’ll try and compare the version of repo.orientdb.json that lives on a new instance to the one from a working instance that I backed up and see if that leads me anywhere.

    #13240
     rhamilton828
    Participant

    I copied over the repo.orientdb.json file from the working instance to a new one and now my mappings are working.

    Again, thanks for the help. I wouldn’t have known where to even start looking.

    #13241
     Bill Nelson
    Participant

    No worries. Glad I could help.

    May the IDs be ever in your favor…

Viewing 15 posts - 1 through 15 (of 15 total)

You must be logged in to reply to this topic.

©2022 ForgeRock - we provide an identity and access platform to secure every online relationship for the enterprise market, educational sector and even entire countries. Click to view our privacy policy and terms of use.

Log in with your credentials

Forgot your details?