We have a simple database table with 5 columns

• Auto generated: row_index and when_created timestamp
• Managed by grouper: group_name, subject_source_id, subject_id

Provisioning to this table has been working (seemingly perfect) for about month in v2.6.19.

After our recent upgrade to 4.5.5, we're seeing fairly persistent database errors during this provisioner's FullSync runs:

Cannot invoke "com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getNestedDataSource()" because "wrapperConnectionPoolDataSource" is null

(GrouperLoaderDb)

2023-09-13 17:46:04.881: ERROR: Provisioner 'iam_unifieddb_group_memberships' (vwsayaya) Error inserting memberships, e.g. Mship(matchingAttrs: LinkedHashSet(1): [0]: [id, val: MultiKey[gt:ref:affiliations:employee@gt, 128173C66F91C4F0A916DF8D0A407C34], compareVal: MultiKey[gt:ref:affiliations:employee@gt, 128173C66F91C4F0A916DF8D0A407C34], currentValue: true], attr[group_name]: "gt:ref:affiliations:employee@gt", attr[subject_id]: "128173C66F91C4F0A916DF8D0A407C34", attr[subject_source_id]: "gted-people", ins group_name "gt:ref:affiliations:employee@gt", ins subject_id "128173C66F91C4F0A916DF8D0A407C34", ins subject_source_id "gted-people", recalcObject: true, create: true, selectProcessed: true)
(vwsayaya): java.lang.RuntimeException: sql: insert into grouper.memberships_from_grouper(group_name,subject_id,subject_source_id) values (?, ?, ?), 
(vwsayaya):     at edu.internet2.middleware.grouperClient.jdbc.GcDbAccess.callbackResultSet(GcDbAccess.java:2361)
(vwsayaya):     at edu.internet2.middleware.grouperClient.jdbc.GcDbAccess.executeBatchSql(GcDbAccess.java:2407)
(vwsayaya):     at edu.internet2.middleware.grouper.app.sqlProvisioning.SqlProvisionerCommands.insertObjects(SqlProvisionerCommands.java:508)
(vwsayaya):     at edu.internet2.middleware.grouper.app.sqlProvisioning.SqlProvisioningDao.insertMemberships(SqlProvisioningDao.java:1102)
(vwsayaya):     at edu.internet2.middleware.grouper.app.provisioning.targetDao.GrouperProvisionerTargetDaoAdapter.insertMembershipsHelper(GrouperProvisionerTargetDaoAdapter.java:4012)
(vwsayaya):     at edu.internet2.middleware.grouper.app.provisioning.targetDao.GrouperProvisionerTargetDaoAdapter$31.callLogic(GrouperProvisionerTargetDaoAdapter.java:4072)
(vwsayaya):     at edu.internet2.middleware.grouper.app.provisioning.targetDao.GrouperProvisionerTargetDaoAdapter$31.callLogic(GrouperProvisionerTargetDaoAdapter.java:4066)
(vwsayaya):     at edu.internet2.middleware.grouper.util.GrouperCallable$1.callback(GrouperCallable.java:203)
(vwsayaya):     at edu.internet2.middleware.grouper.GrouperSession.callbackGrouperSession(GrouperSession.java:1000)
(vwsayaya):     at edu.internet2.middleware.grouper.util.GrouperCallable.callLogicWithSessionIfExists(GrouperCallable.java:200)
(vwsayaya):     at edu.internet2.middleware.grouper.util.GrouperCallable.call(GrouperCallable.java:166)
(vwsayaya):     at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
(vwsayaya):     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
(vwsayaya):     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
(vwsayaya):     at java.base/java.lang.Thread.run(Thread.java:833)
(vwsayaya): Caused by: java.lang.RuntimeException: Error connecting to: null
(vwsayaya):     at edu.internet2.middleware.grouperClient.jdbc.GcDbAccess.connection(GcDbAccess.java:2067)
(vwsayaya):     at edu.internet2.middleware.grouperClient.jdbc.GcDbAccess.callbackResultSet(GcDbAccess.java:2301)
(vwsayaya):     ... 14 more
(vwsayaya): Caused by: java.lang.RuntimeException: Error calling constructor and 'connection' on edu.internet2.middleware.grouper.app.loader.db.GrouperLoaderDb
(vwsayaya):     at edu.internet2.middleware.grouperClient.jdbc.GcDbAccess.connectionGetFromPool(GcDbAccess.java:2119)
(vwsayaya):     at edu.internet2.middleware.grouperClient.jdbc.GcDbAccess.connectionHelper(GcDbAccess.java:2743)
(vwsayaya):     at edu.internet2.middleware.grouperClient.jdbc.GcDbAccess.connection(GcDbAccess.java:2050)
(vwsayaya):     ... 15 more
(vwsayaya): Caused by: java.lang.RuntimeException: Problems with db: DB: user: grouper_provisioner, url: jdbc:postgresql://iam-unified-db-test.cluster-cvlmtfrka6np.us-east-1.rds.amazonaws.com:5432/test_iam_unified, databaseConfigType: loader, driver: org.postgresql.Driver, connectionName: unified_db_provisioning
(vwsayaya):     at edu.internet2.middleware.grouper.app.loader.db.GrouperLoaderDb.connection(GrouperLoaderDb.java:399)
(vwsayaya):     at jdk.internal.reflect.GeneratedMethodAccessor428.invoke(Unknown Source)
(vwsayaya):     at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
(vwsayaya):     at java.base/java.lang.reflect.Method.invoke(Method.java:568)
(vwsayaya):     at edu.internet2.middleware.grouperClient.util.GrouperClientCommonUtils.invokeMethod(GrouperClientCommonUtils.java:2787)
(vwsayaya):     at edu.internet2.middleware.grouperClient.util.GrouperClientCommonUtils.callMethod(GrouperClientCommonUtils.java:2745)
(vwsayaya):     at edu.internet2.middleware.grouperClient.util.GrouperClientCommonUtils.callMethod(GrouperClientCommonUtils.java:2922)
(vwsayaya):     at edu.internet2.middleware.grouperClient.jdbc.GcDbAccess.connectionGetFromPool(GcDbAccess.java:2114)
(vwsayaya):     ... 17 more
(vwsayaya): Caused by: java.lang.NullPointerException: Cannot invoke "com.mchange.v2.c3p0.WrapperConnectionPoolDataSource.getNestedDataSource()" because "wrapperConnectionPoolDataSource" is null
(vwsayaya):     at edu.internet2.middleware.grouper.app.loader.db.GrouperLoaderDb.retrieveDataSourceFromC3P0(GrouperLoaderDb.java:200)
(vwsayaya):     at edu.internet2.middleware.grouper.app.loader.db.GrouperLoaderDb.connection(GrouperLoaderDb.java:296)
(vwsayaya):     ... 24 more
(vwsayaya): 

Notes:

1. The External System tests well
2. We have Connection Testing turned on in the External System
3. The incremental provisioning sometimes works
4. Sometimes the FullSyncs do get a connection and submit bad sql (separate Jira)

We've bumped up logging for ...grouper.app.loader.db package, but don't have any analysis that includes this information.