If you write a Keycloak SPI provider factory whose getId() returns the same string as a built-in factory for the same SPI, Keycloak will silently discard one of the two at startup. It picks a winner before it ever calls isSupported(), so the guard you wrote to decide when your provider should activate never runs. There is no warning at default log levels. The server starts clean and serves traffic on the provider you did not choose.
We shipped that bug into our own Redis cache backend for Locke, and it took a Prometheus counter that stubbornly refused to increment before anyone noticed. This is the walkthrough we promised in the launch post.
The symptom, which was no symptom at all
Locke adds a Redis cache backend to Keycloak, selectable at boot with KC_CACHE=redis. Eleven cache types move over: authentication sessions, user sessions, login failures, single-use tokens, and the read-mostly configuration caches for realms, users, and authorization.
Set KC_CACHE=redis and everything looked right. Keycloak started. Logins worked. The benchmark harness ran clean. Redis showed traffic when we watched it with redis-cli MONITOR. Tests passed, the build was green, and five rounds of optimization work had been measured against that setup.
Three of those eleven caches were not on Redis at all. The realm cache, the user cache, and the authorization cache were quietly running on embedded Infinispan, which is precisely the thing the project exists to make optional. The config flag said redis. The code said Infinispan. Nothing said anything.
The reason this is worse than a crash: a crash tells you where to look. This told us our numbers were good.
Why nothing caught it
Every check we had was aimed at the wrong layer.
The benchmarks measured end-to-end login latency. A realm cache on Infinispan is a perfectly fast realm cache, so throughput looked fine. If anything it flattered us, because in-process reads beat network reads.
The integration tests asserted that logins succeeded and sessions persisted across pod restarts. Session caches genuinely were on Redis, so those passed honestly.
The unit tests constructed our provider classes directly and exercised their methods. Those classes were correct. They were simply never instantiated by the running server.
And redis-cli MONITOR showed real Redis traffic, because the session caches were chattering away on the same connection. Watching the wire told us Redis was in use. It did not tell us which caches were using it.
Every one of those signals is a proxy for “is the Redis path working.” None of them was a direct measurement of “is this specific cache on Redis.” That distinction is the whole bug.
What ProviderManager actually does
Keycloak discovers provider factories through ServiceLoader, then deduplicates them in ProviderManager.load(Spi). This is the relevant loop, from services/src/main/java/org/keycloak/provider/ProviderManager.java:
Map<String, ProviderFactory> loaded = new HashMap<>();
for (ProviderLoader loader : loaders) {
List<ProviderFactory> f = loader.load(spi);
if (f != null) {
for (ProviderFactory pf : f) {
String uniqueId = spi.getName() + "-" + pf.getId();
if (!loaded.containsKey(uniqueId)) {
loaded.put(uniqueId, pf);
} else {
ProviderFactory currentFactory = loaded.get(uniqueId);
ProviderFactory factoryToUse = compareFactories(currentFactory, pf);
loaded.put(uniqueId, factoryToUse);
logger.debugf("Found multiple provider factories of same provider ID "
+ "implementing same SPI. ...");
}
}
}
}The dedup key is spi.getName() + "-" + pf.getId(). Two factories that implement the same SPI and return the same getId() collide into one map entry, and compareFactories decides which survives:
public ProviderFactory compareFactories(ProviderFactory p1, ProviderFactory p2) {
if (p1.order() != p2.order()) return (p1.order() > p2.order()) ? p1 : p2;
// Internal factory is supposed to be overriden by custom factory
if (DefaultKeycloakSessionFactory.isInternal(p1) ^ DefaultKeycloakSessionFactory.isInternal(p2)) {
return DefaultKeycloakSessionFactory.isInternal(p1) ? p2 : p1;
}
return p1;
}Read that carefully, because the tiebreakers matter:
- Higher
order()wins. Both of ours returned the default, so this was a tie. - If exactly one factory is “internal” (living under the
org.keycloakpackage namespace), the external one wins. This is the escape hatch that lets your custom JAR override a built-in. Locke’s Redis factories live inorg.keycloak.models.cache.redis, so both sides were internal. The XOR is false. Tie again. - Fall through to
return p1, meaning whichever factory happened to be loaded first keeps the slot. That is classpath and loader ordering, not intent.
Infinispan’s InfinispanCacheRealmProviderFactory returns "default". So did ours. Infinispan got there first. Ours was dropped on the floor.
Now the part that turns a mistake into a trap. Our factory had a guard that was supposed to prevent exactly this:
@Override
public boolean isSupported(Config.Scope config) {
return "redis".equals(config.root().get("cache"));
}That guard is real, it is correct, and it never ran. isSupported() is invoked from DefaultKeycloakSessionFactory.isEnabled(), which is called while iterating the factories that ProviderManager.load() returned:
protected boolean isEnabled(ProviderFactory factory, Config.Scope scope) {
if (!scope.getBoolean("enabled", true)) {
return false;
}
if (factory instanceof EnvironmentDependentProviderFactory) {
return ((EnvironmentDependentProviderFactory) factory).isSupported(scope);
}
return true;
}Dedup happens first. Activation guards run second, and only on the survivors. A factory eliminated in round one is never asked whether it wanted the job. Writing a careful isSupported() feels like declaring your intent to Keycloak, but it is only consulted after Keycloak has already narrowed the field on a criterion you may not have known existed.
And the one place Keycloak does mention the collision, that logger.debugf call, is at DEBUG. At the default log level it prints nothing. Turn on --log-level=DEBUG and the message is right there, clear and accurate. Nobody runs a benchmark at DEBUG.
The counter that would not increment
Iteration 6 of the Redis work was about observability rather than speed. We added a RedisMetrics holder that binds Micrometer meters to Metrics.globalRegistry, the same registry Keycloak’s own keycloak_user_events_* meters use, and wired counters into every adapter on the Redis path. The goal was modest: stop needing a bespoke harness to answer “did that change help.”
After twenty password-grant logins against the iteration 6 image, :9000/metrics returned this:
# TYPE keycloak_redis_l2_ops counter
keycloak_redis_l2_ops_total{cache="authenticationSessions",op="hset_multi"} 40.0
keycloak_redis_l2_ops_total{cache="authenticationSessions",op="hset"} 280.0The auth-session numbers were exactly what we hoped for. 280 single-field HSETs across 20 logins works out to 14 per login on the hot path, which directly confirmed that the previous iteration’s move to field-level writes had landed.
What caught our attention was a label that was not there. No series carried cache="realms". None carried cache="users" or cache="authorization". Caffeine’s stock cache_* meters, which the L1 layer registers on first use, had no entries for those cache names either.
The first theory was a metrics bug: lazy registration failing to fire, or a race in computeIfAbsent. We logged it in the iteration doc as an open question rather than a defect, which in hindsight was the correct instinct pointed at the wrong subject. A counter that never increments is not necessarily a broken counter. Sometimes it is an accurate report that the code path does not execute.
Chasing why RedisCacheRealmProviderFactory.lazyInit never ran led straight into ProviderManager.load(), and the dedup key made the answer obvious in about the time it takes to read one String concatenation.
The observability work found a correctness bug it was not looking for. That is the single most useful thing to take from this post: metrics you add to measure performance will also tell you when code is not running, and “not running” is a failure mode that passing tests are structurally bad at detecting.
The fix that broke startup
The fix looked like a one-word change. Give the Redis factories a distinct ID:
@Override
public String getId() {
return "redis";
}With a unique dedup key, both factories survive ProviderManager.load(), both get asked isSupported(), and exactly one says yes depending on KC_CACHE. That is how the mechanism is meant to work.
Keycloak then refused to start:
ERROR: Failed to start server in (development) mode
ERROR: Failed to serialize object: org.keycloak.models.cache.redis.entities.CachedRealmRole
ERROR: org.keycloak.models.cache.redis.DefaultLazyLoaderActivating a code path that had been dead since it was written exposed the second bug immediately.
Why the entities could not go on the wire
CachedRealm, CachedUser, CachedClient, CachedRealmRole and friends hold lazily-loaded fields through DefaultLazyLoader:
public class DefaultLazyLoader<S, D> implements LazyLoader<S, D> {
private final Function<S, D> loader;
private final Supplier<D> fallback;
private volatile D data;
...
}Those Function and Supplier fields are populated with method references and lambdas at construction time, things like OAuth2DeviceConfig::new or realm -> realm.getDefaultClientScopesStream(...). Java lambdas are not Serializable unless explicitly declared as such, and these are not.
This was never a problem upstream because Infinispan does not use Java-native serialization for these entities. Our LettuceCacheAdapter wrote through ObjectOutputStream, hit the first lambda field, and threw.
We considered three ways to make the entities serializable and rejected all three. Adding @ProtoField annotations across roughly thirty entity classes is a large mechanical refactor with real regression risk. Making the lambda fields transient and null-tolerant means a deserialized entity cannot reload its lazy data, trading a startup crash for silent stale reads. Force-eager-loading before every put defeats the point of lazy loading and pushes the requirement onto every call site.
Then we asked a better question: why are these entities going into Redis in the first place?
The actual fix: stop storing what you never needed to store
Look at what Infinispan does with these caches. It does not replicate them. Realm, user, and authorization config are read-mostly, and PostgreSQL is the source of truth. Each node keeps a local cache, loads from JPA on a miss, and listens for invalidation events from its peers.
We already had both halves of that: a Caffeine L1 from iteration 2, and an L1InvalidationBus on Redis pub/sub from the same iteration. The only missing piece was a way to keep the L1 and the invalidation channel while opting out of L2 storage entirely.
That piece is NoOpRedisCache, about 65 lines of which most are the comment explaining why it exists. Reads return null, writes do nothing, nothing is ever serialized. Routing picks it per cache name:
private static final Set<String> L1_ONLY_PREFIXES = Set.of(
"realms", "realmRevisions",
"users", "userRevisions",
"authorization", "authorizationRevisions",
"keys", "crl"
);When L1RedisCache misses and asks its L2 delegate, the no-op returns null. Keycloak’s cache manager sees the miss, loads from JPA, and calls put to populate the L1. Writes on any pod publish to kc:l1:invalidate and peers evict. That is Infinispan local cache behavior, without Infinispan, and the serialization cascade becomes irrelevant because the entities never touch the wire.
With the getId change safe to keep, the numbers moved sharply. Single-pod mean response time went from 69 ms to 23 ms, landing within 21% of vanilla Keycloak, and the three-pod mean roughly halved from 199 ms to 108 ms. Full detail is in the iteration 6 and iteration 7 write-ups, and the head-to-head against embedded Infinispan is in the benchmark post.
Making sure it stays fixed
A comment saying “do not change this back” is not a control. Two tests are.
The first pins the IDs directly, so reverting any factory to "default" fails the build:
@Test
public void redisCacheRealmProviderFactory_idIsRedis_notDefault() {
assertThat(new RedisCacheRealmProviderFactory().getId(), equalTo("redis"));
assertThat(new RedisCacheRealmProviderFactory().getId(), not(equalTo("default")));
}The second guards the mirror image of the bug, and we wrote it because we hit that one too. RedisProviderParityTest walks every registered Spi on the classpath and asserts that any SPI with an enabled factory under KC_CACHE=infinispan also has one under KC_CACHE=redis. If an SPI’s only implementation is an Infinispan factory that our own guard switches off, then session.getProvider(...) returns null and the feature breaks at runtime instead of at boot. That is what briefly took down external identity provider brokering for us: the sole PublicKeyStorageProvider was the Infinispan one, disabled under redis, with nothing to replace it.
Sweeping every SPI rather than a hand-written list matters, because it means an SPI added upstream later cannot arrive unnoticed with no provider under our configuration. Legitimate exceptions go in an EXPECTED_ABSENT_UNDER_REDIS set with a written reason, which turns each one into a reviewed decision instead of a null pointer.
Both bugs are the same shape: provider resolution silently produced a different answer than the configuration implied. One dropped our factory, the other dropped theirs.
What upstream could change
None of this is a defect in Keycloak. The dedup behavior is deliberate and the isInternal tiebreaker exists specifically so custom extensions can override built-ins. But three small changes would have saved us the trip:
- Log the collision above DEBUG. Two factories claiming the same
(spi, id)pair is close to always a mistake. At INFO or WARN, this bug prints its own diagnosis on the first boot. - Document the ordering. The SPI documentation covers
isSupported()andorder(), but not that dedup runs before either. The rule worth stating plainly is: a provider ID must be unique per SPI, andisSupported()cannot arbitrate between factories that share one. - Say that
getId()is an identity, not a label. It is natural to read"default"as “this is my default configuration.” It is actually a registry key in a namespace shared with every other implementation of that SPI.
We are drafting these as an upstream documentation contribution.
Frequently asked questions
Why is my custom Keycloak provider not being used?
The most common cause is a getId() collision with a built-in factory for the same SPI. Keycloak dedupes on spi-name + "-" + provider-id, and if your factory loses the tiebreak it is discarded before isSupported() runs. Start Keycloak with --log-level=DEBUG and search the output for “Found multiple provider factories of same provider ID.”
Does isSupported() control whether my provider is selected?
Only among factories that survive deduplication. It decides whether a factory is enabled, not which of two same-ID factories wins. If you need conditional activation, you also need a unique provider ID.
How do I force my factory to override a built-in one?
Either return a higher value from order(), or keep your factory outside the org.keycloak package namespace so Keycloak’s internal-versus-external tiebreak favors it. Deliberately sharing an ID to override a built-in works, but only when one of those tiebreakers is decisively in your favor.
Can I see which provider Keycloak actually chose?
Yes. The admin console lists the resolved providers for every SPI on the Provider info tab of the dashboard, and --log-level=DEBUG prints the loaded factories at startup. Both are worth checking after any change to an extension’s ID.
Do passing tests mean my provider is active?
Not necessarily. Unit tests that instantiate your factory directly will pass whether or not the running server ever loads it. Assert on runtime behavior that only your implementation could produce, such as a counter that increments only on your code path.
The lesson we actually took
The bug was one string. The reason it survived was that every check we had confirmed the system worked, and none confirmed it worked the way we configured it. Benchmarks measure outcomes, and an outcome produced by the wrong component looks identical to the right one when the wrong component happens to be fast.
Metrics broke the tie, and not because we were looking for a correctness problem. A counter at zero is a claim about the world. It is worth believing before you assume the counter is broken.
Locke is Apache 2.0 and the full iteration history, including the parts that did not work, is public at github.com/sky-cloak/locke. If you are writing Keycloak extensions of your own, our custom SPI development guide covers the basics, and now you know one thing it does not.
If you would rather not debug provider resolution at all, Skycloak runs managed Keycloak so you do not have to.
