Making sense of JBOSS/Wildfly interfaces in domain mode

In this blog we consider JBoss/Wildfly domain mode in the context of the wonderful Keycloak software. It is not necessarily trivial to understand how the interfaces  should be configured, especially if you want to do something other than the defaults, for example to secure your Wildfly/JBOSS configuration, or if you are dealing with a more complex nodes with multiple network interfaces. 

Assumptions:

  • We want to use Keycloak in domain mode. In this operating mode Keycloak application is clustered and also the caches, like user sessions are replicated in the cluster. 

  • We use static discovery for the slaves in the cluster to find their master

  • We use JDBC_PING with tcp transport for the Infinispan cache for the cluster nodes to find each other

  • Our master has the address 192.168.168.253 

  • Our slave has the address 192.168.168.252 

When configuring Keycloak in domain mode, the interfaces can and must be configured in two places. 

In the domain master:

  • domain.xml

  • host-master.xml 

In the slaves

  • domain.xml

  • host-slave.xml 

The differences of defining interfaces in these configuration file is actually not quite clear. You should be able to configure common logical names in domain.xml as general values, and then be more specific in host-master.xml and host-slave.xml to adapt the logical names for the specific needs of a specific cluster node. 

JBoss/Wildfy uses a multitude of ports for it’s cluster, management and service needs. Therefore we need to take these ports into consideration. The most important ports in our case are:

  • 7600/tcp: this is the unicast peer discovery in HA clusters. This port needs to be reachable from the master. You can’t listen on a localhost and expect it to work.

  • 9990/tcp: this is the web management console port. We do not usually want this to be open to the world, but want it to listen on localhost

  • 9999/tcp - This port is used by the Management Console and Management API. The master needs to be able to be connected to this port by the slaves. So we cannot bind this to an address that is not reachable by the master. Localhost will not work. 

When booting up, the slave will try to connect to the master in tcp port 9999. If it cannot connect, it will fail to boot and emit a message:

WFLYHC0001: Could not connect to remote domain controller remote://<master-ip>:9999

With the default configuration files, there are three logical interfaces defined in domain.xml:

interfaces
<interface name="management"/>
<interface name="private">            
<inet-address value="${jboss.bind.address.private:127.0.0.1}"/>        </interface>
<interface name="public"/>
interfaces

two in host-master.xml:

interfaces
<interface name="management">
<inet-address value="${jboss.bind.address.management:127.0.0.1}"/>
 </interface>
 <interface name="public">
 <inet-address value="${jboss.bind.address:127.0.0.1}"/>
 </interface>

The block in host-slave.xml is identical to that in host-master.xml.

The “management” interface is used for all components and services that are required by the management layer (i.e. the HTTP Management Endpoint)
The “public” interface binding is used for any application related network communication (i.e. Web, Messaging, etc). 
The private interface is used for components that are meant to be, well, private. 

There is nothing special about these names; interfaces can be declared with any names. Other sections of the configuration can then reference those interfaces by their logical name, rather than having to include the full details of the interface (which, on servers in a management domain, may vary on different machines).

When you boot the domain master node with something like this: 

domain.sh --host-config=host-master.xml -b 0.0.0.0 -Djboss.http.port=8080

the -b <bind_address> switch to the domain.sh script is a shortcut to bind the public interface. This binds the public-named logical interface to 0.0.0.0.  

With reference to the above blocks, you can express the same thing like this:

domain.sh --host-config=host-master.xml -Djboss.http.port=8080 -Djboss.bind.address=0.0.0.0

Now, what we specifically want to do is:

  • Bind 9999/tcp to an interface that the slaves can connect to to tak e part in the cluster

  • Bind tcp/7600 to an address that other nodes in the infinispan cluster can connect to to replicate caches. This needs to be a routable address. If using JDBP_PINC, like we are here, the address are entered into a database table named jgroupsping

The approach we are going to use here is to first declare the logical interfaces in domain.xml and clear any attributes and properties:

interfaces
  <interface name="management"/>
  <interface name="private"/>
  <interface name="public"/>
</interfaces>

The second thing we want to do is to ensure our socket-binding group, in our case “ha-sockets” is using “public” as it’s default interface. For specific bindings in the group, we override the default interface. Most importantly this means jgroups-tcp binding. The block looks like this:

<socket-binding-group name="ha-sockets" default-interface="public">
<socket-binding name="ajp" port="${jboss.ajp.port:8009}"/>
<socket-binding name="http" port="${jboss.http.port:8080}"/>
<socket-binding name="https" port="${jboss.https.port:8443}"/>
<socket-binding name="jgroups-mping" interface="private" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45700"/>
<socket-binding name="jgroups-tcp" interface="management" port="7600"/>
<socket-binding name="jgroups-tcp-fd" interface="private" port="57600"/>
<socket-binding name="jgroups-udp" interface="private" port="55200" multicast-address="${jboss.default.multicast.address:230.0.0.4}" multicast-port="45688"/>
<socket-binding name="jgroups-udp-fd" interface="private" port="54200"/>
<socket-binding name="modcluster" multicast-address="${jboss.modcluster.multicast.address:224.0.1.105}" multicast-port="23364"/>
<socket-binding name="txn-recovery-environment" port="4712"/>
<socket-binding name="txn-status-manager" port="4713"/>
<outbound-socket-binding name="mail-smtp">
<remote-destination host="localhost" port="25"/>
</outbound-socket-binding>
</socket-binding-group>

Notice that we have used the management interface for jgroups-tcp binding. 

Now that we have the domain.xml out of the way, we can be more specific in host-master.xml and host-slave.xml:

interfaces
<interface name="management">
<inet-address value="${jboss.bind.address.management:127.0.0.1}"/>
</interface>
<interface name="private">
<inet-address value="${jboss.bind.address.private:127.0.0.1}"/>
</interface>
<interface name="public">
<inet-address value="${jboss.bind.address:127.0.0.1}"/>
</interface>
</interfaces>

The inet-address values all have the default of 127.0.0.1, but allows us to specify our own values by using the environment. This interfaces blocks are identical in host-master.xml and host-slave.xml.

When these definitions are in place, we can simply start the master with:

domain.sh --host-config=host-master.xml -b 0.0.0.0 -Djboss.http.port=8080 -Djboss.bind.address.management=192.168.168.253

and the slave with:

domain.sh --host-config=host-slave.xml -b 0.0.0.0 -Djboss.http.port=8080 -Djboss.domain.master.address=192.168.168.253 -Djboss.bind.address.management=192.168.168.252

Of course there is more to make the domain mode work. More on that later. Hopefully this clears some of the common source of confusion. 

We actually automate our keycloak setups and keep them running with puppet, using the exellent puppet-module-keycloak by treydock, to which we also contribute. If you want to know more about how to make Keycloak domain mode work, you can take a look at the code. Should you have any other questions or needs about infrastructure or cloud management with code, feel free to contact us.