Configuring and using ACE relay servers

When using Apache ACE in a large deployment scenario, in which many targets need to be provisioned, it might be necessary to scale the number of servers with additional relay servers. How many targets can be handled by a single server depends a bit on the number of changes that are made to the metadata, and how often each target contacts the ACE server to get the latest changes. Just to give an impression of what Apache ACE can handle, we tested a single ACE server, during the development of ACE, with several hundred of targets, which caused not real problems apart from a significant load on your disk and CPUs. If you come into a situation in which a single ACE server is not able to cope with the load of your targets, or you want to replicate your metadata across several servers to achieve better availability, you can use this document to set up and configure relay servers.

Overview

In ACE, each target only talks to a single ACE server at a time. From this perspective, an ACE server only serves two major tasks:

1. receiving and storing log information from targets, and;
2. providing deployment packages containing software updates.

In situations where lots of targets need to be provisioned with software, having only a single ACE server can impose major scalability and availability issues. To solve this, ACE can be set up to act as relay server for an upstream (or master) ACE server. This allows a tree-topology of ACE servers to be created that can distribute the load of many targets evenly, as denoted in the following image:

Figure 1: An example server topology, showing multiple servers relaying information from an upstream server. Many other topologies are possible as well.

Note that target logs need to be propagated to the main ACE server in order to make this information available for further inspection. For example, a target operator might want to know the current status of a target, even if this target is not directly served by the main ACE server.

An ACE relay server is a stripped down ACE server that only includes the bare minimum functionality to serve its targets. In addition to accepting log-updates from targets and provisioning software updates to its targets, it relays log information it received from its targets to the upstream ACE server and replicates (at least) the deployment repository from the upstream ACE server for its own use. A schematic overview of the functional blocks in a relay server is shown in figure 2:

Figure 2: A functional overview of how a relay server functions.

In the following sections, each of these functional blocks will be explained in more depth along with how to configure them. The configuration examples are shown as simple key-value pairs separated with an equals (=) sign. Lines starting with a hash (#) are considered comments.

Log synchronisation between target and relay

Like in a single-server scenario, the relay must allow a target to upload its log before it can be synchronised with the main ACE server.

To configure the storage of logs from targets on the relay server, we need to instantiate a log store by supplying the following configuration to the org.apache.ace.log.server.store.factory managed service factory:

# a symbolic name used to reference to this store from other services
name=auditlog

This will instantiate a new log store service named auditlog¹ that can be used to store the log information of targets.

To allow targets to upload their logs to the relay server, a log-servlet needs to be instantiated. This is done by supplying the following configuration to the org.apache.ace.log.server.servlet.factory managed service factory:

# the symbolic name of the actual log store to store the uploaded logs in
name=auditlog
# the alias/endpoint at which the servlet is registered
org.apache.ace.server.servlet.endpoint=/auditlog
# whether or not to use authentication
authentication.enabled=false

This will instantiate a servlet that listens on the /auditlog² endpoint (for example, http://my.relay.server:8080/auditlog). The given name configuration key is used to lookup the actual log store we've created previously.

Log synchronisation between relay and main server

With the log store and servlet configured and in place, a target is now able to synchronise its logs with the relay server. However, once uploaded to the relay server, the logs will not propagate automatically to the main ACE server. A separate log-synchronisation task is responsible for this. To enable this task, we need to supply the following configuration to the org.apache.ace.log.server.task.factory managed service factory:

# the symbolic name of the actual log store to synchronise
name=auditlog
# how to synchronise logs, can be 'pull', 'push' or 'pushpull'
mode=push

This will create a "task" service that will push all log information from the auditlog¹ store to the main server. The mode configuration key denotes how to synchronise with the main server:

• push synchronises (or "pushes") the logs from the relay server to the main server and is typically used in a relaying situation as described in this document;
• pull synchronises (or "pulls") the logs from the main server to the relay server and is typically used to create secondary backup servers and is not further discussed in this document;
• pushpull will perform a two-way synchronisation between relay and main server and can be used to support fail-over situations.

In addition, the log synchronisation task (and other tasks as well, see below) needs to know what upstream server it should synchronise with. For this information, it uses the ACE discovery service, which is configured by supplying, for example, the following configuration to the org.apache.discovery.property³ managed service:

# what is the URL to the *main* ACE server
serverURL = http://my.main.server:8080

Deployment provider

The relay needs to have at least a deployment repository⁴. This repository is a verbatim copy of the deployment repository on the master server and as such needs to be replicated periodically. The configuration of the deployment provider consists of three parts: a repository store, the repository servlet, and a repository provider.

To create a "slave" deployment repository⁵ store, the following configuration needs to be supplied to the org.apache.ace.server.repository.factory managed service factory:

# the symbolic name of the repository, should be "deployment"
name=deployment
# the customer name, should be equal to the customer name used on the master server
customer=apache
# indicates that this is a slave/read-only repository
master=false

To make the repository store accessible through a servlet, we need to supply the following configuration to the org.apache.ace.repository.servlet.RepositoryServlet managed service:

# the endpoint on which the deployment repository store is accessible
org.apache.ace.server.servlet.endpoint=/repository
# whether or not to enable authentication for this endpoint
authentication.enabled = false

This will instantiate a servlet that listens on the /repository endpoint (for example, http://my.relay.server:8080/repository).

With the repository store and servlet configured, we can configure the deployment repository provider, which is used to collect information about deployment artefacts that should be deployed on a target. The deployment repository provider is configured by supplying the following configuration to the org.apache.ace.deployment.provider.repositorybased managed service:

# the URL on which the deployment repository store can be accessed
url = http://my.relay.server:8080/repository
# the symbolic name of the repository, should be "deployment"
name = deployment
# the customer name, should be equal to the customer name used on the master server
customer = apache

Deployment servlet

The target is only interested in downloading deployment packages containing the updates for its software and/or its management agent. To support this, two servlets need to be instantiated, one for accessing the deployment packages of the target software, and one for accessing the deployment packages for the management agent.

To configure the servlet responsible for providing deployment packages of the target software, we need to supply the following configuration to the org.apache.ace.deployment.servlet managed service:

# the endpoint on which the deployment servlet for software-updates is accessible
org.apache.ace.server.servlet.endpoint=/deployment
# whether or not to enable authentication for this endpoint
authentication.enabled = false

To configure the servlet responsible for providing agent updates, we need to supply the following configuration to the org.apache.ace.deployment.servlet.agent managed service:

# the endpoint on which the deployment servlet for agent-updates is accessible
org.apache.ace.server.servlet.endpoint=/agent
# whether or not to enable authentication for this endpoint
authentication.enabled = false
#  the OBR used for retrieving the agent software
obr.url = http://my.obr.server:8080/obr/

Task scheduler

With the synchronisation and repository replication tasks in place, we need to tell the ACE scheduler to periodically execute these tasks. This is done by supplying the following configuration to the org.apache.ace.scheduler.cfg managed service:

# execute all LogSyncTasks once every 2 seconds...
org.apache.ace.log.server.task.LogSyncTask=2000
# Synchronise with the master repository every 5 seconds...  
org.apache.ace.repository.task.RepositoryReplicationTask=5000

The relay server is now completely configured and ready to serve its targets.

A complete runnable example can be found in the run-relay project of the ACE source repository. This example project starts a relay server on localhost:8282 and expects its upstream server to run at localhost:8080 A list of all required bundles to run a relay server can be found in the relay.bndrun file.

Target configuration

To only thing that a target needs to know about the relay server is its (base) URL. This can be supplied when starting the management agent on the target by adding the following argument to its command line:

-Dagent.discovery.serverurls=http://my.relay.server:8080/

In fact, the agent.discovery.serverurls can take multiple server URLs (separated by commas) to support a simple form of fail-over, like http://first.relay.server:8080/,http://second.relay.server:8080.

Notes