Chapter 6. Cloning Subsystems. Red Hat 8.0 Install Guide

Red Hat Certificate System 8.0 is a software solution for managing digital certificates. This software can be used to create, issue, and manage certificates for servers, users, routers, and other subsystems. This software also allows you to archive and recover keys, and checking the status of a certificate.

PDF Download

Document

Chapter 6. Cloning Subsystems

When a new subsystem instance is first configured, the Red Hat Certificate System allows subsystems to be cloned, or duplicated, for high availability of the Certificate System. The cloned instances run on different machines to avoid a single point of failure and their databases are synchronized through replication.

6.1. About Cloning

Planning for high availability reduces unplanned outages and other problems by making one or more subsystem clones available. When a host machine goes down, the cloned subsystems can handle requests and perform services, taking over from the master (original) subsystem seamlessly and keeping uninterrupted service.

Using cloned subsystems also allows systems to be taken offline for repair, troubleshooting, or other administrative tasks without interrupting the services of the overall PKI system.

NOTE

All of the subsystems except the TPS and RA can be cloned.

Cloning is one method of providing scalability to the PKI by assigning the same task, such as handling certificate requests, to separate instances on different machines. The internal databases for the master and its clones are replicated between each other, so the information about certificate requests or archived keys on one subsystem is available on all the others.

Typically, master and cloned instances are installed on different machines, and those machines are placed behind a load balancer. The load balancer accepts HTTP and HTTPS requests made to the

Certificate System subsystems and directs those requests appropriately between the master and cloned instances. In the event that one machine fails, the load balancer transparently redirects all requests to the machine that is still running until the other machine is brought back online.

Red Hat Certificate System 8.0 Install Guide

Figure 6.1. Cloning Example

The load balancer in front of a Certificate System subsystem is what provides the actual failover support in a high availability system. A load balancer can also provide the following advantages as part of a

Certificate System subsystem:

DNS round-robin, a feature for managing network congestion that distributes load across several different servers.

Sticky SSL, which makes it possible for a user returning to the system to be routed the same host used previously.

6.1.1. Cloning for CAs

Cloned instances have the exact same private keys as the master, so their certificates are identical. For

CAs, that means that the CA signing certificates are identical for the original master CA and its cloned

CAs. From the perspectives of clients, these look like a single CA.

Every CA, both cloned and master, can issue certificates and process revocation requests.

The main issue with managing cloned CAs is how to assign serial numbers to the certificates they issue.

Different CAs can have different levels of traffic, using serial numbers at different rates, and it is imperative that no CA issue the certificates with the same serial number. These serial number ranges are assigned and managed dynamically by using a shared, replicated entry that defines the ranges for

Chapter 6. Cloning Subsystems each CA and the next available range to reassign when one CA range runs low.

Cloned CAs do have limits on what operations they can perform. Most important, cloned CAs cannot generate or publish CRLs. Any CRL requests submitted to a cloned CA are immediately redirected to the master CA. Anything related to generating or caching CRLs is disabled in the CS.cfg file for the clone.

Clones can revoke, display, import, and download CRLs previously generated by master CAs, but having them generate new CRLs may cause synchronization problems. Only a single CA should generate

CRLs, and this task is always left to the master CA, which also maintains the CRL cache.

Master CAs also manage the relationships and information sharing between the cloned CAs by monitoring replication changes to the internal databases.

6.1.2. Cloning for DRMs

With DRMs, all keys archived in one DRM are replicated to the internal databases of the other DRMs.

This allows a key recovery to be initiated on any clone DRM, regardless of which DRM the key was originally archived on.

After a key recovery is processed, the record of the recovery is stored in the internal database of all of the cloned DRMs.

Although key recovery can be initiated on any clone, once the recovery is initiated, it must be completed on the same single DRM. This is because a recovery operation is recorded in the replicated database only after the appropriate number of approvals have been obtained from the DRM agents. Until then, the

DRM on which the recovery is initiated is the only one which knows about the recovery operation.

6.1.3. Cloning for Other Subsystems

There is no real operational difference between masters and clones for TKSs; the information created or maintained on one is replicated along the other servers.

For OCSPs, only the master OCSP receives CRL updates, and then the published CRLs are replicated to the clones.

6.1.4. Cloning and Key Stores

Cloning a subsystem creates two server processes performing the same functions: another new instance of the subsystem is created and configured to use the same keys and certificates to perform its operations. Depending on where the keys are stored for the master clone, the method for the clone to access the keys is very different.

If the keys and certificates are stored in the internal software token, then the keys must be exported from the master subsystem when it is first configured. When configuring the master instance, there is an option in the Export Keys and Certificates panel to back up the keys and certificates to a

PKCS#12 file. Before the clone instance is configured, the PKCS#12 file is copied to the alias/ directory for the clone instance. Then, the PKCS#12 filename is given in the Restore Keys and

Certificates screen during the clone's configuration.

If the keys and certificates are stored on a hardware token, then the keys and certificates can be copied or referenced directly in the token:

Duplicate all the required keys and certificates, except the SSL server key and certificate to the clone instance. Keep the nicknames for those certificates the same. Additionally, copy all the necessary trusted root certificates from the master instance to the clone instance, such as chains or cross-pair certificates.

If the token is network-based, then the keys and certificates simply need to be available to the token;

Red Hat Certificate System 8.0 Install Guide the keys and certificates do not need to be copied.

When using a network-based hardware token, make sure the high-availability feature is enabled on the hardware token to avoid single point of failure.

6.1.5. LDAP Considerations

As mentioned in Section 6.1, “About Cloning”

, part of the behavior of cloning is to replication information between the master and the clone, so that they work from an identical set of data and records. This means that the LDAP servers for the master and clones need to be able to communicate. If these servers are on different hosts, then make sure that there is appropriate firewall access to allow the

Directory Server instances to connect with each other.

NOTE

Cloned subsystems and their masters use separate LDAP servers.

6.2. Exporting Keys from a Software Database

Ideally, the keys for the master instance are exported when the instance is first created. If the keys were not exported then or if the backup file is lost, then it is possible to extract the keys from the internal software database for the subsystem instance using the PKCS12Export command. For example:

PKCS12Export -debug -d /var/lib/subsystem_name/alias -w p12pwd.txt -p internal.txt

-o master.p12

The PKCS#12 file (in this example, master.p12) can then be copied to the clone instance's alias/ directory and imported during the clone configuration.

NOTE

Keys and certificates do not need to be exported from an HSM, so long as the clone instance is installed using the same HSM as the master. If both instances use the same key store, then the keys are naturally available to the clone.

6.3. Cloning a CA

1. Configure the master CA, as described in Section 3.3, “Configuring a CA”

, and back up the keys.

2. In the CS.cfg file for the master CA, enable the master CA to monitor replication database changes by adding the ca.listenToCloneModifications parameter: cd /etc/subsystem_name ca.listenToCloneModifications=true

3. Create the clone subsystem instance.

Chapter 6. Cloning Subsystems

IMPORTANT

Do not go through the setup wizard for the instance yet.

4. Copy the exported PKCS#12 file containing the master instance's keys to the clone's alias/ directory.

The keys for the master instance could have been exported to a .p12 file when the instance was configured. Alternatively, the keys can be exported using the PKCS12Export command, as in

Section 6.2, “Exporting Keys from a Software Database”

5. Make sure the PKCS#12 file is accessible by the Certificate System user. If necessary, change the file owner to pkiuser and reset the permissions to allow the correct read/write access. For example: chown pkiuser:pkiuser example.p12

chmod 00644 example.p12

6. It may be necessary to reset the SELinux permissions for the exported file so that the setup program can use it. First, check what context is assigned: ls -lZ *

-rw-------. pkiuser pkiuser system_u:object_r:pki_ca_var_lib_t:s0 cert8.db

-rw-------. pkiuser pkiuser system_u:object_r:pki_ca_var_lib_t:s0 key3.db

-rw-r--r--. pkiuser pkiuser system_u:object_r:nfs_t:s0 example.p12

-rw-------. pkiuser pkiuser system_u:object_r:pki_ca_var_lib_t:s0 secmod.db

If it does not match the other security files, then reset the SELinux context to that of the other objects using chcon: chcon "system_u:object_r:pki_ca_var_lib_t:s0" example.p12

ls -lZ *

-rw-------. pkiuser pkiuser system_u:object_r:pki_ca_var_lib_t:s0 cert8.db

-rw-------. pkiuser pkiuser system_u:object_r:pki_ca_var_lib_t:s0 key3.db

-rw-r--r--. pkiuser pkiuser system_u:object_r:pki_ca_var_lib_t:s0

example.p12

-rw-------. pkiuser pkiuser system_u:object_r:pki_ca_var_lib_t:s0 secmod.db

7. Open the setup wizard URL, which was returned when the instance was created. For example: http://server.example.com:9180/ca/admin/console/config/login?

pin=HIsd90RJSioDK==

8. In the Security Domain panel, add the clone to the same security domain to which the master belongs.

9. The Subsystem Type panel sets whether to create a new instance or a clone; select the clone radio button.

Red Hat Certificate System 8.0 Install Guide

10. Give the path and filename of the PKCS #12 backup file which was saved when the master

instance was created or that were exported in 4

If the keys are stored on an HSM that is accessible to the clone, then they are picked up automatically.

NOTE

When cloning a CA, the master and clone instances have the same CA signing key.

11. The subsystem information is automatically supplied from the master instance to the clone instance once the keys are successfully restored. Complete the configuration process.

Chapter 6. Cloning Subsystems

NOTE

By default, the instance configuration wizard uses localhost as the location for the internal LDAP database for a new instance. However, with cloning, the configuration process will spin endlessly and never complete if localhost is used for the internal database location, even if the LDAP database is indeed installed on the localhost.

Use the fully-qualified domain name for the LDAP database in the Internal Database panel when configuring a clone.

12. Edit the CS.cfg file for the clone. Certain parameters must be added to the clone configuration to disable caching and generating CRLs.

Disable control of the database maintenance thread: ca.certStatusUpdateInterval=0

Disable monitoring database replication changes: ca.listenToCloneModifications=false

Disable maintenance of the CRL cache: ca.crl.IssuingPointId.enableCRLCache=false

Disable CRL generation: ca.crl.IssuingPointId.enableCRLUpdates=false

Enable the clone to redirect CRL requests to the master clone: master.ca.agent.host=master_hostname master.ca.agent.port=master_port

13. Restart the clone instance.

service subsystem_name restart

After configuring the clone, test to make sure that the master-clone relationship is functioning:

1. Request a certificate from the cloned CA.

2. Approve the request.

3. Download the certificate to the browser.

4. Revoke the certificate.

5. Check master CA's CRL for the revoked certificate. In the master Certificate Manager's agent services page, click Update Certificate Revocation List. Find the CRL in the list.

The CRL should show the certificate revoked by the cloned Certificate Manager. If that certificate is not listed, check logs to resolve the problem.

6.4. Cloning OCSP Subsystems

Red Hat Certificate System 8.0 Install Guide

1. Configure the master OCSP, as described in Section 3.5, “Configuring a DRM, OCSP, or TKS”

, and back up the keys.

2. In the CS.cfg file for the master OCSP, set the

OCSP.Responder.store.defStore.refreshInSec parameter to any non-zero number other than 21600; 21600 is the setting for a clone.

vim /etc/subsystem_name/CS.cfg

OCSP.Responder.store.defStore.refreshInSec=15000

3. Create the clone subsystem instance.

IMPORTANT

Do not go through the setup wizard for the instance yet.

4. Copy the exported PKCS#12 file containing the master instance's keys to the clone's alias/ directory.

The keys for the master instance could have been exported to a .p12 file when the instance was configured. Alternatively, the keys can be exported using the PKCS12Export command, as in

Section 6.2, “Exporting Keys from a Software Database”

chmod 00644 example.p12

6. It may be necessary to reset the SELinux permissions for the exported file so that the setup program can use it. First, check what context is assigned: ls -lZ *

-rw-------. pkiuser pkiuser system_u:object_r:pki_ocsp_var_lib_t:s0 cert8.db

-rw-------. pkiuser pkiuser system_u:object_r:pki_ocsp_var_lib_t:s0 key3.db

-rw-r--r--. pkiuser pkiuser system_u:object_r:nfs_t:s0 example.p12

-rw-------. pkiuser pkiuser system_u:object_r:pki_ocsp_var_lib_t:s0 secmod.db

If it does not match the other security files, then reset the SELinux context to that of the other objects using chcon: chcon "system_u:object_r:pki_ocsp_var_lib_t:s0" example.p12

ls -lZ *

-rw-------. pkiuser pkiuser system_u:object_r:pki_ocsp_var_lib_t:s0 cert8.db

-rw-------. pkiuser pkiuser system_u:object_r:pki_ocsp_var_lib_t:s0 key3.db

-rw-r--r--. pkiuser pkiuser system_u:object_r:pki_ocsp_var_lib_t:s0

example.p12

-rw-------. pkiuser pkiuser system_u:object_r:pki_ocsp_var_lib_t:s0 secmod.db

7. Open the setup wizard URL, which was returned when the instance was created. For example: http://server.example.com:11180/ocsp/admin/console/config/login?

pin=IOjh7fIOjkld90k

Chapter 6. Cloning Subsystems

8. In the Security Domain panel, add the clone to the same security domain to which the master belongs.

9. The Subsystem Type panel sets whether to create a new instance or a clone; select the clone radio button.

10. Give the path and filename of the PKCS #12 backup file which was saved when the master

instance was created or that were exported in 3

If the keys are stored on an HSM that is accessible to the clone, then they are picked up automatically.

11. The subsystem information is automatically supplied from the master instance to the clone instance once the keys are successfully restored. Complete the configuration process.

Red Hat Certificate System 8.0 Install Guide

NOTE

Use the fully-qualified domain name for the LDAP database in the Internal Database panel when configuring a clone.

12. Edit the CS.cfg file for the clone. Set the

OCSP.Responder.store.defStore.refreshInSec parameter in the clone instance to

21600.

vim /etc/subsystem_name/CS.cfg

OCSP.Responder.store.defStore.refreshInSec=21600

13. Restart the clone instance.

service subsystem_name start

After configuring the clone, test to make sure that the master-clone relationship is functioning:

1. Set up OCSP publishing in the master CA so that the CRL is published to the master OCSP.

2. Once the CRL is successfully published, check both the master and cloned OCSP's List

Certificate Authorities link in the agent pages. The list should be identical.

3. Use the OCSPClient tool to submit OCSP requests to the master and the cloned Online

Certificate Status Manager. The tool should receive identical OCSP responses from both managers.

6.5. Cloning DRM and TKS Subsystems

1. Configure the master subsystem, as described in Section 3.5, “Configuring a DRM, OCSP, or

TKS” , and back up the keys.

2. Create the clone subsystem instance.

IMPORTANT

Do not go through the setup wizard for the instance yet.

3. Copy the exported PKCS#12 file containing the master instance's keys to the clone's alias/ directory.

The keys for the master instance could have been exported to a .p12 file when the instance was configured. Alternatively, the keys can be exported using the PKCS12Export command, as in

Section 6.2, “Exporting Keys from a Software Database”

4. Make sure the PKCS#12 file is accessible by the Certificate System user. If necessary, change the file owner to pkiuser and reset the permissions to allow the correct read/write access. For example:

Chapter 6. Cloning Subsystems chown pkiuser:pkiuser example.p12

chmod 00644 example.p12

5. It may be necessary to reset the SELinux permissions for the exported file so that the setup program can use it. First, check what context is assigned: ls -lZ *

-rw-------. pkiuser pkiuser system_u:object_r:pki_tks_var_lib_t:s0 cert8.db

-rw-------. pkiuser pkiuser system_u:object_r:pki_tks_var_lib_t:s0 key3.db

-rw-r--r--. pkiuser pkiuser system_u:object_r:nfs_t:s0 example.p12

-rw-------. pkiuser pkiuser system_u:object_r:pki_tks_var_lib_t:s0 secmod.db

If it does not match the other security files, then reset the SELinux context to that of the other objects using chcon: chcon "system_u:object_r:pki_tks_var_lib_t:s0" example.p12

ls -lZ *

-rw-------. pkiuser pkiuser system_u:object_r:pki_tks_var_lib_t:s0 cert8.db

-rw-------. pkiuser pkiuser system_u:object_r:pki_tks_var_lib_t:s0 key3.db

-rw-r--r--. pkiuser pkiuser system_u:object_r:pki_tks_var_lib_t:s0

example.p12

-rw-------. pkiuser pkiuser system_u:object_r:pki_tks_var_lib_t:s0 secmod.db

6. Open the setup wizard URL, which was returned when the instance was created. For example: http://server.example.com:10180/kra/admin/console/config/login?

pin=Jhu7HBJiodljnw3oijs

7. In the Security Domain panel, add the clone to the same security domain to which the master belongs.

8. The Subsystem Type panel sets whether to create a new instance or a clone; select the clone radio button.

Red Hat Certificate System 8.0 Install Guide

9. Give the path and filename of the PKCS #12 backup file which was saved when the master

instance was created or that were exported in 3

If the keys are stored on an HSM that is accessible to the clone, then they are picked up automatically.

NOTE

When cloning a DRM, the master and clone instances have the same storage and transport keys.

10. The subsystem information is automatically supplied from the master instance to the clone instance once the keys are successfully restored. Complete the configuration process.

Chapter 6. Cloning Subsystems

NOTE

Use the fully-qualified domain name for the LDAP database in the Internal Database panel when configuring a clone.

11. Restart the clone instance.

service subsystem_name restart

For the DRM clone, test to make sure that the master-clone relationship is functioning:

1. Go to the DRM agent's page.

2. Click List Requests.

3. Select Show all requests for the request type and status.

4. Click Submit.

5. Compare the results from the cloned DRM and the master DRM. The results ought to be identical.

For the TKS, enroll a smart card and then run an ldapsearch to make sure that the same key information is contained in both databases.

6.6. Converting Masters and Clones

There can be any number of clones, but there can only be a single configured master. For DRMs and

TKSs, there is no configuration difference between masters and clones, but CAs and OCSPs do have some configuration differences. This means that when a master is taken offline — because of a failure or for maintenance or to change the function of the subsystem in the PKI — then the existing master must be reconfigured to be a clone, and one of the clones promoted to be the master.

6.6.1. Converting CA Clones and Masters

1. Stop the master CA if it is still running.

2. Open the existing master CA configuration directory: cd /var/lib/pki-ca/conf

3. Edit the CS.cfg file for the master, and change the CRL and maintenance thread settings so that it is set as a clone:

Disable control of the database maintenance thread: ca.certStatusUpdateInterval=0

Disable monitoring database replication changes: ca.listenToCloneModifications=false

Disable maintenance of the CRL cache:

Red Hat Certificate System 8.0 Install Guide ca.crl.IssuingPointId.enableCRLCache=false

Disable CRL generation: ca.crl.IssuingPointId.enableCRLUpdates=false

Set the CA to redirect CRL requests to the new master: master.ca.agent.host=new_master_hostname master.ca.agent.port=new_master_port

4. Stop the cloned CA server.

service subsystem_name stop

5. Open the cloned CA's configuration directory.

cd /etc/subsystem_name

6. Edit the CS.cfg file to configure the clone as the new master.

a. Delete each line which begins with the ca.crl. prefix.

b. Copy each line beginning with the ca.crl. prefix from the former master CA CS.cfg file into the cloned CA's CS.cfg file.

c. Enable control of the database maintenance thread; the default value for a master CA is

600.

ca.certStatusUpdateInterval=600 d. Enable monitoring database replication: ca.listenToCloneModifications=true e. Enable maintenance of the CRL cache: ca.crl.IssuingPointId.enableCRLCache=true f. Enable CRL generation: ca.crl.IssuingPointId.enableCRLUpdates=true g. Disable the redirect settings for CRL generation requests: master.ca.agent.host=hostname master.ca.agent.port=port number

7. Start the new master CA server.

service subsystem_name start

6.6.2. Converting OCSP Clones

Chapter 6. Cloning Subsystems

1. Stop the OCSP master, if it is still running.

2. Open the existing master OCSP configuration directory.

cd /etc/subsystem_name

3. Edit the CS.cfg, and reset the OCSP.Responder.store.defStore.refreshInSec parameter to 21600:

OCSP.Responder.store.defStore.refreshInSec=21600

4. Stop the online cloned OCSP server.

service subsystem_name stop

5. Open the cloned OCSP responder's configuration directory.

cd /etc/subsystem_name

6. Open the CS.cfg file, and delete the OCSP.Responder.store.defStore.refreshInSec parameter or change its value to any non-zero number:

OCSP.Responder.store.defStore.refreshInSec=15000

7. Start the new master OCSP responder server.

service subsystem_name start

6.7. Updating CA Clones

When a CA is cloned, any configuration in its CS.cfg is also copied to the clone CA. This includes any

DRMs which are configured for the CA to use for key archival. However, if a DRM is configured for a master CA after a clone is created, then the new DRM configuration must be copied over to the clone

CAs manually.

To update any clone CAs with new DRM configuration in the master CA:

1. Stop the clone CA.

service subsystem_name stop

Always stop a subsystem instance before editing its configuration files.

2. Copy all of the ca.connecter.KRA.* parameters for the new DRM connection in the CS.cfg for the master CA over to the clone CA CS.cfg file.

3. Restart the clone CA.

service subsystem_name restart

Key features

Certificate management
Key recovery
Certificate status checking
Scalable and customizable
Based on open standards

Frequently asked questions

Red Hat Certificate System 8.0 can manage a wide variety of certificates, including certificates for servers, users, routers, other subsystems, and file or object signing.

Red Hat Certificate System 8.0 has several subsystems that work together to manage certificates. These subsystems include the Certificate Manager, Data Recovery Manager, Online Certificate Status Manager, and Registration Authority.

Red Hat Certificate System 8.0 is supported on Red Hat Enterprise Linux 5.3 (x86, 32-bit), Red Hat Enterprise Linux 5.3 (x86_64, 64-bit), Microsoft Windows Vista 32-bit, Microsoft Windows Vista 64-bit, Microsoft Windows XP 32-bit, Microsoft Windows XP 64-bit, and Apple Mac OS X 10.5.x (Leopard).