High Availablity (HA) - Aftermath

The HA has its own consequences. Higher server costs, memory costs, additional support system, regular updates, intensive monitoring and regular updates. This is not sufficient to end the cause. It will be a practical thought to have a HA system in place , along with disaster recovery only considering there is a possibility of one of the servers failing or hard disk crash. The various scenarios where you might want to have HA are considering:

1.       Corruption of Policy Server files / Authorization / ACLD servers

2.       Corruption of LDAP data

3.       Accidental Fat finger at root of one of the components removing critical files

4.       Corruption of itim data file

5.       Corruption of DB2 server

6.       Crash of DB2 primary server

7.       Crashing WAS / corruption of security.xml file

8.       Physical crash of any hard disk

9.       Natural calamity (like fire / floods ) rendering the entire data center useless

In all of the above situations, the HA, and HADR must be able to ensure business continuity, with little or no impact. Once we have encountered such a situation, and we pat our backs for winning out of the situation, we are now faced with a very difficult situation. The one which will become much more critical as the data and customization grows. Once you have come out of disasters in the work environment, you have the bigger challenge then HA itself. There are different parts which needs attention. Two of the most critical parts are DB2 and LDAP.  DB2 backups should be more regular and pushed outside the datacenter. Similar with LDAP backups. The ldap backups should be extensive, and should be couple with V3.schema file. There are other files which need to be backed up as indicated in the installation guide.

Once a few servers or data center is lost, we should be making very critical decisions. What will be the primary data center, which will be primary server, will the secondary / backup servers take the responsibility of the primary servers or one intend to build out a new primary server. This will be really an interesting choice and you would like to delay your decision on following topics:

1.       Was the secondary server at par with primary server?

2.       Did you have restricted features in failover server than in primary server

3.       Is the performance of secondary server lower than primary

4.       Is the new server build will have better configuration?

5.       The time to build the new servers from scratch (assuming you have the backups configuration for all the servers) so that you can be up in business quickly.

Assuming you gather all the information and decisions on time next is the implementation of these.  As I always say, the backup process should be bottom up. A copy of all the servers is maintained if possible, like getting a snapshot of initial configuration. Once the snapshot is taken, it acts as the baseline for future.  The entire LDAP should be delicately and very carefully taken in account, considering it has all the information you need in future. One important thing while taking the ldap backup is to make sure you have hashed out the password, sealed it out and then encrypt it. The encryption should be very private and out of hands of anyone, to ensure the security of data. This key will be needed if you have to rebuild the identical primary server as crashed. Once you have ensured you have ldap schema, keys and ldap entries. Make sure you back up all domains as well. For TAM LDAP make sure you have backed up secauthority=default, and o=company, dc=com as well.  All the workflows are saved up to this point, but all the customized java code still resides in the data folder, and should be backed up after each release. The data continuity can be maintained from secondary working server as well. Make sure that the new server is cryptographically in sync.

Another thing which you want to save and have a copy of, are the keystores and pass stores. This will be useful for you to have at least in one of the cases, I am going to tell. The certificates will be used for two ways SSL. You can use the certificates from secondary server, but one should make a detailed list of all the servers with SSL locations, and where all servers are defined, and where all clients are installed. The policy server if rebuilt has its own default certificates, which will need to go out, and update with the new one, and you will need to run bassconfig to establish the new connection. Make sure svrsslconfig and pdjrte are done correctly the first time. If the servers were running as non-root, and you need to re-build the servers, make sure to back up the startup scripts as well. It is a small thing but will help you once your system is up and roaring.

Re-Configuring TDI pdjrte with Policy server

One of the most painful things is to have pdjrte installed incorrectly for the first time. If everything is installed as per needs and correctly in one time success , the journey is sweet. If not , or just in case you have to make any change , its more easy to unconfigure and then configure the system back. Trying to get one component, then second and so on will make you lose the mission critical time. To configure the pdjrte with already configured policy server be:
[root@tdiserver sbin]# ./pdjrtecfg -action status
HPDBF0031E   This Java Runtime Environment has already been configured.
[root@tdiserver sbin]# ./pdjrtecfg -action name
Access Manager Runtime for Java
[root@tdiserver sbin]# ./pdjrtecfg -action unconfig
Unconfiguration of:
Access Manager Runtime for JavajvmHome: //V7.0/jvm/jre
 is in progress.  This might take several minutes.


Unconfiguration of: Access Manager Runtime for Java
completed successfully.

[root@tdiserver sbin]# ./pdjrtecfg -action config -interactive
Specify the full path of the Java Runtime Environment (JRE)
to configure for Tivoli Access Manager [//V7.0/jvm/jre]://V7.0/jvm/jre/

Enter 'full' or 'standalone' for the configuration type [full]:

Policy server host name [tdiserver]: policyserverfullname.company.com

Tivoli Access Manager policy server port number [7135]:

Enter the Access Manager policy server domain [null]:

The domain contains invalid characters.

Enter the Access Manager policy server domain [null]: null

Tivoli Common Directory logging is currently configured.
You may enable this application to use Tivoli Common Directory logging
using the currently configured directory for log files.


Do you want to use Tivoli Common Directory logging (y/n) [n]? y

Log files for this application will be created
in directory: /var/ibm/tivoli/common

Configuration of Access Manager Runtime for Java is in progress.
This might take several minutes.
Configuration of Access Manager Runtime for Java completed successfully.
[root@tdiserver sbin]#
--- 
This completes the pdjrte re-configuration to prevent any mishaps and make sure it works correctly

Adding Chain Certificate to Webseal

To add a certificate to the webseal server is clearly defined in the webseal administration guide. What I faced a big challenge was to add a chain certificate. I had no clue what is the process or how should I go about doing it.
To start with , chain certificates are no different than regular certificates, just they are more in number and follow a particular pattern. There is a root certificate, and a signer one. There can also be an intermediate signer, or more levels as the security would demand.
The root certificate goes in first , then intermediate signer and then the signer. Before you start with adding the certificate, make sure you have the following things with you :
1. All root and signer certs
2. Location of keystore and truststore , with the passwords for both
3. Permission to run the commands and add certificates to the server
4. The location of webseal pdsrv file .

Many of the people use iKeyman or gsk7cmd kit , you can use anyone of  those. most People like to use export display to add the certificates , but my system was responding too slow for display that i used command line interface ( my first choice of use as well) .

Adding chain certificates to webseal server::
Exact commands as run on the server.
--------------------
[root@servername /]# export JAVA_HOME=/opt/IBM/WebSphere/AppServer/java/jre/
[root@servername/]# export PATH=$JAVA_HOME/bin:$PATH
[root@servername/]# gsk7cmd -cert -list -db /var/pdweb/www-default/certs/pdsrv.kdb -pw  
Certificates in database /var/pdweb/www-default/certs/pdsrv.kdb:
   WebSEAL-Test-Only
   RSA Secure Server Certification Authority
   Thawte Personal Basic CA
   Thawte Personal Freemail CA
   Thawte Personal Premium CA
   Thawte Premium Server CA
   Thawte Server CA
   Verisign Class 1 Public Primary Certification Authority
[root@servername/]# gsk7cmd -cert -add -db /var/pdweb/www-default/certs/pdsrv.kdb -pw  -file /tmp/RootCertificate.cer -label WEbsealRootCertificate
[root@servername/]# gsk7cmd -cert -add -db /var/pdweb/www-default/certs/pdsrv.kdb -pw   -file /tmp/Root/CertificateFile.cer -label websealChaincertificate
--------
One of the very critical commands here is listing the already existing certificates. This step would enable you to determine whether you are hitting the right keystore, have correct creds and possibly the correct permission to go for the operation. 


This is specific to webseal , but can be repeated over all of Tivoli identity manager , websphere application server or any server.

HA - Reports and Log monitoring

One of the most critical part of the HA is the reports. Possibly the most under-valued , yet the most powerful tool are reports. Reports can be run from various parts of the system. Equally , if not more , important are the logs of various components in ITIM / TAM which make up for the system, and should be monitored very closely. The logs and reports generally provide a heads-up , and system health check to understand and estimate any possible memory leaks / performance issues or malicious attack on the server or the system. When we talk about making the system HA with no / scheduled pre-determined downtime, we need to understand that we still need to bring down each component of the system one by one , repair it ( fixes /maintenance ) and then take down the other server.
What is the best time for maintenance :
If the majority of user base is corporate then weekend might be considered a good option to consider, but its here where the reports come to play in. Monitoring the request logs in webseal can help you estimate the peak time and the traffic strength , essential for determining maintenance window.
Reports : Linux reports can be downloaded directly from server logs , /var/logs directory or if the financial condition allows , we can go forward and install any of the major reporting tools  Tivoli Monitoring (ITCAM) / Prognosis / Wiley / iiAgent are few popular you would like to consider. ITCAM stands as my personal choice , for its ease in configuring.
If you have installed ITIM , then reports can be generated from Console homepage itself ( given your admin has given you access to the reports). If you are admin , navigate to Reports --> data synchronization .  Click on Run Syncronization Now. Run the reports. Refresh synchronization status. Once you have done with sync, you can generate the reports. Make sure you have orphan reports produced before anything else 
Another important aspect is the webseal Log monitoring , LDAP monitoring ( both ITIM / ITAM).
Webseal log is divided into request logs . agent logs and referrer logs. All these logs are sufficient to pinpoint the intrusion. Regular study of logs is required to keep the system healthy. LDAP logs enable to keep intact the enteries , making sure that errors are not encountered.
DB2 logs can be monitored using TDS web interface, even though not essential on day to day basis, but one should  keep a look at them frequently, just to make sure all the processes are running fine.
In next article i will be writing the technical details where the logs  are , and common things inside the log one should be looking for.