HA - Reports and Log monitoring

One of the most critical part of the HA is the reports. Possibly the most under-valued , yet the most powerful tool are reports. Reports can be run from various parts of the system. Equally , if not more , important are the logs of various components in ITIM / TAM which make up for the system, and should be monitored very closely. The logs and reports generally provide a heads-up , and system health check to understand and estimate any possible memory leaks / performance issues or malicious attack on the server or the system. When we talk about making the system HA with no / scheduled pre-determined downtime, we need to understand that we still need to bring down each component of the system one by one , repair it ( fixes /maintenance ) and then take down the other server.
What is the best time for maintenance :
If the majority of user base is corporate then weekend might be considered a good option to consider, but its here where the reports come to play in. Monitoring the request logs in webseal can help you estimate the peak time and the traffic strength , essential for determining maintenance window.
Reports : Linux reports can be downloaded directly from server logs , /var/logs directory or if the financial condition allows , we can go forward and install any of the major reporting tools  Tivoli Monitoring (ITCAM) / Prognosis / Wiley / iiAgent are few popular you would like to consider. ITCAM stands as my personal choice , for its ease in configuring.
If you have installed ITIM , then reports can be generated from Console homepage itself ( given your admin has given you access to the reports). If you are admin , navigate to Reports --> data synchronization .  Click on Run Syncronization Now. Run the reports. Refresh synchronization status. Once you have done with sync, you can generate the reports. Make sure you have orphan reports produced before anything else 
Another important aspect is the webseal Log monitoring , LDAP monitoring ( both ITIM / ITAM).
Webseal log is divided into request logs . agent logs and referrer logs. All these logs are sufficient to pinpoint the intrusion. Regular study of logs is required to keep the system healthy. LDAP logs enable to keep intact the enteries , making sure that errors are not encountered.
DB2 logs can be monitored using TDS web interface, even though not essential on day to day basis, but one should  keep a look at them frequently, just to make sure all the processes are running fine.
In next article i will be writing the technical details where the logs  are , and common things inside the log one should be looking for.

HA - Illusion or realization

So lets say this new hyper-phobia of making the system HA catches you. You go, hire a few professionals to make the system HA, how do you test if your system is a true HA or not. Well , lets get it straight first time, IBM TIM is made up of java classes, has a LDAP server , a DB2 , a directory server and TDI in its most simplest of installations. You would be adding up with TAM combo adapter , which means webseal, policy server, authorization server as well. When you talk about HA , you want to have a backup of most of these components.
Few of these are critical , like mission critical, which if fail will stop the entire functioning of the system. Identification of these is purely dependent on the business logic, and functioning of the system. If your system runs the recon once daily / weekly, and then most of the transactions happen over ITIM server in itself , you might want to replicate your LDAP / Directory Server and DB2 ( HADR) , while TDI can be taken care of later. One has to understand, the team can go and have all the components. If you have a  huge number of requests coming from Webseal , and need to authorize using ITIM, then you will need to make the TDI HA. Similarly , multiple Webseals might be needed for heavy traffic streamlining , but the policy and authorization server can be up only one at a time. If you don't have daily changing ACLs , then this piece does not need to be HA. But if you have a TAI++ configuration set up, you can't afford for this piece to go down either. Regardless of the thought, lets say you have a HA setup with two LDAP (master -master) , two directory server, two DB2 ( HADR) , and two TDI servers as well. How would be start testing. What is the most basic test one can perform.
The first and most basic test is ,if you can , turn off the primary servers, one component at time , and keep on running transactions. By turning off one of the servers the hard way , you know if the system is truly HA. Once this step is complete, then starts the alternate shutting down of services , and restarting them while hitting the tests, and watching the logs .
One of the most important things while doing the HA is to look after the logs, they help you determine if the request is routed to the correct side. The LDAP should essentially be master -master , and more the entries, more should be the number of LDAP servers. Regular backup of LDAP servers is an essential tool, which will help to troubleshoot any unforeseen activity.
Getting the system to HA is not impossible in the books, but one should understand the fact, ITIM is essentially an EAR file deployed on application server, which will have memory leaks. Any code ,custom added will add to the complexity, and the irregular code would make it worse. Additionally DB2 sitting as the backbone should be properly configured, and steps as mentioned in Performance tuning guide be strictly followed.
I will try to push out the basic HA steps as well , but to understand HA is something which can be achieved, and sustained, but it comes with nice hardware costs , and understanding that it might be HA, it still can fail. There are too many pieces which can cause it stop functioning.