Sunday, February 9, 2014

Tablespace Growth History and Forecast for 10g and 11g

For 10g and 11g database growth history and forecast see script Database Growth History and Forecast for 10g and 11g.
For 12c and above; Tablespace and Database growth history and forecast, please use following scripts
Tablespace Growth History and Forecast for 12c and Above.
Database Growth History and Forecast for 12c and Above.

Starting Oracle 10G, Oracle records tablespaces usage (allocated, used etc.) in AWR which can be retrieved by querying the data dictionary view dba_hist_tbspc_space_usage. Following scripts can be used to view the history of tablespace(s) usage and predict the expected growth for the future. Growth forecast is based on daily growth in the past.

Things to note:
1) This script is based on AWR. If your AWR retention period is 7 days, this script can only tell the growth history of last 7 days and predict based on last 7 days growth. I would recommend to change AWR retention to at least 35 days - this will also be more helpful in case of performance tuning situation as you will have a longer window from the past to look into for performance comparisons.
2) You may edit this scrip according to your requirement to forecast for a period  which suites your requirements. By default it will predict expected growth for next 30, 60 and 90 days.
3) Save this code in an sql script.
4) Log in as user SYS on SQLPLUS and execute the script or copy and paste the following code. You will be prompted for the tablespace name

Script for Single Tablespace

##############################################
set serverout on
set verify off
set lines 200
set pages 2000
DECLARE
v_ts_id number;
not_in_awr EXCEPTION;
v_ts_name varchar2(200) := UPPER('&Tablespace_Name');
v_ts_block_size number;
v_begin_snap_id number;
v_end_snap_id number;
v_begin_snap_date date;
v_end_snap_date date;
v_numdays number;
v_ts_begin_size number;
v_ts_end_size number;
v_ts_growth number;
v_count number;
v_ts_begin_allocated_space number;
v_ts_end_allocated_space number;
BEGIN
SELECT ts# into v_ts_id FROM v$tablespace where name = v_ts_name;
SELECT count(*) INTO v_count FROM dba_hist_tbspc_space_usage where tablespace_id=v_ts_id;
IF v_count = 0 THEN 
RAISE not_in_awr;
END IF ;
SELECT block_size into v_ts_block_size FROM dba_tablespaces where tablespace_name = v_ts_name;
SELECT min(snap_id), max(snap_id), min(trunc(to_date(rtime,'MM/DD/YYYY HH24:MI:SS'))), max(trunc(to_date(rtime,'MM/DD/YYYY HH24:MI:SS')))
into v_begin_snap_id,v_end_snap_id, v_begin_snap_date, v_end_snap_date from dba_hist_tbspc_space_usage where tablespace_id=v_ts_id;
v_numdays := v_end_snap_date - v_begin_snap_date;

SELECT round(max(tablespace_size)*v_ts_block_size/1024/1024,2) into v_ts_begin_allocated_space from dba_hist_tbspc_space_usage where tablespace_id=v_ts_id and snap_id = v_begin_snap_id;
SELECT round(max(tablespace_size)*v_ts_block_size/1024/1024,2) into v_ts_end_allocated_space from dba_hist_tbspc_space_usage where tablespace_id=v_ts_id and snap_id = v_end_snap_id;
SELECT round(max(tablespace_usedsize)*v_ts_block_size/1024/1024,2) into v_ts_begin_size from dba_hist_tbspc_space_usage where tablespace_id=v_ts_id and snap_id = v_begin_snap_id;
SELECT round(max(tablespace_usedsize)*v_ts_block_size/1024/1024,2) into v_ts_end_size from dba_hist_tbspc_space_usage where tablespace_id=v_ts_id and snap_id = v_end_snap_id;
v_ts_growth := v_ts_end_size - v_ts_begin_size;
DBMS_OUTPUT.PUT_LINE(CHR(10));
DBMS_OUTPUT.PUT_LINE('Tablespace Block Size: '||v_ts_block_size);
DBMS_OUTPUT.PUT_LINE('---------------------------');
DBMS_OUTPUT.PUT_LINE(CHR(10));
DBMS_OUTPUT.PUT_LINE('Summary');
DBMS_OUTPUT.PUT_LINE('========');
DBMS_OUTPUT.PUT_LINE('1) Allocated Space: '||v_ts_end_allocated_space||' MB'||' ('||round(v_ts_end_allocated_space/1024,2)||' GB)');
DBMS_OUTPUT.PUT_LINE('2) Used Space: '||v_ts_end_size||' MB'||' ('||round(v_ts_end_size/1024,2)||' GB)');
DBMS_OUTPUT.PUT_LINE('3) Used Space Percentage: '||round(v_ts_end_size/v_ts_end_allocated_space*100,2)||' %');
DBMS_OUTPUT.PUT_LINE(CHR(10));
DBMS_OUTPUT.PUT_LINE('History');
DBMS_OUTPUT.PUT_LINE('========');
DBMS_OUTPUT.PUT_LINE('1) Allocated Space on '||v_begin_snap_date||': '||v_ts_begin_allocated_space||' MB'||' ('||round(v_ts_begin_allocated_space/1024,2)||' GB)');
DBMS_OUTPUT.PUT_LINE('2) Current Allocated Space on '||v_end_snap_date||': '||v_ts_end_allocated_space||' MB'||' ('||round(v_ts_end_allocated_space/1024,2)||' GB)');
DBMS_OUTPUT.PUT_LINE('3) Used Space on '||v_begin_snap_date||': '||v_ts_begin_size||' MB'||' ('||round(v_ts_begin_size/1024,2)||' GB)' );
DBMS_OUTPUT.PUT_LINE('4) Current Used Space on '||v_end_snap_date||': '||v_ts_end_size||' MB'||' ('||round(v_ts_end_size/1024,2)||' GB)' );
DBMS_OUTPUT.PUT_LINE('5) Total growth during last '||v_numdays||' days between '||v_begin_snap_date||' and '||v_end_snap_date||': '||v_ts_growth||' MB'||' ('||round(v_ts_growth/1024,2)||' GB)');
IF (v_ts_growth <= 0 OR v_numdays <= 0) THEN
DBMS_OUTPUT.PUT_LINE(CHR(10));
DBMS_OUTPUT.PUT_LINE('!!! NO DATA GROWTH WAS FOUND FOR TABLESPCE '||V_TS_NAME||' !!!');
ELSE
DBMS_OUTPUT.PUT_LINE('6) Per day growth during last '||v_numdays||' days: '||round(v_ts_growth/v_numdays,2)||' MB'||' ('||round((v_ts_growth/v_numdays)/1024,2)||' GB)');
DBMS_OUTPUT.PUT_LINE(CHR(10));
DBMS_OUTPUT.PUT_LINE('Expected Growth');
DBMS_OUTPUT.PUT_LINE('===============');
DBMS_OUTPUT.PUT_LINE('1) Expected growth for next 30 days: '|| round((v_ts_growth/v_numdays)*30,2)||' MB'||' ('||round(((v_ts_growth/v_numdays)*30)/1024,2)||' GB)');
DBMS_OUTPUT.PUT_LINE('2) Expected growth for next 60 days: '|| round((v_ts_growth/v_numdays)*60,2)||' MB'||' ('||round(((v_ts_growth/v_numdays)*60)/1024,2)||' GB)');
DBMS_OUTPUT.PUT_LINE('3) Expected growth for next 90 days: '|| round((v_ts_growth/v_numdays)*90,2)||' MB'||' ('||round(((v_ts_growth/v_numdays)*90)/1024,2)||' GB)');
END IF;

EXCEPTION
WHEN NO_DATA_FOUND THEN
DBMS_OUTPUT.PUT_LINE(CHR(10));
DBMS_OUTPUT.PUT_LINE('!!! TABLESPACE DOES NOT EXIST !!!');
WHEN NOT_IN_AWR THEN
DBMS_OUTPUT.PUT_LINE(CHR(10));
DBMS_OUTPUT.PUT_LINE('!!! TABLESPACE USAGE INFORMATION NOT FOUND IN AWR !!!');

END;
/
##############################################

Sample Output

Enter value for tablespace_name: TEST


Tablespace Block Size: 8192
---------------------------


Summary
========
1) Allocated Space: 2048 MB (2 GB)
2) Used Space: 1558.44 MB (1.52 GB)
3) Used Space Percentage: 76.1 %


History
========
1) Allocated Space on 06-DEC-14: 2048 MB (2 GB)
2) Current Allocated Space on 10-JAN-15: 2048 MB (2 GB)
3) Used Space on 06-DEC-14: 1273 MB (1.24 GB)
4) Current Used Space on 10-JAN-15: 1558.44 MB (1.52 GB)
5) Total growth during last 35 days between 06-DEC-14 and 10-JAN-15: 285.44 MB (.28 GB)
6) Per day growth during last 35 days: 8.16 MB (.01 GB)


Expected Growth
===============
1) Expected growth for next 30 days: 244.66 MB (.24 GB)
2) Expected growth for next 60 days: 489.33 MB (.48 GB)
3) Expected growth for next 90 days: 733.99 MB (.72 GB)

PL/SQL procedure successfully completed.


/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/\/

Script for All Tablespace (Except UNDO and TEMP)
set serverout on
set verify off
set lines 200
set pages 2000
DECLARE
v_ts_id number;
not_in_awr EXCEPTION;
v_ts_block_size number;
v_begin_snap_id number;
v_end_snap_id number;
v_begin_snap_date date;
v_end_snap_date date;
v_numdays number;
v_ts_begin_size number;
v_ts_end_size number;
v_ts_growth number;
v_count number;
v_ts_begin_allocated_space number;
v_ts_end_allocated_space number;
cursor v_cur is select tablespace_name from dba_tablespaces where contents='PERMANENT';

BEGIN
FOR v_rec in v_cur
LOOP
BEGIN
SELECT ts# into v_ts_id FROM v$tablespace where name = v_rec.tablespace_name;
SELECT count(*) INTO v_count FROM dba_hist_tbspc_space_usage where tablespace_id=v_ts_id;
IF v_count = 0 THEN 
RAISE not_in_awr;
END IF ;
SELECT block_size into v_ts_block_size FROM dba_tablespaces where tablespace_name = v_rec.tablespace_name;
SELECT min(snap_id), max(snap_id), min(trunc(to_date(rtime,'MM/DD/YYYY HH24:MI:SS'))), max(trunc(to_date(rtime,'MM/DD/YYYY HH24:MI:SS')))
into v_begin_snap_id,v_end_snap_id, v_begin_snap_date, v_end_snap_date from dba_hist_tbspc_space_usage where tablespace_id=v_ts_id;
v_numdays := v_end_snap_date - v_begin_snap_date;

SELECT round(max(tablespace_size)*v_ts_block_size/1024/1024,2) into v_ts_begin_allocated_space from dba_hist_tbspc_space_usage where tablespace_id=v_ts_id and snap_id = v_begin_snap_id;
SELECT round(max(tablespace_size)*v_ts_block_size/1024/1024,2) into v_ts_end_allocated_space from dba_hist_tbspc_space_usage where tablespace_id=v_ts_id and snap_id = v_end_snap_id;
SELECT round(max(tablespace_usedsize)*v_ts_block_size/1024/1024,2) into v_ts_begin_size from dba_hist_tbspc_space_usage where tablespace_id=v_ts_id and snap_id = v_begin_snap_id;
SELECT round(max(tablespace_usedsize)*v_ts_block_size/1024/1024,2) into v_ts_end_size from dba_hist_tbspc_space_usage where tablespace_id=v_ts_id and snap_id = v_end_snap_id;
v_ts_growth := v_ts_end_size - v_ts_begin_size;
DBMS_OUTPUT.PUT_LINE(CHR(10));
DBMS_OUTPUT.PUT_LINE(v_rec.tablespace_name||' Tablespace');
DBMS_OUTPUT.PUT_LINE('--------------------');
DBMS_OUTPUT.PUT_LINE('Tablespace Block Size: '||v_ts_block_size);
DBMS_OUTPUT.PUT_LINE('---------------------------');
DBMS_OUTPUT.PUT_LINE(CHR(10));
DBMS_OUTPUT.PUT_LINE('Summary');
DBMS_OUTPUT.PUT_LINE('========');
DBMS_OUTPUT.PUT_LINE('1) Allocated Space: '||v_ts_end_allocated_space||' MB'||' ('||round(v_ts_end_allocated_space/1024,2)||' GB)');
DBMS_OUTPUT.PUT_LINE('2) Used Space: '||v_ts_end_size||' MB'||' ('||round(v_ts_end_size/1024,2)||' GB)');
DBMS_OUTPUT.PUT_LINE('3) Used Space Percentage: '||round(v_ts_end_size/v_ts_end_allocated_space*100,2)||' %');
DBMS_OUTPUT.PUT_LINE(CHR(10));
DBMS_OUTPUT.PUT_LINE('History');
DBMS_OUTPUT.PUT_LINE('========');
DBMS_OUTPUT.PUT_LINE('1) Allocated Space on '||v_begin_snap_date||': '||v_ts_begin_allocated_space||' MB'||' ('||round(v_ts_begin_allocated_space/1024,2)||' GB)');
DBMS_OUTPUT.PUT_LINE('2) Current Allocated Space on '||v_end_snap_date||': '||v_ts_end_allocated_space||' MB'||' ('||round(v_ts_end_allocated_space/1024,2)||' GB)');
DBMS_OUTPUT.PUT_LINE('3) Used Space on '||v_begin_snap_date||': '||v_ts_begin_size||' MB'||' ('||round(v_ts_begin_size/1024,2)||' GB)' );
DBMS_OUTPUT.PUT_LINE('4) Current Used Space on '||v_end_snap_date||': '||v_ts_end_size||' MB'||' ('||round(v_ts_end_size/1024,2)||' GB)' );
DBMS_OUTPUT.PUT_LINE('5) Total growth during last '||v_numdays||' days between '||v_begin_snap_date||' and '||v_end_snap_date||': '||v_ts_growth||' MB'||' ('||round(v_ts_growth/1024,2)||' GB)');
IF (v_ts_growth <= 0 OR v_numdays <= 0) THEN
DBMS_OUTPUT.PUT_LINE(CHR(10));
DBMS_OUTPUT.PUT_LINE('!!! NO DATA GROWTH WAS FOUND FOR TABLESPACE '||v_rec.tablespace_name||' !!!');
ELSE
DBMS_OUTPUT.PUT_LINE('6) Per day growth during last '||v_numdays||' days: '||round(v_ts_growth/v_numdays,2)||' MB'||' ('||round((v_ts_growth/v_numdays)/1024,2)||' GB)');
DBMS_OUTPUT.PUT_LINE(CHR(10));
DBMS_OUTPUT.PUT_LINE('Expected Growth');
DBMS_OUTPUT.PUT_LINE('===============');
DBMS_OUTPUT.PUT_LINE('1) Expected growth for next 30 days: '|| round((v_ts_growth/v_numdays)*30,2)||' MB'||' ('||round(((v_ts_growth/v_numdays)*30)/1024,2)||' GB)');
DBMS_OUTPUT.PUT_LINE('2) Expected growth for next 60 days: '|| round((v_ts_growth/v_numdays)*60,2)||' MB'||' ('||round(((v_ts_growth/v_numdays)*60)/1024,2)||' GB)');
DBMS_OUTPUT.PUT_LINE('3) Expected growth for next 90 days: '|| round((v_ts_growth/v_numdays)*90,2)||' MB'||' ('||round(((v_ts_growth/v_numdays)*90)/1024,2)||' GB)');
END IF;
DBMS_OUTPUT.PUT_LINE('/\/\/\/\/\/\/\/\/\/\/\/ END \/\/\/\/\/\/\/\/\/\/\/\');

EXCEPTION
WHEN NOT_IN_AWR THEN
DBMS_OUTPUT.PUT_LINE(CHR(10));
DBMS_OUTPUT.PUT_LINE(v_rec.tablespace_name||' Tablespace');
DBMS_OUTPUT.PUT_LINE('--------------------');
DBMS_OUTPUT.PUT_LINE('Tablespace Block Size: '||v_ts_block_size);
DBMS_OUTPUT.PUT_LINE('---------------------------');
DBMS_OUTPUT.PUT_LINE(CHR(10));
DBMS_OUTPUT.PUT_LINE('!!! TABLESPACE USAGE INFORMATION NOT FOUND IN AWR !!!');
DBMS_OUTPUT.PUT_LINE('/\/\/\/\/\/\/\/\/\/\/\/ END \/\/\/\/\/\/\/\/\/\/\/\');
NULL;
END;
END LOOP;
END;
/

Please give your feedback if you face any difficulty in executing these scripts or if these scripts do not work for you.

Thursday, February 6, 2014

Error "[NS-20802] Grid Infrastructure Configuration Failed" During 11.2 RAC installation on Windows 2008R2

I faced error "[NS-20802] Grid Infrastructure Configuration Failed" during a 2 node Oracle 11.2 RAC installation on Windows 2008 R2 (64 bit). All prerequisites were checked and cluvfy also did not show any problem, but during installation, the installation failed at the following screen (on node1 where installation was initiated.)


A snippet from the installation log is as follows.
---------------------------------------------------------------
INFO: Starting Install on nodes 'hm-cnpb-st286b' 
INFO: Saving Cluster Inventory
INFO: InstallProgressMonitor: Starting phase 512
INFO: Saving Cluster Inventory
INFO: Saving Cluster Inventory
INFO: Running command 'c:\temp\OraInstall2014-01-06_02-40-12PM\oui\bin\platform\win64\setup.exe -jreLoc c:\temp\OraInstall2014-01-06_02-40-12PM\jdk/jre -paramFile c:\temp\OraInstall2014-01-06_02-40-12PM\oui/clusterparam.ini  -silent -ignoreSysPrereqs -attachHome -noClusterEnabled ORACLE_HOME=D:\oracle\11203\grid ORACLE_HOME_NAME=OraCrs11g_home1 CLUSTER_NODES=hm-cnpb-st286a,hm-cnpb-st286b "INVENTORY_LOCATION=C:\Program Files\Oracle\Inventory" LOCAL_NODE=hm-cnpb-st286b -remoteInvocation -invokingNodeName hm-cnpb-st286a -logFilePath "C:\Program Files\Oracle\Inventory/logs" -timestamp 2014-01-06_02-40-12PM' on the nodes 'hm-cnpb-st286b'.
INFO: Deleting service 'OracleOUIOraCrs11g_home1Service' on nodes 'hm-cnpb-st286b'.
INFO: Creating service 'OracleOUIOraCrs11g_home1Service' on nodes 'hm-cnpb-st286b'.
INFO: Starting service 'OracleOUIOraCrs11g_home1Service' on nodes 'hm-cnpb-st286b'.
INFO: Stopping service OracleOUIOraCrs11g_home1Service on nodes hm-cnpb-st286b.
INFO: Deleting service 'OracleOUIOraCrs11g_home1Service' on nodes 'hm-cnpb-st286b'.
INFO: InstallProgressMonitor: Completed phase 512
INFO: Checkpoint:Failed Checkpoint found returning it for getAllFailedCheckPoints.
INFO: Checkpoint:Failed Checkpoint found returning null for getLastFailedCheckPoint.
INFO: Checkpoint:Index file written and updated
INFO: Checkpoint:Transfer of file done from remote node
INFO: Updating the global context
INFO: Path To 'globalcontext.xml' = D:\oracle\11203\grid\install\chainedInstall\globalcontext
INFO: Since operation was successful, move the current OiicAPISessionDetails to installed list
INFO: Number of scripts to be executed as root user = 0
INFO: isSuccessfullInstallation: true
INFO: isSuccessfullRemoteInstallation: true
INFO: Adding ExitStatus SUCCESS to the exit status set
INFO: Completed setting up crssetup
INFO: Performing Configuration
INFO: Entering ConfigClient.init method
INFO: ConfigClient.init sOracleHomeLoc=D:\oracle\11203\grid, sTopAggId=oracle.crs
INFO: Entering ConfigClient.initClientAndSession method
INFO: ConfigClient.initClientAndSession status : connected to the config framework
INFO: ConfigClient.initClientAndSession session begun oracle.sysman.emCfg.client.CfwSession@560c9447
INFO: ConfigClient.initClientAndSession m_oAggregate=[Loracle.sysman.emCfg.common.IAggregate;@53daed73
INFO: Configuration log directory - D:\oracle\11203\grid\cfgtoollogs
INFO: Exiting ConfigClient.init method
INFO: Calling event ConfigSessionInitialized
INFO: ConfigClient.registerOutErrStream method called
INFO: Updating Transient Parameters in the Aggregates
INFO: Updating Transient Parameters for oracle.crs
INFO: Updating Transient Parameters for oracle.assistants.netca.client
INFO: Updating Transient Parameters for OuiConfigVariables
INFO: Updating Transient Parameters for oracle.assistants.asm
INFO: Setting value to transient variable S_ASMMONITORPASSWORD
INFO: Setting value to transient variable S_ASMPASSWORD
INFO: Setting value to transient variable s_asmcaArgs
INFO: Setting value to transient variable s_ASMPasswordFlag
INFO: Setting value to transient variable s_ASMMonitorPasswordFlag
INFO: Successfully updated Transient Parameters in the Aggregates
INFO: Entering ConfigClient.getToolsList method
INFO: Created config job for Grid Infrastructure Configuration
INFO: Created config job for Update Inventory
INFO: Created config job for Oracle Net Configuration Assistant
INFO: Created config job for Automatic Storage Management Configuration Assistant
INFO: Created config job for Oracle Cluster Verification Utility
INFO: Configuration log directory - D:\oracle\11203\grid\cfgtoollogs
INFO: 
 The Runconfig command constructed is D:\oracle\11203\grid\oui\bin\runConfig.bat ORACLE_HOME=D:\oracle\11203\grid MODE=perform ACTION=configure RERUN=true $*
INFO: Created a new file D:\oracle\11203\grid\cfgtoollogs\configToolAllCommands
INFO: Since the option is to overwrite the existing D:\oracle\11203\grid\cfgtoollogs\configToolAllCommands file, backing it up
INFO: The backed up file name is D:\oracle\11203\grid\cfgtoollogs\configToolAllCommands.bak
INFO: Entering ConfigClient.executeToolsInAggregate method
INFO: ConfigClient.executeToolsInAggregate oAggregate=oracle.crs:oracle.crs:11.2.0.3.0:common
INFO: Entering ConfigClient.assignAction method
INFO: ConfigClient.assignAction actionref=[Loracle.sysman.emCfg.common.IActionReference;@353d9906
INFO: ConfigClient.assignAction action is of type :configuration
INFO: Exiting ConfigClient.assignAction method
INFO: ConfigClient.executeToolsInAggregate action assigned
INFO: Started Plugin named: Grid Infrastructure Configuration
INFO: Found associated job
INFO: Starting 'Grid Infrastructure Configuration'
INFO: Starting 'Grid Infrastructure Configuration'
INFO: 2014-01-06 14:55:16: Checking for super user privileges
INFO: 2014-01-06 14:55:16: superUser=$app-hkdl-oracle-svc groupName=Administrators
INFO: 2014-01-06 14:55:16: domain=APAC user=$APP-HKDL-ORACLE-SVC
INFO: 2014-01-06 14:55:16: D:\oracle\11203\grid\bin\crssetup.exe getsystem
INFO: 2014-01-06 14:55:16: Executing cmd: D:\oracle\11203\grid\bin\crssetup.exe getsystem
WARNING: Parameter ORACLE_OWNER not defined at D:\oracle\11203\grid\crs\install/crsconfig_lib.pm line 12044.
INFO: 2014-01-06 14:55:16: Command output:
INFO: >  SYSTEM 
INFO: >End Command output
INFO: 2014-01-06 14:55:16: User has $app-hkdl-oracle-svc privileges
INFO: Using configuration parameter file: D:\oracle\11203\grid\crs\install\crsconfig_params
WARNING: 
INFO: 
INFO: Completed Plugin named: Grid Infrastructure Configuration
INFO: Grid Infrastructure Configuration failed.
INFO: Grid Infrastructure Configuration failed.
INFO: ConfigClient.executeToolsInAggregate action performed
INFO: Exiting ConfigClient.executeToolsInAggregate method
INFO: Calling event ConfigToolsExecuted
INFO: 
 The Runconfig command constructed is D:\oracle\11203\grid\oui\bin\runConfig.bat ORACLE_HOME=D:\oracle\11203\grid MODE=perform ACTION=configure RERUN=false $*
INFO: Created a new file D:\oracle\11203\grid\cfgtoollogs\configToolFailedCommands
INFO: Since the option is to overwrite the existing D:\oracle\11203\grid\cfgtoollogs\configToolFailedCommands file, backing it up
INFO: The backed up file name is D:\oracle\11203\grid\cfgtoollogs\configToolFailedCommands.bak
WARNING: readme.txt file doesn't exits
INFO: ConfigClient.saveSession method called
INFO: Calling event ConfigSessionEnding
INFO: ConfigClient.endSession method called
INFO: Completed Configuration
INFO: Shutting down OUISetupDriver.JobExecutorThread
INFO: Cleaning up, please wait...
INFO: Dispose the install area control object
INFO: Update the state machine to STATE_CLEAN
INFO: All forked task are completed at state setup
INFO: Completed background operations
INFO: Moved to state <setup>
--------------------------------------------------------------------------

As can be seen above, there wasn't any ERROR message in the installation log files (installActions<Date_Time>.log) on either of the nodes, but, there was a WARNING "WARNING: Parameter ORACLE_OWNER not defined at D:\oracle\11203\grid\crs\install/crsconfig_lib.pm line 12044" in the log file of node 1 (please see above log snippet). Even though there wasn't any error message on node2 log file, but there was a message (INFO: Checkpoint:Failed Checkpoint found returning it for getAllFailedCheckPoints) which was not much clear.  

Based on the WARNING message on node 1, I had a hint that there could be something wrong with the OS user which is performing this installation

Solution
Actually the OS user name which I was using for this installation (domain user which was part of local Administrator group) was starting with a dollar sign i.e "$MY_USER_NAME" and this was the actual problem. Once we created a new OS user without $ sign as first letter, all went fine and I was able to perform a successful installation.
I have experienced same error being returned if username starts with a number.

Note
Remember; that in my case, $ sign was first character of the OS user I was using. If you have $ sign or any other special character or a number as first character or anywhere in the OS user name and you face this problem, you can consider creating a new user without any special character or a number as the first character of the username.
I contacted Oracle support but they also did not mentioned any restriction of this type, and until now (while I am writing this post), there is no official document available at MOS which explains this restriction of OS user name.
A similar type of issue is discussed here where OS username started with a number.