Sunday, August 31, 2014

TSM error while running RMAN Backup



TSM error while running RMAN Backup :

RMAN>
RMAN> 2> 3> 4> 5> 6> 7> 8> 9>
starting full resync of recovery catalog
full resync complete
allocated channel: t1
channel t1: sid=826 devtype=SBT_TAPE
channel t1: Data Protection for Oracle: version 5.4.1.0

Starting backup at 02-SEP-11
channel t1: starting incremental level 0 datafile backupset
channel t1: specifying datafile(s) in backupset
input datafile fno=00009 name=/dbfs/oradata/ababil/data/ababil_ts01.dbf
input datafile fno=00008 name=/indexfs/oradata/ababil/index/ababil_index_ts01.dbf
input datafile fno=00005 name=/dbfs/oradata/ababil/data/ababil_ts02.dbf
input datafile fno=00007 name=/datafs/oradata/ababil/data/ababil_image_ts01.DBF
input datafile fno=00004 name=/datafs/audit_ts/ababil/audit_ts01.dbf
input datafile fno=00002 name=/undofs/oradata/ababil/undo/ababil_undo_ts01.dbf
input datafile fno=00003 name=/datafs/oradata/ababil/system/sysaux01.dbf
input datafile fno=00001 name=/datafs/oradata/ababil/system/system01.dbf
channel t1: starting piece 1 at 02-SEP-11
released channel: t1
RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03009: failure of backup command on t1 channel at 09/02/2011 14:48:27
ORA-19502: write error on file "ksmlhtmq_1_1", blockno 13305345 (blocksize=512)
ORA-27030: skgfwrt: sbtwrite2 returned error
ORA-19511: Error received from media manager layer, error text:
   ANS1312E (RC12)   Server media mount not possible




Solution :

Add  a  new  tape to the  Library  Tape ….






Add Tape Drive using Tivoli :


Go to : D:/Tivoli/tsm/baclient

1)    Label  Libvolume  tsmlib  search=bulk  checkin=SCRATCH  labelsource=BARCODE



2)    Query request   (Request ID get at the first stage of the line , like 101)
3)    Query Process
4)    Query  libvolume


To check error  during  backup :

D:\tivoli\tsm\baclient > dsmadmc  -consolemode

Saturday, August 30, 2014

RMAN>Restore rman from Dataguard ASM to Non-asm Environment (Part 1)!

Link : Restore RMAN How To

Restore RAC Dataguard RMAN backup to a new server :

 - Take the RMAN full database backup from Dataguard End.

 - Transfer the backup with the logfile to remote server on the SAME LOCATION .



Step 1 : Create pfile for ISLDB database :

[oracle@PRIM ~]$ cat initisldb.ora
isldb.__oracle_base='/u01/app/oracle'#ORACLE_BASE set from environment
*.compatible='11.2.0.3.0'
*.control_files='/u01/app/oracle/oradata/isldb/control01.ctl','/u01/app/oracle/oradata/isldb/control02.ctl'#Restore Controlfile
*.DB_NAME='isldb'
#_allow_resetlogs_corruption = true
#_no_recovery_through_resetlogs=TRUE



Step 2:

   [oracle@PRIM dbs]$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.3.0 Production on Thu Aug 28 15:22:46 2014

Copyright (c) 1982, 2011, Oracle.  All rights reserved.

Connected to an idle instance.

SQL> startup nomount pfile='initisldb.ora'
ORACLE instance started.

Total System Global Area  329895936 bytes
Fixed Size                  2228024 bytes
Variable Size             255852744 bytes
Database Buffers           67108864 bytes
Redo Buffers                4706304 bytes
SQL>


Step 3 : Restore the Controlfile

[oracle@PRIM dbs]$ rman target /

Recovery Manager: Release 11.2.0.3.0 - Production on Thu Aug 28 15:30:33 2014

Copyright (c) 1982, 2011, Oracle and/or its affiliates.  All rights reserved.

connected to target database: ISLDB (not mounted)

RMAN> restore controlfile from '/cloudfs/rmanback/isl/20140827_172834/controlfile.ctlbk';

Starting restore at 28-AUG-14
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=266 device type=DISK

channel ORA_DISK_1: restoring control file
channel ORA_DISK_1: restore complete, elapsed time: 00:00:01
output file name=/u01/app/oracle/product/11.2.0/db_1/dbs/cntrlisldb.dbf
Finished restore at 28-AUG-14

RMAN> alter database mount;

database mounted
released channel: ORA_DISK_1

RMAN>


Step 4 : Restore all the datafiles :

  1)

run {
set newname for datafile '+DATA/isldg/datafile/system.298.849413815'  to  '/u01/app/oracle/oradata/isldb/system1.dbf' ;
restore datafile 1 ;
}

executing command: SET NEWNAME

Starting restore at 28-AUG-14
using target database control file instead of recovery catalog
allocated channel: ORA_DISK_1
channel ORA_DISK_1: SID=266 device type=DISK

channel ORA_DISK_1: starting datafile backup set restore
channel ORA_DISK_1: specifying datafile(s) to restore from backup set
channel ORA_DISK_1: restoring datafile 00001 to /u01/app/oracle/oradata/isldg/system1.dbf
channel ORA_DISK_1: reading from backup piece /cloudfs/rmanback/isl/20140827_172834/856718916_67_%r.dbf
channel ORA_DISK_1: piece handle=/cloudfs/rmanback/isl/20140827_172834/856718916_67_%r.dbf tag=TAG20140827T172835
channel ORA_DISK_1: restored backup piece 1
channel ORA_DISK_1: restore complete, elapsed time: 00:00:07
Finished restore at 28-AUG-14

RMAN>

2)

run {
set newname for datafile '+DATA/isldg/datafile/sysaux.295.849413813'  to  '/u01/app/oracle/oradata/isldb/sysaux1.dbf' ;
restore datafile 2 ;
}


3)

run {
set newname for datafile '+DATA/isldg/datafile/undotbs1.296.849413813'  to  '/u01/app/oracle/oradata/isldb/undotbs1.dbf' ;
restore datafile 3 ;
}


4)

run {
set newname for datafile '+DATA/isldg/datafile/undotbs2.297.849413815'  to  '/u01/app/oracle/oradata/isldb/undotbs2.dbf' ;
restore datafile 4 ;
}


5)

run {
set newname for datafile '+DATA/isldg/datafile/users.299.849413849'  to  '/u01/app/oracle/oradata/isldb/users1.dbf' ;
restore datafile 5 ;
}


6)

run {
set newname for datafile '+DATA/isldg/datafile/bu_system_tbs.291.849413617'  to '/u01/app/oracle/oradata/isldb/bu_system_tbs1.dbf';
restore datafile 6 ;
}


7)

run {
set newname for datafile '+DATA/isldg/datafile/bu_his_log_tbs.292.849413617'  to '/u01/app/oracle/oradata/isldb/bu_his_log_tbs1.dbf';
restore datafile 7;
}


8)

run {
set newname for datafile '+DATA/isldg/datafile/bu_index_tbs.293.849413617'  to '/u01/app/oracle/oradata/isldb/bu_index_tbs1.dbf';
restore datafile 8 ;
}


9)

run {
set newname for datafile '+DATA/isldg/datafile/bu_image_tbs.294.849413619'  to '/u01/app/oracle/oradata/isldb/bu_image_tbs1.dbf';
restore datafile 9 ;
}


10)

run {
set newname for datafile '+DATA/isldg/tempfile/temp.301.849513611'  to '/u01/app/oracle/oradata/isldb/temp1.dbf';
restore datafile 10 ;
}


Step 5 :

  Run the SWITCH command :

run
{
set newname for datafile '+DATA/isldg/datafile/system.298.849413815'  to  '/u01/app/oracle/oradata/isldb/system1.dbf' ;
set newname for datafile '+DATA/isldg/datafile/sysaux.295.849413813'  to  '/u01/app/oracle/oradata/isldb/sysaux1.dbf' ;
set newname for datafile '+DATA/isldg/datafile/undotbs1.296.849413813'  to  '/u01/app/oracle/oradata/isldb/undotbs1.dbf' ;
set newname for datafile '+DATA/isldg/datafile/undotbs2.297.849413815'  to  '/u01/app/oracle/oradata/isldb/undotbs2.dbf' ;
set newname for datafile '+DATA/isldg/datafile/users.299.849413849'  to  '/u01/app/oracle/oradata/isldb/users1.dbf' ;
set newname for datafile '+DATA/isldg/datafile/bu_system_tbs.291.849413617'  to '/u01/app/oracle/oradata/isldb/bu_system_tbs1.dbf' ;
set newname for datafile '+DATA/isldg/datafile/bu_his_log_tbs.292.849413617'  to '/u01/app/oracle/oradata/isldb/bu_his_log_tbs1.dbf' ;
set newname for datafile '+DATA/isldg/datafile/bu_index_tbs.293.849413617'  to '/u01/app/oracle/oradata/isldb/bu_index_tbs1.dbf' ;
set newname for datafile '+DATA/isldg/datafile/bu_image_tbs.294.849413619'  to '/u01/app/oracle/oradata/isldb/bu_image_tbs1.dbf' ;
set newname for tempfile 1  to  '/u01/app/oracle/oradata/isldb/temp1.dbf' ;
switch datafile all;
switch tempfile all;
}


Step 6 :  Recover the Database :

Error :--------

SQL> recover database until cancel using backup controlfile;
ORA-00279: change 10219039198 generated at 08/27/2014 11:05:14 needed for
thread 1
ORA-00289: suggestion :
/u01/app/oracle/product/11.2.0/db_1/dbs/arch1_1000_839681983.dbf
ORA-00280: change 10219039198 for thread 1 is in sequence #1000


Specify log: {=suggested | filename | AUTO | CANCEL}
AUTO
ORA-00308: cannot open archived log
'/u01/app/oracle/product/11.2.0/db_1/dbs/arch1_1000_839681983.dbf'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3


ORA-00308: cannot open archived log
'/u01/app/oracle/product/11.2.0/db_1/dbs/arch1_1000_839681983.dbf'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3


ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '/u01/app/oracle/oradata/isldg/system1.dbf'


SQL> recover database until cancel using backup controlfile;
ORA-00279: change 10219039198 generated at 08/27/2014 11:05:14 needed for
thread 1
ORA-00289: suggestion :
/u01/app/oracle/product/11.2.0/db_1/dbs/arch1_1000_839681983.dbf
ORA-00280: change 10219039198 for thread 1 is in sequence #1000


Specify log: {=suggested | filename | AUTO | CANCEL}
cancel
ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '/u01/app/oracle/oradata/isldg/system1.dbf'


ORA-01112: media recovery not started


SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '/u01/app/oracle/oradata/isldg/system1.dbf'

-------



Solution :

I manually put the archivelog to '/u01/app/oracle/product/11.2.0/db_1/dbs/thread_1_seq_1000.652.856710149'
directory and rename to is as 'arch1_1000_839681983.dbf' .

After that it checks , 'arch2_771_839681983.dbf' . I put it manually also



 ---
ERROR :
SQL> recover database until cancel using backup controlfile;
ORA-00279: change 10219039198 generated at 08/27/2014 11:05:14 needed for
thread 1
ORA-00289: suggestion :
/u01/app/oracle/product/11.2.0/db_1/dbs/arch1_1000_839681983.dbf
ORA-00280: change 10219039198 for thread 1 is in sequence #1000


Specify log: {=suggested | filename | AUTO | CANCEL}
AUTO
ORA-00279: change 10219039198 generated at 08/27/2014 11:00:42 needed for
thread 2
ORA-00289: suggestion :
/u01/app/oracle/product/11.2.0/db_1/dbs/arch2_771_839681983.dbf
ORA-00280: change 10219039198 for thread 2 is in sequence #771


ORA-00308: cannot open archived log
'/u01/app/oracle/product/11.2.0/db_1/dbs/arch2_771_839681983.dbf'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
Additional information: 3


ORA-01547: warning: RECOVER succeeded but OPEN RESETLOGS would get error below
ORA-01194: file 1 needs more recovery to be consistent
ORA-01110: data file 1: '/u01/app/oracle/oradata/isldg/system1.dbf'

.
.
.
And so on . And After some few logs , make cancel :
-----

SQL>  recover database until cancel using backup controlfile;
ORA-00279: change 10230768222 generated at 08/27/2014 19:01:58 needed for
thread 1
ORA-00289: suggestion :
/u01/app/oracle/product/11.2.0/db_1/dbs/arch1_1002_839681983.dbf
ORA-00280: change 10230768222 for thread 1 is in sequence #1002


Specify log: {=suggested | filename | AUTO | CANCEL}
cancel
Media recovery cancelled.
--


SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-00349: failure obtaining block size for '+red



Step 7 : Open the Database


During open got the below error :

SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-00392: log 2 of thread 2 is being cleared, operation not allowed
ORA-00312: online log 2 thread 2: '+redo'


Sol :

 a) Backup the logfile and recreate it like below :

SQL> ALTER DATABASE BACKUP CONTROLFILE TO TRACE AS '/home/oracle/control.sql' resetlogs ;

Database altered.


 b) Edit the controlfile :

   [oracle@PRIM ~]$ cat control.sql
CREATE CONTROLFILE REUSE DATABASE "ISLDB" RESETLOGS FORCE LOGGING ARCHIVELOG
    MAXLOGFILES 192
    MAXLOGMEMBERS 3
    MAXDATAFILES 1024
    MAXINSTANCES 32
    MAXLOGHISTORY 584
LOGFILE
  GROUP 1 '/u01/app/oracle/oradata/isldg/redo01.log'  SIZE 200M BLOCKSIZE 512,
  GROUP 3 '/u01/app/oracle/oradata/isldg/redo02.log'  SIZE 200M BLOCKSIZE 512,
  GROUP 5 '/u01/app/oracle/oradata/isldg/redo03.log'  SIZE 200M BLOCKSIZE 512
DATAFILE
  '/u01/app/oracle/oradata/isldg/system1.dbf',
  '/u01/app/oracle/oradata/isldg/sysaux1.dbf',
  '/u01/app/oracle/oradata/isldg/undotbs1.dbf',
  '/u01/app/oracle/oradata/isldg/undotbs2.dbf',
  '/u01/app/oracle/oradata/isldg/users1.dbf',
  '/u01/app/oracle/oradata/isldg/bu_system_tbs1.dbf',
  '/u01/app/oracle/oradata/isldg/bu_his_log_tbs1.dbf',
  '/u01/app/oracle/oradata/isldg/bu_index_tbs1.dbf',
  '/u01/app/oracle/oradata/isldg/bu_image_tbs1.dbf'
CHARACTER SET WE8ISO8859P1
;


SQL> shutdown immediate
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.


---
[oracle@PRIM ~]$ vi initisldb.ora
isldb.__oracle_base='/u01/app/oracle'#ORACLE_BASE set from environment
*.compatible='11.2.0.3.0'
*.control_files='/u01/app/oracle/oradata/isldb/control01.ctl','/u01/app/oracle/oradata/isldb/control02.ctl'#Restore Controlfile
*.DB_NAME='isldb'
#_allow_resetlogs_corruption = true
#_no_recovery_through_resetlogs=TRUE
--


SQL> startup nomount pfile='/home/oracle/initisldb.ora'
ORACLE instance started.

Total System Global Area  329895936 bytes
Fixed Size                  2228024 bytes
Variable Size             255852744 bytes
Database Buffers           67108864 bytes
Redo Buffers                4706304 bytes
               4706304 bytes
SQL> @/home/oracle/control.sql

Control file created.

SQL> select status from v$instance;

STATUS
------------
MOUNTED



SQL> alter database open resetlogs;
alter database open resetlogs
*
ERROR at line 1:
ORA-38856: cannot mark instance UNNAMED_INSTANCE_2 (redo thread 2) as enabled




Step 8 :

 Open the database with the "_no_recovery_through_resetlogs=TRUE"

 cat initisldb.ora
isldb.__oracle_base='/u01/app/oracle'#ORACLE_BASE set from environment
*.compatible='11.2.0.3.0'
*.control_files='/u01/app/oracle/oradata/isldb/control01.ctl','/u01/app/oracle/oradata/isldb/control02.ctl'#Restore Controlfile
*.DB_NAME='isldb'
#_allow_resetlogs_corruption = true
_no_recovery_through_resetlogs=TRUE


SQL> shutdown immediate
ORA-01109: database not open


Database dismounted.
ORACLE instance shut down.
SQL> startup pfile='/home/oracle/initisldb.ora'
ORACLE instance started.

Total System Global Area  329895936 bytes
Fixed Size                  2228024 bytes
Variable Size             255852744 bytes
Database Buffers           67108864 bytes
Redo Buffers                4706304 bytes
Database mounted.
ORA-01589: must use RESETLOGS or NORESETLOGS option for database open


SQL> alter database open resetlogs;

Database altered.


Step 9 : Create TEMP File :

SQL>  create temporary tablespace TEMP1  tempfile '/u01/app/oracle/oradata/isldg/temp01.dbf' size 100M;

Tablespace created.

SQL> alter DATABASE DEFAULT TEMPORARY TABLESPACE temp1;

Database altered.

SQL> select name from v$tempfile;

NAME
--------------------------------------------------------------------------------
/u01/app/oracle/oradata/isldg/temp01.dbf


SQL> show parameter spfile

NAME                                 TYPE        VALUE
------------------------------------ ----------- ------------------------------
spfile                               string
SQL> create spfile from pfile='/home/oracle/initisldb.ora';

File created.


Step 10 : RESTART DATABASE :

SQL> shutdown immediate
Database closed.
Database dismounted.
ORACLE instance shut down.
SQL> startup
ORACLE instance started.

Total System Global Area  329895936 bytes
Fixed Size                  2228024 bytes
Variable Size             255852744 bytes
Database Buffers           67108864 bytes
Redo Buffers                4706304 bytes
Database mounted.
Database opened.


Fianlly , Change the required Parameter Like : 'sga_target' ,'log_archive' etc ....






Wednesday, August 13, 2014

RAC/RMAN Troubleshoot Site




Doc ID  

Metalink Notes
>1050908.1 : Troubleshoot Grid Infrastructure Startup Issues
265769.1 - Trobleshooting CRS Reboots
240001.1 - Trobleshooting CRS root.sh problems
341214.1 - How to cleanp after a failed (or sccessfl) Oracle Clsterware installation
294430.1 - MISSCONT Definition and Defalt Vales
357808.1 - CRS Diagnostics
272331.1 - CRS 10g Diagnostic Gide
330358.1 - CRS 10g R2 Diagnostic Collection Gide
331168.1 - Clsterware consolidated logging in 10gR2
357808.1 - Diagnosibility for CRS/EVM/RACG
289690.1 - Data Gathering for Trobleshooting RAC and CRS Isses
284752.1 - Increasing CSS Misscont, Reboottime and Disktimeot
462616.1 - Reconfigring the CSS disktimeot of 10gR2 Clsterware for proper LN failover
317628.1 - How to replace a corrupt OCR mirror file
279793.1 - How to restore a lost voting disk in 10g

Links of RAC Admin :
==============

dbasolution
> jhdba.com
uhess.com
tanelpoder
Arup
tkyte
oracledbabay
halimdba


Links of RMAN Admin :
==============
 puzzling RMAN

RAC TroubleShoot : Cluster Process List

Ques : 
Define the cluster process list when cluster (crsd) is  running or not .

> When cluster is running :

[grid@primoda1 ~]$ ps -ef | grep d.bin
grid      2218     1  0 Jul06 ?        00:14:56 /u01/app/11.2.0.3/grid/bin/tnslsnr LISTENER -inherit
root     15481     1  0 Jun20 ?        03:28:25 /u01/app/11.2.0.3/grid/bin/ohasd.bin reboot
root     15998     1  0 Jun20 ?        01:16:03 /u01/app/11.2.0.3/grid/bin/ologgerd -m primoda2 -r -d /u01/app/11.2.0.3/grid/crf/db/primoda1
root     16018     1  0 Jun20 ?        02:10:04 /u01/app/11.2.0.3/grid/bin/octssd.bin reboot
grid     16130     1  0 Jun20 ?        02:02:20 /u01/app/11.2.0.3/grid/bin/evmd.bin
grid     16275     1  0 Jun20 ?        02:51:56 /u01/app/11.2.0.3/grid/bin/oraagent.bin
grid     16302     1  0 Jun20 ?        00:01:58 /u01/app/11.2.0.3/grid/bin/mdnsd.bin
grid     16452     1  0 Jun20 ?        00:25:09 /u01/app/11.2.0.3/grid/bin/gpnpd.bin
root     16568     1  0 Jun20 ?        06:10:24 /u01/app/11.2.0.3/grid/bin/orarootagent.bin
grid     16571     1  0 Jun20 ?        02:39:16 /u01/app/11.2.0.3/grid/bin/gipcd.bin
root     16586     1  2 Jun20 ?        1-11:29:22 /u01/app/11.2.0.3/grid/bin/osysmond.bin
root     16602     1  0 Jun20 ?        00:43:20 /u01/app/11.2.0.3/grid/bin/cssdmonitor
root     16635     1  0 Jun20 ?        00:42:51 /u01/app/11.2.0.3/grid/bin/cssdagent
root     16880     1  0 Jun20 ?        05:39:25 /u01/app/11.2.0.3/grid/bin/crsd.bin reboot
grid     17019 16130  0 Jun20 ?        00:00:00 /u01/app/11.2.0.3/grid/bin/evmlogger.bin -o /u01/app/11.2.0.3/grid/evm/log/evmlogger.info -l /u01/app/11.2.0.3/grid/evm/log/evmlogger.log
grid     17073     1  0 Jun20 ?        02:11:50 /u01/app/11.2.0.3/grid/bin/oraagent.bin
root     17082     1  0 Jun20 ?        06:05:06 /u01/app/11.2.0.3/grid/bin/orarootagent.bin
grid     17111     1  0 Jun20 ?        12:13:23 /u01/app/11.2.0.3/grid/bin/ocssd.bin
grid     17631     1  0 Jun20 ?        00:06:30 /u01/app/11.2.0.3/grid/bin/tnslsnr LISTENER_SCAN1 -inherit
oracle   17672     1  0 Jun20 ?        06:22:05 /u01/app/11.2.0.3/grid/bin/oraagent.bin
grid     27397 23054  0 12:19 pts/1    00:00:00 grep d.bin
[grid@primoda1 ~]$ ps -ef | grep d.bin | wc -l
21
[grid@primoda1 ~]$


> When cluster is not running :

[oracle@ol5-112-rac1 ~]$ ps -ef | grep d.bin
oracle   30756 30727  0 12:21 pts/2    00:00:00 grep d.bin
[oracle@ol5-112-rac1 ~]$ 



2)  Output of the pmon process where cluster is up :

[oracle@ol5-112-rac2 ~]$ ps -ef | grep pmon
oracle    8085     1  0 Aug13 ?        00:00:00 asm_pmon_+ASM2
oracle   21636     1  0 11:38 ?        00:00:00 ora_pmon_RAC2
oracle   23646 22837  0 12:22 pts/1    00:00:00 grep pmon


See All Hidden parameter in oracle


All hidden parameter as 'SYS' user / SQLNET:

SELECT a.ksppinm Param , b.ksppstvl SessionVal , c.ksppstvl InstanceVal, a.ksppdesc Descr 
FROM x$ksppi a , x$ksppcv b , x$ksppsv c WHERE 
a.indx = b.indx AND a.indx = c.indx AND a.ksppin

To see all SQLNET avaiable parameters :

  sqlnet paramters