27 de jan. de 2010

How to benchmark the performance of the bpbkar process

A good way to determine the theoretical maximum read performance of the bpbkar daemon/process on the client is to run a dev null test. This test is usefully to isolate disk reads performances.

UNIX:
Create: /usr/openv/netbackup/logs/bpbkar directory and the /usr/openv/netbackup/bpbkar_path_tr touch file.

Run:
/usr/openv/netbackup/bin/bpbkar -nocont > /dev/null

example: /usr/openv/netbackup/bin/bpbkar -nocont /home > /dev/null

Note: Running this command will add I/O load to the system.
Before begin test change bpbkar log level to 5

The results can be found in the bpbkar log on the same client that the command was run from. It will give an idea ofhow quickly data is read from the file system being tested.

WINDOWS:
Create: \netbackup\logs\bpbkar folder
Run:
\netbackup\bin\bpbkar32 -nocont > NUL 2>

example: C:\program files\veritas\netbackup\bin\bpbkar32 -nocont E:\ > NUL 2> c:\temp.txt

Note: Running this command will add I/O load to the system.
Before begin test change bpbkar log level to 5

The results can be found in the bpbkar log on the same client that the command was run from. It will give an idea of how quickly data is read from the file system being tested.

http://seer.entsupport.symantec.com/docs/242918.htm

by TOCALS

17 de jan. de 2010

NetBackup Troubleshoot - Howto

This isn't really a how to, is a quick reference with some commands to use to debug the main problems that happens with NetBackup.


1. Log files
- NetBackup Logs
/usr/openv/netbackup/logs (Unix)
Install_path\NetBackup\logs (Win)
- Unified Logs
/usr/openv/logs (Unix)
Install_path\NetBackup\logs (Win)
- NetBackup Media Manager Logs
/usr/openv/volmgr/debug (Unix)
Install_path\volmgr\debug (win)

Obs.: Media manager Logs need be created.
Be aware with log level, they can increase an fill the directory (File system) and the NetBackup will stop to work.

2. Name Resolution
You need ensure that name resolution is working correctly, verify DNS and DNS reverse.

3. Commands you can use for day by day trouble shooting

- Command to list Process are runing
/usr/openv/netbackup/bin/bpps -x

- All process
/usr/openv/netbackup/bin/bpps -a

- List Jobs
/usr/openv/netbackup/bin/admincmd/bpdbjobs

- Command used to confirm the successful lookup of the client name
/usr/openv/netbackup/bin/bpclntcmd -hn client_name
/usr/openv/netbackup/bin/bpclntcmd -ip client_ip

- Commands used to expire or to calculate a new retention for a media
/usr/openv/netbackup/bin/bpexpdate -m ACB001 -d 0
- Change expiration of copy 2, does not afect copy 1
/usr/openv/netbackup/bin/bpexpdate -backupid eel_0904219764 -d 12/20/2010 08:00:00
-copy 2
- If bpxpdate don't work to expire the media execute the command before use bpexpdate
/usr/openv/volmgr/bin/vmquery -deassignbyid media_ID pool_number stat
- To check pool_number and stat
/usr/openv/volmgr/bin/vmquery -m media_ID

- Command to list configured Storage Units
/usr/openv/netbackup/bin/admincmd/bpstulist -L

- Command to list configured Policies
/usr/openv/netbackup/bin/admincmd/bppllist policy_name -L

- Command used to perform operator functions on drives
/usr/openv/volmgr/bin/vmoprcmd

- Command used to display media status
/usr/openv/netbackup/bin/admincmd/bpmedialist -rl 1

- Script to make a media report
/usr/openv/netbackup/bin/goodies/available_media

- Command to unfreeze medias
/usr/openv/netbackup/bin/admincmd/bpmedia -unfreeze -m ACB001

- Text interactive menu to administrate NetBackup when you can't access the GUI.
/usr/openv/netbackup/bin/bpadm

- Command to report media contents of a robotic library
/usr/openv/volmgr/bin/vmcheckxxx -rt tld -rn 1 -list

- Command to inventory a robotic library
/usr/openv/volmgr/bin/vmupdate -rt tld -rn 1 -full [-interactive] -use_barcode_rules

- Command to manager EMM server - bee careful using this command
/usr/openv/netbackup/bin/admincmd/nbemmcmd
This command can be used to list servers, medias, include new servers see nbemmcmd (1M)

- To delete a media manger
nbemmcmd -deletehost -machinename media_server -machinetype media

- To delete a tape
nbemmcmd -deletemedia -mediaid ABC001 -originhost server

- To tell you if which boxes are have a emmserver and if they are reachable
nbemmcmd -getemmserver

- Will let you know if there are any media conflicts
nbemmcmd -listmedia -conflicts

- Will let you know what hosts are seen as media_managers and master servers
nbemmcmd -listhosts

4. Sybase trouble shooting
- Database Files
/usr/openv/db/data (Unix)
Install_path\NetBackupDB\data (Windows)

- Binary Files
/usr/openv/db/bin (Unix)
Install_path\NetBackup\bin (win)

- To create a blank DBDB database for the recovery
http://seer.entsupport.symantec.com/docs/284299.htm

- Database Administration tool (since NetBackup 6.5.2)
/usr/openv/db/bin/dbadm (Unix)
\Veritas\NetBackup\bin\bpadm (Win)

Be careful to use this tool, some operations can stop and destroy database.

5. NetBackup Consistency Check (NBCC) utility
The NBCC command executes the NBCC utility that is used to confirm the integrity of portions of the NetBackup catalog related to tape media. This check includes the review of configured storage units, 5.x volume database hosts, 6.xEMMservers, tape media, and images associated with tape media.

If NBCC detects catalog inconsistencies, it generates a set of output files. It can create a support package bundle of these files by using available system utilities.
More information about the NBCC utility is available in the Using NetBackup Utilities section of the NetBackup Troubleshooting Guide.


By TOCALS

16 de jan. de 2010

A Quick Reference Card for gathering and viewing NetBackup Unified Logs

NetBackup 6.x introduces a new type of logging, referred to as unified logging. This style of logging is used by new NetBackup daemons, such as the Intelligent Resource Manager (IRM) and the Enterprise Media Manager (EMM) daemons. Unified logs are written to the /usr/openv/logs directory on UNIX systems or the \NetBackup\logs directory on Windows systems. An example of a log file that would be generated would be: 51216-111-990921226-060318-0000000000.log

Some techs to help manage the unified logs
- Quick Reference
http://seer.entsupport.symantec.com/docs/332108.htm

- Some examples
http://seer.entsupport.symantec.com/docs/278059.htm

- How to configure log rotation (prevent full file system)
http://seer.entsupport.symantec.com/docs/279541.htm

- How to change log level
http://seer.entsupport.symantec.com/docs/279929.htm

- How to configure to write in a different directory
http://seer.entsupport.symantec.com/docs/281360.htm

by TOCALS

Status Code: 41 - network connection timed out

The server did not receive any information from the client for too long a period of time. This can occur when a network connection is broken during a backup or restore. This can also occur when a NetBackup timeout is exceeded.

See more:
ftp://ftp.support.veritas.com/pub/support/products/NetBackup_BusinesServer/277695.pdf

By TOCALS

13 de jan. de 2010

Backup job will be interrupted suddenly on Netbackup server with SEP client installed

Backup job will be interrupted suddenly on Netbackup server with SEP client installed. During backup job working, there is an alert of MAC spoofing detected, and all traffic is blocked after then. Cause: It might be caused by SEP Anti-MAC spoofing fun


You can disable Anti-MAC spoofing on SEPM console:
Clients -> traget group -> Polices -> firewall rules -> Traffic and Stealth Settings -> Enable Anti-Mac spoofing (disable it)

12 de jan. de 2010

ERR - Can't read a tape block from a backup file.

problems perform restore via rman using oracle database

When the restore using Rman appear erros on dbclient

Dbclient

13:00:53.765 [2196.6784] <16> VxBSAEndData: ERR - EndData unable to successfully close object
13:00:53.765 [2196.6784] <16> xbsa_EndData: ERR - VxBSAEndData: Failed with error:
Server Status: Communication with the server has not been iniatated or the server status has not been retrieved from the server.
13:00:53.765 [2196.6784] <8> close_image: Session being terminated abnormally, cleaning up
13:00:53.765 [2196.6784] <4> close_image: INF - restore FAILED
13:00:53.765 [2196.6784] <4> close_image: INF ---- end of Restore ---

13:00:53.765 [2196.6784] <16> int_ReadData: ERR - Can't read a tape block from a backup file.

Rman Log


ORA-19870: error reading backup piece bk_uiiktc8l7_s21074_p1_t701899431

ORA-19501: read error on file "bk_uiiktc8l7_s21074_p1_t701899431", blockno 21614081 (blocksize=512)

ORA-27190: skgfrd: sbtread2 returned erro

The system's device driver returned an error "13:00:53.765 [2196.6784] <16> int_ReadData: ERR - Can't read a tape block from a backup file"
while NetBackup was writing to removable media

If the media is tape, check for:

A defective or dirty drive, in which case, clean it or have it repaired (refer to the tpclean command for robotic drives).

Defective media. If this is the case, use the bpmedia command to set the volume to the FROZEN state so it
is not used for future backups.

The device has indicated that cleaning is required before further operations are attempted

When the backup/restore appear the message

The device has indicated that cleaning is required before further operations are attempted

Check the log BPTM

13:59:57.843 [552.6660] <2> check_serial_num: serial number match for drive with SCSI coordinates {4,0,5,0}, dos_path <\\.\Tape0>, drive serial number 1310007476, expected serial number 1310007476
14:00:17.843 [552.6660] <16> init_tape: GetTapeStatus() on <\\.\Tape0> (SCSI coordinates {4,0,5,0}), failed, The device has indicated that cleaning is required before further operations are attempted. (1165)
14:00:17.843 [552.6660] <2> db_error_add_to_file: dberrorq.c:midnite = 1245207600
14:00:17.843 [552.6660] <16> io_open: cannot open file C:\Program Files\Veritas\NetBackup\db\media\tpreq\drive_IBM.ULT3580-TD4.000, The device has indicated that cleaning is required before further operations are attempted. (1165)
14:00:17.843 [552.6660] <2> send_MDS_msg: DEVICE_STATUS 1 11169 ferro 0004L4 4000013 IBM.ULT3580-TD4.000 2000004 OPEN_ERROR 0 0
14:00:17.843 [552.6660] <2> log_media_error: successfully wrote to error file - 06/17/09 14:00:17 0004L4 0 OPEN_ERROR IBM.ULT3580-TD4.000

Run the manual cleaning of the tape-drive

pack nb_clt_6.5.3 install incomplete.

when you are installing of NB_CLT_6.5.3 and NB_CLT_6.5.3.1 appear message.

preinstall script.
See /usr/openv/pack/pack.history for more details.
/usr/openv/netbackup/client/*/*/client_bin.tar.Z: No such file or directory
cp: cannot create /usr/openv/pack/NB_CLT_6.5.3/save: No such file or directory
cp: cannot create /usr/openv/pack/NB_CLT_6.5.3/save/NB_update.uninstall: No such file or directory
Save of NB_update failed
Pack NB_CLT_6.5.3 install incomplete.


We need to create manually

mkdir /usr/openv/pack/NB_CLT_6.5.3/save/

mkdir /usr/openv/pack/NB_CLT_6.5.3.1/save/

Execute the installing again

Status code 9 when backing lotus note

Check the bpbkar log from client.

Bpbkar

10:54:05.938: [3508.4052] <4> tar_backup::V_SetupJobData: INF - dwJobData: ffffffff
10:54:05.938: [3508.4052] <4> tar_backup::V_SetupJobData: INF - dwJob: ffffffff
10:54:05.953: [3508.4052] <4> dos_backup::V_PreProcessing: INF - backup privileges enabled, previous = 0
10:54:05.953: [3508.4052] <4> dos_backup::V_PreProcessing: INF - restore privileges enabled, previous = 0
10:54:05.953: [3508.4052] <4> dos_backup::V_PreProcessing: INF - security privileges enabled, previous = 0
10:54:05.953: [3508.4052] <4> dos_backup::V_PreProcessing: INF - tcb privileges enabled, previous = 0
10:54:05.953: [3508.4052] <4> dos_backup::V_PreProcessing: INF - create token privileges enabled, previous = 0
10:54:05.953: [3508.4052] <4> dos_backup::V_PreProcessing: INF - user name: root
10:54:05.953: [3508.4052] <4> dos_backup::V_PreProcessing: INF - no access token: scheduled backup
10:54:05.953: [3508.4052] <2> ov_log::V_GlobalLog: INF - BEDS_Init() Enter InitFlags:0x0
10:54:05.984: [3508.4052] <2> ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: C:
10:54:05.984: [3508.4052] <2> ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: D:
10:54:05.984: [3508.4052] <2> ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: Microsoft Terminal Services
10:54:05.984: [3508.4052] <2> ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: Microsoft Windows Network
10:54:05.984: [3508.4052] <2> ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: VMware Shared Folders
10:54:05.984: [3508.4052] <2> ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: Web Client Network
10:54:05.984: [3508.4052] <2> ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: Shadow?Copy?Components
10:54:05.984: [3508.4052] <2> ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: System?State
10:54:05.984: [3508.4052] <2> ov_log::V_GlobalLog: INF - DumpDleInfo() DLE Device Name: Active Directory Application Mode
10:54:06.156: [3508.4052] <2> tar_backup::V_SetupProcessContinue: TAR - CONTINUE BACKUP received
10:54:06.281: [3508.4052] <2> tar_backup_tfi::setupFileDirectives: TAR - backup filename = D:\Program Files\Lotus\Domino\data
10:54:06.281: [3508.1940] <4> tar_base::keepaliveThread: INF - keepalive thread is active with an interval of 60 seconds
10:54:07.297: [3508.4052] <4> UpdatePath(): INF - Path updated with Lotus Program Dir 'D:\Program Files\Lotus\Domino\nserver.exe'
10:54:07.297: [3508.4052] <2> tar_base::V_vTarMsgW: FTL - The NetBackup Extension for Lotus Notes is not properly installed:SetCurrentDirectory (WIN32 267: The directory name is invalid. )
10:54:07.297: [3508.4052] <2> tar_backup_vxbsa::add: INF - called with 'D:\Program Files\Lotus\Domino\data'
10:54:07.297: [3508.4052] <2> tar_base::V_vTarMsgW: INF - EXIT STATUS 9: an extension package is needed, but was not installed
10:54:07.297: [3508.4052] <4> backup_create: INF - NetBackup Temp Directory: 'C:\Program Files\Veritas\\NetBackup\Temp'
10:54:07.297: [3508.4052] <4> OVStopCmd: INF - EXIT - status = 0
10:54:07.297: [3508.4052] <2> tar_base::V_Close: closing...

Check some steps

* The most common cause of Status 9 on Lotus Notes backup is incorrectly configuring the INI and PATH information. To add/modify the registry values using REGEDIT32:

1. Click on the Windows Start menu and click Run. In the Run box, type REGEDT32 and press Enter.
2. Select the HKEY_LOCAL_MACHINE key, and navigate the tree and select the SOFTWARE\VERITAS\NetBackup\CurrentVersion\Config key
3. Click on the Config key.
4. Look for a LOTUS_NOTES_PATH value - if it does not exist, from the Edit menu, select Add Value. If it does exist, double-click on it and go to step 7.
5. Set the Value Name to LOTUS_NOTES_PATH.
6. Select REG_SZ for the Data Type, and press OK to add the value.
7. Enter the value data. This is the full directory path to where the nserver.exe file is located. So if, for example, in the environment, the nserver.exe file was located in D:\Lotus\Domino, the value for the LOTUS_NOTES_PATH would be D:\Lotus\Domino.
8. Press OK to accept the value.
9. Repeat steps 4 to 8 for the LOTUS_NOTES_INI value. Please note the value for the LOTUS_NOTES_INI value is the full directory path to the notes.ini file, along with the file name. So if, for example, in your environment, the notes.ini file was located in D:\Lotus\Domino, the value for the LOTUS_NOTES_INI would be D:\Lotus\Domino\notes.ini.

Warning: Incorrect use of the Windows registry editor may prevent the operating system from functioning properly. Great care should be taken when making changes to a Windows registry. Registry modifications should only be carried-out by persons experienced in the use of the registry editor application. It is recommended that a complete backup of the registry and workstation be made prior to making any registry changes.


* Look to see if there are multiple copies of the files named js32.dll and nlsccstr.dll available to NetBackup (tm). In some cases, the NetBackup for Lotus Notes agent initiates, using older versions of these files found in the C:\winnt\system32 directory, causing the Status 9 error. By default, these files are not located in C:\winnt\system32 after a base Windows install.


By using the third party utility Dependency Walker (http://www.dependencywalker.com/), Support was able to determine that the nnotes.dll (from Lotus) and the nblotusnotes.dll (from Veritas) files were searching for and using two different versions of the same file names from different locations.

The nnotes.dll file (a Lotus installed binary used by NetBackup during the backup and restore process) was looking for the js32.dll and nlsccstr.dll files, and finding them in the d:\lotus\domino directory. In contrast, the NetBackup nblotusnotes.dll file was looking for them in the c:\winnt\system32 directory. Furthermore, when Dependency Walker was run against the nblotusnotes.dll file, both the js32.dll and nlsccstr.dll files displayed in red, indicating a problem with the relationship between NetBackup and the relative version of these DLL files.

Resolve this by renaming the js32.dll and nlsccstr.dll files in the c:\winnt\system32 directory to js32.old and nlsccstr.old, and then copy js32.dll and nlsccstr.dll from the :\lotus\domino directory into the C:\winnt\system32 directory.

make clearing the path and ini path in the client's properties

host properties -> client properties -> Windows Client -> Lotus Notes
clear the Lotus Notes Path and Lotus Notes INI Path fields