Avamar Grabage collection troubleshooting

If your garbage collection stop working, you know you are in trouble. Your grid will eventually fill up, and die a slow and painfull death.
Use the following commands from the utility node to try and diagnose the problem:

dumpmaintlogs --types=gc --days=7
In case you see along the lines of "<4203> garbage collection skipped because backups in progres", please verify which nodes are allowed overtime, they are certainly creating problems.

Nodes that are allowed overtime can be seen with:
mccli client show --recursive=true --verbose=true | grep -v "No overtime allowed"


If you'd rather just have garbage collection kill the sessions preventing it from running, just type:
avmaint sched gc --ava --kill=300 --permanent

gc_report
less /usr/local/avamar/var/cron/gc.log

Avamar - How to retrieve all of your grid's serial numbers

Sometimes, EMC support will contact you because your equipment dialed in, and need a part replaced. They will mention a node serial number, so if you have multiples sites like me, here is a useful command to retrieve your grid's serials:

Log in to the utility node:

#> su - admin
#> ssh-agent bash
#> ssh-add ~/.ssh/dpnid
#> mapall --noerror --user=root --nodes=all+ --quiet 'echo `hostname -i`": "`/usr/bin/ipmitool fru print 0 | grep "Product Asset Tag" | sed "s/^.*:\s\(.*$\)/\1/"`'

This will get you the node serial number only.
To get more extensive information, you can try this as well:
#> mapall --noerror --user=root --nodes=all+ --quiet 'echo `hostname -i`": "`/usr/bin/ipmitool fru print 0 " grep "Product Asset Tag" " sed "s/^.*:\s\(.*$\)/\1/"`'

(see solution esg111183 on emc support website) 

Customizing the "Documents And Download" page

Suppose you have windows 2k clients, that needs an older version of the avamar client.

If your are like me and want to keep the file easily accessible, you can simple add them to the "Documents and Downloads" page accessible from http://utility_node_ip by uploading the client to the utility node.

Simple use "winscp" or any other tool to transfer the file, and upload them to:
/data01/avamar/src/downloads/


For example, if I want to add the client under Windows 32 bits, I'll place the executable here:
/data01/avamar/src/downloads/WIN32/AvamarClient-windows-x86-4.1.106-27.msi

That's it, simply reload the page and you will see the client appear.

Avamar - How to query the PostgreSQL MCS database

Here is how to connect to the DB if you need custom reports and want to create custom queries:

1.)  SSH to the Utility node


root@dpn:~/#: su - admin
admin@dpn:~/>: psql -p 5555 -U viewuser mcdb
Welcome to psql 8.2.3, the PostgreSQL interactive terminal.

mcdb=>

Note:  Please use "viewuser" (read only) account for your queries !

2.) Here is the example for a simple SQL query, but you can build your own queries and export it as plain text .

    SELECT * FROM "public"."v_clients";
    SELECT v_clients.cid, v_clients.client_name, v_clients.client_addr,v_clients.os_type FROM "public"."v_clients";


select client_name,status_code,domain FROM v_activities_2 WHERE (status_code=30901 or status_code=30915) and recorded_date_time > current_timestamp - interval '24 hours';

avagent Error 5320- The client named: client1 is not registered as CID: NO LOCATION - RECORD MAY BE CORRUPT

While I was trying to fix a corrupted install I got the following error:

2013-11-13 09:40:40 avagent Error <5320>: Received from MCS#103:The client named: client1 is not registered as CID: NO LOCATION - RECORD MAY BE CORRUPT

2013-11-13 09:40:40 avagent Error <6210>: Unable to register 'client1' with MCS at utilitynode_addr:28001

Fix:

 Delete de client ID file (cid.bin), and re-register.

Avamar - How to get around the 1 schedule repetition per hour maximum

I recently stumbled on the need to take SQL transactional backups every 15 minutes for a specific project (RAID protection on an active-active array was not enough it seem), I came up with a  solution using crontabinf the EMC community forums.

1. Create a Dataset such as SQL1hrinc
    a. Select the database to backup
    b. Choose incremental as backup type

2. Under Policy, create a New Group i.e., SQL1hrinc
    a. Add Dataset SQL1hrinc to the group
    b. Select Retention, client etc

3. Under Policy, click on Group and run backup to make sure group meets the requirements

Now, please login to Avamar utility node via Putty as root user

1. Create a batch file in the /usr/local/avamar/bin such as sqlinc and add the following:
/usr/local/avamar/bin/mccli client backup-group-dataset --xml --name=/DOMAIN_NAME/CLIENT_NAME --group-name=/SQL1hrinc

2. Save file and set the execute permissions i.e., chmod 755 sqlinc

3. Run the file and make sure that the backup is performed.

4. As root, type crontab -e and add the following:

*/60 7-19 * * 1-5 /usr/local/avamar/bin/sqlinc >/dev/null 2>&1

5. Save and exit

(As seen on the Avamar forums, thanks Sandeep Sinha)

Avamar - How to verify your backup nodes connection speed

I recently suspected one of my storage node was not connected at 1 Gbps, this can be verified quickly from the utility node using the "mapall" and "ethtool" commands.
mappall send a command to all the storages nodes, and ethtool is a linux command.

Here you go :

su - admin

ssh-agent bash

ssh-add ~/.ssh/dpnid

mapall --noerror --user=root 'ethtool eth0' | grep "Speed"

mapall --noerror --user=root 'ethtool eth2' | grep "Speed"

These are the NICs that face your network

To check the internal NICs: 

mapall --noerror --user=root 'ethtool eth1' | grep "Speed"

mapall --noerror --user=root 'ethtool eth3' | grep "Speed"

Now you can verify that the "Speed" fields all say 1000Mb/s.
Mines did not:

(0.0) ssh  -x  -o GSSAPIAuthentication=no root@192.168.255.2 'ethtool eth2'
        Speed: 1000Mb/s
(0.1) ssh  -x  -o GSSAPIAuthentication=no root@192.168.255.3 'ethtool eth2'
        Speed: 100Mb/s
(0.2) ssh  -x  -o GSSAPIAuthentication=no root@192.168.255.4 'ethtool eth2'
        Speed: 100Mb/s
(0.3) ssh  -x  -o GSSAPIAuthentication=no root@192.168.255.5 'ethtool eth2'
        Speed: 1000Mb/s
(0.4) ssh  -x  -o GSSAPIAuthentication=no root@192.168.255.6 'ethtool eth2'
        Speed: 1000Mb/s
(0.5) ssh  -x  -o GSSAPIAuthentication=no root@192.168.255.7 'ethtool eth2'
        Speed: 1000Mb/s
(0.6) ssh  -x  -o GSSAPIAuthentication=no root@192.168.255.8 'ethtool eth2'
        Speed: 100Mb/s
(0.7) ssh  -x  -o GSSAPIAuthentication=no root@192.168.255.9 'ethtool eth2'
        Speed: 1000Mb/s
(0.8) ssh  -x  -o GSSAPIAuthentication=no root@192.168.255.10 'ethtool eth2'
        Speed: 1000Mb/s
(0.9) ssh  -x  -o GSSAPIAuthentication=no root@192.168.255.11 'ethtool eth2'
        Speed: 1000Mb/s


You can then work with support to solve the issue, or check your cables / switchs, etc.