Avamar - MCCLI - 1,22801,User login failure.

After running /usr/local/avamar/7.0.1-61/bin/avsetup_mccli for the first time, if you make an error in your setup, it will never go back and update files in your home ~/.avamardata/* correctly.


Fix:

1. delete ~/.avamardata/* OR:
2. manually update ~/.avamardata/7.0.1-61/var/mc/cli_data/prefs/mcclimcs.xml with the correct information.

Avamar - How to retrieve all of your grid's serial numbers

Sometimes, EMC support will contact you because your equipment dialed in, and need a part replaced. They will mention a node serial number, so if you have multiples sites like me, here is a useful command to retrieve your grid's serials:

Log in to the utility node:

#> su - admin
#> ssh-agent bash
#> ssh-add ~/.ssh/dpnid
#> mapall --noerror --user=root --nodes=all+ --quiet 'echo `hostname -i`": "`/usr/bin/ipmitool fru print 0 | grep "Product Asset Tag" | sed "s/^.*:\s\(.*$\)/\1/"`'

This will get you the node serial number only.
To get more extensive information, you can try this as well:
#> mapall --noerror --user=root --nodes=all+ --quiet 'echo `hostname -i`": "`/usr/bin/ipmitool fru print 0 " grep "Product Asset Tag" " sed "s/^.*:\s\(.*$\)/\1/"`'

(see solution esg111183 on emc support website) 

Avamar - How to verify your backup nodes connection speed

I recently suspected one of my storage node was not connected at 1 Gbps, this can be verified quickly from the utility node using the "mapall" and "ethtool" commands.
mappall send a command to all the storages nodes, and ethtool is a linux command.

Here you go :

su - admin

ssh-agent bash

ssh-add ~/.ssh/dpnid

mapall --noerror --user=root 'ethtool eth0' | grep "Speed"

mapall --noerror --user=root 'ethtool eth2' | grep "Speed"

These are the NICs that face your network

To check the internal NICs: 

mapall --noerror --user=root 'ethtool eth1' | grep "Speed"

mapall --noerror --user=root 'ethtool eth3' | grep "Speed"

Now you can verify that the "Speed" fields all say 1000Mb/s.
Mines did not:

(0.0) ssh  -x  -o GSSAPIAuthentication=no root@192.168.255.2 'ethtool eth2'
        Speed: 1000Mb/s
(0.1) ssh  -x  -o GSSAPIAuthentication=no root@192.168.255.3 'ethtool eth2'
        Speed: 100Mb/s
(0.2) ssh  -x  -o GSSAPIAuthentication=no root@192.168.255.4 'ethtool eth2'
        Speed: 100Mb/s
(0.3) ssh  -x  -o GSSAPIAuthentication=no root@192.168.255.5 'ethtool eth2'
        Speed: 1000Mb/s
(0.4) ssh  -x  -o GSSAPIAuthentication=no root@192.168.255.6 'ethtool eth2'
        Speed: 1000Mb/s
(0.5) ssh  -x  -o GSSAPIAuthentication=no root@192.168.255.7 'ethtool eth2'
        Speed: 1000Mb/s
(0.6) ssh  -x  -o GSSAPIAuthentication=no root@192.168.255.8 'ethtool eth2'
        Speed: 100Mb/s
(0.7) ssh  -x  -o GSSAPIAuthentication=no root@192.168.255.9 'ethtool eth2'
        Speed: 1000Mb/s
(0.8) ssh  -x  -o GSSAPIAuthentication=no root@192.168.255.10 'ethtool eth2'
        Speed: 1000Mb/s
(0.9) ssh  -x  -o GSSAPIAuthentication=no root@192.168.255.11 'ethtool eth2'
        Speed: 1000Mb/s


You can then work with support to solve the issue, or check your cables / switchs, etc.






 

Avamar and large dataset with multiples files

Here is a gem I found in the Avamar forums, from a "Ask the expert" session answered by Ian anderson concerning large file systems and Avamar. Please share your experiences if you have any.

lmorris99 wrote:

We have one server with 25 million files, scattered through directories six levels deep.
We'd like to throw it at our test Avamar grid; any tuning I should look at on the client (or server) side before we set it up for its first backup?

The most important thing to do on a client with so many files is to make sure that the file cache is sized appropriately. The file cache is responsible for the vast majority (>90%) of the performance of the Avamar client. If there's a file cache miss, the client has to go and thrash your disk for a while chunking up a file that may already be on the server.

So how to tune the file cache size?

The file cache starts at 22MB in size and doubles in size each time it grows. Each file on a client will use 44 bytes of space in the file cache (two SHA-1 hashes consuming 20 bytes each and 4 bytes of metadata). For 25 million files, the client will generate just over 1GB of cache data.

Doubling from 22MB, we get a minimum required cache size of:
22MB => 44MB => 88MB => 176MB => 352MB => 704MB => 1408MB

The naive approach would be to set the filecachemax in the dataset to 1500. However, unless you have an awful lot of memory, you probably don't want to do that since the file cache must stay loaded in memory for the entire run of the backup.

Fortunately there is a feature called "cache prefixing" that can be used to set up a unique pair of cache files for a specific dataset. Since there are so many files, you will likely want to work with support to set up cache prefixing for this client and break the dataset up into more manageable pieces.

One quick word of warning -- as the saying goes, if you have a hammer, everything starts to look like a nail. Cache prefixing is the right tool for this job because of the large dataset but it shouldn't be the first thing you reach for whenever there is client performance tuning to be done.

On to the initial backup.

If you plan to have this client run overtime during its initial, you will have to make sure that there is enough free capacity on the server to allow garbage collection to be skipped for a few days while the initial backup completes.

If there is not enough free space on the server, the client will have to be allowed to time out each day and create partials. Make sure the backup schedule associated with the client is configured to end no later than the start of the blackout window. If a running backup is killed by garbage collection, no partial will be created.

You will probably want to start with a small dataset (one that will complete within a few days) and gradually increase the size of the dataset (or add more datasets if using cache prefixing) to get more new data written to the server each day. The reason for this is that partial backups that will only be retained on the server for 7 days. Unless a backup completes successfully within 7 days of the first partial, any progress made by the backup will be lost when the first partial expires.

After the initial backup completes, typical filesystem backup performance for an Avamar client is about 1 million files per hour. You will likely have to do some tuning to get this client to complete on a regular basis, even doing incrementals. The speed of an incremental Avamar backup is generally limited by the disk performance of the client itself but it's important to run some performance testing to isolate the bottleneck before taking corrective action. If we're being limited by the network performance, obviously we don't want to try to tweak disk performance first.

The support team L2s from the client teams have a good deal of experience with performance tuning and can work with you to run some testing. The tests that are normally run are:

• An iperf test to measure raw network throughput between client and server
• A "randchunk" test, which generates a set of random chunks and sends them to the grid in order to test network backup performance
• A "degenerate" test which, as I mentioned previously, processes the filesystem and discards the results in order to measure disk I/O performance
• OS performance monitoring to ensure we are not being bottlenecked by system resource availability (CPU cycles, memory, etc.)

Edit -- 2013-08-06: The behaviour for partial backups changed in Avamar 6.1. More information in the following forums thread:
Re: Garbage collection does not reclaim expected amount of space

1,23996,CLI failed to connect to MCS

I tried to install a remote MCCLI instance as I do not like to work directly from the utility node, and I kept getting this error and could not find the problem.

Support told me running MCCLI from a remote host was not supported (of course it is).

When I finally looked up the config file "/root/.avamardata/6.1.1-87/var/mc/cli_data/prefs/mcclimcs.xml"
I noticed I had the "mcsaddr" filed wrong.

Make sure you don't have any configuration problems...

Avamar nodes capacity and hardware

For Gen4 3.9TB node :
Raw capacity is 6 TB and licensed capacity is 3.9 TB

For Gen4 7.8 TB node:
Raw capacity is 12 TB and licensed capacity is 7.8 TB

So, In a RAIN config with 1 utility node and 4 storage nodes (7.8 TB), the total capacity should be 48 TB and total licensed capacity is 31.2 TB


Single Node:

• A single stand-alone node of Avamar with the brains and the storage capacity all in one.

RAIN Configuration:

• Redundant Array of Independent Nodes.  Think of a RAID5 array of servers instead of disks

Storage Node:

• A single Avamar server appliance (2U server).  They are available in 1TB, 2TB and now 3.3TB configurations.

Utility Node:    

• An Avamar appliance similar to the storage node (same physical size) that works as the brains of an Avamar system.

With the release of the new Gen4 nodes, we can now take advantage of higher density disks and improved hardware configurations.  There are 6 different type of nodes: Accelerator node, Utility node, 1.3TB, 2.6TB, 3.9TB and 7.8TB storage nodes.

All of the above node types now sport a single Quad Core Processor with the exception of the Accelerator node containing 2 Quad Core Processors, this is a step above the single processor in previous Gen models. 
Disks are now configured in a RAID 1 configuration for increased performance.  All node chassis’ are the R710 model with support up to 6 drives, except the 7.8TB node that has a R510 chassis which can support up to 12 drives.

A thing to note about the 7.8TB node, is the inclusion of a 100GB SSD for increased memory performance focusing on indexes.  With the increased amount of disks and activity, this should keep the performance in line with the other nodes.
Here is a table with the different type of Nodes and their perspective hardware configuration:

Nodes	Disk	RAM	CPU	Ethernet	Chassis	RAIN
7.8TB	12x2TB SATA 6x RAID 1 pairs	32GB 100GB SSD	1	4	R510	Yes
3.9TB	6x2TB SATA 3x RAID 1 pairs	36GB	1	4	R710	Yes
2.6TB	4x2TB SATA 2x RAID 1 pairs	24GB	1	4	R710	No
1.3TB	2x2TB SATA 1x RAID 1 pair	12GB	1	4	R710	No
Utility	2x300GB SAS 1x RAID 1 pair	12GB	1	8	R710	-
Accelerator	2x300GB SAS 1x RAID 1pair	36GB	2	4	R710

Switch:
Manufacturer:              Allied Telesis
Model:                                 AT-9924 (dumbed down to basically a hub)
Type:                                   Multilayer IPv4 and IPv6 Gigabit switch
Ports:                                    24
Power:                                 Dual Power Supplies – 75W maximum
Heat (BTU):                     256 BTU/hr
Size:                                        1 U each
Switching Capacity:   48Gbps