OVIRT engile unable to import old "Export storage": How to fix

WARNING: before to start this tests, backup your datacenters,storages, config files etc..!!!
 
After reinstall ovirt 3.5.x I got following error during the import export domain:
Error while executing action: Cannot add Storage Connection. Storage connection already exists.

This was due to the many experiments what I was doing with old storage, I was trying  to attach or detach/ destroy them in meanwhile the name server went down, so Ovirt-Engine  somehow was screwed out.

Not show time to fix the corrupted database:
1) Try identify ovirt storageconnections using API:
curl -k -v -u "admin@internal:xxxxxxx" -H "Content-type: application/xml" -X GET https://ovirt-engine.cls/api/storageconnections

It shows something like this:

.....

   
       

192.168.101.102

        nfs
        /tank/VMs/EXPORT
   

......


2) So let us delete it:
 curl -k -v -u "admin@internal:xxxxx"  -X DELETE https://ovirt-engine.cls/api/storageconnections/8e5335e0-8821-40ee-8dc8-6b4245032920

 output will be like this:
complete

Important to have here complete, otherwise you should dive more deeper to find out the trouble.

So next try to connect again your old export, if it claims that domain is inuse with other one then make following:
1) mount old export storage to /tmp/media
2) find file: dom_md/metadata
3) remove line _SHA_CKSUM=
4) set empty  POOL_UUID=

Try to import in to new DC.
Enjoy the new setup.

sysbench: benchmark your mysql storage

0) yum install sysbench -y 
1) mysql -uroot -e "CREATE SCHEMA sbtest;"
2)  sysbench --db-driver=mysql --test=oltp --num-threads=8 --max-requests=10000 --oltp-table-size=2000000 --oltp-test-mode=complex --mysql-host=localhost --mysql-db=sbtest --mysql-table-engine=innodb prepare --mysql-port=3306 --mysql-user=root
3) sysbench --db-driver=mysql --test=oltp --num-threads=8 --max-requests=10000 --oltp-table-size=2000000 --oltp-test-mode=complex --mysql-host=localhost --mysql-db=sbtest --mysql-port=3306 --mysql-user=root  run | tee sysbench.log

4) analyse your sysbench.log

Results on striped 2x3TB SATA-HGST disks /dev/md0:

OLTP test statistics:
    queries performed:
        read:                            140014
        write:                           50005
        other:                           20002
        total:                           210021
    transactions:                        10001  (151.00 per sec.)
    deadlocks:                           0      (0.00 per sec.)
    read/write requests:                 190019 (2868.92 per sec.)
    other operations:                    20002  (301.99 per sec.)

Test execution summary:
    total time:                          66.2336s
    total number of events:              10001
    total time taken by event execution: 529.6338
    per-request statistics:
         min:                                 15.57ms
         avg:                                 52.96ms
         max:                               1392.39ms
         approx.  95 percentile:             100.09ms

Threads fairness:
    events (avg/stddev):           1250.1250/5.11
    execution time (avg/stddev):   66.2042/0.01

cli ansible vs clustershell : running commands on multiple hosts.

Let us start with CentOS 6.5
1) enable epel:  yum locainstall -y  http://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
2) yum install clustershell -y
3) yum install ansible

Let us run who command on multiple hosts:
clush -w localhost,anotherhost -B -b w
ansible all -i 'localhost,anotherhost,' -c local -m command -a "w"

Then Analyze output....

OpenSM lid and more...

On centOS 6.5
yum groupinstall "Infiniband Support"
yum install infiniband-diags

Infiniband diagnostics tools are containing nice "weapons" to eliminate network BUGS.
One of them is:
 saquery - query InfiniBand subnet administration attributes



 it looks like this:

NodeRecord dump:
                lid.....................0x7D
                reserved................0x0
                base_version............0x1
                class_version...........0x1
                node_type...............Switch
                num_ports...............36
                sys_guid................0xf452140300365230
                node_guid...............0xf452140300365230
                port_guid...............0xf452140300365230
                partition_cap...........0x8
                device_id...............0xC738
                revision................0xA2
                port_num................0
                vendor_id...............0x2C9
                NodeDescription.........MF0;switch-cf2826:SX6036/U1

Fix infiniband QDR or FDR troubles on fat memory machines.

Fix memory trouble on FAT machines:
The formula to computer the maximum value of pagepool when using RDMA is:   
2^log_num_mtt  x  2^log_mtts_per_seg * x PAGE_SIZE > ( 2x pagepool )

2^20 bytes x 2^4 x 4K = 64GiB
Add:
/etc/modprobe.d/mlx4_core.conf
options mlx4_core log_num_mtt=20 log_mtts_per_seg=4
check changes

more /sys/module/mlx4_core/parameters/log_num_mtt

more /sys/module/mlx4_core/parameters/log_mtts_per_seg

CentOS 7 after install:

CentOS 7 is different management than CentOS 5.x and 6.x.
It is uses systemctl to manage services.

systemctl enable sshd
systemctl list-unit-files
systemctl get-default
systemctl set-default multi-user.target
systemctl disable firstboot-graphical.service
systemctl disable bluetooth.service
systemctl enable network.service
systemctl show
systemd-analyzesystemd-analyze blame

Tune other stuff as you need.

Lustre 2.1.6 server and 2.5.3 client.

Do they compatible ?
The answer is yes!
Lustre client upgrading using yum in centos 6.5:

vim /etc/yum.repo/lustre.repo
[hpddLustreserver]
name=CentOS-$releasever - Lustre
baseurl=https://downloads.hpdd.intel.com/public/lustre/latest-maintenance-release/el6/server/
gpgcheck=0
enabled=1
 
[e2fsprogs]
name=CentOS-$releasever - Ldiskfs
baseurl=https://downloads.hpdd.intel.com/public/e2fsprogs/latest/el6/RPMS/
gpgcheck=0
enabled=1

[hpddLustreclient]
name=CentOS-$releasever - Lustre
baseurl=https://downloads.hpdd.intel.com/public/lustre/latest-maintenance-release/el6/client/
gpgcheck=0
enabled=1

On client:
1) yum update lustre-client -y
2) and finally one liner to restart the lustre mount point:

umount /lustre;lustre_rmmod;service lnet stop;service lnet start;mount /lustre