HDF5 v1.10 or above on the Lustre FS

If you unable to read the files on the lustrefs with the  HDF5 library v1.10 and above you should mount the FS with localflock option. The reason is flock support absence by default.

h5ls --version
h5ls: Version 1.10.1
Error:
h5ls snapshot.hdf5 snapshot_063.hdf5: unable to open file
What?!

mount -tlustrefs  -o localflock 10.10.10.10@o2ib:/lustre /lustre

Now everything is ok!!
 
h5ls snapshot.hdf5
Header                   Group
PartType0                Group 

Performance test: 24 SSD 150 GB Intel on MegaRAID SAS 9361-24i

No RAID, simply jbod mode dstat results:

Reading:
lsscsi | grep SSDSC2BB15| awk '{printf"dd if=%s of=/dev/null bs=16K & \n",$7}'| sh

  0  18  21  60   0   0|6421M    0 | 120B  876B|   0     0 |  36k   53k
  0  18  17  64   0   1|6414M    0 | 120B  346B|   0     0 |  36k   53k
  0  18  14  67   0   1|6413M    0 | 120B  346B|   0     0 |  36k   53k

Writing:
lsscsi | grep SSDSC2BB15| awk '{printf"dd of=%s if=/dev/zero bs=16K & \n",$7}'| sh 

  0   9  11  77   0   3|   0  4503M| 120B  346B|   0     0 |  24k 8055
  0   9  11  77   0   3|   0  4525M| 120B  346B|   0     0 |  24k 8046
  0   9  13  75   0   3|   0  4505M| 120B  346B|   0     0 |  25k 7968

adding more stress:
lsscsi | grep SSDSC2BB15| awk '{printf"dd if=%s of=/dev/null bs=16K & \n",$7}'| sh
lsscsi | grep SSDSC2BB15| awk '{printf"dd of=%s if=/dev/zero bs=16K & \n",$7}'| sh 

  0  19   1  75   0   4|3420M 3401M| 120B  362B|   0     0 |  37k   36k
  0  18   1  77   0   3|3282M 3261M| 180B  362B|   0     0 |  36k   34k
  0  19   2  76   0   3|3297M 3215M| 180B  362B|   0     0 |  35k   33k

Not Bad:)

 

Lustrefs: a big performance hit on lfs find after patch of: CVE-2017-5754 CVE-2017-5753 CVE-2017-5715

Disable: sh set-protection.sh 0

time lfs find /lustre/arm2arm/| wc -l
1454706

real    0m14.941s
user    0m1.633s
sys    0m10.770s

 Enable: sh set-protection.sh 0

time lfs find /lustre/arm2arm/| wc -l
1454706

real    0m10.468s
user    0m0.959s
sys    0m5.521s

Let us hope that the situation will change in the near future....

 And the script set-protection.sh content is:

 #!/bin/bash

[ ! -d  /sys/kernel/debug/x86 ]&& mount -t debugfs debugfs /sys/kernel/debug
echo $1 > /sys/kernel/debug/x86/pti_enabled
echo $1 > /sys/kernel/debug/x86/ibrs_enabled
echo $1 > /sys/kernel/debug/x86/ibpb_enabled

Enable or disable Meltdown and Spectre attack protection on CentOS

After kernel and microcode update one can enable or disable the protection:
Enable:
sh ./set-protection.sh 1
or
Disable:
sh ./set-protection.sh 0 

And the script content is:

 #!/bin/bash

[ ! -d  /sys/kernel/debug/x86 ]&& mount -t debugfs debugfs /sys/kernel/debug
echo $1 > /sys/kernel/debug/x86/pti_enabled
echo $1 > /sys/kernel/debug/x86/ibrs_enabled
echo $1 > /sys/kernel/debug/x86/ibpb_enabled

Q:Why should I disable the protection?
A: If you enable the protection you might accounter the performance degradation.
For some tasks one can enable or disable it:
Synthetic test iperf3 shows network performance over IB degradation about x2:
node01:iperf3 -s
node02:iperf3 -c node01.ib

Enabled:

[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  8.70 GBytes  7.48 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  8.70 GBytes  7.47 Gbits/sec                  receiver

Disabled:
[ ID] Interval           Transfer     Bandwidth       Retr
[  4]   0.00-10.00  sec  17.6 GBytes  15.1 Gbits/sec    0             sender
[  4]   0.00-10.00  sec  17.6 GBytes  15.1 Gbits/sec                  receiver






[node01~]#ibstat
CA 'mlx4_0'
    CA type: MT4099
    Number of ports: 1
    Firmware version: 2.40.7000
    Hardware version: 0
    Node GUID: 0xXXXXXXX
    System image GUID: 0xXXXXXXX
    Port 1:
        State: Active
        Physical state: LinkUp
        Rate: 56
        Base lid: 338
        LMC: 0
        SM lid: 3
        Capability mask: 0xXXXXXX
        Port GUID: 0xXXXXXXX
        Link layer: InfiniBand

Fixing X11 forwarding on CentOS7.4

I have an client Centos 6.5 trying to connect to Centos 7.4 with X11 forwarding, but no luck:

Finally I found that somehow the important option is missing in the server side in /etc/ssh/sshd_config :

X11UseLocalhost no

After adding everything is ok now.

Lustre 2.10.x dkms usage

get the src for the lustre-client-dkms package from the:

https://build.hpdd.intel.com/job/lustre-b2_10/arch=x86_64,build_type=client,distro=el7,ib_stack=inkernel/

1) rpmbuild --rebuild --without servers lustre-client-dkms-2.10.2_RC1-1.el7.src.rpm
2) yum install /root/pmbuild/RPMS/noarch/lustre-client-dkms-2.10.2_RC1-1.el7.centos.noarch.rpm
It will rebuild the modules for the active kernel with the following commands:
 

/bin/bash /sbin/dkms build -m lustre-client -v 2.10.2_RC1 -k 3.10.0-693.5.2.el7.x86_64

PS troubleshooting
if dkms  status shows something like this:

 dkms status
lustre-client, 2.10.2_RC1, 3.10.0-693.5.2.el7.x86_64, x86_64: installed (original_module exists) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!) (WARNING! Diff between built and installed module!)

1) dkms --force remove  -m  lustre-client -v 2.10.2_RC1 -k 3.10.0-693.5.2.el7.x86_64
2) rm -fr  /lib/modules/3.10.0-693.5.2.el7.x86_64/extra/lustre
3) find and remove manually the module leftovers: find /lib/modules/| grep lustre
4) dkms --force install -m  lustre-client -v 2.10.2_RC1 -k 3.10.0-693.5.2.el7.x86_64

Deleting the funny 192.168.122.xxx network from CentOS 7.4

In order to stop unnecessary interface on the VM:
 
yum install -y libvirt-client
systemctl start  libvirtd 
virsh net-destroy default
systemctl stop  libvirtd 
systemctl disable  libvirtd