These all instructions are assuming that you have a machines with IB cards connected with IB switch.
My tests are done with Mellanox DDR(ConnectX-2-20Gb/s) and QDR(ConnectX-3-40Gb/s) cards
Assuming you have on machineA blockdevice/RAID6 on /dev/vg00/lv_01 and you wont to export
it to machineB as a block device/scsi device.
Why do we need it?
Situation1: We bot a bunch of RAID boxes each with 24HDDs very fast on local IO, can I use their full power on MS SQL or MySQL database? Can I combine them in to one striped LVM on DB server?
Situation2: We have a server with huge RAID in LVM and we are running out of space.
Can we expand LVM with another raid without plugging yet another physical raid?
Can we expand LVM without any downtime of server? Yes we can if we will bring the new empty raid over the Infiniband to the server.
There are several solutions for this: ISCSI(TCP/IP)-slow, ISER(RDMA)-fast, SRP(RDMA)-fastest-almost no overhead.
Yes we can export block device with ISCSI protocol. One can export block device from Linux machine to Windows machine using TCP/IP, or better to use ip-over-ib to have 300MB/s prformance. On concurrent and heavy loads ISCSI uses a lot of CPU and performance in some cases is only 10% of original RAID performance. Due to the overhead of ISCSI protocol I never got more than 350MB/s.
There is another more advanced protocol ISER which is ISCSI over RDMA. This one brings more performance from my personal tests it is about 30-40% of original RAID. Unfortunately on heavy loads the kernel panic is accruing with unknown reasons. ISER performance was about 400~420 MB/s
Reading the white paper "
Building a Scalable Storage with InfiniBand" from Mellanox was motiating me to look in SRP protocol.
Let us install SRPT and mount it on another machine.
Assuming machineA is a target and machineB is an initiator.
In normal language
machineA is exporting the block device and machineB is a client.
There are several steps to get running whole machinery.
First of all, do the following in the clean SL 6.3 installation
yum
-y update && yum -y upgrade.
The next step depend on your IB card, if IB card is working with
RDMA then you don't need OFED at all.
On my some machines I have QDR cards(Mellanox Technologies MT27500
Family [ConnectX-3]) they are ok with kernels 2.6.32-279.x.
But another card DDR with dual port does not work without OFED.
Getting RDMA running without and with OFED:
1)
Without OFED.
In order to test your IB installation you can test RDMA connection:
the utils are coming with librdmacm-utils (it contains some examples how to use cm
library)
machineA:> rdma_server
then on server 2:
machineB> rdma_client -s serverA.ibIP
The output should not contain negative results:
rdma_client: start
rdma_server: end 0
rdma_client: end 0
This looks nice.
2)
With OFED we need to be careful, It brings broken SRPT
and we will not be able to compile SCST properly.
2.1) in the OFED folder: ./configure -c myofed.conf
the myofed.conf you can generate like:
./install.pl -p
it will generate ofed-all.conf, rename it to myofed.conf
Edit that file and put and remove all other stuff what you dont
require like mpi or sdp, etc...:
nfsrdma=n
rnfs-utils=n
srp=n
srpt=n
iser=n
I put here nfs as well,so we are not going to run buggy nfs-rdma on
srpt server. The iser brings only 30% of original RAID performance,
so we say
no to iser/iscsi protocol.
For using SCST we need to compile that from source
So let me explain the scripts for compiling:
WARNING: We are going to compile Target part, after this point any kernel
update will break the
SRPT integrity.
On machine where you are going to compile code:
adduser mockbuild
visudo
Add the line
mockbuild ALL=(ALL) NOPASSWD: ALL
We need to set up our rpmbuild environment
mkdir -p ~/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
echo “%_topdir %(echo $HOME)/rpmbuild
%_smp_mflags -j3
%_signature gpg
%_gpg_name Your Name (Your Title) ” > .rpmmacros
Time to get some other directories prepared…
mkdir ~/scst
mkdir ~/mlx
cd ~/rpmbuild/SOURCES
wget http://downloads.sourceforge.net/project/scst/srpt/2.2.0/srpt-2.2.0.tar.bz2
wget http://downloads.sourceforge.net/project/scst/scst/2.2.0/scst-2.2.0.tar.bz2
wget http://downloads.sourceforge.net/project/scst/scstadmin/2.2.0/scstadmin-2.2.0.tar.bz2
wget http://downloads.sourceforge.net/project/scst/iscsi-scst/2.2.0/iscsi-scst-2.2.0.tar.bz2
cp ./srpt-2.2.0.tar.bz2 ~/scst
wget -O ~/rpmbuild/SPECS/scst.spec.newlines
http://pastebin.com/raw.php?i=HLMskJKK
cd ~/rpmbuild/SPECS
tr -d '\r' < scst.spec.newlines > scst.spec
rm scst.spec.newlines
cd ~/scst
tar jxvf srpt-2.2.0.tar.bz2
#Patch the
kernel, you can put this into shell script then
execute later
i
f [ -e /lib/modules/$(uname -r)/build/scripts/Makefile.lib ];
then cd /lib/modules/$(uname -r)/build; else cd
/usr/src/linux-$(uname -r); fi
sudo patch -p1 <
/home/mockbuild/scst/srpt-2.2.0/patches/kernel-$(uname -r | sed -e
's|-.*||')-pre-cflags.patch
#Compile and build RPMS
cd ~/rpmbuild/SPECS
rpmbuild -ba scst.spec
If you see error during the compilation please follow the advice
which will tell you remove ib modules from the kernel path.
WARNING: Before installation
remove /lib/modules/$(uname
-r)/updates/drivers/infiniband/ulp/srpt if exist!!
Almost done.
Let us install rpms on the target machine:
copy rpms from ~mockbuild/rpmbuild/RPMS/..etc
rpm -Uhv
~/kernel-module-scst-core-2.6.32-???.el6-2.2.0-1.ab.x86_64.rpm
rpm -Uhv
~/kernel-module-scst-srpt-2.6.32-???.el6-2.2.0-1.ab.x86_64.rpm
rpm -Uhv ~/scst-tools-2.6.32-220.4.2.el6-???-1.ab.x86_64.rpm
# regenerate dependency of modules
depmod -aq $(uname -r)
Done!
In the next post I will show how to configure scst.
Thanks to Uwe Sauter from HLRS to initiating to publish it.
The credits are going to: http://sonicostech.wordpress.com/2012/02/23/howto-infiniband-srp-target-on-centos-6-incl-rpm-spec/ for detailed explanation which was working for me.
