Freitag, 2. Juni 2017

Torque + Maui with the mixed nodes, eg with different number of cores per numa node


Assuming we have in the compute cluster 2 different types of processors: E-2650v3 and E-2650v4.
Each node has a dual CPU, becouse of 26xx :) . If we divide to numa boards then each "node" will get 10 cores(v3) and 12 cores(v4).

To use the whole cluster as one queue one need to reconfigure the maui:

/usr/local/maui/maui.cfg
ENABLEMULTIREQJOBS              TRUE

Then use the job script with the mixed resources:

#!/bin/bash
#PBS -q mix2
#PBS -r n
#PBS -l nodes=2:ppn=12:dbt+2:ppn=10:db
#PBS -l walltime=1:00:00
#PBS -u arm2arm
Where dbt and db are nodes future flags in the /var/lib/torque/server_priv/nodes file.

And we can can see now our job is  running on different nodes with different number of cores:
lei4-0      db  dbt 15.9  excl 10/10  0.00      6%                9549     arm2arm
lei4-1      db  dbt 16.0  excl 10/10  0.00      2%                9549     arm2arm
lei76-0  b2dis  dbt 63.9  excl 12/12  0.00      2%                9549     arm2arm
lei76-1  b2dis  dbt 64.0  excl 12/12  0.00      2%                9549     arm2arm
lei4   has 2 numa boards each with 10 cores
lei76 has 2 numa boards each with 12 cores
 
 

Keine Kommentare: