Got bad lustre OST? Disable it!!

DEACTIVATE WRITES to OST
=========================

mgs:>lctl conf_param fsname-OST0002-osc.osc.active=0

check active/inactive OST:

cat /proc/fs/lustre/lov/fsname-MDT0000-mdtlov/target_obd

Simply replace fsname of your file system name. 

Before taking off or re-formating OST you can move out the data:  

Assuming we would like to move out all data from OST0002, run script on client: 

lfs find /lustre --ost fsanme-OST0002 | xargs -P4 -I{} rebalance-lustre.sh {}

And script: 

cat /opt/admin/rebalance-lustre.sh 
#!/bin/bash

file="$1"
fn=$(mktemp --dry-run "${file}".lureb-XXXXX )
echo $file $fn
cp -a "$file" "$fn" && unlink "$file" && mv "$fn" "$file"

=====================================================

Note: Script does not work properly with files with special characters like space.