Skip to main content

Multipath ...

Issue: Removing a device, or replacing a device with a different SAN LUN can be a bit challenging.

HISTORY: Take this completely hypothetical situation, which may or may not happen to me quite frequently...
your customer asked for 3 x 64 and 1 x 16 Gig LUNs... and for some reason the SAN admin assigns 4 x 64 Gig LUNs. You don't know this until you scan the bus and see them from the OS. At which point you have to tell the SAN admin that you need to replace 1 of the 64 Gig LUNs with a 16 Gig. And you also ask the admin to let you know when he/she removes the incorrect LUN so you can do your procedure to then remove the device from the OS view, and then he/she can proceed with adding the 16 Gig. Well, inevitably you will end up with syslog complaining about a SAN device no longer being available and the fun begins (this is because the admin removed the 64 and added the 16 probably in the same keystroke, or click of a button and this will not give you an opportuntiy to straighten things out...) Well, the only thing that makes this worse is when it happens to a 3-node RAC cluster and have to clean things up on 3 boxes ;-)

# tail -f /var/log/messages
# multipath -ll > /tmp/mpath.out
-- check that output for 'fail' and you should still see the "sd" aliases.
-- confirm the failed device has the same WWN as you were expecting

# echo offline > /sys/class/scsi_disk/2:0:0:44/device/state
# echo offline > /sys/class/scsi_disk/3:0:0:44/device/state
# echo 1 > /sys/class/scsi_disk/2:0:0:44/device/delete
# echo 1 > /sys/class/scsi_disk/3:0:0:44/device/delete
# echo "- - -" > /sys/class/scsi_host/host#/scan
# rescan-scsi-bus.sh
-- the rescan shell script is part of the sg3_utils package

I believe a point of contention is that the LUN being replaced occupies the same device path, although I have not devoted much time to proving this, yet...

#! /bin/sh
# return all offline scsi devices to the running state
for d in /sys/block/sd*/device/state; do if [ `cat $d` = "offline" ]; then echo running > $d; fi; done

Enable additional logging in the lpfc driver:
# echo 0x1f > /sys/module/lpfc/lpfc_log_verbose

Enable extra logging for the scsi-subsystem:
# echo 7 > /sys/module/scsi_mod/scsi_logging_level
~
Labels:

Comments

Post a Comment

Popular posts from this blog

P2V using dd for KVM-QEMU guest

Preface: I have certainly not exhaustively tested this process.  I had a specific need and found a specific solution that worked. Situation:  I was issued a shiny new laptop running Red Hat Enterprise Linux 7 (with Corp VPN, certs, Authentication configuration, etc...)  The image was great, but I needed more flexibility on my bare metal.  So, my goal was to P2V the corporate image so I could just run it as a VM. * Remove corporate drive and install new SSD * install corp drive in external USB-3 case * Install RHEL 7 on new SSD * dd old drive to a disk-image file in a temp location which will be an image which is the same size as your actual drive (unless you have enough space in your destination to contain a temp and converted image) * convert the raw disk-image to a qcow file while pushing it to the final location - this step should reduce the disk size - however, I believe it will only reduce/collapse zero-byte blocks (not just free space - i.e. if you de...
1 comment
Read more

Extending SNMP to run arbitrary shell script

Why are we here... This is not likely something I would have pursued under normal circumstances.  I happen to be working for a customer/client who is not afforded a lot of flexibility to accomplish their goals.  In this case, the rigor is justified.  They have to sometimes be fairly creative with how they solve problems. In this case they would like to utilize an existing snmp implementation to execute a command (or shell script) on a remote system.  They came to me with the idea of using Net-SNMP extend. https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/sect-System_Monitoring_Tools-Net-SNMP-Extending.html NOTE:  This is NOT a good implementation strategy in the "real world"  it will simply allow you to test the functionality.  There are a TON of security implications which would need to be taken in to consideration. Implementation Steps: [root@rh7tst01 ~]# yum -y install net-snmp net-snmp-utils ...
Post a Comment
Read more

RHN Satellite Server (spacewalk) repomd.xml not found

"repomd.xml not found" If you add a channel, or if your RHN cache gets corrupted, and one of your guests complains that it cannot find repomd.xml for jb-ews-2-x86_64-server-5-rpm (for example) - you need to rebuild your repodata cache. Normally this is an automated job - which is exemplified by the fact that you have obviously built out your entire Satellite environment and never had to do any of the steps you are about to do. So - some prep work: Open 3 terminals to your Satellite Server and run: # Term 1 cd /var/cache/rhn watch "ls -l | wc -l" # Term 2 pwd cd /var/log/rhn tail -f rhn_taskomatic_daemon.log # Term 3 satellite-sync --channel=jb-ews-2-x86_64-server-5-rpm Once the satellite-sync has completed, you >should< see the count increment by one.  If you are unlucky (like me) you will not. You then need to login to the Satellite WebUI as the satellite admin user. Click on the Admin tab (at the top) Task Schedules (on the left) fin...
Post a Comment
Read more