Skip to main content

Better is not always... better? HP Smart Array and Linux

I am currently working through an issue with my 3-node RAC clusters (RHEL 5.6 x86_64 and Oracle RAC 11g running on HP DL580-G7 servers). They seem to enjoy rebooting themselves at will. There is nothing glaring for a root cause, other than some messages in the syslog about blocking for more than 120 seconds. Anyhow - after quite a bit of research I have discovered some things I really like about Linux. They have made the disk scheduler modular (in a sense). Therefore you can utilize your disk access in one of 4 methods. The CCISS is the HP Smart Array driver which is loaded and should be consistent in most Linux releases.

If you initially look at the "scheduler" file - you can see your four options. The one in use is surrounded by brackets. I am hoping that by changing the access for ONLY the cciss device to noop, that my reboots go away - and I leave a positive legacy behind at my customer-site ;-)


root@dbslp0067:/root
# cd /sys/block/cciss\!c0d0/queue/
root@dbslp0067:/sys/block/cciss!c0d0/queue
# cat scheduler
noop anticipatory deadline [cfq]
root@dbslp0067:/sys/block/cciss!c0d0/queue
# echo noop > scheduler
root@dbslp0067:/sys/block/cciss!c0d0/queue
# cat scheduler
[noop] anticipatory deadline cfq
root@dbslp0067:/sys/block/cciss!c0d0/queue

After trying this work-around on my system, I'm disappointed to report it did not help my cause. I will leave this out there, as I may need to tune a system for this at a later point.


Turned out to be a bad "CPU on a SAN switch blade". Not sure why multipath didn't handle the event better than the box locking up and subsequently rebooting itself. Might have to investigate the Multipath tunables?


Labels:

Comments

Post a Comment

Popular posts from this blog

P2V using dd for KVM-QEMU guest

Preface: I have certainly not exhaustively tested this process.  I had a specific need and found a specific solution that worked. Situation:  I was issued a shiny new laptop running Red Hat Enterprise Linux 7 (with Corp VPN, certs, Authentication configuration, etc...)  The image was great, but I needed more flexibility on my bare metal.  So, my goal was to P2V the corporate image so I could just run it as a VM. * Remove corporate drive and install new SSD * install corp drive in external USB-3 case * Install RHEL 7 on new SSD * dd old drive to a disk-image file in a temp location which will be an image which is the same size as your actual drive (unless you have enough space in your destination to contain a temp and converted image) * convert the raw disk-image to a qcow file while pushing it to the final location - this step should reduce the disk size - however, I believe it will only reduce/collapse zero-byte blocks (not just free space - i.e. if you de...
1 comment
Read more

Extending SNMP to run arbitrary shell script

Why are we here... This is not likely something I would have pursued under normal circumstances.  I happen to be working for a customer/client who is not afforded a lot of flexibility to accomplish their goals.  In this case, the rigor is justified.  They have to sometimes be fairly creative with how they solve problems. In this case they would like to utilize an existing snmp implementation to execute a command (or shell script) on a remote system.  They came to me with the idea of using Net-SNMP extend. https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/sect-System_Monitoring_Tools-Net-SNMP-Extending.html NOTE:  This is NOT a good implementation strategy in the "real world"  it will simply allow you to test the functionality.  There are a TON of security implications which would need to be taken in to consideration. Implementation Steps: [root@rh7tst01 ~]# yum -y install net-snmp net-snmp-utils ...
Post a Comment
Read more

RHN Satellite Server (spacewalk) repomd.xml not found

"repomd.xml not found" If you add a channel, or if your RHN cache gets corrupted, and one of your guests complains that it cannot find repomd.xml for jb-ews-2-x86_64-server-5-rpm (for example) - you need to rebuild your repodata cache. Normally this is an automated job - which is exemplified by the fact that you have obviously built out your entire Satellite environment and never had to do any of the steps you are about to do. So - some prep work: Open 3 terminals to your Satellite Server and run: # Term 1 cd /var/cache/rhn watch "ls -l | wc -l" # Term 2 pwd cd /var/log/rhn tail -f rhn_taskomatic_daemon.log # Term 3 satellite-sync --channel=jb-ews-2-x86_64-server-5-rpm Once the satellite-sync has completed, you >should< see the count increment by one.  If you are unlucky (like me) you will not. You then need to login to the Satellite WebUI as the satellite admin user. Click on the Admin tab (at the top) Task Schedules (on the left) fin...
Post a Comment
Read more