Skip to main content

Prelink cache - friend or foe?

Ran in to a very bothersome issue over the weekend.

Pushed a package installation automatically via Satellite to ALL of my Linux hosts at 11pm on Sunday evening.  Around 9am on Monday I was invited to a bridge to discuss some Oracle issue where no new connections were allowed to the database on the Listener.  The fun begins...
So - we discuss what changes had been applied recently and my software push is mentioned.  I had tested this software push on all our non-prod hosts 8 weeks prior - and everything has been fine.  So.. what the hell happened.

As it turns out the number of hosts affected were less than 1% - However, the way the problem was communicated made the problem sound much worse.  So - at this point I assume that the approved deployment to ALL our hosts broke EVERY.SINGLE.BOX in our environment.

I start to review the actual output of the software push and I notice a package that had been updated which was not included in my testing 8 weeks ago.  PCRE...  So, I pull that RPM down and look at the scripts and I notice that it runs ldconfig.  I also check all files that were updated in /etc (using find) and notice that /etc/prelink.cache had been updated.  I review the prelink cache and everything appears fine, but.. that file is just a collection of "paths" in a binary/data file.

We find a Red Hat article indicating that if you have an Oracle host, with and SGA greater than 200GB... that you should disable prelink cache.  Huh?

I uninstall all the packages from the night before - problem persists.  Ugg..
I then disable the prelink cache
# prelink -ua
# ls -l /etc | grep prelink

Problem goes away.  Progress... sort of?
I recreate the cache...

# prelink -va
# ls -l /etc | grep prelink

Still no issue.  Hmm.. So, I re-install all the packages from the earlier deployment.  Still no issue.

Summary:  Somehow the /etc/prelink.cache file became invalid, it would seem.  I'm not sure if there was a special character in there.. or if the path was screwed up by a new package installation (making Oracle find a binary in the wrong order).

Anyhow - I recommend Google Searching whether prelinking is a good/bad thing.  It's quite a spectacle to see all the discussions out there.  I'm still on the fence - but I'm leaning towards thinking that the prelinking is a good thing.
Labels:

Comments

Post a Comment

Popular posts from this blog

P2V using dd for KVM-QEMU guest

Preface: I have certainly not exhaustively tested this process.  I had a specific need and found a specific solution that worked. Situation:  I was issued a shiny new laptop running Red Hat Enterprise Linux 7 (with Corp VPN, certs, Authentication configuration, etc...)  The image was great, but I needed more flexibility on my bare metal.  So, my goal was to P2V the corporate image so I could just run it as a VM. * Remove corporate drive and install new SSD * install corp drive in external USB-3 case * Install RHEL 7 on new SSD * dd old drive to a disk-image file in a temp location which will be an image which is the same size as your actual drive (unless you have enough space in your destination to contain a temp and converted image) * convert the raw disk-image to a qcow file while pushing it to the final location - this step should reduce the disk size - however, I believe it will only reduce/collapse zero-byte blocks (not just free space - i.e. if you de...
1 comment
Read more

Extending SNMP to run arbitrary shell script

Why are we here... This is not likely something I would have pursued under normal circumstances.  I happen to be working for a customer/client who is not afforded a lot of flexibility to accomplish their goals.  In this case, the rigor is justified.  They have to sometimes be fairly creative with how they solve problems. In this case they would like to utilize an existing snmp implementation to execute a command (or shell script) on a remote system.  They came to me with the idea of using Net-SNMP extend. https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Deployment_Guide/sect-System_Monitoring_Tools-Net-SNMP-Extending.html NOTE:  This is NOT a good implementation strategy in the "real world"  it will simply allow you to test the functionality.  There are a TON of security implications which would need to be taken in to consideration. Implementation Steps: [root@rh7tst01 ~]# yum -y install net-snmp net-snmp-utils ...
Post a Comment
Read more

RHN Satellite Server (spacewalk) repomd.xml not found

"repomd.xml not found" If you add a channel, or if your RHN cache gets corrupted, and one of your guests complains that it cannot find repomd.xml for jb-ews-2-x86_64-server-5-rpm (for example) - you need to rebuild your repodata cache. Normally this is an automated job - which is exemplified by the fact that you have obviously built out your entire Satellite environment and never had to do any of the steps you are about to do. So - some prep work: Open 3 terminals to your Satellite Server and run: # Term 1 cd /var/cache/rhn watch "ls -l | wc -l" # Term 2 pwd cd /var/log/rhn tail -f rhn_taskomatic_daemon.log # Term 3 satellite-sync --channel=jb-ews-2-x86_64-server-5-rpm Once the satellite-sync has completed, you >should< see the count increment by one.  If you are unlucky (like me) you will not. You then need to login to the Satellite WebUI as the satellite admin user. Click on the Admin tab (at the top) Task Schedules (on the left) fin...
Post a Comment
Read more