DeVO's thoughts

To save the time, short story, CHECK YOUR SYSTEM TIME and MAKE SURE THEY ARE SYNC’ED!!!!!!

Yes, I am rather feeling stupid today, since I spent good 3 hours trying to figure out what is going on with an automated installation that I was doing…

2 identical hardwares, with two identical setup except the fact that one is booting from SAN and the other one is booting from Local HDD. All the software installations and configurations and scripts… everything was identical.

BUT one would register to the satellite that I kicked off the installation from, and the other one won’t

Error log was suggesting that there were SSL verification failure.
Tried to install again about 3 times with exactly the same issue.. Copied the SSL file from the one which is working to the none working… Still didn’t work… argh argh argh…..

And out of curiosity I checked the system time…
It says 2009…….. WHAT? 2009…………………………………………..
If the Bios setting was never set correctly, and the installation won’t be finished correctly….

So the answer is…
CHECK THE SYSTEM TIME!!!!!!!!!!

Because I work on a lot of automated installation, there are times when I need to grab the ip address off ifconfig then use it as a part of script.
As all other scripts do, there would be a thousand ways to script it, but since I am lazy, don’t want to rethink again and again, so keeping it here so I won’t lose

IP=`ifconfig ppp0 | grep “inet addr” | awk ‘{ print $2 }’ | awk ‘BEGIN { FS=”:” } { print $2 }’`

There are cases where you need to edit files in initrd, and because it says .img in the extension, it is easy to make the assumption that with a loopback mounting the file, it would enable you to be able to edit the file.
BUT think again, based on my recent experience, for RHEL5, initrd is not just a img file.

It is a combination of gzip and cpio, then how can you edit it?

# gunzip -cd /boot/initrd-<version>.img | cpio -idv

This would extract all the files from initrd, then you should be able to change things.
Then to recompress it, you need to run following command;

#  find . | cpio -co | gzip -c9 /boot/initrd-new.img

Easy? to find this detail, it took me awhile, not realizing that mkinitrd was just a script.

It gets more interesting if you want to change/move/add new modules into the initrd, after extracting the initrd file, you need to do following to extract modules then compress it again;

# gunzip -cd modules/modules.cgz | cpio -idv

# mv KERNEL-VER/ARCH/module.name.ko /tmp/

# find KERNEL-VER/ | cpio -H crc -o | gzip -9 > modules/modules.cgz

# rm -rf KERNEL-VER

Above example only covers deleting a module but it should be a good guide to do other things in it.

As it has been mentioned in earlier post, to be able to use the rsyslog as syslog in RHEL5.4/5.5, you need to make changes in /etc/sysconfig/rsyslog and /etc/rsyslog.conf.
Here, more options will be introduced;

- /etc/sysconfig/rsyslog;

SYSLOGD_OPTIONS="-c3"

-c3 : This is a new option to v3, and -c<vers> will turn the backward compatibility mode on. In version3, this will turn the backward compatibility mode, and start the native mode for version3. (RHEL5.5)

- /etc/rsyslog.conf

$ModLoad imudp
$UDPServerRun 514
$ModLoad imuxsock
$ModLoad imklog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

$ModLoad imudp – This will enable the udp mode
$UDPServerRun 514 – UDP will use port 514 to accept logging coming from remote hosts
$ModLoad imuxsock – This provides support for local system logging (e.g. via logger command)
$ModLoad imklog – To enable rklogd
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat – old syslog’s  compatible timestamp format with lower precision.

Additional Options;

$ModLoad immark - To provide -----MARK-----
$MarkMessageInterval 1800 - 1800== 30 minutes

To enable the remote logging with the same syslog structures with HOSTNAME to be used as differentiators

$template DYNmessages,"/var/log/%HOSTNAME%/messages"
$template DYNsecure,"/var/log/%HOSTNAME%/secure"
$template DYNmaillog,"/var/log/%HOSTNAME%/maillog"
$template DYNcron,"/var/log/%HOSTNAME%/cron"
$template DYNspooler,"/var/log/%HOSTNAME%/spooler"
$template DYNboot,"/var/log/%HOSTNAME%/boot.log"

if \
        $source != 'localhost' \
        and ( \
                        ($syslogseverity-text == 'info') \
                or \
                        ($syslogseverity-text == 'notice') \
        ) \
        and ( \
                        $syslogfacility-text != 'mail' \
                and \
                        $syslogfacility-text != 'authpriv' \
                and \
                        $syslogfacility-text != 'cron' \
        ) \
then    ?DYNmessages

if \
        $source != 'localhost' \
                and \
        $syslogfacility-text == 'authpriv' \
then    ?DYNsecure

if \
        $source != 'localhost' \
                and \
        $syslogfacility-text == 'mail' \
then    -?DYNmaillog

if \
        $source != 'localhost' \
                and \
        $syslogfacility-text == 'cron' \
then    ?DYNcron

if \
        $source != 'localhost' \
                and \
        (\
                $syslogfacility-text == 'uucp' \
                        or \
                $syslogfacility-text == 'news' \
        )\
                and \
        $syslogseverity-text == 'crit' \
then    ?DYNspooler

if \
        $source != 'localhost' \
                and \
        $syslogfacility-text == 'local7' \
then    ?DYNboot

To enable rsyslog forwarding through TCP port 2010;

*.* @@1.2.3.4:2010

To enable rsyslog forwarding through UDP port 514;

*.* @1.2.3.4:514

To enable rsyslog accepting through TCP port 10514;
(If you are going to use port 10514, don’t forget to add the information to /etc/services)

$modload imtcp
$InputTCPServerRun 10514

For more detailed options, following links would provide right information;

Compatibility Notes for rsyslog v3

rsyslog configuration samples

Sysklogd drop-in with remote logs separated by dynamic directory

Daily Log Rotation

rsyslog very simple config

rsyslog cookbook

Sending Messages to a Remote Syslog Server

Receiving Messages from a Remote System

rsyslog – Documentation

rsyslog sample config

While I have been working with rsyslog in RHEL5.4 and RHEL5.5, 2 main issues that I was presented with;

1. rklogd has disappeared
2. syslog says that it is running in compatibility mode and not working well with previous version’s config files

rsyslogd: WARNING: rsyslogd is running in compatibility mode. Automatically generated config directives may interfere with your rsyslog.conf settings. We suggest upgrading your config and adding -c3 as the first rsyslogd option.

Then looked around what actually happened, and found out that in RHEL5.5, there was a major version update from 2.x to 3.x and it seems like the QA didn’t catch some of the issues.
As the errta article mentions, it has fixed and brought a lot of nice features into the software. However, it also broke few things.
I am going to list few bugzilla entries below for anyone who is interested, but following are the options that you need to add to make sure that it works as it did in previous versions.

Please add following lines into /etc/sysconfig/rsyslog and /etc/rsyslog.conf

/etc/sysconfig/rsyslog
SYSLOGD_OPTIONS="-c3"

/etc/rsyslog.conf

$ModLoad imudp
$UDPServerRun 514
$ModLoad imuxsock
$ModLoad imklog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

Bugzilla Entries
Bug 592039 – Rklogd is gone and kernel log messages are not being logged by rsyslog
Bug 583621 – rsyslog 3.x has no longer old syslog timestamp by default

For last few weeks I had a ‘pleasure’ of working with rsyslog in RHEL5.4 and RHEL5.5

As you may have noticed, it is in inverted quotes, reasons are because, it is really nice tool, but there seems to be a mishap in later version of RHEL5 how this has brought more pains for not telling people what has been changes and how to configure it.

Due to the fact that I had to implement a centralized logging system for a customer, I had to do bit of research.

I am going to write next few blogs on how to do and also consolidate all the links that I found useful so other people can benefit from it.

There are cases where we were only given a binary RPMs.
And sometimes you may want to just have a look at the contents of the RPM.
Unfortunately, without SRPM, there are very limited things that you can do.

For example, using less, you can see some good information on the rpm;
# less <package name>.rpm

And if you want to actually, look at the files in rpm you can use rpm2cpio;
# rpm2cpio <package name>.rpm | cpio -id

This would ‘un-tar’ the files in to the current directory.

However, you still needs SPEC files and proper SOURCE to rebuild the package

There has been a lot of cases, where there has been issues with timekeeping for linux guests on KVM/Xen/VMware.

The best thing that Red Hat recommends its customer is to setup NTP and get it to automatically fix it.
But, there are also kernel options that you can pass to make it more reliable.

The original information is from VMWare’s knowledge base article

Here are Red Hat Enterprise Linux Related information;

32bit Kernels

64bit Kernels

Background info:
I am working at a site that requires JDK to be deployed by Red Hat Satellite Automatically. So I have resigned the package and try to deploy, and it will fail with following error msg;
warning: jdk-1.6.0_18-fcs.x86_64.rpm: Header V3 DSA signature: NOKEY, key ID XXXXXXXX

To Check which version of RPM was used to sign rpm;

# rpm -q --qf '%{RPMVERSION}' -p jdk-1.6.0_18-fcs.x86_64.rpm
3.0.6

Question: Can you resign a binary rpm that was built with rpm 3.0.x?

Short Answer: You are in bad luck!
Well, it will let you, but it doesn’t mean that it can be used. You can’t do it, this is due to the fact that 3.0.x is so long ago, and technical feature has been added to RPM and due to that, it won’t work properly.

Long Answer:

There is a deep design issue underlying this mess.

RPM has been mired between header+payload and header-only signatures for many
years now.

In order to preserve header+payload MD5 as a universal invariant independent of
what is implemented in rpm when resigning, the legacy header which is part
of the header+payload MD5 digest cannot be changed.

In order to support header-only signatures on an immutable region,
markers are added to the header to identify the immutable header-only region
blob that is signed.

The conclusion is that header-only signature/digest should not be attempted
when resigning legacy packages. Any other scheme will either add Yet Another
Package format special case, or otherwise change the invariant header+payload
MD5 digest.

And the right “fix” is to dump header+payload signatures entirely. There is a deep design issue underlying this mess.

RPM has been mired between header+payload and header-only signatures for many
years now.

In order to preserve header+payload MD5 as a universal invariant independent of
what is implemented in rpm when resigning, the legacy header which is part
of the header+payload MD5 digest cannot be changed.

In order to support header-only signatures on an immutable region,
markers are added to the header to identify the immutable header-only region
blob that is signed.

The conclusion is that header-only signature/digest should not be attempted
when resigning legacy packages. Any other scheme will either add Yet Another
Package format special case, or otherwise change the invariant header+payload
MD5 digest.

And the right “fix” is to dump header+payload signatures entirely.

There are other payload complications that need solving to preserve
header+paload md5 invariance (for true legacy compatibility) as well.
E.g. invariance assumes that zlib never changes what is written to
a resigned package for all rpm implementations.

I’d suggest that the one line patch to handle the region marker change
in the signature header, and living with the accurate (because legacy headers
are invariant all versions of rpm afaik) but mysterious (essential elements to
compute a header-only signature/digest are not present in the definition
of the header, perhaps something other than NOKEY to be returned)
Header V3 DSA signature: NOKEY, key ID 831ffbca
is the only sane path forward.

But feel free to do whatever you want, patches cheerfully accepted!

Above are the comments from Jeff Johnson in Red Hat’s Bugzilla #127113

So the fix?
Resign the pkg with rpm 3.0.x which is Red Hat 6.2 which I don’t think anyone would see it as an answer.

Then only option is ignoring GPG signature path…

Today, I bought a new HDD for my X200.
2.5″ Seagate 500G, it didn’t cost me much, about AUD117.50.

So while I was reinstalling Fedora 11 on my laptop, a colleague of mine asked a question,
“What is the CPU flag that tells the CPU is capable of 64bit?”
Hmm.. the question stumped me, so I started to google and found following site;

http://www.gentoo-wiki.info/Gentoo:/proc/cpuinfo

Also, To outline what those are;

flags

The flags line tells us what extra instructions the processor has to offer. This is a listing of flags with a short explanation of each. There is an even shorter explanation in the file /usr/include/asm/cpufeature.h.
<edit: For F11, the header is in “kernel-devel”, and located in “/usr/src/kernels/$(uname -r)/arch/x86/include/asm/cpufeature.h”>

Intel flags (This table is currently identical with /usr/include/asm/cpufeature.h. Hopefully some hardware god will share his wisdom and expand this table. )