DeVO's thoughts

서마이의 생각들을 담는 곳입니다…

Browsing Posts in Linux

To save the time, short story, CHECK YOUR SYSTEM TIME and MAKE SURE THEY ARE SYNC’ED!!!!!!

Yes, I am rather feeling stupid today, since I spent good 3 hours trying to figure out what is going on with an automated installation that I was doing…

2 identical hardwares, with two identical setup except the fact that one is booting from SAN and the other one is booting from Local HDD. All the software installations and configurations and scripts… everything was identical.

BUT one would register to the satellite that I kicked off the installation from, and the other one won’t

Error log was suggesting that there were SSL verification failure.
Tried to install again about 3 times with exactly the same issue.. Copied the SSL file from the one which is working to the none working… Still didn’t work… argh argh argh…..

And out of curiosity I checked the system time…
It says 2009…….. WHAT? 2009…………………………………………..
If the Bios setting was never set correctly, and the installation won’t be finished correctly….

So the answer is…
CHECK THE SYSTEM TIME!!!!!!!!!!

Because I work on a lot of automated installation, there are times when I need to grab the ip address off ifconfig then use it as a part of script.
As all other scripts do, there would be a thousand ways to script it, but since I am lazy, don’t want to rethink again and again, so keeping it here so I won’t lose :)

IP=`ifconfig ppp0 | grep “inet addr” | awk ‘{ print $2 }’ | awk ‘BEGIN { FS=”:” } { print $2 }’`

There are cases where you need to edit files in initrd, and because it says .img in the extension, it is easy to make the assumption that with a loopback mounting the file, it would enable you to be able to edit the file.
BUT think again, based on my recent experience, for RHEL5, initrd is not just a img file.

It is a combination of gzip and cpio, then how can you edit it?

# gunzip -cd /boot/initrd-<version>.img | cpio -idv

This would extract all the files from initrd, then you should be able to change things.
Then to recompress it, you need to run following command;

#  find . | cpio -co | gzip -c9 /boot/initrd-new.img

Easy? to find this detail, it took me awhile, not realizing that mkinitrd was just a script.

It gets more interesting if you want to change/move/add new modules into the initrd, after extracting the initrd file, you need to do following to extract modules then compress it again;

# gunzip -cd modules/modules.cgz | cpio -idv
# mv KERNEL-VER/ARCH/module.name.ko /tmp/
# find KERNEL-VER/ | cpio -H crc -o | gzip -9 > modules/modules.cgz
# rm -rf KERNEL-VER

Above example only covers deleting a module but it should be a good guide to do other things in it.

As it has been mentioned in earlier post, to be able to use the rsyslog as syslog in RHEL5.4/5.5, you need to make changes in /etc/sysconfig/rsyslog and /etc/rsyslog.conf.
Here, more options will be introduced;

- /etc/sysconfig/rsyslog;

SYSLOGD_OPTIONS="-c3"

-c3 : This is a new option to v3, and -c<vers> will turn the backward compatibility mode on. In version3, this will turn the backward compatibility mode, and start the native mode for version3. (RHEL5.5)

- /etc/rsyslog.conf

$ModLoad imudp
$UDPServerRun 514
$ModLoad imuxsock
$ModLoad imklog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

$ModLoad imudp – This will enable the udp mode
$UDPServerRun 514 – UDP will use port 514 to accept logging coming from remote hosts
$ModLoad imuxsock – This provides support for local system logging (e.g. via logger command)
$ModLoad imklog – To enable rklogd
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat – old syslog’s  compatible timestamp format with lower precision.

Additional Options;

$ModLoad immark - To provide -----MARK-----
$MarkMessageInterval 1800 - 1800== 30 minutes

To enable the remote logging with the same syslog structures with HOSTNAME to be used as differentiators

$template DYNmessages,"/var/log/%HOSTNAME%/messages"
$template DYNsecure,"/var/log/%HOSTNAME%/secure"
$template DYNmaillog,"/var/log/%HOSTNAME%/maillog"
$template DYNcron,"/var/log/%HOSTNAME%/cron"
$template DYNspooler,"/var/log/%HOSTNAME%/spooler"
$template DYNboot,"/var/log/%HOSTNAME%/boot.log"

if \
        $source != 'localhost' \
        and ( \
                        ($syslogseverity-text == 'info') \
                or \
                        ($syslogseverity-text == 'notice') \
        ) \
        and ( \
                        $syslogfacility-text != 'mail' \
                and \
                        $syslogfacility-text != 'authpriv' \
                and \
                        $syslogfacility-text != 'cron' \
        ) \
then    ?DYNmessages

if \
        $source != 'localhost' \
                and \
        $syslogfacility-text == 'authpriv' \
then    ?DYNsecure

if \
        $source != 'localhost' \
                and \
        $syslogfacility-text == 'mail' \
then    -?DYNmaillog

if \
        $source != 'localhost' \
                and \
        $syslogfacility-text == 'cron' \
then    ?DYNcron

if \
        $source != 'localhost' \
                and \
        (\
                $syslogfacility-text == 'uucp' \
                        or \
                $syslogfacility-text == 'news' \
        )\
                and \
        $syslogseverity-text == 'crit' \
then    ?DYNspooler

if \
        $source != 'localhost' \
                and \
        $syslogfacility-text == 'local7' \
then    ?DYNboot

To enable rsyslog forwarding through TCP port 2010;

*.* @@1.2.3.4:2010

To enable rsyslog forwarding through UDP port 514;

*.* @1.2.3.4:514

To enable rsyslog accepting through TCP port 10514;
(If you are going to use port 10514, don’t forget to add the information to /etc/services)

$modload imtcp
$InputTCPServerRun 10514

For more detailed options, following links would provide right information;

Compatibility Notes for rsyslog v3

rsyslog configuration samples

Sysklogd drop-in with remote logs separated by dynamic directory

Daily Log Rotation

rsyslog very simple config

rsyslog cookbook

Sending Messages to a Remote Syslog Server

Receiving Messages from a Remote System

rsyslog – Documentation

rsyslog sample config

While I have been working with rsyslog in RHEL5.4 and RHEL5.5, 2 main issues that I was presented with;

  1. rklogd has disappeared
  2. syslog says that it is running in compatibility mode and not working well with previous version’s config files

rsyslogd: WARNING: rsyslogd is running in compatibility mode. Automatically generated config directives may interfere with your rsyslog.conf settings. We suggest upgrading your config and adding -c3 as the first rsyslogd option.

Then looked around what actually happened, and found out that in RHEL5.5, there was a major version update from 2.x to 3.x and it seems like the QA didn’t catch some of the issues.
As the errta article mentions, it has fixed and brought a lot of nice features into the software. However, it also broke few things.
I am going to list few bugzilla entries below for anyone who is interested, but following are the options that you need to add to make sure that it works as it did in previous versions.

Please add following lines into /etc/sysconfig/rsyslog and /etc/rsyslog.conf

/etc/sysconfig/rsyslog
SYSLOGD_OPTIONS="-c3"
/etc/rsyslog.conf

$ModLoad imudp
$UDPServerRun 514
$ModLoad imuxsock
$ModLoad imklog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

Bugzilla Entries
Bug 592039 – Rklogd is gone and kernel log messages are not being logged by rsyslog
Bug 583621 – rsyslog 3.x has no longer old syslog timestamp by default

For a quite long time, I have been asked why RH wasn’t shipping syslog-ng like other distros do.
(Not that I have a great influence in what goes into RHEL, but because, I just work there, I got aske quite a lot.)

My answer was truely I don’t know.

But, since RHEL5.2 rather than including syslog-ng, rsyslog was added into RHEL5.
(If you are more interested in Fedora, this was included into Fedora 8 or Fedora 7 Rawhides, as it was pointed out in this wikipage)
This is quite interesting, because, based on the RH’s normal Practice adding a new package into RHEL is almost impossible.
Anyway so let’s look at why rsyslog was included instead of syslog-ng;

- TCP based network transport for log messages.
- Secure transport over the network.
- A realtime analysis framework for logmessages (e.g. to launch programs on alerts).
- Database backend.
- Rule (pattern) based de-multiplexing of log messages (e.g. logging to different files based on regexp).
- Backward compatibility with existing syslog configuration

Not only that, also, LWN.net has a nice little article on rsyslog’s features here.

Reasons why not syslog-ng?

  • Code Complexity
  • Performance issues
  • Incompatible format
  • Dual licensing model where adding features available in the other version might cause friction with upstream.
  • What hurts the most is the very last point, this is explained in more plain in one of the email conversations in fedora devel list;

    For one, its dual licensed. If we go adding the features that are in the non-free
    version, I think that will create bad-blood. Its configuration file does not
    appear to be backawards compatible, meaning everyone will have to go reconfigure
    their logging. Anyone that prefers syslog-ng can still use that.

    So rsyslog is in RHEL5.2 onwards, and use it!

    If you want a somewhat formal answer on rsyslog vs syslog-ng, please read a knowledgebase article;

     Where can I find syslog-ng in Red Hat Enterprise Linux 5?

    (Please note, to read above article, you may require an active RHN account)

    For last few weeks I had a ‘pleasure’ of working with rsyslog in RHEL5.4 and RHEL5.5

    As you may have noticed, it is in inverted quotes, reasons are because, it is really nice tool, but there seems to be a mishap in later version of RHEL5 how this has brought more pains for not telling people what has been changes and how to configure it.

    Due to the fact that I had to implement a centralized logging system for a customer, I had to do bit of research.

    I am going to write next few blogs on how to do and also consolidate all the links that I found useful so other people can benefit from it.

    When you create a cluster, the sole purpose is to have a fault tolerance environment. Also, means that you are trying to build a environment with no single point of failure…

    So for all the network you would introduce bonding/channel bonding….

    But, what some people may not realize is that, you also need to configure your CISCO switch so it will pass multicast pings across switches or VLANs. (This is specifically required for Red Hat Cluster or OpenAIS clustering.)

    Howto do it?

    http://www.openais.org/doku.php?id=faq:cisco_switches

    It took me a while to figure it out…

    There are cases where we were only given a binary RPMs.
    And sometimes you may want to just have a look at the contents of the RPM.
    Unfortunately, without SRPM, there are very limited things that you can do.

    For example, using less, you can see some good information on the rpm;
    # less <package name>.rpm

    And if you want to actually, look at the files in rpm you can use rpm2cpio;
    # rpm2cpio <package name>.rpm | cpio -id

    This would ‘un-tar’ the files in to the current directory.

    However, you still needs SPEC files and proper SOURCE to rebuild the package :)

    There has been a lot of cases, where there has been issues with timekeeping for linux guests on KVM/Xen/VMware.

    The best thing that Red Hat recommends its customer is to setup NTP and get it to automatically fix it.
    But, there are also kernel options that you can pass to make it more reliable.

    The original information is from VMWare’s knowledge base article

    Here are Red Hat Enterprise Linux Related information;

    32bit Kernels

    Linux Flavor Version Kernel Parameters Notes
    RHEL
    RHEL 5.4
    No additional kernel parameters required.
    See the note below about the optional use of divider=10.
    RHEL 5.3 divider=10 clocksource=acpi_pm
    RHEL 5.2 divider=10 clocksource=acpi_pm
    RHEL 5.1 divider=10 clocksource=acpi_pm
    RHEL 5.0 clocksource=acpi_pm
    RHEL 4.8 clock=pmtmr divider=10
    RHEL 4.7 clock=pmtmr divider=10
    RHEL 4.6 clock=pmtmr
    RHEL 4.5 clock=pmtmr
    RHEL 4.4 clock=pmtmr
    RHEL 4.3 clock=pmtmr
    RHEL 4.2 clock=pmtmr
    RHEL 4.1 clock=pmtmr
    RHEL 4.0 clock=pmtmr
    RHEL 3 (All updates) No additional kernel parameters required.

    64bit Kernels

    Linux Flavor
    Version
    Kernel Parameters
    Notes
    RHEL
    RHEL 5.4
    No additional kernel parameters required.
    See the note below about the optional use of divider=10.
    RHEL 5.3 notsc divider=10
    RHEL 5.2
    notsc divider=10
    RHEL 5.1 with RHSA-2007:0993-13
    notsc divider=10
    RHEL 5.1 without RHSA-2007:0993-13
    notsc
    RHEL 5.0
    No additional kernel parameters required.
    RHEL 4.8 notsc divider=10
    RHEL 4.7
    notsc divider=10
    RHEL 4.6
    notsc
    RHEL 4.5
    notsc
    RHEL 4.4
    notsc
    RHEL 4.3
    notsc
    RHEL 4.2
    notsc
    RHEL 4.1
    Does not support notsc . Avoid using if possible.
    RHEL 4.0
    Does not support notsc . Avoid using if possible.
    RHEL 3 Update 9 with RHSA-2008-0973
    disable_lost_ticks
    RHEL 3 through Update 8 Has no workaround for lost tick overcompensation. Avoid using if possible.