DeVO's thoughts

서마이의 생각들을 담는 곳입니다…

To save the time, short story, CHECK YOUR SYSTEM TIME and MAKE SURE THEY ARE SYNC’ED!!!!!!

Yes, I am rather feeling stupid today, since I spent good 3 hours trying to figure out what is going on with an automated installation that I was doing…

2 identical hardwares, with two identical setup except the fact that one is booting from SAN and the other one is booting from Local HDD. All the software installations and configurations and scripts… everything was identical.

BUT one would register to the satellite that I kicked off the installation from, and the other one won’t

Error log was suggesting that there were SSL verification failure.
Tried to install again about 3 times with exactly the same issue.. Copied the SSL file from the one which is working to the none working… Still didn’t work… argh argh argh…..

And out of curiosity I checked the system time…
It says 2009…….. WHAT? 2009…………………………………………..
If the Bios setting was never set correctly, and the installation won’t be finished correctly….

So the answer is…
CHECK THE SYSTEM TIME!!!!!!!!!!

Because I work on a lot of automated installation, there are times when I need to grab the ip address off ifconfig then use it as a part of script.
As all other scripts do, there would be a thousand ways to script it, but since I am lazy, don’t want to rethink again and again, so keeping it here so I won’t lose :)

IP=`ifconfig ppp0 | grep “inet addr” | awk ‘{ print $2 }’ | awk ‘BEGIN { FS=”:” } { print $2 }’`

There are cases where you need to edit files in initrd, and because it says .img in the extension, it is easy to make the assumption that with a loopback mounting the file, it would enable you to be able to edit the file.
BUT think again, based on my recent experience, for RHEL5, initrd is not just a img file.

It is a combination of gzip and cpio, then how can you edit it?

# gunzip -cd /boot/initrd-<version>.img | cpio -idv

This would extract all the files from initrd, then you should be able to change things.
Then to recompress it, you need to run following command;

#  find . | cpio -co | gzip -c9 /boot/initrd-new.img

Easy? to find this detail, it took me awhile, not realizing that mkinitrd was just a script.

It gets more interesting if you want to change/move/add new modules into the initrd, after extracting the initrd file, you need to do following to extract modules then compress it again;

# gunzip -cd modules/modules.cgz | cpio -idv
# mv KERNEL-VER/ARCH/module.name.ko /tmp/
# find KERNEL-VER/ | cpio -H crc -o | gzip -9 > modules/modules.cgz
# rm -rf KERNEL-VER

Above example only covers deleting a module but it should be a good guide to do other things in it.

I like my mac to do documentations and photo editing.

For my next week’s training preparation, I had to do a lot of screenshots.
With Windows and Linux, it is quite straight forward to take a screenshot, but realized that it wasn’t as straight forward.

Followings are the key shortcuts that I found through google :)

  • Command-Shift-3: Take a screenshot of the screen, and save it as a file on the desktop
  • Command-Shift-4, then select an area: Take a screenshot of an area and save it as a file on the desktop
  • Command-Shift-4, then space, then click a window: Take a screenshot of a window and save it as a file on the desktop
  • Command-Control-Shift-3: Take a screenshot of the screen, and save it to the clipboard
  • Command-Control-Shift-4, then select an area: Take a screenshot of an area and save it to the clipboard
  • Command-Control-Shift-4, then space, then click a window: Take a screenshot of a window and save it to the clipboard

I have been using Adobe Lightroom for a while.
Even though it is not feature-rich as Photoshop, but this does everything that I need to do for my little photography hobby.

But what is great about is the preset capability.
Like photoshop’s action, it can easily put effects.

Below is the site that I use a lot to get latest and best presets for lightroom

http://www.presetsheaven.com/category/presets/

As it has been mentioned in earlier post, to be able to use the rsyslog as syslog in RHEL5.4/5.5, you need to make changes in /etc/sysconfig/rsyslog and /etc/rsyslog.conf.
Here, more options will be introduced;

- /etc/sysconfig/rsyslog;

SYSLOGD_OPTIONS="-c3"

-c3 : This is a new option to v3, and -c<vers> will turn the backward compatibility mode on. In version3, this will turn the backward compatibility mode, and start the native mode for version3. (RHEL5.5)

- /etc/rsyslog.conf

$ModLoad imudp
$UDPServerRun 514
$ModLoad imuxsock
$ModLoad imklog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

$ModLoad imudp – This will enable the udp mode
$UDPServerRun 514 – UDP will use port 514 to accept logging coming from remote hosts
$ModLoad imuxsock – This provides support for local system logging (e.g. via logger command)
$ModLoad imklog – To enable rklogd
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat – old syslog’s  compatible timestamp format with lower precision.

Additional Options;

$ModLoad immark - To provide -----MARK-----
$MarkMessageInterval 1800 - 1800== 30 minutes

To enable the remote logging with the same syslog structures with HOSTNAME to be used as differentiators

$template DYNmessages,"/var/log/%HOSTNAME%/messages"
$template DYNsecure,"/var/log/%HOSTNAME%/secure"
$template DYNmaillog,"/var/log/%HOSTNAME%/maillog"
$template DYNcron,"/var/log/%HOSTNAME%/cron"
$template DYNspooler,"/var/log/%HOSTNAME%/spooler"
$template DYNboot,"/var/log/%HOSTNAME%/boot.log"

if \
        $source != 'localhost' \
        and ( \
                        ($syslogseverity-text == 'info') \
                or \
                        ($syslogseverity-text == 'notice') \
        ) \
        and ( \
                        $syslogfacility-text != 'mail' \
                and \
                        $syslogfacility-text != 'authpriv' \
                and \
                        $syslogfacility-text != 'cron' \
        ) \
then    ?DYNmessages

if \
        $source != 'localhost' \
                and \
        $syslogfacility-text == 'authpriv' \
then    ?DYNsecure

if \
        $source != 'localhost' \
                and \
        $syslogfacility-text == 'mail' \
then    -?DYNmaillog

if \
        $source != 'localhost' \
                and \
        $syslogfacility-text == 'cron' \
then    ?DYNcron

if \
        $source != 'localhost' \
                and \
        (\
                $syslogfacility-text == 'uucp' \
                        or \
                $syslogfacility-text == 'news' \
        )\
                and \
        $syslogseverity-text == 'crit' \
then    ?DYNspooler

if \
        $source != 'localhost' \
                and \
        $syslogfacility-text == 'local7' \
then    ?DYNboot

To enable rsyslog forwarding through TCP port 2010;

*.* @@1.2.3.4:2010

To enable rsyslog forwarding through UDP port 514;

*.* @1.2.3.4:514

To enable rsyslog accepting through TCP port 10514;
(If you are going to use port 10514, don’t forget to add the information to /etc/services)

$modload imtcp
$InputTCPServerRun 10514

For more detailed options, following links would provide right information;

Compatibility Notes for rsyslog v3

rsyslog configuration samples

Sysklogd drop-in with remote logs separated by dynamic directory

Daily Log Rotation

rsyslog very simple config

rsyslog cookbook

Sending Messages to a Remote Syslog Server

Receiving Messages from a Remote System

rsyslog – Documentation

rsyslog sample config

While I have been working with rsyslog in RHEL5.4 and RHEL5.5, 2 main issues that I was presented with;

  1. rklogd has disappeared
  2. syslog says that it is running in compatibility mode and not working well with previous version’s config files

rsyslogd: WARNING: rsyslogd is running in compatibility mode. Automatically generated config directives may interfere with your rsyslog.conf settings. We suggest upgrading your config and adding -c3 as the first rsyslogd option.

Then looked around what actually happened, and found out that in RHEL5.5, there was a major version update from 2.x to 3.x and it seems like the QA didn’t catch some of the issues.
As the errta article mentions, it has fixed and brought a lot of nice features into the software. However, it also broke few things.
I am going to list few bugzilla entries below for anyone who is interested, but following are the options that you need to add to make sure that it works as it did in previous versions.

Please add following lines into /etc/sysconfig/rsyslog and /etc/rsyslog.conf

/etc/sysconfig/rsyslog
SYSLOGD_OPTIONS="-c3"
/etc/rsyslog.conf

$ModLoad imudp
$UDPServerRun 514
$ModLoad imuxsock
$ModLoad imklog
$ActionFileDefaultTemplate RSYSLOG_TraditionalFileFormat

Bugzilla Entries
Bug 592039 – Rklogd is gone and kernel log messages are not being logged by rsyslog
Bug 583621 – rsyslog 3.x has no longer old syslog timestamp by default

For a quite long time, I have been asked why RH wasn’t shipping syslog-ng like other distros do.
(Not that I have a great influence in what goes into RHEL, but because, I just work there, I got aske quite a lot.)

My answer was truely I don’t know.

But, since RHEL5.2 rather than including syslog-ng, rsyslog was added into RHEL5.
(If you are more interested in Fedora, this was included into Fedora 8 or Fedora 7 Rawhides, as it was pointed out in this wikipage)
This is quite interesting, because, based on the RH’s normal Practice adding a new package into RHEL is almost impossible.
Anyway so let’s look at why rsyslog was included instead of syslog-ng;

- TCP based network transport for log messages.
- Secure transport over the network.
- A realtime analysis framework for logmessages (e.g. to launch programs on alerts).
- Database backend.
- Rule (pattern) based de-multiplexing of log messages (e.g. logging to different files based on regexp).
- Backward compatibility with existing syslog configuration

Not only that, also, LWN.net has a nice little article on rsyslog’s features here.

Reasons why not syslog-ng?

  • Code Complexity
  • Performance issues
  • Incompatible format
  • Dual licensing model where adding features available in the other version might cause friction with upstream.
  • What hurts the most is the very last point, this is explained in more plain in one of the email conversations in fedora devel list;

    For one, its dual licensed. If we go adding the features that are in the non-free
    version, I think that will create bad-blood. Its configuration file does not
    appear to be backawards compatible, meaning everyone will have to go reconfigure
    their logging. Anyone that prefers syslog-ng can still use that.

    So rsyslog is in RHEL5.2 onwards, and use it!

    If you want a somewhat formal answer on rsyslog vs syslog-ng, please read a knowledgebase article;

     Where can I find syslog-ng in Red Hat Enterprise Linux 5?

    (Please note, to read above article, you may require an active RHN account)

    For last few weeks I had a ‘pleasure’ of working with rsyslog in RHEL5.4 and RHEL5.5

    As you may have noticed, it is in inverted quotes, reasons are because, it is really nice tool, but there seems to be a mishap in later version of RHEL5 how this has brought more pains for not telling people what has been changes and how to configure it.

    Due to the fact that I had to implement a centralized logging system for a customer, I had to do bit of research.

    I am going to write next few blogs on how to do and also consolidate all the links that I found useful so other people can benefit from it.

    When you create a cluster, the sole purpose is to have a fault tolerance environment. Also, means that you are trying to build a environment with no single point of failure…

    So for all the network you would introduce bonding/channel bonding….

    But, what some people may not realize is that, you also need to configure your CISCO switch so it will pass multicast pings across switches or VLANs. (This is specifically required for Red Hat Cluster or OpenAIS clustering.)

    Howto do it?

    http://www.openais.org/doku.php?id=faq:cisco_switches

    It took me a while to figure it out…