Petri Mäkijärvi - November 2005

ESRF/Linux is a version control scheme used at the European Synchrotron Radiation Facility to manage the Linux workstation and server installations. This document describes how to prepare a KickStart installation CD-ROM juke-box server for Fully Automated Installation (FAI) of multiple versions of Red Hat Enterprise Linux 4 WS operating system.

This is a public document to share our knowledge about the networked KickStart installation method, with minimum of dependencies to the ESRF/Linux version control scheme, which is an internal standard.

Table of Contents:

^Introduction

Where the Red Hat Enterprise Linux 4 System Administration Guide's KickStart section talks about "Making the Installation Tree Available" they mean that you copy the contents of the Red Hat installation CD-ROMs on a file server. When the system gets installed from the Installation Tree, it contains out of date packages and configuration issues. These will get fixed when you attach the system to the Red Hat Network and run the up2date process which will keep your system up to date. If this is what you want you can stop reading now and stick to the Red Hat Network up2date method.

In the ESRF/Linux project we developed another approach where only a single machine is maintained using the Red Hat Network up2date method. At a given time we freeze the system by stopping the up2date processing. Using the frozen machine as a source, we create a new Red Hat distribution.

The aim is to build several Red Hat distributions, similar to the one available on the Red Hat ISO CD image. The distributions are given a version number and they are stored on a networked server, called (here) KickStart Installation Tree server in a juke-box like tree structure.

The systems which will be installed from the KickStart Installation Tree Server's juke-box are immediately in a known state and they do not have to run the Red Hat Network up2date method (but they can, if needed). This contributes to a better application level stability without having to go for a NFS-root type of solution (which is also used at the ESRF).

^Version Numbering

The below table illustrates the version numbering scheme used in this document. It is taken directly from the ESRF/Linux version management scheme. Nothing prevents to use an other version numbering scheme, for example name a version as SQL345A or anything else. It is just that the below descriptions and examples will use the ESRF/Linux version numbering.

ESRF/Linux version CVS tag Patch path Apache link SNMP data
v1.2.3 v1_2_3 /dist/r1/v2/p3 r1v2p3 1 2 3

^Network Boot Method

The target computers of the described networked Red Hat KickStart installation method should be PXE (Pre-boot Execution Environment) compliant. A pre-boot operating system is needed to launch a Red Hat KickStart Linux kernel. At the ESRF we are using Rembo Toolkit 2.0 servers to answer the PXE boot requests. On the pre-boot Rembo OS we are running an open-source application called The Rembo Wizard, which has a KickStart support module.

An alternative networked boot method discussed in this document is to use PXELINUX. Examples are given for this method where applicable. It is also possible to prepare a CD-ROM for installation with the prepared KickStart kernel on it and forget the PXE based network boot. This document discuss only the network boot method, however.

^Networked KickStart Installation Flow Diagram

  1. A network identity is obtained from the DHCP server.
     
  2. The machine is started with The Rembo Wizard using PXE network boot.
    Alternatively, PXELINUX is loaded using TFTP protocol.
     
  3. Available Red Hat Enterprise Linux versions are stored on a KickStart Installation Tree Server.
     
  4. The versions are stored on the server as a set of KickStart installation CD-ROMs.
     
  5. The Rembo Wizard displays a simple menu that allows the person to select the type of the installation (workstation, control, server, ...) and the version to install.
    Alternatively, a fully automated default installation can be done without human intervention both with The Rembo Wizard or with the PXELINUX installation methods.
     
  6. Once the installation is started The Rembo Wizard or the PXELINUX searches a KickStart Linux kernel from the Rembo server and launches it.
    In the case of The Rembo Wizard, the kernel is fetched from a local network Rembo server. With the PXELINUX, the KickStart Installation Tree server should be configured as a TFTP/BOOTP server, providing the KickStart Linux kernel.
    Once launched, the KickStart Linux kernel works with the KickStart Installation Tree Server using HTTP-protocol. It gets all its arguments about which installation to do, which server to use and so on from The Rembo Wizard or from the PXELINUX as kernel arguments.
     
  7. Vendor (RedHat) packages (.rpm files) gets installed from the selected Red Hat Enterprise Linux version.
     
  8. Instead of asking the person who is doing the installation about specific decisions regarding the hard disk partitioning, network configuration, KickStart Answer Files contains the configuration information adapted to the existing infrastructure.
     
  9. Once the system is up and running, the person who is doing the installation can run some post-installation scripts that further tailor the system. The post-install operations are not discussed in this paper.

^Anaconda Compatibility Issues

Anaconda is Red Hat's installer package. It is heavily Python based. It uses many high level software packages, such as Gtk but it needs also device drivers and such from the base installation. Although in theory it is possible to use an Anaconda installer from an earlier release to install an actualized Red Hat distribution, this will only work to a certain extent.

The main problem is Python dependencies in Anaconda. When a new distribution is created, it always needs some packages from the machine on which it is created. For example, you may update Python and Gtk libraries in the distribution images that Anaconda is using during the installation. If you now launch an installation using the Anaconda installer that originates from the Red Hat installation ISO CD there is a chance that the installation will fail due to incompatibility issues in the Gtk libraries.

The obvious reason for the above incompatibility problem is the Gtk package upgrade. In order to take into account a new RPM package that you want to put into a distribution that you are preparing, you need to run an utility called genhdlist to regenerate the RPM package database in the KickStart installation tree. In consequence, stage2.img gets altered. When the Anaconda installer gets in execution, there is an incompatibility between some of the function calls in it and the Gtk run-time library.

^Installer - Installed Version Coherence

From the compatibility issues explained above we can conclude that the Anaconda installer must be recompiled (or regenerated) to match the version that it is installing. Furthermore, the version that we are installing must correspond to the version of the machine on which we are compiling the Anaconda installer. This is because to compile the Anaconda installer, we are actually using the Anaconda installer's run-time that should be installed on the machine on which the Red Hat distribution is located.

Below we explain how we get all the packages for the Red Hat distribution and how to make sure that the server on which we are working uses the very same packages. Then the Installer - Installed Version Coherence is obtained using a script that not only rebuilds the RPM databases for the installation but also rebuilds the Anaconda installer using the packages that we plan to install.

^Creating The Installation Tree

A master system needs to be installed with Red Hat Enterprise Linux 4 WS distribution. Installation is done this time from the Red Hat CD-ROMs. We want all possible RPM packages installed on this system. After the installation the system declared on Red Hat Network and we use up2date to get the latest RPM packages installed on the system. The installed system is intended to become a KickStart Installation Tree server.

The KickStart Installation Tree is created usually on the second, dedicated data disk of the server, mount as /dist. Some of the following actions are done as root (marked with a blue font). Most of the actions are destined to be done as you (marked with a green font) in order to allow you to work with a version control system, such as CVS to manage the configuration files.

^Create /dist - The Installation Tree File System

It is recommended that the /dist file system is stored on a separate disk than the rest of the system. This way, the /dist file system is backed up daily while the system disk is just backed up after the installation.

cd /
mkdir /dist
chown yourname.yourgroup dist
chmod 775 dist

For example, you have a secondary SCSI disk with a large, formatted EXT3 file system, verify that you have an entry such as

/dev/sdb2 /dist ext3 defaults 1 1

in /etc/fstab file.

^Create Global Level mydist Directory

We would have a directory for the common level scripts and configuration file in the root of the Installation Tree File System. The directory will be called here as mydist.

cd /dist
mkdir -p mydist/distbin
chmod 775 mydist
cd mydist
mkdir distcnf
chmod 775 dist*

In the absence of the CVS version control system, the above directories will be populated below, as we proceed with the instructions. There is a download link for each script or configuration file example discussed.

^Create Initial Version Path

You would create the directory structure using the nomenclature rx/vy/pz, where x = release number, y = version number and z = patch level number. This is the base of the Red Hat KickStart installation juke-box, i.e. the installation CD-ROM organized in structured manner. The first patch level to create is level number 0 (zero).

Note: You can change the version numbering scheme but you have to modify the example scripts accordingly.

cd /dist
mkdir -p r1/v1/p0
chmod -R 775 r1
cd r1/v1/p0

There will be scripts and configuration files that are directly attached to the released version. Let's create appropriate directories for them.

mkdir -p mydist/ksbin
chmod 775 mydist
cd mydist
mkdir kscnf
chmod 775 dist*

Next we need to install an initial Red Hat installation tree contents. That can be found from the Red Hat Enterprise Linux 4 WS distribution's ISO CD images. You have to spot the following type of directory structure from the distribution disks:

autorun README-ta.html RELEASE-NOTES-U1-en
EULA README-zh_CN.html RELEASE-NOTES-U1-en.html
GPL README-zh_TW.html RELEASE-NOTES-U1-es.html
i386-redhat-linux-gnu RedHat ..

For your information, it is precisely in the RedHat directory where most of the future work will be done. For the time being, make a directory tar-ball starting from the directory level illustrated above.

tar cvjf /tmp/rhel4ws_dist.tar.bz2 /my_cdrom_mount

Back to the new Installation Tree file system, recover the archive and install its contents to its new location

cd /dist/r1/v1/p0
tar xvjf /tmp/rhel4ws_dist.tar.bz2
rm /tmp/rhel4ws_dist.tar.bz2
cd rhel4ws

^Getting ALLRPMS

In our search of Installer - Installed version coherence we have to obtain all the available RPM packages from the Red Hat Network channel (architecture/product) on which the server we are working with has been subscribed.

Download: /dist/mydist/distbin/up2date_download_all (as text file)

Next operation will take about ten hours to complete. On one nice evening, before the sunset, issue the command

cd ~
nohup /dist/mydist/distbin/up2date_download_all &

The following morning, you would check up with the below commands to find out if the operation has worked OK. Count the number of packages, first in the download directory and then the installed packages in the system.

ls /var/spool/up2date/*.rpm | wc -l
rpm -qa | wc -l

You should have more packages retrieved than actually installed. If there is a problem, check nohup.out for errors. Copy now all the packages in ALLRPMS/i386 directory of the distribution. When going there, you would make sure that the RPMS symbolic link to points to it all right.

cd /dist/r1/v1/p0/rhel4ws/RedHat
ls -l RPMS
lrwxrwxrwx 1 root root 12 Aug 3 12:55 RPMS -> ALLRPMS/i386
cd ALLRPMS/i386
/bin/rm *.rpm
cp /var/spool/up2date/*.rpm .

^Installation Tree Server Up2Date

Update the server you are working with to the same level of RPM version that you have just retrieved, preferably during the same day. The operation should not take too long, because all the RPM packages are already in the up2date-utility's spool directory.

export DISPLAY=mymachine:0
up2date &

Let you self to be guided and upgrade to all proposed packages. Once you are done, reboot the machine.

This is the point where Red Hat Network upgrades should be stopped on the server. From now on, the server has become a KickStart Installation Tree server and it should not been modified automatically. Only patch level modifications are allowed later on if security reasons so require. Take a backup of your system disk now, using The Rembo Wizard, for example.

^Consider comps.xml Adjustements

Within the distribution, in RedHat/base directory, there is a database file called comps.xml. It contains definitions of all packages, their dependencies of each other and to which category of applications each package belongs to. In theory, by modifying the comps.xml file we can add our own packages, modify application categories, and so on. But you must remember that comps.xml is shared by all KickStart installations. Because it is quite large, it is quite error prone to modify it and the consequent problems may take a lot of time to resolve. Simple package and program group level selections can be done using the KickStart answer files which are discussed in the next chapter.

It may be necessary sometimes to fine tune the default package installation scheme. In this case the comps.xml file should be edited. Because of its complexity, it is recommended to separate the comps.xml file from the Red Hat distribution so that it can easily be placed under the CVS version control scheme.

cd /dist/r1/v1/p0/rhel4ws/RedHat/base
mv comps.xml /dist/r1/v1/p0/mydist/kscnf
ln -s ../../../mydist/kscnf/comps.xml .
chown yourname.yourgroup ../../../mydist/kscnf/comps.xml
chmod 664 ../../../mydist/kscnf/comps.xml
ls -l comps.xml
lrwxrwxrwx 1 root root 34 Oct 5 14:24 comps.xml -> ../../../mydist/kscnf/comps.xml

^Creating KickStart Configurations

Use KickStart configuration utility to extract the major characteristics of the server you are working on as root.

system-config-kickstart --generate /tmp/this.cfg

Back as you in kscnf-directory.

cd /dist/r1/v1/p0/mydist/kscnf
cp /tmp/this.cfg server.cfg

You will have now a starting point for a KickStart configuration file, server.cfg. With this KickStart answer file you can easily regenerate an other KickStart Installation Tree server. But this is probably not what you want to do right now. It is recommended that you create a KickStart answer file for default installations. Let's call it workstation.cfg.

Download: example for /dist/r1/v1/p0/mydist/kscnf/workstation.cfg KickStart answer file (as text file).

Open the server.cfg file and workstation.cfg file side by using your favorite text editor.

Note that you should not use the graphical KickStart configuration utility to modify the configuration files that are used for the installations. You may lose some important configuration parameters when saving the files.

You will notice that server.cfg is much bigger than its counterpart, which is closer to the Red Hat's default installation. This is because it contains a dependency list after the application category selection. Other KickStart configuration files use almost exclusively the application category selection. Only some few packages are added with their RPM package names. The application category selection makes reference to the comps.xml database discussed in the earlier chapter. The database is shared by all KickStart configuration files. This is why modifying comps.xml cannot be recommended to add individual package installations. Instead, declare the individual packages you need with their base name in the "%packages --resolvedeps" section.

In the server.cfg file find the section "%packages --resolvedeps". Below that line you find the list of all packages installed in the actual server. Check for the following caveats (there can be others, for you to find them from the future Red Hat systems):

  • There are multiple kernel entries, remove the duplicate ones.
  • Remove kernel-source entry (may be installed but the package is not part of Red Hat Network channel packages installed on ALLRPMS).
  • Remove  gpg-pubkey entry (because it is specific to each machine, say rpm -qa | grep pubkey to understand why).

There are some IP-address dependencies in the above workstation.cfg KickStart answer file. Please check for them and make changes to fit the configuration into your network infrastructure.

^Regenerate Anaconda

In order to maintain the Installer - Installed version coherence we want to regenerate Anaconda, the Red Hat installer based on Python. We need root privileges for that.

Download: /dist/r1/v1/p0/mydist/ksbin/make_install_images (as text file)

cd /dist/r1/v1/p0/mydist/ksbin
./make_install_images

The process takes usually more than ten minutes. Be very suspicious because this is a fragile process. Very many things can get wrong. In the end of the script, the major files are listed with a time stamp. Study the time stamps carefully. If the files have not been generated within the time span that the script has run, something has gone wrong. The script writes several log files in /tmp directory. They need to be studied in case of a problem. Do not attempt any further tests, if not all key files have been regenerated. You would just lose your time.

^Distribute Anaconda Kernel

You can skip this section if you do not plan to use The Rembo Wizard as a launching platform for KickStart kernels. But if you do, issue now the following command to distribute the generated kernel and initial ram disk files on all Rembo servers with KickStart support. You can work as you from now on.

Download:
/dist/r1/v1/p0/mydist/ksbin/upload_iso_kernel_all (as text file)
/dist/mydist/distbin/upload_on_all_remboservers (as text file)
/dist/mydist/distbin/upload_on_remboserver (as text file)

cd /dist/r1/v1/p0/mydist/ksbin
./upload_iso_kernel_all

^Declare New Distribution

This chapter discuss how the new distribution tree is declared to the KickStart installation launchers, alternatively to The Rembo Wizard's KickStart support or to the PXELINUX boot based launching method.

^Declare Distribution for The Rembo Wizard

In some cases you may want to have more than one KickStart Installation Tree server. You should name a master KickStart Installation Tree server. When you declare a new distribution you should work on the master server only. The below explanation supposes, however that the master KickStart Installation Tree server is the same machine on which you have been working on.

In the distcnf directory you would need a configuration file called kickconfig.rbc. This is the unique file for all releases, versions and patch levels, describing to Rembo servers what versions are available and on which servers. It defines the number of patch levels available for each version and convenient names for the KickStart configuration files. The contents of the kickconfig.rbc file is self-explanatory but if problems do arise, kickconfig_example.rbc file, available in The Rembo Wizard's distribution helps to understand how to declare new versions and their servers. Typos and other human mistakes can occur, but when you start your first test installation, The Rembo Wizard's KickStart support module analyzes carefully the kickconfig.rbc file and gives you an error message that allows you to pinpoint the problem.

Example: A simplified kickconfig.rbc file from The Rembo Wizard's KickStart support module documentation. To be saved and edited in /dist/mydist/distcnf directory.

cd /dist/mydist/distcnf
vi kickconfig.rbc

Once you have done with the editing kickconfig.rbc file, you should distribute it on all Rembo servers with KickStart support. This is done with the following script, still being as you.

Download: /dist/mydist/distbin/update_kickconfig (as text file)

cd /dist/mydist/distbin
./update_kickconfig

^Declare Distribution for PXELINUX boot

If you have skipped the above section you would probably want to know how to set up a PXELINUX support to boot the KickStart's Linux kernel over the network. Fortunately Red Hat System Administration Manual explains the PXE Network Installation Method in details. It may occur that you cannot find all the software explained in the documentation from the Red Hat Enterprise Linux 4 WS packaging but an ES-level packaging is required. We have downloaded the missing components from the PXELINUX home and they worked all right on a WS based server.

Tftpboot-method is far from being as flexible as The Rembo Wizard what comes to the configuration but for a smaller infrastructure quite convenient default configuration can be created using symbolic links.

The first step is to copy (instead of using symbolic links) the KickStart kernel and ramdisk, which were generated above into a directory from where PXELINUX can find them.

cd /tftpboot/images
cp /dist/r1/v1/p0/rhel4ws/isolinux/vmlinuz vmlinuz_r1v1p0
cp /dist/r1/v1/p0/rhel4ws/isolinux/initrd initrd_r1v1p0

Now you would need a PXELINUX configuration file that would allow it to load and launch the above kernel with correct arguments. Below is an example for a configuration file. Note that it is not a KickStart answer file despite similar file extension.

Download: /tftpboot/pxelinux.cfg/workstation.cfg (as text file)

Below we use the PXELINUX's ability to group action per sub-network basis and make all systems that launch PXELINUX boot on the network 192.1.1 to use the above, default configuration file.

cd /tftpboot/pxelinux.cfg
ln -s workstation.cfg C00101

^Create Web Server Link

Hold your horses before launching anything, the web server needs to access the distribution files. Usually, in Apache you have to give explicit rights for it to follow symbolic links, and especially if they are pointing outside to its intended web document file system. How exactly configure this is beyond the scope of this document because it changes from one version of Apache to an other. For an example, in  /etc/httpd/conf/httpd.conf file you may find section:

<Directory />
    Options FollowSymLinks
    AllowOverride None
</Directory>

An other method would be to use AllowOverride directive and use a .htaccess file within the document tree, defining the FollowSymLinks option for a specific directory.

Now create a symbolic link for the first patch level on the new KickStart Installation Tree server. You must work as root. The below example is for Apache 1.3 on a Red Hat Enterprise Linux 4 WS system.

cd /var/www/html
ln -s /dist/r1/v1/p0 r1v1p0

Test the above link using a web browser, point to the URL http://mysystem.mydomain.com/r1v1p0  and navigate to the KickStart installation tree (rhel4ws).

^Saving Your Work

It is highly recommended that you save your work and manage various configuration files using CVS, subversion or other similar source code version control system. This is why in the above examples you are working as you when ever possible. The implementation of the version control system within the ESRF/Linux project is so specific to our infrastructure that it will be not discussed in this document. For your information, we are using CVS in the ESRF/Linux project.

^From Patches to Versions

From this point onwards you would create new patch levels, served by the new KickStart Installation Tree server that you have just installed. Creating a new patch level is mainly to copy an existing installation tree and then replace the RPM packages in the tree with symbolic links to this first installation. The question rise when you would need to worry about the installation of a new KickStart Installation Tree server?

The rule is that a new version must be created when a new kernel version must be used.

The logic behind the above rule is in the fact that we need a new compiler farm machine to be able to compile device driver modules for the new kernel versions. As you have learned the hard way by following the above procedure, switching version is a rather complicated operation.

Therefore you would probably like to stick to new patch levels as long as you can. But you must be careful. As soon as you start to update packages used by Anaconda, such as Gtk, you are on a slippery surface. It may very well happen that the match between the Installer - Installed version coherence gets broken. If this happens you must consider creating a new version because you cannot update the installed compiler farm machine with other packages than those related to network security issues. Anyway, say to yourself that if the Anaconda installer fails to work there is a chance that an application compiled on the compiler farm machine would fail as well.

^Post-Install Configuration and Management

This chapter will discuss the automated Post-Install configuration issues of the KickStart installed systems based on the experience that we have acquired in the ESRF/Linux project. Most of the post-install configuration operations are related to the organization's computing infrastructure and therefore they are not coped with. We will discuss below about the general system management automation.

^Using SNMP Distributed Configuration Information

You may have noticed the section "%post" of the workstation.cfg file, the example KickStart answer file of this document. It stands for the KickStart post-install operations (and not for the post-install operations in general). It contains a section where we configure the SNMP daemon's local database to contain information about the installed version. The information is then available for and can be scanned with the network management tools.

But the post-installation scripts can access the local SNMP database, get information about the installation and adapt their functionality accordingly.

View an example: KSH code snippet to extract version, patch and other information from the local SNMP database

^---

Onsite information:

Please see the link in the left-hand navigation column to the internal document about the Kickstart CD-ROM Juke-Box installation. (visible to onsite visitors only).