Table of Contents

Embedded Gentoo

Name:
Embedded Gentoo
Tags:
linux, gentoo
Description:
Attempt at creating an incredibly tiny Gentoo install, suitable for embedded devices.
Last Update:
2008-03-17

CHANGELOG

2008-03-17

  • DONE
    1. Fixed attempt to read past end of device error.
    2. Fixed reboot error.
    3. Reduced size of disk image to 11M.
  • TODO
    1. Add information about creating the qemu boot scripts.
    2. Test SQLite.
    3. Experiment with different space-conservation techniques.
      • unification filesystems
      • squashfs/tmpfs storage of /var/www
      • ideas?

2008-03-12

  • DONE
    1. Add information about copying data to the hard disk and converting to qcow2.
    2. Add information about making the hard disk bootable.
  • TODO
    1. Solve attempt to read past end of device error.
      • possibly kernel-related? maybe hard disk too big and support needs to be enabled?
    2. Add information about creating the qemu boot scripts.
    3. Test SQLite.
    4. Experiment with different space-conservation techniques.
      • unification filesystems
      • squashfs/tmpfs storage of /var/www
      • ideas?

2008-03-11

  • DONE:
    1. Successfully booted the system and tested it using a QEMU emulator. Cherokee/PHP works.
    2. Added some more documentation of the process.
    3. Switched base partition to JFS format for spacial/journaling concerns.
    4. Added SquashFS-LZMA support, still requires testing.
    5. Added information regarding qemu-img virtual hard disk creation.
This page is still very much under development and should not be considered “complete” by any means. Use any advice or information provided on this site at your own risk.

Introduction

The EmGen project is an effort at creating the smallest possible, working distribution of the Gentoo Linux Distribution. This is being done using uClibc, and the current Gentoo Embedded project.

EmGen will attempt to make a fully working system, and as the project grows we will see how much we can fit in a small location.

The final product of this project currently will be to place an entire system, as well as a system emulator, on a 32MB SD Card. The finished product should be able to be inserted into a MMC/SC Card slot (such as one on a laptop) on a Windows system, and be used to run the Gentoo system.

Prerequisites

It has been determined that the following prerequisites are needed to build this project:

Requirement Notes
Hard Disk Space >1.5G You will need a lot of space to build your system. This includes compile/download space.
Network Active Internet connection You are going to need to download a lot of packages, so the faster the better.
System Linux Since you are creating a Gentoo Linux system, you will need a Linux build system.

Environment Setup

The first thing we will do is set up a build environment which will be used to build our actual system. This build environment will have to be built once and from then on can be used to create various versions of the EmGen system.

Create the Base Directory

First, be sure to create a directory that will be the base for your build system. This system should fulfill the space requirements.

export EMGEN_DIR=/emgen
mkdir -p $EMGEN_DIR
Throughout this document the EMGEN_DIR environment variable will be used in all scripts. Please make sure this is properly set to the base of your build system.

Install Stage 1 Embedded Gentoo

In order to create the build system, we must download a stage 1 tar-ball of the Embedded Gentoo system, and install it into the EmGen build environment.

Enter EmGen Build Directory

cd $EMGEN_DIR

Download Stage 1 Embedded Gentoo

Open the Gentoo Mirrors page in a browser of your choice, and select a mirror. Once logged on to the mirror, traverse the directory tree to experimental / x86 / embedded / stages and download the most recent stage1 tar-ball. In the case of this project, I will be using version 2008.0.

Some mirrors listed do not maintain a copy of the experimental branch. If you are unable to find it on your chosen mirror, select another one.
wget http://mirror.mcs.anl.gov/pub/gentoo/experimental/x86/embedded/stages/stage1-x86-uclibc-2008.0.tar.bz2

Unpack the Stage 1 Tar-Ball

If you downloaded a .tar.bz2, use the following command to unpack the file:

tar -xjpvf stage1-x86-uclibc-2008.0.tar.bz2

If you downloaded a .tar.gz or a .tgz, use the following command to unpack the file:

tar -xzpvf stage1-x86-uclibc-2008.0.tar.gz

(Optional Step) Install Portage

If you are not using a Gentoo system as the host system for this project, you will need to download a snapshot of portage. If you are using Gentoo for your host system, then you can skip this step as you will be able to simply bind your current portage tree to the build system.

Enter EmGen Build Directory

cd $EMGEN_DIR

Download Portage Snapshot

Open the Gentoo Mirrors page in a browser of your choice, and select a mirror. Once logged on to the mirror, traverse the directory tree to snapshots and download the portage-latest tar-ball.

wget http://mirror.mcs.anl.gov/pub/gentoo/snapshots/portage-latest.tar.bz2

Unpack Portage Snapshot

If you downloaded a .tar.bz2, use the following command to unpack the file:

tar -xjvf portage-latest.tar.bz2 -C $EMGEN_DIR/usr

If you downloaded a .tar.gz or a .tgz, use the following command to unpack the file:

tar -xzvf portage-latest.tar.gz -C $EMGEN_DIR/usr

Bootstrap Build System

In this stage, we will chroot into the newly created build system, and bootstrap it to enable us to use it for building the embedded target.

Chroot into the Build System

First we must set enter the Build System

Bind Directories

Bind all of the appropriate directories to the build system:

mount -o bind /proc $EMGEN_DIR/proc
mount -o bind /sys $EMGEN_DIR/sys
If your host system is a Gentoo system, be sure to bind the portage tree to the build system as well:
mkdir -p $EMGEN_DIR/usr/portage
mount -o bind /usr/portage $EMGEN_DIR/usr/portage
If you are binding your portage tree, and your distfiles directory is either on a separate partition or in a different location, be sure to bind it as well.
mkdir -p $EMGEN_DIR/usr/portage/distfiles
mount -o bind /usr/portage/distfiles $EMGEN_DIR/usr/portage/distfiles

This is recommended since it would prevent you from re-downloading any packages that you already have on your host system.

Copy Resolv.conf

Next we copy the /etc/resolv.conf file so that we maintain internet connectivity.

cp -L /etc/resolv.conf $EMGEN_DIR/etc/resolv.conf

Chroot

You should be ready to chroot now.

chroot $EMGEN_DIR /bin/bash --login

Update Environment Variables

Now that you are within the chroot, you need to update the environment variables in your system.

env-update
source /etc/profile

It is also a good idea to modify your prompt so that you don't accidentally type commands into your host system that were meant for your build system.

export PS1="(emgen-build) $PS1"

Edit Configuration Files

Now that we are in the build system, we need to update some common configuration files to ensure that the build goes alright.

/etc/make.conf

The /etc/make.conf file is one of the most important files on any Gentoo system. It sets all of the environment variables to use when installing packages.

Open the file.

nano /etc/make.conf

It will look something like this as a default:

# These settings were set by the catalyst build script that automatically built
# Please consult /etc/make.conf.example for a more detailed example
CFLAGS="-Os -pipe"
CHOST="i386-gentoo-linux-uclibc"
CXXFLAGS="-Os -pipe"
LDFLAGS="-Wl,-z,relro"

For the sake of making things easy, I would recommend changing/adding the values to make the file resemble the following:

# These settings were set by the catalyst build script that automatically built
# Please consult /etc/make.conf.example for a more detailed example
CFLAGS="-Os -pipe -march=i686 -fomit-frame-pointer"
CXXFLAGS="${CFLAGS}"
CHOST="i686-gentoo-linux-uclibc"
LDFLAGS="-Wl,-z,relro"

FEATURES="ccache buildpkg"
USE="minimal"
UCLIBC_CPU="686"
PKGDIR="${PORTDIR}/packages/uclibc"

Here is an explanation for the added/modified lines above:

  • CFLAGS
    1. -march=i686 is just a way of telling the GCC compiler to make generic optimizations for a i686-class processor (generally safe for any x86 architecture these days).
    2. -fomit-frame-pointer tells GCC to not use a register for stack tracing, which we won't need as we will be leaving out any sort of debugging for the sake of space.
  • CXXFLAGS
    1. It is generally good practice to set the CXXFLAGS equal to the CFLAGS unless you have good reason not to.
  • CHOST
    1. I set this to i686-gentoo-linux-uclibc as I wish to build a generic i686-based system. You can change this but it should reflect all of the other values (such as -march and UCLIBC_CPU).
  • LDFLAGS
    1. This I would just leave at a default.
  • FEATURES
    1. ccache tells the system that we will be using the ccache system to cache our compiles to make recompiling faster.
    2. buildpkg tells the system to use pre-built packages when possible.
  • USE
    1. minimal enables all minimalistic installs in the system.
  • UCLIBC_CPU
    1. 686 informs the uClibc library to make optimizations for the i686 architecture. If this is not set, uClibc will be very generic and also very bulky.
  • PKGDIR
    1. This is used to tell portage to store packages for this build in a separate location from the default.

Once you have finished making the modifications, hit <key>Ctrl - X</key>, <key>Y</key>, and <key>Enter</key> to exit nano and save the modifications.

/etc/make.profile

The /etc/make.profile file sets default for the /etc/make.conf file as well as some other settings. It is important to set it to the proper value for your build system. To view what it currently is, type:

ls -ld /etc/make.profile

This should be set to a generic x86 uClibc profile. To do so execute the following command:

ln -fns ../usr/portage/profiles/uclibc/x86 /etc/make.profile

It is generally a good idea to double check to make sure it worked:

ls -ld /etc/make.profile

Perform the Bootstrap

The next step in the build process is to bootstrap the build system. This must be done to create a compiler toolkit based on embedded devices.

Enter the Portage Script Directory

This directory is where the bootstrap.sh file is located, which will be used to bootstrap the system.

cd /usr/portage/scripts

Perform a Test Run

In order to see what is going to be bootstrapped, it is generally good to run a pretend run first.

./bootstrap.sh -p -v

This command will locate all of the packages you need to have installed to bootstrap the system. Take a note of this as if anything goes wrong you will need to know where to continue. In my case, the following were displayed:

First the system will update portage (the Gentoo emerge system), by installing/upgrading the following:

[ebuild     U ] sys-apps/sandbox-1.2.18.1-r2 [1.2.17] 0 kB 
[ebuild     U ] sys-apps/portage-2.1.4.4 [2.1.1-r1] USE="build -doc -epydoc% (-selinux)" LINGUAS="-pl" 0 kB 

Next, the system will be bootstrapped by installing/upgrading the following:

[ebuild  N    ] dev-util/unifdef-1.20  0 kB 
[ebuild     U ] sys-kernel/linux-headers-2.6.23-r3 [2.4.26-r1] USE="(-gcc64%)" 0 kB 
[ebuild     U ] sys-apps/texinfo-4.8-r5 [4.8-r2] USE="-build* (-nls) -static" 0 kB 
[ebuild     U ] sys-devel/binutils-2.18-r1 [2.16.1-r3] USE="-multislot -multitarget (-nls) -test -vanilla" 0 kB 
[ebuild     U ] sys-devel/gcc-4.1.2 [4.1.1] USE="bootstrap* (-altivec) -build* -d% -doc -fortran -gcj -gtk (-hardened) -ip28 -ip32r10k -libffi% -mudflap (-multilib) -multislot (-n32) (-n64) (-nls) -nocxx -objc -objc++ -objc-gc -test -vanilla" 0 kB 
[ebuild     U ] sys-libs/uclibc-0.9.28.3 [0.9.28] USE="-build* -debug -hardened (-iconv) -ipv6 -minimal (-nls) -pregen -savedconfig -uclibc-compat -userlocales -wordexp" 0 kB 
[ebuild     U ] sys-apps/sysvinit-2.86-r10 [2.86-r5] USE="(-ibm) (-selinux) -static" 0 kB 
[ebuild  N    ] virtual/init-0  0 kB 
[ebuild     U ] sys-apps/baselayout-1.12.11.1 [1.12.6] USE="bootstrap* -build* -static -unicode" 0 kB 
[ebuild   R   ] sys-libs/zlib-1.2.3-r1  USE="(-build%*)" 0 kB 

Start the Bootstrap

Once you feel confident (and have a lot of free time to fix possible errors), execute the following command to start the bootstrap process:

./bootstrap.sh
Keep in mind that since you are building an experimental system from a stage-1 tar-ball, it is very likely that something WILL go wrong.
If something does go wrong (and trust me it probably will), then you can resume the bootstrapping process by typing the above command again. It will continue on from wherever it left off.

Leave Chroot Environment

Now that we have bootstrapped the system, it is generally a good idea to leave the chroot and backup the directory so that you can set off from this point if something goes wrong in the future.

Exit Chroot

First just exit the chroot.

exit

Unmount All Bound Directories

Now you should unbind all directories that were bound when chrooting.

umount $EMGEN_DIR/proc
umount $EMGEN_DIR/sys
If you bound your portage distfiles directory, you must unbind it now.
umount $EMGEN_DIR/usr/portage/distfiles
If you bound your portage directory, you must unbind it now.
umount $EMGEN_DIR/usr/portage

Reinstall the System

Immediately after bootstrapping the system, you must then reinstall all core system libraries using the newly bootstrapped tools. This is probably the most difficult part of the entire process, as you will more than likely run into some errors.

The uClibc library is a slimmed down version of the glibc library. Most libraries are built expecting glibc to be present, so when it is not there is a high probability that the install will fail. This section requires a lot of patience and willingness to dig into the code base yourself to solve problems.

Chroot into the System

First, be sure to chroot into the system just as you did prior to bootstrapping (follow the section defined above).

MAKE SURE YOU ARE IN THE BUILD SYSTEM. If you are not inside the chroot environment when you run the following commands it could potentially break or damage your host system.

Emerge the System

In order to make sure everything on the system will run under the newly bootstrapped toolkit, we need to reinstall all system libraries. In order to do this, run the following command:

emerge -eav system

The flags passed to the emerge system mean the following:

  • -e - (emptytree) assume the dependency tree is empty, so all dependencies of all files must be installed. This forces portage to install all system files again.
  • -a - (ask) show the user what will be installed prior to starting, and let them decide if portage should continue with the install.
  • -v - (verbose) be verbose in your description of what will be installed.

Once you have viewed everything that is to be installed and made sure it is to your liking (you may wish to enable various use-flags and the like), hit <key>Y</key> or <key>Enter</key> and begin the emerge process.

Solutions to Things that May go Wrong

In a perfect world, the emerge would complete successfully and everything would work fine. Of course we are not in a perfect world, however, and these packages are very much still in the experimental stages. For that reason I have outlined in this section solutions to all of the problems that arose when I was attempting the install.

You may experience more or less of these problems stated below. Because of the nature of Gentoo and the Portage Tree, packages are constantly being tweaked and updated, and as such there is a high likely-hood that your experience will change as time goes on. I have included version numbers in this HOWTO to assist in comparing your install against mine.

What to do if the Emerge Fails

If the emerge does fail (and trust me, it probably will), do not panic. Whatever you do, do not type in emerge -eav system again! This will compute with an empty tree again and force you to start all over instead of where you left off!

In order to resume the emerge that you had before it failed, simply type:

emerge -av --resume

This will allow you to pick up where you left off.

What if you must Perform a Separate Emerge?

In the event that the only way to fix the problem is to emerge a package out-of-order. There are different ways to do this.

Performing a second emerge typically runs the risk of wiping out your possibility to resume the initial one (this could be a problem). The best way to resolve this is to run the resume command, but like this:

emerge -p --resume > remerge

This will print out a list of packages that would be installed if portage were to resume, and save the list into a file called remerge in your current directory.

Now you can emerge whatever you need to, remove those packages from the list in the remerge file, and then use that list to populate an emerge command to resume where you left off.

'AI_ADDRCONFIG' undeclared

If a build fails with a line similar to:

io.c:1150: error: 'AI_ADDRCONFIG' undeclared (first use in this function)

Then the problem is more than likely due to an outdated version of netdb.h. To fix the problem, execute the following commands:

cd /usr/include
cp ./netdb.h ./netdb.h.orig
nano ./netdb.h

Now add the following after line 403 (hitting <key>Ctrl-C</key> in nano give you the current line number):

# define AI_V4MAPPED    0x0008  /* IPv4 mapped addresses are acceptable. */
# define AI_ALL         0x0010  /* Return IPv4 mapped and IPv6 addresses. */
# define AI_ADDRCONFIG  0x0020  /* Use configuration of this host to choose returned address type. */

After adding those lines, hit <key>Ctrl-X</key>, <key>Y</key>, and <key>Enter</key> to save the file.

Now continue the emerge:

emerge -av --resume

For more information regarding this bug, please visit Bug #195368.

It is important to note that this bug may occur more than once. Since you are effectively hand-modifying an include file, there is a good chance that your modification may be overwritten during the emerge process. If that is the case then you will need to re-modify the netdb.h file.

Python Library Could not Locate 'i386-gentoo-linux-uclibc-gcc'

If a compile of a python library fails with a line stating something to the effect of:

gcc-config error: Could not run/locate "i386-gentoo-linux-uclibc-gcc"

If this is the case, then your python install is more than likely hard-coded to use an old version of GCC (the one prior to bootstrapping). To fix this you have to change it's configuration.

First run the following command:

python-config

The output should be something along the lines of:

-lpython2.4 -lm -L/usr/lib/python2.4/config

cd into the directory given that contains the configuration information:

cd /usr/lib/python2.4/config

Now run the following command to retrieve the current gcc profile:

gcc-config -c

You should see output similar to this:

i686-gentoo-linux-uclibc-4.1.2

You will need to replace the missing gcc version with the version just displayed, minus the actual version number.

Now run this command to replace all the old gcc CHOST values with the new ones:

for i in `ls -1`; do cp ${i} ${i}.bak; sed -r 's/i386-gentoo-linux-uclibc/i686-gentoo-linux-uclibc/gi' ${i}.bak > ${i}; done;

This essentially loops through all of the files in this directory and replaces the old value of i386-gentoo-linux-uclibc with the new value of i686-gentoo-linux-uclibc.

Now resume the emerge:

emerge -av --resume

Undefined Reference to '___tls_get_addr' in E2fsprogs

If your compile of sys-fs/e2fsprogs fails due to a message similar to the following:

../../lib/libuuid.so: undefined reference to '___tls_get_addr'

The problem amounts from a faulty portage ebuild which makes sys-fs/e2fsprogs be configured assuming it will be built with glibc. uClibc does not contain TLS support, which is enabled in e2fsprogs by default.

Luckily, however, the nice people at Gentoo have already been working on this and the solution is to simply unmask the more recent package that contains a fix (namely version 1.40.5-r1).

To do this all you need to do is run the following command:

echo "=sys-fs/e2fsprogs-1.40.5-r1 ~x86" >> /etc/portage/package.keywords
echo "=sys-libs/ss-1.40.5 ~x86" >> /etc/portage/package.keywords
echo "=sys-libs/com_err-1.40.5 ~x86" >> /etc/portage/package.keywords

The only problem now is that when you attempt to resume the emerge, it will try to resume based on package version, and by that turn try to resume the old version of e2fsprogs. To solve this problem, you must resume using the following command:

TMP=""; \
for i in `emerge --resume -pv | grep "\[ebuild" - | sed -r "s/\[ebuild[^]]*\] //gi" - | awk '{print $1}' - | sed -r "s/e2fsprogs-.*$/e2fsprogs-1\.40\.5-r1/gi" -`; \
do TMP="${TMP} =${i}"; \
done; \
emerge -av ${TMP}

The above code is basically a very glorified way of automating parsing through the output of emerge –resume -p, removing everything but the package atoms, changing the version of e2fsprogs, compiling them all into a single line, and running emerge again.

For more information regarding this bug, please visit Bug #205102.

'Can't locate ??? in @INC' Error

If your emerge fails with an error similar to the following:

Can't locate auto/POSIX/assert.al in @INC (@INC contains: . /etc/perl
/usr/lib/perl5/vendor_perl/5.8.8/i386-linux /usr/lib/perl5/vendor_perl/5.8.8
/usr/lib/perl5/vendor_perl /usr/lib/perl5/site_perl/5.8.8/i386-linux
/usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl /usr/lib/perl5/5.8.8/i386-linux
/usr/lib/perl5/5.8.8 /usr/local/lib/site_perl .) at ./../mk-script line 52

Then the problem is most likely related to an inadequate install of Perl. There is probably a high likelyhood that the output for emerge -p –resume includes an upgrade for Perl, but that it does not occur for awhile so the current package won't work.

The best way to get around this is to manually reorder the packages to be emerged, with all Perl packages coming next.

To do this, simply run this command:

TMP=""; \
for i in `emerge --resume -p | grep "\[ebuild" - | sed -r "s/\[ebuild[^]]*\] //gi" - | awk '{print $1}' - | grep "perl" -`; \
do TMP="${TMP} =${i}"; \
done; \
echo '#!/bin/bash' > ./install-perl.sh; \
echo "emerge -av ${TMP}" >> ./install-perl.sh; \
chmod 755 ./install-perl.sh; \
TMP=""; \
for i in `emerge --resume -p | grep "\[ebuild" - | sed -r "s/\[ebuild[^]]*\] //gi" - | awk '{print $1}' - | grep -v "perl" -`; \
do TMP="${TMP} =${i}"; \
done; \
echo '#!/bin/bash' > ./resume-nonperl.sh; \
echo "emerge -av ${TMP}" >> ./resume-nonperl.sh; \
chmod 755 ./resume-nonperl.sh

This will create two new scripts in the current directory. install-perl.sh, which runs an emerge to install all Perl-related packages that were pending in the resume, and resume-nonperl.sh, which resumes installation where it left off without the Perl packages. They should also be executable.

It is also generally a good idea to take a look at the files to make sure they were populated correctly:

cat {install-,resume-non}perl.sh

It should show something like this:

#!/bin/bash
emerge -av  =dev-lang/perl-5.8.8-r4 =perl-core/PodParser-1.35 =perl-core/Test-Harness-2.64
#!/bin/bash
emerge -av  =sys-apps/coreutils-6.9-r1 =sys-apps/groff-1.19.2-r1 =sys-apps/shadow-4.0.18.2 =sys-process/psmisc-22.6 =sys-apps/man-1.6f =sys-apps/man-pages-2.76 =sys-apps/diffutils-2.8.7-r2 =dev-lang/python-2.4.4-r6 =dev-python/python-fchksum-1.7.1 =sys-apps/file-4.23 =sys-apps/module-init-tools-3.4 =sys-apps/debianutils-2.28.2 =sys-apps/mktemp-1.5 =sys-apps/baselayout-1.12.11.1 =sys-fs/udev-115-r1

Now you should run them in succession. First, execute install-perl.sh to (re-)install Perl:

./install-perl.sh

Once this has completed (after resolving any errors or whatnot in-between), then run the resume script:

./resume-nonperl.sh
If something does go wrong while any of these scripts are running, you may fix the issue and then resume from where you left off by using
emerge -av --resume

as usual. This is because the created scripts just run emerge, and therefore if the emerge fails, it can be resumed through regular means.

Once you are done it is a good idea to remove the scripts from the filesystem:

rm ./{install-,resume-non}perl.sh

Clean-up

After everything has completed successfully, you will first have to re-source your shell so that changes in path take effect:

source /etc/profile
export PS1="(emgen-build) $PS1"

After that you will also want to update any configuration files currently held on your system:

etc-update

Since you haven't really set any configuration files, however, it should be save to just update all of the listed files automatically. So type in -5 and hit <key>Enter</key> to do so.

Check for Broken Packages

During the emerge -eav system process, it is possible that package dependencies were broken. To check for that, you must use the revdep-rebuild utility.

Emerge app-portage/gentoolkit

The revdep-rebuild program is available from the app-portage/gentoolkit package. You will need to emerge that first:

emerge -av gentoolkit

Run Revdep-Rebuild

Now that it is installed, you should run the following command to check for broken packages:

revdep-rebuild -- -av

The above command will search for any broken linkage between libraries on your system, determine which packages they belong to, and then attempt to emerge them, appending all flags after the switch to the end of the emerge command.

Update All Packages on the Build System

Now that we have a working build system with all of its core libraries bootstrapped, it is time to perform any updates to packages not declared to be part of the core system.

Install CCache

To save time on compiling we will not install the CCache compiling cache utility. This will cache compiled objects to be used during recompiles or future updates.

To install ccache, first you must emerge it:

emerge -av ccache

Now that CCache is installed, we need to set our cache size. Set it to something appropriate for your build environment. For us I will use a cache of 200M.

ccache -M 200M

Upgrade World

Now that we have our caching mechanism setup, it is time to upgrade any and all packages on our build system that are outdated. To do this we perform a global world update:

emerge -DuavN world

Once this returns you will have an up-to-date build system, capable of creating your EmGen target system.

Install a Kernel Source Tree

The next step is to install a Linux kernel source tree. There are several different trees available, although for general purposes it is usually a good idea to use the tree associated with Gentoo, gentoo-sources.

We will be using SquashFS 3.3 and will be adding LZMA support since it offers the best compression. For this reason we will also need to unmask some of the sources:

echo "sys-kernel/gentoo-sources ~x86" >> /etc/portage/package.keywords
emerge -av gentoo-sources

This needs to be installed before we start installing some other features, since some packages build modules which will need a kernel source tree available.

Install GRUB

We will be using the Grand Unified Boot Loader (GRUB) as our boot manager (or at least a simplified version of it). In order to do this, we need to have it installed on the build system so that later it can be copied over to the target system.

emerge -av grub

Install QEMU Toolkit

We will also be using QEMU to emulate our system. In order to build the hard drive and such, we will need some toolkits from QEMU (specifically qemu-img).

The biggest problem is that QEMU requires some features not provided by uClibc or gcc-4. This means that we will not be able to build qemu-img on our build system. At this point you have two options:

  1. Build app-emulation/qemu-softmmu on your host system
  2. Download a static compilation: .tar.bz2

All that essentially matters is that you have access to a qemu-img binary. If you wish to go the easy route, just download the static version. It has been tested and should run within the emgen-build environment.

If you are going to be using the static compilation, be sure to download the file and place it in a location within your path:

tar -jxvf ./qemu-img.tar.bz2
cp -a qemu-img "${EMGEN_BUILD}"/usr/bin

Install JFS Toolkit

We will also be using the JFS filesystem, so we need to install utilities to create/manage those filesystems:

emerge -av jfsutils
At the time of writing this document, the most recent unmasked jfsutils package (1.1.8) would fail with posix compatibility errors (uClibc vs. glibc). Unmasking a more recent version (1.1.12) by the following command fixed the problem:
echo "sys-fs/jfsutils ~x86" >> /etc/portage/package.keywords

Leave Chroot and Backup

Now that you have a build system that is fully functional, it is a good idea to once again leave the chroot environment and backup the system. Follow the same steps as outlined in the previous section on leaving the Chroot to exit the environment and unmount any bound directories.

Prepare the Target System

Once the build system is set up and ready to go, you need to prepare a target filesystem for your EmGen target.

Chroot into the Build System

As per the documentation above, once again chroot into the build system to perform all of the tasks in this section.

Create the Target System Directory

Now that we are in a chroot, we will create a directory in the chroot for our target system.

export EMGEN_TARGET="/etarget"
mkdir -p "${EMGEN_TARGET}"

From here on out I will be referring to the target as ${EMGEN_TARGET}.

If you don't want to have to worry about setting the ${EMGEN_TARGET} variable every time you Chroot into the build system, you can execute the following command to have it set every time you perform the Chroot:
echo "EMGEN_TARGET=\"/etarget\"" >> /etc/env.d/99emgen

(Optional Step) Create the Target Emerge Script

To ease the management of packages on the target system, it is generally a good idea to create a script to handle all packages on the target system.

If you decide not to create this program, be sure to treat subsequent commands that use it accordingly.

First, open nano:

nano

Now create a file that looks like the following:

#!/bin/bash
 
ROOT="${EMGEN_TARGET}" \
    INSTALL_MASK="*.h HACKING.gz INSTALL.gz README.gz TODO.gz" \
    emerge -avkN \
    $@

This file contains the following lines:

  • ROOT=“${EMGEN_TARGET}“ - sets the root for all merges (similar to a –prefix directive).
  • INSTALL_MASK=“*.h …“ - masks all files matching the given patterns, preventing unneeded files from being merged.
  • emerge -avkN - set default emerge options to be verbose, ask for confirmation, use pre-built packages when possible, and check for new USE flags.
  • $@ - append all given parameters to the end of the emerge command.

Save the file as etarget-merge (or something similar, by typing <key>Ctrl-X</key>, <key>Y</key>, type in the name, and then type <key>Enter</key>.

Now set execute permissions on the file, and move it into the path:

chmod 755 ./etarget-merge
mv ./etarget-merge /usr/bin/

Install Required Packages

Now we need to install some required packages using our new script. The following packages MUST be installed, otherwise the target system won't be able to run.

sys-apps/baselayout-lite

Every system needs a baselayout. This package creates a base root filesystem structure for your target. We want to minimize size, however, so we will use the baselayout-lite package. This package is still in testing stages, however so we need to first unmask it:

echo "sys-apps/baselayout-lite **" >> /etc/portage/package.keywords

Now we can install the baselayout into our target environment:

etarget-merge baselayout-lite

sys-libs/uclibc

Since this entire system is built on the uClibc library, it should go without saying that we need to include the library in our target system so that any packages that require a C library will have it available.

uClibc should not be masked, so just install it:

etarget-merge uclibc

sys-apps/busybox

We need to have a base set of utilities / shell / etc. as well, so we use busybox to provide that. Although not usually done, we will also enable the make-symlinks USE flag so that the install will make the symlinks that are usually done by other programs/installs.

USE="make-symlinks" etarget-merge busybox

sys-fs/jfsutils

Since we are using a JFS filesystem, we will be installing the jfsutils package.

etarget-merge jfsutils
You MUST install the filesystem creation/checking utilities for whichever root filesystem you choose! In the event that a system gets shutdown improperly (before the root is remounted read-only), you will be unable to remount the root read/write until a filesystem check has been performed!

Install Optional Packages

In the next section, we will be installing the packages that we will actually be using in the EmGen target environment. You can install whatever you want into your target environment, but the more you install (and what you install) will cause your target root size to become larger.

You should think about what you want your EmGen system to do (hopefully you did this before starting this project). For me, I will be going over how to create a basic (pseudo-)LAMP environment for web development. This way people could develop for their website regardless of what computer they are at.

The requirements for this system is the following:

  • Linux - Well that's already taken care of.
  • Apache - We won't actually be using Apache because of its size, but instead a simplified server called Cherokee.
  • MySQL - We won't be able to use MySQL since it is a bit bulky and I was unable to get the embedded server to install properly. Instead we will use SQLite3 as our SQL solution since it is lightweight and also because it uses single files, making the used disk space quite a bit easier to manage.
  • PHP - PHP is a staple of LAMP programming and it can't really be created without it in its entirety.

www-servers/cherokee

Cherokee is a lightweight and simple HTTP server. Although there are lots of lightweight servers, I chose Cherokee because it is lightweight but also supports FastCGI, which will come in useful in speeding up PHP (since lightweight servers only support PHP through the CGI interface).

Cherokee is still masked by keyword at this time so first unmask it, then install it into the target:

echo "www-servers/cherokee ~x86" >> /etc/portage/package.keywords
etarget-merge cherokee

This may install some dependencies, but it should be minimal. If anything make sure there are no USE flags set for the install as they are probably not needed anyway.

After the install, portage will attempt to create users for the cherokee daemon. The only problem is that it will create them outside of the specified ROOT. It is important to create a user and group for the cherokee daemon within the ROOT. To do this, run the next commands:

grep cherokee /etc/passwd >> "${EMGEN_TARGET}"/etc/passwd
grep cherokee /etc/group >> "${EMGEN_TARGET}"/etc/group
grep cherokee /etc/shadow >> "${EMGEN_TARGET}"/etc/shadow

dev-db/sqlite

SQLite is a simplified SQL Server that is managed by simply using flat-files and some C programs. Although it sounds rather slow and bulky, it is actually comparable to larger server-based programs such as MySQL or PostgreSQL.

Since we need to conserve some space and the minimal install of MySQL only results in client-side programs (I attempted the embedded server but to no avail), we will be using SQLite as our backend.

We don't actually have to explicitly install SQLite as it will be installed as a dependency when we install PHP.

I actually recommend not installing SQLite separately, as I attempted to do so and it caused more than one version to be installed. PHP at the time of this writing only supports SQLite2, while SQLite3 is currently available. Therefore let PHP handle which version to install to avoid accidentally using up extra space.

dev-lang/php

Our last step is to install PHP on our target. This will by far be the most bulky package we will be installing, and quite a few USE flags that may be set. You should make sure you only set the ones you think you'll need for your environment, as every flag used will add more bulk to the install.

I used the following USE flags:

  • bzip2 - I like to have support for bzlib compression.
  • cgi - THIS MUST BE ENABLED OR WE WON'T BE ABLE TO ATTACH PHP TO OUR HTTP SERVER
  • cli - To allow PHP to be accessed via command-line.
  • crypt - Support for encryption.
  • ctype - cType functions can be useful.
  • curl - The cURL library is very useful when doing HTTP or other URL-related requests.
  • force-cgi-redirect - A common thing to enable in the CGI SAPI.
  • gd - Add internal graphics support.
  • gmp - Add support for the GNU Mupltiple Precision Library
  • hash - Enable the hash extension
  • json - Add support for JSON so we can talk with JavaScript easily.
  • mhash - Support the mhash library.
  • ncurses - Add ncurses support.
  • pcre - Add support for Perl Compatible Regular Expressions.
  • readline - Add support for GNU line-editing.
  • reflection - Enable the Reflection API.
  • session - Add support for sessions.
  • simplexml - Add support for SimpleXML.
  • soap - Add support for the Simple Object Access Protocol.
  • sockets - Sockets can be quite useful to have support for.
  • spl - Add support for the Standard PHP Library.
  • sqlite - Add support for SQLite
  • tidy - Add support for HTML Tidy.
  • tokenizer - Add support for the PHP file parser.
  • unicode - Support Unicode characters.
  • xml - Support XML files.
  • xmlreader - Enable XMLReader.
  • xmlwriter - Enable XMLWriter.
  • zip - Add ZIP compression support.
  • zlib - Add zlib compression support.
Certain USE flags in PHP will install extensions that are incompatible with uClibc (as they expect glibc instead). The above USE flags have been tested and do work with uClibc, but if you enable other flags, be prepared for possible installation issues.

To perform the install, simply run the following command:

USE="bzip2 cgi cli crypt ctype curl force-cgi-redirect gd gmp hash \
json mhash ncurses pcre readline reflection session simplexml soap \
sockets spl sqlite tidy tokenizer unicode xml xmlreader xmlwriter \
zip zlib" etarget-merge php
This will definitely install extra packages that your system will need. It is important to note that some packages may appear to be installed twice. This is because your system will have to install some packages to emgen-build so that future packages that depend on them can be compiled and placed on emgen-target.

net-misc/dhcpcd

We need to install a DHCP client so that we can connect our NIC to the virtual DHCP server, and thusly server our pages to the host system:

etarget-merge dhcpcd

Compile the Kernel

Now that we have our packages setup, it is time to get to the nitty-gritty and compile our kernel that will be used to run all of this.

First, cd into the kernel build directory:

cd /usr/src/linux

Configure the Kernel

Since we are attempting to build a tiny kernel, it is a good idea to start off by removing all configuration options, we will set them on our own:

make mrproper
make allnoconfig

This will create a default .config file that will have no options set. Now it is time to set those options. To open up the configuration menu, type the following command:

make menuconfig

This will open up a curses-based menu configuration program. Although configuring a kernel is typically somewhat difficult, we have a trick that will make it easier: qemu! Since we will be using qemu to run our system, we already know what hardware will appear to be present regardless of what system the emulator is run on.

That being said, the configuration should be as follows for the kernel:

    General Setup  --->
    [*] Prompt for development and/or incompleted code/drivers
    [*] Support for paging of anonymous memory (swap)
    [*] System V IPC
    [*] BSD Process Accounting
    [*]     BSD Process Accounting version 3 file format
    (14) Kernel log buffer size
    [*] Initial RAM filesystem and RAM disk support
    [*] Optimize for size (Look out for broken compilers!)
    [*] Configure standard kernel features (for small systems)  --->
        --- Configure standard kernel features (for small systems)
        [*] Enable 16-bit UID system calls
        [*] Support for hot-pluggable devices
        [*] Enable support for printk
        [*] Enable full-sized data structures for core
        [*] Enable futex support
    [*] Enable eventpoll support
    [*] Enable signalfd() system call
    [*] Enable eventfd() system call
    [*] Use full shmem filesystem
    [*] Enable VM event counters for /proc/vmstat
        Choose SLAB allocator (SLUB (Unqueued Allocator))  --->

[*] Enable loadable module support  --->
    [*] Module unloading
    [*] Automatic kernel module loading

[*] Enable the block layer  --->
    --- Enable the block layer
          IO Schedulers --->
            <*> CFQ I/O scheduler
                Default I/O scheduler (CFQ)  --->

    Processor type and features  --->
        Subarchitecture Type (PC-compatible)  --->
        Processor family (586/K5/5x86/6x86/6x86MX)  --->
        Preemption Model (Preemptible Kernel (Low-Latency Desktop))  --->
        Hich Memory Support (off)  --->
        Memory split (3G/1G usr/kernel split)  --->
        Memory model (Flat Memory)  --->
    [*] Math emulation
    [*] MTRR support
        Timer frequency (250 HZ)  --->
    (0x100000) Physical address where the kernel is loaded
    (0x100000) Alignment value to which kernel should be aligned

    Bus options (PCI, PCMCIA, EISA, MCA, ISA)  --->
    [*] PCI support
          PCI access mode (Any)  --->
    [*] ISA support

    Executable file formats  --->
    [*] Kernel support for ELF binaries
    <*> Kernel support for a.out and ECOFF binaries
    <*> Kernel support for MISC binaries

    Networking  --->
    [*] Networking support
        Networking options  --->
        <*> Packet socket
        <*> Unix domain sockets
        [*] TCP/IP networking
        <*>   IP: IPsec transport mode
        <*>   IP: IPsec tunnel mode
        <*>   IP: IPsec BEET mode
        <*>   INET: socket monitoring interface

    Device Drivers  --->
        Generic Driver Options  --->
        (/sbin/hotplug) path to uevent helper
    [*] Block devices  --->
        --- Block devices
        <M>   Normal floppy disk support
        <*>   Loopback device support
        <*>   RAM disk support
        (16)    Default number of RAM disks
        (4096)  Default RAM disk size (kbytes)
        (1024)  Default RAM disk block size (bytes)
    <*> ATA/ATAPI/MFM/RLL support  --->
        --- ATA/ATAPI/MFM/RLL support
        (4)   Max IDE interfaces
        <*>   Enhanced IDE/MFM/RLL disk/cdrom/tape/floppy support
        <*>     Include IDE/ATA-2 DISK support
        <*>     Include IDE/ATAPI CDROM support
        <M>     Include IDE/ATAPI FLOPPY support
        [*]     legacy /proc/ide/ support
        <*>     generic/default IDE chipset support
        <*>     Generic PCI IDE Chipset support
        <*>     Intel PIIXn chipsets support
        SCSI device support  --->
        <*> SCSI device support
        [*] legacy /proc/scsi/ support
        <*> SCSI disk support
        <*> SCSI CDROM support
        <*> SCSI generic support
    [*] Network device support  --->
        --- Network device support
        [*]   Ethernet (10 or 100Mbit)  --->
            --- Ethernet (10 or 100Mbit)
            <M>   AMD LANCE and PCnet (AT1500 and NE2100) support
            [*]   Other ISA cards
            <M>   NE2000/NE1000 support
            [*]   EISA, VLB, PCI and on board controllers
            <M>   PCI NE2000 and clones support
        Input device support  --->
        -*- Generic input layer (needed for keyboard, mouse, ...)
        [*]   Keyboards  --->
            --- Keyboards
            <*>   AT keyboard
        Character devices  --->
        [*] Virtual terminal
        [*]   Support for console on virtual terminal
        [*] Unix98 PTY support
        Graphics support  --->
            Console display driver support  --->
            [*] VGA text console
            [*]   Video mode selection support

    File systems  --->
    <*> JFS filesystem support
    <*> ROM file system support
    [*] Dnotify support
    <*> Kernel automounter version 4 support
        Pseudo filesystems  --->
        [*] /proc file system support
        [*]   Sysctl support (/proc/sys)
        [*] sysfs file system support
        [*] Virtual memory file system support (former shm fs)
        Miscellaneous filesystems  --->
        <*> Compressed ROM file system support (cramfs)

    Kernel hacking --->
    [*] Enable doublefault exception handler

    Library routines  --->
    {*} CRC32 functions

Compile the Kernel Image

Now that you have your kernel configured, make the kernel with the following function:

make -j2

Install the Modules

Now you need to install the modules for the kernel. This is done with the following command:

make modules_install

Install SquashFS + LZMA Module

If you noticed, we did not enable the SquashFS module/builtin capabilities. This is because we will be patching the driver to enable LZMA compression versus GZip compression. LZMA is much more effective. In this section we will be downloading all of the patches, source, etc., needed to compile the SquashFS+LZMA module.

Create a Build Directory

First we will be needing to create a build directory for our module:

mkdir -p /tmp/sqfs-lzma
cd /tmp/sqfs-lzma
<code>
 
=== Download the Sources ===
 
Next we need to download all of the sources for our build:
 
<code bash>
wget http://www.squashfs-lzma.org/dl/squashfs3.3.tar.gz
wget http://www.squashfs-lzma.org/dl/squashfs-cvsfix.patch
wget http://superb-east.dl.sourceforge.net/sourceforge/sevenzip/lzma449.tar.bz2
wget http://www.squashfs-lzma.org/dl/sqlzma3.3-457.tar.bz2
I realize that I am using LZMA-449 even though the patchset says it is for -457, but it would appear that the provided patchset online still only supports -449.

Unpack the Sources

After downloading the sources, we need to unpack everything.

First, we will unpack the sqlzma patches:

mkdir -p sqlzma3.3-457
tar -jxvf sqlzma3.3-457.tar.bz2 -C sqlzma3.3-457

Next, unpack the lzma449 files into a subdirectory of our lzma patchset:

mkdir -p sqlzma3.3-457/lzma449
tar -jxvf lzma449.tar.bz2 -C sqlzma3.3-457/lzma449

Lastly, unpack the squashfs3.3 files:

tar -zxvf squashfs3.3.tar.gz -C sqlzma3.3-457

Once everything is unpacked, be sure to enter the sqlzma directory for making:

cd sqlzma3.3-457

We will also be needing to patch the linux kernel to build the squashfs module. To provide a method for this, create a link, linux, in the sqlzma directory. We will also make a versioned link to make it easier to patch:

ln -s /usr/src/linux linux
ln -s linux linux-2.6.24

Apply CVS Patches to SquashFS

First we need to apply the sqlzma patchset to the LZMA tree:

patch -p0 < ./sqlzma1-449.patch

The next step is to apply the CVS patchset to the SquashFS source tree:

patch -p0 < ../squashfs-cvsfix.patch

After that we need to apply the kernel patches from the SquashFS tree, as well as the sqlzma patshet.

patch -p0 < squashfs3.3/kernel-patches/linux-2.6.24/squashfs3.3-patch
patch -p0 < sqlzma2k-3.3.patch

Lastly, we need to apply the user tools patch:

patch -p0 < sqlzma2u-3.3.patch

There is an error in the Makefile that needs to be corrected regarding the directory structure and the kernel version, to solve that, use this command (assuming the flavor of Gentoo-Sources-2.6.24 is 2.6.24-gentoo-r3:

cp Makefile Makefile.old; \
    sed -r 's/\$\(shell uname -r\)/2.6.24-gentoo-r3/g' Makefile.old | \
    sed -r 's/^SqFs =.*$/SqFs = ${Sqlzma}\/linux\/fs\/squashfs/g' - > Makefile

Sadly, we will also need to fix an error in SquashFS-3.3 regarding the use of get_nprocs(). Thanks to the wonderful people in the Gentoo Developer Community, however, we don't have to worry about how. This problem is already included as a patch in SquashFS-3.3! We can't apply this patch however since we have already applied patches to the required files. The bonus is that they do tell us how to fix the problem.

To fix the problem, we need to open mksquashfs.c, and change the line ourselves:

nano ./squashfs3.3/squashfs-tools/mksquashfs.c

Then we need to go to line ~2847 (use <key>Ctrl-C</key> to see the current line), and change the value from:

                processors = get_nprocs();

to

                processors = sysconf(_SC_NPROCESSORS_CONF);

Then save the file (<key>Ctrl-X</key>, <key>Y</key>, <key>Enter</key>, and that should be it!

Make the Modules

Now that we have everything we need patched, it is time to build the modules.

make

Install to Build Environment

Now we need to copy the modules over to the build environment so that they are loadable:

mkdir /lib/modules/2.6.24-gentoo-r3/misc
cp linux/fs/squashfs/squashfs.ko /lib/modules/2.6.24-gentoo-r3/misc/
cp lzma449/C/Compress/Lzma/kmod/sqlzma.ko /lib/modules/2.6.24-gentoo-r3/misc/
cp lzma449/C/Compress/Lzma/kmod/unlzma.ko /lib/modules/2.6.24-gentoo-r3/misc/

We will need to add the modules to the modules.dep list as well to make sure they get used properly:

echo "/lib/modules/2.6.24-gentoo-r3/misc/squashfs.ko: /lib/modules/2.6.24-gentoo-r3/misc/sqlzma.ko /lib/modules/2.6.24-gentoo-r3/misc/unlzma.ko" >> /lib/modules/2.6.24-gentoo-r3/modules.dep
echo "/lib/modules/2.6.24-gentoo-r3/misc/sqlzma.ko:" >> /lib/modules/2.6.24-gentoo-r3/modules.dep
echo "/lib/modules/2.6.24-gentoo-r3/misc/unlzma.ko:" >> /lib/modules/2.6.24-gentoo-r3/modules.dep

We also need to copy over a few libraries so that they may be used:

cp lzma449/C/Compress/Lzma/libunlzma.a /usr/lib/
cp lzma449/C/Compress/Lzma/libunlzma_r.a /usr/lib/

And lastly we need to copy over the utility binaries:

cp squashfs3.3/squashfs-tools/mksquashfs /usr/bin/
cp squashfs3.3/squashfs-tools/unsquashfs /usr/bin/

And now we're all done!

Install Modules to the Target

Now that we have all of the modules we need installed on our build system, we need to transfer them over to the target system.

To do this, we enter the following commands:

rm -r "${EMGEN_TARGET}"/lib/modules
mkdir -p "${EMGEN_TARGET}"/lib/modules
cp -a /lib/modules/* "${EMGEN_TARGET}"/lib/modules/

Leave the Chroot Environment

You now also have a target system to be installed on the device of your choice! It is once again a good idea to leave the Chroot environment and save a backup of your work thus far. (Trust me, you'll regret it if you make a mistake in the future!). To leave the Chroot completely, follow the section regarding the topic in the above statements.

Modify Target

Now that we have our target machine setup pretty much exactly the way we want it, we need to make a few extra changes to it to ensure that it will boot and load properly.

Remember that we did not install any form of text editor into the target system, so all modifications will need to be made outside of the system. After a modification has been made, it can be tested from within the system itself by chrooting into it.
All commands from here on out assume that you are within the directory of your emgen-target system and that an environment variable, $EMGEN_TARGET is set to that directory. If you do not set this and run the commands as below you will damage your system! To set this value, run the following command:
export EMGEN_TARGET="/emgen/etarget"

Create Init Scripts

The installs for our system do not include any init scripts for cherokee or mysql. In order to make sure they start at boot time, we need these scripts to be created.

BusyBox (which we installed for our target), provides a mechanism by which all scripts in /etc/init.d are started at boot and killed at system halt. For that reason we will create two scripts to do just that.

/etc/init.d/mount

Because of how our system is going to be created, we will need to remount the root filesystem to make it writable.

rm -f ${EMGEN_TARGET}/etc/init.d/mount
nano ${EMGEN_TARGET}/etc/init.d/mount

Type (or copy) the following into the file:

#!/bin/sh
 
case $1 in
start)
        # perform a filesystem check on the root device
        fsck.jfs -fa /dev/root
        # make the root read/write
        echo "Remounting Root Filesystem Read/Write..."
        mount -oremount,rw,noatime -n /dev/root /
        # mount our squashed filesystem to the root
        echo "Loading SquashFS-LZMA Module..."
        modprobe squashfs
        echo "Mounting /usr squashed filesystem..."
        mount -t squashfs -oloop /usr.sfs /usr
        echo "Mounting /etc/cherokee squashed filesystem..."
        mount -t squashfs -oloop /etc/cherokee.sfs /etc/cherokee
        ;;
stop)
        # umount our squashed filesystem
        echo "Unmounting /etc/cherokee squashed filesystem..."
        umount /etc/cherokee
        echo "Unmounting /usr squashed filesystem..."
        umount /usr
        echo "Unloading SquashFS-LZMA Module..."
        modprobe -r squashfs
        # remount root read-only
        echo "Remounting Root Filesystem Read-only..."
        mount -oremount,ro -n /dev/root /
        ;;
esac
Don't worry about the user filesystem being mounted stuff. The creation of that filesystem will be handled later and it is going to be done that way for spacial reasons. For now just type in script.

Now save the file (<key>Ctrl-X</key>, <key>Y</key>, <key>Enter</key>), then make it executable:

chmod +x ${EMGEN_TARGET}/etc/init.d/mount

/etc/init.d/net

We will also need a network to be present, so let's start that up here:

rm -f ${EMGEN_TARGET}/etc/init.d/net
nano ${EMGEN_TARGET}/etc/init.d/net

Type (or copy) the following into the file:

#!/bin/sh
 
case $1 in
start)
        echo "Loading Network Modules..."
        modprobe ne2k-pci
 
        echo "Bringing Up Eth0 Network Interface..."
        ifconfig eth0 up
        dhcpcd -L eth0
        ;;
stop)
        echo "Shutting Down Eth0 Network Interface..."
        dhcpcd -k eth0
        ifconfig eth0 down
 
        echo "Unloading Network Modules..."
        modprobe -r ne2k-pci
        ;;
esac
You will notice that I added some module autoloading. This is because I installed some of the NIC drivers as modules. If you compiled them in, you will not need to have this be a part of the script. Also, the modules I am loading are for the default NIC used by QEMU.

Now save the file (<key>Ctrl-X</key>, <key>Y</key>, <key>Enter</key>), then make it executable:

chmod +x ${EMGEN_TARGET}/etc/init.d/net

/etc/init.d/cherokee

We will need to create a script to boot our cherokee server:

rm -f ${EMGEN_TARGET}/etc/init.d/cherokee
nano ${EMGEN_TARGET}/etc/init.d/cherokee

Type (or copy) the following into the file:

#!/bin/sh
 
PIDFILE=/var/run/cherokee.pid
 
case $1 in
start)
        echo "Starting Cherokee HTTPD Server..."
        start-stop-daemon --quiet --start --pidfile ${PIDFILE} \
                --exec /usr/sbin/cherokee -- -b &
        ;;
stop)
        echo "Stopping Cherokee HTTPD Server..."
        start-stop-daemon --quiet --stop --pidfile ${PIDFILE} &
        ;;
esac

Now save the file (<key>Ctrl-X</key>, <key>Y</key>, <key>Enter</key>), then make it executable:

chmod +x ${EMGEN_TARGET}/etc/init.d/cherokee

Create /var/run

Since we will be storing our cherokee PID file under /var/run, we need to make sure it exists:

mkdir -p ${EMGEN_TARGET}/var/run

Last but not least we need to create some symbolic links so that the system will automatically start/stop the scripts we created in the right order on startup nad shutdown.

ln -s mount ${EMGEN_TARGET}/etc/init.d/S01
ln -s net ${EMGEN_TARGET}/etc/init.d/S02
ln -s cherokee ${EMGEN_TARGET}/etc/init.d/S03
ln -s cherokee ${EMGEN_TARGET}/etc/init.d/K01
ln -s net ${EMGEN_TARGET}/etc/init.d/K02
ln -s mount ${EMGEN_TARGET}/etc/init.d/K03

The init system for BusyBox will execute all scripts in /etc/init.d, in numerical order, with S?? being the startup scripts and K?? being the kill scripts.

Test Binaries

Now it is generally a good idea to test your binaries that will be run to ensure that no dependencies have been left out in the fray (because they were installed to the build system but not the target system).

In order to perform the tests, you will need to chroot into the target system:

chroot $EMGEN_TARGET /bin/ash --login

Now the files you will want to test are:

/usr/sbin/cherokee
/usr/lib/php5/bin/php
/usr/lib/php5/bin/php-cgi
/usr/sbin/dhcpcd
/sbin/fsck.jfs (or similar)

Run each of these programs. If one of them fails to run because of an error such as:

./php: can't load library 'libpcre.so.0'

Then you will have to re-install each library that it states using the build environment. To re-install a package from the build environment, first you must make sure you are chrooted into the emgen-build environment.

Next, run the following command to determine which package contains the library you need (for example, with libpcre.so.0):

equery belongs libpcre.so.0

This should yield where the library is and to which package it belongs:

[ Searching for file(s) libpcre.so.0 in *... ]
dev-libs/libpcre-7.6-r1 (/usr/lib/libpcre.so.0 -> libpcre.so.0.0.1)

At this point you can either:

  1. Copy the library itself to the system (recommended for space)
  2. Install the package to the system (recommended for stability)
I recommend having two terminal windows open, one with you chrooted into the emgen-target environment, and one of you chrooted into the emgen-build environment. This way you can continue to test on emgen-target and whenever something goes wrong, you can change terminals to emgen-build to re-install a library to the target.

Option 1: Copy the library

It is recommended to copy the library to the system since it will conserve a lot of space and prevent potentially unneeded dependencies from being installed on the target.

To copy the libraries to the system:

cp -a /usr/lib/libpcre.so.0 "${EMGEN_TARGET}"/usr/lib
cp -a /usr/lib/libpcre.so.0.0.1 "${EMGEN_TARGET}"/usr/lib
Notice how I copied over the symbolic link and the actual library. You must copy with the -a flag to preserve users and other values, but it means that a symbolic link will be copied as a symbolic link and not as the library it linked to, so the library itself must be copied over seperately.

Option 2: Install the Package

This option is not recommended for space since a lot of extra (potentially unnecessary packages) will probably be installed as dependencies.

To install the package, simply:

etarget-merge libpcre

Configure Cherokee

Next you need to configure cherokee to run PHP scripts. To do this, first open the cherokee configuration file:

nano "${EMGEN_TARGET}"/etc/cherokee/sites-enabled/default

Now scroll to the bottom of the default file where it has something to the effect of:

Extension php, php3, php4 {
                Handler phpcgi
}

Change it to this:

Extension php, php3, php4, php5 {
        Handler phpcgi {
                Interpreter "/usr/lib/php5/bin/php-cgi"
        }
}

Now save (<key>Ctrl-X</key>, <key>Y</key>, <key>Enter</key>).

Your web server should now support PHP!

I attempted to get PHP's built-in FastCGI server working however whenever a page would be loaded the system would return a “500 Internal Server Error” error message and cherokee would complain about not being able to connect to the transport.

Set Target Hostname

Just for good measure, it is generally a good idea to set the target's hostname, for networking purposes:

echo "emgen" >> "${EMGEN_TARGET}"/etc/hostname

Create Root Password

In order to be able to log in once you boot the target, you need to have the root password set. To do this, first chroot into the target:

chroot "${EMGEN_TARGET}" /bin/ash --login

Now that you are logged in to the system, create a root password:

passwd

Then exit the chroot:

exit

The system also needs a few links for the loop devices to ensure that they function properly with some of the losetup/mount operations:

mkdir -p "{$EMGEN_TARGET}"/dev/loop
for i in "{$EMGEN_TARGET}"/dev/loop?*; do ln -s ../`basename $i` "${EMGEN_TARGET}"/dev/loop/`echo $i | sed -r 's/^.*loop([0-9]+).*$/\1/gi' - `; done

This should make symbolic links for each of the loopback devices, which may be useful in the future.

Modify /etc/fstab

We need to modify the emgen-target's Filesystem Tab file (/etc/fstab) in order to make sure that some filesystems are mounted automatically at boottime.

Open the file as such:

nano "${EMGEN_TARGET}"/etc/fstab

And edit it to look something like the following:

# default ram disk as root

/dev/root               /               jfs             noatime         1 2
none                    /proc           proc            defaults        0 0

# glibc 2.2 and above expects tmpfs to be mounted at /dev/shm for
# POSIX shared memory (shm_open, shm_unlink).
# (tmpfs is a dynamically expandable/shrinkable ramdisk, and will
#  use almost no memory if not populated with files)
# Adding the following line to /etc/fstab should take care of this:

none                    /dev/shm        tmpfs           defaults        0 0
I set the root partition to /dev/root (which will be /dev/hda1) since we will be storing our system on an emulated hard drive image. We also set it to an JFS filesystem, which is semi-arbitrary. You can set the filesystem type to whatever you wish provided support for that filesystem has been built into your kernel, and you remember use that filesystem from here on out instead of JFS. JFS was chosen because it take up very little of the hard drive space for file accounting (for our size), and also supported journaling.

We also uncomment /proc and /dev/shm since /proc is useful for monitoring system information and /dev/shm is a RAM filesystem and takes virtually no space unless needed (no real downside to not use it).

Then save the file (<key>Ctrl-X</key>, <key>Y</key>, <key>Enter</key>).

Now for BusyBox compatibility, we need to create a symbolic link for the root device:

ln -s hda1 "${EMGEN_TARGET}"/dev/root

Setup Temporary Creation Environment

Now that we have everything that we need to setup the hard drive, we need to create a temporary creation environment.

Chroot to the Build Environment

Follow the aforementioned instructions to do this (once again).

Create the Creation Directory

We will be building a copy of our emgen-target system so that we can modify it at will without having to worry about making mistakes. All of our work will be done under the directory that we create for the build. To reset this directory, execute the following commands:

rm -fr /tmp/ecreate
mkdir -p /tmp/ecreate

Also move into the directory you just created.

cd /tmp/ecreate

Copy Over the Target

Now that we have a temporary environment, we need to copy over everything we've created in the target. We do an archival copy to ensure that we save all of our permissions and such.

cp -a "${EMGEN_TARGET}" ./etarget

Finalize Installation on the Target

This section remains within the build environment chroot.

Install the Boot Loader

We now need to install our boot loader, GRUB. This will allow our hard drive to boot the kernel directly.

Copy the Boot Directory

First step is to copy our build environment's /boot directory to our target system (don't worry about sizes, we will clean it up manually):

cp -a /boot ./etarget/

Clean-up GRUB

Now it is time to get a little messy and remove some of the pieces of GRUB that are not needed. First, move into the GRUB directory:

cd ./etarget/boot/grub

Now we don't need most of these files, so first save the stage1 and stage2 installs:

mv stage1 ..
mv stage2 ..

Now depending on which filesystem you are using for your root directory, your stage1_5 is going to be different. The directory generally contains stages of the form “${FS_TYPE}_stage1_5”. Since we are using JFS, we need to save jfs_stage1_5:

mv jfs_stage1_5 ..

Now clean out the directory:

rm *

Then reinstate our saved files:

mv ../stage1 ./
mv ../stage2 ./
mv ../*_stage1_5 ./

Now we need to create the grub.conf, menu.lst, and device.map files. To start with, just create them, then we will edit them:

touch ./grub.conf
touch ./device.map
ln -s ./grub.conf ./menu.lst

Easiest is to populate is the device.map file, so let's do that:

echo "(hd0)   /dev/hda" > ./device.map

Now we need to create our configuration file:

nano ./grub.conf

Fill the file with something similar to this:

timeout 10
default 0
fallback 0

title EmGen
        root (hd0,0)
        kernel /boot/vmlinuz root=/dev/hda1

Then save the file (<key>Ctrl-X</key>, <key>Y</key>, <key>Enter</key>).

Be sure to remember what you name the kernel (in our case vmlinuz), because you have to install the kernel to that location!

Leave the GRUB Configuration Directory

Now that we are done configuring GRUB, we need to return to our original directory:

cd ../../../

Install the Kernel

Our system would be nothing without the Kernel we created, so let's install it:

cp /usr/src/linux/arch/i386/boot/bzImage ./etarget/boot/vmlinuz
Remember to rename your kernel to what you called it in your GRUB configuration! (in our case we renamed it to vmlinuz)

Squash the Filesystems

This is the step where we shrink down our system. Using SquashFS, we can reduce some of our file tree. I recommend compressing any directories (of substance) that require read-only access.

/usr

First off, we will clean any unneeded files from the /usr filesystem. After scanning through most of the directory, the safest thing to remove from the directory is the include directory, since we won't be performing any compiles on the target system:

rm -fr ./etarget/usr/include

Another step that can save some space on your system, is to move all of the library files not needed until later on in the boot process to the /usr directory, so that they will be squashed. For example, in order to boot your system, you should only need the libraries needed to run the files in the /bin and /sbin directories, as well as any corresponding libraries for inter-library links. Therefore, any other extra libraries can be thrown into /usr/lib.

In order to perform this, we need to check each runtime to get a list of libraries that need to be saved:

for i in ./etarget/{bin,sbin}/*; do if [ ! -L $i ]; then ldd $i | awk '{print $3}' - >> libs; fi; done
for i in `cat libs`; do if [ -e $i ]; then echo "$i" >> libs_2; fi; done
touch libs_3
for i in `cat libs_2`; do if [ -z "`grep $i libs_3`" ]; then echo "$i" >> libs_3; fi; done

These commands first list all libraries, then get rid of any excess garbage, then removes duplicates. Now that we have our list of libraries we need to add to that list any libraries that are required by them:

for i in `cat libs_3`; do ldd $i | awk '{print $3}' - >> libs_4; done
for i in `cat libs_4`; do if [ -e $i ]; then echo "$i" >> libs_5; fi; done
cp libs_3 libs_6
for i in `cat libs_5`; do if [ -z "`grep $i libs_6`" ]; then echo "$i" >> libs_6; fi; done

Now at this point, the file libs_6 contains all of the libraries that we need for our root filesystem to function. Anything else can be placed into /usr/lib. There is really no easy way that I know of to do this, except to compare the library list and the files in /lib.

It is assumed at this point that you will move these files yourself. Please remember to move all of the libraries as well as their symbolic links, or else things won't work properly!
It is generally a good idea after performing the copy-overs to double-check the /lib and /usr/lib for symbolic link breaks caused by the copies.

Now we can create our SquashFS file.

To conserve the greatest amount of space, we will also be changing the default block size to the largest amount supported, 1024k.

mksquashfs ./etarget/usr/* ./etarget/usr.sfs -b 1048576

Now we have no more need for the /usr directory, so empty it:

rm -fr ./etarget/usr/*

/etc/cherokee

The cherokee configuration directory does not need to be read-write, since it will only be read by the server, and not written to:

mksquashfs ./etarget/etc/cherokee/* ./etarget/etc/cherokee.sfs -b 1048576
rm -fr ./etarget/etc/cherokee/*

Cleanup Unneeded Files

Since we used Gentoo to create our target system, there are some unneeded files left over from the process. It is now time to remove them in order to free up some space:

rm ./etarget/etc/*-
rm -fr ./etarget/etc/portage
rm -fr ./etarget/var/cache/edb
rm -fr ./etarget/var/db/pkg
rm -fr ./etarget/var/lib/portage

Here is an explanation of what each directory is and why we don't need them:

  • /etc/*- - Sometimes backup files are saved with a dash (-) afterwards.
  • /etc/portage - A list of keywords/masks/etc for portage.
  • /var/cache/edb - A cache of various emerge data.
  • /var/db/pkg - This stores information about portage installs.
  • /var/lib/portage - Another location of information regarding portage installs.

Install to a Virtual Hard Drive

This section remains within the build environment's chroot.

Create the Virtual Hard Drive

Now we need to create a virtual hard drive to store our system on.

In order to make sure that everything will fit on the drive, you should first check how much space your system takes up:

du -shc etarget

Now take the total amount into consideration when determining how big you should make your disk image.

Since my goal is to fit everything plus emulation software onto a 32M flash card, I will calculate my size like this:
  • After the card is formatted using vfat it only holds about 30M, so assume I have 29M of space for that.
  • The QEMU windows binaries take up about 2-3M, so assume 4M and our total size drops to 26M.
  • Give some room for buffer space and extra files, and we will make our image 24M.

Since the system I created this far is only 11M, a 24M hard disk is plenty big.

Now make the image:

qemu-img create -f raw hda.img 24M

Next we need to create a partition table on the hard drive, so first create a loopback device for the hard drive image, and then execute fdisk on it:

losetup /dev/loop0 ./hda.img
fdisk /dev/loop0

Create a new partition (filling up all of the space), and then exit, writing the data to disk. To do this, simply type the following keys in sequence: <key>n</key>, <key>Enter</key>, <key>p</key>, <key>Enter</key>, <key>1</key>, <key>Enter</key>, <key>Enter</key>, <key>Enter</key>, <key>w</key>, <key>Enter</key>.

Now that you have the partition table created, remove the loop device:

losetup -d /dev/loop0

You know have a simulated hard disk with one empty partition.

Format the Target Partition

Now we need to format the target partition to our filesystem type (we chose JFS).

To do this, we first need a loopback device setup that starts at the partition. To find the starting point of that partition, execute the following command:

sfdisk -l -uS ./hda.img

This should yield something like:

Disk hda.img: cannot get geometry

Disk hda.img: 3 cylinders, 255 heads, 63 sectors/track
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
 hda.img1            63     48194      48132  83  Linux
 hda.img2             0         -          0   0  Empty
 hda.img3             0         -          0   0  Empty
 hda.img4             0         -          0   0  Empty

We can see that according to sfdisk, our virtual hard drive is made up of sectors of 512 bytes each and that our primary partition starts at sector 63. Using some basic math, we can deduce that the primary partition of our virtual hard drive starts at byte 512 * 63 = 32256.

Now we just create a loopback device with that offset:

losetup -o 32256 /dev/loop0 ./hda.img

To make sure that we format the right number of blocks, we need to check how many there are on our partition, so perform the following command:

sfdisk -l ./hda.img

This should yield something like:

Disk ./hda.img: cannot get geometry

Disk ./hda.img: 3 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
./hda.img1          0+      2       3-     24066   83  Linux
./hda.img2          0       -       0          0    0  Empty
./hda.img3          0       -       0          0    0  Empty
./hda.img4          0       -       0          0    0  Empty

We can see that there are 24066 blocks. It is generally a good idea to make your block count be a multiple of 4, so you should set your block count to ( total_blocks - (total_blocks mod 4) ) = ( 24066 - 2 ) = 24064.

The problem is that qemu-img uses a block size of 1024 bytes, while mkfs_jfs uses a default block size of 4096 (which is 1024*4). So to figure out how many JFS blocks to install on the partition, we divide our number by 4, and get 6016.

Then format the partition, using the specified block size:

mkfs.jfs /dev/loop0 6016

When asked whether to continue, hit <key>Y</key>, <key>Enter</key>.

Then once the formatting is completed, delete the loopback device:

losetup -d /dev/loop0

Load the Partition

Now we have a formatted, partitioned virtual hard drive, and the next step is to install all of our system files onto it.

Mount the Partition

In order to load the files onto the partition, we will first need to mount it. Before that can be done we need a place to mount it to, so create the mount point:

mkdir -p ./mnt

Now you can actually mount the partition. For this to work you will also need to use the offset from before, but you can skip the creation of a loopback device by letting the mount command handle it.

mount -o loop,offset=32256 ./hda.img ./mnt
If your mount fails it may be due to an incompatible partition type. We are using IBM's JFS filesystem, which is not nearly as common as Ext2 or Ext3. Your host system's kernel will need to have support for JFS for this mount to work (either built-in or through a module).

If the support is not build-in, you can check if there is a module by running the command:

modprobe -l | grep jfs -

On your host system. If it comes up with a module such as jfs.ko, then load the module using modprobe:

modprobe jfs

Copy Over the Files

Now that we have our mounted partition, it is time to copy over all of our hard work. This can be done simply by doing this:

cp -a ./etarget/* ./mnt/

Unmount the Partition

Now we no longer need to access the virtual hard drive, so just unmount it:

umount ./mnt

Convert the Hard Drive Image

To conserve some extra space, it is generally a good idea to convert the virtual hard drive image from RAW format (which it is in now) to QCOW2 format (a QEMU expandable format). The benefit is that QCOW2 will only take up as much space as it needs to, but will dynamically expand to the maximum.

qemu-img convert -f raw hda.img -O qcow2 hda2.img
rm hda.img
mv hda2.img hda.img

Leave the Chroot

Follow the aforementioned instructions to do this (once again).

Install GRUB to Virtual Hard Drive MBR

Although our hard drive does have GRUB installed, it is not installed to the MBR so therefore it will not work. To get it to work, we need to do some extra work since we now have a QCOW2 image.

Create a Bootable Floppy

First off, we need a bootable floppy so that we can boot into a grub command line and then run the install from there.

To do this, run the following commands:

dd if="${EMGEN_BUILD}"/boot/grub/stage1 of="${EMGEN_BUILD}"/tmp/ecreate/fda.img bs=512 count=1
dd if="${EMGEN_BUILD}"/boot/grub/stage2 of="${EMGEN_BUILD}"/tmp/ecreate/fda.img bs=512 seek=1

Run QEMU to Install GRUB

Now that we have our bootable floppy, we need to run QEMU to install the GRUB to the MBR.

This requires QEMU to be installed. If your host system is a Gentoo system, you can install QEMU by running:
emerge -av qemu

If you are not running a Gentoo system, you will need to check your system's information regarding how to install a copy of QEMU.

If you are running this howto as root but have sudo'd into it, the following command MAY LOCK UP YOUR COMPUTER! Be forewarned that in my experience when I was handling the install as root, QEMU would lock the computer when trying to access my DISPLAY. To get around this, copy the files to your main user's home directory and run qemu as that use (that user will need to belong to the qemu group):
mkdir -p ~/emgen
cp "${EMGEN_BUILD}"/tmp/ecreate/*.img ~/emgen/
chown myuser:mygrp -R ~/emgen

Then when you are completed with the GRUB MBR installation, run this command to move the hard drive image back:

rm "${EMGEN_BUILD}"/tmp/ecreate/*.img
mv ~/emgen/*.img "${EMGEN_BUILD}"/tmp/ecreate/
chown root:root "${EMGEN_BUILD}"/tmp/ecreate/*.img

Now that all of that is cleared up, run QEMU with both the hard disk and floppy images, booting from the floppy:

qemu -fda fda.img -hda hda.img -boot a

When GRUB boots up, install it to the hard drive's MBR using these commands:

root (hd0,0)
setup (hd0)
halt

Now the system should exit and you officially have a complete Hard Drive Image that is bootable, and contains your virtualize system! Congrats!

References

  • TinyGentoo - The HOWTO that this project is based on. Their idea was to create a system for a flash drive, I'd like to take it a bit further.
  • Embedded Gentoo - A tutorial on how to install Gentoo on an embedded device from stage 1. Also a strong foundation for this document.
  • Official Squashfs LZMA - Official site dedicated to using SquashFS with LZMA.
  • Linux Kernel Documentation - Provides a lot of useful information about how the Linux Kernel works.
  • ro root fs w/ busybox? - A mailing list reply explaining how to remount /dev/root as read/write.
  • QEMU on Windows - A port of QEMU to Windows so that we can run our Linux system on Windows.
  • QEMU Documentation - Useful information about all of the options that QEMU provides.
  • Creating a GRUB boot floppy - Information from the GRUB manual on how to create a bootable floppy disk.
  • A Minimal Linux System From Scratch - A similar project that was very useful in explaining some specifics regarding QEMU image creation.
 
projects/linux/emgen.txt · Last modified: 2014/05/13 12:36 (external edit)
 

Hosted on Microsoft Azure Powered by PHP Driven by DokuWiki RSS Feed

© 2011 Austen Dicken | cvpcs.org