Embedded Gentoo

• DONE
  1. Fixed attempt to read past end of device error.
  2. Fixed reboot error.
  3. Reduced size of disk image to 11M.
• TODO
  1. Add information about creating the qemu boot scripts.
  2. Test SQLite.
  3. Experiment with different space-conservation techniques.
     • unification filesystems
     • squashfs/tmpfs storage of /var/www
     • ideas?

• DONE
  1. Add information about copying data to the hard disk and converting to qcow2.
  2. Add information about making the hard disk bootable.
• TODO
  1. Solve attempt to read past end of device error.
     • possibly kernel-related? maybe hard disk too big and support needs to be enabled?
  2. Add information about creating the qemu boot scripts.
  3. Test SQLite.
  4. Experiment with different space-conservation techniques.
     • unification filesystems
     • squashfs/tmpfs storage of /var/www
     • ideas?

• DONE:
  1. Successfully booted the system and tested it using a QEMU emulator. Cherokee/PHP works.
  2. Added some more documentation of the process.
  3. Switched base partition to JFS format for spacial/journaling concerns.
  4. Added SquashFS-LZMA support, still requires testing.
  5. Added information regarding qemu-img virtual hard disk creation.

The EmGen project is an effort at creating the smallest possible, working distribution of the Gentoo Linux Distribution. This is being done using uClibc, and the current Gentoo Embedded project.

EmGen will attempt to make a fully working system, and as the project grows we will see how much we can fit in a small location.

The final product of this project currently will be to place an entire system, as well as a system emulator, on a 32MB SD Card. The finished product should be able to be inserted into a MMC/SC Card slot (such as one on a laptop) on a Windows system, and be used to run the Gentoo system.

It has been determined that the following prerequisites are needed to build this project:

The first thing we will do is set up a build environment which will be used to build our actual system. This build environment will have to be built once and from then on can be used to create various versions of the EmGen system.

First, be sure to create a directory that will be the base for your build system. This system should fulfill the space requirements.

export EMGEN_DIR=/emgen
mkdir -p $EMGEN_DIR

In order to create the build system, we must download a stage 1 tar-ball of the Embedded Gentoo system, and install it into the EmGen build environment.

cd $EMGEN_DIR

Open the Gentoo Mirrors page in a browser of your choice, and select a mirror. Once logged on to the mirror, traverse the directory tree to experimental / x86 / embedded / stages and download the most recent stage1 tar-ball. In the case of this project, I will be using version 2008.0.

wget http://mirror.mcs.anl.gov/pub/gentoo/experimental/x86/embedded/stages/stage1-x86-uclibc-2008.0.tar.bz2

If you downloaded a .tar.bz2, use the following command to unpack the file:

tar -xjpvf stage1-x86-uclibc-2008.0.tar.bz2

If you downloaded a .tar.gz or a .tgz, use the following command to unpack the file:

tar -xzpvf stage1-x86-uclibc-2008.0.tar.gz

If you are not using a Gentoo system as the host system for this project, you will need to download a snapshot of portage. If you are using Gentoo for your host system, then you can skip this step as you will be able to simply bind your current portage tree to the build system.

cd $EMGEN_DIR

Open the Gentoo Mirrors page in a browser of your choice, and select a mirror. Once logged on to the mirror, traverse the directory tree to snapshots and download the portage-latest tar-ball.

wget http://mirror.mcs.anl.gov/pub/gentoo/snapshots/portage-latest.tar.bz2

If you downloaded a .tar.bz2, use the following command to unpack the file:

tar -xjvf portage-latest.tar.bz2 -C $EMGEN_DIR/usr

If you downloaded a .tar.gz or a .tgz, use the following command to unpack the file:

tar -xzvf portage-latest.tar.gz -C $EMGEN_DIR/usr

In this stage, we will chroot into the newly created build system, and bootstrap it to enable us to use it for building the embedded target.

First we must set enter the Build System

Bind all of the appropriate directories to the build system:

mount -o bind /proc $EMGEN_DIR/proc
mount -o bind /sys $EMGEN_DIR/sys

mkdir -p $EMGEN_DIR/usr/portage
mount -o bind /usr/portage $EMGEN_DIR/usr/portage

mkdir -p $EMGEN_DIR/usr/portage/distfiles
mount -o bind /usr/portage/distfiles $EMGEN_DIR/usr/portage/distfiles

Next we copy the /etc/resolv.conf file so that we maintain internet connectivity.

cp -L /etc/resolv.conf $EMGEN_DIR/etc/resolv.conf

You should be ready to chroot now.

chroot $EMGEN_DIR /bin/bash --login

Now that you are within the chroot, you need to update the environment variables in your system.

env-update
source /etc/profile

It is also a good idea to modify your prompt so that you don't accidentally type commands into your host system that were meant for your build system.

export PS1="(emgen-build) $PS1"

Now that we are in the build system, we need to update some common configuration files to ensure that the build goes alright.

The /etc/make.conf file is one of the most important files on any Gentoo system. It sets all of the environment variables to use when installing packages.

Open the file.

nano /etc/make.conf

It will look something like this as a default:

# These settings were set by the catalyst build script that automatically built
# Please consult /etc/make.conf.example for a more detailed example
CFLAGS="-Os -pipe"
CHOST="i386-gentoo-linux-uclibc"
CXXFLAGS="-Os -pipe"
LDFLAGS="-Wl,-z,relro"

For the sake of making things easy, I would recommend changing/adding the values to make the file resemble the following:

# These settings were set by the catalyst build script that automatically built
# Please consult /etc/make.conf.example for a more detailed example
CFLAGS="-Os -pipe -march=i686 -fomit-frame-pointer"
CXXFLAGS="${CFLAGS}"
CHOST="i686-gentoo-linux-uclibc"
LDFLAGS="-Wl,-z,relro"

FEATURES="ccache buildpkg"
USE="minimal"
UCLIBC_CPU="686"
PKGDIR="${PORTDIR}/packages/uclibc"

Here is an explanation for the added/modified lines above:

• CFLAGS
  1. -march=i686 is just a way of telling the GCC compiler to make generic optimizations for a i686-class processor (generally safe for any x86 architecture these days).
  2. -fomit-frame-pointer tells GCC to not use a register for stack tracing, which we won't need as we will be leaving out any sort of debugging for the sake of space.
• CXXFLAGS
  1. It is generally good practice to set the CXXFLAGS equal to the CFLAGS unless you have good reason not to.
• CHOST
  1. I set this to i686-gentoo-linux-uclibc as I wish to build a generic i686-based system. You can change this but it should reflect all of the other values (such as -march and UCLIBC_CPU).
• LDFLAGS
  1. This I would just leave at a default.
• FEATURES
  1. ccache tells the system that we will be using the ccache system to cache our compiles to make recompiling faster.
  2. buildpkg tells the system to use pre-built packages when possible.
• USE
  1. minimal enables all minimalistic installs in the system.
• UCLIBC_CPU
  1. 686 informs the uClibc library to make optimizations for the i686 architecture. If this is not set, uClibc will be very generic and also very bulky.
• PKGDIR
  1. This is used to tell portage to store packages for this build in a separate location from the default.

Once you have finished making the modifications, hit <key>Ctrl - X</key>, <key>Y</key>, and <key>Enter</key> to exit nano and save the modifications.

The /etc/make.profile file sets default for the /etc/make.conf file as well as some other settings. It is important to set it to the proper value for your build system. To view what it currently is, type:

ls -ld /etc/make.profile

This should be set to a generic x86 uClibc profile. To do so execute the following command:

ln -fns ../usr/portage/profiles/uclibc/x86 /etc/make.profile

It is generally a good idea to double check to make sure it worked:

ls -ld /etc/make.profile

The next step in the build process is to bootstrap the build system. This must be done to create a compiler toolkit based on embedded devices.

This directory is where the bootstrap.sh file is located, which will be used to bootstrap the system.

cd /usr/portage/scripts

In order to see what is going to be bootstrapped, it is generally good to run a pretend run first.

./bootstrap.sh -p -v

This command will locate all of the packages you need to have installed to bootstrap the system. Take a note of this as if anything goes wrong you will need to know where to continue. In my case, the following were displayed:

First the system will update portage (the Gentoo emerge system), by installing/upgrading the following:

[ebuild     U ] sys-apps/sandbox-1.2.18.1-r2 [1.2.17] 0 kB 
[ebuild     U ] sys-apps/portage-2.1.4.4 [2.1.1-r1] USE="build -doc -epydoc% (-selinux)" LINGUAS="-pl" 0 kB 

Next, the system will be bootstrapped by installing/upgrading the following:

[ebuild  N    ] dev-util/unifdef-1.20  0 kB 
[ebuild     U ] sys-kernel/linux-headers-2.6.23-r3 [2.4.26-r1] USE="(-gcc64%)" 0 kB 
[ebuild     U ] sys-apps/texinfo-4.8-r5 [4.8-r2] USE="-build* (-nls) -static" 0 kB 
[ebuild     U ] sys-devel/binutils-2.18-r1 [2.16.1-r3] USE="-multislot -multitarget (-nls) -test -vanilla" 0 kB 
[ebuild     U ] sys-devel/gcc-4.1.2 [4.1.1] USE="bootstrap* (-altivec) -build* -d% -doc -fortran -gcj -gtk (-hardened) -ip28 -ip32r10k -libffi% -mudflap (-multilib) -multislot (-n32) (-n64) (-nls) -nocxx -objc -objc++ -objc-gc -test -vanilla" 0 kB 
[ebuild     U ] sys-libs/uclibc-0.9.28.3 [0.9.28] USE="-build* -debug -hardened (-iconv) -ipv6 -minimal (-nls) -pregen -savedconfig -uclibc-compat -userlocales -wordexp" 0 kB 
[ebuild     U ] sys-apps/sysvinit-2.86-r10 [2.86-r5] USE="(-ibm) (-selinux) -static" 0 kB 
[ebuild  N    ] virtual/init-0  0 kB 
[ebuild     U ] sys-apps/baselayout-1.12.11.1 [1.12.6] USE="bootstrap* -build* -static -unicode" 0 kB 
[ebuild   R   ] sys-libs/zlib-1.2.3-r1  USE="(-build%*)" 0 kB 

Once you feel confident (and have a lot of free time to fix possible errors), execute the following command to start the bootstrap process:

./bootstrap.sh

Now that we have bootstrapped the system, it is generally a good idea to leave the chroot and backup the directory so that you can set off from this point if something goes wrong in the future.

First just exit the chroot.

exit

Now you should unbind all directories that were bound when chrooting.

umount $EMGEN_DIR/proc
umount $EMGEN_DIR/sys

umount $EMGEN_DIR/usr/portage/distfiles

umount $EMGEN_DIR/usr/portage

Immediately after bootstrapping the system, you must then reinstall all core system libraries using the newly bootstrapped tools. This is probably the most difficult part of the entire process, as you will more than likely run into some errors.

First, be sure to chroot into the system just as you did prior to bootstrapping (follow the section defined above).

In order to make sure everything on the system will run under the newly bootstrapped toolkit, we need to reinstall all system libraries. In order to do this, run the following command:

emerge -eav system

The flags passed to the emerge system mean the following:

• -e - (emptytree) assume the dependency tree is empty, so all dependencies of all files must be installed. This forces portage to install all system files again.
• -a - (ask) show the user what will be installed prior to starting, and let them decide if portage should continue with the install.
• -v - (verbose) be verbose in your description of what will be installed.

Once you have viewed everything that is to be installed and made sure it is to your liking (you may wish to enable various use-flags and the like), hit <key>Y</key> or <key>Enter</key> and begin the emerge process.

In a perfect world, the emerge would complete successfully and everything would work fine. Of course we are not in a perfect world, however, and these packages are very much still in the experimental stages. For that reason I have outlined in this section solutions to all of the problems that arose when I was attempting the install.

If the emerge does fail (and trust me, it probably will), do not panic. Whatever you do, do not type in emerge -eav system again! This will compute with an empty tree again and force you to start all over instead of where you left off!

In order to resume the emerge that you had before it failed, simply type:

emerge -av --resume

This will allow you to pick up where you left off.

In the event that the only way to fix the problem is to emerge a package out-of-order. There are different ways to do this.

Performing a second emerge typically runs the risk of wiping out your possibility to resume the initial one (this could be a problem). The best way to resolve this is to run the resume command, but like this:

emerge -p --resume > remerge

This will print out a list of packages that would be installed if portage were to resume, and save the list into a file called remerge in your current directory.

Now you can emerge whatever you need to, remove those packages from the list in the remerge file, and then use that list to populate an emerge command to resume where you left off.

If a build fails with a line similar to:

io.c:1150: error: 'AI_ADDRCONFIG' undeclared (first use in this function)

Then the problem is more than likely due to an outdated version of netdb.h. To fix the problem, execute the following commands:

cd /usr/include
cp ./netdb.h ./netdb.h.orig
nano ./netdb.h

Now add the following after line 403 (hitting <key>Ctrl-C</key> in nano give you the current line number):

# define AI_V4MAPPED    0x0008  /* IPv4 mapped addresses are acceptable. */
# define AI_ALL         0x0010  /* Return IPv4 mapped and IPv6 addresses. */
# define AI_ADDRCONFIG  0x0020  /* Use configuration of this host to choose returned address type. */

After adding those lines, hit <key>Ctrl-X</key>, <key>Y</key>, and <key>Enter</key> to save the file.

Now continue the emerge:

emerge -av --resume

For more information regarding this bug, please visit Bug #195368.

If a compile of a python library fails with a line stating something to the effect of:

gcc-config error: Could not run/locate "i386-gentoo-linux-uclibc-gcc"

If this is the case, then your python install is more than likely hard-coded to use an old version of GCC (the one prior to bootstrapping). To fix this you have to change it's configuration.

First run the following command:

python-config

The output should be something along the lines of:

-lpython2.4 -lm -L/usr/lib/python2.4/config

cd into the directory given that contains the configuration information:

cd /usr/lib/python2.4/config

Now run the following command to retrieve the current gcc profile:

gcc-config -c

You should see output similar to this:

i686-gentoo-linux-uclibc-4.1.2

You will need to replace the missing gcc version with the version just displayed, minus the actual version number.

Now run this command to replace all the old gcc CHOST values with the new ones:

for i in `ls -1`; do cp ${i} ${i}.bak; sed -r 's/i386-gentoo-linux-uclibc/i686-gentoo-linux-uclibc/gi' ${i}.bak > ${i}; done;

This essentially loops through all of the files in this directory and replaces the old value of i386-gentoo-linux-uclibc with the new value of i686-gentoo-linux-uclibc.

Now resume the emerge:

emerge -av --resume

If your compile of sys-fs/e2fsprogs fails due to a message similar to the following:

../../lib/libuuid.so: undefined reference to '___tls_get_addr'

The problem amounts from a faulty portage ebuild which makes sys-fs/e2fsprogs be configured assuming it will be built with glibc. uClibc does not contain TLS support, which is enabled in e2fsprogs by default.

Luckily, however, the nice people at Gentoo have already been working on this and the solution is to simply unmask the more recent package that contains a fix (namely version 1.40.5-r1).

To do this all you need to do is run the following command:

echo "=sys-fs/e2fsprogs-1.40.5-r1 ~x86" >> /etc/portage/package.keywords
echo "=sys-libs/ss-1.40.5 ~x86" >> /etc/portage/package.keywords
echo "=sys-libs/com_err-1.40.5 ~x86" >> /etc/portage/package.keywords

The only problem now is that when you attempt to resume the emerge, it will try to resume based on package version, and by that turn try to resume the old version of e2fsprogs. To solve this problem, you must resume using the following command:

TMP=""; \
for i in `emerge --resume -pv | grep "\[ebuild" - | sed -r "s/\[ebuild[^]]*\] //gi" - | awk '{print $1}' - | sed -r "s/e2fsprogs-.*$/e2fsprogs-1\.40\.5-r1/gi" -`; \
do TMP="${TMP} =${i}"; \
done; \
emerge -av ${TMP}

The above code is basically a very glorified way of automating parsing through the output of emerge –resume -p, removing everything but the package atoms, changing the version of e2fsprogs, compiling them all into a single line, and running emerge again.

For more information regarding this bug, please visit Bug #205102.

If your emerge fails with an error similar to the following:

Can't locate auto/POSIX/assert.al in @INC (@INC contains: . /etc/perl
/usr/lib/perl5/vendor_perl/5.8.8/i386-linux /usr/lib/perl5/vendor_perl/5.8.8
/usr/lib/perl5/vendor_perl /usr/lib/perl5/site_perl/5.8.8/i386-linux
/usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl /usr/lib/perl5/5.8.8/i386-linux
/usr/lib/perl5/5.8.8 /usr/local/lib/site_perl .) at ./../mk-script line 52

Then the problem is most likely related to an inadequate install of Perl. There is probably a high likelyhood that the output for emerge -p –resume includes an upgrade for Perl, but that it does not occur for awhile so the current package won't work.

The best way to get around this is to manually reorder the packages to be emerged, with all Perl packages coming next.

To do this, simply run this command:

TMP=""; \
for i in `emerge --resume -p | grep "\[ebuild" - | sed -r "s/\[ebuild[^]]*\] //gi" - | awk '{print $1}' - | grep "perl" -`; \
do TMP="${TMP} =${i}"; \
done; \
echo '#!/bin/bash' > ./install-perl.sh; \
echo "emerge -av ${TMP}" >> ./install-perl.sh; \
chmod 755 ./install-perl.sh; \
TMP=""; \
for i in `emerge --resume -p | grep "\[ebuild" - | sed -r "s/\[ebuild[^]]*\] //gi" - | awk '{print $1}' - | grep -v "perl" -`; \
do TMP="${TMP} =${i}"; \
done; \
echo '#!/bin/bash' > ./resume-nonperl.sh; \
echo "emerge -av ${TMP}" >> ./resume-nonperl.sh; \
chmod 755 ./resume-nonperl.sh

This will create two new scripts in the current directory. install-perl.sh, which runs an emerge to install all Perl-related packages that were pending in the resume, and resume-nonperl.sh, which resumes installation where it left off without the Perl packages. They should also be executable.

It is also generally a good idea to take a look at the files to make sure they were populated correctly:

cat {install-,resume-non}perl.sh

It should show something like this:

#!/bin/bash
emerge -av  =dev-lang/perl-5.8.8-r4 =perl-core/PodParser-1.35 =perl-core/Test-Harness-2.64
#!/bin/bash
emerge -av  =sys-apps/coreutils-6.9-r1 =sys-apps/groff-1.19.2-r1 =sys-apps/shadow-4.0.18.2 =sys-process/psmisc-22.6 =sys-apps/man-1.6f =sys-apps/man-pages-2.76 =sys-apps/diffutils-2.8.7-r2 =dev-lang/python-2.4.4-r6 =dev-python/python-fchksum-1.7.1 =sys-apps/file-4.23 =sys-apps/module-init-tools-3.4 =sys-apps/debianutils-2.28.2 =sys-apps/mktemp-1.5 =sys-apps/baselayout-1.12.11.1 =sys-fs/udev-115-r1

Now you should run them in succession. First, execute install-perl.sh to (re-)install Perl:

./install-perl.sh

Once this has completed (after resolving any errors or whatnot in-between), then run the resume script:

./resume-nonperl.sh

emerge -av --resume

Once you are done it is a good idea to remove the scripts from the filesystem:

rm ./{install-,resume-non}perl.sh

After everything has completed successfully, you will first have to re-source your shell so that changes in path take effect:

source /etc/profile
export PS1="(emgen-build) $PS1"

After that you will also want to update any configuration files currently held on your system:

etc-update

Since you haven't really set any configuration files, however, it should be save to just update all of the listed files automatically. So type in -5 and hit <key>Enter</key> to do so.

During the emerge -eav system process, it is possible that package dependencies were broken. To check for that, you must use the revdep-rebuild utility.

The revdep-rebuild program is available from the app-portage/gentoolkit package. You will need to emerge that first:

emerge -av gentoolkit

Now that it is installed, you should run the following command to check for broken packages:

revdep-rebuild -- -av

The above command will search for any broken linkage between libraries on your system, determine which packages they belong to, and then attempt to emerge them, appending all flags after the – switch to the end of the emerge command.

Now that we have a working build system with all of its core libraries bootstrapped, it is time to perform any updates to packages not declared to be part of the core system.

To save time on compiling we will not install the CCache compiling cache utility. This will cache compiled objects to be used during recompiles or future updates.

To install ccache, first you must emerge it:

emerge -av ccache

Now that CCache is installed, we need to set our cache size. Set it to something appropriate for your build environment. For us I will use a cache of 200M.

ccache -M 200M

Now that we have our caching mechanism setup, it is time to upgrade any and all packages on our build system that are outdated. To do this we perform a global world update:

emerge -DuavN world

Once this returns you will have an up-to-date build system, capable of creating your EmGen target system.

The next step is to install a Linux kernel source tree. There are several different trees available, although for general purposes it is usually a good idea to use the tree associated with Gentoo, gentoo-sources.

We will be using SquashFS 3.3 and will be adding LZMA support since it offers the best compression. For this reason we will also need to unmask some of the sources:

echo "sys-kernel/gentoo-sources ~x86" >> /etc/portage/package.keywords
emerge -av gentoo-sources

This needs to be installed before we start installing some other features, since some packages build modules which will need a kernel source tree available.

We will be using the Grand Unified Boot Loader (GRUB) as our boot manager (or at least a simplified version of it). In order to do this, we need to have it installed on the build system so that later it can be copied over to the target system.

emerge -av grub

We will also be using QEMU to emulate our system. In order to build the hard drive and such, we will need some toolkits from QEMU (specifically qemu-img).

The biggest problem is that QEMU requires some features not provided by uClibc or gcc-4. This means that we will not be able to build qemu-img on our build system. At this point you have two options:

1. Build app-emulation/qemu-softmmu on your host system
2. Download a static compilation: .tar.bz2

All that essentially matters is that you have access to a qemu-img binary. If you wish to go the easy route, just download the static version. It has been tested and should run within the emgen-build environment.

If you are going to be using the static compilation, be sure to download the file and place it in a location within your path:

tar -jxvf ./qemu-img.tar.bz2
cp -a qemu-img "${EMGEN_BUILD}"/usr/bin

We will also be using the JFS filesystem, so we need to install utilities to create/manage those filesystems:

emerge -av jfsutils

echo "sys-fs/jfsutils ~x86" >> /etc/portage/package.keywords

Now that you have a build system that is fully functional, it is a good idea to once again leave the chroot environment and backup the system. Follow the same steps as outlined in the previous section on leaving the Chroot to exit the environment and unmount any bound directories.

Once the build system is set up and ready to go, you need to prepare a target filesystem for your EmGen target.

As per the documentation above, once again chroot into the build system to perform all of the tasks in this section.

Now that we are in a chroot, we will create a directory in the chroot for our target system.

export EMGEN_TARGET="/etarget"
mkdir -p "${EMGEN_TARGET}"

From here on out I will be referring to the target as ${EMGEN_TARGET}.

echo "EMGEN_TARGET=\"/etarget\"" >> /etc/env.d/99emgen

To ease the management of packages on the target system, it is generally a good idea to create a script to handle all packages on the target system.

First, open nano:

nano

Now create a file that looks like the following:

#!/bin/bash
 
ROOT="${EMGEN_TARGET}" \
    INSTALL_MASK="*.h HACKING.gz INSTALL.gz README.gz TODO.gz" \
    emerge -avkN \
    $@

This file contains the following lines:

• ROOT=“${EMGEN_TARGET}“ - sets the root for all merges (similar to a –prefix directive).
• INSTALL_MASK=“*.h …“ - masks all files matching the given patterns, preventing unneeded files from being merged.
• emerge -avkN - set default emerge options to be verbose, ask for confirmation, use pre-built packages when possible, and check for new USE flags.
• $@ - append all given parameters to the end of the emerge command.

Save the file as etarget-merge (or something similar, by typing <key>Ctrl-X</key>, <key>Y</key>, type in the name, and then type <key>Enter</key>.

Now set execute permissions on the file, and move it into the path:

chmod 755 ./etarget-merge
mv ./etarget-merge /usr/bin/

Now we need to install some required packages using our new script. The following packages MUST be installed, otherwise the target system won't be able to run.

Every system needs a baselayout. This package creates a base root filesystem structure for your target. We want to minimize size, however, so we will use the baselayout-lite package. This package is still in testing stages, however so we need to first unmask it:

echo "sys-apps/baselayout-lite **" >> /etc/portage/package.keywords

Now we can install the baselayout into our target environment:

etarget-merge baselayout-lite

Since this entire system is built on the uClibc library, it should go without saying that we need to include the library in our target system so that any packages that require a C library will have it available.

uClibc should not be masked, so just install it:

etarget-merge uclibc

We need to have a base set of utilities / shell / etc. as well, so we use busybox to provide that. Although not usually done, we will also enable the make-symlinks USE flag so that the install will make the symlinks that are usually done by other programs/installs.

USE="make-symlinks" etarget-merge busybox

Since we are using a JFS filesystem, we will be installing the jfsutils package.

etarget-merge jfsutils

In the next section, we will be installing the packages that we will actually be using in the EmGen target environment. You can install whatever you want into your target environment, but the more you install (and what you install) will cause your target root size to become larger.

You should think about what you want your EmGen system to do (hopefully you did this before starting this project). For me, I will be going over how to create a basic (pseudo-)LAMP environment for web development. This way people could develop for their website regardless of what computer they are at.

The requirements for this system is the following:

• Linux - Well that's already taken care of.
• Apache - We won't actually be using Apache because of its size, but instead a simplified server called Cherokee.
• MySQL - We won't be able to use MySQL since it is a bit bulky and I was unable to get the embedded server to install properly. Instead we will use SQLite3 as our SQL solution since it is lightweight and also because it uses single files, making the used disk space quite a bit easier to manage.
• PHP - PHP is a staple of LAMP programming and it can't really be created without it in its entirety.

Cherokee is a lightweight and simple HTTP server. Although there are lots of lightweight servers, I chose Cherokee because it is lightweight but also supports FastCGI, which will come in useful in speeding up PHP (since lightweight servers only support PHP through the CGI interface).

Cherokee is still masked by keyword at this time so first unmask it, then install it into the target:

echo "www-servers/cherokee ~x86" >> /etc/portage/package.keywords
etarget-merge cherokee

This may install some dependencies, but it should be minimal. If anything make sure there are no USE flags set for the install as they are probably not needed anyway.

After the install, portage will attempt to create users for the cherokee daemon. The only problem is that it will create them outside of the specified ROOT. It is important to create a user and group for the cherokee daemon within the ROOT. To do this, run the next commands:

grep cherokee /etc/passwd >> "${EMGEN_TARGET}"/etc/passwd
grep cherokee /etc/group >> "${EMGEN_TARGET}"/etc/group
grep cherokee /etc/shadow >> "${EMGEN_TARGET}"/etc/shadow

SQLite is a simplified SQL Server that is managed by simply using flat-files and some C programs. Although it sounds rather slow and bulky, it is actually comparable to larger server-based programs such as MySQL or PostgreSQL.

Since we need to conserve some space and the minimal install of MySQL only results in client-side programs (I attempted the embedded server but to no avail), we will be using SQLite as our backend.

We don't actually have to explicitly install SQLite as it will be installed as a dependency when we install PHP.

Our last step is to install PHP on our target. This will by far be the most bulky package we will be installing, and quite a few USE flags that may be set. You should make sure you only set the ones you think you'll need for your environment, as every flag used will add more bulk to the install.

I used the following USE flags:

• bzip2 - I like to have support for bzlib compression.
• cgi - THIS MUST BE ENABLED OR WE WON'T BE ABLE TO ATTACH PHP TO OUR HTTP SERVER
• cli - To allow PHP to be accessed via command-line.
• crypt - Support for encryption.
• ctype - cType functions can be useful.
• curl - The cURL library is very useful when doing HTTP or other URL-related requests.
• force-cgi-redirect - A common thing to enable in the CGI SAPI.
• gd - Add internal graphics support.
• gmp - Add support for the GNU Mupltiple Precision Library
• hash - Enable the hash extension
• json - Add support for JSON so we can talk with JavaScript easily.
• mhash - Support the mhash library.
• ncurses - Add ncurses support.
• pcre - Add support for Perl Compatible Regular Expressions.
• readline - Add support for GNU line-editing.
• reflection - Enable the Reflection API.
• session - Add support for sessions.
• simplexml - Add support for SimpleXML.
• soap - Add support for the Simple Object Access Protocol.
• sockets - Sockets can be quite useful to have support for.
• spl - Add support for the Standard PHP Library.
• sqlite - Add support for SQLite
• tidy - Add support for HTML Tidy.
• tokenizer - Add support for the PHP file parser.
• unicode - Support Unicode characters.
• xml - Support XML files.
• xmlreader - Enable XMLReader.
• xmlwriter - Enable XMLWriter.
• zip - Add ZIP compression support.
• zlib - Add zlib compression support.

To perform the install, simply run the following command:

USE="bzip2 cgi cli crypt ctype curl force-cgi-redirect gd gmp hash \
json mhash ncurses pcre readline reflection session simplexml soap \
sockets spl sqlite tidy tokenizer unicode xml xmlreader xmlwriter \
zip zlib" etarget-merge php

We need to install a DHCP client so that we can connect our NIC to the virtual DHCP server, and thusly server our pages to the host system:

etarget-merge dhcpcd

Now that we have our packages setup, it is time to get to the nitty-gritty and compile our kernel that will be used to run all of this.

First, cd into the kernel build directory:

cd /usr/src/linux

Since we are attempting to build a tiny kernel, it is a good idea to start off by removing all configuration options, we will set them on our own:

make mrproper
make allnoconfig

This will create a default .config file that will have no options set. Now it is time to set those options. To open up the configuration menu, type the following command:

make menuconfig

This will open up a curses-based menu configuration program. Although configuring a kernel is typically somewhat difficult, we have a trick that will make it easier: qemu! Since we will be using qemu to run our system, we already know what hardware will appear to be present regardless of what system the emulator is run on.

That being said, the configuration should be as follows for the kernel:

    General Setup  --->
    [*] Prompt for development and/or incompleted code/drivers
    [*] Support for paging of anonymous memory (swap)
    [*] System V IPC
    [*] BSD Process Accounting
    [*]     BSD Process Accounting version 3 file format
    (14) Kernel log buffer size
    [*] Initial RAM filesystem and RAM disk support
    [*] Optimize for size (Look out for broken compilers!)
    [*] Configure standard kernel features (for small systems)  --->
        --- Configure standard kernel features (for small systems)
        [*] Enable 16-bit UID system calls
        [*] Support for hot-pluggable devices
        [*] Enable support for printk
        [*] Enable full-sized data structures for core
        [*] Enable futex support
    [*] Enable eventpoll support
    [*] Enable signalfd() system call
    [*] Enable eventfd() system call
    [*] Use full shmem filesystem
    [*] Enable VM event counters for /proc/vmstat
        Choose SLAB allocator (SLUB (Unqueued Allocator))  --->

[*] Enable loadable module support  --->
    [*] Module unloading
    [*] Automatic kernel module loading

[*] Enable the block layer  --->
    --- Enable the block layer
          IO Schedulers --->
            <*> CFQ I/O scheduler
                Default I/O scheduler (CFQ)  --->

    Processor type and features  --->
        Subarchitecture Type (PC-compatible)  --->
        Processor family (586/K5/5x86/6x86/6x86MX)  --->
        Preemption Model (Preemptible Kernel (Low-Latency Desktop))  --->
        Hich Memory Support (off)  --->
        Memory split (3G/1G usr/kernel split)  --->
        Memory model (Flat Memory)  --->
    [*] Math emulation
    [*] MTRR support
        Timer frequency (250 HZ)  --->
    (0x100000) Physical address where the kernel is loaded
    (0x100000) Alignment value to which kernel should be aligned

    Bus options (PCI, PCMCIA, EISA, MCA, ISA)  --->
    [*] PCI support
          PCI access mode (Any)  --->
    [*] ISA support

    Executable file formats  --->
    [*] Kernel support for ELF binaries
    <*> Kernel support for a.out and ECOFF binaries
    <*> Kernel support for MISC binaries

    Networking  --->
    [*] Networking support
        Networking options  --->
        <*> Packet socket
        <*> Unix domain sockets
        [*] TCP/IP networking
        <*>   IP: IPsec transport mode
        <*>   IP: IPsec tunnel mode
        <*>   IP: IPsec BEET mode
        <*>   INET: socket monitoring interface

    Device Drivers  --->
        Generic Driver Options  --->
        (/sbin/hotplug) path to uevent helper
    [*] Block devices  --->
        --- Block devices
        <M>   Normal floppy disk support
        <*>   Loopback device support
        <*>   RAM disk support
        (16)    Default number of RAM disks
        (4096)  Default RAM disk size (kbytes)
        (1024)  Default RAM disk block size (bytes)
    <*> ATA/ATAPI/MFM/RLL support  --->
        --- ATA/ATAPI/MFM/RLL support
        (4)   Max IDE interfaces
        <*>   Enhanced IDE/MFM/RLL disk/cdrom/tape/floppy support
        <*>     Include IDE/ATA-2 DISK support
        <*>     Include IDE/ATAPI CDROM support
        <M>     Include IDE/ATAPI FLOPPY support
        [*]     legacy /proc/ide/ support
        <*>     generic/default IDE chipset support
        <*>     Generic PCI IDE Chipset support
        <*>     Intel PIIXn chipsets support
        SCSI device support  --->
        <*> SCSI device support
        [*] legacy /proc/scsi/ support
        <*> SCSI disk support
        <*> SCSI CDROM support
        <*> SCSI generic support
    [*] Network device support  --->
        --- Network device support
        [*]   Ethernet (10 or 100Mbit)  --->
            --- Ethernet (10 or 100Mbit)
            <M>   AMD LANCE and PCnet (AT1500 and NE2100) support
            [*]   Other ISA cards
            <M>   NE2000/NE1000 support
            [*]   EISA, VLB, PCI and on board controllers
            <M>   PCI NE2000 and clones support
        Input device support  --->
        -*- Generic input layer (needed for keyboard, mouse, ...)
        [*]   Keyboards  --->
            --- Keyboards
            <*>   AT keyboard
        Character devices  --->
        [*] Virtual terminal
        [*]   Support for console on virtual terminal
        [*] Unix98 PTY support
        Graphics support  --->
            Console display driver support  --->
            [*] VGA text console
            [*]   Video mode selection support

    File systems  --->
    <*> JFS filesystem support
    <*> ROM file system support
    [*] Dnotify support
    <*> Kernel automounter version 4 support
        Pseudo filesystems  --->
        [*] /proc file system support
        [*]   Sysctl support (/proc/sys)
        [*] sysfs file system support
        [*] Virtual memory file system support (former shm fs)
        Miscellaneous filesystems  --->
        <*> Compressed ROM file system support (cramfs)

    Kernel hacking --->
    [*] Enable doublefault exception handler

    Library routines  --->
    {*} CRC32 functions

Now that you have your kernel configured, make the kernel with the following function:

make -j2

Now you need to install the modules for the kernel. This is done with the following command:

make modules_install

If you noticed, we did not enable the SquashFS module/builtin capabilities. This is because we will be patching the driver to enable LZMA compression versus GZip compression. LZMA is much more effective. In this section we will be downloading all of the patches, source, etc., needed to compile the SquashFS+LZMA module.

First we will be needing to create a build directory for our module:

mkdir -p /tmp/sqfs-lzma
cd /tmp/sqfs-lzma
<code>
 
=== Download the Sources ===
 
Next we need to download all of the sources for our build:
 
<code bash>
wget http://www.squashfs-lzma.org/dl/squashfs3.3.tar.gz
wget http://www.squashfs-lzma.org/dl/squashfs-cvsfix.patch
wget http://superb-east.dl.sourceforge.net/sourceforge/sevenzip/lzma449.tar.bz2
wget http://www.squashfs-lzma.org/dl/sqlzma3.3-457.tar.bz2

After downloading the sources, we need to unpack everything.

First, we will unpack the sqlzma patches:

mkdir -p sqlzma3.3-457
tar -jxvf sqlzma3.3-457.tar.bz2 -C sqlzma3.3-457

Next, unpack the lzma449 files into a subdirectory of our lzma patchset:

mkdir -p sqlzma3.3-457/lzma449
tar -jxvf lzma449.tar.bz2 -C sqlzma3.3-457/lzma449

Lastly, unpack the squashfs3.3 files:

tar -zxvf squashfs3.3.tar.gz -C sqlzma3.3-457

Once everything is unpacked, be sure to enter the sqlzma directory for making:

cd sqlzma3.3-457

We will also be needing to patch the linux kernel to build the squashfs module. To provide a method for this, create a link, linux, in the sqlzma directory. We will also make a versioned link to make it easier to patch:

ln -s /usr/src/linux linux
ln -s linux linux-2.6.24

First we need to apply the sqlzma patchset to the LZMA tree:

patch -p0 < ./sqlzma1-449.patch

The next step is to apply the CVS patchset to the SquashFS source tree:

patch -p0 < ../squashfs-cvsfix.patch

After that we need to apply the kernel patches from the SquashFS tree, as well as the sqlzma patshet.

patch -p0 < squashfs3.3/kernel-patches/linux-2.6.24/squashfs3.3-patch
patch -p0 < sqlzma2k-3.3.patch

Lastly, we need to apply the user tools patch:

patch -p0 < sqlzma2u-3.3.patch

There is an error in the Makefile that needs to be corrected regarding the directory structure and the kernel version, to solve that, use this command (assuming the flavor of Gentoo-Sources-2.6.24 is 2.6.24-gentoo-r3:

cp Makefile Makefile.old; \
    sed -r 's/\$\(shell uname -r\)/2.6.24-gentoo-r3/g' Makefile.old | \
    sed -r 's/^SqFs =.*$/SqFs = ${Sqlzma}\/linux\/fs\/squashfs/g' - > Makefile

Sadly, we will also need to fix an error in SquashFS-3.3 regarding the use of get_nprocs(). Thanks to the wonderful people in the Gentoo Developer Community, however, we don't have to worry about how. This problem is already included as a patch in SquashFS-3.3! We can't apply this patch however since we have already applied patches to the required files. The bonus is that they do tell us how to fix the problem.

To fix the problem, we need to open mksquashfs.c, and change the line ourselves:

nano ./squashfs3.3/squashfs-tools/mksquashfs.c

Then we need to go to line ~2847 (use <key>Ctrl-C</key> to see the current line), and change the value from:

                processors = get_nprocs();

to

                processors = sysconf(_SC_NPROCESSORS_CONF);

Then save the file (<key>Ctrl-X</key>, <key>Y</key>, <key>Enter</key>, and that should be it!

Now that we have everything we need patched, it is time to build the modules.

make

Now we need to copy the modules over to the build environment so that they are loadable:

mkdir /lib/modules/2.6.24-gentoo-r3/misc
cp linux/fs/squashfs/squashfs.ko /lib/modules/2.6.24-gentoo-r3/misc/
cp lzma449/C/Compress/Lzma/kmod/sqlzma.ko /lib/modules/2.6.24-gentoo-r3/misc/
cp lzma449/C/Compress/Lzma/kmod/unlzma.ko /lib/modules/2.6.24-gentoo-r3/misc/

We will need to add the modules to the modules.dep list as well to make sure they get used properly:

echo "/lib/modules/2.6.24-gentoo-r3/misc/squashfs.ko: /lib/modules/2.6.24-gentoo-r3/misc/sqlzma.ko /lib/modules/2.6.24-gentoo-r3/misc/unlzma.ko" >> /lib/modules/2.6.24-gentoo-r3/modules.dep
echo "/lib/modules/2.6.24-gentoo-r3/misc/sqlzma.ko:" >> /lib/modules/2.6.24-gentoo-r3/modules.dep
echo "/lib/modules/2.6.24-gentoo-r3/misc/unlzma.ko:" >> /lib/modules/2.6.24-gentoo-r3/modules.dep

We also need to copy over a few libraries so that they may be used:

cp lzma449/C/Compress/Lzma/libunlzma.a /usr/lib/
cp lzma449/C/Compress/Lzma/libunlzma_r.a /usr/lib/

And lastly we need to copy over the utility binaries:

cp squashfs3.3/squashfs-tools/mksquashfs /usr/bin/
cp squashfs3.3/squashfs-tools/unsquashfs /usr/bin/

And now we're all done!

Now that we have all of the modules we need installed on our build system, we need to transfer them over to the target system.

To do this, we enter the following commands:

rm -r "${EMGEN_TARGET}"/lib/modules
mkdir -p "${EMGEN_TARGET}"/lib/modules
cp -a /lib/modules/* "${EMGEN_TARGET}"/lib/modules/

You now also have a target system to be installed on the device of your choice! It is once again a good idea to leave the Chroot environment and save a backup of your work thus far. (Trust me, you'll regret it if you make a mistake in the future!). To leave the Chroot completely, follow the section regarding the topic in the above statements.

Now that we have our target machine setup pretty much exactly the way we want it, we need to make a few extra changes to it to ensure that it will boot and load properly.

export EMGEN_TARGET="/emgen/etarget"

The installs for our system do not include any init scripts for cherokee or mysql. In order to make sure they start at boot time, we need these scripts to be created.

BusyBox (which we installed for our target), provides a mechanism by which all scripts in /etc/init.d are started at boot and killed at system halt. For that reason we will create two scripts to do just that.

Because of how our system is going to be created, we will need to remount the root filesystem to make it writable.

rm -f ${EMGEN_TARGET}/etc/init.d/mount
nano ${EMGEN_TARGET}/etc/init.d/mount

Type (or copy) the following into the file:

#!/bin/sh
 
case $1 in
start)
        # perform a filesystem check on the root device
        fsck.jfs -fa /dev/root
        # make the root read/write
        echo "Remounting Root Filesystem Read/Write..."
        mount -oremount,rw,noatime -n /dev/root /
        # mount our squashed filesystem to the root
        echo "Loading SquashFS-LZMA Module..."
        modprobe squashfs
        echo "Mounting /usr squashed filesystem..."
        mount -t squashfs -oloop /usr.sfs /usr
        echo "Mounting /etc/cherokee squashed filesystem..."
        mount -t squashfs -oloop /etc/cherokee.sfs /etc/cherokee
        ;;
stop)
        # umount our squashed filesystem
        echo "Unmounting /etc/cherokee squashed filesystem..."
        umount /etc/cherokee
        echo "Unmounting /usr squashed filesystem..."
        umount /usr
        echo "Unloading SquashFS-LZMA Module..."
        modprobe -r squashfs
        # remount root read-only
        echo "Remounting Root Filesystem Read-only..."
        mount -oremount,ro -n /dev/root /
        ;;
esac

Now save the file (<key>Ctrl-X</key>, <key>Y</key>, <key>Enter</key>), then make it executable:

chmod +x ${EMGEN_TARGET}/etc/init.d/mount

We will also need a network to be present, so let's start that up here:

rm -f ${EMGEN_TARGET}/etc/init.d/net
nano ${EMGEN_TARGET}/etc/init.d/net

Type (or copy) the following into the file:

#!/bin/sh
 
case $1 in
start)
        echo "Loading Network Modules..."
        modprobe ne2k-pci
 
        echo "Bringing Up Eth0 Network Interface..."
        ifconfig eth0 up
        dhcpcd -L eth0
        ;;
stop)
        echo "Shutting Down Eth0 Network Interface..."
        dhcpcd -k eth0
        ifconfig eth0 down
 
        echo "Unloading Network Modules..."
        modprobe -r ne2k-pci
        ;;
esac

Now save the file (<key>Ctrl-X</key>, <key>Y</key>, <key>Enter</key>), then make it executable:

chmod +x ${EMGEN_TARGET}/etc/init.d/net

We will need to create a script to boot our cherokee server:

rm -f ${EMGEN_TARGET}/etc/init.d/cherokee
nano ${EMGEN_TARGET}/etc/init.d/cherokee

Type (or copy) the following into the file:

#!/bin/sh
 
PIDFILE=/var/run/cherokee.pid
 
case $1 in
start)
        echo "Starting Cherokee HTTPD Server..."
        start-stop-daemon --quiet --start --pidfile ${PIDFILE} \
                --exec /usr/sbin/cherokee -- -b &
        ;;
stop)
        echo "Stopping Cherokee HTTPD Server..."
        start-stop-daemon --quiet --stop --pidfile ${PIDFILE} &
        ;;
esac

Now save the file (<key>Ctrl-X</key>, <key>Y</key>, <key>Enter</key>), then make it executable:

chmod +x ${EMGEN_TARGET}/etc/init.d/cherokee

Since we will be storing our cherokee PID file under /var/run, we need to make sure it exists:

mkdir -p ${EMGEN_TARGET}/var/run

Last but not least we need to create some symbolic links so that the system will automatically start/stop the scripts we created in the right order on startup nad shutdown.

ln -s mount ${EMGEN_TARGET}/etc/init.d/S01
ln -s net ${EMGEN_TARGET}/etc/init.d/S02
ln -s cherokee ${EMGEN_TARGET}/etc/init.d/S03
ln -s cherokee ${EMGEN_TARGET}/etc/init.d/K01
ln -s net ${EMGEN_TARGET}/etc/init.d/K02
ln -s mount ${EMGEN_TARGET}/etc/init.d/K03

The init system for BusyBox will execute all scripts in /etc/init.d, in numerical order, with S?? being the startup scripts and K?? being the kill scripts.

Now it is generally a good idea to test your binaries that will be run to ensure that no dependencies have been left out in the fray (because they were installed to the build system but not the target system).

In order to perform the tests, you will need to chroot into the target system:

chroot $EMGEN_TARGET /bin/ash --login

Now the files you will want to test are:

/usr/sbin/cherokee
/usr/lib/php5/bin/php
/usr/lib/php5/bin/php-cgi
/usr/sbin/dhcpcd
/sbin/fsck.jfs (or similar)

Run each of these programs. If one of them fails to run because of an error such as:

./php: can't load library 'libpcre.so.0'

Then you will have to re-install each library that it states using the build environment. To re-install a package from the build environment, first you must make sure you are chrooted into the emgen-build environment.

Next, run the following command to determine which package contains the library you need (for example, with libpcre.so.0):

equery belongs libpcre.so.0

This should yield where the library is and to which package it belongs:

[ Searching for file(s) libpcre.so.0 in *... ]
dev-libs/libpcre-7.6-r1 (/usr/lib/libpcre.so.0 -> libpcre.so.0.0.1)

At this point you can either:

1. Copy the library itself to the system (recommended for space)
2. Install the package to the system (recommended for stability)

It is recommended to copy the library to the system since it will conserve a lot of space and prevent potentially unneeded dependencies from being installed on the target.

To copy the libraries to the system:

cp -a /usr/lib/libpcre.so.0 "${EMGEN_TARGET}"/usr/lib
cp -a /usr/lib/libpcre.so.0.0.1 "${EMGEN_TARGET}"/usr/lib

This option is not recommended for space since a lot of extra (potentially unnecessary packages) will probably be installed as dependencies.

To install the package, simply:

etarget-merge libpcre

Next you need to configure cherokee to run PHP scripts. To do this, first open the cherokee configuration file:

nano "${EMGEN_TARGET}"/etc/cherokee/sites-enabled/default

Now scroll to the bottom of the default file where it has something to the effect of:

Extension php, php3, php4 {
                Handler phpcgi
}

Change it to this:

Extension php, php3, php4, php5 {
        Handler phpcgi {
                Interpreter "/usr/lib/php5/bin/php-cgi"
        }
}

Now save (<key>Ctrl-X</key>, <key>Y</key>, <key>Enter</key>).

Your web server should now support PHP!

Just for good measure, it is generally a good idea to set the target's hostname, for networking purposes:

echo "emgen" >> "${EMGEN_TARGET}"/etc/hostname

In order to be able to log in once you boot the target, you need to have the root password set. To do this, first chroot into the target:

chroot "${EMGEN_TARGET}" /bin/ash --login

Now that you are logged in to the system, create a root password:

passwd

Then exit the chroot:

exit

The system also needs a few links for the loop devices to ensure that they function properly with some of the losetup/mount operations:

mkdir -p "{$EMGEN_TARGET}"/dev/loop
for i in "{$EMGEN_TARGET}"/dev/loop?*; do ln -s ../`basename $i` "${EMGEN_TARGET}"/dev/loop/`echo $i | sed -r 's/^.*loop([0-9]+).*$/\1/gi' - `; done

This should make symbolic links for each of the loopback devices, which may be useful in the future.

We need to modify the emgen-target's Filesystem Tab file (/etc/fstab) in order to make sure that some filesystems are mounted automatically at boottime.

Open the file as such:

nano "${EMGEN_TARGET}"/etc/fstab

And edit it to look something like the following:

# default ram disk as root

/dev/root               /               jfs             noatime         1 2
none                    /proc           proc            defaults        0 0

# glibc 2.2 and above expects tmpfs to be mounted at /dev/shm for
# POSIX shared memory (shm_open, shm_unlink).
# (tmpfs is a dynamically expandable/shrinkable ramdisk, and will
#  use almost no memory if not populated with files)
# Adding the following line to /etc/fstab should take care of this:

none                    /dev/shm        tmpfs           defaults        0 0

Then save the file (<key>Ctrl-X</key>, <key>Y</key>, <key>Enter</key>).

Now for BusyBox compatibility, we need to create a symbolic link for the root device:

ln -s hda1 "${EMGEN_TARGET}"/dev/root

Now that we have everything that we need to setup the hard drive, we need to create a temporary creation environment.

Follow the aforementioned instructions to do this (once again).

We will be building a copy of our emgen-target system so that we can modify it at will without having to worry about making mistakes. All of our work will be done under the directory that we create for the build. To reset this directory, execute the following commands:

rm -fr /tmp/ecreate
mkdir -p /tmp/ecreate

Also move into the directory you just created.

cd /tmp/ecreate

Now that we have a temporary environment, we need to copy over everything we've created in the target. We do an archival copy to ensure that we save all of our permissions and such.

cp -a "${EMGEN_TARGET}" ./etarget

We now need to install our boot loader, GRUB. This will allow our hard drive to boot the kernel directly.

First step is to copy our build environment's /boot directory to our target system (don't worry about sizes, we will clean it up manually):

cp -a /boot ./etarget/

Now it is time to get a little messy and remove some of the pieces of GRUB that are not needed. First, move into the GRUB directory:

cd ./etarget/boot/grub

Now we don't need most of these files, so first save the stage1 and stage2 installs:

mv stage1 ..
mv stage2 ..

Now depending on which filesystem you are using for your root directory, your stage1_5 is going to be different. The directory generally contains stages of the form “${FS_TYPE}_stage1_5”. Since we are using JFS, we need to save jfs_stage1_5:

mv jfs_stage1_5 ..

Now clean out the directory:

rm *

Then reinstate our saved files:

mv ../stage1 ./
mv ../stage2 ./
mv ../*_stage1_5 ./

Now we need to create the grub.conf, menu.lst, and device.map files. To start with, just create them, then we will edit them:

touch ./grub.conf
touch ./device.map
ln -s ./grub.conf ./menu.lst

Easiest is to populate is the device.map file, so let's do that:

echo "(hd0)   /dev/hda" > ./device.map

Now we need to create our configuration file:

nano ./grub.conf

Fill the file with something similar to this:

timeout 10
default 0
fallback 0

title EmGen
        root (hd0,0)
        kernel /boot/vmlinuz root=/dev/hda1

Then save the file (<key>Ctrl-X</key>, <key>Y</key>, <key>Enter</key>).

Now that we are done configuring GRUB, we need to return to our original directory:

cd ../../../

Our system would be nothing without the Kernel we created, so let's install it:

cp /usr/src/linux/arch/i386/boot/bzImage ./etarget/boot/vmlinuz

This is the step where we shrink down our system. Using SquashFS, we can reduce some of our file tree. I recommend compressing any directories (of substance) that require read-only access.

First off, we will clean any unneeded files from the /usr filesystem. After scanning through most of the directory, the safest thing to remove from the directory is the include directory, since we won't be performing any compiles on the target system:

rm -fr ./etarget/usr/include

Another step that can save some space on your system, is to move all of the library files not needed until later on in the boot process to the /usr directory, so that they will be squashed. For example, in order to boot your system, you should only need the libraries needed to run the files in the /bin and /sbin directories, as well as any corresponding libraries for inter-library links. Therefore, any other extra libraries can be thrown into /usr/lib.

In order to perform this, we need to check each runtime to get a list of libraries that need to be saved:

for i in ./etarget/{bin,sbin}/*; do if [ ! -L $i ]; then ldd $i | awk '{print $3}' - >> libs; fi; done
for i in `cat libs`; do if [ -e $i ]; then echo "$i" >> libs_2; fi; done
touch libs_3
for i in `cat libs_2`; do if [ -z "`grep $i libs_3`" ]; then echo "$i" >> libs_3; fi; done

These commands first list all libraries, then get rid of any excess garbage, then removes duplicates. Now that we have our list of libraries we need to add to that list any libraries that are required by them:

for i in `cat libs_3`; do ldd $i | awk '{print $3}' - >> libs_4; done
for i in `cat libs_4`; do if [ -e $i ]; then echo "$i" >> libs_5; fi; done
cp libs_3 libs_6
for i in `cat libs_5`; do if [ -z "`grep $i libs_6`" ]; then echo "$i" >> libs_6; fi; done

Now at this point, the file libs_6 contains all of the libraries that we need for our root filesystem to function. Anything else can be placed into /usr/lib. There is really no easy way that I know of to do this, except to compare the library list and the files in /lib.

Now we can create our SquashFS file.

To conserve the greatest amount of space, we will also be changing the default block size to the largest amount supported, 1024k.

mksquashfs ./etarget/usr/* ./etarget/usr.sfs -b 1048576

Now we have no more need for the /usr directory, so empty it:

rm -fr ./etarget/usr/*

The cherokee configuration directory does not need to be read-write, since it will only be read by the server, and not written to:

mksquashfs ./etarget/etc/cherokee/* ./etarget/etc/cherokee.sfs -b 1048576
rm -fr ./etarget/etc/cherokee/*

Since we used Gentoo to create our target system, there are some unneeded files left over from the process. It is now time to remove them in order to free up some space:

rm ./etarget/etc/*-
rm -fr ./etarget/etc/portage
rm -fr ./etarget/var/cache/edb
rm -fr ./etarget/var/db/pkg
rm -fr ./etarget/var/lib/portage

Here is an explanation of what each directory is and why we don't need them:

• /etc/*- - Sometimes backup files are saved with a dash (-) afterwards.
• /etc/portage - A list of keywords/masks/etc for portage.
• /var/cache/edb - A cache of various emerge data.
• /var/db/pkg - This stores information about portage installs.
• /var/lib/portage - Another location of information regarding portage installs.

Now we need to create a virtual hard drive to store our system on.

In order to make sure that everything will fit on the drive, you should first check how much space your system takes up:

du -shc etarget

Now take the total amount into consideration when determining how big you should make your disk image.

Now make the image:

qemu-img create -f raw hda.img 24M

Next we need to create a partition table on the hard drive, so first create a loopback device for the hard drive image, and then execute fdisk on it:

losetup /dev/loop0 ./hda.img
fdisk /dev/loop0

Create a new partition (filling up all of the space), and then exit, writing the data to disk. To do this, simply type the following keys in sequence: <key>n</key>, <key>Enter</key>, <key>p</key>, <key>Enter</key>, <key>1</key>, <key>Enter</key>, <key>Enter</key>, <key>Enter</key>, <key>w</key>, <key>Enter</key>.

Now that you have the partition table created, remove the loop device:

losetup -d /dev/loop0

You know have a simulated hard disk with one empty partition.

Now we need to format the target partition to our filesystem type (we chose JFS).

To do this, we first need a loopback device setup that starts at the partition. To find the starting point of that partition, execute the following command:

sfdisk -l -uS ./hda.img

This should yield something like:

Disk hda.img: cannot get geometry

Disk hda.img: 3 cylinders, 255 heads, 63 sectors/track
Units = sectors of 512 bytes, counting from 0

   Device Boot    Start       End   #sectors  Id  System
 hda.img1            63     48194      48132  83  Linux
 hda.img2             0         -          0   0  Empty
 hda.img3             0         -          0   0  Empty
 hda.img4             0         -          0   0  Empty

We can see that according to sfdisk, our virtual hard drive is made up of sectors of 512 bytes each and that our primary partition starts at sector 63. Using some basic math, we can deduce that the primary partition of our virtual hard drive starts at byte 512 * 63 = 32256.

Now we just create a loopback device with that offset:

losetup -o 32256 /dev/loop0 ./hda.img

To make sure that we format the right number of blocks, we need to check how many there are on our partition, so perform the following command:

sfdisk -l ./hda.img

This should yield something like:

Disk ./hda.img: cannot get geometry

Disk ./hda.img: 3 cylinders, 255 heads, 63 sectors/track
Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
./hda.img1          0+      2       3-     24066   83  Linux
./hda.img2          0       -       0          0    0  Empty
./hda.img3          0       -       0          0    0  Empty
./hda.img4          0       -       0          0    0  Empty

We can see that there are 24066 blocks. It is generally a good idea to make your block count be a multiple of 4, so you should set your block count to ( total_blocks - (total_blocks mod 4) ) = ( 24066 - 2 ) = 24064.

The problem is that qemu-img uses a block size of 1024 bytes, while mkfs_jfs uses a default block size of 4096 (which is 1024*4). So to figure out how many JFS blocks to install on the partition, we divide our number by 4, and get 6016.

Then format the partition, using the specified block size:

mkfs.jfs /dev/loop0 6016

When asked whether to continue, hit <key>Y</key>, <key>Enter</key>.

Then once the formatting is completed, delete the loopback device:

losetup -d /dev/loop0

Now we have a formatted, partitioned virtual hard drive, and the next step is to install all of our system files onto it.

In order to load the files onto the partition, we will first need to mount it. Before that can be done we need a place to mount it to, so create the mount point:

mkdir -p ./mnt

Now you can actually mount the partition. For this to work you will also need to use the offset from before, but you can skip the creation of a loopback device by letting the mount command handle it.

mount -o loop,offset=32256 ./hda.img ./mnt

modprobe -l | grep jfs -

modprobe jfs

Now that we have our mounted partition, it is time to copy over all of our hard work. This can be done simply by doing this:

cp -a ./etarget/* ./mnt/

Now we no longer need to access the virtual hard drive, so just unmount it:

umount ./mnt

To conserve some extra space, it is generally a good idea to convert the virtual hard drive image from RAW format (which it is in now) to QCOW2 format (a QEMU expandable format). The benefit is that QCOW2 will only take up as much space as it needs to, but will dynamically expand to the maximum.

qemu-img convert -f raw hda.img -O qcow2 hda2.img
rm hda.img
mv hda2.img hda.img

Follow the aforementioned instructions to do this (once again).

Although our hard drive does have GRUB installed, it is not installed to the MBR so therefore it will not work. To get it to work, we need to do some extra work since we now have a QCOW2 image.

First off, we need a bootable floppy so that we can boot into a grub command line and then run the install from there.

To do this, run the following commands:

dd if="${EMGEN_BUILD}"/boot/grub/stage1 of="${EMGEN_BUILD}"/tmp/ecreate/fda.img bs=512 count=1
dd if="${EMGEN_BUILD}"/boot/grub/stage2 of="${EMGEN_BUILD}"/tmp/ecreate/fda.img bs=512 seek=1

Now that we have our bootable floppy, we need to run QEMU to install the GRUB to the MBR.

emerge -av qemu

mkdir -p ~/emgen
cp "${EMGEN_BUILD}"/tmp/ecreate/*.img ~/emgen/
chown myuser:mygrp -R ~/emgen

rm "${EMGEN_BUILD}"/tmp/ecreate/*.img
mv ~/emgen/*.img "${EMGEN_BUILD}"/tmp/ecreate/
chown root:root "${EMGEN_BUILD}"/tmp/ecreate/*.img

Now that all of that is cleared up, run QEMU with both the hard disk and floppy images, booting from the floppy:

qemu -fda fda.img -hda hda.img -boot a

When GRUB boots up, install it to the hard drive's MBR using these commands:

root (hd0,0)
setup (hd0)
halt

Now the system should exit and you officially have a complete Hard Drive Image that is bootable, and contains your virtualize system! Congrats!

• TinyGentoo - The HOWTO that this project is based on. Their idea was to create a system for a flash drive, I'd like to take it a bit further.
• Embedded Gentoo - A tutorial on how to install Gentoo on an embedded device from stage 1. Also a strong foundation for this document.
• Official Squashfs LZMA - Official site dedicated to using SquashFS with LZMA.
• Linux Kernel Documentation - Provides a lot of useful information about how the Linux Kernel works.
• ro root fs w/ busybox? - A mailing list reply explaining how to remount /dev/root as read/write.
• QEMU on Windows - A port of QEMU to Windows so that we can run our Linux system on Windows.
• QEMU Documentation - Useful information about all of the options that QEMU provides.
• Creating a GRUB boot floppy - Information from the GRUB manual on how to create a bootable floppy disk.
• A Minimal Linux System From Scratch - A similar project that was very useful in explaining some specifics regarding QEMU image creation.