====== Embedded Gentoo ====== ---- dataentry project ---- Name_ : Embedded Gentoo Tags_tags : linux, gentoo Description_ : Attempt at creating an incredibly tiny Gentoo install, suitable for embedded devices. Last Update_dt : 2008-03-17 ---- ===== CHANGELOG ===== ==== 2008-03-17 ==== * **DONE** - Fixed //attempt to read past end of device// error. - Fixed reboot error. - Reduced size of disk image to 11M. * **TODO** - Add information about creating the qemu boot scripts. - Test SQLite. - Experiment with different space-conservation techniques. * //unification filesystems// * //squashfs/tmpfs storage of /var/www// * //ideas?// ==== 2008-03-12 ==== * **DONE** - Add information about copying data to the hard disk and converting to qcow2. - Add information about making the hard disk bootable. * **TODO** - Solve //attempt to read past end of device// error. * //possibly kernel-related? maybe hard disk too big and support needs to be enabled?// - Add information about creating the qemu boot scripts. - Test SQLite. - Experiment with different space-conservation techniques. * //unification filesystems// * //squashfs/tmpfs storage of /var/www// * //ideas?// ==== 2008-03-11 ==== * **DONE:** - Successfully booted the system and tested it using a QEMU emulator. Cherokee/PHP works. - Added some more documentation of the process. - Switched base partition to JFS format for spacial/journaling concerns. - Added SquashFS-LZMA support, still requires testing. - Added information regarding qemu-img virtual hard disk creation. **This page is still very much under development and should not be considered "complete" by any means. Use any advice or information provided on this site at your own risk.** ===== Introduction ===== The EmGen project is an effort at creating the smallest possible, working distribution of the [[http://gentoo.org|Gentoo Linux Distribution]]. This is being done using [[http://www.uclibc.org|uClibc]], and the current [[http://gentoo-wiki.com/Embedded_Gentoo|Gentoo Embedded]] project. EmGen will attempt to make a fully working system, and as the project grows we will see how much we can fit in a small location. The final product of this project currently will be to place an entire system, as well as a system emulator, on a 32MB SD Card. The finished product should be able to be inserted into a MMC/SC Card slot (such as one on a laptop) on a Windows system, and be used to run the Gentoo system. ===== Prerequisites ===== It has been determined that the following prerequisites are needed to build this project: ^ ^ Requirement ^ Notes ^^ ^ Hard Disk Space | >1.5G | You will need a lot of space to build your system. This includes compile/download space. || ^ Network | Active Internet connection | You are going to need to download a lot of packages, so the faster the better. || ^ System | Linux | Since you are creating a Gentoo Linux system, you will need a Linux build system. || ===== Environment Setup ===== The first thing we will do is set up a build environment which will be used to build our actual system. This build environment will have to be built once and from then on can be used to create various versions of the EmGen system. ==== Create the Base Directory ==== First, be sure to create a directory that will be the base for your build system. This system should fulfill the space requirements. export EMGEN_DIR=/emgen mkdir -p $EMGEN_DIR Throughout this document the **EMGEN_DIR** environment variable will be used in all scripts. Please make sure this is properly set to the base of your build system. ==== Install Stage 1 Embedded Gentoo ==== In order to create the build system, we must download a stage 1 tar-ball of the Embedded Gentoo system, and install it into the EmGen build environment. === Enter EmGen Build Directory === cd $EMGEN_DIR === Download Stage 1 Embedded Gentoo === Open the [[http://www.gentoo.org/main/en/mirrors2.xml|Gentoo Mirrors]] page in a browser of your choice, and select a mirror. Once logged on to the mirror, traverse the directory tree to **experimental / x86 / embedded / stages** and download the most recent **stage1** tar-ball. In the case of this project, I will be using version **2008.0**. Some mirrors listed do not maintain a copy of the **experimental** branch. If you are unable to find it on your chosen mirror, select another one. wget http://mirror.mcs.anl.gov/pub/gentoo/experimental/x86/embedded/stages/stage1-x86-uclibc-2008.0.tar.bz2 === Unpack the Stage 1 Tar-Ball === If you downloaded a **.tar.bz2**, use the following command to unpack the file: tar -xjpvf stage1-x86-uclibc-2008.0.tar.bz2 If you downloaded a **.tar.gz** or a **.tgz**, use the following command to unpack the file: tar -xzpvf stage1-x86-uclibc-2008.0.tar.gz ==== (Optional Step) Install Portage ==== If you are not using a Gentoo system as the host system for this project, you will need to download a snapshot of portage. If you are using Gentoo for your host system, then you can skip this step as you will be able to simply bind your current portage tree to the build system. === Enter EmGen Build Directory === cd $EMGEN_DIR === Download Portage Snapshot === Open the [[http://www.gentoo.org/main/en/mirrors2.xml|Gentoo Mirrors]] page in a browser of your choice, and select a mirror. Once logged on to the mirror, traverse the directory tree to **snapshots** and download the **portage-latest** tar-ball. wget http://mirror.mcs.anl.gov/pub/gentoo/snapshots/portage-latest.tar.bz2 === Unpack Portage Snapshot === If you downloaded a **.tar.bz2**, use the following command to unpack the file: tar -xjvf portage-latest.tar.bz2 -C $EMGEN_DIR/usr If you downloaded a **.tar.gz** or a **.tgz**, use the following command to unpack the file: tar -xzvf portage-latest.tar.gz -C $EMGEN_DIR/usr ===== Bootstrap Build System ===== In this stage, we will chroot into the newly created build system, and bootstrap it to enable us to use it for building the embedded target. ==== Chroot into the Build System ==== First we must set enter the Build System === Bind Directories === Bind all of the appropriate directories to the build system: mount -o bind /proc $EMGEN_DIR/proc mount -o bind /sys $EMGEN_DIR/sys If your host system is a Gentoo system, be sure to bind the portage tree to the build system as well: mkdir -p $EMGEN_DIR/usr/portage mount -o bind /usr/portage $EMGEN_DIR/usr/portage If you are binding your portage tree, and your distfiles directory is either on a separate partition or in a different location, be sure to bind it as well. mkdir -p $EMGEN_DIR/usr/portage/distfiles mount -o bind /usr/portage/distfiles $EMGEN_DIR/usr/portage/distfiles This is recommended since it would prevent you from re-downloading any packages that you already have on your host system. === Copy Resolv.conf === Next we copy the /etc/resolv.conf file so that we maintain internet connectivity. cp -L /etc/resolv.conf $EMGEN_DIR/etc/resolv.conf === Chroot === You should be ready to chroot now. chroot $EMGEN_DIR /bin/bash --login === Update Environment Variables === Now that you are within the chroot, you need to update the environment variables in your system. env-update source /etc/profile It is also a good idea to modify your prompt so that you don't accidentally type commands into your host system that were meant for your build system. export PS1="(emgen-build) $PS1" ==== Edit Configuration Files ==== Now that we are in the build system, we need to update some common configuration files to ensure that the build goes alright. === /etc/make.conf === The **/etc/make.conf** file is one of the most important files on any Gentoo system. It sets all of the environment variables to use when installing packages. Open the file. nano /etc/make.conf It will look something like this as a default: # These settings were set by the catalyst build script that automatically built # Please consult /etc/make.conf.example for a more detailed example CFLAGS="-Os -pipe" CHOST="i386-gentoo-linux-uclibc" CXXFLAGS="-Os -pipe" LDFLAGS="-Wl,-z,relro" For the sake of making things easy, I would recommend changing/adding the values to make the file resemble the following: # These settings were set by the catalyst build script that automatically built # Please consult /etc/make.conf.example for a more detailed example CFLAGS="-Os -pipe -march=i686 -fomit-frame-pointer" CXXFLAGS="${CFLAGS}" CHOST="i686-gentoo-linux-uclibc" LDFLAGS="-Wl,-z,relro" FEATURES="ccache buildpkg" USE="minimal" UCLIBC_CPU="686" PKGDIR="${PORTDIR}/packages/uclibc" Here is an explanation for the added/modified lines above: * **CFLAGS** - //-march=i686// is just a way of telling the GCC compiler to make generic optimizations for a i686-class processor (generally safe for any x86 architecture these days). - //-fomit-frame-pointer// tells GCC to not use a register for stack tracing, which we won't need as we will be leaving out any sort of debugging for the sake of space. * **CXXFLAGS** - It is generally good practice to set the **CXXFLAGS** equal to the **CFLAGS** unless you have good reason not to. * **CHOST** - I set this to //i686-gentoo-linux-uclibc// as I wish to build a generic i686-based system. You can change this but it should reflect all of the other values (such as //-march// and **UCLIBC_CPU**). * **LDFLAGS** - This I would just leave at a default. * **FEATURES** - //ccache// tells the system that we will be using the ccache system to cache our compiles to make recompiling faster. - //buildpkg// tells the system to use pre-built packages when possible. * **USE** - //minimal// enables all minimalistic installs in the system. * **UCLIBC_CPU** - //686// informs the uClibc library to make optimizations for the i686 architecture. If this is not set, uClibc will be very generic and also very bulky. * **PKGDIR** - This is used to tell portage to store packages for this build in a separate location from the default. Once you have finished making the modifications, hit Ctrl - X, Y, and Enter to exit nano and save the modifications. === /etc/make.profile === The **/etc/make.profile** file sets default for the **/etc/make.conf** file as well as some other settings. It is important to set it to the proper value for your build system. To view what it currently is, type: ls -ld /etc/make.profile This should be set to a generic x86 uClibc profile. To do so execute the following command: ln -fns ../usr/portage/profiles/uclibc/x86 /etc/make.profile It is generally a good idea to double check to make sure it worked: ls -ld /etc/make.profile ==== Perform the Bootstrap ==== The next step in the build process is to bootstrap the build system. This must be done to create a compiler toolkit based on embedded devices. === Enter the Portage Script Directory === This directory is where the //bootstrap.sh// file is located, which will be used to bootstrap the system. cd /usr/portage/scripts === Perform a Test Run === In order to see what is going to be bootstrapped, it is generally good to run a pretend run first. ./bootstrap.sh -p -v This command will locate all of the packages you need to have installed to bootstrap the system. Take a note of this as if anything goes wrong you will need to know where to continue. In my case, the following were displayed: First the system will update portage (the Gentoo emerge system), by installing/upgrading the following: [ebuild U ] sys-apps/sandbox-1.2.18.1-r2 [1.2.17] 0 kB [ebuild U ] sys-apps/portage-2.1.4.4 [2.1.1-r1] USE="build -doc -epydoc% (-selinux)" LINGUAS="-pl" 0 kB Next, the system will be bootstrapped by installing/upgrading the following: [ebuild N ] dev-util/unifdef-1.20 0 kB [ebuild U ] sys-kernel/linux-headers-2.6.23-r3 [2.4.26-r1] USE="(-gcc64%)" 0 kB [ebuild U ] sys-apps/texinfo-4.8-r5 [4.8-r2] USE="-build* (-nls) -static" 0 kB [ebuild U ] sys-devel/binutils-2.18-r1 [2.16.1-r3] USE="-multislot -multitarget (-nls) -test -vanilla" 0 kB [ebuild U ] sys-devel/gcc-4.1.2 [4.1.1] USE="bootstrap* (-altivec) -build* -d% -doc -fortran -gcj -gtk (-hardened) -ip28 -ip32r10k -libffi% -mudflap (-multilib) -multislot (-n32) (-n64) (-nls) -nocxx -objc -objc++ -objc-gc -test -vanilla" 0 kB [ebuild U ] sys-libs/uclibc-0.9.28.3 [0.9.28] USE="-build* -debug -hardened (-iconv) -ipv6 -minimal (-nls) -pregen -savedconfig -uclibc-compat -userlocales -wordexp" 0 kB [ebuild U ] sys-apps/sysvinit-2.86-r10 [2.86-r5] USE="(-ibm) (-selinux) -static" 0 kB [ebuild N ] virtual/init-0 0 kB [ebuild U ] sys-apps/baselayout-1.12.11.1 [1.12.6] USE="bootstrap* -build* -static -unicode" 0 kB [ebuild R ] sys-libs/zlib-1.2.3-r1 USE="(-build%*)" 0 kB === Start the Bootstrap === Once you feel confident (and have a lot of free time to fix possible errors), execute the following command to start the bootstrap process: ./bootstrap.sh Keep in mind that since you are building an experimental system from a stage-1 tar-ball, it is very likely that something WILL go wrong. If something does go wrong (and trust me it probably will), then you can resume the bootstrapping process by typing the above command again. It will continue on from wherever it left off. ==== Leave Chroot Environment ==== Now that we have bootstrapped the system, it is generally a good idea to leave the chroot and backup the directory so that you can set off from this point if something goes wrong in the future. === Exit Chroot === First just exit the chroot. exit === Unmount All Bound Directories === Now you should unbind all directories that were bound when chrooting. umount $EMGEN_DIR/proc umount $EMGEN_DIR/sys If you bound your portage distfiles directory, you must unbind it now. umount $EMGEN_DIR/usr/portage/distfiles If you bound your portage directory, you must unbind it now. umount $EMGEN_DIR/usr/portage ===== Reinstall the System ===== Immediately after bootstrapping the system, you must then reinstall all core system libraries using the newly bootstrapped tools. This is probably the most difficult part of the entire process, as you will more than likely run into some errors. The uClibc library is a slimmed down version of the glibc library. Most libraries are built expecting glibc to be present, so when it is not there is a high probability that the install will fail. This section requires a lot of patience and willingness to dig into the code base yourself to solve problems. ==== Chroot into the System ==== First, be sure to chroot into the system just as you did prior to bootstrapping (follow the section defined above). **MAKE SURE YOU ARE IN THE BUILD SYSTEM**. If you are not inside the chroot environment when you run the following commands it could potentially break or damage your host system. ==== Emerge the System ==== In order to make sure everything on the system will run under the newly bootstrapped toolkit, we need to reinstall all system libraries. In order to do this, run the following command: emerge -eav system The flags passed to the emerge system mean the following: * **-e** - (emptytree) assume the dependency tree is empty, so all dependencies of all files must be installed. This forces portage to install all system files again. * **-a** - (ask) show the user what will be installed prior to starting, and let them decide if portage should continue with the install. * **-v** - (verbose) be verbose in your description of what will be installed. Once you have viewed everything that is to be installed and made sure it is to your liking (you may wish to enable various use-flags and the like), hit Y or Enter and begin the emerge process. ==== Solutions to Things that May go Wrong ==== In a perfect world, the emerge would complete successfully and everything would work fine. Of course we are not in a perfect world, however, and these packages are very much still in the **experimental** stages. For that reason I have outlined in this section solutions to all of the problems that arose when I was attempting the install. You may experience more or less of these problems stated below. Because of the nature of Gentoo and the Portage Tree, packages are constantly being tweaked and updated, and as such there is a high likely-hood that your experience will change as time goes on. I have included version numbers in this HOWTO to assist in comparing your install against mine. === What to do if the Emerge Fails === If the emerge does fail (**and trust me, it probably will**), do not panic. Whatever you do, do not type in //emerge -eav system// again! This will compute with an empty tree again and force you to start all over instead of where you left off! In order to resume the emerge that you had before it failed, simply type: emerge -av --resume This will allow you to pick up where you left off. === What if you must Perform a Separate Emerge? === In the event that the only way to fix the problem is to emerge a package out-of-order. There are different ways to do this. Performing a second emerge typically runs the risk of wiping out your possibility to resume the initial one (this could be a problem). The best way to resolve this is to run the resume command, but like this: emerge -p --resume > remerge This will print out a list of packages that would be installed if portage were to resume, and save the list into a file called //remerge// in your current directory. Now you can emerge whatever you need to, remove those packages from the list in the //remerge// file, and then use that list to populate an emerge command to resume where you left off. === 'AI_ADDRCONFIG' undeclared === If a build fails with a line similar to: io.c:1150: error: 'AI_ADDRCONFIG' undeclared (first use in this function) Then the problem is more than likely due to an outdated version of //netdb.h//. To fix the problem, execute the following commands: cd /usr/include cp ./netdb.h ./netdb.h.orig nano ./netdb.h Now add the following after line 403 (hitting Ctrl-C in nano give you the current line number): # define AI_V4MAPPED 0x0008 /* IPv4 mapped addresses are acceptable. */ # define AI_ALL 0x0010 /* Return IPv4 mapped and IPv6 addresses. */ # define AI_ADDRCONFIG 0x0020 /* Use configuration of this host to choose returned address type. */ After adding those lines, hit Ctrl-X, Y, and Enter to save the file. Now continue the emerge: emerge -av --resume For more information regarding this bug, please visit [[http://bugs.gentoo.org/show_bug.cgi?id=195368|Bug #195368]]. It is important to note that this bug may occur more than once. Since you are effectively hand-modifying an include file, there is a good chance that your modification may be overwritten during the emerge process. If that is the case then you will need to re-modify the //netdb.h// file. === Python Library Could not Locate 'i386-gentoo-linux-uclibc-gcc' === If a compile of a python library fails with a line stating something to the effect of: gcc-config error: Could not run/locate "i386-gentoo-linux-uclibc-gcc" If this is the case, then your python install is more than likely hard-coded to use an old version of GCC (the one prior to bootstrapping). To fix this you have to change it's configuration. First run the following command: python-config The output should be something along the lines of: -lpython2.4 -lm -L/usr/lib/python2.4/config //cd// into the directory given that contains the configuration information: cd /usr/lib/python2.4/config Now run the following command to retrieve the current gcc profile: gcc-config -c You should see output similar to this: i686-gentoo-linux-uclibc-4.1.2 You will need to replace the missing gcc version with the version just displayed, minus the actual version number. Now run this command to replace all the old gcc CHOST values with the new ones: for i in `ls -1`; do cp ${i} ${i}.bak; sed -r 's/i386-gentoo-linux-uclibc/i686-gentoo-linux-uclibc/gi' ${i}.bak > ${i}; done; This essentially loops through all of the files in this directory and replaces the old value of //i386-gentoo-linux-uclibc// with the new value of //i686-gentoo-linux-uclibc//. Now resume the emerge: emerge -av --resume === Undefined Reference to '___tls_get_addr' in E2fsprogs === If your compile of //sys-fs/e2fsprogs// fails due to a message similar to the following: ../../lib/libuuid.so: undefined reference to '___tls_get_addr' The problem amounts from a faulty portage ebuild which makes sys-fs/e2fsprogs be configured assuming it will be built with glibc. uClibc does not contain TLS support, which is enabled in e2fsprogs by default. Luckily, however, the nice people at Gentoo have already been working on this and the solution is to simply unmask the more recent package that contains a fix (namely version 1.40.5-r1). To do this all you need to do is run the following command: echo "=sys-fs/e2fsprogs-1.40.5-r1 ~x86" >> /etc/portage/package.keywords echo "=sys-libs/ss-1.40.5 ~x86" >> /etc/portage/package.keywords echo "=sys-libs/com_err-1.40.5 ~x86" >> /etc/portage/package.keywords The only problem now is that when you attempt to resume the emerge, it will try to resume based on package version, and by that turn try to resume the old version of e2fsprogs. To solve this problem, you must resume using the following command: TMP=""; \ for i in `emerge --resume -pv | grep "\[ebuild" - | sed -r "s/\[ebuild[^]]*\] //gi" - | awk '{print $1}' - | sed -r "s/e2fsprogs-.*$/e2fsprogs-1\.40\.5-r1/gi" -`; \ do TMP="${TMP} =${i}"; \ done; \ emerge -av ${TMP} The above code is basically a very glorified way of automating parsing through the output of //emerge --resume -p//, removing everything but the package atoms, changing the version of e2fsprogs, compiling them all into a single line, and running //emerge// again. For more information regarding this bug, please visit [[http://bugs.gentoo.org/show_bug.cgi?id=204102|Bug #205102]]. === 'Can't locate ??? in @INC' Error === If your emerge fails with an error similar to the following: Can't locate auto/POSIX/assert.al in @INC (@INC contains: . /etc/perl /usr/lib/perl5/vendor_perl/5.8.8/i386-linux /usr/lib/perl5/vendor_perl/5.8.8 /usr/lib/perl5/vendor_perl /usr/lib/perl5/site_perl/5.8.8/i386-linux /usr/lib/perl5/site_perl/5.8.8 /usr/lib/perl5/site_perl /usr/lib/perl5/5.8.8/i386-linux /usr/lib/perl5/5.8.8 /usr/local/lib/site_perl .) at ./../mk-script line 52 Then the problem is most likely related to an inadequate install of Perl. There is probably a high likelyhood that the output for //emerge -p --resume// includes an upgrade for Perl, but that it does not occur for awhile so the current package won't work. The best way to get around this is to manually reorder the packages to be emerged, with all Perl packages coming next. To do this, simply run this command: TMP=""; \ for i in `emerge --resume -p | grep "\[ebuild" - | sed -r "s/\[ebuild[^]]*\] //gi" - | awk '{print $1}' - | grep "perl" -`; \ do TMP="${TMP} =${i}"; \ done; \ echo '#!/bin/bash' > ./install-perl.sh; \ echo "emerge -av ${TMP}" >> ./install-perl.sh; \ chmod 755 ./install-perl.sh; \ TMP=""; \ for i in `emerge --resume -p | grep "\[ebuild" - | sed -r "s/\[ebuild[^]]*\] //gi" - | awk '{print $1}' - | grep -v "perl" -`; \ do TMP="${TMP} =${i}"; \ done; \ echo '#!/bin/bash' > ./resume-nonperl.sh; \ echo "emerge -av ${TMP}" >> ./resume-nonperl.sh; \ chmod 755 ./resume-nonperl.sh This will create two new scripts in the current directory. //install-perl.sh//, which runs an emerge to install all Perl-related packages that were pending in the resume, and //resume-nonperl.sh//, which resumes installation where it left off without the Perl packages. They should also be executable. It is also generally a good idea to take a look at the files to make sure they were populated correctly: cat {install-,resume-non}perl.sh It should show something like this: #!/bin/bash emerge -av =dev-lang/perl-5.8.8-r4 =perl-core/PodParser-1.35 =perl-core/Test-Harness-2.64 #!/bin/bash emerge -av =sys-apps/coreutils-6.9-r1 =sys-apps/groff-1.19.2-r1 =sys-apps/shadow-4.0.18.2 =sys-process/psmisc-22.6 =sys-apps/man-1.6f =sys-apps/man-pages-2.76 =sys-apps/diffutils-2.8.7-r2 =dev-lang/python-2.4.4-r6 =dev-python/python-fchksum-1.7.1 =sys-apps/file-4.23 =sys-apps/module-init-tools-3.4 =sys-apps/debianutils-2.28.2 =sys-apps/mktemp-1.5 =sys-apps/baselayout-1.12.11.1 =sys-fs/udev-115-r1 Now you should run them in succession. First, execute //install-perl.sh// to (re-)install Perl: ./install-perl.sh Once this has completed (after resolving any errors or whatnot in-between), then run the resume script: ./resume-nonperl.sh If something does go wrong while any of these scripts are running, you may fix the issue and then resume from where you left off by using emerge -av --resume as usual. This is because the created scripts just run //emerge//, and therefore if the emerge fails, it can be resumed through regular means. Once you are done it is a good idea to remove the scripts from the filesystem: rm ./{install-,resume-non}perl.sh ==== Clean-up ==== After everything has completed successfully, you will first have to re-source your shell so that changes in path take effect: source /etc/profile export PS1="(emgen-build) $PS1" After that you will also want to update any configuration files currently held on your system: etc-update Since you haven't really set any configuration files, however, it should be save to just update all of the listed files automatically. So type in //-5// and hit Enter to do so. ==== Check for Broken Packages ==== During the //emerge -eav system// process, it is possible that package dependencies were broken. To check for that, you must use the //revdep-rebuild// utility. === Emerge app-portage/gentoolkit === The //revdep-rebuild// program is available from the //app-portage/gentoolkit// package. You will need to emerge that first: emerge -av gentoolkit === Run Revdep-Rebuild === Now that it is installed, you should run the following command to check for broken packages: revdep-rebuild -- -av The above command will search for any broken linkage between libraries on your system, determine which packages they belong to, and then attempt to emerge them, appending all flags after the //--// switch to the end of the //emerge// command. ==== Update All Packages on the Build System ==== Now that we have a working build system with all of its core libraries bootstrapped, it is time to perform any updates to packages not declared to be part of the core //system//. === Install CCache === To save time on compiling we will not install the CCache compiling cache utility. This will cache compiled objects to be used during recompiles or future updates. To install ccache, first you must emerge it: emerge -av ccache Now that CCache is installed, we need to set our cache size. Set it to something appropriate for your build environment. For us I will use a cache of 200M. ccache -M 200M === Upgrade World === Now that we have our caching mechanism setup, it is time to upgrade any and all packages on our build system that are outdated. To do this we perform a global //world// update: emerge -DuavN world Once this returns you will have an up-to-date build system, capable of creating your EmGen target system. ==== Install a Kernel Source Tree ==== The next step is to install a Linux kernel source tree. There are several different trees available, although for general purposes it is usually a good idea to use the tree associated with Gentoo, //gentoo-sources//. We will be using SquashFS 3.3 and will be adding LZMA support since it offers the best compression. For this reason we will also need to unmask some of the sources: echo "sys-kernel/gentoo-sources ~x86" >> /etc/portage/package.keywords emerge -av gentoo-sources This needs to be installed before we start installing some other features, since some packages build modules which will need a kernel source tree available. ==== Install GRUB ==== We will be using the Grand Unified Boot Loader (GRUB) as our boot manager (or at least a simplified version of it). In order to do this, we need to have it installed on the build system so that later it can be copied over to the target system. emerge -av grub ==== Install QEMU Toolkit ==== We will also be using QEMU to emulate our system. In order to build the hard drive and such, we will need some toolkits from QEMU (specifically qemu-img). The biggest problem is that QEMU requires some features not provided by uClibc or gcc-4. This means that we will not be able to build //qemu-img// on our build system. At this point you have two options: - Build //app-emulation/qemu-softmmu// on your host system - Download a static compilation: {{project:qemu-img.tar.bz2|.tar.bz2}} All that essentially matters is that you have access to a //qemu-img// binary. If you wish to go the easy route, just download the static version. It has been tested and should run within the //emgen-build// environment. If you are going to be using the static compilation, be sure to download the file and place it in a location within your path: tar -jxvf ./qemu-img.tar.bz2 cp -a qemu-img "${EMGEN_BUILD}"/usr/bin ==== Install JFS Toolkit ==== We will also be using the JFS filesystem, so we need to install utilities to create/manage those filesystems: emerge -av jfsutils At the time of writing this document, the most recent unmasked jfsutils package (//1.1.8//) would fail with posix compatibility errors (uClibc vs. glibc). Unmasking a more recent version (//1.1.12//) by the following command fixed the problem: echo "sys-fs/jfsutils ~x86" >> /etc/portage/package.keywords ==== Leave Chroot and Backup ==== Now that you have a build system that is fully functional, it is a good idea to once again leave the chroot environment and backup the system. Follow the same steps as outlined in the previous section on leaving the Chroot to exit the environment and unmount any bound directories. ===== Prepare the Target System ===== Once the build system is set up and ready to go, you need to prepare a target filesystem for your EmGen target. ==== Chroot into the Build System ==== As per the documentation above, once again chroot into the build system to perform all of the tasks in this section. ==== Create the Target System Directory ==== Now that we are in a chroot, we will create a directory in the chroot for our target system. export EMGEN_TARGET="/etarget" mkdir -p "${EMGEN_TARGET}" From here on out I will be referring to the target as ${EMGEN_TARGET}. If you don't want to have to worry about setting the ${EMGEN_TARGET} variable every time you Chroot into the build system, you can execute the following command to have it set every time you perform the Chroot: echo "EMGEN_TARGET=\"/etarget\"" >> /etc/env.d/99emgen ==== (Optional Step) Create the Target Emerge Script ==== To ease the management of packages on the target system, it is generally a good idea to create a script to handle all packages on the target system. If you decide not to create this program, be sure to treat subsequent commands that use it accordingly. First, open nano: nano Now create a file that looks like the following: #!/bin/bash ROOT="${EMGEN_TARGET}" \ INSTALL_MASK="*.h HACKING.gz INSTALL.gz README.gz TODO.gz" \ emerge -avkN \ $@ This file contains the following lines: * **ROOT="${EMGEN_TARGET}"** - sets the root for all merges (similar to a --prefix directive). * **INSTALL_MASK="*.h ..."** - masks all files matching the given patterns, preventing unneeded files from being merged. * **emerge -avkN** - set default //emerge// options to be verbose, ask for confirmation, use pre-built packages when possible, and check for new USE flags. * **$@** - append all given parameters to the end of the //emerge// command. Save the file as //etarget-merge// (or something similar, by typing Ctrl-X, Y, type in the name, and then type Enter. Now set execute permissions on the file, and move it into the path: chmod 755 ./etarget-merge mv ./etarget-merge /usr/bin/ ==== Install Required Packages ==== Now we need to install some required packages using our new script. The following packages **MUST** be installed, otherwise the target system won't be able to run. === sys-apps/baselayout-lite === Every system needs a baselayout. This package creates a base root filesystem structure for your target. We want to minimize size, however, so we will use the //baselayout-lite// package. This package is still in testing stages, however so we need to first unmask it: echo "sys-apps/baselayout-lite **" >> /etc/portage/package.keywords Now we can install the baselayout into our target environment: etarget-merge baselayout-lite === sys-libs/uclibc === Since this entire system is built on the uClibc library, it should go without saying that we need to include the library in our target system so that any packages that require a C library will have it available. uClibc should not be masked, so just install it: etarget-merge uclibc === sys-apps/busybox === We need to have a base set of utilities / shell / etc. as well, so we use //busybox// to provide that. Although not usually done, we will also enable the //make-symlinks// USE flag so that the install will make the symlinks that are usually done by other programs/installs. USE="make-symlinks" etarget-merge busybox === sys-fs/jfsutils === Since we are using a JFS filesystem, we will be installing the //jfsutils// package. etarget-merge jfsutils You **MUST** install the filesystem creation/checking utilities for whichever root filesystem you choose! In the event that a system gets shutdown improperly (before the root is remounted read-only), you will be unable to remount the root read/write until a filesystem check has been performed! ==== Install Optional Packages ==== In the next section, we will be installing the packages that we will actually be using in the EmGen target environment. You can install whatever you want into your target environment, but the more you install (and what you install) will cause your target root size to become larger. You should think about what you want your EmGen system to do (hopefully you did this before starting this project). For me, I will be going over how to create a basic (pseudo-)LAMP environment for web development. This way people could develop for their website regardless of what computer they are at. The requirements for this system is the following: * **L**inux - Well that's already taken care of. * **A**pache - We won't actually be using Apache because of its size, but instead a simplified server called Cherokee. * **M**ySQL - We won't be able to use MySQL since it is a bit bulky and I was unable to get the //embedded// server to install properly. Instead we will use SQLite3 as our SQL solution since it is lightweight and also because it uses single files, making the used disk space quite a bit easier to manage. * **P**HP - PHP is a staple of LAMP programming and it can't really be created without it in its entirety. === www-servers/cherokee === [[http://www.cherokee-project.com/|Cherokee]] is a lightweight and simple HTTP server. Although there are lots of [[wp>Tiny_web_servers|lightweight servers]], I chose Cherokee because it is lightweight but also supports FastCGI, which will come in useful in speeding up PHP (since lightweight servers only support PHP through the CGI interface). Cherokee is still masked by keyword at this time so first unmask it, then install it into the target: echo "www-servers/cherokee ~x86" >> /etc/portage/package.keywords etarget-merge cherokee This may install some dependencies, but it should be minimal. If anything make sure there are no USE flags set for the install as they are probably not needed anyway. After the install, portage will attempt to create users for the //cherokee// daemon. The only problem is that it will create them outside of the specified //ROOT//. It is important to create a user and group for the //cherokee// daemon within the //ROOT//. To do this, run the next commands: grep cherokee /etc/passwd >> "${EMGEN_TARGET}"/etc/passwd grep cherokee /etc/group >> "${EMGEN_TARGET}"/etc/group grep cherokee /etc/shadow >> "${EMGEN_TARGET}"/etc/shadow === dev-db/sqlite === SQLite is a simplified SQL Server that is managed by simply using flat-files and some C programs. Although it sounds rather slow and bulky, it is actually comparable to larger server-based programs such as MySQL or PostgreSQL. Since we need to conserve some space and the minimal install of MySQL only results in client-side programs (I attempted the //embedded// server but to no avail), we will be using SQLite as our backend. We don't actually have to explicitly install SQLite as it will be installed as a dependency when we install PHP. I actually recommend not installing SQLite separately, as I attempted to do so and it caused more than one version to be installed. PHP at the time of this writing only supports SQLite2, while SQLite3 is currently available. Therefore let PHP handle which version to install to avoid accidentally using up extra space. === dev-lang/php === Our last step is to install PHP on our target. This will by far be the most bulky package we will be installing, and quite a few USE flags that may be set. You should make sure you only set the ones you think you'll need for your environment, as every flag used will add more bulk to the install. I used the following USE flags: * **bzip2** - I like to have support for bzlib compression. * **cgi** - **THIS MUST BE ENABLED OR WE WON'T BE ABLE TO ATTACH PHP TO OUR HTTP SERVER** * **cli** - To allow PHP to be accessed via command-line. * **crypt** - Support for encryption. * **ctype** - cType functions can be useful. * **curl** - The cURL library is very useful when doing HTTP or other URL-related requests. * **force-cgi-redirect** - A common thing to enable in the CGI SAPI. * **gd** - Add internal graphics support. * **gmp** - Add support for the GNU Mupltiple Precision Library * **hash** - Enable the hash extension * **json** - Add support for JSON so we can talk with JavaScript easily. * **mhash** - Support the mhash library. * **ncurses** - Add ncurses support. * **pcre** - Add support for Perl Compatible Regular Expressions. * **readline** - Add support for GNU line-editing. * **reflection** - Enable the Reflection API. * **session** - Add support for sessions. * **simplexml** - Add support for SimpleXML. * **soap** - Add support for the Simple Object Access Protocol. * **sockets** - Sockets can be quite useful to have support for. * **spl** - Add support for the Standard PHP Library. * **sqlite** - Add support for SQLite * **tidy** - Add support for HTML Tidy. * **tokenizer** - Add support for the PHP file parser. * **unicode** - Support Unicode characters. * **xml** - Support XML files. * **xmlreader** - Enable XMLReader. * **xmlwriter** - Enable XMLWriter. * **zip** - Add ZIP compression support. * **zlib** - Add zlib compression support. Certain USE flags in PHP will install extensions that are incompatible with uClibc (as they expect glibc instead). The above USE flags have been tested and do work with uClibc, but if you enable other flags, be prepared for possible installation issues. To perform the install, simply run the following command: USE="bzip2 cgi cli crypt ctype curl force-cgi-redirect gd gmp hash \ json mhash ncurses pcre readline reflection session simplexml soap \ sockets spl sqlite tidy tokenizer unicode xml xmlreader xmlwriter \ zip zlib" etarget-merge php This will definitely install extra packages that your system will need. It is important to note that some packages may appear to be installed twice. This is because your system will have to install some packages to //emgen-build// so that future packages that depend on them can be compiled and placed on //emgen-target//. === net-misc/dhcpcd === We need to install a DHCP client so that we can connect our NIC to the virtual DHCP server, and thusly server our pages to the host system: etarget-merge dhcpcd ==== Compile the Kernel ==== Now that we have our packages setup, it is time to get to the nitty-gritty and compile our kernel that will be used to run all of this. First, //cd// into the kernel build directory: cd /usr/src/linux === Configure the Kernel === Since we are attempting to build a tiny kernel, it is a good idea to start off by removing all configuration options, we will set them on our own: make mrproper make allnoconfig This will create a default //.config// file that will have no options set. Now it is time to set those options. To open up the configuration menu, type the following command: make menuconfig This will open up a curses-based menu configuration program. Although configuring a kernel is typically somewhat difficult, we have a trick that will make it easier: qemu! Since we will be using //qemu// to run our system, we already know what hardware will appear to be present regardless of what system the emulator is run on. That being said, the configuration should be as follows for the kernel: General Setup ---> [*] Prompt for development and/or incompleted code/drivers [*] Support for paging of anonymous memory (swap) [*] System V IPC [*] BSD Process Accounting [*] BSD Process Accounting version 3 file format (14) Kernel log buffer size [*] Initial RAM filesystem and RAM disk support [*] Optimize for size (Look out for broken compilers!) [*] Configure standard kernel features (for small systems) ---> --- Configure standard kernel features (for small systems) [*] Enable 16-bit UID system calls [*] Support for hot-pluggable devices [*] Enable support for printk [*] Enable full-sized data structures for core [*] Enable futex support [*] Enable eventpoll support [*] Enable signalfd() system call [*] Enable eventfd() system call [*] Use full shmem filesystem [*] Enable VM event counters for /proc/vmstat Choose SLAB allocator (SLUB (Unqueued Allocator)) ---> [*] Enable loadable module support ---> [*] Module unloading [*] Automatic kernel module loading [*] Enable the block layer ---> --- Enable the block layer IO Schedulers ---> <*> CFQ I/O scheduler Default I/O scheduler (CFQ) ---> Processor type and features ---> Subarchitecture Type (PC-compatible) ---> Processor family (586/K5/5x86/6x86/6x86MX) ---> Preemption Model (Preemptible Kernel (Low-Latency Desktop)) ---> Hich Memory Support (off) ---> Memory split (3G/1G usr/kernel split) ---> Memory model (Flat Memory) ---> [*] Math emulation [*] MTRR support Timer frequency (250 HZ) ---> (0x100000) Physical address where the kernel is loaded (0x100000) Alignment value to which kernel should be aligned Bus options (PCI, PCMCIA, EISA, MCA, ISA) ---> [*] PCI support PCI access mode (Any) ---> [*] ISA support Executable file formats ---> [*] Kernel support for ELF binaries <*> Kernel support for a.out and ECOFF binaries <*> Kernel support for MISC binaries Networking ---> [*] Networking support Networking options ---> <*> Packet socket <*> Unix domain sockets [*] TCP/IP networking <*> IP: IPsec transport mode <*> IP: IPsec tunnel mode <*> IP: IPsec BEET mode <*> INET: socket monitoring interface Device Drivers ---> Generic Driver Options ---> (/sbin/hotplug) path to uevent helper [*] Block devices ---> --- Block devices Normal floppy disk support <*> Loopback device support <*> RAM disk support (16) Default number of RAM disks (4096) Default RAM disk size (kbytes) (1024) Default RAM disk block size (bytes) <*> ATA/ATAPI/MFM/RLL support ---> --- ATA/ATAPI/MFM/RLL support (4) Max IDE interfaces <*> Enhanced IDE/MFM/RLL disk/cdrom/tape/floppy support <*> Include IDE/ATA-2 DISK support <*> Include IDE/ATAPI CDROM support Include IDE/ATAPI FLOPPY support [*] legacy /proc/ide/ support <*> generic/default IDE chipset support <*> Generic PCI IDE Chipset support <*> Intel PIIXn chipsets support SCSI device support ---> <*> SCSI device support [*] legacy /proc/scsi/ support <*> SCSI disk support <*> SCSI CDROM support <*> SCSI generic support [*] Network device support ---> --- Network device support [*] Ethernet (10 or 100Mbit) ---> --- Ethernet (10 or 100Mbit) AMD LANCE and PCnet (AT1500 and NE2100) support [*] Other ISA cards NE2000/NE1000 support [*] EISA, VLB, PCI and on board controllers PCI NE2000 and clones support Input device support ---> -*- Generic input layer (needed for keyboard, mouse, ...) [*] Keyboards ---> --- Keyboards <*> AT keyboard Character devices ---> [*] Virtual terminal [*] Support for console on virtual terminal [*] Unix98 PTY support Graphics support ---> Console display driver support ---> [*] VGA text console [*] Video mode selection support File systems ---> <*> JFS filesystem support <*> ROM file system support [*] Dnotify support <*> Kernel automounter version 4 support Pseudo filesystems ---> [*] /proc file system support [*] Sysctl support (/proc/sys) [*] sysfs file system support [*] Virtual memory file system support (former shm fs) Miscellaneous filesystems ---> <*> Compressed ROM file system support (cramfs) Kernel hacking ---> [*] Enable doublefault exception handler Library routines ---> {*} CRC32 functions === Compile the Kernel Image === Now that you have your kernel configured, make the kernel with the following function: make -j2 === Install the Modules === Now you need to install the modules for the kernel. This is done with the following command: make modules_install ==== Install SquashFS + LZMA Module ==== If you noticed, we did not enable the SquashFS module/builtin capabilities. This is because we will be patching the driver to enable LZMA compression versus GZip compression. LZMA is much more effective. In this section we will be downloading all of the patches, source, etc., needed to compile the SquashFS+LZMA module. === Create a Build Directory === First we will be needing to create a build directory for our module: mkdir -p /tmp/sqfs-lzma cd /tmp/sqfs-lzma === Download the Sources === Next we need to download all of the sources for our build: wget http://www.squashfs-lzma.org/dl/squashfs3.3.tar.gz wget http://www.squashfs-lzma.org/dl/squashfs-cvsfix.patch wget http://superb-east.dl.sourceforge.net/sourceforge/sevenzip/lzma449.tar.bz2 wget http://www.squashfs-lzma.org/dl/sqlzma3.3-457.tar.bz2 I realize that I am using LZMA-449 even though the patchset says it is for -457, but it would appear that the provided patchset online still only supports -449. === Unpack the Sources === After downloading the sources, we need to unpack everything. First, we will unpack the //sqlzma// patches: mkdir -p sqlzma3.3-457 tar -jxvf sqlzma3.3-457.tar.bz2 -C sqlzma3.3-457 Next, unpack the //lzma449// files into a subdirectory of our lzma patchset: mkdir -p sqlzma3.3-457/lzma449 tar -jxvf lzma449.tar.bz2 -C sqlzma3.3-457/lzma449 Lastly, unpack the //squashfs3.3// files: tar -zxvf squashfs3.3.tar.gz -C sqlzma3.3-457 Once everything is unpacked, be sure to enter the sqlzma directory for making: cd sqlzma3.3-457 We will also be needing to patch the linux kernel to build the squashfs module. To provide a method for this, create a link, //linux//, in the //sqlzma// directory. We will also make a versioned link to make it easier to patch: ln -s /usr/src/linux linux ln -s linux linux-2.6.24 === Apply CVS Patches to SquashFS === First we need to apply the sqlzma patchset to the LZMA tree: patch -p0 < ./sqlzma1-449.patch The next step is to apply the CVS patchset to the SquashFS source tree: patch -p0 < ../squashfs-cvsfix.patch After that we need to apply the kernel patches from the SquashFS tree, as well as the //sqlzma// patshet. patch -p0 < squashfs3.3/kernel-patches/linux-2.6.24/squashfs3.3-patch patch -p0 < sqlzma2k-3.3.patch Lastly, we need to apply the user tools patch: patch -p0 < sqlzma2u-3.3.patch There is an error in the Makefile that needs to be corrected regarding the directory structure and the kernel version, to solve that, use this command (assuming the flavor of Gentoo-Sources-2.6.24 is 2.6.24-gentoo-r3: cp Makefile Makefile.old; \ sed -r 's/\$\(shell uname -r\)/2.6.24-gentoo-r3/g' Makefile.old | \ sed -r 's/^SqFs =.*$/SqFs = ${Sqlzma}\/linux\/fs\/squashfs/g' - > Makefile Sadly, we will also need to fix an error in SquashFS-3.3 regarding the use of get_nprocs(). Thanks to the wonderful people in the Gentoo Developer Community, however, we don't have to worry about how. This problem is already included as a patch in SquashFS-3.3! We can't apply this patch however since we have already applied patches to the required files. The bonus is that they do tell us how to fix the problem. To fix the problem, we need to open //mksquashfs.c//, and change the line ourselves: nano ./squashfs3.3/squashfs-tools/mksquashfs.c Then we need to go to line ~2847 (use Ctrl-C to see the current line), and change the value from: processors = get_nprocs(); to processors = sysconf(_SC_NPROCESSORS_CONF); Then save the file (Ctrl-X, Y, Enter, and that should be it! === Make the Modules === Now that we have everything we need patched, it is time to build the modules. make === Install to Build Environment === Now we need to copy the modules over to the build environment so that they are loadable: mkdir /lib/modules/2.6.24-gentoo-r3/misc cp linux/fs/squashfs/squashfs.ko /lib/modules/2.6.24-gentoo-r3/misc/ cp lzma449/C/Compress/Lzma/kmod/sqlzma.ko /lib/modules/2.6.24-gentoo-r3/misc/ cp lzma449/C/Compress/Lzma/kmod/unlzma.ko /lib/modules/2.6.24-gentoo-r3/misc/ We will need to add the modules to the modules.dep list as well to make sure they get used properly: echo "/lib/modules/2.6.24-gentoo-r3/misc/squashfs.ko: /lib/modules/2.6.24-gentoo-r3/misc/sqlzma.ko /lib/modules/2.6.24-gentoo-r3/misc/unlzma.ko" >> /lib/modules/2.6.24-gentoo-r3/modules.dep echo "/lib/modules/2.6.24-gentoo-r3/misc/sqlzma.ko:" >> /lib/modules/2.6.24-gentoo-r3/modules.dep echo "/lib/modules/2.6.24-gentoo-r3/misc/unlzma.ko:" >> /lib/modules/2.6.24-gentoo-r3/modules.dep We also need to copy over a few libraries so that they may be used: cp lzma449/C/Compress/Lzma/libunlzma.a /usr/lib/ cp lzma449/C/Compress/Lzma/libunlzma_r.a /usr/lib/ And lastly we need to copy over the utility binaries: cp squashfs3.3/squashfs-tools/mksquashfs /usr/bin/ cp squashfs3.3/squashfs-tools/unsquashfs /usr/bin/ And now we're all done! ==== Install Modules to the Target ==== Now that we have all of the modules we need installed on our build system, we need to transfer them over to the target system. To do this, we enter the following commands: rm -r "${EMGEN_TARGET}"/lib/modules mkdir -p "${EMGEN_TARGET}"/lib/modules cp -a /lib/modules/* "${EMGEN_TARGET}"/lib/modules/ ==== Leave the Chroot Environment ==== You now also have a target system to be installed on the device of your choice! It is once again a good idea to leave the Chroot environment and save a backup of your work thus far. (Trust me, you'll regret it if you make a mistake in the future!). To leave the Chroot completely, follow the section regarding the topic in the above statements. ===== Modify Target ===== Now that we have our target machine setup pretty much exactly the way we want it, we need to make a few //extra// changes to it to ensure that it will boot and load properly. Remember that we did not install any form of text editor into the target system, so all modifications will need to be made outside of the system. After a modification has been made, it can be tested from within the system itself by chrooting into it. All commands from here on out assume that you are within the directory of your //emgen-target// system and that an environment variable, **$EMGEN_TARGET** is set to that directory. **If you do not set this and run the commands as below you will damage your system!** To set this value, run the following command: export EMGEN_TARGET="/emgen/etarget" ==== Create Init Scripts ==== The installs for our system do not include any init scripts for cherokee or mysql. In order to make sure they start at boot time, we need these scripts to be created. BusyBox (which we installed for our target), provides a mechanism by which all scripts in ///etc/init.d// are started at boot and killed at system halt. For that reason we will create two scripts to do just that. === /etc/init.d/mount === Because of how our system is going to be created, we will need to remount the root filesystem to make it writable. rm -f ${EMGEN_TARGET}/etc/init.d/mount nano ${EMGEN_TARGET}/etc/init.d/mount Type (or copy) the following into the file: #!/bin/sh case $1 in start) # perform a filesystem check on the root device fsck.jfs -fa /dev/root # make the root read/write echo "Remounting Root Filesystem Read/Write..." mount -oremount,rw,noatime -n /dev/root / # mount our squashed filesystem to the root echo "Loading SquashFS-LZMA Module..." modprobe squashfs echo "Mounting /usr squashed filesystem..." mount -t squashfs -oloop /usr.sfs /usr echo "Mounting /etc/cherokee squashed filesystem..." mount -t squashfs -oloop /etc/cherokee.sfs /etc/cherokee ;; stop) # umount our squashed filesystem echo "Unmounting /etc/cherokee squashed filesystem..." umount /etc/cherokee echo "Unmounting /usr squashed filesystem..." umount /usr echo "Unloading SquashFS-LZMA Module..." modprobe -r squashfs # remount root read-only echo "Remounting Root Filesystem Read-only..." mount -oremount,ro -n /dev/root / ;; esac Don't worry about the user filesystem being mounted stuff. The creation of that filesystem will be handled later and it is going to be done that way for spacial reasons. For now just type in script. Now save the file (Ctrl-X, Y, Enter), then make it executable: chmod +x ${EMGEN_TARGET}/etc/init.d/mount === /etc/init.d/net === We will also need a network to be present, so let's start that up here: rm -f ${EMGEN_TARGET}/etc/init.d/net nano ${EMGEN_TARGET}/etc/init.d/net Type (or copy) the following into the file: #!/bin/sh case $1 in start) echo "Loading Network Modules..." modprobe ne2k-pci echo "Bringing Up Eth0 Network Interface..." ifconfig eth0 up dhcpcd -L eth0 ;; stop) echo "Shutting Down Eth0 Network Interface..." dhcpcd -k eth0 ifconfig eth0 down echo "Unloading Network Modules..." modprobe -r ne2k-pci ;; esac You will notice that I added some module autoloading. This is because I installed some of the NIC drivers as modules. If you compiled them in, you will not need to have this be a part of the script. Also, the modules I am loading are for the default NIC used by QEMU. Now save the file (Ctrl-X, Y, Enter), then make it executable: chmod +x ${EMGEN_TARGET}/etc/init.d/net === /etc/init.d/cherokee === We will need to create a script to boot our //cherokee// server: rm -f ${EMGEN_TARGET}/etc/init.d/cherokee nano ${EMGEN_TARGET}/etc/init.d/cherokee Type (or copy) the following into the file: #!/bin/sh PIDFILE=/var/run/cherokee.pid case $1 in start) echo "Starting Cherokee HTTPD Server..." start-stop-daemon --quiet --start --pidfile ${PIDFILE} \ --exec /usr/sbin/cherokee -- -b & ;; stop) echo "Stopping Cherokee HTTPD Server..." start-stop-daemon --quiet --stop --pidfile ${PIDFILE} & ;; esac Now save the file (Ctrl-X, Y, Enter), then make it executable: chmod +x ${EMGEN_TARGET}/etc/init.d/cherokee === Create /var/run === Since we will be storing our //cherokee// PID file under ///var/run//, we need to make sure it exists: mkdir -p ${EMGEN_TARGET}/var/run === Create Script Links === Last but not least we need to create some symbolic links so that the system will automatically start/stop the scripts we created in the right order on startup nad shutdown. ln -s mount ${EMGEN_TARGET}/etc/init.d/S01 ln -s net ${EMGEN_TARGET}/etc/init.d/S02 ln -s cherokee ${EMGEN_TARGET}/etc/init.d/S03 ln -s cherokee ${EMGEN_TARGET}/etc/init.d/K01 ln -s net ${EMGEN_TARGET}/etc/init.d/K02 ln -s mount ${EMGEN_TARGET}/etc/init.d/K03 The init system for BusyBox will execute all scripts in ///etc/init.d//, in numerical order, with S?? being the startup scripts and K?? being the kill scripts. ==== Test Binaries ==== Now it is generally a good idea to test your binaries that will be run to ensure that no dependencies have been left out in the fray (because they were installed to the build system but not the target system). In order to perform the tests, you will need to chroot into the target system: chroot $EMGEN_TARGET /bin/ash --login Now the files you will want to test are: /usr/sbin/cherokee /usr/lib/php5/bin/php /usr/lib/php5/bin/php-cgi /usr/sbin/dhcpcd /sbin/fsck.jfs (or similar) Run each of these programs. If one of them fails to run because of an error such as: ./php: can't load library 'libpcre.so.0' Then you will have to re-install each library that it states using the build environment. To re-install a package from the build environment, first you must make sure you are chrooted into the //emgen-build// environment. Next, run the following command to determine which package contains the library you need (for example, with libpcre.so.0): equery belongs libpcre.so.0 This should yield where the library is and to which package it belongs: [ Searching for file(s) libpcre.so.0 in *... ] dev-libs/libpcre-7.6-r1 (/usr/lib/libpcre.so.0 -> libpcre.so.0.0.1) At this point you can either: - Copy the library itself to the system (recommended for space) - Install the package to the system (recommended for stability) I recommend having two terminal windows open, one with you chrooted into the //emgen-target// environment, and one of you chrooted into the //emgen-build// environment. This way you can continue to test on //emgen-target// and whenever something goes wrong, you can change terminals to //emgen-build// to re-install a library to the target. === Option 1: Copy the library === It is recommended to copy the library to the system since it will conserve a lot of space and prevent potentially unneeded dependencies from being installed on the target. To copy the libraries to the system: cp -a /usr/lib/libpcre.so.0 "${EMGEN_TARGET}"/usr/lib cp -a /usr/lib/libpcre.so.0.0.1 "${EMGEN_TARGET}"/usr/lib Notice how I copied over the symbolic link and the actual library. You must copy with the //-a// flag to preserve users and other values, but it means that a symbolic link will be copied as a symbolic link and not as the library it linked to, so the library itself must be copied over seperately. === Option 2: Install the Package === This option is not recommended for space since a lot of extra (potentially unnecessary packages) will probably be installed as dependencies. To install the package, simply: etarget-merge libpcre ==== Configure Cherokee ==== Next you need to configure //cherokee// to run PHP scripts. To do this, first open the //cherokee// configuration file: nano "${EMGEN_TARGET}"/etc/cherokee/sites-enabled/default Now scroll to the bottom of the default file where it has something to the effect of: Extension php, php3, php4 { Handler phpcgi } Change it to this: Extension php, php3, php4, php5 { Handler phpcgi { Interpreter "/usr/lib/php5/bin/php-cgi" } } Now save (Ctrl-X, Y, Enter). Your web server should now support PHP! I attempted to get PHP's built-in FastCGI server working however whenever a page would be loaded the system would return a "500 Internal Server Error" error message and //cherokee// would complain about not being able to connect to the transport. ==== Set Target Hostname ==== Just for good measure, it is generally a good idea to set the target's hostname, for networking purposes: echo "emgen" >> "${EMGEN_TARGET}"/etc/hostname ==== Create Root Password ==== In order to be able to log in once you boot the target, you need to have the root password set. To do this, first chroot into the target: chroot "${EMGEN_TARGET}" /bin/ash --login Now that you are logged in to the system, create a root password: passwd Then exit the chroot: exit ==== Create Loop Links ==== The system also needs a few links for the loop devices to ensure that they function properly with some of the losetup/mount operations: mkdir -p "{$EMGEN_TARGET}"/dev/loop for i in "{$EMGEN_TARGET}"/dev/loop?*; do ln -s ../`basename $i` "${EMGEN_TARGET}"/dev/loop/`echo $i | sed -r 's/^.*loop([0-9]+).*$/\1/gi' - `; done This should make symbolic links for each of the loopback devices, which may be useful in the future. ==== Modify /etc/fstab ==== We need to modify the //emgen-target//'s Filesystem Tab file (///etc/fstab//) in order to make sure that some filesystems are mounted automatically at boottime. Open the file as such: nano "${EMGEN_TARGET}"/etc/fstab And edit it to look something like the following: # default ram disk as root /dev/root / jfs noatime 1 2 none /proc proc defaults 0 0 # glibc 2.2 and above expects tmpfs to be mounted at /dev/shm for # POSIX shared memory (shm_open, shm_unlink). # (tmpfs is a dynamically expandable/shrinkable ramdisk, and will # use almost no memory if not populated with files) # Adding the following line to /etc/fstab should take care of this: none /dev/shm tmpfs defaults 0 0 I set the root partition to /dev/root (which will be /dev/hda1) since we will be storing our system on an emulated hard drive image. We also set it to an //JFS// filesystem, which is semi-arbitrary. You can set the filesystem type to whatever you wish provided support for that filesystem has been built into your kernel, and you remember use that filesystem from here on out instead of //JFS//. //JFS// was chosen because it take up very little of the hard drive space for file accounting (for our size), and also supported journaling.\\ \\ We also uncomment ///proc// and ///dev/shm// since ///proc// is useful for monitoring system information and ///dev/shm// is a RAM filesystem and takes virtually no space unless needed (no real downside to not use it). Then save the file (Ctrl-X, Y, Enter). ==== Create a Symbolic Link for Root ==== Now for BusyBox compatibility, we need to create a symbolic link for the root device: ln -s hda1 "${EMGEN_TARGET}"/dev/root ===== Setup Temporary Creation Environment ===== Now that we have everything that we need to setup the hard drive, we need to create a temporary creation environment. ==== Chroot to the Build Environment ==== Follow the aforementioned instructions to do this (once again). ==== Create the Creation Directory ==== We will be building a copy of our //emgen-target// system so that we can modify it at will without having to worry about making mistakes. All of our work will be done under the directory that we create for the build. To reset this directory, execute the following commands: rm -fr /tmp/ecreate mkdir -p /tmp/ecreate Also move into the directory you just created. cd /tmp/ecreate ==== Copy Over the Target ==== Now that we have a temporary environment, we need to copy over everything we've created in the target. We do an archival copy to ensure that we save all of our permissions and such. cp -a "${EMGEN_TARGET}" ./etarget ===== Finalize Installation on the Target ===== This section remains within the build environment //chroot//. ==== Install the Boot Loader ==== We now need to install our boot loader, GRUB. This will allow our hard drive to boot the kernel directly. === Copy the Boot Directory === First step is to copy our build environment's ///boot// directory to our target system (don't worry about sizes, we will clean it up manually): cp -a /boot ./etarget/ === Clean-up GRUB === Now it is time to get a little messy and remove some of the pieces of GRUB that are not needed. First, move into the GRUB directory: cd ./etarget/boot/grub Now we don't need most of these files, so first save the //stage1// and //stage2// installs: mv stage1 .. mv stage2 .. Now depending on which filesystem you are using for your root directory, your //stage1_5// is going to be different. The directory generally contains stages of the form "${FS_TYPE}_stage1_5". Since we are using JFS, we need to save //jfs_stage1_5//: mv jfs_stage1_5 .. Now clean out the directory: rm * Then reinstate our saved files: mv ../stage1 ./ mv ../stage2 ./ mv ../*_stage1_5 ./ Now we need to create the //grub.conf//, //menu.lst//, and //device.map// files. To start with, just create them, then we will edit them: touch ./grub.conf touch ./device.map ln -s ./grub.conf ./menu.lst Easiest is to populate is the //device.map// file, so let's do that: echo "(hd0) /dev/hda" > ./device.map Now we need to create our configuration file: nano ./grub.conf Fill the file with something similar to this: timeout 10 default 0 fallback 0 title EmGen root (hd0,0) kernel /boot/vmlinuz root=/dev/hda1 Then save the file (Ctrl-X, Y, Enter). Be sure to remember what you name the kernel (in our case //vmlinuz//), because you have to install the kernel to that location! === Leave the GRUB Configuration Directory === Now that we are done configuring GRUB, we need to return to our original directory: cd ../../../ ==== Install the Kernel ==== Our system would be nothing without the Kernel we created, so let's install it: cp /usr/src/linux/arch/i386/boot/bzImage ./etarget/boot/vmlinuz Remember to rename your kernel to what you called it in your GRUB configuration! (in our case we renamed it to //vmlinuz//) ==== Squash the Filesystems ==== This is the step where we shrink down our system. Using SquashFS, we can reduce some of our file tree. I recommend compressing any directories (of substance) that require read-only access. === /usr === First off, we will clean any unneeded files from the ///usr// filesystem. After scanning through most of the directory, the safest thing to remove from the directory is the //include// directory, since we won't be performing any compiles on the target system: rm -fr ./etarget/usr/include Another step that can save some space on your system, is to move all of the library files not needed until later on in the boot process to the ///usr// directory, so that they will be squashed. For example, in order to boot your system, you should only need the libraries needed to run the files in the ///bin// and ///sbin// directories, as well as any corresponding libraries for inter-library links. Therefore, any other extra libraries can be thrown into ///usr/lib//. In order to perform this, we need to check each runtime to get a list of libraries that need to be saved: for i in ./etarget/{bin,sbin}/*; do if [ ! -L $i ]; then ldd $i | awk '{print $3}' - >> libs; fi; done for i in `cat libs`; do if [ -e $i ]; then echo "$i" >> libs_2; fi; done touch libs_3 for i in `cat libs_2`; do if [ -z "`grep $i libs_3`" ]; then echo "$i" >> libs_3; fi; done These commands first list all libraries, then get rid of any excess garbage, then removes duplicates. Now that we have our list of libraries we need to add to that list any libraries that are required by them: for i in `cat libs_3`; do ldd $i | awk '{print $3}' - >> libs_4; done for i in `cat libs_4`; do if [ -e $i ]; then echo "$i" >> libs_5; fi; done cp libs_3 libs_6 for i in `cat libs_5`; do if [ -z "`grep $i libs_6`" ]; then echo "$i" >> libs_6; fi; done Now at this point, the file //libs_6// contains all of the libraries that we need for our root filesystem to function. Anything else can be placed into ///usr/lib//. There is really no easy way that I know of to do this, except to compare the library list and the files in ///lib//. It is assumed at this point that you will move these files yourself. Please remember to move all of the libraries as well as their symbolic links, or else things won't work properly! It is generally a good idea after performing the copy-overs to double-check the ///lib// and ///usr/lib// for symbolic link breaks caused by the copies. Now we can create our SquashFS file. To conserve the greatest amount of space, we will also be changing the default block size to the largest amount supported, 1024k. mksquashfs ./etarget/usr/* ./etarget/usr.sfs -b 1048576 Now we have no more need for the ///usr// directory, so empty it: rm -fr ./etarget/usr/* === /etc/cherokee === The //cherokee// configuration directory does not need to be read-write, since it will only be read by the server, and not written to: mksquashfs ./etarget/etc/cherokee/* ./etarget/etc/cherokee.sfs -b 1048576 rm -fr ./etarget/etc/cherokee/* ==== Cleanup Unneeded Files ==== Since we used Gentoo to create our target system, there are some unneeded files left over from the process. It is now time to remove them in order to free up some space: rm ./etarget/etc/*- rm -fr ./etarget/etc/portage rm -fr ./etarget/var/cache/edb rm -fr ./etarget/var/db/pkg rm -fr ./etarget/var/lib/portage Here is an explanation of what each directory is and why we don't need them: * **/etc/*-** - Sometimes backup files are saved with a dash (-) afterwards. * **/etc/portage** - A list of keywords/masks/etc for portage. * **/var/cache/edb** - A cache of various emerge data. * **/var/db/pkg** - This stores information about portage installs. * **/var/lib/portage** - Another location of information regarding portage installs. ===== Install to a Virtual Hard Drive ===== This section remains within the build environment's //chroot//. ==== Create the Virtual Hard Drive ==== Now we need to create a virtual hard drive to store our system on. In order to make sure that everything will fit on the drive, you should first check how much space your system takes up: du -shc etarget Now take the total amount into consideration when determining how big you should make your disk image. Since my goal is to fit everything plus emulation software onto a 32M flash card, I will calculate my size like this: * After the card is formatted using vfat it only holds about 30M, so assume I have 29M of space for that. * The QEMU windows binaries take up about 2-3M, so assume 4M and our total size drops to 26M. * Give some room for buffer space and extra files, and we will make our image 24M. Since the system I created this far is only 11M, a 24M hard disk is plenty big. Now make the image: qemu-img create -f raw hda.img 24M Next we need to create a partition table on the hard drive, so first create a loopback device for the hard drive image, and then execute //fdisk// on it: losetup /dev/loop0 ./hda.img fdisk /dev/loop0 Create a new partition (filling up all of the space), and then exit, writing the data to disk. To do this, simply type the following keys in sequence: n, Enter, p, Enter, 1, Enter, Enter, Enter, w, Enter. Now that you have the partition table created, remove the loop device: losetup -d /dev/loop0 You know have a simulated hard disk with one empty partition. ==== Format the Target Partition ==== Now we need to format the target partition to our filesystem type (we chose //JFS//). To do this, we first need a loopback device setup that starts at the partition. To find the starting point of that partition, execute the following command: sfdisk -l -uS ./hda.img This should yield something like: Disk hda.img: cannot get geometry Disk hda.img: 3 cylinders, 255 heads, 63 sectors/track Units = sectors of 512 bytes, counting from 0 Device Boot Start End #sectors Id System hda.img1 63 48194 48132 83 Linux hda.img2 0 - 0 0 Empty hda.img3 0 - 0 0 Empty hda.img4 0 - 0 0 Empty We can see that according to //sfdisk//, our virtual hard drive is made up of sectors of 512 bytes each and that our primary partition starts at sector 63. Using some basic math, we can deduce that the primary partition of our virtual hard drive starts at byte 512 * 63 = 32256. Now we just create a loopback device with that offset: losetup -o 32256 /dev/loop0 ./hda.img To make sure that we format the right number of blocks, we need to check how many there are on our partition, so perform the following command: sfdisk -l ./hda.img This should yield something like: Disk ./hda.img: cannot get geometry Disk ./hda.img: 3 cylinders, 255 heads, 63 sectors/track Units = cylinders of 8225280 bytes, blocks of 1024 bytes, counting from 0 Device Boot Start End #cyls #blocks Id System ./hda.img1 0+ 2 3- 24066 83 Linux ./hda.img2 0 - 0 0 0 Empty ./hda.img3 0 - 0 0 0 Empty ./hda.img4 0 - 0 0 0 Empty We can see that there are //24066// blocks. It is generally a good idea to make your block count be a multiple of 4, so you should set your block count to ( //total_blocks// - (//total_blocks// mod 4) ) = ( 24066 - 2 ) = 24064. The problem is that //qemu-img// uses a block size of 1024 bytes, while //mkfs_jfs// uses a default block size of 4096 (which is 1024*4). So to figure out how many JFS blocks to install on the partition, we divide our number by 4, and get 6016. Then format the partition, using the specified block size: mkfs.jfs /dev/loop0 6016 When asked whether to continue, hit Y, Enter. Then once the formatting is completed, delete the loopback device: losetup -d /dev/loop0 ==== Load the Partition ==== Now we have a formatted, partitioned virtual hard drive, and the next step is to install all of our system files onto it. === Mount the Partition === In order to load the files onto the partition, we will first need to mount it. Before that can be done we need a place to mount it to, so create the mount point: mkdir -p ./mnt Now you can actually mount the partition. For this to work you will also need to use the offset from before, but you can skip the creation of a loopback device by letting the //mount// command handle it. mount -o loop,offset=32256 ./hda.img ./mnt If your mount fails it may be due to an incompatible partition type. We are using IBM's JFS filesystem, which is not nearly as common as Ext2 or Ext3. Your host system's kernel will need to have support for JFS for this mount to work (either built-in or through a module). If the support is not build-in, you can check if there is a module by running the command: modprobe -l | grep jfs - On your host system. If it comes up with a module such as //jfs.ko//, then load the module using modprobe: modprobe jfs === Copy Over the Files === Now that we have our mounted partition, it is time to copy over all of our hard work. This can be done simply by doing this: cp -a ./etarget/* ./mnt/ === Unmount the Partition === Now we no longer need to access the virtual hard drive, so just unmount it: umount ./mnt ==== Convert the Hard Drive Image ==== To conserve some extra space, it is generally a good idea to convert the virtual hard drive image from RAW format (which it is in now) to QCOW2 format (a QEMU expandable format). The benefit is that QCOW2 will only take up as much space as it needs to, but will dynamically expand to the maximum. qemu-img convert -f raw hda.img -O qcow2 hda2.img rm hda.img mv hda2.img hda.img ==== Leave the Chroot ==== Follow the aforementioned instructions to do this (once again). ===== Install GRUB to Virtual Hard Drive MBR ===== Although our hard drive does have GRUB installed, it is not installed to the MBR so therefore it will not work. To get it to work, we need to do some extra work since we now have a QCOW2 image. ==== Create a Bootable Floppy ==== First off, we need a bootable floppy so that we can boot into a grub command line and then run the install from there. To do this, run the following commands: dd if="${EMGEN_BUILD}"/boot/grub/stage1 of="${EMGEN_BUILD}"/tmp/ecreate/fda.img bs=512 count=1 dd if="${EMGEN_BUILD}"/boot/grub/stage2 of="${EMGEN_BUILD}"/tmp/ecreate/fda.img bs=512 seek=1 ==== Run QEMU to Install GRUB ==== Now that we have our bootable floppy, we need to run QEMU to install the GRUB to the MBR. This requires QEMU to be installed. If your host system is a Gentoo system, you can install QEMU by running: emerge -av qemu If you are not running a Gentoo system, you will need to check your system's information regarding how to install a copy of QEMU. If you are running this howto as //root// but have //sudo//'d into it, the following command **MAY LOCK UP YOUR COMPUTER**! Be forewarned that in my experience when I was handling the install as //root//, QEMU would lock the computer when trying to access my //DISPLAY//. To get around this, copy the files to your main user's home directory and run qemu as that use (that user will need to belong to the //qemu// group): mkdir -p ~/emgen cp "${EMGEN_BUILD}"/tmp/ecreate/*.img ~/emgen/ chown myuser:mygrp -R ~/emgen Then when you are completed with the GRUB MBR installation, run this command to move the hard drive image back: rm "${EMGEN_BUILD}"/tmp/ecreate/*.img mv ~/emgen/*.img "${EMGEN_BUILD}"/tmp/ecreate/ chown root:root "${EMGEN_BUILD}"/tmp/ecreate/*.img Now that all of that is cleared up, run QEMU with both the hard disk and floppy images, booting from the floppy: qemu -fda fda.img -hda hda.img -boot a When GRUB boots up, install it to the hard drive's MBR using these commands: root (hd0,0) setup (hd0) halt Now the system should exit and you officially have a complete Hard Drive Image that is bootable, and contains your virtualize system! Congrats! ===== References ===== * [[gw>TinyGentoo]] - The HOWTO that this project is based on. Their idea was to create a system for a flash drive, I'd like to take it a bit further. * [[http://www.bulah.com/embeddedgentoo.html|Embedded Gentoo]] - A tutorial on how to install Gentoo on an embedded device from stage 1. Also a strong foundation for this document. * [[http://www.squashfs-lzma.org/|Official Squashfs LZMA]] - Official site dedicated to using SquashFS with LZMA. * [[http://www.kernel.org/pub/linux/docs/|Linux Kernel Documentation]] - Provides a lot of useful information about how the Linux Kernel works. * [[http://uclibc.org/lists/busybox/2005-December/017397.html|ro root fs w/ busybox?]] - A mailing list reply explaining how to remount /dev/root as read/write. * [[http://www.h7.dion.ne.jp/~qemu-win/|QEMU on Windows]] - A port of QEMU to Windows so that we can run our Linux system on Windows. * [[http://fabrice.bellard.free.fr/qemu/qemu-doc.html|QEMU Documentation]] - Useful information about all of the options that QEMU provides. * [[http://www.gnu.org/software/grub/manual/html_node/Creating-a-GRUB-boot-floppy.html|Creating a GRUB boot floppy]] - Information from the GRUB manual on how to create a bootable floppy disk. * [[http://www.geocities.com/gtalon51/Articles/MLS/Minimal_Linux_system.html|A Minimal Linux System From Scratch]] - A similar project that was very useful in explaining some specifics regarding QEMU image creation.