Note (30 july 2009): made some changes because of the repository changes (dream is now called dream-open)

Note (16 july 2009): based on the comments of Foogod a section is added to this blogpost about how to create an updated wifi module. The wlan.ko module which is downloaded by default is not compatible (by version) with the compiled kernel. The new wlan.ko module will resolve the Wifi issue

Note (2 july 2009): I updated the htc_dream.mk script. The updated version will also add the Calculator, Email, ImProvider (which is not GTalk Client) and the VoiceDialer packages to your build. These application are open source and are also part of the Android source tree. Don’t know why these are not by default in the product configuration file.

Note (23 june 2009): the SdkSetup package is not by default added to the system image.  By editing the htc_dream.mk build script the SdkSetup package will be added. Previously I added the package by adding a LOCAL_MODULE_TAGS line to the build script of the SdkSetup package. But by changing the htc_dream.mk buildscript the process is more easier, clearer and the build script will also be used later for adding other packages.

My previous two blogitems about Android described the steps for setting up the build environment and getting the Android source. The next step is to build the source code. Let’s build our own kernel first.

Building the kernel

The kernel can be seen as the core of the OS. Execute the following steps to configure and perform a kernel build which is compatible for the dream hardware (ARM)

cd ~/mydroid/kernel
export ARCH=arm
export CROSS_COMPILE=arm-eabi-
export PATH=$PATH:~/mydroid/prebuilt/linux-x86/toolchain/arm-eabi-4.2.1/bin
cp arch/arm/configs/msm_defconfig .config
make oldconfig && make

Result of this build will be an image of the kernel  found at ~/mydroid/arch/arm/boot/zImage. This kernel will be used for building the other parts of the operating system.

Building the Wifi Module

The ~/mydroid/vendor/htc/dream-open contains a wlan.ko module by default when downloading the source from the Android repository. The problem is that this version is not compatible with the compiled kernel (different version number). Because the wlan.ko is not a proprietary file, you can compile your own version of this module which is compatible with the kernel.

The sourcecode of the wlan.ko module can be found at ~/mydroid/system/wlan/ti/sta_dk_4_0_4_32. Execute the following commands to create a version compatible wifi module

cd ~/mydroid/system/wlan/ti/sta_dk_4_0_4_32
export KERNEL_DIR=~/mydroid/kernel/
make

After make is complete there will be a binary version of the wlan.ko module inside the ~/mydroid/system/wlan/ti/sta_dk_4_0_4_32. This module should be copied to the ~/mydroid/vendor/htc/dream-open location

cp wlan.ko ~/mydroid/vendor/htc/dream-open

Note: make sure, when building the wifi module, that the same terminal is used (the terminal which was also used for building the kernel). This will keep the PATH location to the arm-eabi-gcc compiler available. If you closed the terminal/used another one execute the export PATH command first (before executing make)

export PATH=$PATH:~/mydroid/prebuilt/linux-x86/toolchain/arm-eabi-4.2.1/bin

Changing the HTC build script

The SdkSetup application (.apk) is needed to make incoming call possible (it sets the phone to the “provisioned” state) and makes the home button work. Because the package is not deployed by default, the htc_dream.mk build script should be changed. Edit the file htc_dream.mk inside the ~/mydroid/vendor/htc/dream-open directory by adding the following lines at top of the document

PRODUCT_PACKAGES := \
 Calculator \
 Email \
 ImProvider \
 SdkSetup \
 VoiceDialer

Save the file and exit the editor. The SdkSetup package will now be part of the build.

Building the system image

It is time to start a long task which will compile and create the system image of the Android OS. Two lines added to a configuration file are needed and than the compiler can do it job (and you can watch television or walk with the dog because this takes some time).  Go to the top level of the Android source

cd ~/mydroid

Use a texteditor to create a file called buildspec.mk. Add the following two lines to this file, save the file and exit the editor

TARGET_PRODUCT:=htc_dream
TARGET_PREBUILT_KERNEL:=kernel/arch/arm/boot/zImage

The TARGET_PREBUILT_KERNEL point to the location of the kernel we build earlier.

Let’s start the build by running the command

make

What’s next?

Because I don’t want to wait until your build is over, I am now going to play on my Nintendo Wii. In the next post I will give details about the flash process of your Android Device. While you are waiting you can take some time to give some feedback on these blogposts (even if you want to talk about the Nintendo Wii).

Blogpost in the Building Android 1.5 serie

• Building Android 1.5 – Build environment
• Building Android 1.5 – Getting the source
• Building Android 1.5 – Building the source
• Building Android 1.5 – Flashing the phone
• Building Android 1.5 – Google Apps and audio files

Note (7 augusts 2009): patch 10889 is now part of the source tree and cupcake branch is used instead of cupcake-release

Note (30 july 2009): because of the changes made inside the Android repository, I decided to change the 10501 patch so that the wlan.ko module is not extracted from the device. The wlan.ko module is part of the dream-open project or you can build the latest version of the wlan.ko from source (as described in the blog posts). Instead of patch 10501, patch 10889 is used.

Note (27 july 2009): based on this forum post from JBQ (Software Engineer at Google), I decided to take the cupcake-release branch for the Building 1.5 series. I will write a different blogpost later about the other branches

Note (15 july 2009): the platform/vendor/htc/dream project inside the Android source tree is renamed to platform/vendor/htc/dream-open. Only the local_manifest.xml and the command for applying the patch should be changed (changes are made in this blogpost). The project is renamed because of some cleanup actions by Google. Problem is now with the latest Android1.5r2 source code, that Wifi is not working. Try to found a solution for that.

Note (24 june 2009): contributed a patch to the Android Source (change 10501) for updating the extact-files.sh and AndroidBoard.mk scripts which where outdated for the proprietary files. This simplifies the process of getting the source a little bit.

Note (9 june 2009): I updated this blogpost to make wifi and the led light of your Android Dev Phone working inside your own build.

In the previous blogpost I described how to set up the build environment. The next step will be downloading the source and apply some changes so that it will succesfully build. Getting the source is described at http://source.android.com/download but I will give a more detailed overview (and the details for making the build ready for the Android Dev Phone)

Installing repo

The repo tool will be used to get the Android source from Git. Before you can use Repo it should be installed, initialized and configured. I did not described the installation of repo in setting up the build environment because the repo script is Android specific.

Open a terminal inside Ubuntu and make sure that you are inside your home directory

cd ~

Inside the home directory, create a new directory called bin

mkdir bin

The bin directory should be part of your path ($PATH).  Edit the .bashrc file inside your user home directory (which is the current directoy you are working) and add the following line to the end of the file

export PATH=$PATH:~/bin

Save .bashrc and open a new terminal (this will execute .bashrc). Check if the bin directory is inside your path

echo $PATH

The results will be different for each installation because a different username will be used (or you would like my username jcdekoning) and maybe even more locations are inside your path. The result within my installation gives back

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/home/jcdekoning/bin

The last part (home/jcdekoning/bin) is pointing to the bin directory inside the users home directory (in my case /home/jcdekoning).

The next step is to download the repo script (with curl) and make it executable (chmod). Use to following two commands

curl http://android.git.kernel.org/repo > ~/bin/repo
chmod a+x ~/bin/repo

Initializing the Repo client

You need to make a location where you want to store the files (source, configuration, etc) from Git. I create a directoy inside the users home which is called mydroid

cd ~
mkdir mydroid
cd mydroid

Inside the mydroid directory you need to execute repo init to specify where the android files are remotely located. By running repo init -u git://android.git.kernel.org/platform/manifest.git you will get the master version of the Android source. The problem with this version is that it is mostly not stable and can have expiremental code. Therefore I use a specific branch of the Android source which hold a stable build for Android 1.5

repo init -u git://android.git.kernel.org/platform/manifest.git -b cupcake

Repo will ask for your name and email address. When you are planning to  submit code back to the Android source, use a email address which is assciated with a Google account.

If repo init is succesfully finished you will receive a message with at the

repo initialized in /home/jcdekoning/mydroid

You did not download the source yet, you only configured the repo tool so that it knowns where the source can be found. Inside the mydroid directoy you will find a .repo directory

Repo configuration changes for Android Dev Phone

Note: the download of the kernel source files is only needed if you want to compile the kernel by your own. You can also skip this configuration change and use the prebuilt kernel which is part of the dream-open project. It is possible that this prebuilt kernel is not the latest version

It is possible to download the source at this point, but we need to make some changes to retrieve specific source for the Android Dev Phone (internally called Dream).

Inside the .repo directory (within mydroid) you need to create a local_manifest.xml. This file containsa description of the three extra project needed for the Dev Phone. The contents of the local_manifest.xml

<?xml version="1.0" encoding="UTF-8"?>
 <manifest>
  <project path="kernel" name="kernel/msm" revision="refs/heads/android-msm-2.6.27"/>
 </manifest>

With the local_manifest.xml in place it is time to get the source code. Inside the mydroid directory (and not the .repo directory) execute the command

repo sync

No sit back, relax, drink some coffee or read some of my other blog posts because downloading the code will take some time.

Proprietary binaries

Note: the patch changes are now part of the source tree.

Some closed source files which are specific for the Dream need to be extracted from a Android device running 1.5. These files are not part of the source tree but are needed for running Android on the Android Dev Phone. Within you mydroid directory you find /vendor/htc/dream-open. There is a script called extract-files.sh. This script will use the adb tool to retrieve the files from an android device. But the content of this script is outdated for version 1.5.

To update the content of the extract-files.sh script (and the AndroidBoard.mk file) get a patch (10889):

repo download platform/vendor/htc/dream-open 10889/4

You can now get the proprietary files from you Android device by connecting it to Ubuntu and execute the extract.files.sh script

./extract-files.sh

Sometimes the Android connection between my Ubuntu in VMWare is not working properly. Therefore I use a different method for extracting the files from the official Android 1.5 image. By extracting the image provided by HTC

Using unyaffs

This part is only relevant when you don’t want or not able to extract the proprietary files from an Android device. So skip this part if you already have the files extracted. These files should be downloaded to /vendor/htc/dream-open/proprietary

The Android OS makes use of a yaffs file system. You can download the unyaffs tool at http://code.google.com/p/unyaffs/ to extract the content of an Android image. There is a prebuildversion available at the Downloads section. Place the unyaffs tool inside the ~/bin directory so that the tool can be found because it is inside the PATH definition

When the unyaffs is downloaded, make sure it can be executed

cd ~/bin
chmod a+x unyaffs

Official Android images provided by HTC for the Android Dev Phone can be found at http://www.htc.com/www/support/android/adp.html You will extract the system image to get the proprietary files. At the HTC download page, make sure you download the Android 1.5 system image (signed-dream_devphone_userdebug-img-148830.zip). Inside my home directory I create a new directory where the content of the zip file will be stored (~/htc). It is important to use the same directory name otherwise the script for copying the proprietary files will not work.

The downloaded system image archive contains four different .img files

• boot.img
• recovery.img
• system.img
• userdata.img

When our own build is compiled we will end up with our build Android OS which also consists of these four different image files. The system.img will be extracted using the unyaffs tool

cd ~/htc
sudo ~/bin/unyaffs system.img

If the unyaffs is executed succesfully it will give a message like end of image. When looking at the content of the htc directory you will find new created files and folders (like app, bin, etc, lib, usr). The proprietary files are stored inside the etc and lib directory. Create with your favorite text editor a new file called htc-copy.sh (the location where you save this file doesn’t matter. I saved it inside ~/htc) and give it the following content.

#!/bin/sh

mkdir -p ~/mydroid/vendor/htc/dream-open/proprietary

cp ~/htc/bin/akmd ~/mydroid/vendor/htc/dream-open/proprietary/akmd

cp ~/htc/etc/AudioFilter.csv ~/mydroid/vendor/htc/dream-open/proprietary/AudioFilter.csv
cp ~/htc/etc/AudioPara4.csv ~/mydroid/vendor/htc/dream-open/proprietary/AudioPara4.csv
cp ~/htc/etc/AudioPreProcess.csv ~/mydroid/vendor/htc/dream-open/proprietary/AudioPreProcess.csv
cp ~/htc/etc/firmware/brf6300.bin ~/mydroid/vendor/htc/dream-open/proprietary/brf6300.bin
cp ~/htc/etc/gps.conf ~/mydroid/vendor/htc/dream-open/proprietary/gps.conf
cp ~/htc/etc/wifi/Fw1251r1c.bin ~/mydroid/vendor/htc/dream-open/proprietary/Fw1251r1c.bin
cp ~/htc/etc/wifi/tiwlan.ini ~/mydroid/vendor/htc/dream-open/proprietary/tiwlan.ini

cp ~/htc/lib/libaudioeq.so ~/mydroid/vendor/htc/dream-open/proprietary/libaudioeq.so
cp ~/htc/lib/libgps.so ~/mydroid/vendor/htc/dream-open/proprietary/libgps.so
cp ~/htc/lib/libhgl.so ~/mydroid/vendor/htc/dream-open/proprietary/libhgl.so
cp ~/htc/lib/libhtc_acoustic.so ~/mydroid/vendor/htc/dream-open/proprietary/libhtc_acoustic.so
cp ~/htc/lib/libhtc_ril.so ~/mydroid/vendor/htc/dream-open/proprietary/libhtc_ril.so
cp ~/htc/lib/libjni_pinyinime.so ~/mydroid/vendor/htc/dream-open/proprietary/libjni_pinyinime.so
cp ~/htc/lib/libmm-adspsvc.so ~/mydroid/vendor/htc/dream-open/proprietary/libmm-adspsvc.so
cp ~/htc/lib/libOmxCore.so ~/mydroid/vendor/htc/dream-open/proprietary/libOmxCore.so
cp ~/htc/lib/libOmxH264Dec.so ~/mydroid/vendor/htc/dream-open/proprietary/libOmxH264Dec.so
cp ~/htc/lib/libOmxMpeg4Dec.so ~/mydroid/vendor/htc/dream-open/proprietary/libOmxMpeg4Dec.so
cp ~/htc/lib/libOmxVidEnc.so ~/mydroid/vendor/htc/dream-open/proprietary/libOmxVidEnc.so
cp ~/htc/lib/libopencorehw.so ~/mydroid/vendor/htc/dream-open/proprietary/libopencorehw.so
cp ~/htc/lib/libpvasf.so ~/mydroid/vendor/htc/dream-open/proprietary/libpvasf.so
cp ~/htc/lib/libpvasfreg.so ~/mydroid/vendor/htc/dream-open/proprietary/libpvasfreg.so
cp ~/htc/lib/libqcamera.so ~/mydroid/vendor/htc/dream-open/proprietary/libqcamera.so
cp ~/htc/lib/libspeech.so ~/mydroid/vendor/htc/dream-open/proprietary/libspeech.so

cp ~/htc/lib/hw/lights.goldfish.so ~/mydroid/vendor/htc/dream-open/proprietary/lights.goldfish.so
cp ~/htc/lib/hw/lights.msm7k.so ~/mydroid/vendor/htc/dream-open/proprietary/lights.msm7k.so
cp ~/htc/lib/hw/sensors.trout.so ~/mydroid/vendor/htc/dream-open/proprietary/sensors.trout.so

chmod 755 ~/mydroid/vendor/htc/dream-open/proprietary/akmd

Execute the htc-copy.sh script so that the files will be copied to the ~/mydroid/vendor/htc/dream-open location.

chmod a+x htc-copy.sh
./htc-copy.sh

You can check if the script is succesful by going to ~/mydroid/vendor/htc/dream-open/proprietary and take a look at the contents of this directory.

What’s next?

So the source is downloaded, patched for the dream device and also the proprietary binary files are in place. The next step will be configuring the build and finally, the real work, compiling the source. I will write down more information about these steps in a next blog. Feel free to give comments on this post even if those are possitive.

Blogpost in the Building Android 1.5 serie

• Building Android 1.5 – Build environment
• Building Android 1.5 – Getting the source
• Building Android 1.5 – Building the source
• Building Android 1.5 – Flashing the phone
• Building Android 1.5 – Google Apps and audio files