Building Android 1.5 – Getting the source

Written on June 7, 2009 by Johan de Koning | 40 Comments

Note (7 augusts 2009): patch 10889 is now part of the source tree and cupcake branch is used instead of cupcake-release

Note (30 july 2009): because of the changes made inside the Android repository, I decided to change the 10501 patch so that the wlan.ko module is not extracted from the device. The wlan.ko module is part of the dream-open project or you can build the latest version of the wlan.ko from source (as described in the blog posts). Instead of patch 10501, patch 10889 is used.

Note (27 july 2009): based on this forum post from JBQ (Software Engineer at Google), I decided to take the cupcake-release branch for the Building 1.5 series. I will write a different blogpost later about the other branches

Note (15 july 2009): the platform/vendor/htc/dream project inside the Android source tree is renamed to platform/vendor/htc/dream-open. Only the local_manifest.xml and the command for applying the patch should be changed (changes are made in this blogpost). The project is renamed because of some cleanup actions by Google. Problem is now with the latest Android1.5r2 source code, that Wifi is not working. Try to found a solution for that.

Note (24 june 2009): contributed a patch to the Android Source (change 10501) for updating the extact-files.sh and AndroidBoard.mk scripts which where outdated for the proprietary files. This simplifies the process of getting the source a little bit.

Note (9 june 2009): I updated this blogpost to make wifi and the led light of your Android Dev Phone working inside your own build.

In the previous blogpost I described how to set up the build environment. The next step will be downloading the source and apply some changes so that it will succesfully build. Getting the source is described at http://source.android.com/download but I will give a more detailed overview (and the details for making the build ready for the Android Dev Phone)

Installing repo

The repo tool will be used to get the Android source from Git. Before you can use Repo it should be installed, initialized and configured. I did not described the installation of repo in setting up the build environment because the repo script is Android specific.

Open a terminal inside Ubuntu and make sure that you are inside your home directory

cd ~

Inside the home directory, create a new directory called bin

mkdir bin

The bin directory should be part of your path ($PATH).  Edit the .bashrc file inside your user home directory (which is the current directoy you are working) and add the following line to the end of the file

export PATH=$PATH:~/bin

Save .bashrc and open a new terminal (this will execute .bashrc). Check if the bin directory is inside your path

echo $PATH

The results will be different for each installation because a different username will be used (or you would like my username jcdekoning) and maybe even more locations are inside your path. The result within my installation gives back

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/home/jcdekoning/bin

The last part (home/jcdekoning/bin) is pointing to the bin directory inside the users home directory (in my case /home/jcdekoning).

The next step is to download the repo script (with curl) and make it executable (chmod). Use to following two commands

curl http://android.git.kernel.org/repo > ~/bin/repo
chmod a+x ~/bin/repo

Initializing the Repo client

You need to make a location where you want to store the files (source, configuration, etc) from Git. I create a directoy inside the users home which is called mydroid

cd ~
mkdir mydroid
cd mydroid

Inside the mydroid directory you need to execute repo init to specify where the android files are remotely located. By running repo init -u git://android.git.kernel.org/platform/manifest.git you will get the master version of the Android source. The problem with this version is that it is mostly not stable and can have expiremental code. Therefore I use a specific branch of the Android source which hold a stable build for Android 1.5

repo init -u git://android.git.kernel.org/platform/manifest.git -b cupcake

Repo will ask for your name and email address. When you are planning to  submit code back to the Android source, use a email address which is assciated with a Google account.

If repo init is succesfully finished you will receive a message with at the

repo initialized in /home/jcdekoning/mydroid

You did not download the source yet, you only configured the repo tool so that it knowns where the source can be found. Inside the mydroid directoy you will find a .repo directory

Repo configuration changes for Android Dev Phone

Note: the download of the kernel source files is only needed if you want to compile the kernel by your own. You can also skip this configuration change and use the prebuilt kernel which is part of the dream-open project. It is possible that this prebuilt kernel is not the latest version

It is possible to download the source at this point, but we need to make some changes to retrieve specific source for the Android Dev Phone (internally called Dream).

Inside the .repo directory (within mydroid) you need to create a local_manifest.xml. This file containsa description of the three extra project needed for the Dev Phone. The contents of the local_manifest.xml

<?xml version="1.0" encoding="UTF-8"?>
 <manifest>
  <project path="kernel" name="kernel/msm" revision="refs/heads/android-msm-2.6.27"/>
 </manifest>

With the local_manifest.xml in place it is time to get the source code. Inside the mydroid directory (and not the .repo directory) execute the command

repo sync

No sit back, relax, drink some coffee or read some of my other blog posts because downloading the code will take some time.

Proprietary binaries

Note: the patch changes are now part of the source tree.

Some closed source files which are specific for the Dream need to be extracted from a Android device running 1.5. These files are not part of the source tree but are needed for running Android on the Android Dev Phone. Within you mydroid directory you find /vendor/htc/dream-open. There is a script called extract-files.sh. This script will use the adb tool to retrieve the files from an android device. But the content of this script is outdated for version 1.5.

To update the content of the extract-files.sh script (and the AndroidBoard.mk file) get a patch (10889):

repo download platform/vendor/htc/dream-open 10889/4

You can now get the proprietary files from you Android device by connecting it to Ubuntu and execute the extract.files.sh script

./extract-files.sh

Sometimes the Android connection between my Ubuntu in VMWare is not working properly. Therefore I use a different method for extracting the files from the official Android 1.5 image. By extracting the image provided by HTC

Using unyaffs

This part is only relevant when you don’t want or not able to extract the proprietary files from an Android device. So skip this part if you already have the files extracted. These files should be downloaded to /vendor/htc/dream-open/proprietary

The Android OS makes use of a yaffs file system. You can download the unyaffs tool at http://code.google.com/p/unyaffs/ to extract the content of an Android image. There is a prebuildversion available at the Downloads section. Place the unyaffs tool inside the ~/bin directory so that the tool can be found because it is inside the PATH definition

When the unyaffs is downloaded, make sure it can be executed

cd ~/bin
chmod a+x unyaffs

Official Android images provided by HTC for the Android Dev Phone can be found at http://www.htc.com/www/support/android/adp.html You will extract the system image to get the proprietary files. At the HTC download page, make sure you download the Android 1.5 system image (signed-dream_devphone_userdebug-img-148830.zip). Inside my home directory I create a new directory where the content of the zip file will be stored (~/htc). It is important to use the same directory name otherwise the script for copying the proprietary files will not work.

The downloaded system image archive contains four different .img files

When our own build is compiled we will end up with our build Android OS which also consists of these four different image files. The system.img will be extracted using the unyaffs tool

cd ~/htc
sudo ~/bin/unyaffs system.img

If the unyaffs is executed succesfully it will give a message like end of image. When looking at the content of the htc directory you will find new created files and folders (like app, bin, etc, lib, usr). The proprietary files are stored inside the etc and lib directory. Create with your favorite text editor a new file called htc-copy.sh (the location where you save this file doesn’t matter. I saved it inside ~/htc) and give it the following content.

#!/bin/sh

mkdir -p ~/mydroid/vendor/htc/dream-open/proprietary

cp ~/htc/bin/akmd ~/mydroid/vendor/htc/dream-open/proprietary/akmd

cp ~/htc/etc/AudioFilter.csv ~/mydroid/vendor/htc/dream-open/proprietary/AudioFilter.csv
cp ~/htc/etc/AudioPara4.csv ~/mydroid/vendor/htc/dream-open/proprietary/AudioPara4.csv
cp ~/htc/etc/AudioPreProcess.csv ~/mydroid/vendor/htc/dream-open/proprietary/AudioPreProcess.csv
cp ~/htc/etc/firmware/brf6300.bin ~/mydroid/vendor/htc/dream-open/proprietary/brf6300.bin
cp ~/htc/etc/gps.conf ~/mydroid/vendor/htc/dream-open/proprietary/gps.conf
cp ~/htc/etc/wifi/Fw1251r1c.bin ~/mydroid/vendor/htc/dream-open/proprietary/Fw1251r1c.bin
cp ~/htc/etc/wifi/tiwlan.ini ~/mydroid/vendor/htc/dream-open/proprietary/tiwlan.ini

cp ~/htc/lib/libaudioeq.so ~/mydroid/vendor/htc/dream-open/proprietary/libaudioeq.so
cp ~/htc/lib/libgps.so ~/mydroid/vendor/htc/dream-open/proprietary/libgps.so
cp ~/htc/lib/libhgl.so ~/mydroid/vendor/htc/dream-open/proprietary/libhgl.so
cp ~/htc/lib/libhtc_acoustic.so ~/mydroid/vendor/htc/dream-open/proprietary/libhtc_acoustic.so
cp ~/htc/lib/libhtc_ril.so ~/mydroid/vendor/htc/dream-open/proprietary/libhtc_ril.so
cp ~/htc/lib/libjni_pinyinime.so ~/mydroid/vendor/htc/dream-open/proprietary/libjni_pinyinime.so
cp ~/htc/lib/libmm-adspsvc.so ~/mydroid/vendor/htc/dream-open/proprietary/libmm-adspsvc.so
cp ~/htc/lib/libOmxCore.so ~/mydroid/vendor/htc/dream-open/proprietary/libOmxCore.so
cp ~/htc/lib/libOmxH264Dec.so ~/mydroid/vendor/htc/dream-open/proprietary/libOmxH264Dec.so
cp ~/htc/lib/libOmxMpeg4Dec.so ~/mydroid/vendor/htc/dream-open/proprietary/libOmxMpeg4Dec.so
cp ~/htc/lib/libOmxVidEnc.so ~/mydroid/vendor/htc/dream-open/proprietary/libOmxVidEnc.so
cp ~/htc/lib/libopencorehw.so ~/mydroid/vendor/htc/dream-open/proprietary/libopencorehw.so
cp ~/htc/lib/libpvasf.so ~/mydroid/vendor/htc/dream-open/proprietary/libpvasf.so
cp ~/htc/lib/libpvasfreg.so ~/mydroid/vendor/htc/dream-open/proprietary/libpvasfreg.so
cp ~/htc/lib/libqcamera.so ~/mydroid/vendor/htc/dream-open/proprietary/libqcamera.so
cp ~/htc/lib/libspeech.so ~/mydroid/vendor/htc/dream-open/proprietary/libspeech.so

cp ~/htc/lib/hw/lights.goldfish.so ~/mydroid/vendor/htc/dream-open/proprietary/lights.goldfish.so
cp ~/htc/lib/hw/lights.msm7k.so ~/mydroid/vendor/htc/dream-open/proprietary/lights.msm7k.so
cp ~/htc/lib/hw/sensors.trout.so ~/mydroid/vendor/htc/dream-open/proprietary/sensors.trout.so

chmod 755 ~/mydroid/vendor/htc/dream-open/proprietary/akmd

Execute the htc-copy.sh script so that the files will be copied to the ~/mydroid/vendor/htc/dream-open location.

chmod a+x htc-copy.sh
./htc-copy.sh

You can check if the script is succesful by going to ~/mydroid/vendor/htc/dream-open/proprietary and take a look at the contents of this directory.

What’s next?

So the source is downloaded, patched for the dream device and also the proprietary binary files are in place. The next step will be configuring the build and finally, the real work, compiling the source. I will write down more information about these steps in a next blog. Feel free to give comments on this post even if those are possitive.

Blogpost in the Building Android 1.5 serie

Read more from the Uncategorized category. If you would like to leave a comment, click here: 40 Comments. or stay up to date with this post via RSS, or you can Trackback from your site.
Social Bookmark : Technorati, Digg, de.licio.us, Yahoo, Blinkbits, Blogmarks, Google, Magnolia.

Leave a Comment

Name (required)

Email (required)

Website

Comments

40 Comments so far

  1. Marco June 10, 2009 8:14 pm

    isn’t there an error at:
    “adb pull /system/lib/hw/lights.msm7k.so proprietary/lighs.msm7k.so”

  2. Johan de Koning June 10, 2009 8:24 pm

    I extracted the official HTC Android 1.5 image and there is a lights.msm7k.so inside the lib/hw directory.

    I did not try it by my own on the HTC build (to be more specific I tried it now but that’s on my own build and that’s one is working).

    What is the error when running that specific pull command?

  3. Jamie June 11, 2009 12:16 pm

    I presume he means the typo “lighs” instead of “lights”

  4. Johan de Koning June 11, 2009 12:39 pm

    Hummm… stupid typo. Fixed it. Thanks

  5. Shimon Shnitzer June 15, 2009 12:31 am

    Hi,

    When you are reffering to AndroidBoard.mk, I guess you mean it’s in the “~/mydroid/vendor/htc/dream” dir and not just “~/mydroid/vendor/htc”, right ?

  6. Johan de Koning June 15, 2009 9:11 pm

    You are right. It should be ~/mydroid/vendor/htc/dream. I will change this.

  7. Johan de Koning June 23, 2009 7:14 am

    FYI: changes to the extract-files.sh and AndroidBoard.mk are now done by applying patch 10501. Above user comments can be outdated according to the updated patch process.

  8. Contribute to Android – My repo/git adventure — Johan de Koning June 23, 2009 9:01 pm

    [...] if I can easily contribute some changes to the Android source tree. In my blog about getting the Android Source, I applied a patch and edit the extract-files.sh and AndroidBoard.mk inside vendor/htc/dream by [...]

  9. Aaron June 26, 2009 9:44 pm

    Hi Johan — thanks for this article. I build a kernel using these instructions, and everything went fine except that when the Dream boots, the SD card is not auto-mounted to /sdcard. It is still recognized and loaded, just not mounted. Any idea what might be causing this?

  10. Johan de Koning June 29, 2009 8:49 pm

    I did not mention the mount issue with the sdcard. When I flash my phone again (i am switching between own build and HTC version) to my own builded version I will look at this issue.

  11. Aaron June 29, 2009 9:25 pm

    Great, thanks for looking into it. I’m glad to hear I’m not the only one with this problem. One possible clue — my build identified the SD card as:

    /class/mmc_host/mmc1/mmc1:9f40

    The stock kernel identifies it as:

    /devices/platform/msm_sdcc.2/mmc_host/mmc1/mmc1:9f40

  12. Aaron June 30, 2009 5:33 pm

    Sorry, this was entirely my fault and I sorted it out. I was using your old local_manifest.xml, which pointed to a different version of the kernel. Now that I’m using 2.6.27 the SD card mounts just fine. Thanks for your excellent walkthrough!

  13. Adrian July 7, 2009 11:50 am

    Hi,
    I am confused by this repo version control. I am used to using SVN where I can view the log when i am searching for a stable release.
    I stumble upon ur website before finding out i should use android-1.5r2. Where can I see this branch names?

    Thank you so much. Your blog helped me a lot.

  14. Johan de Koning July 7, 2009 12:46 pm

    I am also a SVN or Microsoft Team Foudation Server user. I use GitWeb to find out which tags are available on a project.

    For example project platform/frameworks details can be found at http://android.git.kernel.org/?p=platform/frameworks/base.git;a=summary which also gives a list of tags

    I hope there is also a repo command for this. But I could not find it yet.

  15. Adrian July 12, 2009 9:18 am

    Hi,
    With the help of this blog and http://source.android.com/using-eclipse, I am now able to build the platform project on my ubuntu machine.

    Now to the next step, can I use this platform project as an sdk to run the sample apps, eg. lunar lander, on my ubuntu machine? If yes could you guide me to it? I tried to run the Calculator as a java application but the error I get is “Selection does not contain a main type”. Pardon me, I am pretty new to this android development and eclipse

    my goal is to write my own RIL and via a serial driver, get the android project making phone calls via a GSM modem on my ubuntu machine.

    Sorry this is a double post which is also at androidforums.com
    Thank you.

    regards,
    Adrian

  16. Shimon Shnitzer July 12, 2009 10:48 am

    Hi,

    Did you or anyone else try building for the HTC Magic ?

    Do you know/thing I can just follow the same process, but just extract the same device-specific files from the ‘real’ device ?

    TIA

  17. Building Android 1.5 – Google Apps and audio files « Johan de Koning July 12, 2009 1:33 pm

    [...] Comments Shimon Shnitzer on Building Android 1.5 – Getting the sourceAdrian on Building Android 1.5 – Getting the sourceJohan de Koning on Android Dev Phone, [...]

  18. Johan de Koning July 12, 2009 11:50 pm

    I don’t have experience with the HTC Magic. Is it even possible to flash those devices (because there is no Dev Phone version)?

  19. Foogod July 15, 2009 12:08 am

    Thanks a lot for this series! Very helpful to have all this info in one place..

    FYI, it appears that the platform/vendor/htc/dream project has been renamed to platform/vendor/htc/dream-open . I had to change the reference in the local_manifest.xml file to get repo to pull down the sources properly..

  20. Yuan Song July 15, 2009 1:14 pm

    Strangely enough, when I tried to initialize and sync the “android-1.5r2″ branch yesterday, I didn’t get anything except for a 900M big “.repo” directory in the working directory. I’m going to try again today.

  21. Yuan Song July 16, 2009 12:00 am

    The previously mentioned problem is solved. It was actually caused by the outdated “platform/vendor/htc/dream” in “local_manifest.xml”. The sync was interrupted, and that’s why only “.repo” directory exists.

  22. Building Android 1.5 – Building the source « Johan de Koning July 16, 2009 11:51 pm

    [...] « Building Android 1.5 – Getting the source Contribute to Android – My repo/git adventure [...]

  23. Christian T. July 26, 2009 2:06 pm

    Something is strange… I always get:
    Lilith:mydroid Chrigi$ repo sync
    Traceback (most recent call last):
    File “/Volumes/android/mydroid/.repo/repo/main.py”, line 235, in
    _Main(sys.argv[1:])
    File “/Volumes/android/mydroid/.repo/repo/main.py”, line 217, in _Main
    repo._Run(argv)
    File “/Volumes/android/mydroid/.repo/repo/main.py”, line 123, in _Run
    cmd.Execute(copts, cargs)
    File “/Volumes/android/mydroid/.repo/repo/subcmds/sync.py”, line 195, in Execute
    all = self.GetProjects(args, missing_ok=True)
    File “/Volumes/android/mydroid/.repo/repo/command.py”, line 63, in GetProjects
    all = self.manifest.projects
    File “/Volumes/android/mydroid/.repo/repo/manifest_xml.py”, line 158, in projects
    self._Load()
    File “/Volumes/android/mydroid/.repo/repo/manifest_xml.py”, line 197, in _Load
    self._ParseManifest(False)
    File “/Volumes/android/mydroid/.repo/repo/manifest_xml.py”, line 228, in _ParseManifest
    (name)
    error.ManifestParseError: project kernel/common not found

    And if I remove this line then I get:

    Lilith:mydroid Chrigi$ repo sync
    Traceback (most recent call last):
    File “/Volumes/android/mydroid/.repo/repo/main.py”, line 235, in
    _Main(sys.argv[1:])
    File “/Volumes/android/mydroid/.repo/repo/main.py”, line 217, in _Main
    repo._Run(argv)
    File “/Volumes/android/mydroid/.repo/repo/main.py”, line 123, in _Run
    cmd.Execute(copts, cargs)
    File “/Volumes/android/mydroid/.repo/repo/subcmds/sync.py”, line 195, in Execute
    all = self.GetProjects(args, missing_ok=True)
    File “/Volumes/android/mydroid/.repo/repo/command.py”, line 63, in GetProjects
    all = self.manifest.projects
    File “/Volumes/android/mydroid/.repo/repo/manifest_xml.py”, line 158, in projects
    self._Load()
    File “/Volumes/android/mydroid/.repo/repo/manifest_xml.py”, line 197, in _Load
    self._ParseManifest(False)
    File “/Volumes/android/mydroid/.repo/repo/manifest_xml.py”, line 255, in _ParseManifest
    (project.name, self.manifestFile)
    error.ManifestParseError: duplicate project platform/vendor/htc/dream-open in /Volumes/android/mydroid/.repo/local_manifest.xml

  24. Christian T. July 26, 2009 8:35 pm

    Ok, I was able to fix it :) JPQ has changed the dream building process so less manual work is needed. So just change the local_manifest.xml so that only this line remains:
    <project path=”kernel” name=”kernel/msm” revision=”refs/heads/android-msm-2.6.27″>
    and it will work flawlessly :)
    I didn’t check if:
    <project path=”hardware/msm7k” name=”platform/hardware/msm7k” revision=”refs/heads/master”>
    is needed or not… is it?

  25. Christian T. July 28, 2009 1:21 pm

    checked and the msm7k line also has to be deleted.

  26. gunnar-medial July 28, 2009 1:32 pm

    Thanks for documenting for the benefit of others.
    Do you think I could transpose this for building Donut ?
    I seem to get similar build issues there regarding HTC private stuff
    unless (I presume) will apply close to what you outline above ?

  27. Johan de Koning July 28, 2009 4:29 pm

    Will look at the Donut branch later this week after I updated the Building Android 1.5 blogposts because of the repository changes

  28. Johan de Koning July 28, 2009 4:30 pm

    Is the msm7k also not needed for example the led light on the HTC Dream?

    I will try to create a build without this project

  29. Johan de Koning July 29, 2009 7:13 am

    Checked this but the msm7k project is still needed to build some hardware libs

  30. Building Android 1.5 – Flashing the phone « Johan de Koning July 30, 2009 10:42 pm

    [...] Building Android 1.5 – Getting the source [...]

  31. Damwid July 31, 2009 9:25 am

    Seems change 10889 added a wrong tab at the end of line 77, make build error.

  32. Johan de Koning July 31, 2009 9:40 am

    Thanks for the reply. Found also the tab problem in change 10889 and was trying to update the changeset. Only my knowledge of repo/git is limitted. I started a discussion at http://groups.google.com/group/repo-discuss/browse_thread/thread/96c8dcaffee8cefb

  33. Johan de Koning July 31, 2009 7:36 pm

    The 10889 is solved. Use repo download platform/vendor/htc/dream-open 10889/4 to get the correct file

  34. anousak August 5, 2009 11:10 am

    Can’t wait to build Donut soon. Johan please post your Donut build results when you are ready ….

    Thanks,
    Anousak

  35. Randomly Designless » Blog Archive » Rolling your own Android rom for HTC Magic (SAPPHIRE) August 11, 2009 2:06 am

    [...] I followed this guide here [...]

  36. Johan de Koning » Building donut for dream – First attempt: a black screen August 11, 2009 7:31 am

    [...] [...]

  37. Jubeh August 18, 2009 7:31 am

    Johan. Will pulling the vendor files from the APD 1.5 OTA image from the same page as the fastboot system image work too? They’re inside an OTA update zip, so pulling the files is just as easy as opening (or extracting) the zip and putting them in the necessary directory. I, for example, worked it this way in windows 7:
    I downloaded the OTA update for 1.5 (don’t know yet if it’s for r3 CRC1 or HTC is still distributing r2 CBR43), then unzipped the system folder to the desktop. I created a directory called “proprietary” also at the destkop, and then made a windows batch file based on your linux script, but replacing ~/htc with system/ and cp with copy. Then I moved the files to my Ubuntu Virtualbox Guest through the shared folder.
    It’s a little convoluted (I could have just downloaded the zip in the ubuntu guest and moved the files manually myself), but I added all that info if you want to update your post.

  38. Frank January 20, 2010 3:09 am

    Hi Jon,

    There are two kind of Android Development Phone. One is HTC dream while the other one is HTC Magic. So does the cupcake works in both phone? and is there any differences when i build kernel image for those two kind of phones?

  39. Florian Rohrweck April 21, 2010 5:48 pm

    Help!
    I’ve got a serious problem… I ported android 1.5 successfully to the mini2440 and tried to add the google apps, but I am stuck in a loop starting the loaderanimation and launcher again and again.
    It spits out some problem with binder when I am connected to the serial port…
    Without google apps it boots just fine… but with google apps it keeps reloading just after booting…. Got any idea how to track down this problem?
    Greets Florian

  40. Akilesh Sethu July 29, 2010 11:26 am

    hey when i initialized repo to mydroid…
    .repo directory wasnt there in the folder…
    i am using ubuntu 10.04..32 bit…could that be a problem??