Johan de Koning
Note (30 july 2009): made some changes because of the repository changes (dream is now called dream-open)
Note (16 july 2009): based on the comments of Foogod a section is added to this blogpost about how to create an updated wifi module. The wlan.ko module which is downloaded by default is not compatible (by version) with the compiled kernel. The new wlan.ko module will resolve the Wifi issue
Note (2 july 2009): I updated the htc_dream.mk script. The updated version will also add the Calculator, Email, ImProvider (which is not GTalk Client) and the VoiceDialer packages to your build. These application are open source and are also part of the Android source tree. Don’t know why these are not by default in the product configuration file.
Note (23 june 2009): the SdkSetup package is not by default added to the system image. By editing the htc_dream.mk build script the SdkSetup package will be added. Previously I added the package by adding a LOCAL_MODULE_TAGS line to the build script of the SdkSetup package. But by changing the htc_dream.mk buildscript the process is more easier, clearer and the build script will also be used later for adding other packages.
My previous two blogitems about Android described the steps for setting up the build environment and getting the Android source. The next step is to build the source code. Let’s build our own kernel first.
Building the kernel
The kernel can be seen as the core of the OS. Execute the following steps to configure and perform a kernel build which is compatible for the dream hardware (ARM)
cd ~/mydroid/kernel export ARCH=arm export CROSS_COMPILE=arm-eabi- export PATH=$PATH:~/mydroid/prebuilt/linux-x86/toolchain/arm-eabi-4.2.1/bin cp arch/arm/configs/msm_defconfig .config make oldconfig && make
Result of this build will be an image of the kernel found at ~/mydroid/arch/arm/boot/zImage. This kernel will be used for building the other parts of the operating system.
Building the Wifi Module
The ~/mydroid/vendor/htc/dream-open contains a wlan.ko module by default when downloading the source from the Android repository. The problem is that this version is not compatible with the compiled kernel (different version number). Because the wlan.ko is not a proprietary file, you can compile your own version of this module which is compatible with the kernel.
The sourcecode of the wlan.ko module can be found at ~/mydroid/system/wlan/ti/sta_dk_4_0_4_32. Execute the following commands to create a version compatible wifi module
cd ~/mydroid/system/wlan/ti/sta_dk_4_0_4_32 export KERNEL_DIR=~/mydroid/kernel/ make
After make is complete there will be a binary version of the wlan.ko module inside the ~/mydroid/system/wlan/ti/sta_dk_4_0_4_32. This module should be copied to the ~/mydroid/vendor/htc/dream-open location
cp wlan.ko ~/mydroid/vendor/htc/dream-open
Note: make sure, when building the wifi module, that the same terminal is used (the terminal which was also used for building the kernel). This will keep the PATH location to the arm-eabi-gcc compiler available. If you closed the terminal/used another one execute the export PATH command first (before executing make)
export PATH=$PATH:~/mydroid/prebuilt/linux-x86/toolchain/arm-eabi-4.2.1/bin
Changing the HTC build script
The SdkSetup application (.apk) is needed to make incoming call possible (it sets the phone to the “provisioned” state) and makes the home button work. Because the package is not deployed by default, the htc_dream.mk build script should be changed. Edit the file htc_dream.mk inside the ~/mydroid/vendor/htc/dream-open directory by adding the following lines at top of the document
PRODUCT_PACKAGES := \ Calculator \ Email \ ImProvider \ SdkSetup \ VoiceDialer
Save the file and exit the editor. The SdkSetup package will now be part of the build.
Building the system image
It is time to start a long task which will compile and create the system image of the Android OS. Two lines added to a configuration file are needed and than the compiler can do it job (and you can watch television or walk with the dog because this takes some time). Go to the top level of the Android source
cd ~/mydroid
Use a texteditor to create a file called buildspec.mk. Add the following two lines to this file, save the file and exit the editor
TARGET_PRODUCT:=htc_dream TARGET_PREBUILT_KERNEL:=kernel/arch/arm/boot/zImage
The TARGET_PREBUILT_KERNEL point to the location of the kernel we build earlier.
Let’s start the build by running the command
make
What’s next?
Because I don’t want to wait until your build is over, I am now going to play on my Nintendo Wii. In the next post I will give details about the flash process of your Android Device. While you are waiting you can take some time to give some feedback on these blogposts (even if you want to talk about the Nintendo Wii).
Blogpost in the Building Android 1.5 serie
Hi,
Thanks for a very detailed explanation.
Waiting for the kernel to compile now...
One question - can you post another post explaining how I can compile a build for running it on the emulator within Eclipse ?
TIA
Hi,
Well - the kernel compiled (at least I see that zimage file).
But a make does not succeed -
says:
build/core/product_config.mk:261: WARNING: adding test OTA key
============================================
TARGET_PRODUCT=htc_dream
TARGET_BUILD_VARIANT=eng
TARGET_SIMULATOR=
TARGET_BUILD_TYPE=release
TARGET_ARCH=arm
HOST_ARCH=x86
HOST_OS=linux
HOST_BUILD_TYPE=release
BUILD_ID=
============================================
build/core/Makefile:17: Warning: overwriting command for target `out/target/product/dream/system/etc/apns-conf.xml'
build/core/Makefile:17: Warning: ignoring previous commands for target `out/target/product/dream/system/etc/apns-conf.xml'
build/core/Makefile:17: Warning: overwriting command for target `out/target/product/dream/system/lib/libjni_pinyinime.so'
build/core/base_rules.mk:384: Warning: ignoring previous commands for target `out/target/product/dream/system/lib/libjni_pinyinime.so'
build/core/Makefile:17: Warning: overwriting command for target `out/target/product/dream/system/etc/wifi/tiwlan.ini'
build/core/base_rules.mk:384: Warning: ignoring previous commands for target `out/target/product/dream/system/etc/wifi/tiwlan.ini'
No private recovery resources for TARGET_DEVICE dream
make: execvp: /bin/bash: Argument list too long
host Java: clearsilver (out/host/common/obj/JAVA_LIBRARIES/clearsilver_intermediates/classes)
Unrecognized option : 9999999
make: *** [out/host/common/obj/JAVA_LIBRARIES/clearsilver_intermediates/javalib.jar] Error 41
Looks like the wrong Java version is used. Did you use update-java-alternatives to set Java to version 5. Java 6 cannot be used.
Yes - I am using Java 6.
Pardon my ignorance, but how do I use update-java-alternatives ?
First of all are you using Ubuntu as development environment? If that's the case, just open a terminal and use the command
sudo update-java-alternatives -l
to see which java environments are installed and which is used. With
sudo update-java-alternatives -s
You can change the default java version. should be replaced by the java name shown in the update-java-alternatives -l list
Java 6 cannot be used for building the Android OS.
Hi,
Did as you suggested, got the same warnings, but NOT the error 41 at the end. Seems to be compiling.
While I am waiting for it to compile - do you happen to know how I would go about compiling this NOT for flashing on my ADP1, but for loading it into the Eclipse Emulator on my Ubuntu, and see my compiled source run there ?
Also - is there a way I can actually DEBUG the code then ? (i.e. set breakpoints in the source, browse it, etc)
I tried understanding this from the Google docs, but I can't figure out anything there Bear in mind - I am not a unix programmer - I come from the Windows world (been developing Android apps on Eclipse there - a12Keys for free, and others as shareware).
TIA
Hi again,
Hope I'm not being a pest.
Restarted everything, used Java 5.
Al went well - including kernel build, but - when I got to do the MAKE - got an error:
host C: acp <= build/tools/acp/acp.c
In file included from /usr/include/features.h:354,
from /usr/include/stdlib.h:25,
from build/tools/acp/acp.c:11:
/usr/include/gnu/stubs.h:7:27: error: gnu/stubs-32.h: No such file or directory
make: *** [out/host/linux-x86/obj/EXECUTABLES/acp_intermediates/acp.o] Error 1
Looks like I have stubs-64.h in that directory in stead of stubs-32.h
What do I need to change to make this work ?
TIA
Are you using a 64 bit version? Because I did my build on the 32bit version of Ubuntu.
I guess I am - using a package called wubi.exe to run ubuntu as dual boot on my Windows XP laptop.
BTW - if I do manage to compile and flash on my ADP1, is there a way I can change some stuff in the code (i.e. change some code lines to add/fix features), and debug it - is there any GUI IDE I can get/buy to allow me to run the code (either java or cpp) and set breakpoints, etc ?
Do you know what do I need to change to run the compiled stuff on the emulator in ubuntu (in stead of flashing it to the device) ?
TIA
Hi again.
Did the whole thing on a 32bit machine, compilation had tons of warnings, but more important - at the end it says:
host C: emulator <= external/qemu/sockets.c
external/qemu/sockets.c: In function ‘sock_address_init_resolve’:
external/qemu/sockets.c:637: error: ‘EAI_NODATA’ undeclared (first use in this function)
external/qemu/sockets.c:637: error: (Each undeclared identifier is reported only once
external/qemu/sockets.c:637: error: for each function it appears in.)
make: *** [out/host/linux-x86/obj/EXECUTABLES/emulator_intermediates/sockets.o] Error 1
1. Why is there this "host C:emulator" there ? is was supposed to be compiling for the real device...
2. Can I ignore this error ? or did it stop my compilation ?
3. What do I do next ?
Thanks
Looks like it has something to do with the Linux version. I use an older version of Ubuntu (8.04) because as what I know Google mentioned that there can be build errors with later versions.
But for this issue there is a workaround (I don't know if it woks because I don't have this issue) http://groups.google.com/group/android-porting/browse_thread/thread/8432b5f2accb718a/8bbec3737be7fc28?lnk=gst&q=external/qemu/sockets.c+‘#
BTW It is normal that you get a lot of compiler warnings.
Hi,
It worked ! got it compiles, and flashed onto my device !
Thanks !!!
Some apps like Calculator, Email, gMail, Calendar are missing, though.
I know some of them are not included in the source code, and I have to copy them from the original device (which ones ? how ?)
But I do see the Calculator app in the source - could it be it was not compiled correctly ?
Also - can I use this images to load them on the emulator on ubuntu ? how ?
Thanks again.
BTW - do you do consulting ? if yes - please email me and I would like to discuss something with you.
I didn't had time to create new blog posts. One will be about the flashing part (what you already did) and the other one is about adding Google Apps (and also applications like the Calculator for which the source will be used).
Maybe I have some time tonight. But these posts will come.
It should also be possible to make an SDK version of the Android Source. I think that that should even be easier because the hardware is "emulated". Hope to find some time to look at creating a SDK version.
Nice detailed work, Johan. This is very useful stuff. I wonder how you figured all this out since it is not updated on source.android.com? Thank you much for the help. I will be trying once I get my hands on a device.
Also, I too am waiting for info on getting it working with the emulator. If you find the time I'm sure it will be useful for many. I was able to follow your instructions all the way to getting all the code. All that is left is extracting the proprietary files, building the kernel and building the system image.
Thanks for your reply. Lot's of information about the build process can be found/asked at http://groups.google.com/group/android-platform
Hi,
thanks for your kind in sharing this with us. But I have one question, would you be able to localize your build as well? If so, what are tools you've used?
Thanks,
Anousak
Hi Johan et al,
I would like to add additional locale, e.g lo-LA or other, before building it from the source so how do we add additional LOCALE?
Thanks in advance.
Anousak
Good question. Will try to look for more details about how to add an additional locale.
Thanks, Johan.
I am trying to fetch all files and try to build it according to your guidelines and hope it is all working in the first try.
Hope that we can add additional locales soon.
Thanks,
Anousak
Johan,
Was wondering if what we are building is of the latest HTC's Sense GUI or not.
Thanks,
anousak
No the version you are building is the one which can be compared with the official HTC Android 1.5 build.
I also thought that the HTC Sense GUI will not be part of the Android source tree (closed source by HTC). But correct me if I am wrong
When fetching files from the Android source tree, make sure that you read this comment (i will try to update my blog later today)
http://www.johandekoning.nl/index.php/2009/07/03/building-android-1-5-flashing-the-phone/comment-page-1/#comment-329
Looks like the htc dream project is called dream-open. Don't know why they changed it.
Thanks. I have modified and still fetching the files (almost 13 hours as it is being clogged my 12kb ADSL bandwidth).
TODOs/wish list:
1) Add additional locale for localization (MUST)
3) Add more apps before we make the build
4) Be able to modify or add certain features to our desired such as boot screens (first and second) and add new logos and backgrounds and etc.
4) Incorporate Sense UI...if only if it is possible
5) etc etc
Thanks for your kind attention ...look forward for my build.
Anousak
Hi,
Thank you very much for your post.
I try to build the source base on your post but got the following exception:
build/core/product_config.mk:173: *** No matches for product "htc_dream". Stop.
Seems there are some changes recently in the repo. How to deal with it then?
Thank you!
The htc dream project is renamed from dream to dream open. Will update the post about getting the source today
Thanks for your feedback.
Updated the "getting the source" blogpost today with the dream-open change. Only wifi is now not working in the build (version issue). Try to find a fix for this
thank you for your quick feedback!
is it possible to change the kernel to version "2.6.29"?
Hi Johan,
Thank you very much for your quick response.
Could you give us some hint what need to be changed if we want to build the source for htc_magic.
(I know there are some close source libs, but I already have a rom dump of magic with version 1.0)
I did the following but it doesn't work:
1. build the source for htc_dream as your post described.
2. replace the boot.img from the dump rom
3. flash the boot.img, system.img, recovery.img and userdata.img
restart... it hangs from the very very beginning (hami logo)
if I use the system folder and the boot.img to make a update.zip and apply, it tells me that the updat.zip is broken
Thank you!
I tried running 2.6.29 but than the device did not boot. Will try again later if I find some sparetime for this.
I don't have a HTC Magic and I didn't know that those can be flashed by an own build. Maybe I should check Ebay and buy one (if I have the money
)
Maybe you can use the unyaffs tool (as described in my posts) to unzip the system.img (the magic rom dump) and your own build. This give you the possiblity to compare both system images to see the differences.
I think my problem is how to build the boot.img for magic
seems the generated boot.img is for G1 only and the dump boot.img is not compatible with the build system.img
(You can use this mail acc as me gtalk, we can communicate online), thank you!
Hi,
I tried to build Wifi module but failed with the following errors message:
anousak@laonux:~/mydroid/system/wlan/ti/sta_dk_4_0_4_32$ make
rm -f ./libestadrv.a
make -C /home/anousak/mydroid/kernel M=`pwd` ARCH=arm CROSS_COMPILE=arm-eabi- modules
make[1]: arm-eabi-gcc: Command not found
make[1]: Entering directory `/home/anousak/mydroid/kernel'
CC [M] /home/anousak/mydroid/system/wlan/ti/sta_dk_4_0_4_32/./common/src/core/data_ctrl/Tx/MsduList.o
/bin/sh: arm-eabi-gcc: not found
make[2]: *** [/home/anousak/mydroid/system/wlan/ti/sta_dk_4_0_4_32/./common/src/core/data_ctrl/Tx/MsduList.o] Error 127
make[1]: *** [_module_/home/anousak/mydroid/system/wlan/ti/sta_dk_4_0_4_32] Error 2
make[1]: Leaving directory `/home/anousak/mydroid/kernel'
make: *** [libestadrv.a] Error 2
Thanks in advance.
Anousak
Make sure that you are using the same terminal as for making the kernel build so that the same $PATH is active (and the arm-eabi-gcc compiler can be found).
Otherwise execute the following command (before the make) inside the terminal so that te arm-eabi-gcc compiler can be found
export PATH=$PATH:~/mydroid/prebuilt/linux-x86/toolchain/arm-eabi-4.2.1/bin
Thanks, Johan. That does it.
I builded the whole thing with the Kernel 2.6.29. It boots up normally. But i can't sign in because the 3G Connection fails. Although i made up the correct APN.
This is interesting. Maybe you can use DDMS to get log information while booting
When building the 2.6.29 kernel, didn't you get configuration options? I didn't had a change to look at those in details so I choosed the default but I was not sure if those where right.
I choosed a lot times no. Only at the new bluetooth driver i choosed yes. And I also added Netfilter for Wifi Tethering to the Kernel.
3G is working on my build (with the 2.6.29 kernel). At first sight it looks like it is not working and get an error that website cannot be found, but after a minute or two the 3G icon is shown and the connection is working
The problem was also that the camera wasn't working. This weekend I will build the Android 1.5r2 Release with the "old" 2.6.27 Kernel. Later on if I have time i will try to port it to the Kernel 2.6.29. We could tty this together. Do you have ICQ or MSN oder something like that.
It looks like the proprietary files are the problem. Inside the log if see that the libOmxVidEnc.so module is loaded. After that errors are given back. I think the interface to this modules is different which results that the htc files are outdated.
It is a pity that the HTC files are closed source. This means that you cannot update to a newer kernel version if HTC will not update the proprietary files.
I was looking at a decompiler to see if it is possible to get some idea about which functions are inside the so files. But I didn't get any results.
i have this error when building wifi module:
jekyll@jekyll-desktop:~/mydroid/system/wlan/ti/sta_dk_4_0_4_32$ make
rm -f ./libestadrv.a
make -C /home/jekyll/mydroid/kernel/ M=`pwd` ARCH=arm CROSS_COMPILE=arm-eabi- modules
make[1]: ingresso nella directory «/home/jekyll/mydroid/kernel»
CC [M] /home/jekyll/mydroid/system/wlan/ti/sta_dk_4_0_4_32/./pform/linux/src/esta_drv.o
/home/jekyll/mydroid/system/wlan/ti/sta_dk_4_0_4_32/./pform/linux/src/esta_drv.c: In function 'tiwlan_drv_net_open':
/home/jekyll/mydroid/system/wlan/ti/sta_dk_4_0_4_32/./pform/linux/src/esta_drv.c:497: error: 'struct net_device' has no member named 'priv'
/home/jekyll/mydroid/system/wlan/ti/sta_dk_4_0_4_32/./pform/linux/src/esta_drv.c: In function 'tiwlan_drv_net_xmit':
/home/jekyll/mydroid/system/wlan/ti/sta_dk_4_0_4_32/./pform/linux/src/esta_drv.c:595: error: 'struct net_device' has no member named 'priv'
/home/jekyll/mydroid/system/wlan/ti/sta_dk_4_0_4_32/./pform/linux/src/esta_drv.c: In function 'tiwlan_drv_net_get_stats':
/home/jekyll/mydroid/system/wlan/ti/sta_dk_4_0_4_32/./pform/linux/src/esta_drv.c:743: error: 'struct net_device' has no member named 'priv'
/home/jekyll/mydroid/system/wlan/ti/sta_dk_4_0_4_32/./pform/linux/src/esta_drv.c: In function 'setup_netif':
/home/jekyll/mydroid/system/wlan/ti/sta_dk_4_0_4_32/./pform/linux/src/esta_drv.c:762: error: 'struct net_device' has no member named 'priv'
make[2]: *** [/home/jekyll/mydroid/system/wlan/ti/sta_dk_4_0_4_32/./pform/linux/src/esta_drv.o] Errore 1
make[1]: *** [_module_/home/jekyll/mydroid/system/wlan/ti/sta_dk_4_0_4_32] Errore 2
make[1]: uscita dalla directory «/home/jekyll/mydroid/kernel»
make: *** [libestadrv.a] Errore 2
jekyll@jekyll-desktop:~/mydroid/system/wlan/ti/sta_dk_4_0_4_32$
What is the problem?
thank you.
bye
Did you try to copy over these files from the official HTC Build? It didn't work? Yes maybe these files are really too old. But how did Cyanogen did this?? I don't know. Did you unpacked Cyanogens boot.img and looked into it? Try this. Maybe it's also a Kernel error.
Are you using the 2.6.27 or 2.6.29 source? I also got this error when building the wifi module for the 2.6.29 kernel. I solved it by applying a patch (10344)
repo download platform/system/wlan/ti 10344
I am not familiar with the Cyanogen build. I will give it a try (if source of this build is available). I see in the change log that they did something with the libcameraservice
Changes in v3.5.1
* Disabled camera noise in libcameraservice since the HTC camera does it internally.
i use 2.6.29 source. with the patch 10344 is ok now! Thank you!
can you add to this article? This is the only problem i found with 2.6.29 kernel source. There are other issue with this version of kernel?
Thank you very much Johan!
Ohh man. Now I think I know what to do. I forgot that in the Cyanogen Build a different Camera App is used. The HTC Camera. Look here: http://forum.xda-developers.com/showthread.php?t=521837
Its a new Camera app. They got it from an Image of a new Android Phone I think. Give this one a try.
Sorry look at this link: http://code.google.com/p/cupcake-frosting/
There you can download an Camera.apk. Try this one and look at this thread how to install: http://forum.xda-developers.com/showthread.php?t=515840
Ok forget my other post. Look at this link: http://forum.xda-developers.com/showthread.php?t=521421
If you download it there is a HTC Camera apk. Try this one. This is the Camera of the Hero Build and there is also a thread how to use it at: http://forum.xda-developers.com/showthread.php?t=521837
Look at both links. Maybe this is the key for our problem.