Archive for July 2009

New Android 1.5 branch available

Posted by on July 21st, 2009 | 12 Comments

Inside the Android repository there is a new branch available with the name cupcake-release. On the Android Platform Forum (http://groups.google.com/group/android-platform/browse_thread/thread/4853d33c28abb5f5) this is what being said about this new branch

cupcake-release is the branch that is used to build devices, whereas
cupcake is the branch that is used to build the SDK (we separate them
for a variety of reasons). Until 1.5r2 (mid-May) they were identical
for all practical purposes, so the distinction (which had existed on
our internal servers for a long time) didn’t get reflected on the
outside, but since mid-May they’ve been drifting a bit and we decided
to reflect the differences on the outside.

The current state of cupcake-release is a loose match for the CRC01
release that is being deployed to some consumer devices.

Admittedly, we still need to do a better job at matching the various
forms of the version numbers, especially for the device builds.

I updated the blogposts about building Android 1.5. It will now use the cupcake-release branch. So read the blogposts (again) if you want to build your own Android version based on this source.

Building Android 1.5 – Chinese translation

Posted by on July 15th, 2009 | No Comments

Weiguo (a visitor of this website) made a chinese translation of the building Android 1.5 blogpost serie. So if you prefer chinese you can find the details here http://www.hiapk.com/bbs/thread-7714-1-1.html

FYI This version and the chinese version are not updated together (by one person). So difference between the two versions will probaly occur

Building Android 1.5 – Google Apps and audio files

Posted by on July 12th, 2009 | 18 Comments

Note 30 july 2009: updated the post. It now reflects the changes made to the repository (dream is now called dream-open)

Note 13 july 2009: Vladimir Stepanov post a reply today, telling me that the media files are already part of the source tree and can be found at ~/mydroid/fameworks/base/data/sounds. Adding the media files is a lot easier this way. I changed this blogpost about how to include those media files inside the system image.

This will be (for now) the last post of the building Android 1.5 for the Android Dev Phone. In the previous posts I explained how to prepare your building environment (Ubuntu 8.04), get the source and make it ready for building, building the source and finaly flash it to your device. Your own build should be working fine, but comparing to the official builds you are missing the Google applications like GMail, Google Maps, Android Market and YouTube. Besides the applications missing there are also no ringtones available on your device.

In this blogpost you will extract the Google Applications and media files from the official HTC Android 1.5 image (this can be done by extracting them from your device or with the use of unyaffs). The extracted files will be added to your own Android image. This results in an Android build which is almost the same as the HTC version.

Note: it is also possible to install the applications and add the media files while your device is running. But the purpose of this blog post is to create your own image which can be flashed over and over again (or even exchanged between other Android Dev Phone users)

Audio Files

In the official HTC image (http://www.htc.com/www/support/android/adp.html) a system/media directory exists which contains audio files for the alarms, notifications, ringtones and ui. These audio files are also part of the source tree but are not included by default.

Inside the ~/mydroid/frameworks/base/data/sounds folder you will find the different audio files. Together with those audio files there are two build scripts: OriginalAudio.mk and AudioPackage2.mk (containing a larger audio pack). Just rename one of those two script to Android.mk.

mv ~/mydroid/frameworks/base/data/sounds/OriginalAudio.mk ~/mydroid/frameworks/base/data/sounds/Android.mk

You can create a new image by running the make command from ~\mydroid location. The make command will not compile everything again. Only the changed files. In our case nothing will be recompiled, only the media files will be added to the system.img.

After the make command is finished you can flash the new image to your phone. If you are already running your own build, you only need to flash the system.img. You can found more details about the flash process at the Flashing The Phone blogpost.

Google Applications

Besides the missing audio files, are also the closed source Google Applications not installed. The following list of applications are not installed

checkin.apk (needed for login with Google Account)
Gmail.apk
GmailProvider.apk (the synchronization module for GMail)
GoogleApps.apk
GoogleContactsProvider.apk (replacement for the ContactsProvider. This version will synchronize with your Google Contacts)
GooglePartnerSetup.apk
GoogleSettingsProvider.apk
GoogleSubscribedFeedsProvider.apk (replacement for the  SubscribedFeedsProvider.apk)
gtalkservice.apk (for Google Talk)
Maps.apk (Google Maps)
MediaUploader.apk
NetworkLocation.apk (for determining your location based on network cells)
SetupWizard.apk (for setting up your Google Account connected with your device)
Street.apk (Google Maps StreetView)
Talk.apk (Google Talk application)
Vending.apk (Android Market applicatioN)
VoiceSearch.apk (Voice Search)
YouTube.apk (YouTube application)

Note: the HTC image does have more  application installed by default. But I think those are not relevant for your own build but HTC related: BugReport.apk, DebugTool.apk, FieldTest.apk, HtcLog.apk and QxdmLog.apk. Let me know if you think (with some information) that these packages should be part of your own build. Also the use/need of the Ftp.apk package is not clear. And I did not add the SystemUpdater.apk because this is your own build so you should provide updates for it (and not HTC/Google).

The Google applications make use of extra libraries and permission files. So also these files should be part of our build:

com.google.android.gtalkservice.jar (is placed inside the framework directory of the system image and will be used with GoogleTalk)
com.google.android.maps.jar (also part of the framework and is use by Google Maps)
com.google.android.gtalkservice.xml  (permission file for gtalkservice library, placed inside the etc\permissions directory)
com.google.android. maps.xml (permission file for maps library, placed inside the etc\permissions directory)

Just like with the media files, the extracting of these files can be done with adb or unyaffs. The keep thing seperated, I decided to put the extracted files inside the ~/mydroid/vendor/google directory. Because the google directory does not exist, go to the ~/mydroid/vendor location and create the google folder.

cd ~/mydroid/vendor
mkdir google
cd google

Extracting with adb

The procedure for extracting the files with the adb tool is the same as with the media files. Just create a new script called extract-google-files.sh inside the ~/mydroid/vendor/google folder. Place the following commands inside the file

#!/bin/sh

mkdir -p app
adb pull /system/app/checkin.apk app/checkin.apk
adb pull /system/app/Gmail.apk app/Gmail.apk
adb pull /system/app/GmailProvider.apk app/GmailProvider.apk
adb pull /system/app/GoogleApps.apk app/GoogleApps.apk
adb pull /system/app/GoogleContactsProvider.apk app/GoogleContactsProvider.apk
adb pull /system/app/GooglePartnerSetup.apk app/GooglePartnerSetup.apk
adb pull /system/app/GoogleSettingsProvider.apk app/GoogleSettingsProvider.apk
adb pull /system/app/GoogleSubscribedFeedsProvider.apk app/GoogleSubscribedFeedsProvider.apk
adb pull /system/app/gtalkservice.apk app/gtalkservice.apk
adb pull /system/app/Maps.apk app/Maps.apk
adb pull /system/app/MediaUploader.apk app/MediaUploader.apk
adb pull /system/app/NetworkLocation.apk app/NetworkLocation.apk
adb pull /system/app/SetupWizard.apk app/SetupWizard.apk
adb pull /system/app/Street.apk app/Street.apk
adb pull /system/app/Talk.apk app/Talk.apk
adb pull /system/app/Vending.apk app/Vending.apk
adb pull /system/app/VoiceSearch.apk app/VoiceSearch.apk
adb pull /system/app/YouTube.apk app/YouTube.apk

mkdir -p etc/permissions
adb pull /system/etc/permissions/com.google.android.gtalkservice.xml etc/permissions/com.google.android.gtalkservice.xml
adb pull /system/etc/permissions/com.google.android.maps.xml etc/permissions/com.google.android.maps.xml

mkdir -p framework
adb pull /system/framework/com.google.android.gtalkservice.jar framework/com.google.android.gtalkservice.jar
adb pull /system/framework/com.google.android.maps.jar framework/com.google.android.maps.jar

Save and exit the editor. Make sure that your device is connected to Ubuntu. Execute the script by executing the command (from the location ~/mydroid/vendor/google)

./extract-google-files.sh

The different files, needed for the Google applications, will be extracted from your device and placed inside app, etc or framework directory. The build script should be updated so that those files are included inside your own build. This will be explained further in this blog post. First I will explain how to get the files with the use of unyaffs.

Extracting with unyaffs

For the media files I already explained how to use the unyaffs tool. So I presume that the system image is extracted inside the ~/htc directory. You only need to copy the files to the ~/vendor/google location. This will need some more commands than with the media files.

mkdir -p ~/mydroid/vendor/google/app
cp ~/htc/app/checkin.apk ~/mydroid/vendor/google/app/checkin.apk
cp ~/htc/app/Gmail.apk ~/mydroid/vendor/google/app/Gmail.apk
cp ~/htc/app/GmailProvider.apk ~/mydroid/vendor/google/app/GmailProvider.apk
cp ~/htc/app/GoogleApps.apk ~/mydroid/vendor/google/app/GoogleApps.apk
cp ~/htc/app/GoogleContactsProvider.apk ~/mydroid/vendor/google/app/GoogleContactsProvider.apk
cp ~/htc/app/GooglePartnerSetup.apk ~/mydroid/vendor/google/app/GooglePartnerSetup.apk
cp ~/htc/app/GoogleSettingsProvider.apk ~/mydroid/vendor/google/app/GoogleSettingsProvider.apk
cp ~/htc/app/GoogleSubscribedFeedsProvider.apk ~/mydroid/vendor/google/app/GoogleSubscribedFeedsProvider.apk
cp ~/htc/app/gtalkservice.apk ~/mydroid/vendor/google/app/gtalkservice.apk
cp ~/htc/app/Maps.apk ~/mydroid/vendor/google/app/Maps.apk
cp ~/htc/app/MediaUploader.apk ~/mydroid/vendor/google/app/MediaUploader.apk
cp ~/htc/app/NetworkLocation.apk ~/mydroid/vendor/google/app/NetworkLocation.apk
cp ~/htc/app/SetupWizard.apk ~/mydroid/vendor/google/app/SetupWizard.apk
cp ~/htc/app/Street.apk ~/mydroid/vendor/google/app/Street.apk
cp ~/htc/app/Talk.apk ~/mydroid/vendor/google/app/Talk.apk
cp ~/htc/app/Vending.apk ~/mydroid/vendor/google/app/Vending.apk
cp ~/htc/app/VoiceSearch.apk ~/mydroid/vendor/google/app/VoiceSearch.apk
cp ~/htc/app/YouTube.apk ~/mydroid/vendor/google/app/YouTube.apk

mkdir -p ~/mydroid/vendor/google/etc/permissions
cp ~/htc/etc/permissions/com.google.android.gtalkservice.xml ~/mydroid/vendor/google/etc/permissions/com.google.android.gtalkservice.xml
cp ~/htc/etc/permissions/com.google.android.maps.xml ~/mydroid/vendor/google/etc/permissions/com.google.android.maps.xml

mkdir -p ~/mydroid/vendor/google/framework
cp ~/htc/framework/com.google.android.gtalkservice.jar ~/mydroid/vendor/google/framework/com.google.android.gtalkservice.jar
cp ~/htc/framework/com.google.android.maps.jar ~/mydroid/vendor/google/framework/com.google.android.maps.jar

The different files needed for the application are now placed inside the app, etc and framework directory (at ~/mydroid/vendor/google). Next step is to edit the buildscripts

Changing the build script for the Google applications

The htc_dream.mk build script should be edited. I decided to change the htc_dream.mk build script so that it does not depend on the build/target/product/generic.mk script (which depends on the build/target/product.core.mk file). The reason I do this, is that I don’t want to have the generic providers be part of my own build. Those providers will be replaced with the Google specific version (so the ContactsProvider is not needed anymore because the GoogleContactsProvider will be used instead). I could off course remove them from the generic.mk and core.mk files, but I don’t want to edit those files to keep things seperated. Because the core.mk also contains packages which I don’t want to be part of the build, I cannot change the dependency of the htc_dream.mk from generic.mk to core.mk. So the dependencies will be completly removed.

At the point also the SdkSetup package is not needed anymore, because we are using the SetupWizard package. (The SdkSetup package does some small things like making it possible to make phone calls. This functionality is also part of the SetupWizard package).

To add the Google applications a PRODUCT_COPY_FILES section should be added to the htc_dream.mk file. We use PRODUCT_COPY_FILES because the packages are already in a binary format and only need to be copied to the system image.

One of the packages is the NetworkLocation package. This package contains functionality to determine your position based on network GSM cells. To get this package working an extra build property should be set. This can be done by adding the PRODUCT_PROPERTY_OVERRIDES variable to the htc_dream.mk file which contains the property ro.com.google.locationfeatures=1

The complete htc_dream.mk script (which can be find inside the ~/mydroid/vendor/htc/dream-open location) is placed below.

PRODUCT_PROPERTY_OVERRIDES := \
    ro.config.notification_sound=F1_New_SMS.ogg \
    ro.com.google.locationfeatures=1

PRODUCT_PACKAGES := \
    framework-res \
    Browser \
    Contacts \
    Launcher \
    HTMLViewer \
    Phone \
    DownloadProvider \
    GoogleSearch \
    MediaProvider \
    SettingsProvider \
    TelephonyProvider \
    UserDictionaryProvider \
    PackageInstaller \
    Bugreport \
    AlarmClock \
    AlarmProvider \
    Calendar \
    Camera \
    DrmProvider \
    LatinIME \
    Mms \
    Music \
    Settings \
    Sync \
    Updater \
    CalendarProvider \
    SyncProvider \
    Calculator \
    Email \
    ImProvider \
    VoiceDialer

PRODUCT_COPY_FILES := \
	vendor/google/app/checkin.apk:system/app/checkin.apk \
	vendor/google/app/Gmail.apk:system/app/Gmail.apk \
	vendor/google/app/GmailProvider.apk:system/app/GmailProvider.apk \
	vendor/google/app/GoogleApps.apk:system/app/GoogleApps.apk \
	vendor/google/app/GoogleContactsProvider.apk:system/app/GoogleContactsProvider.apk \
	vendor/google/app/GooglePartnerSetup.apk:system/app/GooglePartnerSetup.apk \
	vendor/google/app/GoogleSettingsProvider.apk:system/app/GoogleSettingsProvider.apk \
	vendor/google/app/GoogleSubscribedFeedsProvider.apk:system/app/GoogleSubscribedFeedsProvider.apk \
	vendor/google/app/gtalkservice.apk:system/app/gtalkservice.apk \
	vendor/google/app/Maps.apk:system/app/Maps.apk \
	vendor/google/app/MediaUploader.apk:system/app/MediaUploader.apk \
	vendor/google/app/NetworkLocation.apk:system/app/NetworkLocation.apk \
	vendor/google/app/SetupWizard.apk:system/app/SetupWizard.apk \
	vendor/google/app/Street.apk:system/app/Street.apk \
	vendor/google/app/Talk.apk:system/app/Talk.apk \
	vendor/google/app/Vending.apk:system/app/Vending.apk \
	vendor/google/app/VoiceSearch.apk:system/app/VoiceSearch.apk \
	vendor/google/app/YouTube.apk:system/app/YouTube.apk \
	vendor/google/etc/permissions/com.google.android.gtalkservice.xml:system/etc/permissions/com.google.android.gtalkservice.xml \
	vendor/google/etc/permissions/com.google.android.maps.xml:system/etc/permissions/com.google.android.maps.xml \
	vendor/google/framework/com.google.android.gtalkservice.jar:system/framework/com.google.android.gtalkservice.jar \
	vendor/google/framework/com.google.android.maps.jar:system/framework/com.google.android.maps.jar

# Overrides
PRODUCT_NAME := htc_dream
PRODUCT_DEVICE := dream-open
PRODUCT_MANUFACTURER := htc
PRODUCT_BRAND := generic
PRODUCT_POLICY := android.policy_phone

Save the file and exit the editor. The build script for the google applications is now ready. Before running the make command again remove the ~/mydroid/out/target/product/dream-open/system directory first. If you don’t remove this directory when you remove packages (like the SdkSetup) from your build, those packages will still be copied to your final build. The system directory contains the binary version of your system image but when you remove this directory the binary files will not be recompiled again (there are intermediates version inside the ~/mydroid/out/target/product/dream/obj directory

rm -rf ~/mydroid/out/target/product/dream-open/system

Now you can run the make command again (from the ~\mydroid locatin).  Like I already said, the make command will not compile everything again. Only the changed files. In our case nothing will be recompiled, only the google files will be added to the system.img.

After the make command is finished you can flash the new image to your phone. If you are already running your own build, you only need to flash the system.img. You can found more details about the flash process at the Flashing The Phone blogpost.

What’s next?

If you start your device with the image containing the Google applications files, the Setup Wizard will be shown. Complete the steps of this wizard to connect your Google Account to your device. You will now have an image running that contains the different Google applications which should run correctly (also the Calendar application should work now).

So that brings us to the end of the Building Android 1.5 series. I hoped you liked it and got enough information for making your own build. If not, just post a reply on one of the blog posts so that I can give you more details.  Although this is the final post of this blogpost serie, I will continue writing about Android (and other technologies). So look at my website often or follow me on Twitter to get updates.

Blogpost in the Building Android 1.5 serie

Working on the layout

Posted by on July 6th, 2009 | No Comments

Something went wrong when changing the layout. Therefore the basic theme is temporary used.

Building Android 1.5 – Flashing the phone

Posted by on July 3rd, 2009 | 31 Comments

Note 30 july 2009: updated the post. It now reflects the changes made to the repository (dream is now called dream-open)

In the previous blog posts I described how to setup your build environment, get the Android source and how to build it for the Android Dev Phone. When you followed these steps you should now have an own version of the Android OS image available which can be flashed on your Android Dev Phone. But first, as usual, some preperations to make the flashing succesful.

Note: flashing your Android phone will remove your data from the phone (not the SD card). For example the SMS messages will be destroyed. It is possible to keep this “userdata” by NOT flashing the userdata partition. I found out that this can sometimes result in strange behaviors of the OS. Another option is to backup the data first (with applications from the Android Market)

Note: in my opinion flashing your Android will not brick your phone easily (brick means that you can only use your phone as a building block). You should be able to access the bootloader of the device even when your build does not work. But there is a possibility that things can go wrong. So I am not responsable for any damage to your phone. Know what you are doing. Besides the step by step walkthrough, I try to give you as much information so that you know why you are doing those steps. If you are not sure that what you are doing is right, just post a question to this blog or ask it on the Android Platform Forum  (http://groups.google.com/group/android-platform)

Radio image

The radio part of the Android OS for the Android Dev Phone is, just like the proprietary files, closed source. The radio part makes communication with the GSM network possible. This part is not build with the Android source code but can be downloaded at the HTC support site as separate download (http://www.htc.com/www/support/android/adp.html).

Because your build can be seen as an Android 1.5 version, it is important that you use the 1.5 version of the Radio image. Previous versions are not compatible.

The steps which I took to make sure that I have the right radio version on my phone and that it is working properly, is to flash the phone first to the official HTC 1.5 version of Android. If you already did that you can skip those steps. If you are not sure which radio version is running on your phone, follow these steps on your dev phone

Press menu (from the home screen)
Choose settings
Scroll down to the last option About Phone
At the bottom you will find information about which version is installed

The baseband version shows the radio image version which is in use. It should end with 2.22.19.16I (just like the download name ota-radio-2_22_19_26I). If this is not the case, you should first flash the phone with the HTC image for Android 1.5

The steps for flashing the phone with the HTC image are clearly described at http://www.htc.com/www/support/android/adp.html If you follow these steps (and use the Android 1.5 version) you should end with your dev phone running the HTC Android 1.5 version. I will give you some extra information which can be usefull during the flashing procedure.

About phone screen

About phone screen

Fastboot

For the update process described by HTC you will need the tool fastboot. You can download this tool from the same website as where you find the HTC images. Another option is to use your own version of fastboot which is created during the build process. You will find this application (which is in our case the Linux version) in ~/mydroid/out/host/linux-x86/bin. If you are a Windows user, you should use the fastboot executable provided by HTC. I don’t know if building the source on Mac OS X is as clear as on Ubuntu (it should be possible to compile the source) and that the fastboot version can be used.

Make sure that the fastboot utility location is inside the PATH variable. So that it can be found on the whole system independent on the location of which the terminal is currently active.

Connect ADP with Ubuntu

In a previous blog post I explained how to connect your device with Ubuntu which runs within VMWare as Virtual Machine. Those steps (http://www.johandekoning.nl/index.php/2009/02/07/android-dev-phone-ubuntu-and-vmware-server/) are also relevant when running Ubuntu natively (except for the VMWare steps).  Or follow the steps (when for example you want to use fastboot with Windows) which are written for the Android SDK http://developer.android.com/guide/developing/device.html

Flashing the device

So the HTC Android 1.5 version is running well? You made sure that the right radio version is installed? Fastboot is working properly? And made (optionally) a backup of your important data?

If that’s the case it is time for flashing the phone. The flashing part is quit easy and will take only a couple of minutes. First the device should be started in fastboot mode. The Android Dev Phone contains a bootloader which makes it possible to flash the phone. I think, but I am not sure, that this bootloader is always accessible (or you should execute some kind of operation which will destroy the bootloader part but I don’t know how to do that) making it possible to flash your Android over and over again with for example your own build (if your build is not working or you make changes) or the official HTC one.

To use fastboot execute the following steps on your device

If your device is running, shut is down
Power up the device while holding the BACK key
Hold the BACK key until you see a screen with the message Fastboot

The device is now in fastboot mode. The bootloader exists in different version. You can have a version with a rainbow background or one with skating Androids. It doesn’t matter, the flashing will work for both. Of course only when using a Dev Phone, the G1 does not support the flashing part.

The images which will be flashed to your device are inside the ~/mydroid/out/target/product/dream-open directory. So move to that directory and check if the following four images exists

  • boot.img (for booting the device, contains the kernel)
  • system.img (contains the Android framework, drivers, default applications and utilities)
  • userdata.img (used for the userdata storage. You don’t have to flash the userdata partition, so that your data will not be removed. But I got sometimes strange effects when there is still userdata available from a different Android image. )
  • recovery.img (used to restore and update the Android device. Don’t know when this will be used).

You don’t have to flash all partitions on your Android device every time. If you don’t change the kernel than you can skip the boot image. But off course you should flash the boot partition for the first time because it is different than the HTC one.

From the ~/mydroid/out/target/product/dream-open directory, use the following commands to flash the phone

fastboot flash boot boot.img
fastboot flash system system.img
fastboot flash recovery recovery.img
fastboot flash userdata userdata.img

Every fastboot command will send the image first to the device and (after succesfully received) will be written to the specific device partition. After all partitions are flashed succesfully you can reboot the device. This can be done by executing

fastboot reboot

It is also possible to exit fastboot by pressing Menu+CALL+END CALL button on your device.

It’s alive!

The first boot can take some time. You will get an Android splash screen, followed by another (grey logo) Android splash screen. Finally the home screen will be available. The Android system will be different comparing to the HTC version. A summary of what you can do and can’t with your own build.

Can

  • Wifi and bluetooth should work properly (because we added the right driver versions)
  • The Home button will work and you will have the Silent and Airplane option when holding the END CALL button (this is done by the SdkSetup package)
  • The led light should work when you get an SMS or connect the device
  • You should be able to make and receive calls
  • Camera, GPS and all the other sensors should be working fine
  • Start applications like Browser, Calculator, etc
  • Make use of 3G connection
  • Use android in a different language. The HTC version of Android has only the English language pack. Your own version has multiple language support.

Can’t

  • Use the Calendar application. Because this application depends on some Google libraries which are not on the system. I was looking how to seperate the Calendar application so that it is not Google dependent (and submit the source back to the source tree)
  • No Market application.
  • No Google applications (like GMail, GTalk, Maps).
  • No nice ringtones. The media directory which contains all the ringtones which are normally there is not part of the built.
  • No synchronization with Google (like contacts and calendar)

What’s next?

I will write another post about how to add the Google Apps, Market and ringtones to your own build. For now the flashing part is completed (so turn on the lights again :D ). Questions and other feedback can be given to this post.

Blogpost in the Building Android 1.5 serie