Posts Tagged ‘English’

Building Android 1.5 – Getting the source

Note (7 augusts 2009): patch 10889 is now part of the source tree and cupcake branch is used instead of cupcake-release

Note (30 july 2009): because of the changes made inside the Android repository, I decided to change the 10501 patch so that the wlan.ko module is not extracted from the device. The wlan.ko module is part of the dream-open project or you can build the latest version of the wlan.ko from source (as described in the blog posts). Instead of patch 10501, patch 10889 is used.

Note (27 july 2009): based on this forum post from JBQ (Software Engineer at Google), I decided to take the cupcake-release branch for the Building 1.5 series. I will write a different blogpost later about the other branches

Note (15 july 2009): the platform/vendor/htc/dream project inside the Android source tree is renamed to platform/vendor/htc/dream-open. Only the local_manifest.xml and the command for applying the patch should be changed (changes are made in this blogpost). The project is renamed because of some cleanup actions by Google. Problem is now with the latest Android1.5r2 source code, that Wifi is not working. Try to found a solution for that.

Note (24 june 2009): contributed a patch to the Android Source (change 10501) for updating the extact-files.sh and AndroidBoard.mk scripts which where outdated for the proprietary files. This simplifies the process of getting the source a little bit.

Note (9 june 2009): I updated this blogpost to make wifi and the led light of your Android Dev Phone working inside your own build.

In the previous blogpost I described how to set up the build environment. The next step will be downloading the source and apply some changes so that it will succesfully build. Getting the source is described at http://source.android.com/download but I will give a more detailed overview (and the details for making the build ready for the Android Dev Phone)

Installing repo

The repo tool will be used to get the Android source from Git. Before you can use Repo it should be installed, initialized and configured. I did not described the installation of repo in setting up the build environment because the repo script is Android specific.

Open a terminal inside Ubuntu and make sure that you are inside your home directory

cd ~

Inside the home directory, create a new directory called bin

mkdir bin

The bin directory should be part of your path ($PATH).  Edit the .bashrc file inside your user home directory (which is the current directoy you are working) and add the following line to the end of the file

export PATH=$PATH:~/bin

Save .bashrc and open a new terminal (this will execute .bashrc). Check if the bin directory is inside your path

echo $PATH

The results will be different for each installation because a different username will be used (or you would like my username jcdekoning) and maybe even more locations are inside your path. The result within my installation gives back

/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/home/jcdekoning/bin

The last part (home/jcdekoning/bin) is pointing to the bin directory inside the users home directory (in my case /home/jcdekoning).

The next step is to download the repo script (with curl) and make it executable (chmod). Use to following two commands

curl http://android.git.kernel.org/repo > ~/bin/repo
chmod a+x ~/bin/repo

Initializing the Repo client

You need to make a location where you want to store the files (source, configuration, etc) from Git. I create a directoy inside the users home which is called mydroid

cd ~
mkdir mydroid
cd mydroid

Inside the mydroid directory you need to execute repo init to specify where the android files are remotely located. By running repo init -u git://android.git.kernel.org/platform/manifest.git you will get the master version of the Android source. The problem with this version is that it is mostly not stable and can have expiremental code. Therefore I use a specific branch of the Android source which hold a stable build for Android 1.5

repo init -u git://android.git.kernel.org/platform/manifest.git -b cupcake

Repo will ask for your name and email address. When you are planning to  submit code back to the Android source, use a email address which is assciated with a Google account.

If repo init is succesfully finished you will receive a message with at the

repo initialized in /home/jcdekoning/mydroid

You did not download the source yet, you only configured the repo tool so that it knowns where the source can be found. Inside the mydroid directoy you will find a .repo directory

Repo configuration changes for Android Dev Phone

Note: the download of the kernel source files is only needed if you want to compile the kernel by your own. You can also skip this configuration change and use the prebuilt kernel which is part of the dream-open project. It is possible that this prebuilt kernel is not the latest version

It is possible to download the source at this point, but we need to make some changes to retrieve specific source for the Android Dev Phone (internally called Dream).

Inside the .repo directory (within mydroid) you need to create a local_manifest.xml. This file containsa description of the three extra project needed for the Dev Phone. The contents of the local_manifest.xml

<?xml version="1.0" encoding="UTF-8"?>
 <manifest>
  <project path="kernel" name="kernel/msm" revision="refs/heads/android-msm-2.6.27"/>
 </manifest>

With the local_manifest.xml in place it is time to get the source code. Inside the mydroid directory (and not the .repo directory) execute the command

repo sync

No sit back, relax, drink some coffee or read some of my other blog posts because downloading the code will take some time.

Proprietary binaries

Note: the patch changes are now part of the source tree.

Some closed source files which are specific for the Dream need to be extracted from a Android device running 1.5. These files are not part of the source tree but are needed for running Android on the Android Dev Phone. Within you mydroid directory you find /vendor/htc/dream-open. There is a script called extract-files.sh. This script will use the adb tool to retrieve the files from an android device. But the content of this script is outdated for version 1.5.

To update the content of the extract-files.sh script (and the AndroidBoard.mk file) get a patch (10889):

repo download platform/vendor/htc/dream-open 10889/4

You can now get the proprietary files from you Android device by connecting it to Ubuntu and execute the extract.files.sh script

./extract-files.sh

Sometimes the Android connection between my Ubuntu in VMWare is not working properly. Therefore I use a different method for extracting the files from the official Android 1.5 image. By extracting the image provided by HTC

Using unyaffs

This part is only relevant when you don’t want or not able to extract the proprietary files from an Android device. So skip this part if you already have the files extracted. These files should be downloaded to /vendor/htc/dream-open/proprietary

The Android OS makes use of a yaffs file system. You can download the unyaffs tool at http://code.google.com/p/unyaffs/ to extract the content of an Android image. There is a prebuildversion available at the Downloads section. Place the unyaffs tool inside the ~/bin directory so that the tool can be found because it is inside the PATH definition

When the unyaffs is downloaded, make sure it can be executed

cd ~/bin
chmod a+x unyaffs

Official Android images provided by HTC for the Android Dev Phone can be found at http://www.htc.com/www/support/android/adp.html You will extract the system image to get the proprietary files. At the HTC download page, make sure you download the Android 1.5 system image (signed-dream_devphone_userdebug-img-148830.zip). Inside my home directory I create a new directory where the content of the zip file will be stored (~/htc). It is important to use the same directory name otherwise the script for copying the proprietary files will not work.

The downloaded system image archive contains four different .img files

  • boot.img
  • recovery.img
  • system.img
  • userdata.img

When our own build is compiled we will end up with our build Android OS which also consists of these four different image files. The system.img will be extracted using the unyaffs tool

cd ~/htc
sudo ~/bin/unyaffs system.img

If the unyaffs is executed succesfully it will give a message like end of image. When looking at the content of the htc directory you will find new created files and folders (like app, bin, etc, lib, usr). The proprietary files are stored inside the etc and lib directory. Create with your favorite text editor a new file called htc-copy.sh (the location where you save this file doesn’t matter. I saved it inside ~/htc) and give it the following content.

#!/bin/sh

mkdir -p ~/mydroid/vendor/htc/dream-open/proprietary

cp ~/htc/bin/akmd ~/mydroid/vendor/htc/dream-open/proprietary/akmd

cp ~/htc/etc/AudioFilter.csv ~/mydroid/vendor/htc/dream-open/proprietary/AudioFilter.csv
cp ~/htc/etc/AudioPara4.csv ~/mydroid/vendor/htc/dream-open/proprietary/AudioPara4.csv
cp ~/htc/etc/AudioPreProcess.csv ~/mydroid/vendor/htc/dream-open/proprietary/AudioPreProcess.csv
cp ~/htc/etc/firmware/brf6300.bin ~/mydroid/vendor/htc/dream-open/proprietary/brf6300.bin
cp ~/htc/etc/gps.conf ~/mydroid/vendor/htc/dream-open/proprietary/gps.conf
cp ~/htc/etc/wifi/Fw1251r1c.bin ~/mydroid/vendor/htc/dream-open/proprietary/Fw1251r1c.bin
cp ~/htc/etc/wifi/tiwlan.ini ~/mydroid/vendor/htc/dream-open/proprietary/tiwlan.ini

cp ~/htc/lib/libaudioeq.so ~/mydroid/vendor/htc/dream-open/proprietary/libaudioeq.so
cp ~/htc/lib/libgps.so ~/mydroid/vendor/htc/dream-open/proprietary/libgps.so
cp ~/htc/lib/libhgl.so ~/mydroid/vendor/htc/dream-open/proprietary/libhgl.so
cp ~/htc/lib/libhtc_acoustic.so ~/mydroid/vendor/htc/dream-open/proprietary/libhtc_acoustic.so
cp ~/htc/lib/libhtc_ril.so ~/mydroid/vendor/htc/dream-open/proprietary/libhtc_ril.so
cp ~/htc/lib/libjni_pinyinime.so ~/mydroid/vendor/htc/dream-open/proprietary/libjni_pinyinime.so
cp ~/htc/lib/libmm-adspsvc.so ~/mydroid/vendor/htc/dream-open/proprietary/libmm-adspsvc.so
cp ~/htc/lib/libOmxCore.so ~/mydroid/vendor/htc/dream-open/proprietary/libOmxCore.so
cp ~/htc/lib/libOmxH264Dec.so ~/mydroid/vendor/htc/dream-open/proprietary/libOmxH264Dec.so
cp ~/htc/lib/libOmxMpeg4Dec.so ~/mydroid/vendor/htc/dream-open/proprietary/libOmxMpeg4Dec.so
cp ~/htc/lib/libOmxVidEnc.so ~/mydroid/vendor/htc/dream-open/proprietary/libOmxVidEnc.so
cp ~/htc/lib/libopencorehw.so ~/mydroid/vendor/htc/dream-open/proprietary/libopencorehw.so
cp ~/htc/lib/libpvasf.so ~/mydroid/vendor/htc/dream-open/proprietary/libpvasf.so
cp ~/htc/lib/libpvasfreg.so ~/mydroid/vendor/htc/dream-open/proprietary/libpvasfreg.so
cp ~/htc/lib/libqcamera.so ~/mydroid/vendor/htc/dream-open/proprietary/libqcamera.so
cp ~/htc/lib/libspeech.so ~/mydroid/vendor/htc/dream-open/proprietary/libspeech.so

cp ~/htc/lib/hw/lights.goldfish.so ~/mydroid/vendor/htc/dream-open/proprietary/lights.goldfish.so
cp ~/htc/lib/hw/lights.msm7k.so ~/mydroid/vendor/htc/dream-open/proprietary/lights.msm7k.so
cp ~/htc/lib/hw/sensors.trout.so ~/mydroid/vendor/htc/dream-open/proprietary/sensors.trout.so

chmod 755 ~/mydroid/vendor/htc/dream-open/proprietary/akmd

Execute the htc-copy.sh script so that the files will be copied to the ~/mydroid/vendor/htc/dream-open location.

chmod a+x htc-copy.sh
./htc-copy.sh

You can check if the script is succesful by going to ~/mydroid/vendor/htc/dream-open/proprietary and take a look at the contents of this directory.

What’s next?

So the source is downloaded, patched for the dream device and also the proprietary binary files are in place. The next step will be configuring the build and finally, the real work, compiling the source. I will write down more information about these steps in a next blog. Feel free to give comments on this post even if those are possitive.

Blogpost in the Building Android 1.5 serie

Building Android 1.5 – Build environment

Last year I bought an Android Dev Phone. I wanted to be one of the first to use this great device and operating system. The Android Dev Phone makes it possible to flash it with your own build version of Android. So why not building the Android OS from source?

This first part of Building Android 1.5 is about setting up the build environment. There is a description available at http://source.android.com/download. I will write down the steps I have taken to give you a step by step manual

Installation of Ubuntu

I installed Ubuntu as a Virtual Machine (because I am running Windows Vista). I used Ubuntu 8.04 because this is also the environment which Google is using (as I should believe). There are some issues with Ubuntu 8.10 for example when building the source.

I will not explain the installation of Ubuntu. This is really straight forward (perhaps even easier than installing Windows) when putting in the Ubuntu CD and boot from it.

When Ubuntu is installed I did sudo apt-get update and sudo apt-get upgrade to have the latest versions of all the packages which are installed by default

Required packaged

In order to retrieve the source code from git and to build the Android OS, the installation of some packages is required. Install the following packages by executing sudo apt-get install <pakagename>

  • git-core
  • gnupg (this package was already installed)
  • sun-java5-sdk (don’t use java 6 because this will result in errors during build)
  • flex
  • bison
  • gperf
  • libsdl-dev
  • libesd0-dev (the last 0 is a zero)
  • libwxgtk2.6-dev
  • build-essential
  • zip (already installed)
  • curl
  • libncurses5-dev
  • zlib1g-dev
  • valgrind (this packge is optional)

Instead of installing the packages seperately, you can also comine the apt-get install command

sudo apt-get install git-core gnupg sun-java5-jdk flex bison gperf libsdl-dev libesd0-dev libwxgtk2.6-dev build-essential zip curl libncurses5-dev zlib1g-dev valgrind

At http://source.android.com/download is written that Ubuntu 8.10 users should install a newer version of libreadline. I found out that I also needed this package because otherwise repo gives errors.

sudo apt-get install libreadline5-dev

Note: the Get source page on the android website uses a different package name (lib32readline5-dev) but that one does not exists

Java Environment

It could be possible that java 6 is installed on your own Ubuntu installation. To check which Java version is used, run the command

update-java-alternatives -l

In my Ubuntu enviroment only one java version is given back in the result

java-1.5.0-sun 53 /usr/lib/jvm/java-1.5.0-sun

If multiple versions are installed, use update-java-alternatives -s to change the default Java Environment. For example

update-java-alternatives -s java-1.5.0-sun

The location of the Java installation should also be set exported as $JAVA_HOME. Edit the .bashrc file inside your users home directory. Add the following line to the end of the file

export JAVA_HOME=/usr/lib/jvm/java-1.5.0-sun-1.5.0.16

The location of the Java installation can be different (depending on the version number which is in my case 1.5.0.16). Check the location first to see if it contains the Java installation.

Start a new terminal and check if $JAVA_HOME is working and pointing to the right location

echo $JAVA_HOME

When the result given back is correct (for example /usr/lib/jvm/java-1.5.0-sun.1.5.0.16) the $JAVA_HOME is correctly set.

You can also check if the correct Java version is used by running the command java -version

java -version

Result

java version "1.5.0_16"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_16-b02)
Java HotSpot(TM) Client VM (build 1.5.0_16-b02, mixed mode, sharing)

What’s next?

By installing Ubuntu, de required packages and setting up Java everyting is ready for downloading the source from Git and building your own version of the Android source. I will explain the steps of retrieving the source code from Git in a next blog item.  Feel free to give feedback on this post.

Blogpost in the Building Android 1.5 serie

Enable Latitude on Android

With the Android 1.1 and Android 1.5 images for the Android Dev Phone 1 (ADP1), Google Latitude is not available by default. It is possible to enable Latitude inside the Google Maps application by adding one record to a settings database.

At http://forum.xda-developers.com/showthread.php?p=3438480#post3438480 there are several steps described to enable Latitude. Adding a record to a settings database and updating a XML file (MapsActivity.xml) are needed to make Latitude available as menu item inside the Google Maps application.

Updating the MapsActivity.xml file is not that easy. The adb shell does not support a copy command and there is also no editor available inside the shell. With the use of cat it is possible to make a copy of the MapsActivity.xml to the sdcard of your Android device:

cat /data/data/com.google.android.apps.maps/shared_prefs/com.google.android.maps.MapsActivity.xml > /sdcard/MapsActivity.xml

Next you should mount the sdcard, edit the file on your desktop pc, unmount the sdcard and replace the MapsActivity.xml with the one which you changed on your sdcard:

cat /sdcard/MapsActivity.xml > /data/data/com.google.android.apps.maps/shared_prefs/com.google.android.maps.MapsActivity.xml

Because editing needs a lot of actions, I found a different way of enabling Latitude by only adding the record to the database. I performed the following steps on a device running Android 1.5, but I think it will also work on the Android 1.1 version.

  1. On your ADP1, choose Menu -> Settings -> Applications -> Manage applications -> Maps
  2. Push the Clear data button.This will remove the /data/data/com.google.android.apps.maps/shared_prefs/ folder which contains the MapsActivity.xml file.
    Don’t worry, the different xml files in this folder will be auto generated again with Latitude (Friend Finder) enabled.
  3. Clear Data of Maps Application

  4. Connect your ADP1 to your PC (with the use of the USB cable) and execute adb shell inside a Windows shell or Linux terminal
  5. Execute su inside the adb shell to give you more rights
  6. Change the current directory: cd /data/data/com.android.providers.settings/databases
  7. Run the sqlite3 application on the settings database: sqlite3 settings.db
  8. To see if the setting maps_enable_friend_finder is not already inside the settings database, make a dump of all the records inside the gservices table on the screen and look if the record already exists: .dump gservices
  9. Insert a new record: INSERT INTO “gservices” (name,value) VALUES(‘maps_enable_friend_finder’,’1′);. This record will hold the flag to enable Latitude
  10. Exit the sqlite3 application: .quit
  11. Exit the su mode: exit
  12. Exit the adb shell: exit
  13. On your Android device, run the Maps application. A popup will be shown that Latitude is available. Pressing Menu will show you the Join Latitude option.
  14. Latitude PopupJoin Latitude

By clearing the data of the Maps application (with the Clear data option) the /data/data/com.google.android.apps.maps/shared_prefs/ is removed containing settings for the map application. By adding the record to the database and starting the Maps application again, the /data/data/com.google.android.apps.maps/shared_prefs/ is generated again. Because of the maps_enable_friend_finder flag set to true (1) the MapsActivity.xml is also generated which enables Latitude.

WBXML .Net release 1.0

Today I released the first version of the WBXML .Net 1.0 library. With this library you can encode and decode WBXML messages.

More information can be found at http://wbxml.codeplex.com

Android Dev Phone, Ubuntu and VMWare Server

Compiling the source code for creating your own Android build is not supported within a Windows operating system (I don’t know if it is possible by the use of Cygwin). Because my notebook is running Windows XP, I use VMWare Server 2.0 which runs a Ubuntu 8.10 (Desktop) as virtual machine.

VMWare makes it possible to connect USB devices inside your virtual machine. Because I want to connect my Android Dev Phone to Ubuntu, I performed the following steps to make that possible.

VMWare hardware configuration

When creating a virtual machine (or before starting the virtual machine) make sure that the USB controller is added to the hardware list.

If this is not the fact. Add the USB Controller by choosing Add hardware and select the USB Controller from the Hardware type list.

Sharing the USB device

Connect your Android Dev Phone to your computer and start the Ubuntu Virtual Machine by pressing the Play button inside the VMWare Server web interface. When booting the Virtual Machine you will notice that a new icon (with an USB sign) is placed on the toolbar inside the VMWare Web Interface next to the Restart button. When you click on this icon a context menu will be shown with a list of devices that are plugged inside the host OS (in my case connected to Windows XP).

Mark the checkbox in front of the High Android Phone to make it available inside Ubuntu.

Udev configuration

You can perform a check, to see that the Android Dev Phone is shared with Ubuntu, by opening a Terminal (inside Ubuntu. Open a VMWare Remote Console to make use of the Ubuntu) and type the following command

lsusb

A list of USB devices will be shown. For example:

Bus 002 Device 001: ID ld6b:0001 Linux Foundation 1.1 root hub
Bus 001 Device 004: ID 0bb4:0c02 High Tech Computer Corp.
Bus 001 Device 001: ID ld6b:0002 Linux Foundation 2.0 root hub

The Android Dev Phone is the device with the name High Tech Computer Corp (HTC). Make a note of the first part of the device ID (in this case 0bb4) because this is needed for the udev rule.

Create a new file (as root)

sudo gedit /etc/udev/rules.d/50-android.rules

with the following content

SUBSYSTEM=="usb", SYSFS{idVendor}=="0bb4", MODE="0666"

Make sure that the text after {idVendor} is the same as the device id found when using the lsusb command. Save and exit the editor

To make execution of the rule possible, perform a chmod

sudo chmod a+rx /etc/udev/rules.d/50-android.rules

To make the rule active, restart udev

sudo /etc/init.d/udev restart

Final check

To check if the device is available use the adb tool (which is part of the Android SDK. Download and installation instructions at http://code.google.com/android)

adb devices

This will return a list of attached devices.

List of devices attached
HT845GZ51782 device

You should now be able to use the different tools of the Android SDK (and application deployment with the use of Eclipse) within Ubuntu running inside VMWare Server.

WBXML for .Net

Synchronisation of data between multiple devices is important to have information at every place at every time. For example, I want to have my work agenda items on my mobile phone so that I have a quick overview (and get a notification) when I have a meeting. 

There are different protocols for data synchronization. To synchronize my mobile agenda with work (which makes use of Oracle Collaboration Suite) I have to use SyncML. There are some commercial products available (I used the SyncML client of Synthesis AG on my Windows Mobile Device, which is very good) but I am a developer and I wanted to know how SyncML works. 

When reading the SyncML specification  I tried to initiate a fresh synchronisation (from the server to a mobile which has an empty agenda). Sending message to the Oracle server was working, but the response I got back was not correct (the SyncML messages where accepted but the agenda items where not send back). I wanted to compare those messages to the ones (commercial) SyncML applications where sending… but I found out that these where using WBXML.

What is WBXML?

WBXML stands for WAP Binary XML. It is used to send XML in a compact way to reduce bandwidth (which is limited for a mobile device when using GPRS). For example the content of the following XML (as defined in the WBXML specification as an example)

<br />
<XYZ><CARD><br />
     X &amp; Y<BR /><br />
     X=1<br />
</CARD></XYZ><br />

will be in WBXML (hex notation):

03 01 03 00 47 46 03 0D 0A 20 20 20 20 20 58 20 26 20 59 00 05 03 0D 0A 20 20 20 20 20 58 00 02 81 20 03 3D 00 02 81 20 03 31 20 0D 0A 20 20 00 01 01

Interested by the use of this encryption I started reading the WBXML specification. I searched on the Internet if there where any WBXML libraries for .Net but couldn’t find one. So i decided to write my own.

At CodeProject I found a blog written by Tamir Khason about a WBXML parser for C#. Comparing it to the WBXML specification (and also referencing to his own remarks on the blog) the parser is not following the specification. I liked the idea of extending the XMLDocument of .Net and used that as a base of the WBXMLDocument class.

The code of the WBXML .Net library can be found at http://www.codeplex.com/wbxml. I will finalize this library so that it can be used for WBXML message which are used for SyncML.

Ubuntu, Apache, SSL, Subversion and LDAP

At work we needed a new source control system and we decided to use Subversion. This blog will give details about the installation of how to use Apache (with SSL connection) to control subversion. Access to the subversion repository is done with LDAP connected to an Active Directory

Unbuntu installation

I used the ubuntu 8.0.4 desktop version. Installation with the wizard is straight forward and therefore not discussed. The hostname in this example is: jdk-svn . After installation of Ubuntu I checked for new package update (sudo apt-get update and sudo apt-get upgrade).

Installation of Apache

To install apache2 you need to install the apache2 package

sudo apt-get install apache2

Besides the package apache2 also the packages apache2-mpm-worker, apache2-utils, apache2.2-common, libapr1, libaprutil1 and libpq5 are installed

After installation you can check if the Apache webserver is running by pointing your browser to the http://jdk-svn location (from the server also http://localhost will work). When the server is running a webpage will be served with the simple textmessage It Works

Installation SSL

For authentication we don’t want to send plain text on the network. SSL need to be installed to support secure connections. SSL needs the packages openssl and ssl-cert which are installed by default with the Ubuntu 8.0.4 installation (otherwise use sudo apt-get install openssl and sudo apt-get install ssl-cert to install these packages)

A certificate is needed for SSL. I will use a self signed certificate which can be created with make-ssl-cert

Note: a lot of websites are talking about the script apache2-ssl-certificate to make a certificate. The problem is that for some reason this script is not anymore part of the apache2 package. There are ways to extract the script from other apache2 packages but I found it easier to create a certificate with make-ssl-cert

Note 2: make-ssl-cert generated a certificate which is by default expired after 30 days. After 30 days it can still be used but it will give another warning inside the browser. Does somebody know how to generate a certificate which expires after 365 days (which I though is the maximum days)?

sudo make-ssl-cert /usr/share/ssl-cert/ssleay.cnf /etc/ssl/private/localhost.pem

When using make-ssl-cert some question will be asked by the script which need to be answered for the generation of the certicate:

  • Country Name
  • State or Province Name
  • Locality Name
  • Organisation Name
  • Organisation Unit Name
  • Host name: jdk-svn
  • Emailaddress

The apache website configuration should be changed to make use of SSL and the certificate. Apache can have multiple website configuration (stored inside /etc/apache2/sites-available). A site is accessable (enabled) when it is inside the etc/apache2/sites-enabled directory. This, enabling a site, can be done with the a2ensite tool. With a2dissite, a site can be disabled.

The default website configuration (created by default serving the It works website) will be used for the new SSL configuration

cd /etc/apache2/sites-available
sudo cp default ssl

Edit the ssl configuration with your favorite texteditor (for example gedit: sudo gedit ssl). Make the following changes (port 443 is the port number used for https)

NameVirtualHost * -> NameVirtualHost *:443
<VirtualHost *> -> <VirtualHost *:443>

Add (between <VirtualHost> tags. For example directly below ServerAdmin):

SSLEngine On
SSLCertificateFile /etc/ssl/private/localhost.pem

The ssl website should be enabled. Besides enabling the website also the apache ssl module should be enabled (which can be done with the a2enmod script)

sudo a2ensite ssl
sudo a2enmod ssl

Note 3: I only want to use the server for SVN access. Therefore I want to disable the website running on port 80. Use sudo a2dissite default to disable the website at port 80.

After creating the ssl configuration and enabling the website, Apache should be restarted

sudo /etc/init.d/apache2 restart

Note 4: when (re)starting Apache I got an error that the server name could not be determined. I fixed this by adding the ServerName to the httpd.conf. Open the httpd.conf file (sudo gedit /etc/apache2/httpd.conf), this file is empty by default, and add the line ServerName jdk-svn

After restarting Apache the ssl website should be accessable by pointing your webbrowser to https://jdk-svn (or from the machine: https://localhost). The same It works! website will be served (because we did not changed what will be served inside the ssl configuration). When you disabled the default site, you will get a Not found website when requesting http://jdk-svn. By the way, the first time you will access the website which will make use of SSL, your browser will give some security exception about that the certificate cannot be verified. This is true because we created a self signed certificate. Make an exception for this website by allowing access to this website inside your browser configuration.

Installation of Subversion

Apache2 and SSL is running, it is time for Subversion. Install the subversion package:

sudo apt-get install subversion

After subversion is installed a subversion repository should be made. This repository will be accessible by Apache (SSL) with the use of LDAP for authentication and authorization. I created the subversion repository inside the /var directory

sudo mkdir /var/svn
sudo mkdir /var/svn/myrepository

The repository will be called myrepository. Because users will add and modify files inside SVN by the use of Apache, the Apache user (www-data) should be the owner of the myrepository directory. If permissions are not set correctly, you will finish with a read only repository.

First set the ownership of the directory to the www-data user

sudo chown -R www-data /var/svn/myrepository

The www-data group (of which the user www-data is member of) should be set a ownership group of this directory

sudo chgrp -R www-data /var/svn/myrepository

It is important that when users are adding files to the repository, that these files have proper permissions set. Use the following command to do that

sudo chmod -R g+rws /var/svn/myrepository

Use svnadmin to make it a Subversion repository

sudo svnadmin create /var/svn/myrepository

Because svnadmin added directories and files to the myrepository folder which do not have the right group write access, it is important to repeat the chmod command

sudo chmod -R g+rws /var/svn/myrepository

The repository is now created. Let’s integrate it with Apache. We don’t use LDAP yet to see if the integration is working.

For Apache and Subversion integration, the libapache2-svn package is needed. Install this package

sudo apt-get install libapache2-svn

Change the ssl website configuration (sudo gedit /etc/apache2/sites-available) by adding the following part between the <VirtualHost> tags. For example just below the last </Directory> tag

<Location /repos>
   DAV svn
   SVNParentPath /var/svn
   SVNListparentPath on
</Location>

The location /repos means that you can access the repositories by going to https://jdk-svn/repos from within your browser. You cahttp://www.johandekoning.nl/wp-admin/post.php?action=edit&post=84&message=4n change this if you prefer a different location. SVNParentPath /var/svn is the location where the repositories are stored on the local file system. With SVNListParentPath on you will get a website showing the different repositories when opening https://jdk-svn/repos. If you set this to off (or remove the line) you will get a Not Found error. In this situation you can access a repository by going directly to the repository location (http://jdk-svn/repos/myrepository).

Restart apache (sudo /etc/init.d/apache2 restart) to make the changes visible. When opening the url http://jdk-svn/repos inside your browser, you will get a list of repositories available (in this case only the myrepository)

LDAP integration

Apache and subversion are working together but everybody has access to it. LDAP will be used to only allow access to members of an Active Directory. By using LDAP we can centralize authorization/authentication and don’t have to configure access inside different configuration files.

For LDAP integration two apache modules should be enabled: ldap and authnz_ldap. Ldap will automatically been enabled when enabling the authnz_ldap module:

sudo a2enmod authnz_ldap

Note 5: I am using Microsoft Active Directory, which does not allow anonymous access for retrieving the members inside a Active Directory. To solve this I created a new Windows Account inside the Active Directory which does not have terminal access (cannot login on Windows Workstations). Problem with not having anonymous access (or a more limited account than disabling terminal access) the username and password should be set as plain text inside the ssl configuration. For know I think that is fine but off course I prefer a saver solution. Post your ideas by adding a comment to this blog item).

Edit the ssl configuration file (sudo gedit /var/apache2/sites-available)

...
AuthType Basic
AuthBasicProvider ldap
AuthzLDAPAuthoritative on
AuthName "jdk-svn"
AuthUserFile /dev/null
AuthLDAPURL "ldap://activedirectory.johandekoning.nl:3268/DC=johandekoning,DC=nl?sAMAccountName?sub?(objectClass=*)"
AuthLDAPBindDN "CN=<ldap_username>,CN=users,DC=johandekoning,DC=nl"
AuthLDAPBindPassword  <ldap_password>
AuthLDAPGroupAttributeIsDN on
AuthLDAPGroupAttribute member

SSLRequireSSL
Require valid-user

Make changes to this configuration file confirming your active directory installation (the AuthLDAPURL, AuthLDAPBindDN and the AUTHLDAPBindPassword)

Restart apache (sudo /etc/init.d/apache2 restart) and go to https://svn-jdk/repos with a webbrowser. A login dialog will be shown asking a username and password. Use a windows username and password of a user which is part of the active directory. When login is successfully you will see the repository list again.

Clean up apache ssl configuration

The ssl configuration created still contains configuration which is not needed when you only want to use apache for subversion integration. For example the It works website is still served when opening https://jdk-svn. Clean up the ssl configuration (sudo gedit /etc/apache2/sites-available/ssl) by removing the following parts:

DocumentRoot /var/www/
<Directory />
   Options FollowSymLinks
   AllowOverride None
</Directory>
<Directory /var/www>
   ...
</Directory>
ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
<Directory "/usr/lib/cgi-bin">
   ...
</Directory>

And remove the part

Alias /doc "/usr/share/doc/"
<Directory "/usr/share/doc/">
   ...
</Directory> 

Restart apache (sudo /etc/init.d/apache2 restart) and check that the It works site is not served anymore (https://jdk-svn will give back a Not found message)

What’s next?

You finished the installation and configuration of Apache, SSL, Subversion and LDAP integration. You can now use your favorite Subversion applications to modify the content of the repository.

The LDAP integration gives users access to Subversion which are inside the Active Directory. I am still searching for solutions to make more use of the Active Directory and LDAP mechanism. For example defining inside Active Directory the access to repositories for each user (and not inside Apache configuration files because I want to keep these settings centralized). If you have any idea how to define this within Active Directory, please post it as a comment. Besides accessing repositories you can also think about read-only access.

The SSL configuration

To summarize the changes made to the ssl configuration, the final ssl configuration file is added:

NameVirtualHost *:443
<VirtualHost *:443>
        ServerAdmin webmaster@localhost.com
        SSLEngine On
        SSLCertificateFile /etc/ssl/private/localhost.pem

        ErrorLog /var/log/apache2/error.log

        # Possible values include: debug, info, notice, warn, error, crit,
        # alert, emerg.
        LogLevel warn

        CustomLog /var/log/apache2/access.log combined
        ServerSignature On

        <Location /repos>
                DAV svn
                SVNParentPath /var/svn
                SVNListparentPath on

                AuthType Basic
                AuthBasicProvider ldap
                AuthzLDAPAuthoritative on
                AuthName "jdk-svn"
                AuthUserFile /dev/null
                AuthLDAPURL "ldap://activedirectory.johandekoning.nl:3268/DC=johandekoning,DC=nl?sAMAccountName?sub?(objectClass=*)"
                AuthLDAPBindDN "CN=<ldap_username>,CN=users,DC=johandekoning,DC=nl"
                AuthLDAPBindPassword <ldap_password>
                AuthLDAPGroupAttributeIsDN on
                AuthLDAPGroupAttribute member

                SSLRequireSSL
                Require valid-user
        </Location>
</VirtualHost>

Command-line arguments Windows Mobile

In September I will go on holiday to the United States for 3 weeks. It would be nice to send updates to family and friends by placing images on Picasa. These web images will be taken with the photocamera from my mobile (which is Windows Mobile 6 based). I will upload the pictures when a free WiFi access point is available.

The program will place taken photos in a queue. When I have an internet connection available I can upload all the pictures from this queue to Picasa. This results in an application that can be seperated in two parts: the take a photo functionality and the upload photo-queue functionality.

Use cases explaining functionality

My mobile has a photo camera button which will trigger the default photo camera application. This default camera application can be replaced by changing the button settings on the mobile phone. When replaced (and when clicking the camera button) I want my application to start up ready to take a picture (so it will behave the same as the default camera application). After taking a picture you can type in some comment and add it to the queue.

When arriving at the hotel (or some other connective place) I want to upload the taken images. I can off course push the photo camera button again, but this time it is not my intention to take a new picture. In this case I want to start the application from the Start -> Programs menu and see the state of the queue with the option to upload the photos. The same application but with a different startup state.

The use of command-line arguments should solve this problem. But is this also possible with the Compact .Net Framework

Not a solution: Environment.GetCommandLineArgs

For Windows Form based application (which my mobile application is) the use of the Environment class is suggested to get the command-line arguments. The static method

<br />
public static string[] GetCommandLineArgs()<br />

The problem is that this method is not implemented inside the .Net Compact Framework and can’t therefore not been used.

Solution: static void Main(String[] args)

Just like console application (written in .Net) you have a Main method for your Windows Forms application. Besides the files created for the Form (by default Form1.cs and Form1.Designer.cs) a Program.cs file is created which contains a Main method. By adding a String[] args to the Main method, you can work with command-line arguments. An example of the new Main method

<br />
[MTAThread]<br />
static void Main(String[] args)<br />
   {<br />
      if (args != null &#038;&#038; args.Length > 0)<br />
     {<br />
         foreach (String arg in args)<br />
         {<br />
            MessageBox.Show(arg);<br />
         }<br />
     }<br />
     else<br />
     {<br />
         MessageBox.Show("No program arguments");<br />
     }<br />
     Application.Run(new Form1());<br />
}<br />

The arguments can be given to the form (Form1 in this example) by changing the constructor.

Creating a shortcut

The easiest way to create a shortcut is by using the Windows Explorer on your mobile phone. Navigate to the executable file of the application. Tap and hold the file selected with your styles to get the context menu. Select copy.

At the same location (or on a different location) tap (and hold) on an empty space to get the context menu. This context menu will give you the option to paste as a shortcut.

To add the command line argument inside the shortcut, i copied the shortcut (.lnk file) with ActiveSync to my desktop pc and edit the content with notepad. The file will now look like this:

59#”\Programmabestanden\ProgramArguments\ProgramArguments.exe” this_is_a_program_argument

My Windows Mobile OS is Dutch. On an English based OS the Programmabestanden part will be replaced by something like Program files. I don’t know if the number 59 before the line is mandatory. The number of characters with quotes is 59 long. So maybe it is used for reading optimalization. I really don’t know. I just added the this_is_a_program_argument at the end of the shortcut, saved the file and copied it back to my pocket pc.

Starting the application with the shortcut will show a messagebox with the text this_is_a_program_argument. When starting the application from the normal executable you will get a messagebox saving No program arguments.

Attachment: example project

You can download the example application from here

Special thanks

I want to thank Paul for discussing this problem and finding a solution.

LINQ to XSD versus XSD.exe (part 1)

While LINQ to XSD is still in alpha, it is always a good idea to look at the (new) possibilities of this new technology. Before LINQ to XSD I used the XSD.exe to generate source files based on a XML schema. By generating this code you can work with XML files on a typed safe manner.

I will try to compare the differents between LINQ to XSD and the XSD tool. I will look at the (generated) code needed to work with XML files and try to compare the performances of both techniques. Based on the Linq to XSD overview document I will use the following XML schema:

<xs:schema
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
  targetNamespace="http://www.example.com/Orders"
  xmlns="http://www.example.com/Orders"
  elementFormDefault="qualified">
   <xs:element name="Batch">
      <xs:complexType>
         <xs:sequence>
           <xs:element ref="PurchaseOrder" minOccurs="0" maxOccurs="unbounded"/>
         </xs:sequence>
       </xs:complexType>
     </xs:element>
   <xs:element name="PurchaseOrder">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="CustId" type="xs:string"/>
        <xs:element ref="Item" minOccurs="0" maxOccurs="unbounded"/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
<xs:element name="Item">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="ProdId" type="xs:string"/>
        <xs:element name="Price" type="xs:double"/>
        <xs:element name="Quantity" type="xs:int"/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>

Code generation difficulties

The XML Schema placed above is different than the one I start using by default (for the PageType XML Schema of the Aincha Web Framework). Because the Purchase orders schema is not that complex I would have implemented it this way:

<xs:schema
  xmlns:xs="http://www.w3.org/2001/XMLSchema"
  targetNamespace="http://www.example.com/Orders"
  xmlns="http://www.example.com/Orders"
  elementFormDefault="qualified">
<xs:element name="Batch">
    <xs:complexType>
      <xs:sequence>
        <xs:element name="PurchaseOrder" minOccurs="0" maxOccurs="unbounded">
          <xs:complexType>
            <xs:sequence>
              <xs:element name="CustId" type="xs:string"/>
              <xs:element name="Item" minOccurs="0" maxOccurs="unbounded">
                <xs:complexType>
                  <xs:sequence>
                    <xs:element name="ProdId" type="xs:string"/>
                    <xs:element name="Price" type="xs:double"/>
                    <xs:element name="Quantity" type="xs:int"/>
                  </xs:sequence>
                </xs:complexType>
              </xs:element>
            </xs:sequence>
          </xs:complexType>
        </xs:element>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
</xs:schema>

This schema has the same definition as the previous one only the PurchaseOrder and Item are part of the complexType parts. Although this schema’s are both valid and they look the same, when using this schema with the XSD tool, different code is generated.

The first schema will generate the classes:

  • Batch
  • PurchaseOrder
  • OrderItem

While the second schema will generate classes:

  • Batch
  • BatchPurchaseOrder
  • BatchPurchaseOrderItem

Which is in my opinion not the names you would expect. If the schema is way more complex, you get even bigger names. And I know for sure that programmers don’t want to work with this unusual names.

At first, I though that this was a limitation of the XSD tool. But using the second schema within LINQ to XSD, the class names are also different and generated as private classes of the Batch object. With LINQ to XSD you get the following:

  • Batch
  • Batch.PurchaseOrderLocalType
  • Batch.PurchaseOrderLocalType.ItemLocalType

So it is important to keep it mind that different code will be generated bases on the structure of the XSD. Even when both schemas are valid.

What’s next?

In the next part of this comparison research I will investigate the following things:

  • amount of code used to work with XML data (based on the generated schema code);
  • time to load/parse the XML data (with different sizes);
  • time to print all the items inside the purchase orders to the console;
  • time to calculate the total purchase orders batch value and time to calculate the purchase order value for each customer;
  • CPU and Memory consumptions;

If you have other suggestions? Don’t hesitate to post them as a comment to this blog item.

Searching for a webbased diagram editor

I really like the Google Docs application. Working together with a team on a spreadsheet is really useful for status overviews. Love to see that this is working without browser plugins. I was really surprised to see the direct updates when a colleague was editing a cell of the shared spreadsheet.

Working together with web based collaboration tools makes it easy to share information directly to each other. At work we use Drupal to share information. Adding new content is easy and you can do it anywhere you want without installing software. Only an Internet connection is required.

When it comes to software development, I miss the functionality of creating diagrams within the web based collaboration tools. For example the steps I have to take to show an UML diagram within the webpage document:

  1. Create an UML diagram with an external application (Enterprise Architecture for example)
  2. Copy and paste the content of the diagram to an application which can save it as an image (Photoshop or MSPaint)
  3. Save the image
  4. Within the webbased collaboration tool use the browser upload functionality.
  5. Finally add the image to the document.

What if i made a mistake or want to update these diagrams? Or what if a colleague wants to update them?

It should be possible to make such an editor. UML can for example be converted to XMI which makes it possible to use for a webbased diagram editor to work with. I tried searching for such a web based tool but at this moment could not find anything useful. Do they exist? Or should i try to create a proof of concept to see if it is possible?

BTW: besides creating diagrams online, I also want to see the functionality of pasting an image within the webdocument. Clipboard functionality is limited inside a browser (because of security reasons). Would this be possible by the use on a Firefox plugin? The website should have some (AJAX) method implemented which accept a BASE64 encoded string containing the image data. This image data was stored on the clipboard. Don’t see any security risks because the user triggers the paste operation and websites only implements these methods on parts for which the user is logged in. So another research topic: is a FireFox plugin capable of making access to the clipboard?