Debug a Native Application for Android

Bill Cox wrote the procedure how to debug his statically linked application by gdb. In this article, I show the same procedure can be applied to dynamically linked executable and shared library. At last I have some consideration to the warnings which gdb shows.

Notice it is impossible to debug multi-threaded application for Android with this gdbserver. It is because the gdbserver requires NPTL on linux target to debug multi-threaded application. Porting gdbserver to Android may be possible with source code of the libc. Unfortunately I have no idea how to debug multi-threaded application for Android right now.

I assume that you know how to make dynamically linked executable and shared library for Android. If not, please read first my previous articles here and here.

In the following procedure, $ denotes the prompt of host and # denotes the prompt of emulator.

---

1. Download gdbserver from here and send it to the emulator.

$ adb push gdbserver /data/tmp

2. Prepare Android system image and hello world programs.

I placed three directories such like the following.

$ ls tmp/android.hello
system
hello1
hello2

The system directory is the android /system directory which is copied from the emulator. At least, there must be bin/linker and lib/libc.so under this directory in order to compile and debug hello world programs. If you do not know how to get the system image of android, refer to the previous articles. The hello1 directory contains the sources of dynamically linked type of program and the hello2 directory contains the sources of shared library type of program.

3. Compile programs with -g option.

$ cd hello1
$ arm-none-linux-gnueabi-gcc -c -fno-builtin-printf -g  -o start.o start.c
$ arm-none-linux-gnueabi-gcc -c -fno-builtin-printf -g  -o hello.o hello.c
$ arm-none-linux-gnueabi-ld --dynamic-linker /system/bin/linker -nostdlib\
 -rpath /system/lib -rpath ~/tmp/android.hello/system/lib\
 -L~/tmp/android.hello/system/lib\
 -lc -o hello1 start.o hello.o

$ cd hello2
$ arm-none-linux-gnueabi-gcc -c -g  -o start.o start.c
$ arm-none-linux-gnueabi-gcc -c -g  -o main.o main.c
$ arm-none-linux-gnueabi-gcc -c -fpic -g  -fno-builtin-printf\
 -o hello.o hello.c
$ arm-none-linux-gnueabi-ld -shared  -T armelf_linux_eabi.xsc\
 -o libhello.so hello.o
$ arm-none-linux-gnueabi-ld --dynamic-linker /system/bin/linker -nostdlib\
 -rpath /system/lib -rpath ~/tmp/android.hello/system/lib\
 -L . -L~/tmp/android.hello/system/lib\ 
 -lc -lhello -o hello2 start.o main.o libhello.so

4. Send executables and shared libraries to the emulator.

$ adb hello1 /data/tmp
$ adb hello2 /data/tmp
$ adb libhello.so /data/tmp

You may strip the debug info from the binaries which are placed on the emulator. Gdb will get the debug info from the binaries on the host.

5. Connect to the emulator's console by telnet and redirect the debug port as you like.

$ telnet localhost 5554
 redir add tcp:1234:1234
 exit

6 Run gdbserver on the emulator.

# cd /data/tmp
# ./gdbserver 10.0.2.2:1234 hello1

The same procedure can be applied to hello2. Note that 10.0.2.2 is the ip address of the host from the view point of emulator.

You can attach to the running program such like the following.

# ./gdbserver 10.0.2.2:1234 <pid>

See 11. below, for the example.

7. Run gdb on the host and connect to the target.

$cd hello1
 $arm-none-linux-gnueabi-gdb hello1
 (gdb) set sysroot ../
 (gdb) set solib-search-path ../system/lib
 (gdb) target remote localhost:1234

'set sysroot' set the root directory of the taget image. 'set ../system/lib' set the search path of shared libraries. See 'info gdb' for the detail. The same procedure can be applied to hello2.

8. Start debug as you need.

For example, set breakpoints such like in the following.

In hello1 case,
 (gdb) b main
In hello2 case,
 (gdb) b hello

9. Result will be such like the following.

In hello1 case,
 (gdb) target remote localhost:1234
 Remote debugging using localhost:1234
 warning: Unable to find dynamic linker breakpoint function.
 GDB will be unable to debug shared library initializers
 and track explicitly loaded dynamic code.
 0xb0000100 in ?? () from ../system/bin/linker
 (gdb) b main
 Breakpoint 1 at 0x8344: file hello.c, line 5.
 (gdb) c
 Continuing.
 warning: .dynamic section for "/home/motz/tmp/android.hello/system/lib/libc.so" is not at 
 the expected address (wrong library or version mismatch?)

 Breakpoint 1, main (argc=33540, argv=0x0) at hello.c:5
 5               printf("Hello, world!\n");
 (gdb) c
 Continuing.

 Program exited normally.

In hello2 case,
 (gdb) target remote localhost:1234
 Remote debugging using localhost:1234
 warning: Unable to find dynamic linker breakpoint function.
 GDB will be unable to debug shared library initializers
 and track explicitly loaded dynamic code.
 0xb0000100 in ?? () from ../system/bin/linker
 (gdb) b hello
 Breakpoint 1 at 0x82f8
 (gdb) c
 Continuing.
 warning: .dynamic section for "/home/motz/tmp/android.hello/system/lib/libc.so" is 
 not at the expected address (wrong library or version mismatch?)
 warning: .dynamic section for "libhello.so" is not at the 
 expected address (wrong library or version mismatch?)

 Breakpoint 1, hello (name=0x835c "World!") at hello.c:5
 5               printf("Hello %s!\n", name);
 (gdb) b printf
 Breakpoint 2 at 0xafe0f754
 (gdb) c
 Continuing.

 Breakpoint 2, 0xafe0f754 in printf ()
    from /home/kurotsu/tmp/android.hello/system/lib/libc.so
 (gdb) c
 Continuing.

 Program exited normally.

10. Several points to note.

1) "warning: Unable to find dynamic linker breakpoint function. GDB will be unable to debug shared library initializers and track explicitly loaded dynamic code."

This means that the android linker doesn't provide the breakpoint function which gdb assumes to exist. Therefore it is impossible for gdb to set a breakpoint in the linker. Since the android linker is proprietary, gdb doesn't know the internal.

2) "warning: .dynamic section for "xxx/libc.so" is not at the expected address ..."

The cause of this warning is that libc.so is prelinked. The prelink used againt libc.so is proprietry. It breaks the gdb's assumption of relocation.

3) "warning: .dynamic section for "libhello.so" is not at the expected address ..."

The cause of this warning is that the page align between .text and .data of this library is turned off. It is a fake to deceive the android linker. See the previous artical for the detail.

11. Try to debug system executables/libraries.

Note the /system/bin/runtime is multi-threaded, it is impossible to debug with this gdbserver. Please see below as the example how to attach the running process.

# ps | grep runtime
root     455   1     18528 2068  ffffffff afe0861c S /system/bin/runtime
# gdbserver 10.0.2.2:1234 --attach 455
$ cd system
$ telnet localhost:1234 bin/runtime
$ arm-none-linux-gnueabi-gdb bin/runtime
(gdb) set sysroot ../
(gdb) set solib-search-path ./lib
(gdb) target remote localhost:1234
...
bunch of warnings here
...
(gdb) bt
#0  0xafe0861c in __ioctl ()
   from /home/motz/tmp/android.hello/system/lib/libc.so
#1  0xafe208bc in ioctl ()
   from /home/motz/tmp/android.hello/system/lib/libc.so
#2  0xa9d2b0d6 in android::IPCThreadState::talkWithDriver ()
   from /home/motz/tmp/android.hello/system/lib/libutils.so
#3  0x000151c8 in ?? ()
(gdb) signal SIGKILL
Continuing with signal SIGKILL.

Program terminated with signal SIGKILL, Killed.
The program no longer exists.

That's it.

Initialize libc for Android.

This article describes how to initialize the libc for Android. At the previous article, the initialization of libc is unnecessary because the "Hello, world!" program is so simple. Normally the initialization of libc must be done before jumping to the main function.

I would like to suggest you to read the previous article at first, if you have not yet.

---

1. Prepare initialization routine

crt0.s:
                .text
                .global _start
        _start:
                mov     r0, sp
                mov     r1, #0
                add     r2, pc, #4
                add     r3, pc, #4
                b       __libc_init
                b       main
                .word   __preinit_array_start
                .word   __init_array_start
                .word   __fini_array_start
                .word   __ctors_start
                .word   0
                .word   0

                .section .preinit_array
        __preinit_array_start:
                .word   0xffffffff
                .word   0x00000000

                .section .init_array
        __init_array_start:
                .word   0xffffffff
                .word   0x00000000

                .section .fini_array
        __fini_array_start:
                .word   0xffffffff
                .word   0x00000000

                .section .ctors
        __ctors_start:
                .word   0xffffffff
                .word   0x00000000

This part is the entry of the program. Actually, this part is common to all executables provided by Android. Except some modification, the source above have been gotten by disassembling one of executables in Android.

You see that this routine has two branches. By the first one, it jumps to _libc_init function which is provided in libc.so. By the second one, it jumps to the main function. Comparatively speaking, by the second branch, the executables provided by Android jump not to main but to the stub for arm-to-thumb interwork. In this case, we do not use thumb mode, so we directly jump to main. Note that the __libc_init is so special. It requires the four addresses at the specific position. These are the top address of the sections; .preinit_array, .init_arrays, .fini_arrays, .ctors in the order.

2. Prepare test program

hello.c:

        #include <stdio.h>
        #include <unistd.h>
        
        int main(int argc, char **argv, char**env) {
                int i;
        
                for (i = 0; i < argc; i++) {
                        printf("argv[%d] = %s\n", i, argv[i]);
                }
                for (i = 0; env[i] != NULL; i++) {
                        printf("env[%d] = %s\n", i, env[i]);
                }
                printf("PATH = %s\n", getenv("PATH"));
                setenv("HELLO", "world!", 1);
                putenv("GOODBYE=cruel world!");
                printf("HELLO = %s\n", getenv("HELLO"));
                printf("GOODBYE = %s\n", getenv("GOODBYE"));
        
                return 0;
        }

We try to access to arguments and environment variables, now.

3. Compile and Run.

$ arm-none-linux-gnueabi-gcc -c -fno-builtin  -o crt0.o crt0.s
$ arm-none-linux-gnueabi-gcc -c -fno-builtin  -o hello.o hello.c
$ arm-none-linux-gnueabi-ld \
  --dynamic-linker /system/bin/linker -nostdlib \
  -rpath /system/lib -rpath ~/tmp/android/system/lib \
  -L ~/tmp/android/system/lib  -lc -o hello3 crt0.o hello.o
$ adb push hello3 /data/hello
# ./hello3
argv[0] = ./hello3
env[0] = ANDROID_ROOT=/system
env[1] = PWD=/data/hello
env[2] = LD_LIBRARY_PATH=/system/lib
env[3] = PATH=/sbin:/system/sbin:/system/bin:/data/busybox
env[4] = DRM_CONTENT=/data/drm/content
env[5] = ANDROID_BOOTLOGO=1
env[6] = ANDROID_ASSETS=/system/app
env[7] = EXTERNAL_STORAGE=/sdcard
env[8] = ANDROID_DATA=/data
PATH = /sbin:/system/sbin:/system/bin:/data/busybox
HELLO = world!
GOODBYE = cruel world!

That's it.

Shared library "Hello World!" for Android

This article describes the trace I followed when I managed to compile and run the shared library type of "Hello, world!" for Android. I confirmed this procedure only on my host (Fedora 8). Sorry for Windows and Mac users.

I would like to appreciate developers who posted useful messages on Android Developers and Android Internal news groups. Especially I would like to thank Benno who provided busybox and strace binary which work on Android.

I would like to suggest you to read the "Native C applications for Android" written by Benno and my previous article before reading this one.

The following description assumes that you have toolchains, strace, busybox and system image of Android in the hand. Otherwise, read my previous article at first.

I hope this article is useful for all Android developers.

---

1. Prepare source files.

hello.h:
       extern void hello(const char* name);

hello.c:
       #include <stdio.h>
      
       void hello(const char* name)
       {
               printf("Hello %s!\n", name);
       }

start.c:
       #include <stdlib.h>
      
       extern int main(int argc, char **argv);
      
       void _start(int argc, char **argv)
       {
               exit (main (argc, argv));
       }

main.c:
       #include "hello.h"
      
       int main(void)
       {
               hello("World!");
               return 0;
       }

2. Make shared library.

$ arm-none-linux-gnueabi-gcc -fpic -c hello.c
$ arm-none-linux-gnueabi-ld -shared -o libhello.so hello.o

3. Make dynamically linked executable.

 $ arm-none-linux-gnueabi-gcc -c start.c
$ arm-none-linux-gnueabi-gcc -c main.c
$ arm-none-linux-gnueabi-ld \
  --dynamic-linker /system/bin/linker -nostdlib \
  -rpath /system/lib -rpath ~/tmp/android/system/lib \
  -L . -L ~/tmp/android/system/lib -lc -lhello -o hello2 start.o main.o

4. Copy the shared library and the executable to Android. The following description assumes that there is /data/hello directory on Android.
The '#' is the prompt of Android shell.

 $ adb push libhello.so /data/hello
$ adb push hello2 /data/hello
# ./hello2
WARNING: `libhello.so` is not a prelinked library
[2]   Segmentation fault      ./hello2

5. What happened? Anyway let's take a look at the output of strace.

    11 open("libc.so", O_RDONLY|O_LARGEFILE)   = -1 ENOENT (No such file or directory)
   12 open("/system/lib/libc.so", O_RDONLY|O_LARGEFILE) = 3
   13 lseek(3, -8, SEEK_END)                  = 231908
   14 read(3, "\0\0\340\257PRE ", 8)          = 8
   15 mmap2(0xafe00000, 233472, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xafe00000
   16 close(3)                                = 0
   17 mmap2(0xafe39000, 45056, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, 0, 0) = 0xafe39000
   18 mprotect(0xafe00000, 221184, PROT_READ|PROT_EXEC) = 0
   19 open("libhello.so", O_RDONLY|O_LARGEFILE) = 3
   20 lseek(3, -8, SEEK_END)                  = 2344
   21 read(3, "a_start\0", 8)                 = 8
   22 fstat64(1, {st_mode=S_IFREG|0666, st_size=1432, ...}) = 0
   23 brk(0)                                  = 0x11000
   24 brk(0x11000)                            = 0x11000
   25 brk(0x13000)                            = 0x13000
   26 write(1, "WARNING: `libhello.so` is not a "..., 50WARNING: `libhello.so` is not a prelinked library
   27 ) = 50
   28 lseek(3, 0, SEEK_END)                   = 2352
   29 mmap2(0x80100000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x80100000
   30 close(3)                                = 0
   31 --- SIGSEGV (Segmentation fault) @ 0 (801082bc) ---

At the line 12-14, linker opens libc.so and reads the 8 bytes at the end of the file. It is "\0\0\340\257PRE ". It is interesting that this linker checks the fixed pattern at the end of the library. Then at the line 15, linker mmaps the library and at line 17, drops writable flag. On the other hand, at line 19, linker opens libhello.so. At line 20 and 21, linker checks 8 bytes at the end of the file. It is "a_start\0" and the linker warns "`libhello.so` is not a prelinked library". What is prelinked library? How to make it? Some developers have argued this issue. A developer suggested it is just a warning. So I guessed there may be another problem which causes SIGSEGV rather than "not a prelinked library" issue.

By the way, I found a bug in the source of strace 4.5.15 on which Benno's strace is based. The bug hides the address of segmentation fault. The patch and patched strace binary is available.

Now, you see the segmentaion fault occurs at 0x801082bc at the line 31 on the list above. See the line 29. The fault address is beyond the mmaped area. Let's see the program header of libhello.so by readelf. The fault address is the top of dynamic section.

 Elf file type is DYN (Shared object file)
Entry point 0x274
There are 4 program headers, starting at offset 52

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  LOAD           0x000000 0x00000000 0x00000000 0x002bc 0x002bc R E 0x8000
  LOAD           0x0002bc 0x000082bc 0x000082bc 0x00088 0x00088 RW  0x8000
  DYNAMIC        0x0002bc 0x000082bc 0x000082bc 0x00078 0x00078 RW  0x4
  GNU_STACK      0x000000 0x00000000 0x00000000 0x00000 0x00000 RW  0x4

 Section to Segment mapping:
  Segment Sections...
   00     .hash .dynsym .dynstr .rel.plt .plt .text .rodata
   01     .dynamic .got
   02     .dynamic
   03     

Uh... The linker does not mmap the second segment. How about standard shared libraries in Android, for example, libc.so?

 Elf file type is DYN (Shared object file)
Entry point 0x92d0
There are 3 program headers, starting at offset 52

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  EXIDX          0x035df8 0x00035df8 0x00035df8 0x00110 0x00110 R   0x4
  LOAD           0x000000 0x00000000 0x00000000 0x382a0 0x43978 RWE 0x8000
  DYNAMIC        0x036de8 0x00036de8 0x00036de8 0x000a8 0x000a8 RW  0x4

 Section to Segment mapping:
  Segment Sections...
   00     .ARM.exidx
   01     .hash .dynsym .dynstr .rel.dyn .rel.plt .plt .text .rodata
.ARM.exidx .data.rel.ro .dynamic .got .data .bss
   02     .dynamic 

Look, there is only one segment with LOAD type. In another word, the text and data sections are not page aligned and the combined segment has read/write/exec flag.

Now, the default linker script need to be modified. The default linker script is available at $toolchains_home/arm-none-linux-gnueabi/lib/ldscripts/armelf_linux_eabi.xsc. Copy it to the current directory. Comment out three lines and add one line replacing the first commented out line.

/*  . = ALIGN (CONSTANT (MAXPAGESIZE)) - ((CONSTANT (MAXPAGESIZE) - .) & (CONSTANT (MAXPAGESIZE) - 1)); . = DATA_SEGMENT_ALIGN (CONSTANT (MAXPAGESIZE), CONSTANT (COMMONPAGESIZE)); */
 . = .;    /* this line replaces the line above */

/*  . = DATA_SEGMENT_RELRO_END (0, .); */

/*  . = DATA_SEGMENT_END (.); */

The patch is available, either.

6. Rebuild shared library with -T option.

 $ arm-none-linux-gnueabi-ld \
  -T armelf_linux_eabi.xsc \
  -shared -o libhello.so hello.o

Check new library and confirm the data and text sections are in the same segment with read/write/exec flag.

 Elf file type is DYN (Shared object file)
Entry point 0x254
There are 3 program headers, starting at offset 52

Program Headers:
  Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
  LOAD           0x000000 0x00000000 0x00000000 0x00324 0x00324 RWE 0x8000
  DYNAMIC        0x00029c 0x0000029c 0x0000029c 0x00078 0x00078 RW  0x4
  GNU_STACK      0x000000 0x00000000 0x00000000 0x00000 0x00000 RW  0x4

 Section to Segment mapping:
  Segment Sections...
   00     .hash .dynsym .dynstr .rel.plt .plt .text .rodata .dynamic .got
   01     .dynamic
   02     

7. Rebuild executable, send library and executable to Android and run it.

 $ arm-none-linux-gnueabi-ld \
  --dynamic-linker /system/bin/linker -nostdlib \
  -rpath /system/lib -rpath ~/tmp/android/system/lib \
  -L . -L ~/tmp/android/system/lib -lc -lhello -o hello2 start.o main.o
$ adb push libhello.so /data/hello
$ adb push hello2 /data/hello
# ./hello2
WARNING: `libhello.so` is not a prelinked library
Hello World!!

There is still the warning but the program prints "Hello World!".

8. Check the output of strace.

 # strace ./hello2
   11 open("libc.so", O_RDONLY|O_LARGEFILE)   = -1 ENOENT (No such file or directory)
   12 open("/system/lib/libc.so", O_RDONLY|O_LARGEFILE) = 3
   13 lseek(3, -8, SEEK_END)                  = 231908
   14 read(3, "\0\0\340\257PRE ", 8)          = 8
   15 mmap2(0xafe00000, 233472, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0xafe00000
   16 close(3)                                = 0
   17 mmap2(0xafe39000, 45056, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, 0, 0) = 0xafe39000
   18 mprotect(0xafe00000, 221184, PROT_READ|PROT_EXEC) = 0
   19 open("libhello.so", O_RDONLY|O_LARGEFILE) = 3
   20 lseek(3, -8, SEEK_END)                  = 2312
   21 read(3, "a_start\0", 8)                 = 8
   22 fstat64(1, {st_mode=S_IFREG|0666, st_size=1432, ...}) = 0
   23 brk(0)                                  = 0x11000
   24 brk(0x11000)                            = 0x11000
   25 brk(0x13000)                            = 0x13000
   26 write(1, "WARNING: `libhello.so` is not a "..., 50WARNING: `libhello.so` is not a prelinked library
   27 ) = 50
   28 lseek(3, 0, SEEK_END)                   = 2320
   29 mmap2(0x80100000, 4096, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE, 3, 0) = 0x80100000
   30 close(3)                                = 0
   31 mprotect(0, 0, PROT_READ|PROT_EXEC)     = 0
   32 fstat64(1, {st_mode=S_IFREG|0666, st_size=1970, ...}) = 0
   33 brk(0)                                  = 0x13000
   34 brk(0x13000)                            = 0x13000
   35 brk(0x15000)                            = 0x15000
   36 exit_group(0)                           = ?
   37 Process 6028 detached

You see the program ends in success at the line 36.

9. To see memory map, modify main.c.

 #include <stdio.h>
#include <unistd.h>

int main(int argc, char **argv) {
        hello("World!");
        for(;;) sleep(1);      // Zzz...
        return 0;
}

This program sleeps until killed.

 # ./hello2 &
# WARNING: `libhello.so` is not a prelinked library
Hello World!!

# ps | grep hello2
root     5957  450   484   228   c004bc44 afe08b9c S ./hello2
# cat /proc/5957/maps
00008000-00009000 r-xp 00000000 1f:01 512        /data/hello/hello2
00010000-00011000 rw-p 00000000 1f:01 512        /data/hello/hello2
00011000-00015000 rwxp 00011000 00:00 0          [heap]
80100000-80101000 rwxp 00000000 1f:01 728        /data/hello/libhello.so
afe00000-afe36000 r-xp 00000000 1f:00 365        /system/lib/libc.so
afe36000-afe39000 rwxp 00036000 1f:00 365        /system/lib/libc.so
afe39000-afe44000 rw-p afe39000 00:00 0
b0000000-b0014000 rwxp 00000000 1f:00 272        /system/bin/linker
b0014000-b0019000 rwxp b0014000 00:00 0
be7f7000-be80c000 rw-p befeb000 00:00 0          [stack]

# kill 5957

The interesting point is that the libc.so is mapped to two areas, one is
read/exec and another is read/write/exec. On the other hand, libhello.so is
mapped to one area with read/write/exec flag. Further research is needed.

Dynamically linked "Hello, world!" for Android

This article describes the step-by-step procedure to compile and run the dynamically linked "Hello, world!" for Android. I confirmed this procedure only on my host (Fedora 8). Sorry for Windows and Mac users.

I would like to appreciate developers who posted useful messages on Android Developers and Android Internal news groups. Especially I would like to thank to Benno who provided busybox and strace binary which work on Android.

I would like to suggest you to read the "Native C applications for Android" written by Benno before reading this article.

I hope this article is useful for all Android developers.

---

1. Get toolchains from CodeSourcery(Choose ARM GNU/Linux and IA32 GNU/Linux). The following description assumes that the bin directory of toolchains is added to your PATH.

2. Get strace, busybox and system image of Android from Benno's site. Copy strace and busybox to the runnning emulator. Deploy the system image on your disk. The following description assumes the system image is deployed under ~/tmp/android.

3. Prepare source files.

hello.c:

        #include <stdio.h>
        
        int main(int argc, char **argv) {
                printf("Hello, world!\n");
                return 0;
        }

start.c:

        #include <stdlib.h>
        extern int main(int argc, char **argv);
        
        void _start(int argc, char **argv)
        {
                exit (main (argc, argv));
        }

4. Compile these source files.

$ arm-none-linux-gnueabi-gcc -c hello.c
$ arm-none-linux-gnueabi-gcc -c start.c
$ arm-none-linux-gnueabi-ld \
    --entry=_start \
    --dynamic-linker /system/bin/linker -nostdlib \
    -rpath /system/lib -rpath ~/tmp/android/system/lib \
    -L ~/tmp/android/system/lib -lc -o hello hello.o start.o

The --entry option gives the entry point of program. You can omit this option in this case because the default is _start. The --dynamic-linker option gives the place of dynamic linker. Give it from the viewpoint of the running program on Android. The -nostdlib option prevents ld from searching other library paths rather than the ones given on the command line. The first -rpath gives the place where dynamic linker search libraries. Give it from the viewpoint of the running program on Android. The second -rpath gives the place where ld search libraries needed by the library which has already loaded. Give it from the viewpoint of ld on your host.The -L option the place where ld searches the library which is given by -l option at compile time.

5. Copy binary to Android. The following description assumes that there is /data/hello directory on Android. The '#' is the prompt of Android shell.

$ adb push hello /data/hello/
# cd /data/hello
# ./hello Hello, world!

6. Check by strace whether the program ends in success.

# strace ./hello
execve("./hello", ["./hello"], [/* 9 vars */]) = 0
getpid() = 1803
syscall_983045(0xb0016b48, 0xb0013760, 0x3e4, 0, 0xbea0ae48, 0x1, 0, 0xf0005,
0xb0013760, 0, 0, 0xbea0ae44, 0, 0xbea0adf8, 0xb0000d89, 0xb00016ec, 0x10,
0xb0016b48, 0, 0, 0, 0xeb48, 0xcd1f8, 0xcd1d0, 0, 0, 0, 0, 0, 0, 0, 0) = 0
gettid() = 1803
sigaction(SIGILL, {0xb0001469, [], SA_RESTART}, {SIG_DFL}, 0) = 0
sigaction(SIGABRT, {0xb0001469, [], SA_RESTART}, {SIG_DFL}, 0) = 0
sigaction(SIGBUS, {0xb0001469, [], SA_RESTART}, {SIG_DFL}, 0) = 0
sigaction(SIGFPE, {0xb0001469, [], SA_RESTART}, {SIG_DFL}, 0) = 0
sigaction(SIGSEGV, {0xb0001469, [], SA_RESTART}, {SIG_DFL}, 0) = 0
sigaction(SIGSTKFLT, {0xb0001469, [], SA_RESTART}, {SIG_DFL}, 0) = 0
open("libc.so", O_RDONLY|O_LARGEFILE) = -1 ENOENT (No such file or
directory)
open("/system/lib/libc.so", O_RDONLY|O_LARGEFILE) = 3
lseek(3, -8, SEEK_END) = 231908
read(3, "\0\0\340\257PRE ", 8) = 8
mmap2(0xafe00000, 233472, PROT_READ|PROT_WRITE|PROT_EXEC, MAP_PRIVATE, 3, 0) =
0xafe00000
close(3) = 0
mmap2(0xafe39000, 45056, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, 0,
0) = 0xafe39000
mprotect(0xafe00000, 221184, PROT_READ|PROT_EXEC) = 0
mprotect(0, 0, PROT_READ|PROT_EXEC) = 0
fstat64(1, {st_mode=S_IFREG|0666, st_size=1337, ...}) = 0
brk(0) = 0x11000
brk(0x11000) = 0x11000
brk(0x13000) = 0x13000
exit_group(0) = ?
Process 1803 detached

You see exit_group(0) at the end. The program ended in success.

7. To see memory map, modify hello.c.

        #include <stdio.h>
        #include <unistd.h>
        
        int main(int argc, char **argv) {
                printf("Hello, world!\n");
                for(;;) sleep(1);        // Zzz...
                return 0;
        }

This program sleeps until killed.

# ./hello &
# Hello, world!

# ps | grep hello
root 1944 450 472 184 00000000 00008318 R ./hello
# cat /proc/1944/maps
00008000-00009000 r-xp 00000000 1f:01 775 /data/hello/hello
00010000-00011000 rw-p 00000000 1f:01 775 /data/hello/hello
00011000-00013000 rwxp 00011000 00:00 0 [heap]
afe00000-afe36000 r-xp 00000000 1f:00 365 /system/lib/libc.so
afe36000-afe39000 rwxp 00036000 1f:00 365 /system/lib/libc.so
afe39000-afe44000 rw-p afe39000 00:00 0 
b0000000-b0014000 rwxp 00000000 1f:00 272 /system/bin/linker
b0014000-b0019000 rwxp b0014000 00:00 0 
bea6f000-bea84000 rw-p befeb000 00:00 0 [stack]

# kill 1944

Compile Android kernel from the source

I confirmed the following procedure only on my linux host (FC8). Sorry for Windows and Mac users.

1. Get toolchains from CodeSourcery (Choose ARM GNU/Linux and IA32 GNU/Linux).

2. Get kernl source code from here.

3. Deploy toolcains and kernel source and enter in the kernel source tree.

$ mkdir -p android
$ cd android
$ tar xjvf ../arm-2007q3-51-arm-none-linux-gnueabi-i686-pc-linux-gnu.tar.bz2
$ tar xzvf ../linux-2.6.23-android-m3-rc20.tar.gz
$ cd kernel

4. Get the config.gz from the running emulator by the following way.

$ adb pull /proc/config.gz .

5. Decommpress the config.gz and rename it as .config

$ gunzip config.gz
$ mv config .config

6. Specify CROSS_COMPILE in the Makefile.

CROSS_COMPILE=../arm-2007q3/bin/arm-none-linux-gnueabi-

7. Comment out LDFLAGS_BUILD_ID^[*1] in the same Makefile.

#LDFLAGS_BUILD_ID = $(patsubst -Wl$(comma)%,%,\
$(call ld-option, -Wl$(comma)--build-id,))

8. Make the kernel

$ make

9. Check zImage is created and the size is nearly same as the kernel-qemu in the Andorid SDK.

$ ls -l arch/arm/boot/zImage
-rwxrwxr-x 1 motz motz 1234712 2007-12-01 18:06 arch/arm/boot/zImage
$ ls -l $SDK/tools/lib/images/kernel-qemu
-rwxrwxr-x 1 motz motz 1245836 2007-11-12 5:59 ...sdk/tools/lib/images/kernel-qemu

10. Run emulator with the created kernel.

$ emulator -kernel arch/arm/boot/zImage

---

[*1]: The LDFLAGS_BUILD_ID enables --build-id option of ld if available. The --build-id option is relatively new. The current emulator does not seem to support the kernel linked with this option. See here if you are interested in the detail of this option.