Google ProtoBuff + ZeroMq -- Python

In our last post, we combined ZeroMq & Google Protobuf, a combination that enables heterogeneous distributed computing systems.  The last post integrated the two using C++, this post will focus on integrating the two using Python.  In the end, we'll have an example that can communicate with the previous C++ example seamlessly.

Below is a Makefile, used to create and clean up Google Protobuf message files.

$ cat Makefile

msgs:

 ${SH} protoc -I=. --python_out=. Messages.proto



clean:

 ${RM} Messages_pb2.py

Our sender script is below, notice the change is that of the content exchanged, specifically the serialized protobuff message.

$ cat sender

#!/usr/bin/python



import zmq;

import time;

import Messages_pb2



context = zmq.Context();

pub=context.socket(zmq.PUB);

pub.bind("tcp://127.0.0.1:8000");



for i in range(0,10):



  p=Messages_pb2.Person();

  p.id=i;

  p.name="fatslowkid";



  print "iteration",i

  pub.send(p.SerializeToString());

  time.sleep(1);

The receiver;

$ cat receiver

#!/usr/bin/python



import zmq;

import time;

import Messages_pb2



context = zmq.Context();

sub=context.socket(zmq.SUB);

sub.connect("tcp://127.0.0.1:8000");

filter=""

sub.setsockopt(zmq.SUBSCRIBE, filter);





for i in range(0,20):

  print "waiting on msg"

  M=sub.recv();

  p=Messages_pb2.Person();

  p.ParseFromString(M);

  print "received",p

  print "> " + p.name;

  print p.id;

The sender/receiver can be used together, or used with the previous C++ example.

Cheers.

Google ProtoBuff + ZeroMq -- C++

In the last series of posts we demonstrated ZeroMq as a technology that supports 'sockets on steroids', supporting multiple platforms as well as multiple languages.  The examples to-date have been transmitting strings between senders and receivers.  While interesting, to effectively create a distributed heterogeneous system we need to be capable of transmitting meaningful messages, preferably complex data structures rather than just strings.  That's where Google's Protobuff comes into play: http://code.google.com/p/protobuf/

Building off our previously created Ubuntu 12.04 32-bit VM, let's start by installing the additional necessary packages;

$ sudo apt-get install libprotoc-dev

With the developer libraries installed, we can now extend our previous C++ example to transmit a ProtoBuff message.

We'll extend our Makefile to add the necessary libraries and a target (e.g. msgs) to generate the C++ files for the message.

$ cat Makefile 

CC=g++



SRCS=main.cpp Messages.pb.cc

OBJS=$(subst .cpp,.o,$(SRCS))

INCLUDES += -I.

LIBS += -lpthread -lrt -lzmq -lprotobuf



.cpp.o:

 $(CC) -c $<



main: msgs ${OBJS} 

 ${CC} ${CFLAGS} -o $@ ${OBJS} ${LIBS}



msgs:

 ${SH} protoc -I. --cpp_out=. Messages.proto



clean:

 ${RM} ${OBJS} main *.pb.*

Oh, we should take a look at our simple Protobuff message file:

$ cat Messages.proto 

message Person {

  required int32 id=1;

  required string name=2;

}

Finally, our extended main file:

$ cat main.cpp 

#include 

#include 

#include 

#include 

#include 

#include 

#include "Messages.pb.h"



void* ctx=zmq_init(1);

char* EndPoint="tcp://127.0.0.1:8000";

static const int N=100;

static const int BufferSize=128;



void* sender(void*)

{

  printf("(%s:%d) running\n",__FILE__,__LINE__);

  void* pub=zmq_socket(ctx, ZMQ_PUB);

  assert(pub);

  int rc=zmq_bind(pub,EndPoint);

  assert(rc==0);

  Person p;

  p.set_name("fatslowkid");

  p.set_id(01);

  for(int i=0; i


  {

    zmq_msg_t msg;

    std::string S=p.SerializeAsString();

    char* content=(char*)S.c_str();

    int rc=zmq_msg_init_size(&msg, BufferSize);

    assert(rc==0);

    rc=zmq_msg_init_data(&msg, content, strlen(content), 0,0);

    assert(rc==0);

    rc=zmq_send(pub, &msg, 0);

    assert(rc==0);

    ::usleep(100000);

  }

}



void* receiver(void*)

{

  printf("(%s:%d) running\n",__FILE__,__LINE__);

  void* sub=zmq_socket(ctx, ZMQ_SUB);

  assert(sub);

  int rc=zmq_connect(sub,EndPoint);

  assert(rc==0);

  char* filter="";

  rc=zmq_setsockopt(sub, ZMQ_SUBSCRIBE, filter, strlen(filter));

  assert(rc==0);

  for(int i=0; i


  {

    zmq_msg_t msg;

    zmq_msg_init_size(&msg, BufferSize);

    const int rc=zmq_recv (sub, &msg, 0);

    char* content=(char*)zmq_msg_data(&msg);

    Person p;

    p.ParseFromString(content);

    printf("(%s:%d) received: '%s'\n",__FILE__,__LINE__,p.name().c_str());

    zmq_msg_close(&msg);

  }

}



int main(int argc, char* argv[])

{

  printf("(%s:%d) main process initializing\n",__FILE__,__LINE__);

  int major, minor, patch;

  zmq_version (&major, &minor, &patch);

  printf("(%s:%d) zmq version: %d.%d.%d\n",__FILE__,__LINE__,major,minor,patch);



  pthread_t rId;

  pthread_create(&rId, 0, receiver, 0);



  pthread_t sId;

  pthread_create(&sId, 0, sender, 0);



  pthread_join(rId,0);

  pthread_join(sId,0);



  printf("(%s:%d) main process terminating\n",__FILE__,__LINE__);

}

Notice that we now transmit and receive Protobuf messages, serialized as strings.  The value of this is that the serialization mechanism is multi-platform & multi-language support.

Cheers.

ZeroMq Installation -- Python

In a previous post I documented the procedure for installing ZeroMq libraries for C/C++ applications. This post will focus on installing ZeroMq for Python development.


Let's get started;

Our target installation machine is Ubuntu 12.04 64-bit Desktop edition; http://www.ubuntu.com/download/desktop.  Your mileage may vary with alternative distributions.  I'm starting with a new default installation and will work through the installation process from here on.

$ sudo apt-get install python-zmq

After the APT package handling utility is done humping package installations you should have a function, albeit dated, version to begin working with.

Let's take a look at what version we're dealing with here.

$ cat go
#!/usr/bin/python
import zmq;
print zmq.pyzmq_version()

Running this beauty will show us the version of Zmq we've got installed.

$ ./go
2.1.11

Let's continue our sandbox by creating a sender and receiver Python script as follows;

$ more sender receiver
::::::::::::::
sender
::::::::::::::
#!/usr/bin/python

import zmq;
import time;

context = zmq.Context();
pub=context.socket(zmq.PUB);
pub.bind("tcp://127.0.0.1:8000");

for i in range(0,20):
  print "iteration",i
  pub.send("some message");
  time.sleep(1);
::::::::::::::
receiver
::::::::::::::
#!/usr/bin/python

import zmq;
import time;

context = zmq.Context();
sub=context.socket(zmq.SUB);
sub.connect("tcp://127.0.0.1:8000");
filter=""
sub.setsockopt(zmq.SUBSCRIBE, filter);


for i in range(0,20):
  print "waiting on msg"
  M=sub.recv();
  print "received",M

Running these two concurrently demonstrates the message delivery from the sender to the receiver;

::::::::::::::
sender.log
::::::::::::::
iteration 0
iteration 1
iteration 2
iteration 3
iteration 4
iteration 5
iteration 6
iteration 7
iteration 8
iteration 9
iteration 10
iteration 11
iteration 12
iteration 13
iteration 14
iteration 15
iteration 16
iteration 17
iteration 18
iteration 19
::::::::::::::
receiver.log
::::::::::::::
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg
received some message
waiting on msg

What may be of interest you as well is that since the 'endpoint' is defined identically to the one used in the C++ example you can send messages from the Python sender script to the C++ main application.

Enjoy.

ZeroMq Installation -- C++

ZeroMq, or 0Mq if you prefer, is considered 'sockets on steroids'.  The framework is aimed at simplifying distributed communications, serves as a concurrency framework, and supports more languages than any normal mortal would ever require.
http://zeromq.org

The topic of this post will be guiding through the installation on Ubuntu 12.04 and writing enough example code to verify all is happy.  Future posts will focus on installation of alternative, and interesting, languages with the overall intent of demonstrating interoperability between the languages.

Let's get started;

Our target installation machine is Ubuntu 12.04 64-bit Desktop edition; http://www.ubuntu.com/download/desktop.  Your mileage may vary with alternative distributions.  I'm starting with a new default installation and will work through the installation process from here on.

$ sudo apt-get install libzmq-dev

After the APT package handling utility is done humping package installations you should have a function, albeit dated, version to begin working with.

While ZeroMq supports a plethora of languages, the focus of this post is on C++.  That said, we'll need to install GCC C++ compiler.

$ sudo apt-get install g++

As I said earlier, the Ubuntu repository has a fairly old version.  At time of this writing, v3.2.3 was the latest stable release but for the purposes of this post we'll continue to use this version.

Let's confirm what version we have by authoring our first C++ application and ensure we can successfully compile/link with the installed libraries.

Start with a simple Makefile;

$ cat Makefile



CC=g++



SRCS=main.cpp

OBJS=$(subst .cpp,.o,$(SRCS))

INCLUDES += -I.

LIBS += -lpthread -lrt -lzmq



.cpp.o:

 $(CC) -c $<



main: ${OBJS}

 ${CC} ${CFLAGS} -o $@ ${OBJS} ${LIBS}



clean:

 ${RM} ${OBJS} main

And now let's look as a trival main program that confirms we can compile/link with Zmq libraries;

$ cat main.cpp



#include 

#include 



int main(int argc, char* argv[])

{

  printf("(%s:%d) main process initializing\n",__FILE__,__LINE__);

  int major, minor, patch;

  zmq_version (&major, &minor, &patch);

  printf("(%s:%d) zmq version: %d.%d.%d\n",__FILE__,__LINE__,major,minor,patch);

  printf("(%s:%d) main process terminating\n",__FILE__,__LINE__);

}

Running it confirms the installed Zmq libraries are near ancient, but again ok for our short-term purposes.

$ ./main



(main.cpp:6) main process initializing


(main.cpp:6) zmq version: 2.1.11

(main.cpp:6) main process terminating

Let's extend this to make it a bit more interesting, classic sender/receiver communication model using the PUB/SUB comm model provided by ZeroMq.

$ cat main.cpp



#include 

#include 

#include 

#include 

#include 

#include 



void* ctx=zmq_init(1);

char* EndPoint="tcp://127.0.0.1:8000";

static const int N=100;

static const int BufferSize=128;



void* sender(void*)

{

  printf("(%s:%d) running\n",__FILE__,__LINE__);

  void* pub=zmq_socket(ctx, ZMQ_PUB);

  assert(pub);

  int rc=zmq_bind(pub,EndPoint);

  assert(rc==0);

  for(int i=0; i  {

    char content[BufferSize];

    sprintf(content, "message %03d",i);

    zmq_msg_t msg;

    int rc=zmq_msg_init_size(&msg, BufferSize);

    assert(rc==0);

    rc=zmq_msg_init_data(&msg, content, strlen(content), 0,0);

    assert(rc==0);

    printf("(%s:%d) sending '%s'\n",__FILE__,__LINE__,content);

    rc=zmq_send(pub, &msg, 0);

    assert(rc==0);

    ::usleep(100000);

  }

}



void* receiver(void*)

{

  printf("(%s:%d) running\n",__FILE__,__LINE__);

  void* sub=zmq_socket(ctx, ZMQ_SUB);

  assert(sub);

  int rc=zmq_connect(sub,EndPoint);

  assert(rc==0);

  char* filter="";

  rc=zmq_setsockopt(sub, ZMQ_SUBSCRIBE, filter, strlen(filter));

  assert(rc==0);

  for(int i=0; i  {

    zmq_msg_t msg;

    zmq_msg_init_size(&msg, BufferSize);

    const int rc=zmq_recv (sub, &msg, 0);

    char* content=(char*)zmq_msg_data(&msg);

    printf("(%s:%d) received: '%s'\n",__FILE__,__LINE__,content);

    zmq_msg_close(&msg);

  }

}



int main(int argc, char* argv[])

{

  printf("(%s:%d) main process initializing\n",__FILE__,__LINE__);

  int major, minor, patch;

  zmq_version (&major, &minor, &patch);

  printf("(%s:%d) zmq version: %d.%d.%d\n",__FILE__,__LINE__,major,minor,patch);



  pthread_t rId;

  pthread_create(&rId, 0, receiver, 0);



  pthread_t sId;

  pthread_create(&sId, 0, sender, 0);



  pthread_join(rId,0);

  pthread_join(sId,0);



  printf("(%s:%d) main process terminating\n",__FILE__,__LINE__);

}

Running this beauty will result in the following;

$ cat /var/tmp/ZmqMb/main.log 
(main.cpp:59) main process initializing
(main.cpp:62) zmq version: 2.1.11
(main.cpp:15) running
(main.cpp:29) sending 'message 000'
(main.cpp:38) running
(main.cpp:29) sending 'message 001'
(main.cpp:52) received: 'message 001'
(main.cpp:29) sending 'message 002'
(main.cpp:52) received: 'message 002'
(main.cpp:29) sending 'message 003'
(main.cpp:52) received: 'message 003'
(main.cpp:29) sending 'message 004'
(main.cpp:52) received: 'message 004'
(main.cpp:29) sending 'message 005'
(main.cpp:52) received: 'message 005'
(main.cpp:29) sending 'message 006'
(main.cpp:52) received: 'message 006'
(main.cpp:29) sending 'message 007'
(main.cpp:52) received: 'message 007'
(main.cpp:29) sending 'message 008'
(main.cpp:52) received: 'message 008'
(main.cpp:29) sending 'message 009'
(main.cpp:52) received: 'message 009'
(main.cpp:29) sending 'message 010'
(main.cpp:52) received: 'message 010'
(main.cpp:29) sending 'message 011'
(main.cpp:52) received: 'message 011'

You'll notice that the first message was lost.  I've never had a sufficient understanding of why this happens, but typically the first message can be lost, primarily due to the pub/sub sockets not necessarily being fully established.  Issuing a 'sleep' between socket creation and between the receiver and sender socket usage doesn't solve the problem, it's more involved than that and I have little to offer as an explanation.

So, there you go; a simple usage of ZeroMq.

Cheers.

Soupin' Up Your Linux Harddrive Performance

I came across a little utility that I was until now unaware of. I discovered it when I was doing a search for increasing hard drive performance on our Linux workstations, the utility 'hdparm'.

The remainder of this posting should be performed in single-user mode to avoid interrupting remote users and get proper performance measurements for your system.

First, let's baseline our system hard drive performance, do so performing the following command as root.

# /sbin/hdparm -Tt /dev/hda

The first parameter '-T' measures the performance of the cache system, how the memory, CPU and buffer cache perform together.

The second parameter '-t' measures the performance of the disk, reading data not in cache. According to my drives specs which advertises XXX MB/sec this is pathetic in comparison.

# /sbin/hdparm /dev/hda

It shouldn't surprise you to see a good number of the drive features are turned off. These default settings are nice, safe but by no means optimal. These settings are pretty much guaranteed to work for near any system you throw at it, from a x386 to bleeding edge hardware.

The first option, 'multicount' is short for multiple sector count. This controls how many sectors are fetched from the disk for a single I/O interrupt. The man page suggests that enabling this feature may reduce disk overhead by 30-50% and provide 5-50% data throughput improvements.
The second option, 'I/O support' controls how data passes from the PCI bus to the controller. Near all modern controller chipsets support mode 3, 32-bit mode w/sync. Turning this on will likely near double your throughput.
The third option, 'unmaskirq' allows Linut to unmask other interrups, allowint it to attend to other interrupt driven tasks whil waiting for the disk to return the requested data. While not all hardware configurations will be able to handle this, if your hardware supports it you should note better response time.
The fourth option, 'use_dma' should be used with a bit more caution, we'll be ignoring this feature as it's documented as risky.

So, let's get our hands dirty.
Let's first set 32-bit w/sync on and multicount on.

# /sbin/hdparm -c3 -m16 /dev/hda
# /sbin/hdparm -Tt /dev/hda

'K, let's take another step.

# /sbin/hdparm -X66 -d1 -u1 -m16 -c3 /dev/hda
# /sbin/hdparm -Tt /dev/hda

It's worth noting that these settings don't persist, rebooting your box will default back to your original settings. This allows you to play around with it 'til you find a suite of settings you're happy with. When you're happy, edit your /etc/rc.d/* scripts, adding the assignments after the fsck check.

Viewing WebCam Video with Mplayer

On a number of occasions I find it useful to view my laptop's camera in real-time using Mplayer. Seems this is accomplishable using the following command:

$ mplayer -fps 15 tv:// -tv  driver=v4l2:device=/dev/video0

Cheers.

Making Use Of Idle Workstation

Have you ever given thought to the amount of computational horsepower your machine packs and how much of it is wasted when not in use? I thought it would be useful to write a script that could interrogate the current state of a Linux workstation to determine if it was available to run some background activities, similar to SETI desktop behavior.

Seems determining if the screensaver is active is a pretty good indicator of whether a user is currently using the system. Bundle that with evaluation of the system load and you can get a pretty good indication of an idle system.

Following is a Tcl-script that determines if the system screen saver is active. One could imaging bundling this will suspending/resuming a process or virtual machine to make use of the idle time.

Let me know what you think and usages you may think of.

#!/usr/bin/tclsh



proc isScreenSaverRunning { } {

  set response [exec gnome-screensaver-command -q]

# puts $response

  foreach e [split $response \n] {

    switch -glob -- $e {

      "* inactive"

       {

         set running 0

       }

      "* active"

       {

         set running 1

       }

      default

       {

       }

    }

  }

  return $running

}



proc check { } {

  set running [isScreenSaverRunning]

  if { $running } {

    puts "screen saver running"

    exit

  }

  after 1000 [list check]

}



#-----main------

after 1000 [list check]

vwait forever

Simulating Mouse Events

I've been struggling to find a suitable testing method to test the Android applications I'm developing. As I'm relatively new to testing UI's I didn't have much to fall back on.

I first began looking at LDTP, but found that the raw mouse events were limited to click and double-clicking left and right buttons. I needed something more flexible to support swiping the Android Emulator.

What I found was 'xdotool' which seems to hit all the marks; activating a window, depressing either mouse button, moving the mouse, and releasing the button(s). Introducting a delay between mouse down and up allows for long press events.

e.g.

$ xdotool windowactivate `exec xdotool search --title 5554:myAvd`
$ xdotool mouse move 100 100
$ xdotool mousedown 1
$ xdotool mouse move 200 200
$ xdotool mouseup 1

Cheers.

Hola Android -- Stage 3

So far, we've created the equivalent of a 'Hello World' application, extended the UI to include a menu button, and bound an action with the pressing of the menu button.

Let's expand our example to include a sub-menu system. First, add the additional menu button to the res/layout/main.xml file as follows:

user@River:~/HolaAndroid$ cat -n res/layout/main.xml
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
android:orientation="vertical"
android:layout_width="fill_parent"
android:layout_height="fill_parent"
>
<TextView
android:layout_width="fill_parent"
android:layout_height="wrap_content"
/>
<Button android:id="@+id/button1"
android:layout_width="fill_parent"
android:layout_height="wrap_content"
android:layout_alignParentBottom="true"
android:text="Menu1" />
<Button android:id="@+id/button2"
android:layout_width="fill_parent"
android:layout_height="wrap_content"
android:layout_alignParentBottom="true"
android:text="Menu2" />
</LinearLayout>

Rebuilding and running and you'll get twin menu buttons, the first bound to quitting the application, the second unbound to any action.













Next, we need to bind the pressing of button 2 to an action, namely the creation and rendering of another submenu.

Let's contain the behaviors of the second menu button to a newly defined class, as follows:

user@River:~/HolaAndroid$ cat src/com/example/holaandroid/Menu2.java
package com.example.holaandroid;

import com.example.holaandroid.R;

import android.app.Activity;
import android.database.Cursor;
import android.os.Bundle;
import android.view.View;
import android.widget.Button;
import android.widget.EditText;

public class Menu2 extends Activity {
private Button addButton;
   @Override
   protected void onCreate(Bundle savedInstanceState) {
       super.onCreate(savedInstanceState);
       setContentView(R.layout.menu2);
       addButton = (Button)this.findViewById(R.id.m2_button1);
   }
}

Easy thing to forget, but now that we're introducing a new Activity you must define it in the manifest (line 14).

$ user@River:~HolaAndroid$ cat AndroidManifest.xml
1    <?xml version="1.0" encoding="utf-8"?>
2    <manifest xmlns:android="http://schemas.android.com/apk/res/android"
3          package="com.example.holaandroid"
4          android:versionCode="1"
5          android:versionName="1.0">
6        <application android:label="@string/app_name">
7            <activity android:name=".HolaAndroid"
8                      android:label="@string/app_name">
9                <intent-filter>
10                    <action android:name="android.intent.action.MAIN" />
11                    <category android:name="android.intent.category.LAUNCHER" />
12                </intent-filter>
13            </activity>
14            <activity android:name=".Menu2"/>
15        </application>
16    </manifest>

Since we're referencing a button in our source code, we need to have it defined in our layout xml. Create a new XML file which defines our button as follows:

user@River:~/HolaAndroid$ cat res/layout/menu2.xml
<?xml version="1.0" encoding="utf-8"?>

<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
android:orientation="vertical" android:layout_width="fill_parent"
android:layout_height="fill_parent">
<Button android:text="M2.Button1"
android:id="@+id/m2_button1"
android:layout_width="fill_parent"
android:layout_height="wrap_content"
android:layout_alignParentBottom ="true"/>
</LinearLayout>

The final main class follows, the updates include importing the newly relevant packages, creating a createMenu2() method and assigning the main menu button click to the generation of the submenu (lines 9-12, 18, 36-43 & 45-52).

user@River:~/HolaAndroid$ cat -n src/com/example/holaandroid/HolaAndroid.java
    1 package com.example.holaandroid;
    2 
    3 import android.app.Activity;
    4 import android.os.Bundle;
    5 import android.widget.Button;
    6 import android.view.View;
    7 import android.view.View.OnClickListener;
    8 import android.util.Log;
    9 import android.content.Intent;
   10 import android.view.Menu;
   11 import android.view.MenuItem;
   12 import android.view.MenuInflater;
   13 
   14 public class HolaAndroid extends Activity
   15 {
   16     private static final String ClassName = "HolaAndroid";
   17     private Button b1_;
   18     private Button b2_;
   19 
   20     /** Called when the activity is first created. */
   21     @Override
   22     public void onCreate(Bundle savedInstanceState)
   23     {
   24         final String MethodName = "onStart";
   25 
   26         super.onCreate(savedInstanceState);
   27         setContentView(R.layout.main);
   28 
   29         b1_ = (Button)this.findViewById(R.id.button1);
   30         b1_.setOnClickListener(new OnClickListener() {
   31                 public void onClick (View v){
   32                   Log.d(ClassName+"::"+MethodName, "entry");
   33                   finish();
   34                 }
   35             });
   36         b2_ = (Button)this.findViewById(R.id.button2);
   37         b2_.setOnClickListener(new OnClickListener() {
   38                 public void onClick (View v){
   39                   Log.d(ClassName+"::"+MethodName, "entry");
   40                   createMenu2();
   41                   Log.d(ClassName+"::"+MethodName, "entry");
   42                 }
   43             });
   44     }
   45     private static final int ACTIVITY_CREATE=0;
   46     private void createMenu2() {
   47       final String MethodName = "createMenu2";
   48       Log.d(ClassName+"::"+MethodName, "entry");
   49       Intent i = new Intent(this, Menu2.class);
   50       startActivityForResult(i, ACTIVITY_CREATE);
   51       Log.d(ClassName+"::"+MethodName, "exit");
   52     }
   53 
   54 }

While you're still in a similar predicament as our previous post, namely a submenu button with no action defined, you can navigate to/from the submenu using the main button and the back keypad button.

Cheers.

Hola Android -- Stage 2

Ok, so you've got your first Android application that prints 'Hello World'; nothing to write home about.

Let's expand on it a bit and make use of some of the User Interface capabilities. First though, let's remove the printing of 'Hello World'.

If you search the code you'll find no reference to our elusive string, instead it's hiding in the resources layout definition file res/layout/main.xml.

user@River:~/HolaAndroid$ cat -n res/layout/main.xml
1  <?xml version="1.0" encoding="utf-8"?>
2  <LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
3      android:orientation="vertical"
4      android:layout_width="fill_parent"
5      android:layout_height="fill_parent"
6      >
7  <TextView
8      android:layout_width="fill_parent"
9      android:layout_height="wrap_content"
10      android:text="Hello World, HolaAndroid"
11      />
12  </LinearLayout>
13
user@River:~/HolaAndroid$

Removing line 10 and you'll now have an application that renders a blank screen. Rebuild and re-run and you'll get:

user@River:~/HolaAndroid$ vi res/layout/main.xml
user@River:~/HolaAndroid$ make
ant debug
Buildfile: build.xml
  [setup] Android SDK Tools Revision 6
  [setup] Project Target: Android 1.5
  [setup] API level: 3
  [setup] WARNING: No minSdkVersion value set. Application will install on all Android versions.
  [setup] Importing rules file: platforms/android-3/ant/ant_rules_r2.xml

-compile-tested-if-test:

-dirs:
   [echo] Creating output directories if needed...

-resource-src:
   [echo] Generating R.java / Manifest.java from the resources...

-aidl:
   [echo] Compiling aidl files into Java classes...

compile:
  [javac] Compiling 1 source file to /home/user/HolaAndroid/bin/classes

-dex:
   [echo] Converting compiled files and external libraries into /home/user/HolaAndroid/bin/classes.dex...

-package-resources:
   [echo] Packaging resources
[aaptexec] Creating full resource package...

-package-debug-sign:
[apkbuilder] Creating HolaAndroid-debug-unaligned.apk and signing it with a debug key...
[apkbuilder] Using keystore: /home/user/.android/debug.keystore

debug:
   [echo] Running zip align on final apk...
   [echo] Debug Package: /home/user/HolaAndroid/bin/HolaAndroid-debug.apk

BUILD SUCCESSFUL
Total time: 1 second
user@River:~/HolaAndroid$ make run
adb install -r ./bin/HolaAndroid-debug.apk
97 KB/s (4342 bytes in 0.043s)
 pkg: /data/local/tmp/HolaAndroid-debug.apk
Success
adb shell am start -a android.intent.action.HolaAndroid -n com.example.holaandroid/com.example.holaandroid.HolaAndroid
Starting: Intent { act=android.intent.action.HolaAndroid cmp=com.example.holaandroid/.HolaAndroid }
user@River:~/HolaAndroid$

Now, you can play with your screen design by editing the res/layout/main.xml file. Note below we've added lines 11-15 to add a button to the main window.

user@River:~/HolaAndroid$ cat -n res/layout/main.xml
1    <?xml version="1.0" encoding="utf-8"?>
2    <LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
3        android:orientation="vertical"
4        android:layout_width="fill_parent"
5        android:layout_height="fill_parent"
6        >
7    <TextView
8        android:layout_width="fill_parent"
9        android:layout_height="wrap_content"
10        />
11    <Button android:id="@+id/button1"
12        android:layout_width="fill_parent"
13        android:layout_height="wrap_content"
14        android:layout_alignParentBottom="true"
15        android:text="Menu1" />
16    </LinearLayout>
user@River:~/HolaAndroid$

Rebuild and run and you'll get:



Alas, while you've created a button, the button has no action associated with it. Press it as you'd like you'll never get it to do anything.

So, let's make it do something. In order to tie your newly created button to an action you'll need to edit the src/com/example/holaandroid/HolaAndroid.java source. We'll add a button member, or attribute, and assign an action method to be called on the button click.

Below is our modified source code:

user@River:~/HolaAndroid$ cat -n src/com/example/holaandroid/HolaAndroid.java 
     1  package com.example.holaandroid;
     2
     3  import android.app.Activity;
     4  import android.os.Bundle;
     5  import android.widget.Button;
     6  import android.view.View;
     7  import android.view.View.OnClickListener;
     8  import android.util.Log;
     9
    10  public class HolaAndroid extends Activity
    11  {
    12      private static final String ClassName = "HolaAndroid";
    13      private Button b1_;
    14
    15      /** Called when the activity is first created. */
    16      @Override
    17      public void onCreate(Bundle savedInstanceState)
    18      {
    19          final String MethodName = "onStart";
    20
    21          super.onCreate(savedInstanceState);
    22          setContentView(R.layout.main);
    23
    24          b1_ = (Button)this.findViewById(R.id.button1);
    25          b1_.setOnClickListener(new OnClickListener() {
    26                  public void onClick (View v){
    27                    Log.d(ClassName+"::"+MethodName, "entry");
    28                  }
    29              });
    30      }
    31  }
user@River:~/HolaAndroid$ 

Now, if you're running the debugger you'll notice upon button click the debugger will receive a 'D/HolaAndroid::onStart( 1525): entry' message. To make our action a bit more obvious, follow line 27 with:

    ..
    ..
    25         b1_.setOnClickListener(new OnClickListener() {
    26                 public void onClick (View v){
    27                   Log.d(ClassName+"::"+MethodName, "entry");
    28                   finish();
    29                 }
    30             });
    ..
    ..

Now, your application will terminate when you press the button.

Hola Android -- Stage 1

I've begun my upward journey developing Android applications for my Samsung Galaxy phone. My intention is to review some of what I've learned and follow on with posts as I learn more. I'll skip the development setup as I've documented that in previous posts.

As with every good Computer Science example, we'll begin with the infamous 'hello world'.

Our target, a simple application that prints 'hello world' to the screen. Slowly I intend on building on it with no clear final destination than playing with features as they present themselves.

Assuming you've got your Android development environment properly set up we embark by creating a new project:

user@River:~$ ./android-sdk-linux_86/tools/android create project --package com.example.holaandroid --activity HolaAndroid --target 2 --path ~/HolaAndroidCreated project directory: /home/user/HolaAndroid
Created directory /home/user/HolaAndroid/src/com/example/holaandroid
Added file /home/user/HolaAndroid/src/com/example/holaandroid/HolaAndroid.java
Created directory /home/user/HolaAndroid/res
Created directory /home/user/HolaAndroid/bin
Created directory /home/user/HolaAndroid/libs
Created directory /home/user/HolaAndroid/res/values
Added file /home/user/HolaAndroid/res/values/strings.xml
Created directory /home/user/HolaAndroid/res/layout
Added file /home/user/HolaAndroid/res/layout/main.xml
Added file /home/user/HolaAndroid/AndroidManifest.xml
Added file /home/user/HolaAndroid/build.xml
user@River:~$ 

Now we've got a nice clean project directory structure that looks like the following:

user@River:~$ cd HolaAndroid/
user@River:~/HolaAndroid$ find .
.
./build.xml
./bin
./default.properties
./libs
./build.properties
./AndroidManifest.xml
./local.properties
./res
./res/values
./res/values/strings.xml
./res/layout
./res/layout/main.xml
./src
./src/com
./src/com/example
./src/com/example/holaandroid
./src/com/example/holaandroid/HolaAndroid.java
user@River:~/HolaAndroid$ 

Since I don't utilize the Eclipse development environment (personal choice) I authored a Makefile to ease some of the routine tasks.

user@River:~/HolaAndroid$ cat Makefile 
all:
        ant debug

setup:
        emulator -avd myAvd &
        xterm -e "adb logcat" &

run:
        adb install -r ./bin/HolaAndroid-debug.apk
        adb shell am start -a android.intent.action.HolaAndroid -n com.example.holaandroid/com.example.holaandroid.HolaAndroid

clean:
        adb shell rm data/app/com.example.holaandroid.apk
        ant clean
user@River:~/HolaAndroid$ 

Attempting to build the application by issuing a 'make' command and you'll be met with success.

user@River:~/HolaAndroid$ make
ant debug
Buildfile: build.xml
    [setup] Android SDK Tools Revision 6
    [setup] Project Target: Android 1.5
    [setup] API level: 3
    [setup] WARNING: No minSdkVersion value set. Application will install on all Android versions.
    [setup] Importing rules file: platforms/android-3/ant/ant_rules_r2.xml

-compile-tested-if-test:

-dirs:
     [echo] Creating output directories if needed...
    [mkdir] Created dir: /home/user/HolaAndroid/gen
    [mkdir] Created dir: /home/user/HolaAndroid/bin
    [mkdir] Created dir: /home/user/HolaAndroid/bin/classes

-resource-src:
     [echo] Generating R.java / Manifest.java from the resources...

-aidl:
     [echo] Compiling aidl files into Java classes...

compile:
    [javac] Compiling 2 source files to /home/user/HolaAndroid/bin/classes

-dex:
     [echo] Converting compiled files and external libraries into /home/user/HolaAndroid/bin/classes.dex...

-package-resources:
     [echo] Packaging resources
 [aaptexec] Creating full resource package...

-package-debug-sign:
[apkbuilder] Creating HolaAndroid-debug-unaligned.apk and signing it with a debug key...
[apkbuilder] Using keystore: /home/user/.android/debug.keystore

debug:
     [echo] Running zip align on final apk...
     [echo] Debug Package: /home/user/HolaAndroid/bin/HolaAndroid-debug.apk

BUILD SUCCESSFUL
Total time: 2 seconds

Running it and you'll get

user@River:~/HolaAndroid$ make run
adb install -r ./bin/HolaAndroid-debug.apk
99 KB/s (4385 bytes in 0.043s)
        pkg: /data/local/tmp/HolaAndroid-debug.apk
Success
adb shell am start -a android.intent.action.HolaAndroid -n com.example.holaandroid/com.example.holaandroid.HolaAndroid
Starting: Intent { act=android.intent.action.HolaAndroid cmp=com.example.holaandroid/.HolaAndroid }
user@River:~/HolaAndroid$ 

Just about every Android introduction tutorial demonstrates this, but now you've got another.

Cheers.

Journey of Android Development -- First Emulator App

Still trying to get the feel for the development environment. Much of this is likely fully integrated with Eclipse, but Eclipse is historically a resource hog. 'sides, I more of a command line kinda guy.

I authored my very first application that entails multiple buttons and debug logging. The app simply creates three buttons, registers a click handler for each and demonstrates the ability to log to the adb logfile stream.

Feel free to download it here.


You may also find the Makefile useful, which supports:
1) setting up the environment (ie. starting AVD and logcat utility)
2) rebuilding the project
3) reinstalling and remotely starting the app on the emulator
4) cleanup

The project assumes you've greated an AVD called myAVD.

Cheers.

Ellusive 'Error: java.lang.NullPointerException' for Android Development Installation

I just spent some time debugging why my android development setup was failing on Debian 5.0

I repeatedly got a "XML verification failed for https://dl-ssl.google.com/android/repository/repository.xml.Error: java.lang.NullPointerException" error.

After navigating to the referenced web site and confirmed it wasn't a web site mis-reference I consulted Google.

A couple forums elluded to updating the SDK and Java, but neither was the case for me. My problem was I failed to configure the Sun packages as the default java compiler. This is done by:

debian:/root#  update-alternatives --config java

There are 3 alternatives which provide `java'.

  Selection    Alternative
-----------------------------------------------
          1    /usr/bin/gij-4.3
*+        2    /usr/lib/jvm/java-gcj/jre/bin/java
          3    /usr/lib/jvm/java-6-sun/jre/bin/java

Press enter to keep the default[*], or type selection number: 3
Using '/usr/lib/jvm/java-6-sun/jre/bin/java' to provide 'java'.

Next issue a SDK update by issuing the 'android update sdk' command and install the targets.

The symptom of this issue was an unpopulated target list (e.g. 'android list targets' resulting in no targets available).

Kudos.

Installing WindowMaker on Debian

Real short, but it took me a bit of searching to find how to replace Gnome as my window manager.

Start by installing the WindowMaker package:

# apt-get install wmaker

Next, as the user that you wish to run WindowMaker as the window manager:

$ echo "exec wmaker" >> ~/.xsession
$ echo "exec wmaker" >> ~/.xinitrc

Log off, then back on for it to take effect.

Cheers.

Iso Magic

I find that over the past couple weeks I've had to generate, burn, and examine ISO images to support releasing our software product on external media. I thought it useful to share some useful mechanisms.

If you have a directory structure that you want bundled into a ISO image, issue the command:

mkisofs -o MyImage.iso -J dirName

To burn the image onto a DVD/CDROM use the following command:

cdrecord -v dev=X,X,X MyImage.iso

where X,X,X is determined by issuing the command:

cdrecord --scanbus

and selecting the appropriate device

You can examine the contents of an ISO image by mounting loopback as follows:

mount -o loop MyIsoImage.iso /mnt/iso

Issuing a 'find /mnt/iso' will show the contents of the image.

Create ISO Image From Directory

Occasionally it's useful to create a CD/DVD ISO image from a list of files contained in a directory. For instance, if you're interested in archiving a bunch of files to be later burned to a CD/DVD.

This can be done by using the mkisofs command which stands for 'make ISO filesystem'.

If you have a list of directories like the following:

user@debian:/media/ehd/tmp$ ls -l
total 2802468
drwxr-xr-x 3 root root       4096 2009-12-29 13:01 disc01
drwxr-xr-x 3 root root       4096 2009-12-29 13:01 disc02
drwxr-xr-x 3 root root       4096 2009-12-29 13:02 disc03
drwxr-xr-x 3 root root       4096 2009-12-29 13:03 disc04
drwxr-xr-x 3 root root       4096 2009-12-29 13:05 disc05
drwxr-xr-x 3 root root       4096 2009-12-29 13:06 disc06
drwxr-xr-x 3 root root       4096 2009-12-29 13:08 disc07
drwxr-xr-x 3 root root       4096 2009-12-29 13:09 disc08
drwxr-xr-x 3 root root       4096 2009-12-29 13:10 disc09
drwxr-xr-x 3 root root       4096 2009-12-29 13:10 disc10
drwxr-xr-x 3 root root       4096 2009-12-29 13:12 disc11
user@debian:/media/ehd/tmp$

You can create individual ISO images for each directory by issuing the following command:

$ mkisofs -o disc02.iso -J disc02

The ISO image will preserve the directory structures and file names found under disc02.

Cheers

AutoStart VM with Xen

I've been evaluating Xen, VirtualBox, and VMware as part of work. We require a system to autoboot virtual machines as a feature of the virtualization technology we choose. I just found how to autoboot using Xen and thought it worthy of a quick post.

# mkdir /etc/xen/auto
# ln -s /etc/xen/mymachine.cfg /etc/xen/auto/

You can test this by restarting the xendomain service as follows:

# /etc/init.d/xendomains stop

Starting VirtualBox as a Service

I've been trying to compare VirtualBox features with VMware Server 2.0. One of the first features that doesn't exist (without some work) is the concept of auto-starting virtual machines at system start-up.

With head-less virtual machine modes and RDP connectivity, it appears that all the core features exist.

By incorporating the starting of virtual machines in head-less modes as a service that is started at system start-up you'd have the equivalent of VMware Server. This post will document the means to do just that.

First, since system start-up is executed by root, the virtual machine will need to be associated with the root user. You can accomplish this by firing up VirtualBox interactively and create the desired virtual machines. I, however, had the virtual machines I wanted to start created in an alternative user account, so I created a soft-link between the root user home directory and the user directory (keep you security concerns to yourself, this is simply proof of concept).

# cd /root
# ln -s /home/user/.VirtualBox .VirtualBox

Next, create a script that serves as the necessary service daemon:

#!/bin/bash
# scriptName: VBoxD

VBOXHEADLESS="/usr/bin/VBoxHeadless"

vbox_start() {
  echo "...starting $1" >> /var/tmp/myLog
  $VBOXHEADLESS -s $1 > /dev/null &
}

#---main---
case "$1" in
  start)
    echo "...starting my service" >> /var/tmp/myLog
    logger -p user.notice "starting service"
    vbox_start WinXp
  ;;
  stop)
    echo "...stopping my service" >> /var/tmp/myLog
    logger -p user.notice "stopping service"
  ;;
  restart)
    echo "...restarting my service" >> /var/tmp/myLog
  ;;
esac

exit 0

Note that this script has hard-coded virtual machine named WinXp; again, this is proof of concept. Please keep in mind, this isn't a complete service script solely enough to demonstrate the ability of introducing a service into system start-up.

You can test the script behavior can by invoking:

# ./VBoxD start
# ./VBoxD stop

To incorporate this into the system start-up, you first find what runlevel you are currently running:

# ps -aef | grep init
root         1     0  0 20:23 ?        00:00:00 init [2]
root      5047  4725  0 20:58 pts/0    00:00:00 grep init

Shows run-level 2; meaning that the service must reside in /etc/rc2.d

Changing to /etc/rc2.d you add the service by creating a soft-link to /root/VBoxD with a prefix S followed by a 2-digit numeric (the larger the numeric the later the service is started). In this example, I created S91VBoxD as follows:

xion:/etc/rc2.d# ls -l S91VBoxD
lrwxrwxrwx 1 root root 11 2009-04-12 19:47 S91VBoxD -> /root/VBoxD

If you've properly enabled RDP for this machine at port 3389 you should be able to connect to the head-less virtual machine by:

$ rdesktop 127.0.0.1:3389

Creating CD-ROM ISO Image - the cowboy way

I've been investigating installation of Xen for work, in the process I thought it would be useful to rather than run around with the WinXp installation disk to create an ISO image of it for installation purposes.

The utility 'dd' suits the bill, you have to first interface directly with the device driver so the cd cannot be mounted. If the cd is mounted, I'd first recommend finding the device driver reference by issuing the mount command:

$ mount
. . .
/dev/scd0 on /media/cdrom0 . . . ..
$ umount /media/cdrom0

As you can see, this was an external cdrom on /dev/scd0.

Next, issue the appropriate dd command:

$ dd if=/dev/scd0 of=WinXpProInstall.iso

The arguments if=/dev/scd0 tells dd to read from the device in raw format, of=WinXpInstall.iso specifies the output file.

You'll be left with an exact image of what resides on the cd, use at your leisure.

Timing VM Startup Script

In my previous post I eluded to creating a script for timing the starting of a virtual machine.

This is it, use at your leisure:

#!/bin/csh

set i=0
while ( $i < starttime="`date" addr="192.168.2.24" response="1" response ="=">> /dev/null
  set response=$status
end
set stopTime=`date +%s`

set secs=`expr $stopTime - $startTime`
echo "startUp: $secs (secs)"

sleep 10
vmrun -T 127.0.0.1 -h 'https://127.0.0.1:8333/sdk' -u vmadmin -p 'vmadminpassword' stop "[standard] JltvWinXp/JltvWinXp.vmx"
set i=`expr $i + 1`
end

The script loops 5 times to get a crude sampling.  Each iteration, the virtual machine is started then monitored via a ping until the VM is pingable.  It then sleeps some 10 seconds before commanding the virtual machine to shutdown.