11 October, 2015

FFMpeg is Digital Meth -- Part 1


FFMpeg Options

Global Options






AVOptions

These options are specific to the given container, device or codec.  The '-profile' option above is an example of such an option.  These options are provided directly to the libavformat, libavdevice, and libavcodec libraries.

Main Options

'-fmt (input/output)'

FFMpeg normally auto detects the file format for input or output files, guessed from the file extension so often it is not required to specify the format.  The format can be explicitly specified by using '-fmt', for input or output files.

'-y (global)'

If the output file(s) already exist, FFMpeg will halt and present the user with the option to overwrite the file.  Specifying '-y' overwrites output files without user intervention.

'-t duration (input/output)'

This argument applies both to input and output files.  If specifying a duration as an input option (before -i), the result is limiting the duration of the data read from the input file.  When specified as an output option (before the output filename) the FFMpeg will stop writing after its reached the specified duration.

Specifying '-t 20' as an input file, with two output files will result in decoding and re-encoding the 1st 20 seconds of video creating two equivalent output files:

$ ffmpeg -y -t 20 -i big_buck_bunny_1080p_h264.mov -an /tmp/foo1.mov -an /tmp/foo2.mov

Repeating the command while specifying '-t 10' as an output argument to one of the files will result in one input file being 20 sec duration, the other a 10 second duration.

$ ffmpeg -y -t 20 -i big_buck_bunny_1080p_h264.mov -an /tmp/foo1.mov -an -t 10 /tmp/foo2.mov

'-fs fileSize (input/output)'

Another useful output option allows setting the file size limit.  The input file is read, writing the output file until the file size limit specified has been reached.  The result in the following case will be a video consisting of the first ~22 seconds of the input video.


$ ffmpeg -y -i big_buck_bunny_1080p_h264.mov -an -fs 10M /tmp/foo-10M.mov

$ ls -lh /tmp/foo-10M.mov

-rw-rw-r-- 1 user user 9.8M Sep  5 17:45 /tmp/foo-10M.mov

But what if you want to fast forward to 30 seconds into the input file followed by encoding the following 10 seconds? The '-ss' fits the bill when specified as an input file option.  When specified as an output file option, the end result is similar, accomplished by decoding but discarding the input until the timestamps position is reached.


$ ffmpeg -y -ss 30 -i big_buck_bunny_1080p_h264.mov -t 10 -an /tmp/foo-30-40.mov

Setting the timestamp

Python Progress Bar

Was recently writing some python to process some test cases.  One long'ish testcase would benefit from a progress bar, slapped one together below.

One thing to note, progress doesn't allow going backwards, but indicates in ASCII text the current progress by writing characters and a series of backspaces.



#!/usr/bin/python
import time
import sys
 
class ProgressBar():
  barLen_=0;
  progress_=0;
  def __init__(self,N=10):
    self.barLen_=N;
    print '['+' '*self.barLen_+']',
    print '\b'*(self.barLen_+2),
  def setProgress(self,x):
    sys.stdout.flush();
    k=int(x/100.0*self.barLen_);
    if (k > self.progress_):
      update='\b'+'.'*(k-self.progress_);
      print update,
      self.progress_=k;

N=50
p=ProgressBar(N);

for i in range(0,101):
  p.setProgress(i);
  time.sleep(.05);

The result is of the form:

user@river:~$ ./pBar 
[..............                                  ] 
Enjoy.

19 September, 2015

Limiting Network Bandwidth

Occasionally it's necessary to determine the affects of reduced network bandwidth. Authoring software that works for a networked system backed by Gigabit Ethernet as well as DSL requires testing on both forms of systems.  The alternative is to test on the most capable networking system and then test lower-bandwidth systems on the same network by artificially limiting the bandwidth.

Consulting the all-knowing Google provided a couple general options: trickle and wondershaper.  While there are other utilities, I mainly focused on these two.

Trickle is a 'lightweight user bandwidth shaper', it's particular value is the ability to limit processes independently and it doesn't require superuser privileges.


$ sudo apt-get install trickle


You can limit uploads and downloads to 100 KB/s for a command such as;

$ trickle -u 100 -d 100 wget http://download.blender.org/peach/bigbuckbunny_movies/big_buck_bunny_1080p_stereo.ogg


Independently, you could specify alternative limits for another command.  Trickle is limited to TCP sockets so if you need to limit bandwidth for UDP sockets you'll need to look for an alternative.  Applying system-wide bandwidth, TCP in this case, supposedly is supported via trickled daemon utility but after numerous hours I gave up on getting it to work.  Regardless, I abandoned it's use not only because I couldn't get it to work but also it's limitation to TCP and applying the daemon and client command required prepending 'trickle' to the command line.  This would disqualify applying the network bandwidth constraints to network services like sshd and such without modifying each service starting the process.

Seeking a system-wide bandwidth traffic shaper brought me back to Wondershaper once again.  My first experience with wondershaper I found it didn't always work, this round I tested the usage 1-10 Mbps and I quickly found that it limited the bandwidth like expected until reaching 10 Mbps where it jumped to realized bandwidth of 40 Mbps.  Head-scratcher; working fine for bandwidths under 10Mbps but broke at >= 10Mbps.  Consulting Google again brought me to a fix proposed by 'buzzy'; https://github.com/magnific0/wondershaper/issues/2.  Modifying the wondershaper script as 'buzzy' indicated resolved the issue and I could consistently specify bandwidths 1-40 Mbps and observe realized bandwidths accordingly.


$ cat go

#!/bin/bash



for i in `seq 1 50`; do

  echo "$i Mbps"

  K=`expr $i \* 1024`

  sudo wondershaper wlan0 $K $K

  iperf -t 10 -c 192.168.1.132 2> /dev/null | grep "Mbits/sec"

  sleep 2

done

sudo wondershaper wlan0 clear



lipeltgm@river:~$ ./go 
1 Mbps
[  3]  0.0-11.2 sec  1.38 MBytes  1.03 Mbits/sec
2 Mbps
[  3]  0.0-10.7 sec  2.62 MBytes  2.06 Mbits/sec
3 Mbps
[  3]  0.0-10.4 sec  3.75 MBytes  3.02 Mbits/sec
4 Mbps
[  3]  0.0-10.4 sec  5.00 MBytes  4.04 Mbits/sec
5 Mbps
[  3]  0.0-10.5 sec  5.12 MBytes  4.11 Mbits/sec
6 Mbps
[  3]  0.0-10.3 sec  7.38 MBytes  6.00 Mbits/sec
7 Mbps
[  3]  0.0-10.3 sec  8.50 MBytes  6.95 Mbits/sec
8 Mbps
[  3]  0.0-10.3 sec  9.75 MBytes  7.97 Mbits/sec
9 Mbps
[  3]  0.0-10.2 sec  10.6 MBytes  8.76 Mbits/sec
10 Mbps
[  3]  0.0-10.1 sec  12.0 MBytes  9.94 Mbits/sec
11 Mbps
[  3]  0.0-10.1 sec  13.2 MBytes  11.0 Mbits/sec
12 Mbps
[  3]  0.0-10.2 sec  14.5 MBytes  12.0 Mbits/sec
13 Mbps
[  3]  0.0-10.2 sec  15.5 MBytes  12.8 Mbits/sec
14 Mbps
[  3]  0.0-10.3 sec  16.1 MBytes  13.2 Mbits/sec
15 Mbps
[  3]  0.0-10.1 sec  16.4 MBytes  13.6 Mbits/sec
16 Mbps
[  3]  0.0-10.1 sec  19.0 MBytes  15.8 Mbits/sec
17 Mbps
[  3]  0.0-10.1 sec  19.8 MBytes  16.4 Mbits/sec
18 Mbps
[  3]  0.0-10.1 sec  21.1 MBytes  17.6 Mbits/sec
19 Mbps
[  3]  0.0-10.1 sec  22.1 MBytes  18.3 Mbits/sec
20 Mbps
[  3]  0.0-10.1 sec  23.1 MBytes  19.3 Mbits/sec
...


While the use of WonderShaper requires superuser privileges, it limits the system network bandwidth in it's entirety. Enjoy.

12 September, 2015

Makefile Mystique

Originating in the work by Stuart Feldman at Bell Labs in 1976, the make utility has existed in a number of fashions since and is one of the most common build utilities for *nix based systems.  Despite that however, in my 16+ years of professional software development, authoring or maintaining makefiles takes on a classic game of 'not it!!' seemingly everywhere I work.

Few would argue that the utility lacks flexibility or power, the general complaint is the syntax/semantics are confusing and unmaintainable, one of the primary reasons that popular IDEs synthesize their own makefiles in an attempt to isolate the user from the pain and misery of doing it themselves.  The goal of this post isn't to complain about the utility, but instead to work through a few examples in an attempt to better understand it myself.  While authoring and maintaining a makefile may feel like a prostate exam, it's also likely as necessary as one.  In preparing for this post I referred the documentation here and I invite you to do the same.

Part of make's popularity and power is because of it's implicit rules.  Making use of these rules you'll find that like good liquor, a little goes a long way.  This is evident for example when your project utilizes C/C++.

With a simple source file and relying on implicit make rules the necessary makefile is simplistic;
$ cat main.c 
#include <stdio.h>

int main()
{
  printf("(%s:%d) main process initializing\n",__FILE__,__LINE__);
  printf("(%s:%d) main process terminating\n",__FILE__,__LINE__);
}

A single rule comprises the makefile and provides a simplistic, minimalistic build system.
$ cat Makefile 
main: main.o

Each makefile consists of 'rules' taking the form;
     target ... : prerequisites ...
     <tab> recipe
     <tab> ...

Examining the rule we find the target is defined as 'main' with a prerequisite of 'main.o'.  This simply means that in order to create 'main' the 'main.o' file must exist.  The absence of a recipe relies on the implicit rules.  This can be observed by looking at the output when running make as below;
$ make
cc    -c -o main.o main.c
cc   main.o   -o main

The existence of implicit rules comes with some disadvantages, namely it's easy to not understand what is being done for you.  Conceptually, the implicit rule that generates the object files takes the form of the prefix rule below;
$ cat Makefile 
main: main.o

.c.o:
${CC} ${CPPFLAGS} ${CFLAGS} -c $^ 

Understanding what is going on allows tailoring the behavior without explicitly defining a rule.  Note the usage of the CPPFLAGS and CFLAGS variables.  Tweaking the original makefile will allow us to add debugging info and specifying an optimization level 3 as below;

$ cat Makefile 
CFLAGS += -g -o3
main: main.o

This results in a slight difference when we run make;
$ make
cc -g -o3   -c -o main.o main.c
cc   main.o   -o main

The foundation of make is detecting changes to the prerequisites and determining when the targets need to be remade.  This can be observed by re-running make immediately after running make, the result is a notification that "'main' is up to date".  Affecting the main.c file timestamp by modifying the file or simply touching it will result  in the need for the rule to be applied once again.

$ make
cc -g -o3   -c -o main.o main.c
cc   main.o   -o main
user@kaylee:~/make.blog/C$ make
make: `main' is up to date.
user@kaylee:~/make.blog/C$ touch main.c 
user@kaylee:~/make.blog/C$ make
cc -g -o3   -c -o main.o main.c
cc   main.o   -o main

Likely, you've seen this all before, but stay with me I assure you there's more interesting things to come.

Often, it's preferred to have a target that cleans up the directory and allows building from scratch.  The convention is to name such a target clean.  Below is a modified makefile that defines a clean target that simply deletes the executable and the object files.

CFLAGS += -g -o3
main: main.o

clean:
        ${RM} main main.o

Executing 'make clean' will result in deleting main and main.o files.  Adding a file to your project can be accomplished by adding the object file to the prerequisites for main and recipe for the clean target or we can make use of pattern.  We'll do this by explicitly defining each of the C source files in a variable, then perform a list replacement substituting the *.c with *.o extensions to get our object file list.  The object file list can then be used in the target prerequisites and in the clean target recipe.  Adding a file to the SRCS variable rather than duplication in multiple locations.

$ cat Makefile 
CFLAGS += -g -o3
SRCS=main.c 
OBJS=$(subst .c,.o,${SRCS})
main: ${OBJS}

clean:
${RM} main ${OBJS}


Still however there is duplication, namely the multiple references of main, that can be addressed by a new variable definition.

$ cat Makefile 
CFLAGS += -g -o3
PROGS=main
SRCS=main.c 
OBJS=$(subst .c,.o,${SRCS})
${PROGS}: ${OBJS}

clean:
${RM} ${PROGS} ${OBJS}

Definitely on the right path, but the addition of a file requires modification to the makefile.  The wildcard expansion demonstrated in the following makefile.  The addition or removal of a file with the .c extension in the current directory will take effect in the wildcard expansion.

$ cat Makefile 
CFLAGS += -g -o3
PROGS=main
SRCS=${wildcard *.c}
OBJS=$(subst .c,.o,${SRCS})
${PROGS}: ${OBJS}

clean:
${RM} ${PROGS} ${OBJS}



Let's look at some less typical usages of make which gives us a bit more insight into the creation of targets, prerequisites, and recipes.  Imagemagick is a common utility that we'll be making use in the following examples.

We'll build up the makefile as we go, incorporating what we've learned above.  We'll be satisfying the same objectives using two forms of makefiles; one that makes use of suffix rules, one that makes use of pattern rules.

Let's begin by defining our objectives.  Suppose our project requires taking in a list of JPG files and converting each into a series of other image file formats, namely PNG, GIF, JP2 and XWD files.

Using the suffix rule syntax, the makefile can begin taking the following form;

$ cat Makefile.suffix 
.SUFFIXES:
.SUFFIXES: .jpg .png .gif .jp2 .xwd 

all: image.xwd 

.jpg.png:
${SH} convert $< $@

.png.gif:
${SH} convert $< $@

.gif.jp2:
${SH} convert $< $@

.jp2.xwd:
${SH} convert $< $@

clean:
${RM} *.gif *.jp2 *.xwd

The all target consists of the default target, the prerequisite of image.xwd.  In other words, make is complete when an up-to-date image.xwd file exists.  How it arrives at it is make magic, more precisely a series of suffix rules.  A series of prefix rules chaining is required to get to the final XWD file, each target we can kick off by explicitly specifying on the command line.  Specifying 'make -f Makefile.suffix image.png' results in firing of the .jpg.png suffix rule.  The suffix rules are chained as each must fire to arrive at the final XWD file.  Running 'make' performs this by stepping through a series of recipes; JPG => PNG => GIF => JP2 => XWD.
$ make -f Makefile.suffix
convert image.jpg image.png
convert image.png image.gif
convert image.gif image.jp2
convert image.jp2 image.xwd
rm image.jp2 image.gif image.png

Notice the final step removes intermediate files which can be preserved which can be prevented by adding ".PRECIOUS: %.jpg %.png %.gif %.jp2 %.xwd" line which tells make not to remove the intermediate files with the specified extensions.  The example is a bit fictional but done to demonstrate suffix rules and chaining.  Modifying the source file image.jpg followed by rerunning make will result in converting the new file to each of the alternative file formats.

As is, each prerequisite is generated via suffix rule chaining to completion before moving on to the next prerequisite.  In other words, if you specified image.xwd and image01.xwd the image.xwd would be generated to completion (ie. JPG => PNG => GIF => JP2 => XWD) before moving on to image01.xwd.

Meeting the same goals, let's utilize pattern rules rather than suffix rules which are somewhat dated in use.

$ cat Makefile.pattern 
.PRECIOUS: %.jpg %.png %.gif %.jp2 %.xwd

all: image.xwd

%.png:%.jpg
${SH} convert $< $@

%.gif:%.png
${SH} convert $< $@

%.jp2:%.gif
${SH} convert $< $@

%.xwd:%.jp2
${SH} convert $< $@
        
clean:
${RM} *.gif *.jp2 *.xwd

The most noteworthy difference between the target/prerequisites.  The prefix rules define a target, the pattern rule defines a target and prerequisite making the illusion of the rules being reversed.

What if you want to convert each input images to Jpgs before moving on to Gifs before moving on to Jp2s before the Xwds.  This can be done by specifying a wildcard expansion for the source files and using the substitution expression for each of the formats then specifying each of the formats in as prerequisites for the all target, as follows;
$ cat Makefile.pattern 
.PRECIOUS: %.jpg %.png %.gif %.jp2 %.xwd

SRCS=${wildcard *.jpg}
PNGS=$(subst .jpg,.png,${SRCS})
GIFS=$(subst .jpg,.gif,${SRCS})
JP2S=$(subst .jpg,.jp2,${SRCS})
XWDS=$(subst .jpg,.xwd,${SRCS})

all: ${PNGS} ${GIFS} ${JP2S} ${XWDS}

%.png:%.jpg
${SH} convert $< $@

%.gif:%.png
${SH} convert $< $@

%.jp2:%.gif
${SH} convert $< $@

%.xwd:%.jp2
${SH} convert $< $@
        
clean:
${RM} ${PNGS} ${GIFS} ${JP2S} ${XWDS}

This way, each format is fully satisfied before moving on to the next.  Perhaps less necessary for image files, more applicable for generating source files.  For example, the Protobuf message compiler allows generation of header/c++ files which also allows interdependencies between message files.  This requires all the message files to be converted to C/H files before firing the compilation, otherwise a source file may reference a header file that hasn't been created yet.


Hope you find this useful.  Good luck with your make projects!



01 February, 2015

Setting Up Android For the Nth Time

Since I've been authoring Android apps, I've gone through numerous occasions of installing/re-installing/re-re-installing of the SDK and more recently AndroidStudio.  I've been a bit out of the loop, a bit of a hiatus from development but recently reacquainting with the SDK.

Start by downloading and extracting the SDK;

$ wget http://dl.google.com/android/android-sdk_r24.0.2-linux.tgz

$ tar -zxvf android-sdk_r24.0.2-linux.tgz


Append the locations of the new utilities to your Bash profile;


$ echo "export PATH=\$PATH:~/android-sdk-linux/tools" >> ~/.bashrc

$ echo "export PATH=\$PATH:~/android-sdk-linux/platform-tools" >> ~/.bashrc

The default install generally installs the latest APIs, but you can install additional ones;

$ android

Likely, if you've recently updated or reinstalled the Android SDK, you'll have to update the project XML files.  

The 'update project' command takes a numeric target id, listing the targets and 
find the appropriate id.

$ android list targets | grep id:
id: 1 or "android-7"
id: 2 or "android-8"
id: 3 or "android-21"
id: 4 or "Google Inc.:Google APIs:7"
id: 5 or "Google Inc.:Google APIs:8"
id: 6 or "Google Inc.:Google APIs:21"
The current API version is stored in the project.properties file within the project directory.


$ cat project.properties
.
.
target=Google Inc.:Google APIs:8
.
.

Updating the project to continue to use API 8 would therefore be done by;


$ android update project -p . -t 5

Get the list of devices available by issuing the following command;


$ adb devices
List of devices attached 
01aa7bcb8c8c07be device

$ adb -s 01aa7bcb8c8c07be install -r ./bin/Test01-debug-unaligned.apk


Whole buncha interesting stuff is available by examining the APK;

$ aapt dump badging ./bin/Test01-debug-unaligned.apk
package: name='com.abc.test01' versionCode='3' versionName='1.2' platformBuildVersionName=''
application-label:'Test01'
application-icon-160:'res/drawable/icon.png'
application: label='Test01' icon='res/drawable/icon.png'
application-debuggable
launchable-activity: name='com.abc.test01.Test01'  label='Test01' icon=''
uses-permission: name='android.permission.INTERNET'
uses-permission: name='android.permission.READ_PHONE_STATE'
uses-permission: name='android.permission.ACCESS_NETWORK_STATE'
sdkVersion:'7'
targetSdkVersion:'7'
feature-group: label=''
  uses-feature: name='android.hardware.screen.portrait'
  uses-implied-feature: name='android.hardware.screen.portrait' reason='one or more activities have specified a portrait orientation'
  uses-feature: name='android.hardware.touchscreen'
  uses-implied-feature: name='android.hardware.touchscreen' reason='default feature for all apps'
main
other-activities
supports-screens: 'small' 'normal' 'large'
supports-any-density: 'true'
locales: '--_--'
densities: '160'
That's all for now.  Cheers.






31 January, 2015

Android -- Introduction to Fragments (Part 2)

Building off our last example, we'll spend some time demonstrating how to introduce multiple fragments into a single activity.  Along the way, we'll pick up a little more knowledge on how the activities and fragment interaction with one another.

Let's start by extending the res/layout/main.xml to have two layouts as follows;


$ cat res/layout/main.xml 
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:orientation="vertical"
    android:layout_width="fill_parent"
    android:layout_height="fill_parent"
    >
    <LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
        android:id="@+id/container"
        android:orientation="vertical"
        android:layout_width="fill_parent"
        android:layout_height="wrap_content"
        >
    <TextView
        android:layout_width="fill_parent"
        android:layout_height="wrap_content"
        android:text="Hello World, FragTest01"
        />
    </LinearLayout>
    <LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
        android:id="@+id/container2"
        android:orientation="vertical"
        android:layout_width="fill_parent"
        android:layout_height="wrap_content"
        >
    <TextView
        android:layout_width="fill_parent"
        android:layout_height="wrap_content"
        android:text="Hello World, FragTest02"
        />
    </LinearLayout>
</LinearLayout>

Notice we've duplicated the original linear layout (id=container) and identified the copy as container2.  In addition, both of these layouts have been contained within a parent layout, the layout_height attributes were modified to keep the first layout from using the full screen.

As you may recall, the activity FragTest01.java specifies adding an object of the MyFragment class into the desired container;


.
.
            getFragmentManager().beginTransaction()
                     .add(R.id.container, MyFragment.newInstance())
                     .commit();
.
.


If we extend on this call as follows we're left with creating two copies of the MyFragment fragment and placing into two independent containers.


.
.
            getFragmentManager().beginTransaction()
                     .add(R.id.container, MyFragment.newInstance())
                     .add(R.id.container2, MyFragment.newInstance())
                     .commit();
.
.



Ehh, not terribly interesting, two copies of the same fragment with the same layout.

We can now create specialized fragments, one for each layout.  We'll leave MyFragment.java/myfrag.xml in the top layout and create new elements MyFragment02.java/myfrag2.xml for the lower frame.

Duplicating res/layout/myfrag.xml, we simply change the text to demonstrate the difference from the original.

$ cat res/layout/myfrag2.xml 
<?xml version="1.0" encoding="utf-8"?>
   <LinearLayout
   xmlns:android="http://schemas.android.com/apk/res/android"
   android:orientation="horizontal"
   android:layout_width="fill_parent"
   android:layout_height="fill_parent"
   android:background="#666666">
   <TextView
   android:layout_width="fill_parent"
   android:layout_height="wrap_content"
   android:text="My Fragment Example 02"
   android:textColor="#000000"
   android:textSize="20px" />
</LinearLayout>

Similarly, we'll duplicate MyFragment.java for a new class, inflating from the new layout file.


$ cat src/com/fsk/example/FragTest01/MyFragment02.java

package com.fsk.example.FragTest01;

import android.os.Bundle;
import android.app.Fragment;
import android.view.ViewGroup;
import android.view.View;
import android.view.LayoutInflater;

public class MyFragment02 extends Fragment {
    public View onCreateView(LayoutInflater inflater, ViewGroup container,
        Bundle savedInstanceState) {
        // Inflate the layout for this fragment
        return inflater.inflate(R.layout.myfrag2, container, false);
    }

   public static Fragment newInstance() {
        MyFragment02 fragment = new MyFragment02();
        fragment.setRetainInstance(true);
        return fragment;
    }

}


Finally, return to the activity and add unique fragments rather than adding two copies of the original.

             .

             .

            getFragmentManager().beginTransaction()

                     .add(R.id.container, MyFragment.newInstance())

                     .add(R.id.container2, MyFragment02.newInstance())

                     .commit();

             .

             .



We end with two specialized fragments within the same activity;

Nothing sexy, but a toy example that demonstrates tying and linking activities and fragments.

Cheers.

Android -- Introduction to Fragments (Part 1)

Life gets busy. As a result, I took a hiatus from Android development for a period of several months. What I returned to; the concept of fragments, support libraries, Android Studio and with it a new file structure, new IDE and numerous new technologies for Android development.

Surrounded by new concepts and tools, one must focus on ramping back up one area at a time. With the objective of increased productivity, tools such as Android Studio do much for the developer hiding subtleties from the developer by automating a good deal of the heavy lifting. While I fully recommend use of such tools, the course of this post will be working with the SDK, no IDE, to better understand the subtleties and reduce the complexity of the files and structures.

Let's start by creating a project;

$ android list
targets | grep id:
id: 1 or "android-7"
id: 2 or "android-8"
id: 3 or "android-21"
id: 4 or "Google Inc.:Google APIs:7"
id: 5 or "Google Inc.:Google APIs:8"
id: 6 or "Google Inc.:Google APIs:21"

Targeting Google API v21, we specify target 6.

$ android create project --name FragTest01 --path ./ws/FragTest01 --target 6 --package com.fsk.example.FragTest01 --activity FragTest01
Build it by executing the build via Ant;

$ cd ws/FragTest01
$ ant debug
$ adb devices
List of devices attached 
01aa7bcb8c8c07be device

With a connected device, start the logcat service, followed by installing and running the app;

$ xterm -e adb logcat &amp;
$ adb -s 01aa7bcb8c8c07be install -r ./bin/FragTest01-debug-unaligned.apk
$ adb -s 01aa7bcb8c8c07be shell am start -a android.intent.action.MAIN -n com.fsk.example.FragTest01/.FragTest01


We now have an initial project set up, we can continue to play.

First modification we'll make is to the res/layout/main.xml file that was generated as part of the project creation. We'll simply be adding an id to the layout, the usage will become apparent later.


$ cat res/layout/main.xml 
<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:id="@+id/container"
    android:orientation="vertical"
    android:layout_width="fill_parent"
    android:layout_height="fill_parent"
    >
<TextView  
    android:layout_width="fill_parent" 
    android:layout_height="wrap_content" 
    android:text="Hello World, FragTest01"
    />
</LinearLayout>


We'll move along next by creating our first fragment. Before we do so though, let's spend a bit of time discussing the concept of fragments.

The Android developers website does solid job describing the concept of fragments, I'd recommend spending some candle-lit quality time with it, but for now the Cliff Notes for fragments are:

  • Fragments represent behavior, or portion of user interface in an activity. Think of fragments as a new container to stuff behaviors previously localized in activities with the overall goal of better supporting evolving screen sizes and resolutions.
  • A fragments meaning of life is to be contained within an activity, the activity gives the fragment(s) a purpose in life.

Since a fragment represents a portion of the UI, let's create a layout file for it;

$ cat res/layout/myfrag.xml 
<?xml version="1.0" encoding="utf-8"?>
   <LinearLayout
   xmlns:android="http://schemas.android.com/apk/res/android"
   android:orientation="horizontal"
   android:layout_width="fill_parent"
   android:layout_height="fill_parent"
   android:background="#666666">
   <TextView
   android:layout_width="fill_parent"
   android:layout_height="wrap_content"
   android:text="My Fragment Example"
   android:textColor="#000000"
   android:textSize="20px" />
</LinearLayout>

Linear layout, text view with some static text.
Next, we'll create a fragment that uses this layout.


$ cat src/com/fsk/example/FragTest01/MyFragment.java

package com.fsk.example.FragTest01;

import android.os.Bundle;
import android.app.Fragment;
import android.view.ViewGroup;
import android.view.View;
import android.view.LayoutInflater;

public class MyFragment extends Fragment {
    public View onCreateView(LayoutInflater inflater, ViewGroup container,
        Bundle savedInstanceState) {
        // Inflate the layout for this fragment
        return inflater.inflate(R.layout.myfrag, container, false);
    }
   public static Fragment newInstance() {
        MyFragment fragment = new MyFragment();
        fragment.setRetainInstance(true);
        return fragment;
    }
}
The two key methods of interest; onCreateView() and newInstance().   
The onCreateView() method is part of the Fragment Lifecycle, it's purpose is to return a View that is the root of the fragment layout. This is done in this case by inflating the myfrag.xml layout file.


The newInstance() method returns a new instance of the fragment, used by the activity which is shown below;

$ cat src/com/fsk/example/FragTest01/FragTest01.java 
package com.fsk.example.FragTest01;

import android.app.Activity;
import android.os.Bundle;
import java.util.Timer;
import java.util.TimerTask;

public class FragTest01 extends Activity
{
    /** Called when the activity is first created. */
    @Override
    public void onCreate(Bundle savedInstanceState)
    {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);
        init();
    }

    private void init()
    {
       Timer t=new Timer();
       t.schedule(new TimerTask() 
       {
           public void run()
           {
            getFragmentManager().beginTransaction()
                     .add(R.id.container, MyFragment.newInstance())
                     .commit();
           }
       },5000);
    }
}


The activity simply sets the content view, represented by main.xml, then schedules a timer to add (read that as create) the fragment after 5 seconds. Adding/removing/replacing of fragments is done in the form of 'transactions' which are executed on the UI thread some time after being committed.


About as simple of an example as there is. We'll expand on it in later posts.

Full source for this example can be retrieved here.

Cheers.

26 May, 2014

DVD Backup on Linux

It's been years now, all my primary computers have Linux as the host operating system.  Few things can't be done in Linux, but one in particular is backing up dvd's which required the installation of VirtualBox and Windows VM.  I've been using that setup for some time to backup dvd's, with applications such as DVDFab, DvdShrink or the like.  With WinXp at EOL and Nero failing to work with Win7 VMs I finally gave it up after finding an alternative application for native Linux.

K9copy seems to scratch the itch.  Using Ubuntu 12.04, I found setting it up takes the form:


$ sudo apt-get install libdvdcss libdvdcss2
$ sudo apt-get update
$ sudo apt-get install libdvdread4
$ sudo /usr/share/doc/libdvdread4/install-css.sh
$ sudo apt-get install k9copy


http://k9copy.sourceforge.net/web/index.php/en/

Happy viewing.

03 May, 2014

Google Archive

Did you know that you can export all your Google artifacts?

https://www.google.com/takeout
Check the items you want to export and kick out a Zip file.

Cheers.

11 April, 2014

Generating Multi-Plot Real-Time Plots with Python

In my last post the real-time plotting capabilities were demonstrated, we're extending on this by showing how to generate multiple plots simultaneously.  A couple noteworthy observations, in the past post the X and Y scaling was automatically scaled after each element addition.  While you can still do this, typically for multiplots we would prefer maintaining a shared X range.  While somewhat unnecessary, I've elected to maintain a uniform Y range.



#!/usr/bin/python
from pylab import *;
import time;

def log(M):
  print "__(log) " + M;

def test02():
  plt.ion();
  fig=plt.figure(1);
  ax1=fig.add_subplot(311);
  ax2=fig.add_subplot(312);
  ax3=fig.add_subplot(313);
  l1,=ax1.plot(100,100,'r-');
  l2,=ax2.plot(100,100,'r-');
  l3,=ax3.plot(100,100,'r-');
  time.sleep(3);

  D=[];
  i=0.0;
  while (i < 50.0):
    D.append((i,sin(i),cos(i),cos(i*2)));
    T1=[x[0] for x in D];
    L1=[x[1] for x in D];
    L2=[x[2] for x in D];
    L3=[x[3] for x in D];

    l1.set_xdata(T1);
    l1.set_ydata(L1);

    l2.set_xdata(T1);
    l2.set_ydata(L2);

    l3.set_xdata(T1);
    l3.set_ydata(L3);

    ax1.set_xlim([0,50]);
    ax2.set_xlim([0,50]);
    ax3.set_xlim([0,50]);
    ax1.set_ylim([-1.5,1.5]);
    ax2.set_ylim([-1.5,1.5]);
    ax3.set_ylim([-1.5,1.5]);

    plt.draw();
    i+=0.10;
  show(block=True);

#---main---
log("main process initializing");
test02();
log("main process terminating");

Easy Peasy;

04 April, 2014

Generating Real-Time Plots with Python

In my previous post we described plotting data using MatplotLib utilities and Python.  While this may be valuable, it becomes notably more valuable when you can generate 'live' plots during run-time.  In a past employment I worked with a series of controls engineers that utilized real-time data plots to debug and develop a highly complex multi-axis weapons system and it was the first time I understood how a real-time plot of sequence of steps simplified the development effort.

Let's get started.
Unlike the previous post, let's create the data and plot it as it is generated.


#!/usr/bin/python
from pylab import *;
import time;

def log(M):
  print "__(log) " + M;

def test01():
  plt.ion();
  fig=plt.figure(1);
  ax1=fig.add_subplot(111);
  l1,=ax1.plot(100,100,'r-');

  D=[];
  i=0.0;
  while (i < 50.0):
    D.append((i,sin(i)));
    T=[x[0] for x in D];
    L=[x[1] for x in D];
    l1.set_xdata(T);
    l1.set_ydata(L);
    ax1.relim();
    ax1.autoscale_view();
    plt.draw();
    i+=0.10;
    time.sleep(1/10.0);
  show(block=True);

#---main---
log("main process initializing");
test01();
log("main process terminating");

The result is a dynamically generated plot that resembles the following;

Tie this plotting routine to a system providing run-time information via a socket, or perhaps monitoring network traffic via pcapture libraries and you've got yourself the foundation of a real-time data monitoring system.

Cheers.

01 April, 2014

Plotting with Python

Scripting languages are incredibly powerful, but more powerful when you can visualize the data you are processing.  In this post, we will demonstrate how to quickly plot data sets via Python.

Start with installing Python and a plotting utility known as MatplotLib;


$ sudo apt-get install python python-matplotlib 
Then, let's start with a classic plot, sin(x);



#!/usr/bin/python
from pylab import *;
import time;

def log(M):
  print "__(log) " + M;


def test00():
  D=[];
  i=0.0;
  while (i < 50.0):
    D.append((i,sin(i)));
    i+=0.10;
 
  plt.ion();
  xlabel('radians');
  ylabel('sin(x)');
  grid(True);
  plt.figure(1);
  show();
  T=[x[0] for x in D];
  L=[x[1] for x in D];
  plt.plot(T,L,'b-');
  show(block=True);

#---main---
log("main process initializing");
test00();
log("main process terminating");
The result is calculating a data set followed by plotting the data and allowing the user to manipulate the plots (e.g. zooming, panning, ...).



For more detailed features, refer to the MatlabLib site: http://matplotlib.org/contents.html

Cheers.

27 March, 2014

Tcpdump Examples

As much as I like using Wireshark, I'm a sucker for a command interface.  The trouble is properly defining a filter has been difficult to say the least, boring simple post, but below is a list of brief examples and descriptions.

#!/bin/bash

#--grab the first 10 packets and write to file
  tcpdump -c 10 -i wlan0 -w /tmp/data.pcap

#--grab the first 10 packets to/from host and write to file
  tcpdump -c 10 -i wlan0 host 192.168.1.125 -w /tmp/data.pcap

#--grab the first 10 packets from host and write to file
  tcpdump -c 10 -i wlan0 src host 192.168.1.125 -w /tmp/data.pcap

#--grab the first 10 packets to host and write to file
  tcpdump -c 10 -i wlan0 dst host 192.168.1.125 -w /tmp/data.pcap

#--grab the first 10 packets to/from port and write to file
  tcpdump -c 10 -i wlan0 port 80 -w /tmp/data.pcap

#--grab the first 10 packets to/from net and write to file
  tcpdump -c 10 -i wlan0 net 192.168.1 -w /tmp/data.pcap

#--grab the first 10 packets from network and write to file
  tcpdump -c 10 -i wlan0 src net 192.168.1 -w /tmp/data.pcap

#--grab the first 10 packets from network and write to file
  tcpdump -c 10 -i wlan0 dst net 192.168.1 -w /tmp/data.pcap

#--grab the first 10 packets from network and write to file
  tcpdump -c 10 -i wlan0 '(dst host 192.168.1.125) and (port 443)' -w /tmp/data.pcap

08 December, 2013

Useful FFMpeg Filter






FFMpeg offers more than standard encoding/decoding.  It also allows for running a series of filters, both audio and video, that may prove useful in many a situation.

Let's start with an good quality input video;


$ ffmpeg -y -i video.avi -ss 10 -t 40 -qscale 0 -s 360x240 clip01.mpg















Ok, now that we have a video let's look at a few things we may choose to do with it using our newly discovered filters.

Adjusting Video Speed

Suppose you wish to speed the video up, perhaps to expedite review of a surveillance video or perhaps minimizing the exposure to the latest Sandra Bullock film.

We can double the playback speed by specifying a 0.5 playback coefficient;

$ ffmpeg -y -i clip01.mpg -filter:v "setpts=0.5*PTS" clip02.mpg
















You'll notice however from the previous video that while the video element is playing at double speed, the audio hasn't changed.  We can speed up the audio similarly by adding an appropriate audio filter.



$ ffmpeg -y -i clip01.mpg -filter:v "setpts=0.5*PTS" -filter:a "atempo=2.0" clip03.mpg


















Attempting to speed up faster than 2x will present a hurdle with speeding up the audio filter, which specifies a coefficient range of [0.5, 2.0].  You can get around that by chaining filters;

$ ffmpeg -i clip01.mpg -filter:v "setpts=0.25*PTS" -filter:a "atempo=2.0,atempo=2.0" clip04.mpg






Cropping Video

Suppose we wish to crop the inner middle of the video.  Given the video is 360x240, we need to grab 180x120 inset by 90,60.

$ ffmpeg -y -i clip01.mpg -filter:v "crop=180:120:90:60" clip05.mpg






Processing Edge Detection

Perhaps less useful, but certainly interesting, you can perform edge detection via a Canny Edge Detection algorithm;
$ ffmpeg -y -i clip01.mpg -filter:v "edgedetect=low=0.1:high=0.4" clip06.mpg







Flipping the Video

You can flip the video left-right or top-down;
$ ffmpeg -y -i clip01.mpg -filter:v "hflip" clip07.mpg















$ ffmpeg -y -i clip01.mpg -filter:v "vflip" clip08.mpg











Creating Video Using Source Filter

Occasionally, you want to generate a video using a test source; in other words, without an input video to start with.
$ ffmpeg -f lavfi -i rgbtestsrc -t 30 -pix_fmt yuv420p clip09.mpg















$ ffmpeg -f lavfi -i testsrc=duration=20:size=1280x720:rate=30 -vcodec mpeg2video clip10.mpg

29 November, 2013

Extracting Video from DVDs

Time and again I find a need to snag video from a DVD and perform some routine video processing.

FFMpeg works nicely with standard video files, less nicely with DVD structures.  While you can certainly extract video directly from the DVD VOB files (ref: http://en.wikibooks.org/wiki/Inside_DVD-Video/Directory_Structure) it's less than desirable since the locating the specific VOB file that contains the segment of video your interested in can be a bit of a setback.

Enter Mplayer;
Mplayer can readily play DVD contents in a user friendly way.  It can also dump the file to a more usable file if you can persuade it to do so.

You can extract the full main title into a single VOB file can be done by playing the DVD somewhat normally, but adding a few arguments;


$ mplayer dvd:// -dumpstream -dumpfile /tmp/video.vob


user@River:~$ 
The above will extract the main title into a single VOB file.  Examining with FFMpeg and you'll find that the output file has a video stream and a single audio stream;



$ ffprobe -i /tmp/video.vob

ffprobe version 1.1.2 Copyright (c) 2007-2013 the FFmpeg developers

  built on Feb 10 2013 17:42:38 with gcc 4.4.5 (Debian 4.4.5-8)

  configuration: --enable-filter=split

  libavutil      52. 13.100 / 52. 13.100

  libavcodec     54. 86.100 / 54. 86.100

  libavformat    54. 59.106 / 54. 59.106

  libavdevice    54.  3.102 / 54.  3.102

  libavfilter     3. 32.100 /  3. 32.100

  libswscale      2.  1.103 /  2.  1.103

  libswresample   0. 17.102 /  0. 17.102

[mpeg @ 0x2cd4c60] max_analyze_duration 5000000 reached at 5004678

Input #0, mpeg, from '/tmp/video.vob':

  Duration: 01:37:58.02, start: 0.280633, bitrate: 5012 kb/s

    Stream #0:0[0x1e0]: Video: mpeg2video (Main), yuv420p, 720x480 [SAR 32:27 DAR 16:9], 26.50 fps, 59.94 tbr, 90k tbn, 59.94 tbc

    Stream #0:1[0x80]: Audio: ac3, 48000 Hz, 5.1(side), fltp, 448 kb/s


You may choose an alternative container rather than a VOB by specifying an alternative extension, such as an AVI container by specifying output file video.avi.

Now, we have something we can readily work with.

Let's snag a 30 sec clip starting at 90 seconds in:




$ ffmpeg -i /tmp/video.vob -qscale 0 -ss 90 -t 30 /tmp/clip.mpg



If you're only interested in the video, you can extract only the video by specifying;


$ mplayer dvd:// -dumpvideo -dumpfile /tmp/video01.avi


Or, if you're only interested in listening to the Big Bang Theory like an audio tape, extract only the audio;




$ mplayer dvd:// -dumpaudio -dumpfile /tmp/video01.aac




Cheers.

24 October, 2013

Create Counting Video


On a couple of occasions, I've had the need to create a source video for video processing and testing video processing commands.  A common need is to create a counting video by generating a series of frames and stitching them together at a pre-defined frame rate.

Below is a mechanism for doing so;


$ cat createVideo 
#!/bin/bash

ws=/var/tmp/cache
rm -rf $ws
mkdir -p $ws

for i in `seq 300`; do
  N=$(printf %03d $i)
  convert -size 640x480 -background white -fill black -pointsize 120 -gravity center label:$i $ws/frame-$N.jpg
done

ffmpeg -y -r 2/1 -i $ws/frame-%03d.jpg -r 30/1 $ws/video.avi
mplayer $ws/video.avi


Cheers.

18 October, 2013

Formatting Numerics With Bash

A repeated need I've encountered is a need to format a numeric with leading zeros, similar to the common form used in C.  Typically, I take a over-complicated approach of comparing the number to >100 or >10 and pre-pending leading zeros.

After investigating alternative approaches with good 'ole Google, the better approach is shown below;


$ cat /tmp/go 
#!/bin/bash

for i in `seq 100`; do
  N=$(printf %03d $i)
  echo $N
done

Cheers.

11 October, 2013

FFMpeg Cropping Videos

Recently had a need to crop videos using FFMpeg, thought it worth sharing. Starting with the initial video:

$ ffmpeg -i ExSlyoVTX3I.flv -vf crop=160:120:10:10 -sameq /tmp/out.mp4

The result is a cropped video 160x120 with an offset of 10,10 from the upper left-hand corner.


Cheers.

05 October, 2013

Google ProtoBuff + ZeroMq -- Python

In our last post, we combined ZeroMq & Google Protobuf, a combination that enables heterogeneous distributed computing systems.  The last post integrated the two using C++, this post will focus on integrating the two using Python.  In the end, we'll have an example that can communicate with the previous C++ example seamlessly.

Below is a Makefile, used to create and clean up Google Protobuf message files.


$ cat Makefile

msgs:

 ${SH} protoc -I=. --python_out=. Messages.proto



clean:

 ${RM} Messages_pb2.py


Our sender script is below, notice the change is that of the content exchanged, specifically the serialized protobuff message.


$ cat sender
#!/usr/bin/python
import zmq;
import time;
import Messages_pb2
context = zmq.Context();
pub=context.socket(zmq.PUB);
pub.bind("tcp://127.0.0.1:8000");
for i in range(0,10):
  p=Messages_pb2.Person();
  p.id=i;
  p.name="fatslowkid";
  print "iteration",i
  pub.send(p.SerializeToString());
  time.sleep(1);

The receiver;

$ cat receiver
#!/usr/bin/python
import zmq;
import time;
import Messages_pb2
context = zmq.Context();
sub=context.socket(zmq.SUB);
sub.connect("tcp://127.0.0.1:8000");
filter=""
sub.setsockopt(zmq.SUBSCRIBE, filter);
for i in range(0,20):
  print "waiting on msg"
  M=sub.recv();
  p=Messages_pb2.Person();
  p.ParseFromString(M);
  print "received",p
  print "> " + p.name;
  print p.id;

The sender/receiver can be used together, or used with the previous C++ example.

Cheers.

28 September, 2013

Google ProtoBuff + ZeroMq -- C++

In the last series of posts we demonstrated ZeroMq as a technology that supports 'sockets on steroids', supporting multiple platforms as well as multiple languages.  The examples to-date have been transmitting strings between senders and receivers.  While interesting, to effectively create a distributed heterogeneous system we need to be capable of transmitting meaningful messages, preferably complex data structures rather than just strings.  That's where Google's Protobuff comes into play: http://code.google.com/p/protobuf/

Building off our previously created Ubuntu 12.04 32-bit VM, let's start by installing the additional necessary packages;


$ sudo apt-get install libprotoc-dev

With the developer libraries installed, we can now extend our previous C++ example to transmit a ProtoBuff message.

We'll extend our Makefile to add the necessary libraries and a target (e.g. msgs) to generate the C++ files for the message.


$ cat Makefile 
CC=g++
SRCS=main.cpp Messages.pb.cc
OBJS=$(subst .cpp,.o,$(SRCS))
INCLUDES += -I.
LIBS += -lpthread -lrt -lzmq -lprotobuf
.cpp.o:
$(CC) -c $<
main: msgs ${OBJS} 
${CC} ${CFLAGS} -o $@ ${OBJS} ${LIBS}
msgs:
${SH} protoc -I. --cpp_out=. Messages.proto
clean:
${RM} ${OBJS} main *.pb.*

Oh, we should take a look at our simple Protobuff message file:

$ cat Messages.proto 
message Person {
  required int32 id=1;
  required string name=2;
}

Finally, our extended main file:

$ cat main.cpp 
#include
#include
#include
#include
#include
#include
#include "Messages.pb.h"
void* ctx=zmq_init(1);
char* EndPoint="tcp://127.0.0.1:8000";
static const int N=100;
static const int BufferSize=128;
void* sender(void*)
{
  printf("(%s:%d) running\n",__FILE__,__LINE__);
  void* pub=zmq_socket(ctx, ZMQ_PUB);
  assert(pub);
  int rc=zmq_bind(pub,EndPoint);
  assert(rc==0);
  Person p;
  p.set_name("fatslowkid");
  p.set_id(01);
  for(int i=0; i
  {
    zmq_msg_t msg;
    std::string S=p.SerializeAsString();
    char* content=(char*)S.c_str();
    int rc=zmq_msg_init_size(&msg, BufferSize);
    assert(rc==0);
    rc=zmq_msg_init_data(&msg, content, strlen(content), 0,0);
    assert(rc==0);
    rc=zmq_send(pub, &msg, 0);
    assert(rc==0);
    ::usleep(100000);
  }
}
void* receiver(void*)
{
  printf("(%s:%d) running\n",__FILE__,__LINE__);
  void* sub=zmq_socket(ctx, ZMQ_SUB);
  assert(sub);
  int rc=zmq_connect(sub,EndPoint);
  assert(rc==0);
  char* filter="";
  rc=zmq_setsockopt(sub, ZMQ_SUBSCRIBE, filter, strlen(filter));
  assert(rc==0);
  for(int i=0; i
  {
    zmq_msg_t msg;
    zmq_msg_init_size(&msg, BufferSize);
    const int rc=zmq_recv (sub, &msg, 0);
    char* content=(char*)zmq_msg_data(&msg);
    Person p;
    p.ParseFromString(content);
    printf("(%s:%d) received: '%s'\n",__FILE__,__LINE__,p.name().c_str());
    zmq_msg_close(&msg);
  }
}
int main(int argc, char* argv[])
{
  printf("(%s:%d) main process initializing\n",__FILE__,__LINE__);
  int major, minor, patch;
  zmq_version (&major, &minor, &patch);
  printf("(%s:%d) zmq version: %d.%d.%d\n",__FILE__,__LINE__,major,minor,patch);
  pthread_t rId;
  pthread_create(&rId, 0, receiver, 0);
  pthread_t sId;
  pthread_create(&sId, 0, sender, 0);
  pthread_join(rId,0);
  pthread_join(sId,0);
  printf("(%s:%d) main process terminating\n",__FILE__,__LINE__);
}

Notice that we now transmit and receive Protobuf messages, serialized as strings.  The value of this is that the serialization mechanism is multi-platform & multi-language support.

Cheers.