Augmenting the iPAQ with Sensor Boards via the Serial Port

back to notes

 

The iPAQ
Compaq's iPAQ is a PDA that is easy to wear and yet powerful enough to do serious processing of sensor data. Many people have suggested that it can easily be used as a wearable computer. Also see these notes for details on running Linux on the iPAQ.

 

    

Augmenting the iPAQ
The aim is to make the iPAQ a bit more aware of what is going on around it, with minimal feedback from its user. A sensor board is connected to it (via the main ttySA0 serial port) that provides it with data that it can analyse and that can be labelled by the user.

This way, a very flexible annotation/classification process is created that could be used for a wide range of wearable applications.

The cradle can be dissected to use it as a compact and reliable connector for both a serial sensor board (such as the XBow or the TEA2 board discussed here) and power. You will need a special screwdriver and a special connector, unless you don't mind cutting the cable from the cradle.

 

  

The iPAQ, with the small board from an iPAQ cradle attached to it as connector for power and serial data from a sensorboard.

The Crossbow ADXL202EB-232

The Crossbow ADXL202EB-232 is an evaluation board of the ADXL202 by Analog Devices, a very popular dual axis accelerometer, sensing tilt and acceleration in a plane (on the X- and Y-axes). You could use it for detecting and recognizing motions or positions of whatever you attach the device to. See here for some applications.

The board has a serial port that can be connected to a PC to read, store or analyse the sensor data. The protocol is very simple: send an ASCII 'G' over the serial line, and the board will send back four bytes: 2 bytes per sensor. The serial port is set to 38400, 8 data bits, no stop bits, no parity, no flow control. You can use these settings for instance in HyperTerminal (MS Windows) or Minicom (Linux) to test the Crossbow board.

Although the ADXL202EB-232 contains only 2 sensors, it is a widely available off-the-shelf component, so this should make it easier to reproduce research data that is mentioned on these pages later on.

 

  

The TEA2 Board
The TEA2 Board was developed in the TEA project, funded by the European Commission. It contains several sensors:
  • accelerometers (ADXL202),
  • light sensors (2),
  • microphones(2),
  • a temperature sensor (DS1820),
  • and a touch sensor (measuring skin conductivity)
  • also several connections for additional sensors are available

See these notes for the technical details of the board - on both hardware and software level.

 

  

The TEA2 Board with its battery

 

The Sourcecode

Just small test programs/scripts to check on the serial connections between the iPAQ and the XBow and TEA2:
  • xbow.cpp: a small test file that displays the XBow accelerometer readings on the iPAQ screen (via a terminal)
  • tea2.cpp, Fifo.cpp, Fifo.h: a small project that displays and pre-processes the TEA2 sensor readings on the iPAQ screen via a terminal
  • xbow.m, tea2.m: small scripts that read sensor values from the xbow and tea2 board in Matlab

The software that also does clustering and classification is described here, as an example, here's the C++ source code tailored for simple activities:

  • main.cpp: the main file
  • Neuron.cpp, Neuron.h: small class for an artificial neuron
  • SOM.cpp, SOM.h: self-organizing map for clustering the sensor data
  • KMeans.cpp, KMeans.h: k-means clustering for more clustering and storing the labels
  • Fifo.cpp, Fifo.h: first-in-first-out buffer with additional functions
  • Graph.cpp, Graph.h: construction of a Markov chain
  • Coniok.h, Sercom.h: additional i/o routines

 

    

Building the executable
  • Copy the sourcefiles from the previous section
  • and build the executable, by compiling it with an ARM cross-compiler (from ftp://ftp.handhelds.org/pub/linux/arm/toolchain/, for example)
  • connect the Crossbow or TEA2 to COM1 (/dev/ttyS0)
  • run the program
  • annotate at will and see what the iPAQ predicts..

 

Don't forget to check out these notes on building a GUI front-end with Python for a better user interface!

 

  

The output on the iPAQ should look something like this

 

Demonstration

 

Here is a movie (AVI, 34MB!) that briefly illustrates how the system works, training it on the spot for three contexts: "sitting", "standing up", and "running" using the TEA2 board, and FLITE, a text-to-speech engine by CMU.

Remarks:

  • Initially the system has no idea about any context - it also has no idea about what kind of sensor values can typically enter the system

  • The values from all eight sensors were used as input, all equally pre-processed using minimum, maximum, mean, and standard deviation (so: 8x4=32 features)

  • although probably one accelerometer would have been enough,

  • and sensors like the thermometer did probably not contribute to the recognition.

  • The features were clustered on a topographic map, its cells were labelled whenever I pressed a button on the screen.

  • Not really visible: the left column next to the buttons shows a measure of confidence in the classification

 

A movie showing the same with locations: here .

 

  

Example of how the sensor can be attached quickly for experiments and gathering datasets.

 
 

Compiled by Kristof Van Laerhoven.