Salons-tfm
Project card
- Project Name: Autonomous fall detection system that sends alarms to a cell phone
- Authors: Sara Marugán Alonso (smarugan [at] gsyc [point] es) and Jose María Cañas Plaza (jmplaza [at] gsyc [point] es)
- Academic Years: 2009-2010
- Degree: Master
- Jde Version: jderobot-5.0
- SVN Repository: source code (restricted)
- Tags: computer vision, fall detection, 3D tracking, Android
- Technology: Ice, C++, Java, Android
- State: Finished
- Source License: not defined
- Abstract:
The aim of this project is to upgrade Eldercare-3.0 application to platform jderobot-5.0. This permits a higher level of processing distribution and sending fall detection alarm to android cell phone.
Project evolution
Getting started in jderobot-5.0 [6/02/2010]
This fist week I have been studying jderobot-5.0 infrastructure that is based on the Internet Communications Engine (Ice) middleware. ICE is a set of CORBA like components that include object-oriented remote-object-invocation, replication, grid-computing, failover, load-balancing, firewall-traversals, and publish-subscribe services. To gain access to those services, applications are linked to a stub library or assembly, which is generated from a language-independent IDL-like syntax called slice.
Eldercare application will be an ICE component that uses jderobot-5.0 ICE interface definitions and libraries.
First version of eldercare component [12/02/2010]
This version gets images from four different varcolorserver components and reuses eldercare-3.0 GUI to show them. This images belong to video files.
At the moment eldercare component does not support GUI events.
This is the eldercare-4.0 configuration file in jderobot-5.0:
VarColorView.VarColorA.Proxy=varcolorA:tcp -h 127.0.0.1 -p 9999 VarColorView.VarColorB.Proxy=varcolorB:tcp -h 127.0.0.1 -p 9998 VarColorView.VarColorC.Proxy=varcolorC:tcp -h 127.0.0.1 -p 9997 VarColorView.VarColorD.Proxy=varcolorD:tcp -h 127.0.0.1 -p 9996
Also I have written two simple scripts to launch the varcolorserver components and kill them.
Eldercare component based on Model-View-Controller model [18/02/2010]
Second version of eldercare component has been designed in terms of Model-View-Controller. It is interesting because it separates application algorithms and data from GUI visualization. Therefore there are three classes, two of them correspond to the data application model and the GUI manager. The controller class connect them. Also there is a main class that creates and initializes instances of Model-View-Controller classes and connect to image providers.
At the moment eldercare component receives images from video files and applies a motion filter to them. In addition, GUI events are now supported.
First complete Eldercare component [27/02/2010]
This release integrates people evolutive 3D tracking. I have refactored old source code in several files and rebuilt it with C++ compiler.
Now the application Model saves necessary information for the tracking algorithm like configuration constants. That information is available and configurable by the controller. Also receives new images from the controller and use them to make a new algorithm iteration. Then, the controller pass result images to the view class in order to show them on GUI.
I have introduced four auxiliar threads in the component. Each one only receives images from one camera server. It improves time performance from 4 fps to 15 fps for getting new images. So at the moment we are limited to 15 fps, but algorithm sometimes walks more quickly redoing work. We are considering Ice publish/subscribe service (IceStorm) for incrementing this frame rate.
Tracking algorithm works quite well for two people but works slowly for more people as you see in the video. In the near future we will experiment with process parallelization for tracking algorithm.
Time performance analysis [12/03/2010]
| FPS | QuadCore with 5 threads | CoreDuo with 5 threads | CoreDuo with 1 thread |
|---|---|---|---|
| 1 person | 25 | 14 | 2 |
| 2 people | 16 | 9 | 1 |
| 3 people | 12 | 6 | 1 |
This results shows that at the moment Eldercare works quite fine in time performance in a QuadCore processor. But it needs 4 auxiliar threads for getting images from cameras, so I am going to implement an image server component for jderobot-5.0 that uses Ice publish/subscribe service (IceStorm). Then we will compare it to the current image server.
Using Ice publish/subscribe service (IceStorm) [23/03/2010]
I have implemented an image server using IceStorm and I have compared it to traditional client/server camera server.
First thing I did was introducing a new interface for consuming images from publisher. I defined a slice interface like this:
//! Interface to the image consumer.
interface ImageConsumer
{
//! Transmits the data to the consumer.
void report( ImageData obj );
};
Publisher and subscriber use this interface to communicate to each other. Subscribers have to implement the report method it in order to manage the data.
To run IceStorm service is necessary to execute IceBox that is the Ice service container. Also I had to create image topics through icestormadmin tool. An application indicates its interest in receiving messages by subscribing to a topic. An IceStorm server supports any number of topics, which are created dynamically and distinguished by unique names. Each topic can have multiple publishers and subscribers.
First difference we noticed is that RPC call from client/server method cost 35 millisecond. With publish/subscribe service we avoid this cost and it influences on Eldercare time performance. Here we have results for Eldercare time performance analysis. I have used several combinations to get entry images. Some of them use IceStorm publish/subscribe method and others uses traditional client/server method.
| FPS | Eldercare FPS (QuadCore) | Eldercare FPS (CoreDuo) |
|---|---|---|
| ICESTORM 1 THREAD | 17 | 12 |
| ICESTORM 2 THREADS | 17 | 12 |
| CLIENT/SERVER 1 THREAD | 7 | 4 |
| CLIENT/SERVER 5 THREADS | 16 | 12 |
- for this experiment we used 2 people inside the room as the regular example case.
According to the results we will choose IceStorm with one auxiliar thread because it separates getting images task from main algorithm and is equivalent in time performance to use IceStorm with unique process.
Alarm managing from Android cell phone by EldercareClient [4/04/2010]
One important aim for this project is managing alarms from a cell phone. This capacity raises the Eldercare application utility because people can consult the state of the monitored person from their cellulars and receive alarms in case of danger situations.
Jderobot-5.0 offers a component to manage recordings and save information in the data base. This component is called RecordingManager. As you can see in the image below Eldercare makes a request to RecordingManager in order to start or end a recordings or set events about an existing recording. This information is stored on the data base and recovered when RecordingManager receives a cell phone petition.
We have connected an Android cell phone to the application through Ice for Android middelware. We called the client application EldercareClient. Its source code is written in Java through the Eclipse IDE. This component uses Jderobot-5.0 interfaces and interacts to the RecordingManager to get the alarm events or to receive video streaming from a monitoring camera in real time.
These videos shows how the Eldercare alarm managing works. In the first video you can see video streaming from one camera and the second video shows the list of events stored over an alarm recording. Each event has its own asociated image.
Sending an email to warn about a dangerous situation [11/04/2010]
Storing alarms in a BBDD is quite interesting but it is a passive way to know what it is happening in the watched area. We though that the system needs an active alarm to let the system user know that a dangerous situation has occurred.
Assuming that the user has his Android cell phone connected to Internet, we though that the best way to achieve an active alarm is sending an email to his personal electronic address. At the moment Eldercare has its own email account in a server from this university (GSYC). When the system detects a dangerous situation send the email from this account of this server that is public known as a email server.
To do this it needs a Public Key Infrastructure (PKI) that permits the system enter into the server and send the email automatically. Thanks to that, the ssh command does not ask for the password. The following line shows the command that send the email:
ssh eldercare@gsyc.es 'echo "Someone is in danger." | mailx -s "Eldercare alam" user@email'
After this kind of alarm the user can open the cell phone application that we present before. Then alarms and video streaming could be seen.
Recording video [23/04/2010]
We have seen how the system stores alarms in the BBDD and how from the cell phone these alarms can be consulted. Also we showed that the system permits video streaming to visualize a real camera.
Now I am going to present complete recording manager functionality, that consist of recording video after an alarm. I have implemented a recorder object that grabs video from Jderobot camera. It only needs to know the camera endpoint to get the camera proxy and receive images.
The mecanism to record video consist of writting camera images on a fifo and executing mencoder over this fifo in other process (see the diagram below).
Recordings are configurable through Eldercare configuration file. Here we have a snip from this configuration:
Eldercare.RecordingManager.Proxy=recordManager1:tcp -h 0.0.0.0 -p 9990 Eldercare.RecordingManager.dirRecordings=/tmp/eldercare-alarms/ Eldercare.RecordingManager.namePattern=video-XXXXXX Eldercare.RecordingManager.name=cameraA Eldercare.RecordingManager.cameraProxy=IceStorm/TopicManager:default -t 5000 -p 10000 Eldercare.RecordingManager.width=320 Eldercare.RecordingManager.height=240 Eldercare.RecordingManager.fps=25 Eldercare.RecordingManager.duration=15 /*segs*/
This video shows the Eldercare system activating an alarm and them from the cell phone the recorded video can be visualized.
Firewire camera support for CameraServer [5/05/2010]
In order to make experiments with real cameras at the Robotics Laboratory it has been necessary to add firewire support to Cameraserver component of Jderobot-5.0.
We decided that the best option was using libdc1394-22 directly. We wanted to keep the same configuration style so if the user needs to use a firewire camera only has to indicate it in the uri field of configuration file. Like this:
CameraSrv.Camera.n.Name=cameraA CameraSrv.Camera.n.ShortDescription=Camera pluged to /dev/video1394. Only permits three modes: 320x240-15fps, 320x240-30fps or 640x480-15fps. CameraSrv.Camera.n.Uri=dv:///dev/video1394/0 CameraSrv.Camera.n.FramerateN=15 CameraSrv.Camera.n.FramerateD=1 CameraSrv.Camera.n.ImageWidth=640 CameraSrv.Camera.n.ImageHeight=480 CameraSrv.Camera.n.Format=RGB888
Then, Cameraserver initializes the firewire device and obtain frames from it using libdc1394 functions. Therefore, Cameraserver can be use with v4l and firewire devices and with local o remote video files.
System validation at the Robotics Laboratory [13/05/2010]
He have connected each camera to a different machine and executed cameraserver to publish the images over the local net. These machines and main computer have been connected through the same router, as the diagram below shows.
We wanted to check if the system works fine when a person falls in the floor. The application was expected to record video and send an alarm to cell phone. In the following video you can hear a warning sound when the person falls in the floor and then the system alert someone through sending an email to the cell phone. Then, the Eldercare client can consult the alarms in the database and see the associated recordings. Also it can receive live streaming from a camera.
Virtual camera in EldercareClient [23/05/2010]
I have added a new functionality to Android application EldercareClient. Using OpenGL ES API, it allows to visualize the Eldercare virtual camera. Specifically, I use GLSurfaceView class.
I wanted to draw people trajectories on this camera in real time, so I needed to define a new slice interface to communicate Eldercare tracking results to EldercareClient. I defined a race state for a particular instant. It is composed by 3D position (X,Y,Z) and a color (RGB).
//! Description of the race position
class Point3D
{
float x;
float y;
float z;
};
//! Description of the race position
class ColorRGB
{
float r;
float g;
float b;
};
//! Description of the race state
class RaceState
{
//! Identifier of race
string id;
//! Position
Point3D position;
//! Color
ColorRGB color;
};
//! A sequence of race states
sequence<RaceState> RaceSequence;
interface EldercareProvider
{
//! Returns the list of race states
idempotent RaceSequence getRaceStates();
};
When EldercareClient calls getRaceStates() method, receives a sequence of race states that consist of one race state for each race in this instant. EldercareClient has to store the different states of a race along time to construct race trajectory.
Here we have a video that shows how it works. At the backend Eldercare component is executing and its virtual camera shows race states in real time.
Settings in EldercareClient [27/05/2010]
I have added a settings activity to EldercareClient because I consider it necessary for configure application parameters. Until now if I wanted to move a component to another computer I had to recompile EldercareClient in order to indicate the new IP address. The settings activity permits to modify the addresses of the components that EldercareClient connect to. These components are cameraserver (four instances), recordingmanager and Eldercare. Only because of this the application raises its usability and flexibility.
This settings can be saved permanently thanks to an Android tool called Preference Manager. It permits to save key-value pairs and after exiting the application this configuration is saved. Here we have an snip of source code that use it:
SharedPreferences prefs = PreferenceManager.getDefaultSharedPreferences(this); SharedPreferences.Editor spe = prefs.edit(); spe.putString(PROXY_A, string_proxyA); ... spe.commit();
String camAproxy = prefs.getString(PROXY_A, this.getResources().getString(R.string.uri_camA));
On the left this is the new main GUI and on the right the settings activity GUI. On the menu you can confirm changes or cancel it.
Calibrator component [30/05/2010]
Until now we have been calibrating the cameras using an application called extrinsics in jderobot-4.3. Because of this functionality is useful we considered to create a new component in jderobot-5.0 that reuses that source code. We renamed it because it can adjust both intrinsic and extrinsic camera parameters, not only extrinsic ones.
Now, we have an early version of a camera calibrator in platform jderobot-5.0.
System robustness experiments [3/06/2010]
These experiments follow the aim of checking robustness system. We have already proved that all system components integrated works fine but now we want to measure application robustness and improve it. First step was changing firewire cameras to IP Axis 207MW cameras and connected them to main machine through a switch. We could check that the system is robust to hardware changes.
Illumination changes [3/06/2010]
IP cameras are more sensitive to illumination changes and its images have less quality in general. Axis cameras permitted us to notice that the motion filter was not enough robust to illumination changes. Until now motion filter was based in pixel differences on RGB color model. Illumination changes caused by shadows or switching on the lights provoke that motion filter detects movement where it is not. The following images shows this problem. First example shows how person's shadow generate movement and on the second example the person has switched on the lights.
To solve this problem we use HSV (Hue Color Value) color model instead of RGB. The Value coordinate is the key to solve this problem, raising the tolerance umbral for this coordinate the tolerance of motion filter to illumination changes is improved. The same situations are well passed by motion filter now.
Alarm activation [8/06/2010]
A simple experiment that permitted us to know the system adjustment to detect a fall was sitting a person on the floor. In this case the application does not activate the alarm. Nevertheless, if the person lay on the floor the system do activate the alarm. The following image show this behavior.
In order to assess the influence of the furniture on the system we simulated a table and a sofa in the middle of the laboratory. We wanted to know if the application detects the person fall between the sofa and the table. The result of the experiment was successful despite of the occlusion. This video shows the experiment.
Fall detection percentage [11/06/2010]
Another robustness experiment consist of creating a BBDD of falls in order to find out the percentage of system successful fall detections. The following frames belong to several videos where different people fall on the floor.
Because the algorithm works with a random factor we execute the system 10 times over the same enter data. In mean the result was that 90% of the falls were detected. In some cases the person falling on the floor was covered only by two cameras and position estimation error caused that system missed it.
