Prototype for Real Time Data Streaming (Data Push) Part 1

Real time data streaming from remote devices (ie. data pushing) has been a fascinating topic. In this series of blogs, I will examine how to implement a prototype to demonstrate the capability of pushing data from devices (java client) via embedded light weight web server JETTY (JETTY 7 / COMETD 2 / Bayeux Protocol) to your web browser (Javascript / JQuery). The series includes three parts:

Prototype for Real Time Data Streaming (Data Push) Part 1: maven2 generated Jetty based application

Prototype for Real Time Data Streaming (Data Push) Part 2: multi-channel subscription based web application

Prototype for Real Time Data Streaming (Data Push) Part 3: channel feeder java based application
The prerequisites for the prototype are MAVEN2, JETTY 7, JAVA SDK and JQUERY. I am doing on Ubunto 11.04. I believe it can be generalized on any linux distro. Also don’t worry too much on the minor version of all the prerequisites because MAVEN2 will take care of all the software and version dependencies. That’s why we use MVN. We just need to tell it what our goal is, and let it take care of the rest.

Maven 2 Jetty based Web Application

Get Started

I will create a server side web application for Jetty 7 by using mavern2.

 mvn archetype:generate -DarchetypeCatalog=http://cometd.org

It will present a few archetypes to choose. choose the following

4: http://cometd.org -> org.cometd.archetypes:cometd-archetype-jquery-jetty7 (2.4.3 - CometD archetype for creating a server-side event-driven web application)

Then provide some parameters like the following:

Define value for property 'groupId': : henry416      
Define value for property 'artifactId': : DeviceMonitor
Define value for property 'version': 1.0-SNAPSHOT: 
Define value for property 'package': DeviceMonitor: 
[INFO] Using property: cometdVersion = 2.4.3
[INFO] Using property: jettyVersion = 7.6.4.v20120524
[INFO] Using property: slf4jVersion = 1.6.4
....

From it, a project called DeviceMonitort is created.  We can really test drive this web application now. Here is how we start jetty embedded server:

mvn install jetty:run
....
2012-09-23 12:18:28.567:INFO:oejs.AbstractConnector:Started SelectChannelConnector@0.0.0.0:8080
[INFO] Started Jetty Server
[INFO] Starting scanner at interval of 10 seconds.

Test drive by http://localhost:8080/ from web browser:

CometD Connection Established
Server Says: Hello, World
This is just like any programming where we always start with HELLO WORLD. Don’t stop here. Let’s explore what was created:

Exploring

In web.xml, it defines two servlets:


<servlet>
<servlet-name>cometd</servlet-name>
<servlet-class>org.cometd.server.CometdServlet</servlet-class>
<load-on-startup>1</load-on-startup>
</servlet>
<servlet-mapping>
<servlet-name>cometd</servlet-name>
<url-pattern>/cometd/*</url-pattern>
</servlet-mapping>

<servlet>
<servlet-name>initializer</servlet-name>
<servlet-class>DevServ.BayeuxInitializer</servlet-class>
<load-on-startup>2</load-on-startup>
</servlet>

In DevServ/BayeuxInitializer.java, it creates new HelloService(bayeux);
In DevServ/HelloService.java, it adds service : addService(“/service/hello”, “processHello”);
In processHello, it reads from input (name), writes to output (“greeting”, “Hello, ” + name), and remote.deliver(getServerSession(), “/hello”, output, null); Here /hello is the channel.

In index.jsp, it obtains context-path contextPath: ‘${pageContext.request.contextPath}’ dynamically, and pass control to application.js
In application.js, it does:

...
...
        $(window).unload(function()
        {
            cometd.disconnect(true);
        });

/// 1. configure URL for cometd protocol
        var cometURL = location.protocol + "//" + location.host + config.contextPath + "/cometd";
        cometd.configure({
            url: cometURL,
            logLevel: 'debug'
        });

/// 2. add meta listener
        cometd.addListener('/meta/handshake', _metaHandshake);
        cometd.addListener('/meta/connect', _metaConnect);

/// 3 handshake
        cometd.handshake();
...

Here is what metaHandshake do: subscribe to /hello channel, and publish { name: ‘World’ } to channel ‘/service/hello’, when the message push back, it displays $(‘#body’).append(‘<div>Server Says: ‘ + message.data.greeting + ‘</div>’);

...
 function _metaHandshake(handshake)
 {
 if (handshake.successful === true)
 {
 cometd.batch(function()
 {
 cometd.subscribe('/hello', function(message)
 {
 $('#body').append('<div>Server Says: ' + message.data.greeting + '</div>');
 });
 // Publish on a service channel since the message is for the server only
 cometd.publish('/service/hello', { name: 'World' });
 });
 }
 }
 ...

Deploy a CometD 2.x web application to Tomcat

I was writing a small system to simulate how the multiple data devices from the fields feeding data second by second to web server, and pass though to various clients (desktop, laptop, tablet, smart phones) in real time. The simulation is using data streaming through web servers by using Bayeux in CometD. It works great in jetty 7. However, when I deployed to Tomcat 7, I encountered the following error:

org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [cometd] in context with path [/DevServTest] threw exception
java.lang.IllegalStateException: Not supported.
    at org.apache.catalina.connector.Request.startAsync(Request.java:1664)
    at org.apache.catalina.connector.Request.startAsync(Request.java:1657)
    at org.apache.catalina.connector.RequestFacade.startAsync(RequestFacade.java:1022)
    at org.eclipse.jetty.continuation.Servlet3Continuation.suspend(Servlet3Continuation.java:171)
    at org.cometd.server.transport.LongPollingTransport.handle(LongPollingTransport.java:288)
    at org.cometd.server.CometdServlet.service(CometdServlet.java:181)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.eclipse.jetty.servlets.CrossOriginFilter.handle(CrossOriginFilter.java:212)
    at org.eclipse.jetty.servlets.CrossOriginFilter.doFilter(CrossOriginFilter.java:179)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
    at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:927)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:579)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:309)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:636)

Although all the programs seem work still, the above error messages are annoying. How can I get rid of the messages?

In theory, CometD 2.x runs natively in Jetty 7.x and in Servlet 3 compliant servlet containers (including Tomcat 7.x) and only need some special configuration for Jetty 6.x and Tomcat 6.x.

In reality, you cannot just copy the war file for CometD 2.x and run from Tomcat 7.x. The challenge on how to embed Jetty 7.x into Tomcat is more on deciding which jars are needed.

Since most of Jetty based web applications are developed using mvn and tested through mvn jetty:run, the dependency are on pom.xml and the target war generated was supposed to include all the jars needed in WEF-INF/lib.

I found the walk-around by using Netbeans to import the mvn project and let Netbeans deploy directly to Tomcat 7. It works for me! Here is a Guide for Deployment Using Netbeans.

Browser Based Communication

History

The web has been largely built around the so-called request/response paradigm of HTTP. A client loads up a web page and then nothing happens until the user clicks onto the next page.

Around 2005, AJAX started to make the web feel more dynamic. Still, all HTTP communication was steered by the client, which required user interaction or periodic polling to load new data from the server.

Technologies that enable the server to send data to the client in the very moment when it knows that new data is available have been around for quite some time. They go by names such as “Push” or “Comet”. One of the most common hacks to create the illusion of a server initiated connection is called long polling. With long polling, the client opens an HTTP connection to the server which keeps it open until sending response. Whenever the server actually has new data it sends the response. Long polling and the other techniques work quite well. You use them every day in applications such as GMail chat.

Protocol: Bayeux vs WebSocket

The CometD framework is an implementation of the Bayeux protocol, which allows for a multi-channel asynchronous communication stream over unreliable networks between a client and server. Implementations of this are used in many languages (JavaScript, Java, Perl …) but predominantly browser-based AJAX applications. Bayeux has the advantage that it can run in any AJAX capable browser using nothing more than the underlying HTTP communication facilities provided by the browser to achieve asynchronous/background updates as new information comes in (like Google Mail’s new mail notifications). In fact, the same protocol can be used to connect devices in other languages and where network communication may be spotty, such as mobile devices.

WebSockets is a draft standard which is sponsored by Google, Apple and others at the WhatWG working group that is standardising HTML 5. As a result, HTML 5 capable browsers (Chrome, Safari) are starting to include built-in support for the WebSocket protocol.

Both protocols aim to allow web-based AJAX applications to communicate with other services via asynchronous messaging or socket-based connections, rather than having to roll your own communications layer on top of an existing application. This allows the design of an application to focus on the component parts, and hand off messages to the communication layer for delivery. In addition, both can set up long-running connections such that events can be delivered asynchronously subsequently to the application. This is nothing new: HTTP 1.1 supported connection pipelining (the ability to keep open a connection after each request, and the ability to send multiple requests before the first was acknowledged); and other protocols like IMAP supported the IDLE command to put a connection into a hibernate state, where there is no ongoing communication but the server can push new messages at any time. Indeed, prior to either Bayeux or WebSockets, the term “HTTP Push” was used to indicate a general mechanism for long-lasting communication channel across HTTP.

Problems and Challenges

The real challenge is HOW TO MAINTAIN THE CONNECTION!

A connection that hasn’t had any data over a period of time may be considered to have died, and arbitrarily terminated, at some point in the future. To address this, the IMAP IDLE suggests that clients send a re-negotiated IMAP IDLE command every 29 minutes to avoid disconnection. With other proxies in the way of HTTP, it may be that a proxy determines that a connection is idle and drops the connection, even if the client and server agree to maintain an ongoing connection.

The other problem is resource-based; generally, browsers limit the number of concurrent HTTP connections to a single server to avoid overbearing the server (or the network link). It’s common for browsers to limit such concurrent connections to between 2 and 4 at any one time.

Both Bayeux and WebSockets attempt to avoid the resource limits by using a fall-back mechanism for long polling (in the case of Bayeux) or switching to a non-HTTP based secondary protocol instead. As a result, users of these libraries don’t generally have to worry about limitations placed on them by the browser or infrastructure.

COMETD 2.x

CometD 2 now supports both Bayeux, and Websocket (which as an alternative transport to the currently supported JSON long polling and JSONP callback polling). CometD is transparent to browsers with or without websocket support. Websocket usage will be able to give us even better throughput and latency for cometd than the already impressive results achieved with long polling.

References

Bayeux Protocol Specification

W3C The WebSocket API

Alex Russell’s original 2006 comet blog post

J2EE Big Picture

I came across this graph from “Lesson 26 – Java EE 6 Overview” in the book Java Programming: 24-Hour Trainer. Wrox Press. © 2011, by Fain, Yakov.  It is a wonderfully concise graph.
J2EE Big Picture from Java Programming: 24-Hour Trainer

J2EE Big Picture from Java Programming: 24-Hour Trainer

 

How to connect Blackberry Phone to Ubuntu Linux

RIM only makes a desktop software for Microsoft Windows Platform. For those people running Linux desktop, the following is a guide to make your Linux Desktop to recognize your Blackberry smartphone, charge your battery, backup your device, and sync your data between Evolution Email application and your blackberry.

I tested the following process on my Ubuntu 10.10. with my Blackberry Torch 9800 on Blackberry OS 6.X.

1. Download

Download the following from http://sourceforge.net/projects/barry/files/barry/barry-0.17.1/ubuntu1004/

  • barry-util_0.17.1-0_ubuntu1004_i386.deb
  • barrybackup-gui_0.17.1-0_ubuntu1004_i386.deb
  • libbarry0_0.17.1-0_ubuntu1004_i386.deb
  • opensync-plugin-barry_0.17-0_ubuntu1004_i386.deb

Download libopensync0_0.22-4ubuntu0.1_i386.deb  from https://launchpad.net/ubuntu/lucid/i386/libopensync0/0.22-4ubuntu0.1

2. Install using Ubuntu Software Centre

  • Library: libbarry0_0.17.1-0_ubuntu1004_i386.deb
  • Command Line Utility: barry-util_0.17.1-0_ubuntu1004_i386.deb
  • GTK+ based GUI Backup Tools: barrybackup-gui_0.17.1-0_ubuntu1004_i386.deb
  • OpenSync Framework Library: libopensync0_0.22-4ubuntu0.1_i386.deb 
  • OpenSync: opensync-plugin-barry_0.17-0_ubuntu1004_i386.deb

3. What have been installed?

root@Norhead:~# find / -name ‘*barry*’ -print

/var/lib/dpkg/info/barrybackup-gui.md5sums
/var/lib/dpkg/info/libbarry0.list
/var/lib/dpkg/info/barry-util.conffiles
/var/lib/dpkg/info/barrybackup-gui.list
/var/lib/dpkg/info/barry-util.postinst
/var/lib/dpkg/info/barrybackup-gui.postrm
/var/lib/dpkg/info/libbarry0.shlibs
/var/lib/dpkg/info/barrybackup-gui.postinst
/var/lib/dpkg/info/barry-util.list
/var/lib/dpkg/info/barry-util.md5sums
/var/lib/dpkg/info/libbarry0.md5sums
/var/lib/dpkg/info/libbarry0.postinst
/var/lib/dpkg/info/libbarry0.postrm
/etc/ppp/peers/barry-chinamobile
/etc/ppp/peers/barry-verizon
/etc/ppp/peers/barry-rogers
/etc/ppp/peers/barry-o2ireland
/etc/ppp/peers/barry-sprint
/etc/ppp/peers/barry-optus-au
/etc/ppp/peers/barry-telus
/etc/ppp/peers/barry-kpn
/etc/ppp/peers/barry-orange-spain
/etc/ppp/peers/barry-vodafone-au
/etc/ppp/peers/barry-tmobileus
/etc/ppp/peers/barry-minimal
/etc/ppp/peers/barry-att_cingular
/etc/chatscripts/barry-telus.chat
/etc/chatscripts/barry-orange-spain.chat
/etc/chatscripts/barry-chinamobile.chat
/etc/chatscripts/barry-rogers.chat
/etc/chatscripts/barry-tmobileus.chat
/etc/chatscripts/barry-sprint.chat
/etc/chatscripts/barry-optus-au.chat
/etc/chatscripts/barry-kpn.chat
/etc/chatscripts/barry-vodafone-au.chat
/etc/chatscripts/barry-minimal.chat
/etc/chatscripts/barry-att_cingular.chat
/etc/chatscripts/barry-o2ireland.chat
/etc/chatscripts/barry-verizon.chat
/usr/bin/barrybackup
/usr/lib/libbarrysync.so.17.0.1
/usr/lib/libbarrydp.so.17.0.1
/usr/lib/libbarrydp.so.17
/usr/lib/libbarrybackup.so.17.0.1
/usr/lib/libbarryjdwp.so.17.0.1
/usr/lib/libbarrysync.so.17
/usr/lib/libbarrybackup.so.17
/usr/lib/libbarryjdwp.so.17
/usr/lib/libbarryalx.so.17
/usr/lib/libbarry.so.17.0.1
/usr/lib/libbarryalx.so.17.0.1
/usr/lib/libbarry.so.17
/usr/share/locale/fr/LC_MESSAGES/barry.mo
/usr/share/locale/fr/LC_MESSAGES/barry-backup.mo
/usr/share/man/man1/barrybackup.1.gz
/usr/share/barry
/usr/share/menu/barrybackup-gui
/usr/share/doc/barry-util
/usr/share/doc/barry-util/ppp/barry-telus.chat
/usr/share/doc/barry-util/ppp/barry-chinamobile
/usr/share/doc/barry-util/ppp/barry-verizon
/usr/share/doc/barry-util/ppp/barry-orange-spain.chat
/usr/share/doc/barry-util/ppp/barry-rogers
/usr/share/doc/barry-util/ppp/barry-o2ireland
/usr/share/doc/barry-util/ppp/barry-sprint
/usr/share/doc/barry-util/ppp/barry-chinamobile.chat
/usr/share/doc/barry-util/ppp/barry-optus-au
/usr/share/doc/barry-util/ppp/barry-telus
/usr/share/doc/barry-util/ppp/barry-rogers.chat
/usr/share/doc/barry-util/ppp/barry-tmobileus.chat
/usr/share/doc/barry-util/ppp/barry-kpn
/usr/share/doc/barry-util/ppp/barry-sprint.chat
/usr/share/doc/barry-util/ppp/barry-fido.chat
/usr/share/doc/barry-util/ppp/barry-optus-au.chat
/usr/share/doc/barry-util/ppp/barry-kpn.chat
/usr/share/doc/barry-util/ppp/barry-vodafone-au.chat
/usr/share/doc/barry-util/ppp/barry-orange-spain
/usr/share/doc/barry-util/ppp/barry-minimal.chat
/usr/share/doc/barry-util/ppp/barry-vodafone-au
/usr/share/doc/barry-util/ppp/barry-att_cingular.chat
/usr/share/doc/barry-util/ppp/barry-o2ireland.chat
/usr/share/doc/barry-util/ppp/barry-tmobileus
/usr/share/doc/barry-util/ppp/barry-minimal
/usr/share/doc/barry-util/ppp/barry-verizon.chat
/usr/share/doc/barry-util/ppp/barry-att_cingular
/usr/share/doc/barrybackup-gui
/usr/share/doc/libbarry0
/usr/share/pixmaps/barry_logo_icon.png

root@Norhead:~# find / -name ‘*synctool*’ -print

/etc/bash_completion.d/msynctool

4. Test

Now, it is the fun part. Just connect your Blackberry to the Linux desktop, you will be prompted to “USB” or “Sync Media” or ‘Charge Only’ on your Blackberry device.  Choose one of them, and you can enjoy now. I can charge the battery. I can play mp3 files located on Blackberry through my Linux desktop, and also transfer media files between them.

To backup, start type ‘barrybackup’ in a terminal window and GUI should appear for backup. I can backup my device to .Barry/backup on Linux desktop.

You can do more things. I didn’t test the sync between Evolution Email and Blackberry.  Here is the good guide. Another one is here. But I haven’t tried them all yet, because I pretty much just charge the battery most of time. That’s good enough for me.