Running my own SMS Service with Kannel and HUAWEI E180

Mobile Services are becomming more and more common. Seven years ago I had the opportunity to develop some simpel SMS services, and SMS services is still the most common form of Mobile Services, though the mobile web is picking up momentum as well as custom Mobile Aplications. I decided to have a look at Mobile Services and to start with building a simple SMS Service. This blog post vil cover the basic setup and a later post the service part itself (though it's fairly easy).

Back in 2002 the company I worked for used both a commercial SMSC with billing etc., but we also ran our own SMSC to receive data carried by SMS. For our own SMSC we used Kannel which was quite easy to setup and run. I was please to see the Kannel project is still running.

I started by downloading the lastest stable release (version 1.4.3). Installing on my Fedora 10 Linux distribution was an easy classic ./configure, make, make install. The binary distribution comes with a sample SMSC configuration in the gw/smskannel.conf file. I only needed to update it with my local settings which involved figuring out the way to hook it up to my modem (the only part that was a bit problematic back then with a serial interface).

The Huawei E180 Modem

I have a Mobile Broadband ISP (flatrate) that comes with a USB modem (3G). The modem has the name E180, and can be seen on the Huawei device website HUAWEI E180 - HSPA USB Stick. The short specification list is: Support 2Mbps (5.76Mbps ready) HSUPA and 7.2Mbps HSDPA services Support SMS and Voice services. To find the manual is quite har since it's not available from Huawei's own support site, so through Google I found it [PDF] at the South African company Maredi's website. It carries the subtitle as flexiable as wings and it certainly is easy to use on both Linux and Windows, though I'm not sure I really like the metaphor on airplane wings.

After I insert the modem into a USB port the messages log showes something like:

Mar 28 07:47:11 kannel kernel: usb 2-1: new high speed USB device using ehci_hcd and address 3
Mar 28 07:47:11 kannel kernel: usb 2-1: configuration #1 chosen from 1 choice                 
Mar 28 07:47:11 kannel kernel: scsi3 : SCSI emulation for USB Mass Storage devices            
Mar 28 07:47:11 kannel kernel: scsi4 : SCSI emulation for USB Mass Storage devices            
Mar 28 07:47:11 kannel kernel: usb 2-1: New USB device found, idVendor=12d1, idProduct=1003   
Mar 28 07:47:11 kannel kernel: usb 2-1: New USB device strings: Mfr=2, Product=1, SerialNumber=0
Mar 28 07:47:11 kannel kernel: usb 2-1: Product: HUAWEI Mobile                                  
Mar 28 07:47:11 kannel kernel: usb 2-1: Manufacturer: HUAWEI Technology                         
Mar 28 07:47:11 kannel kernel: usb 2-1: USB disconnect, address 3                               
Mar 28 07:47:11 kannel pulseaudio[3309]: module-hal-detect.c: Error getting capability: org.freedesktop.Hal.NoSuchDevice: No device with id /org/freedesktop/Hal/devices/usb_device_ffffffff_ffffffff_noserial_scsi_host                                     
Mar 28 07:47:11 kannel pulseaudio[3309]: module-hal-detect.c: Error getting capability: org.freedesktop.Hal.NoSuchDevice: No device with id /org/freedesktop/Hal/devices/usb_device_ffffffff_ffffffff_noserial_0_scsi_host                                   
Mar 28 07:47:17 kannel kernel: usb 2-1: new high speed USB device using ehci_hcd and address 4                                 
Mar 28 07:47:17 kannel kernel: usb 2-1: configuration #1 chosen from 1 choice                                                  
Mar 28 07:47:17 kannel kernel: usb-storage: probe of 2-1:1.0 failed with error -5                                              
Mar 28 07:47:17 kannel kernel: usb-storage: probe of 2-1:1.1 failed with error -5                                              
Mar 28 07:47:18 kannel kernel: scsi7 : SCSI emulation for USB Mass Storage devices                                             
Mar 28 07:47:19 kannel kernel: scsi8 : SCSI emulation for USB Mass Storage devices                                             
Mar 28 07:47:19 kannel kernel: usb 2-1: New USB device found, idVendor=12d1, idProduct=1003                                    
Mar 28 07:47:19 kannel kernel: usb 2-1: New USB device strings: Mfr=2, Product=1, SerialNumber=0
Mar 28 07:47:19 kannel kernel: usb 2-1: Product: HUAWEI Mobile
Mar 28 07:47:19 kannel kernel: usb 2-1: Manufacturer: HUAWEI Technology
Mar 28 07:47:19 kannel kernel: usbcore: registered new interface driver usbserial
Mar 28 07:47:19 kannel kernel: usbserial: USB Serial support registered for generic
Mar 28 07:47:19 kannel kernel: usbcore: registered new interface driver usbserial_generic
Mar 28 07:47:19 kannel kernel: usbserial: USB Serial Driver core
Mar 28 07:47:19 kannel kernel: usbserial: USB Serial support registered for GSM modem (1-port)
Mar 28 07:47:19 kannel kernel: option 2-1:1.0: GSM modem (1-port) converter detected
Mar 28 07:47:19 kannel kernel: usb 2-1: GSM modem (1-port) converter now attached to ttyUSB0
Mar 28 07:47:19 kannel kernel: option 2-1:1.1: GSM modem (1-port) converter detected
Mar 28 07:47:19 kannel kernel: usb 2-1: GSM modem (1-port) converter now attached to ttyUSB1
Mar 28 07:47:19 kannel kernel: usbcore: registered new interface driver option
Mar 28 07:47:19 kannel kernel: option: USB Driver for GSM modems: v0.7.2
Mar 28 07:47:23 kannel kernel: scsi 7:0:0:0: CD-ROM            HUAWEI   Mass Storage     2.31 PQ: 0 ANSI: 2
Mar 28 07:47:23 kannel kernel: sr0: scsi-1 drive
Mar 28 07:47:23 kannel kernel: Uniform CD-ROM driver Revision: 3.20
Mar 28 07:47:23 kannel kernel: sr 7:0:0:0: Attached scsi generic sg2 type 5
Mar 28 07:47:24 kannel kernel: scsi 8:0:0:0: Direct-Access     HUAWEI   SD Storage       2.31 PQ: 0 ANSI: 2
Mar 28 07:47:24 kannel kernel: sd 8:0:0:0: [sdc] Attached SCSI removable disk
Mar 28 07:47:24 kannel kernel: sd 8:0:0:0: Attached scsi generic sg3 type 0
Mar 28 07:47:25 kannel NetworkManager:   (ttyUSB0): detected GSM modem via HAL capabilities
Mar 28 07:47:25 kannel NetworkManager:   (ttyUSB0): new Modem device (driver: 'option')
Mar 28 07:47:25 kannel NetworkManager:   (ttyUSB0): exported as /org/freedesktop/Hal/devices/usb_device_12d1_1003_noserial_if0_serial_usb_0
Mar 28 07:47:29 kannel NetworkManager:   (ttyUSB0): device state change: 1 -> 2
Mar 28 07:47:29 kannel NetworkManager:   (ttyUSB0): deactivating device (reason: 2).

Configure and run Bearerbox

The bearerbox part of the example configuration with just the needed changes, configuring the bearerbox and the two SMSC connections, the first one the fake SMSC and the other one my virtual SMSC based on my Huawei E180 modem:

# Needed - otherwise Kannel Bearerbox could not configure the modem correctly
include = doc/examples/modems.conf                                       

#---------------------------------------------                           
# CORE                                                                   
#---------------------------------------------                           
group = core
admin-port = 13000
smsbox-port = 13001
admin-password = bar
box-deny-ip = "*.*.*.*"      
box-allow-ip = "127.0.0.1"   

#---------------------------------------------
# SMSC CONNECTIONS                            
#                                             
# SMSC connections are created in bearerbox and they handle SMSC specific
# protocol and message relying. You need these to actually receive and send
# messages to handset, but can use GSM modems as virtual SMSCs             

# This is a fake smsc connection, _only_ used to test the system and services.
# It really cannot relay messages to actual handsets!                         
group = smsc
smsc = fake 
smsc-id = FAKE
port = 10000  
connect-allow-ip = 127.0.0.1


# My own 'virtual' SMSC with the HUAWEI E169 
group = smsc
smsc = at   
modemtype = auto
device = /dev/ttyUSB1
my-number = [TheTelephoneNumberOnTheSIM]
log-level = 0        
pin = [ThePINofTheSIM]

The Bearerbox is started with /opt/kannel-1.4.3/sbin/bearerbox -v 1 /home/kannel/Download/gateway-1.4.3/gw/smskannel.conf and the output is:

2009-03-28 07:55:26 [4193] [0] INFO: Debug_lvl = 1, log_file = , log_lvl = 0
2009-03-28 07:55:26 [4193] [0] WARNING: DLR: using default 'internal' for storage type.
2009-03-28 07:55:26 [4193] [0] INFO: DLR using storage type: internal
2009-03-28 07:55:26 [4193] [0] INFO: HTTP: Opening server at port 13000.
2009-03-28 07:55:26 [4193] [0] INFO: BOXC: 'smsbox-max-pending' not set, using default (100).
2009-03-28 07:55:26 [4193] [0] INFO: Set SMS resend frequency to 60 seconds.
2009-03-28 07:55:26 [4193] [0] INFO: SMS resend retry set to unlimited.
2009-03-28 07:55:26 [4193] [0] INFO: DLR rerouting for smsc id  disabled.
2009-03-28 07:55:26 [4193] [0] INFO: DLR rerouting for smsc id <(null)> disabled.
2009-03-28 07:55:26 [4193] [0] INFO: AT2[/dev/ttyUSB1]: configuration doesn't show modemtype. will autodetect
2009-03-28 07:55:26 [4193] [0] INFO: ----------------------------------------
2009-03-28 07:55:26 [4193] [0] INFO: Kannel bearerbox II version 1.4.3 starting
2009-03-28 07:55:26 [4193] [0] INFO: MAIN: Start-up done, entering mainloop
2009-03-28 07:55:26 [4193] [7] INFO: AT2[/dev/ttyUSB1]: opening device
2009-03-28 07:55:27 [4193] [7] INFO: AT2[/dev/ttyUSB1]: speed set to 115200
2009-03-28 07:55:29 [4193] [7] INFO: AT2[/dev/ttyUSB1]: Closing device
2009-03-28 07:55:29 [4193] [7] INFO: AT2[/dev/ttyUSB1]: detect speed is 115200
2009-03-28 07:55:29 [4193] [7] INFO: AT2[/dev/ttyUSB1]: opening device
2009-03-28 07:55:30 [4193] [7] INFO: AT2[/dev/ttyUSB1]: speed set to 115200
2009-03-28 07:55:32 [4193] [7] INFO: AT2[/dev/ttyUSB1]: Closing device
2009-03-28 07:55:32 [4193] [7] INFO: AT2[/dev/ttyUSB1]: opening device
2009-03-28 07:55:32 [4193] [7] INFO: AT2[/dev/ttyUSB1]: Logging in
2009-03-28 07:55:33 [4193] [7] INFO: AT2[/dev/ttyUSB1]: init device
2009-03-28 07:55:33 [4193] [7] INFO: AT2[/dev/ttyUSB1]: speed set to 115200
2009-03-28 07:55:33 [4193] [7] ERROR: AT2[/dev/ttyUSB1]: Generic error: ERROR
2009-03-28 07:55:33 [4193] [7] INFO: AT2[/dev/ttyUSB1]: cannot enable hardware handshake
2009-03-28 07:55:34 [4193] [7] INFO: AT2[/dev/ttyUSB1]: AT SMSC successfully opened.

Configuring and running the Smsbox

The Userguide has the complete options for SMSBOX configruation and the example configuration is just fine:

#---------------------------------------------
# SMSBOX SETUP                                
#                                             
# Smsbox(es) do higher-level SMS handling after they have been received from
# SMS centers by bearerbox, or before they are given to bearerbox for delivery

group = smsbox
bearerbox-host = 127.0.0.1
sendsms-port = 13013      
global-sender = 13013     
#log-file = "/tmp/smsbox.log"   
#log-level = 0                  

To use the SMSC for pushing SMS the SendSMS-user configuration is needed:

#---------------------------------------------
# SEND-SMS USERS
#
# These users are used when Kannel smsbox sendsms interface is used to
# send PUSH sms messages, i.e. calling URL like
# http://kannel.machine:13013/cgi-bin/sendsms?username=tester&password=foobar...

group = sendsms-user
username = tester
password = foobar
#user-allow-ip = ""

The normal SMS service is the ability to respond to SMS requests (SMS PULL) which is done with SMS-service configurations:

#---------------------------------------------
# SERVICES
#
# These are 'responses' to sms PULL messages, i.e. messages arriving from
# handsets. The response is based on message content. Only one sms-service is
# applied, using the first one to match.

group = sms-service
keyword = nop
text = "You asked nothing and I did it!"

# There should be always a 'default' service. This service is used when no
# other 'sms-service' is applied.

group = sms-service
keyword = default
text = "No service specified"

This is the most simple service just serving a static string (the value of text). The other and dynamic ways to respond to these requests I'll cover in another post.

Now I can start the SMSBox with /opt/kannel-1.4.3/sbin/smsbox -v 1 gw/smskannel.conf

2009-03-28 08:03:44 [4265] [0] INFO: Debug_lvl = 1, log_file = , log_lvl = 0
2009-03-28 08:03:44 [4265] [0] INFO: Service global sender set as '13013'
2009-03-28 08:03:44 [4265] [0] INFO: HTTP: Opening server at port 13013.
2009-03-28 08:03:44 [4265] [0] INFO: Set up send sms service at port 13013
2009-03-28 08:03:44 [4265] [0] INFO: Connected to bearerbox at 127.0.0.1 port 13001.

After starting these two services there are now a total of 4 open ports for fakesmsc, bearerbox and smsbox:

tcp        0      0 0.0.0.0:10000               0.0.0.0:*                   LISTEN
tcp        0      0 0.0.0.0:13000               0.0.0.0:*                   LISTEN
tcp        0      0 0.0.0.0:13001               0.0.0.0:*                   LISTEN
tcp        0      0 0.0.0.0:13013               0.0.0.0:*                   LISTEN

An example message flow

A simple demo can be run with the fakesmsc commandline tool like in ./test/fakesmsc -m 1 "12345678 87654321 text nop [the_parameters]":

2009-03-28 08:05:55 [4277] [0] INFO: Debug_lvl = -1, log_file = , log_lvl = 0
2009-03-28 08:05:55 [4277] [0] INFO: Host localhost Port 10000 interval 1.000 max-messages 1
2009-03-28 08:05:55 [4277] [0] INFO: fakesmsc starting
2009-03-28 08:05:55 [4277] [0] INFO: fakesmsc: sent message 1
2009-03-28 08:05:55 [4277] [0] INFO: Got message 1: <13013 12345678 text No service specified>

The Bearerbox logs:

2009-03-28 08:03:44 [4193] [5] INFO: Client connected from <127.0.0.1>
2009-03-28 08:05:55 [4193] [6] INFO: Fakesmsc client connected from 127.0.0.1

and the smsbox

2009-03-28 08:05:55 [4265] [4] INFO: Starting to service <nop [the_parameters]> from <12345678> to <87654321>

Summary

Using Kannel is still very easy to both compile, configure and run.

Updated W3C draft on "Cross-Origin Resource Sharing"

About a year ago I wrote about W3C draft on a 'policy' to allow webapps to selectively loosen the same-origin restriction and now they've release a new working draft: Cross-Origin Resource Sharing. I haven't checked that status of the project but hopefully they are closing up to the final recommandation after more than a year's work. The only clue is from the section on status of the document where it states It is expected that this document will progress along the W3C Recommendation track, which isn't all that exact.

The abstract defines the essence as:

This document defines a mechanism to enable client-side cross-origin requests. Specifications that want to enable cross-origin requests in an API they define can use the algorithms defined by this specification. If such an API is used on http://example.org resources, a resource on http://hello-world.example can opt in using the mechanism described by this specification (e.g., specifying Access-Control-Allow-Origin: http://example.org as response header), which would allow that resource to be fetched cross-origin from http://example.org.

The specification defines af total of 3 HTTP request headers and 5 HTTP response headers:

• Origin
• Access-Control-Request-Method
• Access-Control-Request-Headers

• Access-Control-Allow-Origin
• Access-Control-Max-Age
• Access-Control-Allow-Credentials
• Access-Control-Allow-Methods
• Access-Control-Allow-Headers

I haven't read the specification in great detail but it looks like this will make web 2.0 mashups a lot easier, makeing it possible to make requests to different hosts based on a formal policy. As for inter-document interactions like in iframe scenarios I'm not sure, but I think it follows along since if to hostnames declare that they are an associated common domman I would expect these restraints to loosen as well freeing us for using semi-dirty hacks like double iframe communication.

I sure would like to remov the "_WidgetManager._Init" javascript blob

When I did a W3C Validation check on my last blog post I once more stumbled upon the quite large JavaScript blob that blogger.com drops at the buttom of my pages. Some time ago I looked at this with the hope that I could remove it through my template but I never succeded and just gave up. I stille hope that they either drop it all together og externalize it so it doesn't break validation and make it possible to seperate XHTML markup from JavaScript.

When I first looked at it I found an entry on the Blogger Help Group with the title "Widget Manager Code Added To HTML". In my last post it starts like:

 <script type='text/javascript'>
_WidgetManager._Init('http://www.blogger.com/rearrange?blogID=591744930960839717', 'http://blog.sweetxml.org/2009/03/my-first-wsdl-20-generating-axis2.html','591744930960839717');
.......
</script>

A ran it through the wc command that gave me the output 11 779 25931, that is:

11    newlines
779   words
25931 bytes

So that's a good 26 kilo bytes of JavaScript!.

My first WSDL 2.0 - generating Axis2 client code for a REST API

The former InfoStructureBase (ISB) along with the catalogue of OIO standards has been replaced with digitaliser.dk. Closing up to a formal version 1.0 the current beta 4 has as the first release a public API. The ISB API was SOAP based with Web Services Security UsernameToken for authentication. The the WSDL's can still be seen in the former repository part:

• documentmanager.wsdl
• namespacemanager.wsdl
• reviewrequestmanager.wsdl
• searchmanager.wsdl
• subscriptionmanager.wsdl
• taxonomymanager.wsdl

As annonced in the last release note "Digitaliser.dk - nu med offentligt API og medlemspolitik for grupper og rig tekstredigering" this is based on (OIO)REST with POX responses. The documentation is in the form of endpoints and XML Schemas for the responses, and can be read online: http://api.digitaliser.dk/rest.

I have practically no experience with REST, so I started to google for hints on have to call a REST webservice with Axis2. I did'nt find much and it seem way to complex for likeing. I did found out that REST/POX services can be described with WSDL 2.0 Part: 1 and WSDL Version 2.0 Part 2: Adjuncts. I do not find these as easy reads so I looked for an example and found a post on Keith Chapmans blog "RESTfull Mashup with WSDL 2.0 - WSO2 Mashup Server". The WSDL for this exmaple. Another resource that I haven't really read yet is Lawrence Mandes article on IBM DeveloperWorks called "Describe REST Web services with WSDL 2.0".

Keith's example is quite complex and I wanted something simple to increase the likelihood of succes, so I choose to implement fetching a resource on digitaliser.dk. After cleaning up the example WSDL, removing complex parts, renaming and some trial and error I ended up with this WSDL:

    1 <?xml version="1.0"?>
    2 <wsdl2:description
    3   xmlns:wsdl2="http://www.w3.org/ns/wsdl"
    4   xmlns:wrpc="http://www.w3.org/ns/wsdl/rpc"
    5   xmlns:wsaw="http://www.w3.org/2006/05/addressing/wsdl"
    6   xmlns:whttp="http://www.w3.org/ns/wsdl/http"
    7   xmlns:wsdlx="http://www.w3.org/ns/wsdl-extensions"
    8   xmlns:xs="http://www.w3.org/2001/XMLSchema"
    9   xmlns:api1="http://api.digitaliser.dk/rest/1.0"
   10   xmlns:xtns="http://rep.oio.dk/sweetxml.org/xml/xsd/20090306/digitaliser.dk/1.0/"
   11   xmlns:wtns="http://rep.oio.dk/sweetxml.org/xml/wsdl/20090306/digitaliser.dk/1.0/"
   12   targetNamespace="http://rep.oio.dk/sweetxml.org/xml/wsdl/20090306/digitaliser.dk/1.0/">
   13   <wsdl2:types>
   14     <xs:import
   15       schemaLocation="http://api.digitaliser.dk/schemas/rest/1.0/resource.xsd"
   16       namespace="http://api.digitaliser.dk/rest/1.0" />
   17     <xs:schema
   18       attributeFormDefault="unqualified"
   19       elementFormDefault="qualified"
   20       targetNamespace="http://rep.oio.dk/sweetxml.org/xml/xsd/20090306/digitaliser.dk/1.0/">
   21       <xs:element
   22         name="identifier"
   23         type="xs:string" />
   24       <xs:complexType
   25         name="idWrapperType">
   26         <xs:sequence>
   27           <xs:element
   28             name="id"
   29             type="xs:string" />
   30         </xs:sequence>
   31       </xs:complexType>
   32       <xs:element
   33         name="idWrapper"
   34         type="xtns:idWrapperType" />
   35     </xs:schema>
   36   </wsdl2:types>
   37   <wsdl2:interface
   38     name="ResourceInterface">
   39     <wsdl2:operation
   40       name="getResourceOperation"
   41       pattern="http://www.w3.org/ns/wsdl/in-out"
   42       wsdlx:safe="true">
   43       <wsdl2:input
   44         element="xtns:idWrapper"
   45         wsaw:Action="" />
   46       <wsdl2:output
   47         element="api1:resource"
   48         wsaw:Action="" />
   49     </wsdl2:operation>
   50   </wsdl2:interface>
   51   <wsdl2:binding
   52     name="ResourceBinding"
   53     interface="wtns:ResourceInterface"
   54     type="http://www.w3.org/ns/wsdl/http">
   55     <wsdl2:operation
   56       ref="wtns:getResourceOperation"
   57       whttp:method="GET"
   58       whttp:location="resources/{id}" />
   59   </wsdl2:binding>
   60   <wsdl2:service
   61     name="ResourceService"
   62     interface="wtns:ResourceInterface">
   63     <wsdl2:endpoint
   64       name="RestEndpoint"
   65       binding="wtns:ResourceBinding"
   66       address="http://api.digitaliser.dk/rest/resources" />
   67   </wsdl2:service>
   68 </wsdl2:description>

The three major things for me where:

• Eksternal schemas can/must now be imported directly in the types section - no need for a wrapper schema anymore (and it will not work). Look at example 3-4 in the WSDL 2.0 primer or Importing XML Schema in the standard.
• Even though it would be the easiest just to have a simpletype element for the request parameter in this example, I have to use a wrapper. The reason I guess is that this model can handle any number of parameters, where the case with just one is the simplest.
• @signature attribute does a reference to child elements in the wrapper defined in the former bullet.

With this WSDL I could generate Axis2 stub code with xmlbenas binding with the ant codegen utility:

<codegen wsdlfilename="${resource.wsdl}" databindingName="xmlbeans" output="${generated.dir}" wsdlVersion="2.0" />

from here it's easy writing a simple client program. The request param is a little clunky since I'm using XMLBeans, but that's fine with me.

    1 package org.sweetxml.rest;
    2 
    3 import java.rmi.RemoteException;
    4 import java.util.ArrayList;
    5 import java.util.Collection;
    6 import java.util.Iterator;
    7 
    8 import org.apache.axis2.Constants;
    9 import org.apache.xmlbeans.XmlOptions;
   10 import org.apache.xmlbeans.XmlValidationError;
   11 
   12 import dk.digitaliser.api.rest._1_0.ResourceDocument;
   13 import dk.oio.rep.sweetxml_org.xml.wsdl._20090306.digitaliser_dk._1_0.ResourceServiceStub;
   14 import dk.oio.rep.sweetxml_org.xml.xsd._20090306.digitaliser_dk._1_0.IdWrapperDocument;
   15 
   16 public class ResourceClient {
   17 
   18   public static void main(String[] args) throws RemoteException {
   19     String resourceIdentifier = "42079";
   20     ResourceServiceStub stub = new ResourceServiceStub();
   21     stub._getServiceClient().getOptions().setProperty(Constants.Configuration.TRANSPORT_URL, "http://localhost:9999/rest/");
   22 
   23     IdWrapperDocument requestIdentifierDoc = IdWrapperDocument.Factory.newInstance();
   24     requestIdentifierDoc.addNewIdWrapper().setId(resourceIdentifier);
   25 
   26     ResourceDocument respDoc = stub.getResourceOperation(requestIdentifierDoc);
   27     System.out.println("Title on resource " + resourceIdentifier + " is : " + respDoc.getResource().getTitle());
   28     validate(respDoc);
   29   }
   30 
   31   private static void validate(ResourceDocument resource) {
   32     Collection<XmlValidationError> errors = new ArrayList<XmlValidationError>();
   33 
   34     XmlOptions validationOptions = new XmlOptions();
   35     validationOptions.setErrorListener(errors);
   36     validationOptions.setLoadLineNumbers();
   37     validationOptions.setSaveNamespacesFirst();
   38     validationOptions.setSavePrettyPrint();
   39     validationOptions.setSaveAggressiveNamespaces();
   40     validationOptions.setUseDefaultNamespace();
   41 
   42     boolean isValid = false;
   43     isValid = resource.validate(validationOptions);
   44 
   45     if (!isValid) {
   46       Iterator<XmlValidationError> iter = errors.iterator();
   47       System.out.println(errors);
   48       while (iter.hasNext()) {
   49         XmlValidationError err = iter.next();
   50         System.out.println(err);
   51       }
   52     } else {
   53       System.out.println("Respons is XML Schema Valid");
   54     }
   55   }
   56 }

I added a very basic validation method that should be enforced if I were to use this for real. Notice that I changed the endpoint to let it run through TCPMon and now finally lets have a look at how it looks like on the wire.

The request

Since this is REST and I'm only fetching a resource htis is just a GET with only HTTP headers (and no SOAPAction):

GET /rest/resources/42079 HTTP/1.1
Content-Type: application/x-www-form-urlencoded; charset=UTF-8
SOAPAction: ""
User-Agent: Axis2
Host: api.digitaliser.dk:9999

The reponse

The HTTP headers sent back is just plain, except that were informed that it's based on the Noelios Restlet Engine (NRE).

HTTP/1.1 200 OK
Server: Apache-Coyote/1.1
Date: Fri, 13 Mar 2009 19:11:44 GMT
Accept-Ranges: bytes
Server: Noelios-Restlet-Engine/1.1..2
Content-Type: text/xml;charset=UTF-8
Transfer-Encoding: chunked

Here's hte body sent back containg the resource strcuture as defined in the XML Schema.

    1 <?xml version="1.0" encoding="UTF-8" standalone="yes"?>
    2 <ns1:resource
    3   xmlns:ns1="http://api.digitaliser.dk/rest/1.0">
    4   <ns1:title>Demo WSDL for a simple service for OIOUBL Simpel Ordering</ns1:title>
    5   <ns1:body>A naive WSDL for a demonstration service based on OIOUBL Simple Ordering to get a
    6     realistic complexity.</ns1:body>
    7   <ns1:ownerGroup
    8     ns1:ref="http://api.digitaliser.dk/rest/groups/42078">
    9     <ns1:title>Sweetxml.org</ns1:title>
   10   </ns1:ownerGroup>
   11   <ns1:createdDate>2009-01-22T08:38:59.585+01:00</ns1:createdDate>
   12   <ns1:published
   13     ns1:state="published"
   14     ns1:date="2009-01-22T08:42:28.616+01:00" />
   15   <ns1:taggedBy>
   16     <ns1:tag
   17       ns1:taggedItem="http://api.digitaliser.dk/rest/resources/42079"
   18       ns1:owner="http://api.digitaliser.dk/rest/users/19802"
   19       ns1:ref="http://api.digitaliser.dk/rest/tags/246143">
   20       <ns1:label>service</ns1:label>
   21     </ns1:tag>
   22     <ns1:tag
   23       ns1:taggedItem="http://api.digitaliser.dk/rest/resources/42079"
   24       ns1:owner="http://api.digitaliser.dk/rest/users/19802"
   25       ns1:ref="http://api.digitaliser.dk/rest/tags/246139">
   26       <ns1:label>demo</ns1:label>
   27     </ns1:tag>
   28     <ns1:tag
   29       ns1:taggedItem="http://api.digitaliser.dk/rest/resources/42079"
   30       ns1:owner="http://api.digitaliser.dk/rest/groups/42078"
   31       ns1:ref="http://api.digitaliser.dk/rest/tags/246996">
   32       <ns1:label>WSDL</ns1:label>
   33     </ns1:tag>
   34     <ns1:tag
   35       ns1:taggedItem="http://api.digitaliser.dk/rest/resources/42079"
   36       ns1:owner="http://api.digitaliser.dk/rest/users/19802"
   37       ns1:ref="http://api.digitaliser.dk/rest/tags/246145">
   38       <ns1:label>webservice</ns1:label>
   39     </ns1:tag>
   40     <ns1:tag
   41       ns1:taggedItem="http://api.digitaliser.dk/rest/resources/42079"
   42       ns1:owner="http://api.digitaliser.dk/rest/groups/42078"
   43       ns1:ref="http://api.digitaliser.dk/rest/tags/246995">
   44       <ns1:label>OIOUBL</ns1:label>
   45     </ns1:tag>
   46   </ns1:taggedBy>
   47   <ns1:artefacts>
   48     <ns1:artefact
   49       ns1:ref="http://api.digitaliser.dk/rest/resources/42079/artefacts/SX_OIOUBLorderDemo.wsdl">
   50       <ns1:title>SX_OIOUBLorderDemo.wsdl</ns1:title>
   51     </ns1:artefact>
   52   </ns1:artefacts>
   53   <ns1:uri>http://api.digitaliser.dk/rest/resources/42079</ns1:uri>
   54   <ns1:versionGroup
   55     ns1:ref="http://api.digitaliser.dk/rest/versiongroups/ff5aba78-c20c-4c70-9190-84527b81d731" />
   56   <ns1:classificationInstance />
   57   <ns1:version></ns1:version>
   58 </ns1:resource>

Summary

I was very pleased that I was able to acces the REST API with Axis2, and that I was able to do it in contract-first style. Even more I was gladly surprised that WSDL 2.0 had the functionality to describe it well, since my only prior knowledge was from articles going against WSDL 2.0

Another WS-Star candidate for Event Driven Architecture - WS-Eventing

In my last post I had a look at Web Services Notification (WSN). In this post I'll see what Web Services Eventing (WS-Eventing) has to offer. Where WSN is a formal OASIS Standard Web Services Eventing is currently only a Member Submission to W3C back in March 2006. The list of vendors behind the specification are most of the big players like for WSN with the big difference that Microsoft is participating.

The specification tries to meet the following requirements:

• Define means to create and delete event subscriptions.
• Define expiration for subscriptions and allow them to be renewed.
• Define how one Web service can subscribe on behalf of another.
• Define how an event source delegates subscription management to another Web service.
• Allow subscribers to specify how event messages should be delivered.
• Leverage other Web service specifications for secure, reliable, transacted message delivery.
• Support complex eventing topologies that allow the originating event source and the final event sink to be decoupled.
• Provide extensibility for more sophisticated and/or currently unanticipated subscription scenarios.
• Support a variety of encoding formats, including (but not limited to) both SOAP 1.1 [SOAP 1.1] and SOAP 1.2 [SOAP 1.2] Envelopes.

I haven't done a formal comparison but there seems to be quite a overlap between Web Services Notification and Web Services Eventing.

An attempt for convergense

It isn't the first time that we see two competing standards in the WS-Star stack, not that it makes it any better. I found that at the same time that the specification was submitted to W3C, the four companies Intel, Microsoft, IBM and HP put out a a letter of intent with the title "Toward Converging Web Service Standards for Resources, Events, and Management":

HP, IBM, Intel and Microsoft plan to develop a common set of specifications for resources, events, and management that can be broadly supported across multiple platforms. The parties will do this by building on existing specifications and defining a set of enhancements that enable this convergence. In many scenarios, vendors and customers building solutions using Web services will find that the existing specifications support their scenarios. Vendors and customers may use the new specifications and functions when needing the common capabilities.

The full text [PDF] can be found on intels website or in googles cache from Microsofts website. To my knowledge this never materialized, but maybe this is what turned up as a new working group at W3C.

Web Services Resource Access (WS-RA)

Just as I had concluded that WS-Eventing had stranded at W3C I found that W3C in november 2008 launched the Web Services Resource Access Working Group, and from the charter the scope is:

The Web Services Resource Access Working Group is chartered to standardize a general mechanism for accessing and updating the XML representation of a resource-oriented Web Service and metadata of a Web Service, as well as a mechanism to subscribe to events from a Web Service. The following list of features is intended to focus the work of the Working Group and ensure its timely completion:

1. The definition of basic Create, Read, Update, Delete (CRUD) operations that provide capabilities to create, read, update and delete a Web Services XML representation and the binding of these operations to SOAP.
2. A mechanism to allow a requester to retrieve metadata, or references to metadata, related to a Web Service and a mechanism to allow embedding of such metadata, or references to such metadata, in an Endpoint Reference (EPR). The defined capability must provide for retrieval and embedding of specific items of metadata such as WSDL, XML Schema and WS-Policy. The referencing of metadata must allow for both EPR or URL style references.
3. The definition of Web service operations (e.g. Enumerate, Pull, etc…) to enable a consumer to request and manage an enumeration context, and retrieve data items from an enumeration context for a data source. These operations must define details of SOAP messages for the request and response as well as one or more filter dialects to select the data that would be sent to the consumer. The ability to work efficiently with large datasets is important.
4. The definition of Web Services operations (e.g. Subscribe, Unsubscribe, etc…) to create and manage subscriptions to events that are delivered via Web Services.
   1. The definition of one or more filter dialects, such as XPath, that can be used in subscriptions to indicate interest in messages with specific content.
   2. Mechanisms to allow a subscriber to specify the means by which an event is delivered and the definition of a push-based delivery mode.
5. A mechanism by which a Web Service can advertise, through its metadata, such as WSDL and WS-Policy, the capabilities it supports from among the features defined by the Web Service Resource Access specifications.

The thing that jumps into my eyes is how RESTfull this sounds!

This high speed WG i supposed to deliver the recommendations in little more than a year (June 2010), which in quite fast in terms of standarization speed. The work will build on the following submission:

• Web Services Transfer (WS-Transfer)
• Web Services Resource Transfer (WS-RT)
• Web Services Enumeration (WS-Enumeration)
• Web Services Metadata Exchange 1.1 (WS-MetadataExchange)
• Web Services Eventing (WS-Eventing)

The current editors draft of WS-Eventing is available online as promised. Judging from the issues related to WS-Eventing convergence doesn't seem to be on the agenda.

Interesting enough IBM has part in both specifications and their standpoint can be read in a mail from Steve Holbrook to the W3C maillist to show their support for WS-RA, "Re: Standardization of WS-Eventing":

IBM remains committed to use of the OASIS Standard WS-Notification 1.3 as the interoperability standard for enabling a broad range of applications that include publish/subscribe and event notification in the context of business and complex event processing scenarios. However, we are supportive of the W3C standardization of WS-Eventing because we appreciate that some in the Web Services community are using it in certain domains such as device management and we are interested in ensuring that such a standard compose effectively with other Web services specifications including WS-Notification.

Conclusion

I'm not in a position to really conclude anything except from the obvious that I would have preferred to have one widely supported specification instead of two lightly adopted ones. Ih this I agree with Gartners conclusion from back in 2006, "WS-Notification Standard Ratified by OASIS Still Needs Work";

The delay in reconciling WS-Notification and WS-Eventing shows that the Web services standardization process has lost momentum. Standards have remained static for the past year, permitting only primitive interoperability.

A WS-* offering for Event Driven Architecture - Web Services Notification (WSN)

From my point of view Event Driven Architecture (EDA) is part of SOA and typically run with push(like) setups (pub/sub). Most of the SOA talk has been on request/response oriented setups, probably cause by the common usage of HTTP and no agreed upon interfaces for subscriptions/topics. In this blogpost I'll have a quick look at the OASIS standard Web Services Notification (WSN).

Web Services Notification (WSN) setup

The committee was started back in 2004 when the proposed standard was submitted to OASIS:

The OASIS WSN Technical Committee will coordinate efforts with the OASIS Web Services Resource Framework (WSRF) Technical Committee (also announced today). Members of the OASIS WSN Technical Committee plan to conform their work to the implied resource pattern specified by WSRF and will make use of WSRF specifications concerning lifetime and properties.

So it has a close relation with Web Services Resource Framework (WSRF). The homepage for the OASIS Web Services Notification (WSN) TC has the following overview:

The purpose of the Web Services Notification (WSN) TC is to define a set of specifications that standardise the way Web services interact using "Notifications" or "Events". They form the foundation for Event Driven Architectures built using Web services.

These specifications provide a standardized way for a Web service, or other entity, to disseminate information to a set of other Web services, without having to have prior knowledge of these other Web Services. They can be thought of as defining "Publish/Subscribe for Web services".

Two years later in October 2006 they announced that its members had approved the WS-BaseNotification, WS-BrokeredNotification and WS-Topics specifications as OASIS standards. A simpel and in my opion relevant indicator of chance for succes (for a ws-* standard) is which companies approved the standard, and here it was:

• AmberPoint
• CA
• Fujitsu
• HP
• Hitachi
• IBM
• Oracle
• SAP
• Sonic Software
• Tibco
• webMethods

There are certainly a lot of big companies supporting (with a degree of holeheartness), but in my view Microsoft is missing and to some degree Sun.

Implementation

If a standard is to be used theres a need for (widespread) implementation. OASIS has as part of the standardization process a requiment of several practical implementations (like ex. the w3c). At the time when the standard was approved is was (taken from the announcement):

Amberpoint, Fujitsu, and IBM have all implemented WS-Notification either internally or externally (in keeping with the consortium's requirement that at least three parties assert successful implementation prior to OASIS Standard balloting). Fujitsu's implementation is available as open source tool through unicore.forge.net.

The reference to the Fujitsu open source implementation isn't active anymore but the Apache Muse project does:

The Apache Muse Project is a Java-based implementation of the WS-ResourceFramework (WSRF), WS-BaseNotification (WSN), and WS-DistributedManagement (WSDM) specifications. It is a framework upon which users can build web service interfaces for manageable resources without having to implement all of the "plumbing" described by the aforementioned standards. Applications built with Muse can be deployed in both Apache Axis2 and OSGi environments, and the project includes a set of command line tools that can generate the proper artifacts for your deployment scenario.

The implementation of WSN is only partial, with only WS-BaseNotification of the three specifications that make up the standard.

Update: IBM has support for WSN in their WebSphere suite.

The standard for Web Services Notification (WSN)

To get a more detailed view of the features offered by the three specifications:

• WS-BaseNotification 1.3 OASIS Standard (In PDF it's 68 pages)
• WS-BrokeredNotification 1.3 OASIS Standard (In PDF it's 43 pages)
• WS-Topics 1.3 OASIS Standard (In PDF it's 40 pages)

let's look at their goals and requirements.

WS-BaseNotification

Goal

The goal of WS-BaseNotification is to standardize the terminology, concepts, operations, WSDL and XML needed to express the basic roles involved in Web services publish and subscribe for notification message exchange.

Requirements

Must support resource-constrained devices.

The specifications must be factored in a way that allows resource-constrained devices to participate in the Notification pattern. Such devices will be able to send information to, and receive information from Web services, without having to implement all the features of the specifications.

Must provide runtime metadata

There must be a mechanism that lets a potential Subscriber discover what elements are provided for subscription by a NotificationProducer, and in what formats the subscription for notification can be made.

WS-BaseNotification must be independent of binding-level details

Transport protocol details must be orthogonal to the subscription and the delivery of the notifications, so that the specification can be used over a variety of different transports.

Must allow for Message Oriented Middleware implementations.

The design of the WS-BaseNotification must allow a service that is acting as a NotificationProducer to delegate its implementation of WS-BaseNotification semantics to a Message Oriented Middleware provider.

Relationship to other WS-* specifications

WS-BaseNotification must be composable with other Web services specifications.

WS-BrokeredNotificatio

Goal

The goal of WS-BrokeredNotification is to standardize message exchanges involved in Web services publish and subscribe of a message broker. The overall requirements of WS-Notification are presented in [WS-BaseNotification]. The following section lists the specific subset of those objectives realized by WS-BrokeredNotification.

Requirements

Must allow for a notification broker as an intermediary.

A NotificationBroker is an intermediary Web service that decouples NotificationConsumers from Publishers. A notification broker can relieve a Publisher from having to implement message exchanges associated with NotificationProducer; the NotificationBroker takes on the duties of subscription management and distributing Notifications on behalf of the Publisher. It implements NotificationProducer interface. It may implement SubscriptionManager or may delegate the subscription management work to another component.

Must allow for federation of brokers.

It must be possible to build configurations with multiple intermediary broker services in a dynamic fashion. This specification must allow for a variety of broker topology usage patterns. Among other things, these allow for greater scalability and permit sharing of administrative workload.

Must provide runtime metadata.

There must be a mechanism that lets a potential Subscriber discover what elements available for a subscription are provided by a NotificationBroker, and in what formats the subscription for a notification can be made.

Must conform to WS-BaseNotification.

A NotificationBroker must support required message exchanges defined by the [WS-BaseNotification] specification. It must conform to the NotificationProducer and the NotificationConsumer interfaces defined in WS-BaseNotification.

WS-BrokeredNotification must be independent of binding-level details.

Transport protocol details must be orthogonal to the subscription and the delivery of the notifications, so that the specification can be used over a variety of different transports.

Must not exclude non-service producers and subscribers.

WS-BrokeredNotification design must not exclude a non-service entity to deliver a notification message to a NotificationBroker. It must not exclude a NotificationBroker to send a notification message to a non-service consumer.

Must provide publisher registration.

WS-BrokeredNotification must define standard message exchanges for registering a NotificationPublisher with a NotificationBroker.

WS-Topics

Goal

The goal of the WS-Topics specification is to define a mechanism to organize and categorize items of interest for subscription known as “topics”. It defines a set of topic expression dialects that can be used as subscription expressions in subscribe request messages and other parts of the WS-Notification system.

Requirements

Must support resource-constrained devices.

The specifications must be factored in a way that allows resource-constrained devices to participate in the Notification pattern. Such devices will be able to send information to, and receive information from Web services, without having to implement all the features of the specifications.

Must permit transformation and aggregation of Topics.

It must be possible to construct configurations (using intermediary brokers) where the Topic subscribed to by the NotificationConsumer differs from the Topic published to by the NotificationProducer, yet Notifications from the NotificationProducer are routed to the NotificationConsumer by a broker that is acting according to administratively-defined rules.

Must permit non-centralized development of a topic tree.

It must be possible for actors to define additional topics based on existing topics without requiring coordination with the actor responsible for creating the topics that are being built on.

Conclusion

If I were to spot the "Achilles' heel" of the WS-* stack, it must be the lack of a standard to really replace the classic messaging setups like ex. JMS, though that was what many set out for. It took an incredible long time to finally get something like WSN and even more importantly WS-RX. And that is just the standards - then you have to wait for at product near you to support it! In the meantime some has lost confidence (and maybe gained productivity) but using more propriotairy standards making it even harder to gain critical attraction.

Doing a web search for wsn gives far fewer interesting hits than I would have liked to see. This indicates to me that though WSN has alot to offer the real usage is strongly limited.

A Demo WSDL for part of OIOUBL Simpel Ordering

I want to get back on track with my WCF certification and the best thing to do seems to be rolling a service myself. I've had this post in draft for almost two years now so it's about time i finalize it! To save myself from yet another uninspiring hello world example I've decided to base it on a simple document flow that's part of a basic OIOUBL process " OIOUBL BASPRO - Basic procurement Cycle [PDF]. This will not only give my webservice a touch of the real world but I'll also update my knowledge of OIOUBL slightly.

Here's a couple of diagrams:

There's a complete package to download from oioubl.info and this one is called Basic procurement Cycle (Ref. S03) [BASPRO.zip].

I'll develop this WSDL based on the Contract First principle - that's based on the schemas (and not generate it from code), but where do I find the XML Schemas for OIOUBL? The answer is on the introduction page to OIOUBL in the section "The relationship between OIOUBL and UBL 2.0" (my strong):

As mentioned above, OIOUBL is a subset of UBL 2.0. This means that UBL 2.0 contains a number of business documents in addition to the ones in OIOUBL. For example international transportation documents. These additional business documents are fully compatible with and may be used with the OIOUBL documents. However, no Danish language instructions or guidelines are provided for these documents.

The OIOUBL specifications are based directly upon the UBL 2.0 XML schemas. No additional XML schemas have been developed or are required for OIOUBL.

It should be noted that, for business reasons, a few elements from the UBL 2.0 schemas have been excluded in the OIOUBL customization. These elements have been judged as not applicable in Denmark, either because of specific Danish legislation, or because they are of not relevant in a Danish context. These omitted elements will not be present in the OIOUBL specifications, and if these fields were to be used they will not pass the document validator published by the National IT and Telecom Agency.

So contrary to the first Danish adoption to UBL, here no special schemas so I can use the one from the OASIS website UBL-Order-2.0.xsd. The rest is just boilerplate WSDL.

I've uploaded the WSDL SX_OIOUBLorderDemo.wsdl to digitaliser.dk (the successor of the InfoStructureBase). It's still open to anyone, and the biggest change I've found is that once you've made anything public it can't be edited! Not the file (artifact) nor the resource (metadata), the only way is a new version of the resource.