Thursday, January 26, 2006

App integration: processing a flat file

The ideal case for inter-application or b2b integration would be a process that is compatible enough to take direct advantage of a Service Oriented Architecture. Therefore, integration efforts would be a matter of having an agent that invokes web services on the other end.
Unfortunately, this is rarely the case. Most times synchronous processing is not possible for technical (performance, scalability) reasons or enterprise policies. We can still use SOA and throw some queuing service (like MSMQ or MQSeries) between application A and its Service Agent and/or application B and its Web Service interface. However, most times this is just not enough because the legacy processes that will dump a text delimited file somewhere in a directory. No XML, no web services, just a plain tab delimited file. Under that scenario, an ad-hoc solution would be:
1. Write some windows service that is always watching the income directory, parses the tab delimited file with code, wraps the data into xml documents and pushes them to web service exposed functionality of the target application. This solution is cheap, lame, not scaleable, not extensible and hardly reusable.
2. Use Biztalk Server 2004.
I am going to elaborate on option B. And I know this is not new or fantastic but what I’ll do is just list exactly how to do it so if someone reading this comes across this scenario won’t have to google it and try to fish for this recipe:
Some Biztalk in a nutshell: Biztalk input and output happens through ports. Messages enters Biztalk through a receive port and leaves through a send port. In this case we will focus on a Receive Port. Ports are made of:
Adapter – the component that actually reads the bytes from the source. There are many out of the box adapters.
Pipeline – the component that encodes/decodes the data from and to the Biztalk native format: XML.
Map – transforms the source xml message into something different using XSL.
So, to read an input tab delimited file into Biztalk we have to:
Create the schema that defines the incoming message. Yeah, it’s kind of weird but Biztalk wants us to define flat files with an XSD. We need to specify Flat File Extensions in the schema editor extension property. This is the actual schema for an object called Entity:


<?xml version="1.0" encoding="utf-16"?>
<xs:schema xmlns="http://MessagingPrototype.TabEntities" xmlns:b="http://schemas.microsoft.com/BizTalk/2003" targetNamespace="http://MessagingPrototype.TabEntities" xmlns:xs="http://www.w3.org/2001/XMLSchema">
<xs:annotation>
<xs:appinfo>
<b:schemaInfo count_positions_by_byte="false" standard="Flat File" root_reference="Entities" parser_optimization="speed" lookahead_depth="3" suppress_empty_nodes="false" generate_empty_nodes="true" allow_early_termination="false" />
<schemaEditorExtension:schemaInfo namespaceAlias="b" extensionClass="Microsoft.BizTalk.FlatFileExtension.FlatFileExtension" standardName="Flat File" xmlns:schemaEditorExtension="http://schemas.microsoft.com/BizTalk/2003/SchemaEditorExtensions" />
</xs:appinfo>
</xs:annotation>
<xs:element name="Entities">
<xs:annotation>
<xs:appinfo>
<b:recordInfo structure="delimited" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" sequence_number="1" child_delimiter_type="hex" child_delimiter="0x0D 0x0A" child_order="postfix" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<b:groupInfo sequence_number="0" />
</xs:appinfo>
</xs:annotation>
<xs:element maxOccurs="1" name="Entity">
<xs:annotation>
<xs:appinfo>
<b:recordInfo structure="delimited" preserve_delimiter_for_empty_data="true" suppress_trailing_delimiters="false" child_delimiter_type="hex" child_order="infix" sequence_number="1" child_delimiter="0x09" />
</xs:appinfo>
</xs:annotation>
<xs:complexType>
<xs:sequence>
<xs:annotation>
<xs:appinfo>
<b:groupInfo sequence_number="0" />
</xs:appinfo>
</xs:annotation>
<xs:element name="Name" type="xs:string">
</xs:element>
<xs:element name="Code" type="xs:string">
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:sequence>
</xs:complexType>
</xs:element>
</xs:schema>


Note: this schema will automatically indicate the pipeline that the file should be split into individual Entity messages.
Next, since the default pipelines XML or pass-through are not enough we have to create a custom pipeline but all we do is drag and drop the flat file disassembler and specify the above as document schema.
Next we create a new receive port with new port type, specifying the port binding and providing the directory location in the URI (something like d:/InputFiles/*.tsv) and FILE transport. The receive pipeline should be the custom pipeline created above.
For me, the key was to find the right combination of schema element parameters. This port will read a flat file and split it into multiple xml messages as soon as a file shows up in the specified directory.
This solution can then invoke an orchestration that takes advantage of send-receive ports for synchronous web service calls and send ports for queue and database access. It leverages SOA and enables the potential for using Biztalk as an Enterprise Services Bus. An ESB enables optimization of the distribution of information between different types of applications across multiple locations.

Tuesday, December 20, 2005

Sprint SMS C# function

I was wondering if I could, programatically, send SMS text to Sprint PCS phones. Sprint does not expose that functionality through web services and actually try to prevent users from sending sms messages programatically by using an otherwise useless cookie that is setup by the compose message page..
So the workaround is to get this page first using an HttpRequest object, clone the cookie and send the message request with this cookie. The code is actually very simple:

public static void SendSmsMessage(string paraQuien, string msg, string deQuien) {
    // getting the cookie first otherwise sprint refuses to send the msg
    // they think they're smart and will prevent some msg sending automation
    // so let's pretend we will compose a message
    string url = "http://messaging.sprintpcs.com/textmessaging/compose";
    System.Net.HttpWebRequest request0 = (HttpWebRequest)WebRequest.Create(url);
    request0.Method="GET";
    System.Net.HttpWebResponse response0 = (HttpWebResponse)request0.GetResponse();
    // session cookie comes in the Set-Cookie header in the form of:
    // Set-Cookie: JSESSIONID=DnZqWgveQBOfrvjcq4FEvDB1IPiJ2BRYJBUU15LL21bYqLXYVhQN!1266095294!182801593!7060!7002;path=/textmessaging
    string oreo = response0.Headers["Set-Cookie"];
    // some cheap string manipulation to get the actual value
    string cookieValue = oreo.Substring(oreo.IndexOf("=")+1);
    cookieValue = cookieValue.Substring(0,cookieValue.Length-21);
    // creating new cookie to send in request
    Cookie galleta = new Cookie("JSESSIONID",cookieValue);
    galleta.Domain="messaging.sprintpcs.com";
    galleta.Path="/textmessaging";
    url = "http://messaging.sprintpcs.com/textmessaging/composeconfirm";
    msg = System.Web.HttpUtility.UrlEncode(msg);
    string postData = "randomtoken=&phoneNumber="+paraQuien+"&message="+msg+"&callBackNumber="+deQuien;
    // in the website is a POST but it works with a GET sending
    // the same valiables in the querystring
    System.Net.HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url+"?"+postData);
    CookieContainer gamesa = new CookieContainer();
    gamesa.Add(galleta);
    request.CookieContainer = gamesa;
    request.Referer="http://messaging.sprintpcs.com/textmessaging/compose";
    request.ContentType="application/x-www-form-urlencoded";
    request.Method = "GET";
    System.Net.HttpWebResponse response = (HttpWebResponse)request.GetResponse();
    // Get the stream associated with the response.
    StreamReader readStream = new StreamReader(response.GetResponseStream(), Encoding.UTF8);
    string strHtml = readStream.ReadToEnd();
    response.Close();
    readStream.Close();
}

This function can send you sms messages based on whatever events you code!

Weather Web Service ?

With the advent of XML most IT people are really excited about application and business integration, web services, service oriented architecture and xml related technologies. However, for a normal person it seems like there isn't much WS stuff that is available. At least not for free. I was trying to get some free web services for a weather report and I only found Capescience Global Weather. But this free WS is not very reliable; it goes down too often and it's very slow. So, ironically, for an ordinary geek the best alternative was to use good old screenscraping from the CNN Global Weather web page. So, despite the really cool technologies happening nowadays this post will not relate to AJAX, ATLAS or ASP 2.0 but just give out some very useful C# code for screen scraping.
The code parameter is a CNN specific 3 or 4 character code like ORD for Chicago and MEXX for Mexico City.


public static string GetCNNWeather(string code){
// very cheap screen scraping
string cnnUrl = "http://weather.edition.cnn.com/weather/intl/forecast.jsp?locCode="+code;
System.Net.HttpWebRequest request = (HttpWebRequest)WebRequest.Create(cnnUrl);
System.Net.HttpWebResponse response = (HttpWebResponse)request.GetResponse();
// Get the stream associated with the response.
Stream receiveStream = response.GetResponseStream();
// Pipes the stream to a higher level stream reader with the required encoding format.
StreamReader readStream = new StreamReader (receiveStream, Encoding.UTF8);
string strHtml = readStream.ReadToEnd();
response.Close();
readStream.Close();
int donde = strHtml.IndexOf("°F");
string parte = strHtml.Substring(donde-5,5);
for(int i=0;i<5;i++)if(Char.IsNumber(parte[i])||parte[i]=='-')return parte.Substring(i);
return parte;
}

All I have to do now is wrap this cheap method into a web service and re-expose it through wk4.com whenever I get a new server up and running. The old dual pentuim 200MHz clunky machine that hosted wk4.com is not worth plugging in again. Meanwhile you can go ahead and play with this retro piece of code.

Monday, December 05, 2005

Erratum

Just got this in my mailbox:

Eduardo!
That's all hand crafted code my friend.
http://www.ci-inc.com/imagecenter/barcode/
I had to abandon the Al Tova software because it was too limited
they don't support a lot of things fop does
things I need to do.
So please don't mislead people into thinking Al and I are working together.
Al and I spent an active few weeks together but we have parted ways.
Perhaps in the future we'll work together again but for now we are on different paths.
If there is to be a tip of the (red)hat to people who helped my bar code project it should be to www.reliablepenguin.com, www.krysalis.org, and www.ezgen.com and of course the limitless support from bang boom pow technologies: www.bangboompow.com
Let's give credit where credit is due.
wonderBoy

XML to PDF using XSL

I recently became aware of the masterful work of Wonderboy. He teamed up with my old friend Al Tova and came up with some really cool XSL transformation that produces a PDF out of XML data.

Check it out: http://www.ci-inc.com/imagecenter/barcode/

Validation

Validation is one of the most annoying tasks for developers. Different application architects have expressed different opinions in different Starbucks downtown Chicago. So, here is my 2 cents:
Validation cannot occur at the UI layer. If validation happens at the UI layer, the actual business tier has to assume valid requests. Reusability and encapsulation of business components then is diminished because every client - let’s say a PDA, a web browser, and a smart client - have to re-implement validation hence code is excessive and redundant. So, the UI should just gather the data and submit it. Usually this is done by creating and populating an object. So, maybe this object should have some kind of validation rules coded in a method ? NO.
Validation cannot be hard coded in entities. As entities I mean these objects we create in the UI layer and ship to the subsequent layers for processing. These data transfer objects should have no behavior because they are like java beans that just carry data. Additionally, these objects are reusable and different functions of the application may use the same object in different ways and require different validation. Add and Update may have different fields as required.
Validation belongs in the business layer. Because validation is itself a business rule, it should belong where the business rules are coded. This approach allows for maximum reusability and abstracts validation rules from a particular client. This approach is consistent with SOA.
Validation should not throw an exception. When the user forgets to fill out a field we don’t need to publish it. Handling “decaf” exceptions different would be very annoying.
A good idea is to actually code validation in a separate namespace so it can be invoked within the business layer and from the UI layer also. For this to happen, the entities (or data transfer objects) can use a series of interchangeable validation algorithms suitable for every occasion (like add and update). This can be achieved with an IValidate interface that is implemented by all validations and included in a BaseEntity object. A validation message is also very handy:

public class BaseEntity{
private string validationMessage;
public string ValidationMessage {
set {validationMessage=value; }
get {return validationMessage;}
}
private IValidate validationStrategy = null;
public IValidate Validation {
set { validationStrategy=value; }
}

public bool IsValid {
get {
return validationStrategy==null?true:validationStrategy.Validate(this);
}
}
}

public interface IValidate {
bool Validate(BaseEntity obj);
}

All the DTO / Entity objects then derive from BaseEntity and are able to take a validation object that implements IValidate. The IsValid property then can be used at the UI layer or the business layer.
The IValidate implementation has to cast the object to the specific type it validates.

public class Person:BaseEntity {
public string firstName;
public string lastName;
}

public class AddPersonValidation:IValidate{
public bool Validate(BaseEntity person)
{
if(((Person)person).firstName==null ((Person)person).lastName==null)
{
person.ValidationMessage="First and last name are required";
return false;
}
return true;
}
}

The object takes a specific validation class:

Person p = new Person();
p.lastName="Gomez";
p.Validation=new AddPersonValidation();

bool b = p.IsValid; // p.ValidationMessage can be used from the UI

Are asp.net validation controls useless ? No. We can always do it again with one of those. We’d be redundant but at least this validation framework provides reusability of business components between different client types. It doesn’t harm that an aspx page uses a validation control, but if we validate an the business layer also, we ensure that if the functionality gets exposed through a web service the business layer will not perform the requested function if !IsValid.

Wednesday, November 30, 2005

Announcement

I've decided to drop all skydiving content from this blog. Skydivers belong in the sky and/or dropzone.com forums so this blog will be about technology.

Hello

I will dedicate this space to pontificate on my very own extreme, bizarre and always evolving opinions about .net technology / XML and other ad-hoc topics.
I was reading about AJAX in this article:
http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnaspp/html/ASPNetSpicedAjax.asp
and I think this AJAX implementation leads to bad architectural practices as it would be easy to break the ntier rules of an application. Behind the scenes, somehow Javascript is using the XmlHttpRequest object client side. And that's cool. But the obscure implementation of the actual server side functions I dislike. Using attributes and an http handler is adding more complexity to application architecture and making AJAX a particularity.
I would rather comply with SOA and implement AJAX in such a way that the XmlHttpRequest object invokes web services. This way the clientside javascript would only invoke already exposed functionality. This would increase reusability, maintainability and is architecturally very neat. No need for an http handler.
On the other hand, are we going back to the past ? Client-side http posts "on the fly" were originally implemented with Java applets. So, are java applets cool again ?
Maybe the difference is that this time client side http posts are good because is XML.
Ajax is also a great cleaning product:
http://www.colgate.com.mx/productos_hogar_liquido.html

This page is powered by Blogger. Isn't yours?