BizTalk Flat Text File Processing

By Chaitanya Talasila

Learn the in's and out's of processing flat text files in and out of BizTalk.

Translations in BizTallk Server means transforming a Document format to another format
Any type of file to be translated to XML, or XML to any type of file

Example:

Translating flat files to xml files and any type of files to xml.

You may doubt why the translation is required; the answer is BizTalk Server only understands xml Documents and files coming to BizTallk Server need not be XML Documents. So Changing the format is necessary.


Tthese translations are done at receive port and send ports

For Transforming one format to another format other than xml files at Receive Port, we Must Custom Require Receive pipeline.
In the same way, for translating the xml to any file format at Send port, we Must Require Custom Send Pipeline.

Generally two types of files are these

1. Flat files

2. Xml files

Flat file : It is any file which is readable and does not have any description about the data in the file

Example for Flat File

1 Chaitanya BizTalk 23000

2 Pradeep Silver light 25000

3 Ramu Asp.net 23000

This is an employee information flat file. It gives only data , but does not provide any description on the Data like empID ,EmpName,Technology,Salary

We have data Chaitanya , what is it? It does not have any Description i.e. EmpName

So flat files don’t have any description




See the Above xml file it has Data and Description about the Data

You understand the difference between xml files and flat files

Flat file types

There are two types of flat files

1. Delimiter flat files

2. Positional flat files

delimiter flat files

These are Separated by any Separator( separator is like ,(Comma), (Space),(Enter) Take the previous flat file

1, Chaitanya, Biztalk,23000

2 ,Pradeep,Silverlight,25000

3 ,Ramu,Asp.net,23000

Here the fields like 1, Chaitanya are separated by Separator( ,(COMMA))

And first Line whole is a Record and the Every Record is Separated by Separator(Enter)

So in this flat file, There are 2 separators .

i.e. field separator is (COMMA) and Record Separator is (Enter).

So you can understand this Delimited flat file.

Positional Flat files

Positional flat files are Files , in this Data is separated by Column Index.

Take the previous flat file

1-25 26-50 51-75 76-100

1 Chaitanya Biztalk 23000

2 Pradeep Silverlight 25000

3 Ramu Asp.net 23000

Here each and every field is given with column index.

If you have EmployeeID that EmployeeID (1,2,3) must be between 1-25 columns,

That is for the employee id 25 column index is assigned.

Drawback

EmployeeID 1 takes only one columns and Remaining Columns Memory is wasted.

Memory Management issue comes here.

Note

So from these Delimited Flat files and Positional flat files.. Delimited flat files are best.


Process

For the flat file coming to receive port it must be go to Disassemble stage in order for Translating flat file to XML file

For this Translation first the flat file document must be validated

For validating flat file document , we Require Schema for Flat files.

So Our next step is Creating the Schema for flat files

So take flat file document i.e Previous only

1, Chaitanya, Biztalk,23000

2 ,Pradeep,Silverlight,25000

3 ,Ramu,Asp.net,23000

For this we have to prepare the schema, We have give the specifications for Building the Flat file schema.

The Specifications are

For any flat file schema, Delimiter is Specified at Parent Level

Suppose take above flat file we have delimiter at fields.

So we have to specify the delimiter specification at Record Level.

Record delimiter is Specified at Record Set Level(parent)

Ex: EmployeeRecordSet -------à Record Set

Employee Info ------à Record

ID ------------->Fields

Name ------- ->Fields

Technology ---->Fields

Salary ---------->Fields

To specify the delimiter 3 properties have to set

Field Level delimiter

Select Employee Info Record Properties,

Here delimiter is COMMA for fields

Set the Properties

1. Child Delimiter Type :Character

Because COMMA is Character Type

2. Child Delimiter:,

Here Child delimiter is Comma so place symbol here

3. Child Order : Prefix

Child Order means it asks where the Delimiter is ther

We have 3 options

1. Prefix: if the delimiter is before the field

2. Infix: if the delimiter is between fields

3.Postfix:if delimiter is after fields

So our delimiter COMMA is between fields only so Select Prefix Option

So filed level delimiter is over

Record level delimiter

For this select Record Set EmployeeRecordSet Properties

Set the 3 properties

1.Child Delimiter Type :Hexadecimal (Enter is not Character Type)

For enter button delimiter value is

\n ----àit is working fine for windows OS

\r ----à It is working fine for Linux OS

But in xml we don’t use \ because it doesn’t process after \

So the Hexadecimal Value for this is

\n----à0X0D

\r---àOXOA

So Combined Combination is

0X0D OXOA :It is our Delimiter.

It can work both on linux and Windows.

Windows Check OXOD and Complies

Linux Checks OXOD so wrong and it also checks OXOA so Right. And did compilation.

Space is placed between two hexadecimal values for not giving any problems.

OXODOXOA--àCompiler did not Understand and gives error

So

2. Child Delimiter: 0X0D OXOA

and 3rd option is

3. Child Order : Infix

It is always infix for every record level delimiter

For any Conversion of files to xml document we must use Custom Receive pipelines

So Add Custom Receive pipeline to the BizTalk Project,And in the Receive pipeline, we have 4 stages

In these four stages , Translations are done at Disassemble stage only,i.e we have to implement the disassemble component in this stage.

So, Add flat file disassemble component and in the disassemble component properties select Document Schema property and give our Document schema.




Implementation

Create 2 folders In and Out for input flat file and out put xml file.

So create two ports Receive and send ports and configure those ports with that Folders

Add an orchestration to the BizTalk project

For receive port , select the custom receive pipeline and configure the URI with

C:/In/*.txt

And configure the send port

Creating the flat file instance

Select the schema, select the properties, select output file name,

Select generate output instance type: Native

And click on generate Instance

Drop the flat file document in the In folder, then BizTalk translates the flat file and we got the xml file in the Out folder


Popularity  (3634 Views)
Create New Account
Article Discussion: Flat file Processing
Chaitanya Talasila posted at Monday, November 02, 2009 4:47 AM
sree replied to Chaitanya Talasila at Monday, November 23, 2009 9:17 AM
Hello,

Thank you very much for your article and its very helpful for the newbie

I have almost similar situation of Processing FlatFile to XML file, and I am successful in developing the solution(xml file from Flatfile) as the output of the map, but the same flatfile is routed through ports via pipeline i am receiving an error saying :

 "  Unexpected end of stream while looking for:

'\r\n'

$Delimited$   "

Please let me know the solution for this