Kasun's Blog

Gregor Hohpe : The author of Enterprise Integration Patterns - Keynote speaker at WSO2Con 2011

2011-09-09T20:20:00.000-07:00

"Enterprise integration remains harder than it really should be. While integration is inherently complex, I felt that one of the major stumbling blocks is the lack of a common vocabulary and body of knowledge around asynchronous messaging architectures used to build integration solutions. Under the guidance of Martin Fowler and Kyle Brown, I teamed up with Bobby Woolf to create such a language in the form of 65 integration patterns"

That's a phrase from the introduction of the book "Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions" by Gregor Hohpe, Staff Software Engineer, Google Inc. and Bobby Woolf.

If you talk to any Enterprise Service Bus(ESB) vendor/developer or a SOA architect, you'll never found a person who haven't read 'Enterprise Integration Patterns' book. Even some ESB vendors use the pattern templates(same images etc.) discussed in the book for their graphical flow designing tools of ESB (eg. Fuse IDE, Mule IDE) . That shows the impact of this book in the world of SOA.

In a nutshell, 'Enterprise Integration Patterns' by Gregor is the De facto standard for EIP and ESB.

This year, WSO2Con brings you, "Past, Present and Future of EIPs" with Gregor. Gregor Hope will be doing the keynote speech at WSO2Con 2011. So, this is a yet another "Must Attend" speech at the WSO2Con.

Don't miss.... WSO2Con 2011
Enterprise Integration Patterns: Past, Present and Future : Gregor Hope,
9.00 IST Thursday, September 15, 2011

Don't miss ... WSO2Con 2011

2011-09-08T08:08:00.000-07:00

It's almost a year since the last and the very first WSO2 Con which was held in 2010. It was a successful conference where we divulged the power of WSO2 product stack and WSO2 Cloud Platform: Stratos.

Things have dramatically changed this year and WSO2 Con 2011 will be one of the major event in the world of SOA and Cloud Computing. And could find a best time to blog about it, because WSO2 has just won the "The best open source data center and cloud software" in Bossie Awards 2011.

WSO2Con 2011 is one fantastic week of tutorials, tech talk and networking events. Whether you are a developer, architect, IT manager or technology enthusiast, learn how global enterprises, SaaS providers and innovative startups are using WSO2 platforms to build distributed web apps, java services, bpel flows, Software-as-a-Service (SaaS) and more. And it comprises of the speakers from more than 10 countries... including Industry Leaders from eBay, IBM, Google, Cognizant, T-Mobile, Two Degrees and ALC.

Industry Leaders Share Their Inspirations - Keynote Speakers

IBM Global Technology Outlook – 2011
Dr. C Mohan

Service Orientation – Why is it good for your business
Sastry Malladi

Enterprise Integration Patterns: Past, Present and Future
Gregor Hohpe

SOA & Beyond using open source technologies
Narendra Nethmal

WSO2: Disrupting the middleware industry
Sanjiva Weerawarana Ph.D

WSO2 Vision and Roadmap
Paul Fremantle
Founder & CTO WSO2 Inc.

A closer look at WSO2 ESB 4.0.0

2011-06-19T11:54:00.000-07:00

WSO2 ESB v4.0.0 Released - 13th June 2011

WSO2 ESB is a fast, lightweight and user friendly open source Enterprise Service Bus (ESB) distributed under the Apache Software License v2.0. WSO2 ESB allows system administrators and developers to easily configure message routing, intermediation, transformation, logging, task scheduling, fail over routing and load balancing. It also supports transport switching, eventing, rule based mediation and priority based mediation for advanced integration requirements. The ESB runtime is designed to be completely asynchronous, non-blocking and streaming based on the Apache Synapse mediation engine.

WSO2 ESB 4.0.0 is developed on top of the revolutionary WSO2 Carbon platform (Middleware a' la carte), an OSGi based framework that provides seamless modularity to your SOA via componentization. This release also contains many new features and a range of optional components (add-ons) that can be installed to customize the behavior of the ESB. Further, any existing features of the ESB which are not required to your environment can be easily removed using the underlying provisioning framework of Carbon. In brief, WSO2 ESB can be fully customized and tailored to meet your exact SOA needs.

New Features

HTTP relay transport - Pure streaming HTTP transport based on Java NIO for ultra fast HTTP routing
Deployment synchronizer - Easily synchronize the configuration across a cluster of Carbon Servers
Message stores and processors - For advanced enterprise messaging and QoS enforcement (eg: in-order delivery, exactly once delivery)
Mediation templates - The concept of reusable functions to the ESB configuration language
Message broker - Built-in JMS provider/broker powered by Apache Qpid
FTPS support via file transport - Transfer files over SSH connections
Conditional router mediator - Implement advanced and complex routing logic at ease
URL rewrite mediator - Rewrite URLs and URL fragments efficiently
Custom XPath extensions - User defined custom XPath extensions support
New HL7 transport (optional) - Send and receive HL7 messages over MLLP and other transports
New eventing implementation - A powerful event broker with extensive support for WS-Eventing and topic management
Event mediator - Publish events to topics defined in the built-in event broker
LDAP user store - Built-in LDAP service based on Apache Directory Server
Dynamic registry keys and endpoint URL support - Construct registry lookup keys and endpoint URLs dynamically at mediation time

Enhancements
Improved Smooks mediator
Enhanced support for service chaining with the receiving sequence support in the send mediator
Improved weighted round robin algorithm for the load balance endpoint
Improvements to caching
Automatic schema imports from the WSDLs added to the registry (for proxy services)
Automatic schema imports from the schema files added to the registry (for validate mediator)
Support for setting properties at the endpoint level
Enhanced Enrich Mediator
Improved support for batch transfers in FIX transport
Validation support for the ESB configuration editor (source view)
Proxy service UI enhancements
Enhanced menu layouts
Local transport for fast in-VM service calls

Became an Apache Synapse committer...

2011-05-23T09:17:00.000-07:00

It's an honor to become a committer of Apache Synapse[1], which is the core engine of one of the world's leading open source ESB. I must thank the whole Synapse community for trusting me as a committer. And my special gratitude goes to Supun, Hiranya and Ruwan for their guidance and help given.

[1] http://old.nabble.com/-VOTE--Kasun-Indrasiri-as-a-committer-for-Synapse-td31621022.html#a31621070

WSO2 ESB - An answer to Enterprise Service Integration

2011-02-10T04:09:00.000-08:00

In the modern enterprise, complex business requirements are increasingly satisfied by a combination of new and existing services. Getting the right bits of information at the right place, at the right time, in the right format, and with the right protocol can be a challenge.

The WSO2 Enterprise Service Bus (ESB) provides a simple solution to this integration problem by providing a rich set of integration capabilities for robust, scalable and seamless integration of such services.

Topics covered include:

Understanding the basics: mediators and mediation flows, proxy services, endpoints
End-to-end scenario on service integration using the WSO2 ESB
Heterogeneous service integration and complex message mediation features.

Register Now

Thursday, 10th February 2011
9:00 AM - 10:00 AM (PST)
Presenter: Kasun Indrasiri

Thursday, 10th February 2011
10:00 AM - 11:00 AM (GMT)
Presenter: Kasun Indrasiri

REST calls with CURL - HTTP GET, POST, and PUT with curl

2010-11-06T14:57:00.000-07:00

Curl is a pretty handy command line utility when it comes to web services and network related stuff.

Also you can you java bench as an alternative.

Splitting and merging large files in Linux

2010-09-22T08:48:00.000-07:00

In some cases you won't be able to transfer very large files using storage devices. So the best way is to use split a given file in to arbitrary no of volumes at the source and them merge at the destination.
This will simply do that for you.

Splitting

Say you have a directory/file called 'foo'. First you need to create a '.tar.gz'.


tar -cvzf foo.tar.gz foo

Then you can split it using split command and as a parameter you can give the size of a each volume. (here its 25MB)


 split -d -b25m foo.tar.gz foo.tar.gz.

Merge

Simply you can merger using the cat command.


cat foo.tar.gz.0* >foo.tar.gz

Now you can untar this and use the original file/directory.

Retrospective - WSO2 Con 2010

2010-09-15T11:10:00.000-07:00

The first ever WSO2Con 2010 was successfully concluded on 15th of September 2010. It was a giant step in the Sri Lankan Software Industry where a Sri Lankan based company was able to held a world class conference that touches the cutting edge technologies in the Enterprise Middle-ware world.

Being a member of the WSO2Con samples team, I was involved in the development of the sample scenario on the perspective of WSO2 ESB. It was an exciting couple of days that all of us worked with tremendous determination of making it a success. So, finally it paid off.. everything went very smoothly and the demo was a near perfection. Both Carbon and Stratos demos of the shopping cart app was a real success and we were able to use most of the WSO2 Products in that demo.

Azeez's presentation of the demo on top of Cloud went really well.

Panel Discussion

I developed the ESB related configurations and integrated the all the components through the WSO2 ESB. I would like to explain the complete scenario in ESB perspective from a separate post.

So, I would like to thank each and everyone who help us to make this a success. cheers!!

Understanding Cloud Computing

2010-07-25T01:15:00.001-07:00

'Cloud Computing', a buzz words that is frequently pops out in modern day discussions with IT professionals. Being a buzz word, Cloud Computing has proven to be a confusion term with respective to most IT fields. So, rather than 'defining' Cloud Computing, its worthy to write about how it has evolved during last few years and then go for the conceptual things on Cloud Computing.

Business App Nightmare

Nowadays, every modern business has an Enterprise Business Application as its backbone. For instance, for a given company, it may require a Business App, to manages its human resources and to support its business process in efficient manner. Traditional Business Applications are too complex and too expensive.

A traditional Business App often have most of these characteristics.

Often required to have a dedicated datacentre and office space with power,cooling, bandwidth, network, servers and storage.
Consists of a complicated software stack and required a team of exprets to install,configure and run the system.
Required development,testing, staging, productions and fail-over environments.
In case of a failure in any of these systems, we have to go for expensive commercial support and yet we can't guarantee that our prolem will get solved in time, without affecting our businees.
When new versions comes out, it very likely to bring the whole system down for sometime for inhereient incompatibilities
This is just for one single Business App, but imagine what would happen with multiple business apps.
Organization concentrate more on the Business App rather than the business itself.

So, obviously we can't live with such traditional Business Aplication in this dynamic and competitive business world. We need a better and hassle free way of running our business. And that's where the 'Cloud Computing' comes to save us.

Cloud Computing

Cloud computing is a technology that uses the internet and central remote servers to maintain data and applications. It provides on demand resources and services over the internet with the power of scalability and reliability.

In simple terms, when you are using cloud computing, you don't need to install the required application on your system. Instead, you use the application that runs on a remote location/datacenter which we called the 'Cloud'. You just login, customize and start using it.

Gmail is the perfect example of Cloud Computing. You don't need a software system or mail server to send/receive emails. You just login to it, customize it and start using it. Unlike other traditional email management systems like MS Exchange, Gmail doesn't requires a software system, mail server, regular upgrades or dedicated team to manage it. Instead, everything is placed in the Cloud (and Cloud have all those things) and the users get all the benifits that are provided 'as a service'.

Cloud Computing for Enterprises

In the context of the Enterprises, all the traditional application that required in enterprises like HR,CRM and accoundintg apps can be cloudify. Which means that, running them on a cloud and any given business can customize it and use it in their allocated workspace.

Here we use the concept of 'Multitenancy', where a single instance of the software runs on a server, serving multiple clients(tenants).

So for instance, say a Enterprise Cloud App runs on a cloud and different businesses (tenants) are using it.

In this case, every business use the same enterprise cloud application, but it is flexible enough to everyone to customize it with their different requirements. Also version upgrades is no more a hassle for us, as they are upgraded automatically and our system becomes more reliable, more scalable and more secure.

Cloud Computing Models

Software as a Service (SaaS)

This is the most widely used Cloud Computing approach to date. A business application can be run on the centralized servers(cloud) rather than running them on on-site servers/software systems. On Demand

Software as a service, delivers a single application through the browser to thousands of customers using a multitenant architecture. In the cloud service side, it only requires to maintain and manage one application on the cloud and in the client side, there are no any need of upfront investing on servers, software and license etc.

Salesforce, Google, NetSuit, Taleo, Concur Technologies

Platform as a Service (PaaS)

PaaS delivers development environments as a service. You build your own applications that run on the provider's infrastructure and are delivered to your users via the Internet from the provider's servers.

Salesforce's platform, WSo2 Stratos, Amazon Elastic Compute Cloud (EC2), Google App Engine, Coghead, Yahoo pipes, Windows Azure

Infrastructure as a Service (IaaS)

Infrastructure as a Service, is very much the backbone of the entire cloud computing concept. A well known examples include, Infrastructure vendors environments like Google gears which allow users to build applications and Cloud storages, such as Amazon Simple Storage Service (S3) which allows user to store and retrieve any amount of data, at any time, from anywhere on the web.

Google Gears, Amazon S3

Cloud Computing Deployment Models

Public Clouds

In a public clouds, the services and infrastructure are provided off-site over the Internet. These clouds offer the greatest level of efficiency in shared resources; however, they are less secured and more vulnerable than private clouds.

Private Clouds

Unlike public clouds, in the Private Clouds, the services and infrastructure are maintained on a private network. These clouds offer the greatest level of security and control. However they require the company to still purchase and maintain all the software and infrastructure.

Hybrid Clouds

A hybrid cloud includes a variety of public and private options with multiple providers.

Pricing Schema

Cloud computing is often offered with a pricing model that lets you pay as you go and for just the services that you need. No capital expenditure is required.

WSO2 Stratos : Platform-as-a-Service

The first 100% open source cloud platform for enterprise applications, WSO2 Stratos offers organizations of all sizes a fully hosted application platform-as-a-service (PaaS) that is built on top of and extends WSO2 Carbon—the award-winning, OSGi-compliant middleware.

Increased productivity: cut project time using the automated, integrated and highly scalable infrastructure of WSO2 Stratos.

Self-provisioning and Management : results in a near-zero time delay between developing applications and deploying them out to users.

Multi-tenancy: applications supporting different business units or regional offices can be delivered cost-effectively from a single location.

Cost reduction: optimize datacenter utilization and pay only for what is actually used.

Control: govern all applications and services with built-in governance automation.

WSO2 Cloud Services

WSO2 ESB as a Service

Meeting CIO Challenges with WSO2 Stratos

Next big thing..

source: salesforce

WSO2 Stratos - 100% Open Source Cloud Computing Platform

2010-06-03T10:49:00.000-07:00

WSO2 announced the debut of WSO2 Stratos, the first 100% open source cloud platform for enterprise applications. WSO2 Stratos offers organizations of all sizes a fully hosted application platform-as-a-service (PaaS).

Try it Out

WSO2 Stratos Services

Download - WSO2 Stratos

WSO2 Cloud Connectors

Using WSO2 Stratos, IT professionals can build and deploy applications and services with instant provisioning of enterprise servers, including the portal, enterprise service bus (ESB), and application server. WSO2 Stratos is built on top of and extends WSO2 Carbon—the award-winning, OSGi-compliant middleware.

WSO2 enters the emerging market for enterprise PaaS. WSO2 Stratos provides IT organizations with three key benefits:

Productivity: cut project time using the automated, integrated and highly scalable infrastructure of WSO2 Stratos.
Cost reduction: optimize datacenter utilization and pay only for what is actually used.
Control: govern all applications and services with built-in governance automation.

Because WSO2 Stratos is based on the open source WSO2 Carbon, applications can be migrated on premise, to a private cloud or to the public cloud.

Not only does this provide unprecedented deployment flexibility, but for the first time, enterprises can smoothly and safely migrate to the cloud—without the complexity or fear of vendor lock-in.

WSO2 Stratos goes far beyond basic application frameworks to provide an enterprise cloud platform, which manages and deploys key services based on WSO2’s production-quality runtime engines including:

ESB (based on WSO2 Enterprise Service Bus)
Portals (based on WSO2 Gadget Server)
Web applications and services (based on WSO2 Web Services Application Server)
Identity management (based on WSO2 Identity Server)
Governance and registry (based on WSO2 Governance Registry)

For more info, please visit WSO2 Stratos

Visit the Miracle - Sri Lanka

2010-05-23T06:34:00.000-07:00

Sri Lanka celebrates its 1st anniversary of the end of civil war that was lasted almost 3 decades. Now Sri Lankans enjoy the true taste of 'Peace and Harmony'. Despite of the fact that there are some more activities to be done to establish true democracy and economic stability, Sri Lanka can be easily rated as one of the best places to visit. In January 2010 New York Times rates Sri Lanka as No 1 of 'The 31 Places to Go in 2010'.

Retrospective

Although Sri Lanka came across some political instability last year, people enjoys the new born freedom and many foreigns have visited Sri Lanka in 2010. Its obvious that country is more or less going in the correct direction and its the responsibility of the Sri Lankans to guide the government in the proper direction.

It's a great feeling to live in a country that has just came across such a pathetic situation and now.. moving forward with a great confidence. There are no wonders why NY Times has rated Sri Lanka as the no1 place to visit in 2010. The article it self justifies what were the motivations behind the rankings.
I'm quoting from the article created by New York times and in-between I've added some photos taken by me that shows beauty of the island nation.

For a quarter century, Sri Lanka seems to have been plagued by misfortune, including a brutal civil war between the Sinhalese-dominated government and a separatist Tamil group. But the conflict finally ended last May, ushering in a more peaceful era for this teardrop-shaped island off India’s coast, rich in natural beauty and cultural splendors.

The island, with a population of just 20 million, feels like one big tropical zoo: elephants roam freely, water buffaloes idle in paddy fields and monkeys swing from trees.

And then there’s the pristine coastline. The miles of sugary white sand flanked by bamboo groves that were off-limits to most visitors until recently are a happy, if unintended byproduct of the war.

Among the most scenic, if difficult stretches to reach, is Nilaveli Beach in the Tamil north. While a few military checkpoints remain, vacationers can lounge on poolside hammocks under palm trees or snorkel in its crystal-clear waters. Or they can order cocktails at the Nilaveli Beach Hotel (www.tangerinehotels.com/nilavelibeach), a collection of recently renovated bungalows with private terraces.

An international airport in Hambanthota, on the island’s southern shore, is under construction, which will make the gorgeous beaches near the seaside village of Galle easier to get to.

Decimated by the tsunami in 2004, the surrounding coastline is now teeming with stylish guesthouses and boutique hotels.

Unawatuna, a crescent-shaped beach a few miles south of Galle, may be furthest along. Higher-end hotels there include Thambapanni Retreat (www.thambapanni.biz), which features four-poster beds, yoga and an ayurvedic spa. The Sun House (www.thesunhouse.com), in Galle, looks like a place where the Queen of England might stay, with its mango courtyard and colonial décor. One stylish place tucked within Galle’s city walls is the Galle Fort Hotel (www.galleforthotel.com), a refurbished gem merchant’s house run by a couple of Aussies.

I'm sharing nice compilation of videos created by Sri Lankan Tourism Board.

The Small Miracle

The theme that used by Sri Lankan Tourism Board to promote Sri Lanka's tourism is 'Small Miracle' and they came up with a nice set of videos that covers almost every bit of taste of this paradise island.

Part 1

Part 2

Part 3

Part 4

Iridium flare .... WSO2 ESB 3.0.0 Released!

2010-05-02T08:46:00.000-07:00

The WSO2 ESB team is pleased to announce the release Open Source Enterprise Service Bus (ESB) 3.0.0 - Iridium.

WSO2 ESB 3.0.0 is developed on top of the revolutionary WSO2 Carbon platform (Middleware a' la carte), an OSGi based framework that provides seamless modularity to your SOA via componentization. This release also contains many new features and a range of optional components (add-ons) that can be installed to customize the behavior of the ESB.

Further, any existing features of the ESB which are not required to your environment can be easily removed using the underlying provisioning framework of Carbon. In brief, WSO2 ESB can be fully customized and tailored to meet your exact SOA needs.

You can download this distribution from http://wso2.org/downloads/esb and give it a try.

New Features of WSO2 ESB 3.0.0

Priority based mediation through priority executors
WS-Discovery support and dynamic endpoint discovery
Message Relay for efficient pass through of messages
Component manager to install and uninstall features (provisioning support)
Common Internet File System (CIFS) support through the VFS transport
File locking functionality in the VFS transport to support concurrent polling
Smooks mediator for efficient message transformation
Enrich mediator for smart message manipulation
OAuth mediator for 2-legged OAuth support
Default endpoint UI
Hot deploy and hot update configuration elements (sequences, endpoints, proxy services etc)
Transport level statistics collection and monitoring
POX security support
Dependency detection and alerting for mediation configuration elements
Mediation statistics API and custom mediation statistics consumers
Multiple certificate/identity support in the NHTTP transport sender
Improved logging capabilities for the NHTTP transport
Templates based proxy service development in the UI
Dashboard to monitor server environment and runtime
Easy creation and management capabilities for dynamic sequences and endpoints
Pagination to service management, endpoint management and sequence management UIs
Obtaining resources like WSDL's through web proxy servers

Key Features of WSO2 ESB

Proxy services - facilitating synchronous/asynchronous transport, interface (WSDL/Schema/Policy), message format (SOAP 1.1/1.2, POX/REST, Text, Binary), QoS (WS-Addressing/WS-Security/WS-RM) and optimization switching (MTOM/SwA).
Non-blocking HTTP/S transports based on Apache HttpCore-NIO for ultrafast execution and support for thousands of connections at high concurreny with constant memory usage.
Built in Registry/Repository, facilitating dynamic updating and reloading of the configuration and associated resources (e.g. XSLTs, XSD, WSDL, Policies, JS configurations ..)
Easily extendable via custom Java classes (mediator and command)/Spring configurations, or BSF Scripting languages (Javascript, Ruby, Groovy, etc.)
Built in support for scheduling tasks using the Quartz scheduler.
Load-balancing (with or without sticky sessions)/Fail-over, and clustered Throttling and Caching support
WS-Security, WS-Reliable Messaging, Caching & Throttling configurable via (message/operation/service level) WS-Policies
Lightweight, XML and Web services centric messaging model
Support for industrial standards (Hessian binary web service protocol/ Financial Information eXchange protocol and optional Health Level-7 protocol)
Enhanced support for the VFS (File/FTP/SFTP), JMS, Mail transports with optional TCP/UDP transports and transport switching among any of the above transports
Support for message splitting & aggregation using the EIP and service callouts
Database lookup & store support with DBMediators with reusable database connection pools
WS-Eventing support with event sources and event brokering
Rule based mediation of the messages using the Drools rule engine
Transactions support via the JMS transport and Transaction mediator for database mediators
Internationalized GUI management console with user management for configuration development
Integrated monitoring support with statistics, configurable logging and tracing
JMX monitoring support and JMX management capabilities like, Graceful/Forceful shutdown/restart

Understanding Visitor Pattern

2010-04-12T11:29:00.000-07:00

In Object Oriented Design, the visitor patter is one of the obscure pattern yet powerful enough to solve many complex OO scenarios. Visitor Pattern is also not really easy to understand at the first glance but you need to dig it around with practical examples. (I guess it’s not just for Visitor pattern but also for all the other patterns)

I would like to give a clean and simple introduction to visitor pattern with this post. So let’s start with a simple example ofthe usage of Visitor pattern.

Visitor Pattern is often useful when there are fair numbers of related classes. One of the common examples of such a scenario is ‘drawing different shapes’. In this case we have a set of related classes; Circle, Trangle, Rectangle etc.

Now, if we are going to implement a draw() function for all these classes i.e. circle::draw(), trangle::draw().. etc we may be drawing different shapes but we are replicating a fair amount of code in all the classes. (Because the underlying methods, that we use to draw a ’shape’ is similar)

Visitor pattern is quite capable of solving this kind of scenarios. Here what we are doing is that we port all the draw methods of each shaped inherited classes (Circle, triangle, Rectangle) to one common class called ‘DrawVisitor’ or simply ‘Visitor’.

Then in the ‘DrawVisitor’ we have a set of functions overloads based on the type of object passed to it (Circle, Triangle, Rectangle etc), which implements the ‘drawing logic’. And its obvious that since we use a single visitor class to do the drawing of shapes we can share whatever the resources that we want to draw and can reuse drawing code.

There are a lot of concerns about using Visitor pattern, because applying visitor pattern to a given scenario often makes things ambiguous. However the applicability of Visitor pattern is justified by James Cooper (author of a Java companion to the GoF) by giving us a real world scenario which is essentially solvable only from visitor pattern. His primary example :

Suppose you have a hierarchy of Employee-Senior Manager-Vendor etc. They all enjoy a normal vacation day accrual policy, but, Senior Manger also participates in a “bonus” vacation day program. As a result, the interface of class SeniorManager(as well as ‘Vendor’ and ‘SecurityOfficer’) is different than that of class Employee. We cannot polymorphically traverse a Composite-like organization and compute a total of the organization’s remaining vacation days. This is how we use the visitor pattern to solve this problem.

Create a ‘VacationVisitor’ which handles all the vacation manipulations and implementing different vacation manipulations in polymorphic visitor methods.
Each visited instance (Employee, SeniorManager etc) is implementing an ‘accept’ method which in turns call the polymorphic visitor method.
This method call procedure is termed as ‘Double Dispatch’

The complete class design as follows.

Image transferring on Ubuntu 9.10 - gPhoto2 as Nikon Transfer

2010-04-12T10:36:00.000-07:00

When dealing with multimedia devices Ubuntu becomes a nightmare as most of the devices does not offer you the Ubuntu compatibility. The story was the same for my Nikon Digital Camera transfer tools. I badly wanted to transfer images from my camera to my computer and found out that Ubuntu doesn't allow me to transfer the images.

Finally gPhoto2 came to save me and it is a nice tool that allows you to transfer all your images from your cam to pc. This is what I did.

Install gphoto2. (simply use the package manager)
Unmount the device(camera) if mounted automatically.
Open a terminal and go to the location that you want to store the images.
Then do : gphoto2 --get-all-files
For more information do : gphoto2 --usage

That's all. By the way.. I know this is not a very user friendly way but after all as an Ubuntu user.. this is not an alien thing to use command line to transfer images :).

iTunes on Ubuntu - Songbird 1.4.3

2010-04-04T06:55:00.000-07:00

Missing iTunes on Ubuntu ?

I have completely moved from Windows to Ubuntu 9.10 during last 3-4 months, and now I have alternatives for almost all the application I used to run on Windows. However, I'm missing iTunes so badly and couldn't find an attractive media player for Ubuntu till I came across Songbird.

A few days back I downloaded their latest release( Songbird 1.4.3, Build 1438) and really feel happy about the user interface and the features. I feel like I'm using iTunes on Ubuntu.(apparently the Songbird plugin for iPod support seems broken so I have to user gtkpod for manage iPod on Ubuntu )

For those who are new to Songbird, it is a free and open source software audio player (and web browser too) which was founded by Rob Lord. Songbird utilizes the cross-platform frameworks, Mozilla's XULRunner and GStreamer media framework, and capable of running on Microsoft Windows, Mac OS X, and various Linux flavors(Ubuntu, Fedora).

When we compare Songbird 1.4.3 with many other similar media players, there are so many cool features that any music lover would expect from a matured music player, plus its so hard to find such a full featured player for Linux where user community is far less as compared to Windows. Here are some cool stuff that makes Songbird, 'the' player for Ubuntu.

Is this iTunes ? : almost there!

Don't feel bad.. This is not an iTunes screenshot. ;)

Album-art, Artist info, Lyrics on one player..

It not just a matter of listening to a song.. but also you can enjoy the lyrics, album art and artist info/photos/news on one player. This is a true service based design. It fetches various information for the current track from various service providers like Lastfm etc.

Listen and enjoy lyrics at the same time..

Inline Artist info/photos

Listen to Youtube music from the same player.. (of course this is an alternative to your web browser :))

Bad News

The 'Bad news' comes last... :(
Few days back I hear a heartbreaking news about Songbird. This is the news

On April 2, 2010, it was announced that POTI Inc. stops developing program based on multiplatform orientation and will focus only on its MS Windows and Mac OS X versions of Songbird, providing only unofficial support for Linux releases. Company stopped adding by itself new futures and maintaining Linux version and development of the Linux program rely only on community effort.

However, Songbird still remains the ultimate player for Ubuntu... and feel like this would do well on its target platforms (win/mac os').

Real-time XML Syntax Highlighter in Javascript - EditArea

2010-03-27T07:05:00.000-07:00

If you are looking for a Real-time XML Syntax Highlighter and in the same time you want it as a web based lightweight tool then EditArea is the perfect solution for that.

In WSO2 ESB 3.0.0(up coming release) , I've integrated all the source(xml) fields with EditArea highlighter to give the use a real-time rich xml editing experience. However, the original highlighter comes with some dull colors for XML Syntax as well as for controllers. So, I modified the EditArea source and came up something like this. (It takes few seconds to load..;))

EditArea support syntax highlighting not only in XML but many other programming languages including Java,PHP, C/C++, CSS, Perl, Ruby, Python and many more.

I have hosted a modified version of EditArea which supports Real-time XML Syntax Highlighting here.

So, EditArea is one of the rear opensource tool that is specifically designed to support Real-time syntax highlighting for most of the popular programming languages and yet it runs with its extreme lightweight nature.

You can simply add my version of EditArea highlighter by simply adding the following code to ur blog post or html. Say that the text area that you want to add is id="textarea_1" then you can add this code snippet to you are html.

- First you want to point to the EditArea javascript source in your html/blog template. JS- src="http://dl.dropbox.com/u/5494230/editarea/edit_area_full.js"

- Then you need to initiate and load the EditArea Syntax Highlighter by giving the ID of your textarea.

Preserving CDATA sections - AXIOM StAX parsing

2010-03-14T05:44:00.000-07:00

In XML CDATA sections provide a way to tell the parser that there is no markup in the characters contained by the CDATA section. And that makes it much easier to create documents containing sections where markup characters might appear, but where no markup is intended. CDATA is commonly used for scripting language content and sample XML/HTML content.

So in context of Web Services with Apache Axis, the default settings of the AXIOM StAX parser would ignore the CDATA sections and won't preserve them after the parsing is done. In coalescing mode, the parser will not only coalescing adjacent text nodes, but also CDATA sections.
However there are cases that you need to preserve CDATA section after parsing and creating OMElement. So, in such a scenario you need to switch off the coalescing mode by setting a property in XMLInputFactory. (i.e.: "javax.xml.stream.isCoalescing" = false). The following code will create an AXIOM OMElement with preserving CDATA.

For detailed overview of Axiom please refer Andreas' Axiom User Guide.

Modularity in Software Engineering

2010-02-06T00:21:00.000-08:00

What is Modularity

'Modularity' - a frequently used term in Software Engineering. Its often tends to feel that our Software System is modular owing to that of the physical and logical structure of the system.

We often distinguish interfaces and their implementations, abstracting the functionality behind the interfaces, grouping classes and interfaces under packages based on their functionality etc. Those techniques are really cool and they are the real drivers of modular software, but unfortunately modularity could not be achieved just by applying such techniques. Modularity is software engineering is based on few fundamental concepts.

Self-Contained : "Agile & Autonomous"

A module is a self-contained component of a larger software system. This doesn't mean that it is an atomic component. In fact a module consists a several smaller pieces which are collectively contributed to the functionality/performance of the module.

We cannot remove or modify at least any of these tiny (compared to larger software system) components and if we do so, the 'Module' will cease it expected functionality. A module can be installed, un-installed or moved as a whole(single unit) and it wont affect the functionality of the other modules.

Fig I - Modular Compartments

Lets try to understand this by above real world scenarios. In figure 1, we have a set of compartments that can mimic the modularity. Any of these compartments can be move without affecting other components' functionality (but when we move a module the functionality of the whole system changes).

Fig II - Modularizing ..

And also in the second figure we can form different letters by place different components(modules) at different places and we have the freedom of moving modules freely without affecting the functionality of other modules

Highly Cohesive : "To do one thing and do it well"

Cohesive is strong related to the 'responsibility' in real life. A responsibility is a kind of action that we identify that a given entity is subjected to act on.

'High Cohesiveness' means that a component (module) is strongly related or focussed to carryout a specific task and also not doing any unrelated tasks. Therefore, cohesive modules are fine-grained, robust, reusable and less in complexity.

In our compartment examples, you can see each compartment(module) contains a predefined set of sub components and they are responsible to carryout a well defined task and doing it with absolute efficiency.

Loose Coupling : "Hassle free interaction"

A given module's internal implementation is not dependent on the other module that it interacts with. Modules are interacting with a well defined clean interface and any of module can change its internal implementation without affecting other modules.

Its vital to define the interfaces between the modules with extreme care. In the ideal case an interface should be define based on the what a given module offers to other modules and what it requires from other modules.

So back to our real world scenario, in the compartment example, we can clearly see that the interfaces are well defined and any of the internal modification inside a compartment would not affect other modules.

Power of Modularity

Here is a concise explanation of Modularization benefits.

A well-defined segmentation of the project effort ensures system modularity. Each task forms a separate, distinct program module. At implementation time each module and its inputs and outputs are well-defined; there is no confusion in the intended interface with other system modules. At checkout time the integrity of the module is tested independently; there are few scheduling problems in synchronizing the completion of several tasks before checkout can begin. Finally, the system is maintained in modular fashion; system errors and deficiencies can be traced to specific system modules, thus limiting the scope of detailed error searching.

- Richard Gauthier and Stephen Ponto

Restoring Grub2 on Ubuntu 9.10

2010-01-19T08:30:00.001-08:00

It's often required to have a dual boot on any machine owing to the compatibility of the application software. I recently came across a real messy scenario where had to install Windows several times on my Ubuntu/XP dual-boot system.

This is a bit risky case for two reasons. Ubuntu 9.10 comes with Grub2 and conventional Grub restoring methods would not be applicable and also Windows 2008-sp1 is know to be a very restricted OS with huge amount of security features.

However, I managed to restore everything after installing Win 2008. Here are the steps that I followed.

Following procedure will be valid on Ubuntu 9.10/Win XP, Vista, 2008, 7

You need the Ubuntu 9.10 live CD
Boot from live CD
Go to 'System->Administration -> Disk Utility'
Find out the partition that you install Ubuntu (root). eg. 'sda4'
Then open a terminal and mount the partition that you found above.
'sudo mount /dev/sdXY /mnt' where X - Device Descriptor and Y-Partition Id
'sudo grub-install --root-directory=/mnt/ /dev/sdX'
Restart and boot Ubuntu (previous installation, you won't see the newly installed windows OS on grub screen)
Update Grub - 'sudo update-grub' to get both OS' on dual boot

This works fine for me on 64-bit Ubuntu/Windows system.

B-Trees and Inverted Index – De facto standard for file organization (II)

2010-01-11T07:23:00.000-08:00

In the context of modern IR Systems, using a dictionary or hash table to represent inversion list would be kind of tedious as it request huge amount of memory. There for its obvious that we need to get the help of secondary storage to store the content and retrieve when required. Then again there is a huge overhead of using secondary storage to store and read a dictionary. So the solution that we came across was B-Tree, which is a used almost each and every implementation of inverted index.

Definition of a B-Tree of order ‘m’

A root node with 2 ~ 2m keys
All the other internal nodes have between m ~ 2m keys
All keys are kept in ascending order
All levels have the same level of differ (at most 1)

e.g:

Inversion lists structures are used because they provide optimum performance in searching large databases. The optimality comes from the minimization of data flow in resolving a query. Only data directly related to the query are retrieved from secondary storage. The beauty of B-Trees lies in the methods for insertion and deletion of records which always leaves the tree balanced.

In the most command secondary storage ; the hard disk, the number of disk accesses is measured in terms of the number of pages of information that need to be read from or written to the disk. And the disk access time is not constant-it depends on the distance between the current track and the desired track and also on the initial rotational state of the disk. We shall nonetheless use the number of pages read or written as a first-order approximation of the total time spent accessing the disk.

In a typical IR System B-tree, the amount of data handled is so large that all the data do not fit into main memory at once. The B-tree algorithms copy selected pages from disk into main memory as needed and write back onto disk the pages that have changed. B-tree algorithms are designed so that only a constant number of pages are in main memory at any time; thus, the size of main memory does not limit the size of B-trees that can be handled.

It can be prove that the cost of processing a ‘find’ operation in a B-Tree grows as the logarithm of the file size. For example a B-Tree of order 50 which can index a file of 1 million records can be searched with in 4 disk accesses (worst case). Also for insertion and deletion, a B-Tree of order d, for a file of n records, insertion/deletion time is proportional to log(d) n.

However, for B-Trees there are some practical limitations where as the amount of data that can be transferred to with one secondary storage access is limited as well as the track size of hardware should be taken in to the account. So, in practice, optimum node size is depends critically on the characteristics of hardware.

The Romantic Opera – Kasun Kalhara

2010-01-06T09:02:00.000-08:00

Nowadays Sri Lankan music stream is flooded with a lot RnB and Hiphop music and many people merely want to listen to such styles of music. And all the radio stations are backing such music and ruining the natural music taste of Sri Lankans. All those commercial oriented artists tend to release their 5-6 tracks per a month. This is something simply related to the quantity and they simply want to create a music track using computers and other equipments.

Once they create the track there are people who can write some strange words to tally with the track and the beat. Finally they’ll come up with a music video and all the TV stations eagerly waiting to play their music video.This is the story of Iraj, BnS, .. (you name it)

But .. They have forgotten the fact that there are some (I think its few ) people in Sri Lanka who still loves the instinctive slow music (mood songs) which is a blend of soothing voices and acoustic music.

Kasun Kalhara is one of the exceptions thrown from the Sri Lankan modern music stream. As I see he is the best production by Dr. Premasiri Kemadasaa (Kemadasa master) and doubtless selection of the best voice in the modern generation. Not only his voice but he is a composer too.. But not a synthetic composer who solitary depends on Computer and keyboard.

‘The Romantic Opera’ is his latest album and in my opinion it is the best music release of him. The titled rack, ‘The Romantic Opera’ is inspired from Opera style music.

Opera is an art form in which singers and musicians perform a dramatic work (called an opera) which combines a text (called a libretto) and a musical score -wikipedia

And this album is a combination of a set of distinct music tracks. It has some ‘Latin’ music, some of ‘Chinese or Japanese’ music, some of ‘Indian’ music and a lot of ‘Western’ and ‘Sri Lankan’ music. Most of the songs are composed using acoustic music and very less number of music instruments were used.

You’ll not able to listen to this sort of music when you switch on the radio or TV.. You’ll never hear any of these songs when you are travelling on a bus… Soo… you need to find/buy it and listen to it… because that’s the way that most true Sri Lankan music lovers used to do…

‘The Romantic Opera’ is a must listen music album of true Sri Lankan music fan.. Believe me .. It awesome.

We always call that the ‘music’ is an universal thing… yes it is.. That’s why we love A.R Rahman’s … that’s why we love Josh Groban.. and that why we should love Kasun Kalhara.

I've ported this blog post from my previous blog. (Comments are also included)

10 Responses to “The Romantic Opera – Kasun Kalhara”

Akila Mike Subasinghe said

March 12, 2009 at 2:05 am

I am simply Blown away by this Album. He has truly got Lot from Late Premasiri Kemadasa…and I just love his Variety of voice and Powerful music.

Its a Pity that the SL radio channel’s don’t play these songs often. But given the education qualifications and Language use of presenters in Radio Channels these days, I would say that we can’t expect anything better from them.

I would say, “PITTY ON THEM”…This is an Extraordinary collection of Music, and if you haven’t heard this, then you have heard nothing at all….Its that Good!

Reply

Kasun04 said

March 12, 2009 at 3:10 am

certainly… Its not just the imbecility of the presenters, dj etc… but the society. They are playing a lot of RnB and 6/8 junk because thats the taste of most of the people.

As politicians reflects the society.. the musicians also reflects society too

Reply

Thisara De Silva said

March 14, 2009 at 5:11 pm

Haritha Nimnaya, Radical Premaya, Romantic Opera……..

Somebody can think these albums as typical song albums in Sri Lanka. But i can say that those are not only albums. It is some kind of interpretation of several experiments. In other words those are experimental creations.

We cannot see much of this kind, Because everyone can’t do this. Actually Amaradewa, Victor Rathnayaka, Nanda malini, and some other senior musicians have done it. But it is so hard to find a such musician in present generation.

Kasun Kalhara is the king of this generation. It is obvious. He is so talented and so intelligent. I’m not gonna talk about new generation much, but not all.

When i speak about the ‘Romantic Opera’ it is the best album of Kasun’s so far. Thats for sure. It includes different kinds of mood songs and it is not a easy target for any musician.

Thats’s why Kasun is special.

Romantic Opera & Erambumamma Kandulak wela is my favourite songs of this album.

Reply

Akila Mike Subasinghe said

March 15, 2009 at 3:07 pm

Couldn’t agree with you more Tharindu…They all were Experiments and this Guy is Better by the Second…Romantic Opera is 10 Times or even more better than Radical Premaya…as All Songs in this are different..for different Moods and Styles. I showed his music to a Canadian Student and he was just amazed by kasun’s Music…He said that he has a Lot of Respect for his Music (though he understands no Sinhalese)…Kasun’s Music is UNIVERSAL.

Almost all the other Junkies these days (who call themselves as Artists, who are probably deported from Australia), Try to fill in words into the same beat with odd sounds…

My Favorite Songs are :

Sanda Ona

Ra Ahasin

Adare Sithum

Arambumama Kandulak

Romantic Opera ….. (see…I almost wrote the whole album..:D)

Reply

Kasun04 said

March 16, 2009 at 4:32 am

Its glad to hear that even Canadians love Kasun’s music .. As I heard Kasun Kalhara’s music style is pretty much similar to the music composed and sung during the Romantic Period in western music in 18th century. The musicians were very much concentrated on expressing their feelings and moods without any limitations or boundaries. In Kasun’s case, The Romantic Opera songs is influenced from such style of music.. according to Chaaminda Rathnasuriya (who wrote this song) this is an effort to introduce “romanticism” to Sri Lankan music. (I hope to explain about ‘Romanticism in Music’ from a separate post )

However.. I felt that music of some of the songs in the album were not up to their potential quality.. I’ve listened to the acoustic versions of ‘Arabumama Kadulak’ and ‘Rae Ahasin’… they are far better than the versions in the album.

It’s a pleasure that this sort of discussion taken place on this blog.

Reply

Thisara De Silva said

April 7, 2009 at 7:06 pm

Actually i’m pretty much interested in western music, therefore i’ve learnt a litlle. As far as i concerned about the periods of musics i thing ROMANTIC period is much sensetive than others. I have seen that particular characteristic in ‘Romantic Opera’ too. That’s why i call Kasun as a genius. He should be the pioneer of new generation.

Does anybody know the real story of song ‘Arambumama landulak wela’???? Seems it has written to his mother. But there is a little confusion in beginning of the second verse.

Let me tell something about the acoustic versions of the album versions. I think the acoustic version of ‘Arambumama Kadulak’ contains a different kind of taste. Nalaka anjana kumara, the composer of this song is a top class guitarist in Sri Lanka. so we have nothing to talk about the acoutic version since we know the guitarist is Nalaka !!

I love ‘Romantic Opera’ !!! – The best mood song collection ever !!!

Reply

Eranga Malshan said

April 8, 2009 at 6:47 am

“Romantic Opera” is the best album i’ve ever heard, Best music as well as the vocals. so impress from that. I like voice balancing and it has nice variation. this music can touch my soul.listn this songs in a calm n quite place, better if you have a “joint”.then you can hear lot of sensitive things that in these songs. I’m crazy for that.please my dear Kasun keep doing this. we are waiting for these kind of things…. gud luck.

Reply

Kasun04 said

April 17, 2009 at 7:01 pm

This album contains some top quality mood songs and they are backed by not just the vocal and music, but also the lyrics. Songs like ‘Rae Ahasin’, ‘Romantic Opera’, ‘Arabumama Kadulak Velaa’ contains some exceptionally well written lyrics. I really don’t know about the story behind the ‘Arabumama Kadulak’.. anyway its a bit old song of Kasun.

By releasing this album, Kasun Kalhara has introduced some young endowments of musicians and lyrists like Romaine Mendis, Chaaminda Rathanasooriya, Sanjaya Epa Senevirathna, Eshantha Peris, Achala Solamans, Isuru Kumarasinghe, Nalaka Anjana Kumara, Akila Surein and also Nadeeka.

Reply

Rapture_D said

July 23, 2009 at 5:37 am

Haritha Nimnaya….., Radical Premaya……., The Romantic Opera………… kasun’s third album contains different and the bst stuff…

Reply

Thilina1024 said

October 24, 2009 at 11:35 pm

I’m very glad to reply here. Nice to meet such fellows like u guyzzz . I came across this blog by a link. So, read the posts & thought to reply.

There is nothing to say about KK – Kasun Kalhara. Talent is not enough. The honesty & wise’ness are required to be a great one. If someone review his journey in SL music industry, he/she can see how he got it. There are no doubts, simply he is the person who leads the music in our time, like Pandith Amaradewa, Victor Rathnayaka & Nanda Malini & so on. In all three albums of him, he has done a lot of experiments. & if we talk about “The Romantic Opera”, yeah it shows up how he has improved his talent. The track called by the album’s title, I wonder is there anyone who could even think of singing that song. In most places of the net, I have seen that people are talkin about “Warna” track. If someone has read “Agganna Sutra” of Buddhism, then he/she might not have doubts about what that song is talking. The lyrics of Warna track are based on the begining of life on Earth as it was discoursed by Lord Buddha in “Agganna Sutra”. If you are curious, then take a look in to that sutra & know what it is about. Then, you guyz will be able to taste this song better. ‘Coz, I have read a Sinhalese translation of that. So, that’s how I got it. I prefer a Sinhalese translation. ‘Coz, you will get a better understanding.

Google – Business, Ethics and Life

2010-01-06T08:49:00.000-08:00

‘Google’ – One of the innovative and revolutionary concept which was blossomed in early ‘96, as a research project of Larry Page and Sergey Brin (Google’s President incharge of Technology). Google only consume a spam of 13 years to climb to the peek of internet search technology and now, its in a state which is almost untouchable and unreachable to the competitors.

I recently watch a nice interview with Sergey Brin and its really awesome as it reveals some of interesting facts about Google, it’s challenges and Sergey him self.

Google manufactures Nothing!

Yeah… it manufactures nothing but produce an IDEA. The idea or the mission is,

‘Accumulate all the world’s information and make it accessible and useful to everyone’. This sounds like an non-commercial or non-profitable but we should keep in mind that there is a underlying business process which may based on several revenue sources. In general, “Search Engines” use one or more of the following revenue sources.

Charge advertisers for presenting online “banner” ads to users
Collect marketing data on consumer habits, then selling the data or using it for targeted advertising.
Charging websites to become listed
Charging websites for better placement in lists
Charging websites to purchase keywords for themselves
Charging users for searches
Charging other search engines to use their catalog

Google’s mission is to critical as having the correct information that you need and accessible anytime that you want is very important to each and every aspect of the human life.

Is Google a gigantic “Jewish” company?

Both founders of Google, Sergey and Larry are Jewish. Its first employee was Jewish and many of the seniors were Jewish too. So definitely, Google has some Jewish characteristic in its business process. In this case, Google’s main opponent, Microsoft was also founded by a Jewish person.

Unbiased Search Results!

Google doesn’t to any moderation or alteration to the search results (’search hits’), which means Google provides results based on the query that the user executed. For example, I google for “Internet Browser” and the result set contains the ranked hits in this order. – Opera, Firefox, Safari, IE7 and finally Chrome. This shows the integrity of search engine. Google supposed to provides users with the information they required and the presentation of the result set is not modified by the Search Engine as it uses an unbiased scheme in ‘Query and Result Processing’.

Google- Privacy and Identity

When it comes to internet search, Google track all the activities that you did using Google. (may be based on your IP or if you use gmail while searchine. And also even if you use dynamic IP, it gives some identity of your country, region etc.). Google knows what your interest and may be there the things that you don’t want to reveal to anybody. So, its clear that there is some conflict here. And the most important thing here is that, the commercial value (or may military) of the information kept with Google. It can be used as a global business survey where you can identify the potential clients of a given product and the geological distribution of the community.

In this particular interview, Sergey Brin was questioned about this issue but his response was that there were no any cases of exposing of users search history. Yet he didn’t reject that fact that the amount of user information (and to which extent) is kept with Google.

Google Earth ’s impact on military and security activities

When we use Google Earth for the first time, it’s not a surprise that the impact of Google Earth on military activities comes to our mind.

But according to Sergey, most of the high ranking security agencies are claimed that there is no impact of Google Earth on military and security activities.

Google – No 1 Place to work and 20% of working time is yours!

Google is considered to be the best place to work with numerous facilities and good exposure to almost every technology. Apart from that, the employees are supposed to spend 20% of their working time on some personal work(technical stuff). The products like “Orkut”(social network) and “Google News” are the outcomes of this nice policy.

I guess this is really a cool thing. Every company should think about giving there engineers a free time where they can work on their own projects etc. (Of course this allocated 20% time is not supposed be wasted on facebook or similar stuff )

Finally… Is Sergy Brin a happy person

He claimed that he is a happy person and also comment on the fact that most of the business professionals at his level ‘Are NOT’.

You can watch the original interview here.

B-Trees and Inverted Index – De facto standard for file organization (I)

2010-01-06T08:15:00.000-08:00

‘Secondary Storage’ is still a nightmare in achieving higher performance in modern computer systems and often the power of multi-core CPU is more or less negate due secondary storage low performance IO. Of course the performance of secondary storage IO has improved in the recent past but it is still inferior relative to CPU performances.

A computer must retrieve an item and place it in main memory before it can be processed. In order to overcome the low performances of the system one must organize the files intelligently and making the retrieval efficient. The file organization depends on the retrieval method; sequential or random. Particularly secondary storage IO is a huge overhead in the context of random access method. Therefore associated with a large, randomly accessed file in a computer system is an index. An index is often a file that stored in the disk and it contains a mapping between the terms and content.

Inverted Index

An index if often used in Information Retrieval systems, Database systems, file indexer for user and general purpose access methods. The important thing here is that how we physically represent index in a computer system. That’s where B-Trees come in to play. So, before we going deep into B-Trees it’s worthy to have a close look at index and its structure.

‘Inverted Index’ is the more specific name given to the index as it’s the most common data structure used in Database Management Systems and Information Retrieval Systems (Search Engines etc.). It consists of three different basic files,

Document File
Inversion list (posting files)
Dictionary

The name “inverted file” comes from its underlying methodology of storing an inversion of the documents: inversion of the document from the perspective that, for each word, a list of documents in which the word is found in is stored (the inversion list for that word). Each document in the system is given a unique numerical identifier (Document ID). It is that identifier that is stored in the inversion list. The way to locate the inversion list for a particular word is via the Dictionary. The Dictionary is typically a sorted list of all unique words (processing tokens/terms) in the system and a pointer to the location of its inversion list. Dictionaries can also store other information used in query optimization such as the length of inversion lists.

The above inverted index is the simplest representation of the inverted index but for modern IR systems there can be more attributes in the inversion list itself. For example, to support proximity, contiguous word phrases and term weighting algorithms the inversion list contains the offset of each word in the document. So for Document ID = 1, the term bit is the 4^th, 8^th and 30^th term in the document, then the inversion list would look as follows.

bit -1(4), 1(8), 1(30)

So once we create the inverted index, the search process or system is the actual component that uses the created inverted index. So, when a search is performed for a given query, the inversion lists for the terms in the query are located and the appropriate logic is applied on the inversion list sets. So the outcome is the set of hits for a given query (inversion list may ordered based on ranks), and the hits will be given in the form of document IDs. These document ID will be used to retrieve the accrual documents.

So where are the B-trees ?.. Well that comes in to play when we implement the above conceptual models. Although we stored a set of normalized terms in the form of a ‘dictionary’, it actually uses a B-Tree to hold the normalized set of terms and make Secondary Storage IO more efficient. We should keep in mind that, for an enormous set of document we must get an extremely huge set of terms and the inverted index would be equally huge. If we are going to store these dictionaries in the memory, it would be very costly if not impossible. So we tend to store in the secondary storage and of course we have the performance nightmare which is always bounded to secondary storage. So we’ll see how B-Trees solve our problem in this context from part II of this post.

A new journey.. A new working place.. WSO2 Inc.

2010-01-03T08:49:00.000-08:00

I guess switching blogs is less riskier than switching jobs. Anyway I had to take that risk of switching jobs in last month (12/2009) as I've joined WSO2 as a Senior Software Engineer. My very first job lasted only one and half years owing to various issues I had with the type of work I got during my first employment. However it was a good experience on some rare technologies like Search Engines and Indexing.. and not forgetting C++.

Now.. the context has changed.. technologies has changed.. environment has changed... So I believe it's a whole new experience to me to work for WSO2, which is a renowned Open Source SOA Company.

These are my first few steps in the world of SOA and Web Services...

Blogger..

2009-12-26T10:22:00.000-08:00

I started blogging on WordPress in June, 2008 and had a nice time with WordPress blogging. However, the constrained environment of WordPress motivated me to switch to Blogger. Anyway, I'm not going to do a comparison between WordPress and Blogger as there are so many (I'm feeling Lucky link :D).

I thought of porting some of my posts on my old blog to this one.. may be not all of them but a few top posts. (There are some automated tools but seems like there are some issues during conversion.)

So thanks for visiting my blog!

Keep on Visiting...¡Salud!!!

-Kasun