Geeks With Blogs

Charles Young
Richard Seroter talks here about feeding BizTalk events into StreamInsight or receiving events out as BizTalk messages.   I spent a little time considering the possibility of using StreamInsight in conjunction with BTS messaging infrastructure by building custom BTS adapters or pipeline components that directly hook into StreamInsight.   A preliminary look through the code samples (the best way, currently, of understanding this new engine, given that the CTP documentation is somewhat sparse) suggests to me that this is probably not the way to go.   As I understand things, there seems to be an impedance mismatch between BTS messaging and the concept of an event stream.   The closest thing BTS offers to an ‘event stream’ is its concept of a batch, used by adapters to submit multiple messages to BTS via the transport proxy as a single transactional interaction.   I say ‘closest’, but in reality a BTS batch is quite different to an event stream, and consequently a BTS adapter is not at all similar to a StreamInsight adapter.   Event streams may be continuous over extended periods of time, and don’t naturally fit a transactional model.
Of course, in constrained circumstances, it may be possible for the two models to co-exist. The sample TextFile input adapter, for example, reads a single CSV file and converts its contents into a stream of events.   A second CSV files would be processed as a separate event stream.   In this case, StreamInsight could co-exist nicely with a BTS pipeline that processes each CSV file as a separate message.   However, in other scenarios, you would have to ensure that a StreamInsight adapter continues to run over an open-ended number of BTS batch submissions.   Worse still, how would you cope with multiple instances of the same Receive Ports running in different host instances?   Somehow you would have to share a single event stream between those multiple BTS host instances.
So, I think that as far as BTS is concerned, the obvious places you might use StreamInsight are:
a)  outside of BTS, feeding events to a BTS Receive Port
b)  from within a BTS orchestration, probably exploiting the convoying and long-running
     transaction facilities of the orchestration to handle submission of event messages to
     a single event stream. 
c)  outside of BTS, receiving events emitted by a BTS Send Port. 
A special case of a) would be using StreamInsight in conjunction with RFID Server.   Remember that RFID Server can be used for more than just RFID devices.   c) would, I think, prove to be a rare scenario. b) and c) would both significantly compromise the raw performance of StreamInsight by inserting the relatively high latency BTS message box in front of a StreamInsight input adapter.
Having spent little time yet with the CTP, I reserve the right to make mistakes!   Maybe I have not properly grasped the StreamInsight model, but this is how it looks to me at present.
Posted on Monday, August 24, 2009 12:15 AM | Back to top

Comments on this post: Microsoft StreamInsight and BizTalk Server - Preliminary Thoughts

# re: Microsoft StreamInsight and BizTalk Server - Preliminary Thoughts
Requesting Gravatar...
So … how do I find my way through the massive amounts of incoming data, events, and messages from a myriad of diverse applications - intercepting what’s relevant, making sense of seemingly random events, and channeling the essential artifacts to the appropriate BizTalk orchestrations.

I have in front of me 3 architectural drawings of products that are meant to systematically find and deliver, in real-time, the hidden order embedded in the chaos of a very complex input stream occurring in the context of multiple queued transactions. They are NEsper (open source), ESB Toolkit (BizTalk), and StreamInsight (SQL Server).

In our multi-application system generating events from everywhere, going everywhere, we are currently performing, in the BizTalk engine, all the complex decision-making of sorting through messages and lining them up with the correct workflows. The penalty for doing that is a complexity curve rocketing off the chart.

NEsper, an event stream processor ported from Java, advertises itself as “expresses rich event conditions, enabling your system to react to complex situations” seemed to be just the ticket. It can be used to pipe primitive and complex events directly into BizTalk, ala RFID, after applying pattern-matching rules to domain handler objects for grouping, sorting, filtering. Where the disconnect for me is, our events are really delivered more as a cloud with a small finite number of events in any given transaction. not a stream of millions. NEsper seems more geared towards managing and correlating high volumes of events in realtime, but maybe it could be used as the event stream handler from an event cloud conversion?

StreamInsight seems data-centric, rather than event-driven, yet the core CEDR (complex event detection and resolution) might be what we need for receiving events and correlating them. While its tight coupling with SQL Server seems, to me, to limit its usefulness for total event management, might it act as a receptor for the event cloud, converting the cloud into multiple streams?

The ESB Toolkit provides the bus BizTalk has been lacking - now there’s some fun stuff. Referring to another article by Charles, it was posited that event technology “ leads naturally to the notion of event processing networks, which link multiple event processing agents … overlaps to a significant degree the general concept of ESBs”. Charles used ‘Dublin’ as the logical choice for routing events across a distributed environment, but I wonder … if you already are using BizTalk, why not the Toolkit also for this purpose.

Do I need to synthesize these products to work together, using the parts from each that I need? Or do I mix two of them in tandem (but which two)? Or are they mutually exclusive and therefore have to pick just one?
Left by Gerry Hummell on May 19, 2010 3:41 PM

Your comment:
 (will show your gravatar)

Copyright © Charles Young | Powered by: