In this tutorial we will learn to process a raw market data feed. We will learn to report throughput statistics and to detect when the data rate of a feed falls off unexpectedly. A rate fall-off may mean that the data is stale and we want to alert when there is a possible problem with the feed.
Our input stream consists of 1 event stream that contains 2 simulated market data feeds. Each individual event in the stream indicates the feed that supplies the market data, the security symbol and some pricing information:
String symbol; FeedEnum feed; double bidPrice; double askPrice;
For the throughput statistics and to detect rapid fall-off we calculate a ticks per second rate for each market data feed.
We can use an EQL statement that specifies a view onto the market data event stream that batches together 1 second of events. We specify the feed and a count of events per feed as output values. To make this data available for further processing, we insert output events into the TicksPerSecond event stream:
insert into TicksPerSecond select feed, count(*) as cnt from MarketDataEvent.win:time_batch(1) group by feed
We define a rapid fall-off by alerting when the number of ticks per second for any second falls below 75% of the average number of ticks per second over the last 10 seconds.
We can compute the average number of ticks per second over the last 10 seconds simply by using the TicksPerSecond events computed by the prior statement and averaging the last 10 seconds. Next, we compare the current rate with the moving average and filter out any rates that fall below 75% of the average:
select feed, avg(cnt) as avgCnt, cnt as feedCnt from TicksPerSecond.win:time(10) group by feed having cnt < avg(cnt) * 0.75
This tutorial showed how to compute throughput and detect a rapid fall-off on a single event stream. We calculated the ticks per second and a rolling 10-second average. We alerted when a rates fell below an arbitrary 75% threshold.