The tutorial and case studies are available on the public web site at http://esper.codehaus.org/evaluating/evaluating.html.
This chapter outlines the examples that come with Esper in the eg/src folder of the distribution. The code for examples can be found in the net.esper.example packages.
In order to compile and run the samples please follow the below instructions:
Make sure Java 1.5 or greater is installed and the JAVA_HOME environment variable is set.
Open a console window and change directory to esper/eg/etc.
Run "setenv.bat" (Windows) or "setenv.sh" (Unix) to verify your environment settings.
Run "compile.bat" (Windows) or "compile.sh" (Unix) to compile the examples.
Now you are ready to run the examples. Some examples require mandatory parameters. Further information to running each example can be found in the "eg" folder in file "readme.txt".
Modify the logger logging level in the "log4j.xml" configuration file changing DEBUG to INFO on a class or package level to reduce the volume of text output.
JUnit tests exist for the example code. The JUnit test source code for the examples can be found in the eg/test folder. To build and run the example JUnit tests, use the Maven 2 goal test. The JUnit test source code can also be helpful in understanding the example and in the use of Esper APIs.
This example processes a raw market data feed. It reports throughput statistics and detects when the data rate of a feed falls off unexpectedly. A rate fall-off may mean that the data is stale and we want to alert when there is a possible problem with the feed.
The classes for this example live in package net.esper.example.marketdatafeed. Run "run_mktdatafeed.bat" (Windows) or "run_mktdatafeed.sh" (Unix) in the eg/etc folder to start the market data feed simulator.
The input stream consists of 1 event stream that contains 2 simulated market data feeds. Each individual event in the stream indicates the feed that supplies the market data, the security symbol and some pricing information:
String symbol; FeedEnum feed; double bidPrice; double askPrice;
For the throughput statistics and to detect rapid fall-off we calculate a ticks per second rate for each market data feed.
We can use an EQL statement that specifies a view onto the market data event stream that batches together 1 second of events. We specify the feed and a count of events per feed as output values. To make this data available for further processing, we insert output events into the TicksPerSecond event stream:
insert into TicksPerSecond select feed, count(*) as cnt from MarketDataEvent.win:time_batch(1) group by feed
We define a rapid fall-off by alerting when the number of ticks per second for any second falls below 75% of the average number of ticks per second over the last 10 seconds.
We can compute the average number of ticks per second over the last 10 seconds simply by using the TicksPerSecond events computed by the prior statement and averaging the last 10 seconds. Next, we compare the current rate with the moving average and filter out any rates that fall below 75% of the average:
select feed, avg(cnt) as avgCnt, cnt as feedCnt from TicksPerSecond.win:time(10) group by feed having cnt < avg(cnt) * 0.75
The simulator generates market data events for 2 feeds, feed A and feed B. A target rate parameter defines how many events for each feed the simulator sends to the engine in each second. Depending on your system, a large target rate may result in more events to be sent to the engine then your system can handle in one second. In that case, the target rate will not be achieved. After reaching the target number of events for a given second, the simulator sleeps the thread for an approximate remainder of the current second.
The simulator generates a number of events per second following the formula target rate * 0.9 + target rate * 0.2 * random.nextDouble(). This generates a random rate for each feed in a +-10% range within the target rate.
The rate drop probability parameter specifies the probability in percent that the engine drops the rate for a randomly chosen feed to 60% of the target rate for that second. Thus rate fall-off alerts can be generated.
The classes for this example live in package net.esper.example.transaction. Run "run_txnsim.bat" (Windows) or "run_txnsim.sh" (Unix) to start the transaction simulator. Please see the readme file in the same folder for build instructions and command line parameters.
The use case involves tracking three components of a transaction. It‘s important that we use at least three components, since some engines have different performance or coding for only two events per transaction. Each component comes to the engine as an event with the following fields:
Transaction ID
Time stamp
In addition, we have the following extra fields:
In event A:
Customer ID
In event C:
Supplier ID (the ID of the supplier that the order was filled through)
We need to take in events A, B and C and produce a single, combined event with the following fields:
Transaction ID
Customer ID
Time stamp from event A
Time stamp from event B
Time stamp from event C
What we‘re doing here is matching the transaction IDs on each event, to form an aggregate event. If all these events were in a relational database, this could be done as a simple SQL join… except that with 10,000 events per second, you will need some serious database hardware to do it.
Further, we need to produce the following:
Min,Max,Average total latency from the events (difference in time between A and C) over the past 30 minutes.
Min,Max,Average latency grouped by (a) customer ID and (b) supplier ID. In other words, metrics on the the latency of the orders coming from each customer and going to each supplier.
Min,Max,Average latency between events A/B (time stamp of B minus A) and B/C (time stamp of C minus B).
We need to detect a transaction that did not make it through all three events. In other words, a transaction with events A or B, but not C. Note that, in this case, what we care about is event C. The lack of events A or B could indicate a failure in the event transport and should be ignored. Although the lack of an event C could also be a transport failure, it merits looking into.
To make testing easier, standard and to demonstrate how the example works, the example is including an event generator. The generator generates events for a given number of transactions, using the following rules:
One in 5,000 transactions will skip event A
One in 1,000 transactions will skip event B
One in 10,000 transactions will skip event C.
Transaction identifiers are randomly generated
Customer and supplier identifiers are randomly chosen from two lists
The time stamp on each event is based on the system time. Between events A and B as well as B and C, between 0 and 999 is added to the time. So, we have an expected time difference of around 500 milliseconds between each event
Events are randomly shuffled as described below
To make things harder, we don‘t want transaction events coming in order. This code ensures that they come completely out of order. To do this, we fill in a bucket with events and, when the bucket is full, we shuffle it. The buckets are sized so that some transactions‘ events will be split between buckets. So, you have a fairly randomized flow of events, representing the worst case from a big, distributed infrastructure.
The generator lets you change the size of the bucket (small, medium, large, larger, largerer). The larger the bucket size, the more events potentially come in between two events in a given transaction and so, the more the performance characteristics like buffers, hashes/indexes and other structures are put to the test as the bucket size increases.
The StockTicker example comes from the stock trading domain. The example creates event patterns to filter stock tick events based on price and symbol. When a stock tick event is encountered that falls outside the lower or upper price limit, the example simply displays that stock tick event. The price range itself is dynamically created and changed. This is accomplished by an event patterns that searches for another event class, the price limit event.
The classes net.esper.example.stockticker.event.StockTick and PriceLimit represent our events. The event patterns are created by the class net.esper.example.stockticker.monitor.StockTickerMonitor.
Summary:
Good example to learn the API and get started with event patterns
Dynamically creates and removes event patterns based on price limit events received
Simple, highly-performant filter expressions for event properties in the stock tick event such as symbol and price
In the MatchMaker example every mobile user has an X and Y location, a set of properties (gender, hair color, age range) and a set of preferences (one for each property) to match. The task of the event patterns created by this example is to detect mobile users that are within proximity given a certain range, and for which the properties match preferences.
The event class representing mobile users is net.esper.example.matchmaker.event.MobileUserBean. The net.esper.example.matchmaker.monitor.MatchMakingMonitor class contains the patterns for detecing matches.
Summary:
Dynamically creates and removes event patterns based on mobile user events received
Uses range matching for X and Y properties of mobile user events
This example develops some code for measuring quality-of-service levels such as for a service-level agreement (SLA). A SLA is a contract between 2 parties that defines service constraints such as maximum latency for service operations or error rates.
The example measures and monitors operation latency and error counts per customer and operation. When one of our operations oversteps these constraints, we want to be alerted right away. Additionally, we would like to have some monitoring in place that checks the health of our service and provides some information on how the operations are used.
Some of the constraints we need to check are:
That the latency (time to finish) of some of the operations is always less then X seconds.
That the latency average is always less then Y seconds over Z operation invocations.
The net.esper.example.qos_sla.events.OperationMeasurement event class with its latency and status properties is the main event used for the SLA analysis. The other event LatencyLimit serves to set latency limits on the fly.
The net.esper.example.qos_sla.monitor.AverageLatencyMonitor creates an EQL statement that computes latency statistics per customer and operation for the last 100 events. The DynaLatencySpikeMonitor uses an event pattern to listen to spikes in latency with dynamically set limits. The ErrorRateMonitor uses the timer 'at' operator in an event pattern that wakes up periodically and polls the error rate within the last 10 minutes. The ServiceHealthMonitor simply alerts when 3 errors occur, and the SpikeAndErrorMonitor alerts when a fixed latency is overstepped or an error status is reported.
Summary:
This example combines event patterns with EQL statements for event stream analysis.
Shows the use of the timer 'at' operator and followed-by operator -> in event patterns
Outlines basic EQL statements
Shows how to pull data out of EQL statements rather then subscribing to events a statement publishes
The Linear Road example is a very incomplete implementation of the Stream Data Management Benchmark [3] by Standford University.
Linear Road simulates a toll system for the motor vehicle expressways of a large metropolitan area. The main event in this example is a car location report which the class net.esper.example.linearroad.CarLocEvent represents. Currently the event stream joins are performed by JUnit test classes in the eg/test folder. See the net.esper.example.linearroad.TestAccidentNotify and the TestCarSegmentCount classes. Please consider this a work in progress.
Summary:
Shows more complex joins between event streams.
The RSI gives you the trend for a stock and for more complete explanation, you can visit the link: http://www.stockcharts.com/education/IndicatorAnalysis/indic_RSI.html.
After a definite number of stock events, or accumulation period, the first RSI is computed. Then for each subsequent stock event, the RSI calculations use the previous period’s Average Gain and Loss to determine the “smoothed RSI”.
Summary:
Uses a simple event pattern with a filter which feeds a listener that computes the RSI, which publishes events containing the computed RSI.