Technical

What additional components does Esper require to run?

Please see the feature list for this information. And the "lib" folder in the source distribution contains a readme file that describes the dependencies.


It claims to be fast...how does it do that? Has this claim been tested?

Esper uses the open-source CGLIB byte-code generation library to generate accessors to event properties at statement compile time. These accessors are cached and are almost as fast as compiled property access. This has been tested as part of the performance-asserting unit tests that are part of the source code. However, a more thorough performance evaluation has yet to be performed. There is also currently no standard performance test for CEP/ESP engines out there.


How has this been tested? What guarantees do I have that the next release works just as well?

Esper uses the JUnit testing framework to automate testing of the system. Esper's build process runs the currently around 1120 unit tests in 380 JUnit test classes when a build occurs. The test classes test the Esper engine from an end-to-end perspective as well as perform tests against individual classes and components. See the package "net.esper.regression" in the test source folder as the entry point for end-to-end testing against the client API.

The Esper team follows the practice of test-driven development (TDD) rigorously, ensuring that each feature added has automated test coverage. We develop and evolve the tests for each feature along with the feature that is currently being developed.


Can I run it with multiple threads? What, if anything, is multithread-safe?

Esper supports multiple independent Esper engines per Java VM. Each engine instance itself is NOT multithread-safe. In other words, for a given engine instance we do not currently guarantee the engine instance behaves the same in a multithreaded environment as in a single-threaded environment.

Applications using Esper can synchronize around the sendEvent() method. No ThreadLocal data representing engine state remains after the sendEvent() method completes. Thus multiple threads that synchronize posting events to the same engine instance is supported, with the following limitations: (1) statement creation through the EPAdministrator interface, and statement start/stop must be avoided or synchronized with the posting of events (2) Iterating (pulling) of statement results must be synchronized as well.

Applications can also segregate work to different engine instances allocating a thread to each engine instance.

Applications that perform pure filtering of events via event pattern statement are also thread-safe. In this scenario, the event pattern statement can only consist of a simple filter and no other operators. Also, new patterns cannot be created and patterns cannot be removed while multiple threads are used to send events in for filtering.

We plan to improve this in upcoming versions.


What operating systems has it been tested on?

At this time the engine has only been tested on Windows XP using JDK 1.5 update 6.

ESP and CEP

Could you explain the concept of windows for a database programmer?

One could perhaps think of a table with a timestamp column containing the time when the row was inserted. We could create a view that sorts by timestamp descending and selects all rows between the current timestamp and say up to 1 minute prior to now. Every time we fire a query against this view, the view returns the recent rows added in the last 1 minute. The rows returned are the contents of a 1 minute time window. Every time the query is fired we get a new window. Older rows would seem to leave the window while new rows would seem to enter the window.


What is the difference between Esper and an in-memory database?

The Esper engine works a bit like a database turned upside-down. Instead of storing the data and running queries against stored data, the Esper engine allows applications to store queries and run the data through. Response from the Esper engine is real-time when conditions occur that match queries. The execution model is thus continuous rather then only when a query is submitted.

General

How does Esper work? How does Esper allow you to search and match patterns on temporal events?

Esper is an event stream processing (ESP) and event correlation engine (CEP) written in Java. Basically instead of working as a database where you put stuff in to later poll it using SQL queries, Esper works as real time engine that triggers actions when event conditions occur among event streams. A tailored Event Query Language (EQL) allows registering queries in the engine, using javabeans to represent events. A listener class - which is basically a POJO - will then be called by the engine when the EQL condition is matched as events come in. The EQL allows expressing complex matching conditions that include temporal windows, and join different event streams, as well as filter and sort them.

A simple example could be to compute the average stock price of the BEA tick on a sliding window of 30 seconds. Given a StockTick event bean with a price and symbol property and the EQL "select avg(price) from StockTick.win:time(30 sec) where symbol='BEA'", a POJO would get notified as tick come in - and in real world millions of ticks can come in - so there's no way to store them all to later query them using a classical database architecture. Statements can be much more complex, and also be combined together with "followed by" conditions.

The internals of Esper are made up of fairly complex algorithms primarily relying on state machines and delta networks in which only changes to data are communicated across object boundaries when required.


What's the advantage of using Esper over other products?

Esper can easily be embedded in any Java runtime including J2EE application servers or standalone Java applications. Events in Esper can be represented as plain Java objects (POJO) eliminating the need to format, feed and receive events into a separate possibly remote engine process. Esper is an entirely free open-source component available under the LPGL license. The open-source nature of Esper helps in tailoring the event processing language and other community driven features.


Who's using Esper?

With the Esper 1.0 release and given it's shorter history most users are at the evaluation stage. A major financial institution is looking at Esper for real-time monitoring of proprietary options trading. A network monitoring application had success with Esper, and Esper is also playing a role in RFID research. Our user community is growing rapidly.


What business areas/problems is Esper best suited for?

Esper is best suited for real-time event driven applications. Typical application areas are business process management and automation, finance, network and application monitoring and sensor network applications. Esper take much of the complexity out of developing applications that detect patterns among events, filter events, aggregate time or length windows of events, join event streams, trigger based on absence of events etc.

A primary difference with system relying on classical SQL databases is that we do not query a repository for events matching some conditions, but instead trigger customized actions as the flow of events come in matching event conditions - hence drastically reducing the latency.


What might be some mis-uses for it?

Esper is not designed for storing and retrieval of fairly static data - that is better left to conventional databases. In-memory databases may be better suited to CEP applications than traditional relational databases as they generally have good query performance. Yet they are not optimized to provide immediate, real-time query results required for CEP and event stream analysis.


What is the intended audience and what is their interface?

Esper doesn't have a GUI and access is via API only, currently. The intended audience is developers of CEP or ESP applications.


How does Esper hook into an existing event-based system?

Today, event based system are often confused with message based system such as JMS and alike backends or Enterprise Service Bus (ESB). Messaging system are event based but usually do not correlate events or process event streams at all. They simply process unitary events - and it becomes quickly complex if you need to correlate messages.

Esper can hook into any Java based system as a message consumer - whose main purpose would be to listen efficiently to everything whilst sending back other events when a matching condition has occured. On the other hand, hooking Esper in a classic messaging system allows having it correlate events even if those come from non-Java based systems. This is just an example. Esper is completely container agnostic, can run as a standalone component, and the 1.0 release simply assumes events to be POJO. We are currently working on providing native XML format support as well.


How would you position Esper next to existing solutions (or hacks) in the industry?

Esper is the only open source Event Stream Processing out there - and this is not to be confused with classical rules engines. Compared to commercial ESP engines implementations, Esper may lack high availability options in the 1.0 release, but its open source nature is a driver to get the Event query language right, and get the Java community up to speed on those concepts - before tackling enterprise grade requirements in upcoming versions.

Generally Rete-based production rule engines can indeed be used to address part of what CEP coins. Especially triggering by correlating events, possibly including a temporal relationship between events. That said the ESP side - Event Stream Processing - is a different beast, where what matters is the "S" for Stream. In this side of the Esper engine we provide several language facilities to build expressions using time not for temporal relationship (happened before etc) but for sliding window. As an example, this makes it very valuable for computing things like volume weighted average price (VWAP) of ticks - which would possibly be awful using something Rete-based like a classical rule engine.


What is the concept or philosophy behind the design?

Esper was developed using test-driven development and excellent automated test coverage. Esper's design evolved by refactoring with courage towards higher design quality. Favorite patterns are dependency injection/inversion of control, GOF patterns, Immutable, Specification.


What is the history? Where did it start and come from?

Thomas Bernhardt is the project founder and project lead. He works as a software architect for a major financial institution. On a project in 2004 Thomas had been asked to evaluate different rules engines for use in an application for monitoring a trading system. The rules engines proved cumbersome and slow. There was no budget for a CEP product, thus a custom application to solve the monitoring problem was developed. The project lead's interest in CEP and ESP technologies lead to further independent research into the knowledge space followed by prototypes build for no particular organization or purpose. The prototypes evolved and the Esper project started.