Lexis Nexis Enters The World Of Open Source

LexisNexis Risk Solutions, the system used by law enforcement, insurance companies and the federal government to insure people are who they say they are, is making its big data handling engine available to compete with Hadoop and other emerging data handling systems.

Turn the mobile device management challenge into a business opportunity.

Discover four strategies to secure your mobile environment.

At the same time Escalante, who still serves as senior VP and CTO of LexisNexis Risk Solutions, said LexisNexis invented a language, ECL, which uses one-tenth the number of lines of code needed to build a query program compared to the Java programs that exploit Hadoop. ECL “is a very high level language that tells the HPCC cluster what is to be done, not how to do it … Our solution has a significant programmer productivity boost,” he said.

Nevertheless, the contrast with Hadoop, an open source project of the Apache Foundation, is something of an apples-to-oranges comparison. Hadoop is a file distribution system that can map a query to the cluster nodes that are closest to the data needed. It works on one query at a time, or many queries pushed sequentially through its architecture in a batch-like process. A high level, Hadoop-specific language, Pig, is being developed to speed up the process; likewise, Facebook is developing Hive for the same purpose. With HPCC, on the other hand, there are two related clusters, Thor and Roxie, working together to reduce many queries swiftly down to the relevant answers. Hundreds of concurrent queries can be answered in near real time or milliseconds, not multiple seconds or minutes, said Escalante. The Thor cluster is a brute force, extract, transform, and load system for handling data, and works in background at indexing it. Roxie is a parallel, distributed query engine that operates alongside Thor on the cluster, “tailored to do data delivery with low latency,” he said.
Both clusters are contained within Escalante’s projection of 100 servers needed.

“We believe that HPCC Systems will take big data computing to the next level,” said James Peck, CEO of LexisNexis Risk Solutions, in the announcement of the HPCC project on Wednesday. “We’ve been doing this quietly for years for our customers with great success.” LexisNexis has used its core engine for ten years.

If Peck and Escalante are correct, HPCC Systems will offer to further develop its core system through an open source project. HPCC is potentially a powerful big data engine that requires many fewer commodity servers to do its job than the number previously assumed needed.

Even so, Hadoop has many devotees and will still be used in many big data settings, especially for truly gargantuan tasks such as capturing and sorting all the data gained in a complete crawl of the Internet. Its own high-level languages are being developed at a rapid pace and may bring its capabilities closer in line with those of HPCC.

Escalante said LexisNexis was making its core engine open source code because “we don’t want to run a proprietary system. We want to build a community around the system and get the new innovation that people within the community will produce for it.” HPCC is written in C++ and will require C++ skilled open source programmers to advance its state of the art.