In today’s enterprise, a successful big data strategy can mean the difference between success and failure. For example, Netflix reports the company is able to save $1 billion a year from customer retention thanks to its use of big data analytics, and enterprises in every other vertical market are following suit. Market research firm Statista forecasts big data analytics software spending will hit $68 billion by 2025. But adopting a big data strategy is a big undertaking for enterprises, and there are a host of questions an IT team must answer before they can decide on the best big data platform for their needs.

This paper will examine multiple criteria an enterprise IT team should consider before they adopt any big data platform. By rigorously evaluating a potential platform in each of these categories, IT teams will have a better understanding of the total cost of ownership (TCO) of their chosen platform. The paper will then apply each of those criteria to reveal how well an HPCC Systems data lake platform addresses the criteria. Finally, the paper will examine how an actual HPCC Systems customer evaluated HPCC Systems TCO and decided the platform was the best fit for their big data needs.

Since its beginning, HPCC Systems has given its users a platform consisting of a single homogenous data pipeline. This significantly minimizes the amount of effort users spend on platform management, installation, and maintenance. Perfect for both data lakes and warehouses, HPCC Systems is extremely capable and efficient in processing large amounts of data due to an architectural design that leverages two specialized clusters, named Thor and Roxie, to manage and optimize the platform’s various functions.

This paper serves as a comparison between the architectures and feature support of Spark and HPCC Systems in regard to data lake capabilities.

Today, most organizations recognize that data is key to the ability to innovate and remain competitive in a rapidly changing business landscape. A key challenge:

As datasets become larger and more complex, it’s impossible to quickly respond to changing business needs using traditional relational data store such as data warehouse.

To overcome this challenge, many organizations — including some of the world’s largest companies — are successfully using a proven alternative approach: a data lake. Data lakes support datasets that are extremely large, complex and diverse, and they easily accommodate new data sources such as IoT. They allow IT groups to quickly create new applications that support changing business needs, unlocking the power of complex data for all users within the organization. They also scale much more easily and cost-effectively than relational databases. As a result, data lakes enable greater responsiveness to business groups and external customers, reduced costs, and greater scalability.

A “Data Lake” is an architecture and methodology for the continuous management of complex data that stores data on raw format for increased agility on data exploration. As it enters the lake, each piece of data is readily available for manipulations and insights via a unique identifier and a set of extended metadata tags. In contrast, a “Data Warehouse” stores data in a predefined format for faster delivery of data analysis results.

HPCC Systems offers the best of both worlds by combining the fast performance of a Data Warehouse for information delivery with the ability to treat data as if it were in a Data Lake when it comes to data exploration. HPCC Systems uses distributed data architecture and a parallel processing methodology in order to work with large datasets. Enterprises are adopting data lake technology to manage their rapidly growing internal datasets and to solve complex problems through data analysis to improve their relationships with customers and suppliers.