Big Data as a Service (BDaaS), Featuring Sears

Many think Big Data will one of the next big things in computing and that Big Data as a Service (BDaaS) will be the means by which most organizations implement Big Data solutions.

According to International Data Corporation (IDC), the worldwide market for Big Data technology and services forecast is expected to grow 40% per year (compounded) – reaching almost $17 billion by 2015 from about $7 billion this year. And a lot of capital is flowing to companies developing Big Data technology and DBaaS.

The underlying driver behind BDaaS growth is the explosion in data and the fact that this data is becoming less structured and more varied. As to growth in the amount of data – IDC reportedly has estimated that 90 percent of all the data in the world has been created in the past two years. As to complexity – critical data now includes such things as images and data generated by social media; and comes from mobile devices, cameras, other devices that are becoming “smarter” and more powerful, and a host of other sources.

For more and more organizations, the ability to process, analyze and manipulate this data – and do it quickly – is becoming critical to their mission and competitiveness. But this complex data does not fit easily into traditional database structures and is difficult to exploit. And “scaling up” (e.g., buying bigger and bigger computers) is impractical, especially given the phenomenal increase in the amount of data, and costly.

So we come to Big Data and Big Data as a Service. Big Data can simply refer to the large and complex amounts of data organizations must deal with. Big Data can also refer to the “scale out” systems architecture (i.e., distributed and parallel processing) which allows organizations to address Big Data and their increasing need to exploit it by adding large numbers of relatively cheap computers rather than try to push everything through a single one.

The tools to handle Bid Data are evolving quickly, but there are few, if any, off-the-shelf solutions. And a Big Data solution is much more difficult to implement than solutions for traditional data-sets. Another problem for organizations that want to improve their ability to exploit Big Data on their own is that there is a shortage of skilled Big Data engineers and developers, especially those with experience with the major platforms being developed to allow scalable, distributed processing of large data-sets. There is also a shortage of data scientists to effectively mine and exploit all this data.

In addition, organizations must first set up the necessary cloud computing infrastructure.

Thus the growing need for Big Data as a Service (BDaaS), which joins an ever-increasing list of interrelated “as a Service” terms such as Software as a Service (SaaS), Infrastructure as a Service (IaaS), Platform as a Service (PaaS) and other such terms normally associated with cloud computing. BDaaS providers can implement, in a relatively short time period, systems that might take an organization years to create on its own.

The Sears Experience
A recent example is Sears, which went from developing its own distributed Big Data system over several years to seeing a market opportunity in using its Big Data expertise to offer BDaaS through a newly created subsidiary – MetaScale.

Sears, the well-known retailer with over 4,000 stores, got into the business while developing analytics for its own business. Sears offers a broad range of goods and services spread over its thousands of stores. This broad product range faces different pricing competition in different markets. Plus the company claims to have at least some details on 100 million customers and make over 11 million service calls annually.

In the hyper-competitive world of U.S. retail, which increasingly includes e-stores, Sears wanted to increase its ability to target individual customers with special offers, react to local conditions such as major storms, and adjust prices as needed almost in real time. To do this Sears needed to be able to analyze and exploit the vast amount of data it had available.

Use of advanced “analytics” is becoming increasingly important to many industries, but particularly retail. Analytics can help retailers deliver targeted and/or personalized advertising and promotions; improve customer loyalty; determine optimal pricing through a products lifecycle or in reaction to local weather and competitive conditions; localize store assortments and formats; measure marketing effectiveness; reduce overstock inventory; and determine new store locations.

To remain competitive in the future, retailers will need to be able to analyze massive amounts of data, often a mixture of complex (such as social media) and structured data that doesn’t neatly fit conventional database structures – i.e., Big Data. Even most companies with vast data warehouses typically are currently unable to effectively analyze more than a portion of their data.

To get ahead of the curve, Sears developed its own Big Data processing and analytics system, using Apache Hadoop as a platform. Hadoop is a still-evolving open source data management framework that allows for processing and storing huge amounts of data over a large number of computers, freeing companies from dependence on expensive mainframes. Major companies such as Yahoo have taken part in developing Hadoop into an enterprise-ready software library.

Sears realized, as noted above, that a) there are still not many off-the-shelf applications for platforms like Hadoop, and implementing a Big Data solution is a lot more difficult than for a regular database system, and b) at this time there is a shortage of skilled IT professionals with Hadoop experience and data scientists with the ability to analyze and exploit Big Data.

Seeing the market opportunity, Sears decided to leverage its expertise and established MetaScale as separate business. MetaScale is partnering with companies offering analytical and other tools to maintain and improve its best-in-class status.

It is hoped that an experienced group like MetaScale will capture significant market share and deliver significant profits. Perhaps in time Sears, a traditional retailer, will become known as a world class provider of technology services.