Top 10 Big Data and Analytics Predictions for 2015

Predictions for 2015

Forrester’s forecast that Hadoop will become an “enterprise priority” in the next 12 months, International Data corp. has just gazed into its own crystal ball, and sees a future where spending on Big Data analytics is set to grow three times faster in 2015.

Here’s our prediction of Top 10 Big Data and Analytics trends for 2015

  1. Rise of Chief Data Officer and Data Scientists positions
  2. Cloud Computing and Cloud Data Warehouse will grow adoption
  3. Data Visualization tools will become very popular with business users with advanced visualization capabilities
  4. Data Integration Hub at most large and mid-scale enterprise will expand to include unstructured data and incorporate advanced and predictive analytics using machine learning and advanced capabilities such as Watson
  5. Integration with Enterprise Mobile Apps and Big data technology on the rise
  6. Organizations will move from data lakes to processing data platform. Self-service Big Data with predictive analytics and advanced visualizations will go mainstream
  7. Companies and organizations will struggle to find data engineers and data scientist talent. Many legacy ETL/Integration and EDW resources will fill the gap.
  8. The Internet of Things (IoT) will be the next critical focus for data/analytics services
  9. Data Providers will evolve rapidly into new business model
  10. Data Privacy and security will gain momentum in implementation

While we can’t know for sure if each of these things will come true, we do know that the world of big data is changing. It’s no longer about just having access to data and the ability to store it, but instead the ability to achieve actionable results with data through predictive and prescriptive analytics. Only time will tell how this evolves, but if you aren’t leveraging data to compete and win, it’s time to get on board. The big data and analytics market will reach $125 billion worldwide in 2015, according to IDC and if you are not investing or planning to invest, you are already behind your competitor or losing your spot from being an A player.


Daily Newspaper – From Sushil’s Desk

Daily Newspaper – From Sushil’s Desk

Leading Information – Big Data, Analytics, Business Intelligence and IT Trends



Building your Data Science Team: Business Engineering Unit

Every day, we create 2.5 Quintillion bytes of data – so much that 90% of the data in the world has been created in last 2 years alone. Every organization is now accumulating Terabytes and Petabytes of data coming from various devices – machines, mobile, user, web logs and cookies, social media, application servers and transactional logs, etc. Organizations are rushing to store this wealth of information fearing missed opportunities. The challenge is not in storing this information but able to find usage of this data to its competitive advantage.

Leading organizations are rapidly reorganizing themselves and building the Data Science Team. This is very similar pattern that you may have noticed in early to mid 90’s wherein organization started with Business Intelligence team rebranding MIS or DSS teams. The key to competitive advantage is not by storing all and most of the data but by deriving value and insight from it and be able to tie it with business plan that can drive tangible business outcomes. This new function that is evolving to be a Data Science team primarily needs four tiered layer of experts: Data Science Champion, Business/Product Managers, Analytical Data Modeler and Big Data Engineer. To make the Data Science team successful, it’s key that they operate under a Data Science Champion such as Chief Data Officer or Chief Data Scientist and not under a traditional IT organization.

Let us define what each of these layers mean to an organization.

§         Data Science Champion: This is an Executive level sponsor such as Chief Data Officer or Chief Data Scientist. They lay out the vision and lead the mission for the team. This champion is a domain expert in a data science field. A large organization may have several Chief Data Scientist roles specific to LOB working for a Chief Data Officer.

§         Business/Product Managers: They are the product managers from the business that closely partners with the CDO or Chief Data Scientist

§         Analytical Data Modeler: These are the advanced mathematician and statisticians who can apply their computer skills to create complex analytical models largely contributing to the mission

§         Big Data Engineer: These are the computer science engineers who apply the sophisticated engineering skills to process large volumes of datasets using big data tools.

It’s hard to find a Data Scientist who will have all the above 4 skills – leadership, business knowledge, analytical experience, big data processing skills. A successful Data Science Team is partnership of above defined four distinct roles. When building your team, it’s important to focus on these key data science skills: analytical and curiosity mind, creativity, domain knowledge, advanced math skills including a solid background in calculus, geometry, linear algebra, and statistics and a computer science background. Leadership, Analytical and Computer Science skills are important, particularly for the first members chosen for your data science team.

A new concept of Business Engineering Unit under the leadership of CDO/Chief Data Scientist is slowly evolving that will house the Data Science team. This has been seen in some of the large retailers, telecom, and media/entertainment companies. Data Science deals with identifying the real problem or a business opportunity. Unlike IT as a support organization in many companies, the Business Engineering Unit needs to be a profit center that will contribute to the company’s topline or bottomline. To prove the value from Data Science, the Champion needs to initially focus on the hardest business problems within an organization or unique business opportunity that have the highest return for key stakeholders. Organization boundaries and internal political environment are often the biggest challenges facing a data science team.

Companies that have made investments in big-data computing will reap extraordinary near-term and long-term benefits. Data Science is perhaps the biggest innovation in computing in the last decade.  The benefits from data science have already been proven in some industry sectors; the challenge is to extend the technology and to apply it more widely and in all facets of interaction between humans and machines.

The CIO in the Age of Analytics: From Infrastructure to Insight

Author: Joseph Baird

I have been advising senior executives for much of the past twenty years.  Many of my clients have been Chief Information Officers and it has been remarkable to see this role change during this time.  In the pre-2000 period, IT was seen as a strategic advantage and the move towards the web gave the CIO a very important seat at the table.

But as the decade wore on,  and the globalization of IT occurred, the role became more about managing down costs, the stabilization of ERP and operational systems and the providing of infrastructure for a mobile workforce.  Cost containment and predictability ruled the day and the job was more about Infrastructure vs. Information.

The CIOs that thrived during this time period were skillful administrators, negotiators and technology generalists. Their backgrounds mostly were from within the IT organization or Finance.  This served companies very well in this period of stabilization and commoditization of core IT services.  As the famous Harvard Business Review article  proclaimed IT was perceived to no longer be strategic but rather a cost center that needed to be aggressively managed.

The role of the CIO became more about defending vs. attacking and many who assumed these leadership positions were not natural risk takers but risk managers.

And then the ground shifted…the age of the smart phone and tablet has ushered in a new wave of business models in which the consumer knows more than ever before about what, where and how they want to engage.  We are awash in Big Data Hype and the same HBR has now proclaimed….

Now these same CIOs are being asked to develop new consumer facing technologies and create solutions that can take advantage of Big Data.   But the skills and operating models necessary during a period of modest innovation and cost containment are not the same as required for the Age of Big Data.

The skills for data scientists, app developers and others may not only be hard to find but also are at a much higher price point.  This is creating challenges for CIOs who are far from comfortable taking the risk of these hires or the necessary experimentation and controlled failures.

I believe that multiple operating models are and will continue to emerge.  In the case where the current is not suited to extend his or her skill sets into the required areas, the role may be split. In other words, the traditional CIO may need to focus on infrastructure and senior management will need to hire a leader to drive the analytics initiatives forward.

CIOs should recognize that this model has a risk of driving the relative value down of the traditional role.  This model will likely occur where the CIO does not have a history of an operating role either in sales, product development or supply chain.

For those executives who want to make the leap to Chief Insight Officer, I offer a few words of advice.  First of all, recognize that the core skill gap is not technology but technical.  That is, emerging Big Data technologies are fairly easy to understand. The technical aspects of data mining, visualization and process optimization are not.  Spend more time understanding the appropriate use of techniques than you do on a specific technology or vendor. 

Further, successful analytical transformation is all about adding predictive or prescriptive insight to core business processes. As such, knowledge of these business processes is essential.  The CIO who partners with the business to build deep knowledge about the nature of the business will become essential. The ability to apply new technology and techniques to these processes is the holy grail.   The development of six sigma and other process mapping and optimization skills on your team should be a top priority.

Finally, get comfortable with the development of business cases that justify your investment in people and other resources.  Your success demands the formation of a A team who can deliver extraordinary results.  This will not be the low cost provider.  The Chief Insight Officer must confidently build investment cases to acquire this talent and deliver on the investment required.

Top Big Data Posts for Year 2013 !!!

To close the busy 2013 year, here is the list of Top blogs for the Year 2013 by TBDI.

Total Hits: 16200+

Total Followers: 2100+

Total Basic Member: 10000+

We are growing rapidly! Thanks to all your support!

Home page / Archives 3,660
Big Data Use-Cases – Retail, Manufacturing and Auto Industry 1,515
Big Data Use-Cases in Telecom, Media and Entertainment industry 1,500
Big Data Use-cases – Banking and Financial Services 1,415
Our Favorite 40+ Big Data use-cases. What’s your? 1,129
Big Data Use-cases – Insurance, Energy and Utilities, Travel and Hi-Tech 1,080
Big Data Use-Cases in Healthcare – Provider, Payer and Care Management 896
10 Big Data Implementation Best Practices 887
Introduction to Big Data and Hadoop Ecosystem – For Beginners! 652
Will Hadoop replace or augment your Enterprise Data Warehouse? 486
7 steps to Advanced Predictive Analytics! 377
About TBDI 270
Top 10 Blogs so far this year 212
Addressing the big data security! 189
Wonder how to start your Big Data POC / Lab? 165
Hadoop meets SQL 149
Building your Launch Team! 120
Big Data Scientists, Architects, Lead and Developer Needed for IBM – Immediate Need!!! 119
The Hadoop Data Warehouse! 114
Running Hadoop in the Cloud 108
6 Steps to Start Your Big Data Journey 105
Big Data making the difference in Retail industry! 96
Big Data Analytics – Acquire, Grow and Retain Customers 89
Using Big Data Analytics in Healthcare 83
Join TBDI – The Big Data Institute! 77
Invitation to The Big Data Institute (TBDI) Partnership program 70
Who will drive your Big Data Bus? Executive leadership in a new age – By Joseph Baird 57
Analytics in Banking Services 45
Predicting Customer Behavior with Analytics! 44
Facebook and Twitter Social Media Analytics – From White House to Boardroom! 38
A new type of Blog? What Big Data might mean to the business 29
Welcome to new Partners and Alliances 27
Big Data = more diverse data 24
I wonder if exploiting Big Data will enable big companies to grow even bigger or whether it will enable smaller companies to compete with them to level the playing field? 23
Big Data…Defi… 21
Regulation – a class of Big Data apps 21
Join us this Thursday, May 2nd for our next Big Data Developer Meetup! 17
Making the most of what you have 17
Policy for Establishing New TBDI Chapters 13
TBDI Definition… 10
TBDI Big Data & Data Science Meetup in So California, USA 8
Natural Selection in Business – Does using Big Data provide a sustainable advantage? 5
The TBDI News! 4
Reserve your seat now – TBDI Big Data MeetUp: Big Data Analytics and Data Science – 2014 Trends 4
Being Bold and Big with your Big Data Pilots 4

Regulation – a class of Big Data apps

There are bad guys out there!


Going back to the gist of my last post, one of the pillars that underpinned de-regulation was the idea that companies would work in a ‘correct’ manner and regulate themselves. The truth is that this worked and still does work very well for 95% of companies but there are always bad pennies committing fraud or simply not being careful in accounting practices. Thanks to a few well known financial disasters, even before the global meltdown, the concept of re-regulation loomed large across many industries. There are many sets of rules that are now in place to bring governance to company business – some of the more well known include Sarbanes-Oxley and Basel II and III which have been around for a little while now. We might ask ourselves what do they have in common and the answer is that both and many more such initiatives, demand that very accurate and accountable numbers are produced quickly from very complex underlying data – the need for Business Intelligence rears its head once again and the term ‘Big Data’ can certainly be applied to some of these initiatives.


Re-regulation demands that some very complex numbers are delivered:


  • Quickly
  • Accurately
  • Transparently



Throw into the pot that the data needed often as not comes from tens or even hundreds of operational systems distributed across the world and that some of these initiatives need very complex predictive modelling and detailed segmentation and we see a new class of Big Data applications.

7 steps to Advanced Predictive Analytics!

Predictive analytics encompasses a variety of statistical techniques from modelling, machine learning, data mining and that analyze current and historical facts to make predictions about future events. In business, predictive models exploit patterns found in historical and transactional data to identify risks and opportunities. Models capture relationships among many factors to allow assessment of risk or potential associated with a particular set of conditions, guiding decision making for candidate transactions.

Till late 2010, most of the enterprise business intelligence focus and analytics was around structured and some of semi-structured data (emails, logs and call records). The buzz words were ‘data-driven’ business decisions or actions. With era of big data and technology evolution, organizations are now venturing into unstructured data like sensor data from RFID tags, online web content and need for better analytical capabilities. The new buzz words that’s rapidly gaining popularity from meetings to conferences is ‘advanced analytics’ or ‘analytics-driven’ or ‘big insights’. With big data technology, the power of predictive analytics is getting a lot of coverage, and software vendors are touting the latest and greatest in technology and algorithms. Once seemed as boring and dull job or data geeks, the demand for statisticians and data scientists are high and rising. One of the recent articles from Harvard Business Review (HBR) – Oct 2012, it talks about Data Scientist: The Sexiest Job of the 21st Century.

As noted in HBR article, ‘Data are essential, but performance improvements and competitive advantage arise from analytics models that allow managers to predict and optimize outcomes. More important, the most effective approach to building a model rarely starts with the data; instead it originates with identifying the business opportunity and determining how the model can improve performance. According to research by Andrew McAfee and Erik Brynjolfsson, of MIT, companies that inject big data and analytics into their operations show productivity rates and profitability that are 5% to 6% higher than those of their peers. Often organizations look for help to start with their Advanced Predictive Analytics project. There are very limited production processes that leverage the power of Predictive Analytics that is embedded in BPM or decision making processes. In my recent experience from a Analytics Strategy and Assessment workshop for a client, the issue identified was not the tool or architecture but how to get insights from myriads of data that exist. The questions posed was a) Are we collecting and storing right data? b) What insights can be generated from this data?  What organizations want to know is not what kind of technology to buy first or what techniques and training they need, but what kind of problem to go looking for. What kind of problem will show the greatest return on an investment in predictive analytics? Where can they apply predictive analytics and get a clear and compelling “win”?

Based on my experience, here’s the key 7 steps to get started with Advanced Predictive Analytics

1. Define the Problem or Pain or Opportunity

2. Identify the key Metrics

3. Identify the Right Data that support metrics #2 above

4. Analyze and Enrich the data

5. Build models for Advanced Predictive Analysis

6. Experiment the model with test subject/group

7. Embed and Implement the analytics as part of business process or application

Most of the organizations are lost in the process of identifying or collecting data before documenting the step# 1. As mentioned above from my recent experience, the organization was collecting and storing all possible data without defining what insights they want to generate to drive business at tactical and strategic level.  Operational decisions align well with Predictive Analytics model. Most of the operational and tactical decisions are made by front line field level staff from call centers to customer service to sales reps. Predictive Analysis that can be embedded into business applications as part of workflow processes can add a tremendous value to providing an excellent service to customers as well as significantly increase results and outcome from cross-sell or up-sell opportunities. Many predictive analytic tools support access to a wide range of data sources, including those typically branded “big data,” such as unstructured text , or semi-structured Web logs and sensor data. The problem is that organizations are trying to apply these technologies to the wrong problem. With the urge to prove ROI from investments in Big Data and Analytics, organizations focus on large and wider problems than focusing on Operational level or tactical problem that can give an opportunity to implement and prove the solution.

Predictive models analyze past performance to assess how likely a customer is to exhibit a specific behavior in the future in order to improve marketing effectiveness. This category also encompasses models that seek out subtle data patterns to answer questions about customer performance, such as fraud detection models. Predictive models often perform calculations during live transactions, for example, to evaluate the risk or opportunity of a given customer or transaction, in order to guide a decision. Big data and predictive analytics can be combined together in the operational environment. By focusing on operational decisions, we can put big data to work, using it to drive predictions that improve our ability to make good decisions at the operational level. With advancement in computing speed, individual agent modeling systems can simulate human behavior or reaction to given stimuli or scenarios. The new term for animating data specifically linked to an individual in a simulated environment is Avatar Analytics.

From my recent analytics on Social media marketing for a specific brand, we found out some interesting statistics like 23% of customers repurchase on same day and most of the repurchases happens within 5 days of initial purchase. These are key insights for marketing campaign. Healthcare and Financial organizations has a huge potential to leverage predictive analytics for fraud detection, cross-sell and up-sell and interventions. In my experience at one of the leading healthcare companies where I managed the BI and Analytics team, analytics played a significant role on how members/patients were stratified by risk scores using statistical models. Organizations are gaining momentum in leveraging big data for insights and advanced analytics using statistical model and predictive modeling across many industry domains. Whether it is using analytics to predict customer behavior, set pricing strategy, optimize ad spending or manage risk, analytics is moving to the top of the management agenda.