Validate your Big Data Business use case before you implement

One of the key best practice for successful implementation of Big Data Analytics solution is to validate the business use case for Big Data. It will help organization with 2 important aspects for successful implementation:

1. Keeping the scope limited to help work with limited data set within Big data context

2. Help measure the success of solution that address key business problem

In case the same data set addresses multiple use cases, organization may need to prioritize their use case and apply iterative and phased approach. It’s the theory of getting biggest bang for the buck, tactical and strategic.

While there are extensive industry specific use cases, here are some of the use cases for handy reference.

Retail/Consumer Use Cases

  • Merchandizing and market basket analysis.
  • Campaign management and customer loyalty programs.
  • Supply-chain management and analytics.
  • Event- and behavior-based targeting.
  • Market and consumer segmentations

Financial Services Use Cases

  • Compliance and regulatory reporting
  • Risk analysis and management
  • Fraud detection and security analytics
  • CRM and customer loyalty programs
  • Credit risk, scoring and analysis
  • High speed Arbitrage trading
  • Trade surveillance
  • Abnormal trading pattern analysis

Web & Digital Media Services Use Cases

  • Large-scale clickstream analytics
  • Ad targeting, analysis, forecasting and optimization
  • Abuse and click-fraud prevention
  • Social graph analysis and profile segmentation
  • Campaign management and loyalty programs

Health & Life Sciences Use Cases

  • Clinical Trials Data Analysis
  • Disease Pattern Analysis
  • Campaign and sales program optimization
  • Patient care quality and program analysis
  • Medical Device and Pharma Supply-chain management
  • Drug discovery and development analysis

Telecommunications Use Cases

  • Revenue assurance and price optimization
  • Customer churn prevention
  • Campaign management and customer loyalty
  • Call Detail Record (CDR) analysis
  • Network performance and optimization
  • Mobile User Location analysis

Government Use Cases

  • Fraud detection
  • Threat detection
  • Cybersecurity
  • Compliance and regulatory analysis

New Application Use Cases

  • Online Dating
  • Social Gaming

Fraud Use-Cases

  • Credit and debit payment card fraud;
  • Deposit account fraud;
  • Technical fraud and bad debt;
  • Healthcare fraud;
  • Medicaid and Medicare fraud;
  • Property and Casualty (P&C) insurance fraud,
  • Workers’ compensation fraud.

E-Commerce Use-Cases

  • Cross-Channel Analytics
  • Event Analytics
  • Recommendation Engines using Predictive Analytics
  • Right Offer at the Right time
  • Next Best Offer or Next Best Action (Ebay, Netflix, Amazon and Others)

Building your Data Science Team: Business Engineering Unit

Every day, we create 2.5 Quintillion bytes of data – so much that 90% of the data in the world has been created in last 2 years alone. Every organization is now accumulating Terabytes and Petabytes of data coming from various devices – machines, mobile, user, web logs and cookies, social media, application servers and transactional logs, etc. Organizations are rushing to store this wealth of information fearing missed opportunities. The challenge is not in storing this information but able to find usage of this data to its competitive advantage.

Leading organizations are rapidly reorganizing themselves and building the Data Science Team. This is very similar pattern that you may have noticed in early to mid 90’s wherein organization started with Business Intelligence team rebranding MIS or DSS teams. The key to competitive advantage is not by storing all and most of the data but by deriving value and insight from it and be able to tie it with business plan that can drive tangible business outcomes. This new function that is evolving to be a Data Science team primarily needs four tiered layer of experts: Data Science Champion, Business/Product Managers, Analytical Data Modeler and Big Data Engineer. To make the Data Science team successful, it’s key that they operate under a Data Science Champion such as Chief Data Officer or Chief Data Scientist and not under a traditional IT organization.

Let us define what each of these layers mean to an organization.

§         Data Science Champion: This is an Executive level sponsor such as Chief Data Officer or Chief Data Scientist. They lay out the vision and lead the mission for the team. This champion is a domain expert in a data science field. A large organization may have several Chief Data Scientist roles specific to LOB working for a Chief Data Officer.

§         Business/Product Managers: They are the product managers from the business that closely partners with the CDO or Chief Data Scientist

§         Analytical Data Modeler: These are the advanced mathematician and statisticians who can apply their computer skills to create complex analytical models largely contributing to the mission

§         Big Data Engineer: These are the computer science engineers who apply the sophisticated engineering skills to process large volumes of datasets using big data tools.

It’s hard to find a Data Scientist who will have all the above 4 skills – leadership, business knowledge, analytical experience, big data processing skills. A successful Data Science Team is partnership of above defined four distinct roles. When building your team, it’s important to focus on these key data science skills: analytical and curiosity mind, creativity, domain knowledge, advanced math skills including a solid background in calculus, geometry, linear algebra, and statistics and a computer science background. Leadership, Analytical and Computer Science skills are important, particularly for the first members chosen for your data science team.

A new concept of Business Engineering Unit under the leadership of CDO/Chief Data Scientist is slowly evolving that will house the Data Science team. This has been seen in some of the large retailers, telecom, and media/entertainment companies. Data Science deals with identifying the real problem or a business opportunity. Unlike IT as a support organization in many companies, the Business Engineering Unit needs to be a profit center that will contribute to the company’s topline or bottomline. To prove the value from Data Science, the Champion needs to initially focus on the hardest business problems within an organization or unique business opportunity that have the highest return for key stakeholders. Organization boundaries and internal political environment are often the biggest challenges facing a data science team.

Companies that have made investments in big-data computing will reap extraordinary near-term and long-term benefits. Data Science is perhaps the biggest innovation in computing in the last decade.  The benefits from data science have already been proven in some industry sectors; the challenge is to extend the technology and to apply it more widely and in all facets of interaction between humans and machines.

Top 10 Big Data Trends in 2014

In January 2014, IDG published their latest big data enterprise survey and predictions for 2014 finding that on average, enterprises will spend $8M on big data –related initiatives in 2014. The study also found that 70% of enterprise organizations have either deployed or are planning to deploy big data-related projects and programs.

Here’s our Top 10 Big Data Trends in 2014:

  1. Big Data as a Service and Big Data Analytics will go mainstream
  2. More companies will implement Predictive Analytics, Machine Learning
  3. Data Science and Big Data Analytics will be embedded in BI for actionable Insights into Operational Reports and Executive Dashboards
  4. Cloud computing and Big Data will be tightly integrated with BI solutions
  5. Enterprise will be using big data techniques to secure IT infrastructure
  6. Hadoop will be used for operational system and transactional application
  7. Hadoop will be implemented as extensions to part of Enterprise Information Management solutions
  8. Big Data and Data Scientists skills shortage will grow as companies start ramping up hiring for big data and data science projects
  9. Rise in M&A activity in Big Data space with legacy BI companies acquiring niche big data vendors
  10. Companies will start new roles defined as Chief Data Scientists, Chief Data Officers and Chief Analytic Officer

TechNavio’s analysts forecast the Global Big Data market to grow at a CAGR of 34.17 percent over the period 2013-2018. TechNavio’s report, the Global Big Data Market 2014-2018.

The key vendors dominating this market space are IBM Corp., Hewlett-Packard Co., Oracle Corp., and Teradata Corp.

Other vendors are 1010data Ltd., 10gen Inc., Accenture Inc., Amazon Web Services, Attivio Inc., Calpont Corp., Capgemini Inc., ClickFox Inc., Cloudera Inc., Computer Sciences Corp., Couchbase Inc., Datameer Inc., DataStax Inc., Dell Inc., Digital Reasoning Systems Inc., EMC Corp., Fractal Analytics Inc., Fujitsu Ltd., Hitachi Ltd., Hortonworks Inc., HPCC Systems Inc., Huawei Technologies Co. Ltd., Informatica Corp., Intel Corp., Karmasphere Inc., Logica plc, MapR Technologies Inc., MarkLogic Inc., Microsoft Corp., Mu Sigma Inc., NetApp Inc., Opera Solutions Inc., ParAccel Inc., Pervasive Software Inc., QlikTech Ltd., RainStor Inc., Red Hat Inc., SAP AG, SAS Institute Inc., Seagate Inc., Siemens Information Systems Ltd., Splunk Inc., Supermicro Computer Inc., Tableau Software Inc., Tata Consultancy Services Ltd., Think Big Analytics Inc., and Xerox Corp.

The CIO in the Age of Analytics: From Infrastructure to Insight

Author: Joseph Baird

I have been advising senior executives for much of the past twenty years.  Many of my clients have been Chief Information Officers and it has been remarkable to see this role change during this time.  In the pre-2000 period, IT was seen as a strategic advantage and the move towards the web gave the CIO a very important seat at the table.

But as the decade wore on,  and the globalization of IT occurred, the role became more about managing down costs, the stabilization of ERP and operational systems and the providing of infrastructure for a mobile workforce.  Cost containment and predictability ruled the day and the job was more about Infrastructure vs. Information.

The CIOs that thrived during this time period were skillful administrators, negotiators and technology generalists. Their backgrounds mostly were from within the IT organization or Finance.  This served companies very well in this period of stabilization and commoditization of core IT services.  As the famous Harvard Business Review article  proclaimed IT was perceived to no longer be strategic but rather a cost center that needed to be aggressively managed.

The role of the CIO became more about defending vs. attacking and many who assumed these leadership positions were not natural risk takers but risk managers.

And then the ground shifted…the age of the smart phone and tablet has ushered in a new wave of business models in which the consumer knows more than ever before about what, where and how they want to engage.  We are awash in Big Data Hype and the same HBR has now proclaimed….

Now these same CIOs are being asked to develop new consumer facing technologies and create solutions that can take advantage of Big Data.   But the skills and operating models necessary during a period of modest innovation and cost containment are not the same as required for the Age of Big Data.

The skills for data scientists, app developers and others may not only be hard to find but also are at a much higher price point.  This is creating challenges for CIOs who are far from comfortable taking the risk of these hires or the necessary experimentation and controlled failures.

I believe that multiple operating models are and will continue to emerge.  In the case where the current is not suited to extend his or her skill sets into the required areas, the role may be split. In other words, the traditional CIO may need to focus on infrastructure and senior management will need to hire a leader to drive the analytics initiatives forward.

CIOs should recognize that this model has a risk of driving the relative value down of the traditional role.  This model will likely occur where the CIO does not have a history of an operating role either in sales, product development or supply chain.

For those executives who want to make the leap to Chief Insight Officer, I offer a few words of advice.  First of all, recognize that the core skill gap is not technology but technical.  That is, emerging Big Data technologies are fairly easy to understand. The technical aspects of data mining, visualization and process optimization are not.  Spend more time understanding the appropriate use of techniques than you do on a specific technology or vendor. 

Further, successful analytical transformation is all about adding predictive or prescriptive insight to core business processes. As such, knowledge of these business processes is essential.  The CIO who partners with the business to build deep knowledge about the nature of the business will become essential. The ability to apply new technology and techniques to these processes is the holy grail.   The development of six sigma and other process mapping and optimization skills on your team should be a top priority.

Finally, get comfortable with the development of business cases that justify your investment in people and other resources.  Your success demands the formation of a A team who can deliver extraordinary results.  This will not be the low cost provider.  The Chief Insight Officer must confidently build investment cases to acquire this talent and deliver on the investment required.

Who will drive your Big Data Bus? Executive leadership in a new age

Author: Joseph Baird


First let me start by stating my dislike of the ambiguity and hype of the term “Big Data”.  And by Big Data in the context of this writing is nothing to do with the overused Velocity, Variety, Volume and Veracity.

But it has everything to do with what we can now more easily know about the signals from the physical world and its systems as well as the interactions and behaviors of customers for a wide range of companies around the globe.

I have spent much of the past five years advising senior executives from the world’s largest retailers and media companies.  From an nascent awareness brought about by the work of Tom Davenport a few years ago to the screaming hype from today’s cover of Harvard Business Review, it is clearly on the top of mind for nearly every CEO, CFO and COO.

But there is also a disconcerting pattern of corporate behavior that has largely left many frustrated and entering the land of disenchantment.   Non-IT Executives, uncomfortable with asking “stupid” questions or appearing vulnerable, too often have pushed the Big Data opportunity back to the CIO or others in the IT organization.

Yet, if you view the problem as an opportunity to create new products, customer experiences or market opportunities, the decision may be a bit different.   Let’s say your Walmart and you are making a decision to launch a new group of groceries.  Or maybe you are General Motors bring the new electrical Leaf to market?  Maybe even State Farm Insurance deciding which products to bring into which markets?

Invariably, these companies would bring together engineers, market researchers, and designers under the leadership of a very senior business executive.  And ultimately, the decisions(after thorough analysis, prototype and experimentation) to launch product would make its way to desk of the CEO or maybe the Board of Directors.

So, when I am asked who should lead a company’s Big Data efforts, I invariably ask what is the business trying to achieve in the market?   If there is a real, clear vision of a new market opportunity, leadership must be driven by the executives in charge of the functional area or business unit and not the IT leadership.

While there is a clearly a supporting role to be played by the CIO,  the bus must be driven by the senior executive responsible for the business opportunity. And, those executives must take the time to get smart about the new raw materials that are available to create these Big Data products.

Henry Ford became a student of the automation of the assembly line.  Fred Smith at Fedex immersed himself in the new logistics technology that picked Memphis as the first hub.  They realized that even though it was a technology problem, ultimate ownership was taken by the business leader not the technician.

Be good to remember as you decide who is going to drive your Big Data bus….

Top Big Data Posts for Year 2013 !!!

To close the busy 2013 year, here is the list of Top blogs for the Year 2013 by TBDI.

Total Hits: 16200+

Total Followers: 2100+

Total Basic Member: 10000+

We are growing rapidly! Thanks to all your support!

Home page / Archives 3,660
Big Data Use-Cases – Retail, Manufacturing and Auto Industry 1,515
Big Data Use-Cases in Telecom, Media and Entertainment industry 1,500
Big Data Use-cases – Banking and Financial Services 1,415
Our Favorite 40+ Big Data use-cases. What’s your? 1,129
Big Data Use-cases – Insurance, Energy and Utilities, Travel and Hi-Tech 1,080
Big Data Use-Cases in Healthcare – Provider, Payer and Care Management 896
10 Big Data Implementation Best Practices 887
Introduction to Big Data and Hadoop Ecosystem – For Beginners! 652
Will Hadoop replace or augment your Enterprise Data Warehouse? 486
7 steps to Advanced Predictive Analytics! 377
About TBDI 270
Top 10 Blogs so far this year 212
Addressing the big data security! 189
Wonder how to start your Big Data POC / Lab? 165
Hadoop meets SQL 149
Building your Launch Team! 120
Big Data Scientists, Architects, Lead and Developer Needed for IBM – Immediate Need!!! 119
The Hadoop Data Warehouse! 114
Running Hadoop in the Cloud 108
6 Steps to Start Your Big Data Journey 105
Big Data making the difference in Retail industry! 96
Big Data Analytics – Acquire, Grow and Retain Customers 89
Using Big Data Analytics in Healthcare 83
Join TBDI – The Big Data Institute! 77
Invitation to The Big Data Institute (TBDI) Partnership program 70
Who will drive your Big Data Bus? Executive leadership in a new age – By Joseph Baird 57
Analytics in Banking Services 45
Predicting Customer Behavior with Analytics! 44
Facebook and Twitter Social Media Analytics – From White House to Boardroom! 38
A new type of Blog? What Big Data might mean to the business 29
Welcome to new Partners and Alliances 27
Big Data = more diverse data 24
I wonder if exploiting Big Data will enable big companies to grow even bigger or whether it will enable smaller companies to compete with them to level the playing field? 23
Big Data…Defi… 21
Regulation – a class of Big Data apps 21
Join us this Thursday, May 2nd for our next Big Data Developer Meetup! 17
Making the most of what you have 17
Policy for Establishing New TBDI Chapters 13
TBDI Definition… 10
TBDI Big Data & Data Science Meetup in So California, USA 8
Natural Selection in Business – Does using Big Data provide a sustainable advantage? 5
The TBDI News! 4
Reserve your seat now – TBDI Big Data MeetUp: Big Data Analytics and Data Science – 2014 Trends 4
Being Bold and Big with your Big Data Pilots 4

Big Data beyond the hype

Big Data beyond the hype

The Big Data hype cycle is in full swing. But what is Big Data? How do you know if your data is BIG?

Big Data is not a concretely definable category. You can’t always say exactly what it is, but you know it when you see it. In this episode I define the key characteristics of Big Data that enable us to make more intelligent assessments and decisions regarding Big Data solutions.

Listen to the Podcast:

Key characteristics of Big Data:

  • Physical Attributes

    • Bigness: physical size of data sets

    • Multi-source: data from multiple sources, especially both internal and external to the organization

    • Multi-structure: tabular data, markup data, audio and video data, geospatial, activity, transactions, snapshots, statuses

    • Fast arriving: streaming, frequently updated, time volatile

  • How we process it

    • Real time analysis

    • Real time outputs

      • Delivery to decision makers in real time

      • Delivery to external users (consumers, social/mobile users)

      • Interaction with software APIs

    • Aggregate and details

  • What we do with it

    • Predictive value

    • Pattern recognition, especially unlikely relationships

      • fuzzy matching

      • flexible matching

  • Challenges

    • Storage

    • Processing

    • Integration

    • Analysis



TBDI Definition of Big Data: Big Data is a term applied to voluminous data objects that are variety in nature – structured, unstructured or a semi-structured, including sources internal or external to an organization, and generated at a high degree of velocity with an uncertainty pattern, that does not fit neatly into traditional, structured, relational data stores and requires strong sophisticated information ecosystem with high performance computing platform and analytical capabilities to capture, process, transform, discover and derive business insights and value within a reasonable elapsed time