Regulation – a class of Big Data apps

There are bad guys out there!


Going back to the gist of my last post, one of the pillars that underpinned de-regulation was the idea that companies would work in a ‘correct’ manner and regulate themselves. The truth is that this worked and still does work very well for 95% of companies but there are always bad pennies committing fraud or simply not being careful in accounting practices. Thanks to a few well known financial disasters, even before the global meltdown, the concept of re-regulation loomed large across many industries. There are many sets of rules that are now in place to bring governance to company business – some of the more well known include Sarbanes-Oxley and Basel II and III which have been around for a little while now. We might ask ourselves what do they have in common and the answer is that both and many more such initiatives, demand that very accurate and accountable numbers are produced quickly from very complex underlying data – the need for Business Intelligence rears its head once again and the term ‘Big Data’ can certainly be applied to some of these initiatives.


Re-regulation demands that some very complex numbers are delivered:


  • Quickly
  • Accurately
  • Transparently



Throw into the pot that the data needed often as not comes from tens or even hundreds of operational systems distributed across the world and that some of these initiatives need very complex predictive modelling and detailed segmentation and we see a new class of Big Data applications.


Big Data = more diverse data



Many ‘mega-trends’ are in place today – Globalisation, re-regulation, internet shopping, disengaged customers and more take-overs day by day. The need to have accurate information is paramount simply to survive let alone grow.


Too often we use big words without thinking about what they mean and Globalisation is one of them. Now I am not going to write about globalisation here but it is useful to consider it as a phenomenon, is it real, is it important?


Let’s consider some facts:


  • 70% of the world’s shoes come from one town in China – now if you produce shoes in the UK this fact should be very worrisome.
  • To all intents and purposes the UK no longer has a car industry – we used to have, but in the end British Leyland amongst others proved a tad slow and not too smart. There’s nothing left anymore.
  • In the space of just a few years Vodafone has penetrated nearly the entire know world with its mobile services. Unless you take active steps to prevent it, you are almost guaranteed to end up paying some money one way or another, to Vodafone this year.
  • Most holiday companies now make a sizable proportion of their revenue from banking products or shipping cargo.



Globalisation is the force behind the break-down of trading barriers but globalisation is partly a result of another massive change in business practices over the last twenty years that we call de-regulation. Basically, in the ‘old days’ there were rules about what a company (or type of company) could sell. For example, Building Societies could not lend savers money to borrowing customers directly – you had to have a banking license to do that. Retailers could not sell insurance products. Insurance companies could not provide savings accounts. This all changed in the process of de-regulation and so now retailers can sell banking products, banks can sell insurance products and by and large, anything goes. When you put the two things together, globalisation and de-regulation, we have another world, a world in which the biggest retailer ever seen – Wall-Mart can presumably sell banking services in the UK thus becoming a competitor of Barclays Bank!


Note: Wall-Mart own ASDA – I’m not sure if they sell banking products but I guess so.


So what does that mean today in terms of Big Data. Well now your average retailer knows a lot more about you than ever before. They used to know what you eat, now they know what you ware, where you go on holiday, how much you spend and get paid a month etc, etc and it’s by combining all of this information that a 360 degree of a consumer can be constructed. By trawling social media feeds they can find who your friends and family are and what you are saying about their products………scary!!

Our Favorite 40+ Big Data use-cases. What’s your?

One of the key best practices for successful implementation of a big data analytics solution is to validate the business use case for big data. It will help organization with two important aspects for success:

1. Keeping the scope limited

2. Helping to measure the success of a solution that addresses a key business problem

In case the same data set addresses multiple use cases, an organization may need to prioritize their use case and apply an iterative and phased approach. It’s the theory of getting the biggest bang for the buck, both tactical and strategic. Think Big and Act small!

While there are extensive industry-specific use cases, here are some for handy reference:

EDW Use Cases

  • Augment EDW by offloading processing and storage
  • Support as preprocessing hub before getting to EDW

Retail/Consumer Use Cases

Financial Services Use Cases

  • Compliance and regulatory reporting
  • Risk analysis and management
  • Fraud detection and security analytics
  • CRM and customer loyalty programs
  • Credit risk, scoring and analysis
  • High speed arbitrage trading
  • Trade surveillance
  • Abnormal trading pattern analysis

Web & Digital Media Services Use Cases

  • Large-scale clickstream analytics
  • Ad targeting, analysis, forecasting and optimization
  • Abuse and click-fraud prevention
  • Social graph analysis and profile segmentation
  • Campaign management and loyalty programs

Health & Life Sciences Use Cases

  • Clinical trials data analysis
  • Disease pattern analysis
  • Campaign and sales program optimization
  • Patient care quality and program analysis
  • Medical device and pharma supply-chain management
  • Drug discovery and development analysis

Telecommunications Use Cases

  • Revenue assurance and price optimization
  • Customer churn prevention
  • Campaign management and customer loyalty
  • Call detail record (CDR) analysis
  • Network performance and optimization
  • Mobile user location analysis

Government Use Cases

  • Fraud detection
  • Threat detection
  • Cybersecurity
  • Compliance and regulatory analysis

New Application Use Cases

  • Online dating
  • Social gaming

Fraud Use-Cases

  • Credit and debit payment card fraud
  • Deposit account fraud
  • Technical fraud and bad debt
  • Healthcare fraud
  • Medicaid and Medicare fraud
  • Property and casualty (P&C) insurance fraud
  • Workers’ compensation fraud

E-Commerce and Customer Service Use-Cases

  • Cross-channel analytics
  • Event analytics
  • Recommendation engines using predictive analytics
  • Right offer at the right time
  • Next best offer or next best action

These are some of my favorites and ones that I have come across. Please add your favorites to the comment section. I would like to know from readers what they are seeing in their organization.


Big Data Use-cases – Insurance, Energy and Utilities, Travel and Hi-Tech

Big Data is still in its infancy stage of it’s life cycle and alike traditional EDW/BI, it will take couple years to be part of nervous system of an organization. From what I have seen recently, most of the real big data initiatives that are currently in Production are by start-ups and upcoming companies that are working on new product line around real-time analytics and integration. Some of them are actually offering their products through cloud as SAAS and specifically extending it to AAAS (Analysis as a Service). The Cloud and Big Data computing has opened a new market for innovators.

Let’s look at some of the use cases for few traditional industries here:

Big Data use-cases in Insurance Industry

At the recent insurance industry trade shows, “big data” was the talk of the floor. The broad, vague term means different things to different people, though, and like many buzzwords suffers a bit from overexposure. But most people see the potential of advanced data and analytics to make insurance operations more efficient and effective. Here are few of the use cases that are very relevant to Insurance industry and has high ROI potential

  • Fraud Detection & Analysis
  • Personalized Pricing:
  • Customer Sentiment Analysis
  • Catastrophic Planning
  • Call Detail Record
  • Loyalty Management
  • Social Media Analytics
  • Advertising and Campaign Management
  • Agents Analysis
  • Customer Value Management
  • Underwriting and Loss modeling

Big Data use-cases in Utility Industry:

With the proliferation of smart meters, utilities are finding themselves inundated with data as they build out the Internet-enabled, interactive power system called the smart grid. But according to Oracle survey, many utility companies have yet to exploit that data as they modernize the grid. “The average utility with at least one smart meter program in place has increased the frequency of its data collection by 180x– collecting data once every four hours as opposed to just once a month,” states the survey of 151 utility executives in the U.S. and Canada conducted in April. “Despite improvements, 45%of utilities still struggle to report information to business managers as fast as they need it and 50% miss opportunities to deliver useful information to customers”. Here are few of the use cases that’s candidate for big data implementation and analytics

  • Smart Meters – Notifications and Alerts,
  • Smart Meters – Real-time Usage Pattern Analysis
  • Smart Meters – Predictive Analysis for Distribution of power
  • Smart Grid – Weather Pattern and Real Time Usage and Distribution
  • Manage Disasters and Outages
  • Compliance Checks and Audits
  • Customer Sentiment Analysis
  • Customer Feedback and Call Detail Record Analysis

Big Data use-cases in ECommerce & Hi-Tech Digital Industry

  • Association and Complementary Products – Big Data can be used as input to recommendation engines within websites like Amazon or that can increase average order size by recommending complementary products on real time basis based on predictive analysis for cross-selling.
  • Cross-channel analytics — sales attribution, average order value, lifetime value (e.g., how many in-store purchases resulted from a particular recommendation, advertisement or promotion).
  • Event analytics — what series of steps (golden path) led to a desired outcome (e.g., purchase, registration).
  • Right Offer at the Right Time
  • Next Best Offer – deploying predictive models in combination with recommendation engines that drive automated next best offers and tailored interactions across multiple interaction channels.
  • Large-scale click-stream analytics
  • Ad targeting, analysis, forecasting and optimization
  • Abuse and click-fraud prevention
  • Social graph analysis and profile segmentation
  • Campaign management and loyalty programs

Big Data use-cases in Travel Industry

Just think about every data point produced in one single business or pleasure trip, from the chosen time, airline choice, hotel destination, meals, entertainment decisions, and how that data can then be used to provide a deeper and richer consumer experience in a quicker and easier sale of services. As one industry publication points out it is the ability to personalize that will spur Big Data analytics and foster development of new applications and new services in the travel realm.

  • Personalized Pricing for Travel – Aviation, Hotels, Vacation Packages and Cars….
  • Customer Sentiment and Behavior Analysis
  • Customer Loyalty Management
  • Call Detail Record Analysis for Customer Experience
  • Traffic Pattern and Congestion Management
  • GPS Coordinated data processing for Geo-Fencing
  • Social Media – Advertising and Campaign Management
  • Social Media – Consumer Feedback and Interaction analysis

These are just few of the possibilities of big data use cases in these industries. Though the possibilities are endless, key to success is to have a strong framework for big data implementation and having the first step right in the direction.