Four Myths About Real Time Analytics


Five years ago, businesses beyond the e-commerce world were only dipping their toes into real time analytics. Today, brick and mortar businesses have embraced real time analytics in a range of applications, including error detection, price adjustment, inventory tracking, customer experience management, and more.

So what has changed? Certainly, advances in both hardware and software have helped. But at the heart of this reassessment is the data industry’s ability to dispel a number of myths surrounding real time data processing and analytics.

Here are four of the most deeply entrenched:

  1. Real time intelligence is only a nice-to-have

 A common misunderstanding about real time analytics is that it is the same as batch analytics, just conducted over a shorter time frame. Therefore, as long as important business reports are meeting required deadlines, intelligence in real time is regarded as less than critical.

In fact, in contrast to the analytics conducted on batch (ie static) data, real time analytics is not designed to diagnose key business trends. Rather, real time analytics is performed continually on streamed data in order to detect unusual events as they happen, and to trigger an immediate or quick response.

As such, real time analytics is best suited for processes which stream large volumes of data. Where the analysed data is signalling something amiss, fast action – usually dictated by pre-set business rules – can prevent it from escalating into a full blown problem.

For this reason, real time analytics’ greatest value is its ability to avert losses, for example, by detecting financial fraud or manufacturing defects. There is also the potential to improve routine operational efficiencies, such as through retail inventory management or telecoms network optimisation. Real time analytics is therefore deemed to deliver “operational intelligence”, and is complementary to, rather than a substitute for, “business intelligence”.

  1. The prices of my products/services do not change very often

Businesses are increasingly using real time analytics for another innovation – real time pricing. E-commerce sites have long used real time factors and complex algorithms to drive active price adjustments. What is perhaps less well known is the magnitude and velocity of these adjustments. Amazon, for example, changes the prices of about 40 million products many times during a single day, according to Internet Retailer Magazine.

The confidence that e-commerce sites have in their pricing models is now reflected in many traditional businesses. A report in the Wall Street Journal noted that, for example, adult passes to Indiana Zoo can range between USD 8 and 30, by responding in real time to weather changes, school group bookings, surprise closure of an attraction, and a variety of other factors.

Many other facilities now adopt similar real time pricing mechanisms, including highway tolls, parking lots, taxi services, golf courses, ski resorts, theme parks, and entertainment events. In the case of highway tolls, prices can fluctuate by as much as 500% in a single week. This real time approach to pricing is said to be capable of lifting revenues by 10 to 20%.

However, real time pricing offer benefits beyond that of higher revenues. Bottom line consistency and customer experience are often as important, with dynamic price adjustments used to control demand peaks and troughs, and associated resourcing costs. The question traditional businesses need to ask themselves is therefore not ‘How often should we review our prices?’ but rather ‘How can we approach pricing differently?’

  1. Real time information is unsubstantiated and therefore less valuable 

The term “real time” gives the impression of fleeting observations that are not verifiable. However, in most cases, real time analytics is not simply based on what has just occurred. Rather, such analyses involves correlations between real time and historical datasets, thereby placing a real time event in its proper context.

Take for example how a GPS now re-routes vehicles in order to avoid traffic congestion. The analyses required to do this is based not just on real time traffic information, but on the correlation with previous congestion events, in order to determine the best alternative routes that vehicles should take.

It is the ability to combine streamed real time data with historical data and data from other sources that gives real time analytics its greatest potency.

  1. Real time analytics is too expensive to implement

 Real time analytics is not without its complexities, but does not necessarily require a large business investment. This is because the evolution of open-source data technology has resulted in significantly more cost-efficient real time software. Spark, for example, is an open source project that is set to underpin much of the industry’s real time analytics capabilities.

In addition, rather counter-intuitively, the data architecture stack required to support the integration of real time feeds with existing batch based data processes can actually be a cheaper alternative to current storage technologies.

This is because the architecture needed to run real time analytics eliminates existing storage inefficiencies. Adoption of these stacks, which is entirely reliant on open source platforms, enables analytics to be performed incrementally and in memory, while also avoiding the need for large hardware investments.

Today, there are few excuses left for businesses not to adopt real time analytical capabilities. While the insights generated by analysing historical data can help businesses arrive at strategic decisions, real time analytics produce a different kind of insight. They are the triggers that enable business to tweak processes and refine experiences.


Healthcare and Health Insurance: Big Data’s Sweet Spot


Nowhere is the utility of big data greater than in healthcare, and by inference, health insurance. Venture capitalists appear to have caught on to this, and in the first quarter of 2016, put USD 1.4 billion into the hands of healthcare IT companies. This was almost a third more than in the previous quarter, with much of it directed towards wearables, data analytics and telemedicine.

#1: The Cost Challenge

Three key trends are driving the big data revolution in healthcare. The most important of these is the desperate need, in a world that is rapidly aging, to stem ever-rising healthcare costs. This is a need shared by all healthcare stakeholders, from consumers and providers, to insurers and government agencies.

While cost saving is not a new item on the global healthcare agenda, the recent spotlight on “value-for-money” health services is in part the result of the U.S.’s Affordable Care Act. Faced with ACA requirements, “merger mania” has broken out among healthcare insurers, according to the PwC Health Research Institute.

It would be fair to say that these proposed insurance mega-mergers have been met with some scepticism. Opponents fear that the much-touted economies of scale may not materialise. However, proponents and opponents alike agree that insurance company mergers do offer a key opportunity: to optimise their IT infrastructure, and leverage on innovations in mobile health applications and analytics.

#2: Monitoring on the move

Fitness Coach USA Running

This brings us to the second unmissable trend – the rise of activity trackers and medical devices. Increasingly, health monitoring is shifting from the hospital and clinic to the home and on-the-go. Given that the management of chronic diseases is already the largest component of U.S. healthcare spending, Goldman Sachs suggests that remote health monitoring is “at the bulls-eye of the healthcare cost challenge”.

But the power of health monitoring devices is not just the ability to save money. Health and fitness monitoring devices are enjoying a huge surge in popularity. According to IDC, 78.1 million wearables were shipped in 2015, up from 28.8 million the previous year. PwC’s recent survey suggests that one in three American consumers now have at least one medical, health or fitness app on their mobile devices.

What this means is that the healthcare industry is no longer just reliant on clinical data. Health-related digital data is now (potentially) on tap from medical and fitness devices. The question is whether the data is being put to good use. To date, many healthcare and device providers have been reticent to do so, amid concerns that personal data sharing would be unpalatable to patients and users. The PwC survey appears to lay this to rest, with 83% of respondents indicating their willingness to share their data to help their own diagnosis and treatment, and 73% to help others.

#3: Prevention not cure


This level of acceptance may stem from the already signifiant strides in healthcare analytics, particularly in the area of patient risk forecasting. For example, by analysing patients’ everyday behaviours, socio-economic factors, and clinical updates, hospitals are now predicting and where possible, preventing, their post-operative health deterioration. It is perhaps not surprising that, in a recent survey, advanced analytics was found to be more prevalent among mid-sized U.S. hospitals, ie those between 100 – 500 beds, where there is a strong incentive to minimise patient re-admissions.

Furthermore, in the last few years, healthcare analytics have expanded to include healthy individuals with specific health vulnerabilities. Individuals at risk are offered wearables that track their vital statistics and surroundings, and analytics help primary care doctors to determine if a change in preventive medication or lifestyle is required.

Then there are the individuals who are tracked and rewarded simply for staying healthy. Employers, partnering with insurance players like United Health, Kaiser Foundation Group, Humana Group and Aetna, now account for the fastest sales of wearables. By agreeing to have their activity levels monitored and analysed, employees are able to lower their health insurance premiums as part of so-called “corporate wellness programs”.

A data-hungry future


Given the pace of adoption of wearables and healthcare analytics, there is an appetite for data that may well grow to point of becoming insatiable. Already, insurance companies are said to be busy approaching app developers in an attempt to directly access their fitness data.

By applying sophisticated analytics to large and varied datasets, insurers are hoping to refine their insurance plans. On the horizon looms a cut-throat health insurance marketplace, with new players such as hospitals and primary healthcare providers joining the fray. To stay competitive, health plans could evolve from annual and generalised premiums to daily-adjusted, personalised rates, based on an individual’s real time fitness data. This data will be generated by devices that insurers will offer at discounted rates or for free, further elevating data volumes and the performance of healthcare analytics.

Can you think of a more compelling virtuous cycle?

Are You Open To Open Source?


Navin Suri, Percipient’s CEO, gives six compelling reasons why organisations should leverage open source software

Copy of a copy of a copy…

Already faced with crippling data storage bills, the last thing enterprises need is to waste money on storing multiple copies of the same data

In a ground-breaking study by Veritas Technologies published last month, it was found that on average, 41% of an enterprise’s data is “stale”, that is, has not been modified in the past three years. The study estimates that, per enterprise, this amounts to as much as USD 20.5 million of additional data management costs that could be otherwise saved.

Orphans on the rise

For some industries, for example banking, a portion of this data is stored to meet regulatory requirements. But according to Veritas, the majority is just the result of a passive approach to data storage. For example, “orphaned data”, ie data without an associated owner because of personnel changes, is a particular culprit. Not only is such data on the rise, but comprises of presentations and images that take up a disproportionate share of disk space, and is unattended for even longer than other stale data.

Veritas’s report highlights the fact that data growth, estimated to be as much as 39% p.a. in the US, is caused both by the number of files stored, and the doubling of average file sizes in the past decade. Veritas stresses the need for enterprises to prioritise how they manage their data, that is, what to store, archive or delete.

Needless duplication

What the report fails to mention however is how much of this stale data is actually the result of multiple data duplicates. In large organisations that house several functional departments, analytics teams and data warehouses, it is common for one team to copy already-copied data, which is then copied again for a different purpose. Copies are also made where back-ups are required, further exacerbating storage costs, error risks and the risk of contravening data security laws.

Ironically, many big data management solutions are actually contributing to the problem. For example, in order to combine data stored in a traditional EDW with that moved to a Hadoop data lake, the data is often re-copied and re-stored into expensive solution-specific servers. It is also not unusual for whole databases to be copied in order to query only a limited number of datasets.

Painful lessons

In fact, there could be as many as 30 copies of the same data within a single organisation, according to Iain Chidley, GM at Software company, Delphix. He warns that enterprises can end up storing ten times more data than they had orginally anticipated. While cloud storage is now a cheaper alternative to traditional data warehouses, substantial amounts of money is still being wasted if all this duplicated data is moved wholesale into the cloud.

Clearly, major clean-ups are called for in many large companies. But avoiding the copying process wherever possible can help drastically limit further storage challenges. And for those not yet facing these challenges, prevention is of course better than cure.

RAMp it up

The solution lies in in-memory processing, which eliminates the need to write data to physical disks, track copies, and delete them when no longer required. Instead, data is processed in a computer’s memory or RAM, which has seen price declines of over 200% in the past three years. Meanwhile, newer 64-bit operating systems now offer one terabyte or more of addressable memory, potentially allowing for an entire data warehouse to be cached.

Conducting analytics in-memory also reduces the need for some ETL processes, such as data indexing and storing pre-aggregated data in aggregate tables, thereby further reducing IT costs. The application of the revolutionary open source software, Apache Spark, which supports in-memory computing, is yet another step towards low cost big data processing and analytics.

These technological advances make it possible for enterprises to embrace a vast uptick in the amount of data that can be accessed and analysed, while at the same time making substantial storage cost savings.

Who says you can’t have your cake and eat it too?


Happy Birthday, Hadoop

Ten years on, are big data technologies finally taking root in Asia?

Hadoop celebrated its 10th birthday last week. Back in 2006, Doug Cutting joined Yahoo, and brought with him the development work he was doing on the Google File System and MapReduce. The Yahoo team subsequently launched Hadoop as an open source Apache Software Foundation project, and the rest is history.

In the ten years since, the advent of Hadoop has spawned hundreds of big data hardware and software products and services vendors. These contributed to a global industry that hit USD 38.4 billion of revenues in 2015, according to market researcher Wikibon

Never say never

So how has Asia responded to the lure of big data? Just a few years ago, when asked about the big data potential, the common responses from many Asian executives were, “we will never accept open source”, or “we will never go on the cloud”.

Today a different picture is emerging. A January 2016 report  by the Economist Intelligence Unit found that 63% of Asian executives believe their organisations are already generating revenue from the data they own, ahead of the survey’s US and European respondents (58% and 56% respectively).

Patchy use

However, only about half of those polled see data as ‘vital’, that is, used by almost all parts of their business, compared to 69% of US firms. This may account for IDC’s relatively modest forecast for the big data market in Asia – USD 3.6 billion by 2018 – a small fraction of the global number.

Studies have put Asia’s more tentative big data adoption down to four key factors: the lack of firm evidence that big data analytics pays off, the technological and skills challenges of leveraging big data, the inability to align existing processes to the new analytical output, and finally, the tight personal data regulations.

Put to the test

More recent evidence suggests that these barriers are slowly eroding. With many Asian firms now able to draw on at least six months’ experience of big data application, success stories have abounded.

Take for example the Philippines Long Distance Telephone Company (PLDT). Since June 2015, the company has been analysing network and subscriber calling activities, and have used the results to improve their capacity planning and advertising placements. Several other banks and telcos in the region have similar stories to tell. In fact, 94% of all users in a recent global Accenture survey said that their big data implementation was meeting their needs.

The rise in the number of big data vendors in Asia, and the 4,000-strong sold-out attendence at December’s first ever Asian Hadoop-Strata conference also signal changing times for big data adoption in Asia. Given the red hot speed of development within the Apache open source world, firms in the region that have yet to embrace big data can still catch up to their competitors, by leap frogging into newer technologies. It is never too late.

Big Data 3.0: Delivering on the Promise

There is unusual consensus among economists that 2016 will be a more-of-the-same year of moderate growth, low inflation and emerging market risks. Against this unspectacular backdrop, how do corporations achieve spectacular outperformance?

One clear weapon of choice is big data. Last year, the world received further affirmation of big data’s real world applicability. The largest ever deal in IT industry history – courtesy of Michael Dell and data storage provider, EMC – was just icing on the cake.

Here is a recap of big data’s big achievements in 2015, and what lies in store for 2016:

Managed Healthcare

 percipent cartoons 2a_1

Most heartening for big data proponents were the strides made in healthcare in 2015. In June for example, Forbes reported on the way data is now being collected from every US cancer patient in order to detect patterns, not just of the emergence of the disease, but also of patients’ responses to treatment.

In 2016, the focus looks set to shift from battling illnesses to maintaining good health. Biometric data from fitness apps and IoT-based wearable healthcare devices is likely to come to the fore. The challenge to date has been the technical and intellectual capabilities required to make sense of this overwhelming amount of data. As the Internet-of-Things comes of age, the opportunities for innovation will fall to those able to master AoT – the Analytics of Things.

Machine Learning

 percipent cartoons 4a_1

Machine learning was hard to ignore in 2015. Algorithms moved from a term familiar only to maths geeks to one used in the trendiest of places. Last month, fashion retailer, The North Face, launched its virtual personal shopper service, providing style trends, fashion recommendations and practical travel advice, all powered by a big data machine learning platform.

However, machine learning in 2016 may take on a different flavour. Rather than replacing humans, machine learning is moving to the incorporation of human intervention. “Human-in-the-loop” computing is based on a continual cycle of 1. ascertaining an algorithm’s ability to solve a problem, 2. seeking human input where necessary, and 3. feeding this back into the model. A range of automated services from self driving cars to natural language processing (NLP) will likely continue to rely on humans teaching machines a thing or two.

Dynamic Pricing

percipent cartoons 1a_1 

Talk advanced retail pricing and you are likely to think of price-matching. Some of the biggest names in retail have relied on price-matching strategies to build customer loyalty. However, Walmart took the concept to a new level in 2015 with its Savings Catcher mobile app, which allows shoppers to scan their receipts and claim for items more expensive than at competitor stores. This feature alone is credited with taking Walmart’s app usage from 5 million to 22 million in just one year.

That said, the evolution of big data use is paving the way for retailers to improve their margins. In 2016, rather than the intense focus on price, big data will offer retailers the ability to price based on a variety of highly dynamic and complex factors, including pop-up trends, time of day, weather conditions, underperforming products, etc. Amazon, for example, is known to change prices up to 2.5 million times a day during the busy holiday season. According to IDC, by the end of 2016, product intelligence will inform 90% of pricing decisions made by the top 10 US e-Commerce retailers.

Chief Data Officers

 percipent cartoons 3a_1

A survey conducted in 2015 found that 61% of CIOs wanted to hire chief data officer (CDOs) within the next 12 months. This suggests that in 2016, CDOs will be a significantly more common sight on company payrolls than at present.

Although the CDO designation has been in place in some US banks since the 2008 financial crisis to oversee data quality, the role in its current form has only surfaced in the last couple of years. In 2014, about 25% of Fortune 500 companies claimed to have one appointed to optimise their use of data.

Despite this, a study by IBM suggests that many CDO employers are not clear what this role is all about. Furthermore, while the growing importance of big data will see CDOs gaining more power in 2016, research firm Forrester note in an intriguing report that this power may be eroded by an increased trend to outsource data solutioning, and to succumb to data security concerns.

Big Data 3.0

 Forrester’s report also notes that 2016 may see on the end of big data’s honeymoon. The report warns that big data tools alone will not be enough, if not accompanied by enterprise-wide commitment and visionary management.

If Big Data 1.0 was the era of discovery, and Big Data 2.0 was marked by a period of experimentation and application, then what we see unfolding in 2016 must surely be Big Data 3.0, when tangible results will be key. All I can say is, bring it on!

What airlines can teach banks about big data

In the topsy turvy world of big data, it has become apparent that some industries are unexpectedly lagging behind others in their data mining capabilities.

For example, one would be justified to assume that financial companies, given their access to vast amounts of customer profile and transaction data, are leaders in this field. However, the industry can be divided into two camps.

On the one hand are the traditional global banks and insurance firms, which continue to lumber under a mountain of new, legacy, proprietary, departmental and country data systems that do not communicate with each other.

A good example of this is Deutsche Bank, whose CEO, John Cryan, noted at a recent press conference that, “…about 35 percent of the hardware in the data centers has come close to the end of its lifecycles or is already beyond that”. Jost Hopperman, vice president of banking and applications and architecture at Forrester Research, claims that the problem is widespread, “The situation at Deutsche is not unusual at many other large tier 1 or tier 2 banks simply because many of them have similar issues, including too many silos, too fragmented outsourcing, and lost control of applications”.

On the other hand are the new and nimble digital-only banks, the likes of US-based Ally Bank and UK-based Atom Bank. The latter, even before its launch, targetted for early-2016, has received a 45 million pound investment from Spanish bank BBVA. According to Atom Bank CEO, Mark Mullen, the bank aims to be as disruptive to banking as Uber has been to taxis.

Assuming that brick and mortar banks are able to overcome their data woes and to unify the volumous data currently sitting untapped, then opportunities abound for to them to catch up to their retail and online counterparts.

Top of their list of priorities must be the use of data to achieve a meaningful 360-view of the customer, as a necessary step towards re-imagining the customer experience. While the airlines industry is not obvious as a trend-setter in this field, a number of airlines have started to take customer-360 initiatives to heart.

Southwest Airlines, British Airways and American Airlines, for example, all employ mobile apps to keep frontline staff informed in real time of individual customer circumstances, whether this be food preferences, flight delays, or complaints on social media. These airlines, through a process of trial and error, have also determined the limits of communicating personal data (favourite drinks are okay, pets’ names are not). This “know me on the go” approach is one that could easily be adopted by bank tellers and advisors when dealing with their customers, but remains sadly lacking.

There are other transferable practices. In June this year, Qantas launched a programme for frequent flyers, rewarding them with extra points in exchange for information on their consumer habits. This data is used not only within the airline but is shared (in anonymised form) with partner organisations, for example hotels and car rental services. Banks’ credit card businesses have the opportunity to do the same, and to share this information with their merchants, but again, this is not yet common practice.

Finally, in this age of algorithms, banks, like airlines, can do more to target products to specific customer types. United Airlines has sought to connect together its 3.5 petabytes of customer data in order to use it more effectively. For example, according to executive vice president of marketing, technology and strategy, Jeffrey Foland, sales of United’s economy-plus seats have surged since the airline started using data to target fliers who are more likely to buy them.

This kind of sophisticated customer analytics comes on the back of well-rounded data on an individual, linking together his/her buying patterns with his/her demographics, personal preferences and lifestyle. Many banks have made strong strides in their “digital revolution” journey. But only those able to capture customer data as a consolidated whole are likely to enjoy the customer loyalty and ROI results they had hoped for.