Readying for Sri Lanka’s Big Data Deluge

As the economy takes off, the volume of data will explode. Companies preparing for this deluge will have clear advantages. But small companies and start-ups that don’t have the expertise, nor the money, to harness Big Data may still ride the wave over their larger competitors.

Sri Lanka is yet to encounter Big Data but it will happen soon. Many companies don’t appreciate the data they do generate or have access to. Data analytics here is an out-layer industry. It’s only a matter of time till the floodgates open. The early birds will have a clear advantage like the banks, telcos, insurance and retail companies, which already generate and collect large amounts of data. Big Data will also help small businesses which, despite lacking the expertise and the technology, will be able to take on the established giants. Governments will be able to streamline public transport, improve disaster management and healthcare.

Globally, the emergence of Big Data has seen large tech firms such as Oracle Corporation, IBM, Microsoft, SAP, HP and Dell spend over US$15 billion on software companies specializing in data management and analytics. In 2010, this industry was worth more than US$ 100 billion and growing at 10 per cent a year. Data volumes are booming.

In 2000, the telescope commissioned by Sloan Digital Sky Survey in New Mexico collected more astronomical data in its first few weeks than what was collected in the history of astronomy. Ten years later, its archive contained 140 terabytes of data. A new Large Synoptic Survey telescope to be launched in Chile next year is expected to capture this much data every five days.

33US retail giant Wal-Mart records over one million customer transactions every hour. Facebook had reached 40 billion photos in 2010. Decoding the human genome involved thousands of scientists analysing three billion base pairs over ten years costing $3 billion. In 2010, it cost $50,000 and took under a week to decode. Speeds and volumes keep growing, growing and growing. According to a Gartner survey of over 400 business leaders, the number of Big Data projects has increased to 76 percent in 2015 from 73 percent in 2014 as business leaders demand more from their chief information officers to align Big Data with company finance and marketing strategies.

Big Data analytics helped Obama and Modi win elections in the USA and India and the global debate is around how to use Big Data for good and efficient governance and planning while preventing governments (and even corporates) from playing the big brother.

With Big Data come big responsibilities and big challenges. Companies could wear themselves out without knowing what data to use and how; like the proverbial fox which fell victim to the hounds because it spent so much time weighing its many escape options. Governments and companies increasingly gather personal data from mobile phone and credit card usage blurring the lines of confidentiality. The developed world has been debating and grappling with these issues around Big Data for more than a decade.

Echelon had a round table discussion on the possibilities, opportunities and challenges around Big Data for Sri Lanka with Sohan Dharmaraja Ph.D, Chief Executive Neotenicity, Ranjiva Munasinghe Ph.D, Managing Director Argyle X, Srinath Perera Ph.D, Vice President Research WSO2 Inc., and Sankha Muthu Poruthotage Ph.D, co-Founder Linear2. Excepts of the discussion:

What are the businesses and sectors that will have new opportunities opened up for them as a result of Big Data applications?
Sankha: Firstly, Big Data is a wide variety of data gathered from a wide range of sources at a very high velocity. Think of social media data. It has a great variety: pictures, videos and text, and accumulated at a great velocity. Secondly, Big Data entails a wide range of data analytical techniques to process and make sense of this data. These methods range from some of the classical statistical techniques such as regression models to very modern machine learning type techniques such as neural networks. Thirdly, Big Data also refers to technological advancements in distributed file sharing systems, cloud based data warehousing and processing systems, commodity hardware based warehousing and processing systems. Advancement on any of these three aspects can be regarded as a Big Data initiative.

Sri Lanka is behind the Big Data and analytics curve, we will catch on fast. The big industries like retail, banking and finance will lead the way. Once the flood gates are open we may see some of the Big Data applications being adopted in Sri Lanka faster than in other countries

Any industry generates a lot of data, just by doing business. A Big Data initiative could be as simple as asking the customer where he or she would like to vacation in, or asking to share a vacation picture, to investing in a cloud based data warehousing system and implementing a real time pricing algorithm. The definition of Big Data itself lends to scalability. So any industry can benefit. All that is needed is the conviction that data can be used to do business better.

The tourism industry will benefit from Big Data analytics with better targeting. Shouldn’t we promote our beaches more to a certain profile while promoting ancient sites to some other profile? Using social media and internet we can devise a system to do so. Hotels can aggressively use data to align their services with their business proposition.

The manufacturing industry can use data to make the processes more efficient and cost effective. An example would be predicting machine breakdowns in advance by constantly monitoring and analysing their performances.

The main advantage of Big Data is that it provides a platform for small businesses to think big – and punch above their weight. Smart and intelligent implementation of Big Data ideas can yield results in quick time

The financial sector is primed to reap rewards of the Big Data era. The insurance sector can identify bad risks by analysing historical data and use it as an integral part of pricing. Banking and leasing companies can predict likelihood of default and make lending decisions accordingly. These initiatives will make insurance costs and lending costs lower and the entire economy will benefit.

What about other industries such as fisheries? Can we improve the output by analysing fish movements? What about spice exports? Do we have demand forecasts? Does it make sense to aggressively grow that sector of agriculture? Big Data can make all this possible.

gThe public sector can use data to devise smarter systems, and bring relief almost immediately to the general public. A positive example is dengue eradication. The health sector can use data to analyse time and geographical characteristics of diseases and prevent outbreaks. We can make our public transportation system smarter.

Ranjiva: Businesses today have the ability to generate, store and analyse a lot more useful data than ever before. If they don’t do this, they will lose their competitive edge.

Larger the statistical sample the more accurate the forecast. Businesses can do much better with more data. Having a database of one million loans will enable a bank to make better lending decisions than having a database of 50,000 loans.

Banking, finance, retail and telecommunications are some of the industries that have been using analytics on the data they generate. A telco will see the signs of a customer thinking of switching providers based on their usage patterns and it could take action to retain the customer. If it can keep a significant number of potential leavers from switching that’s a massive saving. In retail, analysing granular loyalty card data at the customer level and their purchase basket can help the retailer up-sell and cross-sell products to customers with similar profiles. So instead of ‘sms-bombing’ everyone on their loyalty program will learn about a deal on Levi’s jeans. They would only send it to people more likely to buy Levi’s jeans. This profiling is possible because of analytics.

Globally, because of the Internet of Things, we are seeing a lot of information being captured. There are wearable devices that monitor pulse and heart-rates and apps that help with medical diagnosis. Many countries in Europe have telematics based motor insurance where good drivers pay lower premiums. The insurance company benefits by having a lower risk portfolio. HR departments are also starting to use analytics to retain good staff and help identify candidates that would make good employees and less likely to leave.

My feeling overall is that while Sri Lanka is behind the Big Data and analytics curve, we will catch on fast. The big industries like retail, banking and finance will lead the way. Once the flood gates are open we may see some of the Big Data applications being adopted in Sri Lanka faster than in other countries.

Srinath: Super markets can use analytics to understand what products sell well, what does not, and under what conditions, and optimize what items to keep in the store and on what and when to give deals.

The term Big Data can be intimidating, but everyone has data. I think businesses need to first use what data they have to gain a competitive edge, maybe by improving their services. There are many insights to be gained by looking at your own data

Companies can use analytics and Big Data to monitor what customers are saying about them in social media. Analytics can be used on various data to optimize marketing strategies and to find out whether marketing campaigns are effective. It can help predict and prevent customer churn, optimize the sales pipeline and analyse competition.

Big Data can help producers save time and money. If a company generates a lot of customer data, they can build profiles and sell the insights to other companies. There will be opportunities around investments, asset and risk allocations. Banks and insurance companies can price their services by using demographics, for example. Fraud detection and prevention in organisations is another opportunity Big Data analytics can bring.

Start-ups have huge opportunities in providing services around Big Data analytics.

Sohan: The obvious sectors that would benefit from Big Data are telcos, banks, retail, healthcare, and insurance. But there are other sectors too that are not so obvious. Companies can use Big Data analytics for human resource management to drive recruiting, staffing and employee development. Non-profit organisations tend to have large amounts of customer interactions. Services and utilities can use Big Data to minimise failures and downtime, and for smart metering. Travel and hospitality is another sector that will gain big. The education system is going to change from the traditional university system to a more targeted, vocational training system. It is in the forefront now and we will have Data Scientists and turn out more hackers, not software developers: people with a mathematical, statistical and business appreciation.

There will be opportunities around investments, asset and risk allocations. Banks and insurance companies can price their services by using demographics, for example. Fraud detection and prevention in organisations is another opportunity Big Data analytics can bring

It is also going to be industries with large customer centric focus, using analytics to lend towards personalisation. Amazon has been at it for a long time. An industry with a large number of personalisation will benefit, for example in sports attaching sensors to people can help monitor performance and predict injuries.

With the Internet of Things being more pervasive in the housewhold and everywhere, there is more and more data you can inter connect, like how a person drives and shops. People have come to the point where they expect solutions to be beautiful, elegant, intuitive and personalized and any customer facing interface, be it an app, insurance pricing or hotel service or marketing message can leverage that.

What should companies do right now to leverage the opportunities that will arise out of Big Data?
sfvSrinath: A company’s ultimate competitive advantage is its data. If a company generates data and does not use it, they are losing opportunities. Telcos are sitting on a gold mine of data. They know who I called, how I used the internet and where I have been. They cannot release this data because of privacy concerns, but at an aggregate level they could sell it to companies for targeted marketing campaigns.

Sankha: The term Big Data can be intimidating, but everyone has data. I think businesses need to first use what data they have to gain a competitive edge, maybe by improving their services. There are many insights to be gained by looking at your own data.

Ranjiva: Companies individually and collectively generate a lot of data so the challenge is to find a usefulness for the data. This can be done by looking at the strategic objectives of the company and what data can be used to identify and exploit opportunities.

Sohan: Companies need to really look at their business models and reengineer themselves as more and more opportunities open up. They need to innovate and figure out new revenue streams as well. The Ma-and-Pa shop will not have large volumes of data but their data can still be valuable. Businesses, whatever their size, need to go through their data and figure out what will be of value to it or someone else. Businesses need to lookout for such opportunities.

Small business, how will Big Data help them?
Srinath: A small company may not generate a large volume of data but it can improve its processes by looking at its own data and data scientists may assist them to look at other open data sources for insights on how the business can benefit. Start small.

Any business will need to have an interactive front like a website or even a Facebook page. Try to understand what’s going on there and understand the consumer. Start-ups can also start-up as big data companies. Big Data is changing the status quo and can give small and medium businesses more access to the market and help them grab a share of the pie from the big players.

With the Internet of Things being more pervasive in the household and everywhere, there is more and more data you can inter connect, like how a person drives and shops. People have come to the point where they expect solutions to be beautiful, elegant, intuitive and personalized and any customer facing interface, be it an app, insurance pricing or hotel service or marketing message can leverage that

Sohan: Small businesses cannot afford the technology and data bases. Volume is just one aspect. Variety and speed are other aspects of Big Data and small businesses may be able to leverage on the brokered data. There is a lot of aggregate data out there which they may not be able to process themselves but there are Big Data companies that can help. Analytics for the masses will also be possible with cloud based solutions benefitting small business that will never be able to afford Oracle, MS-SQL, let alone staff for this requirement.

Srinath: Most analysis is available for free on the internet and you need to only spend on consultants. There maybe a point when you have to spend on analytic tools, but it’s worthwhile to understand what is available out there.

Sankha: Back in the old days you had to commission a market research based on face-to-face interviews and questionnaires, but now with the internet and social media, data collecting is no longer an expensive exercise.

The main advantage of Big Data is that it provides a platform for small businesses to think big – and punch above their weight. Smart and intelligent implementation of Big Data ideas can yield results in quick time. Organizations should not think of Big Data initiatives as R&D (research and development) projects. They are operational level initiatives which essentially enable them to do what they are already doing – only better.

dsIt is never too late to start. The amount of data that organizations have will only grow. The core concepts of Big Data—data, analytics and technology—will always be there, although it may be called by some other name later on. Big Data definitely does not mean ‘start big by investing big’. The most important message would be to start: start somewhere, start small.

Big Data has its challenges, especially on ethics. What kind of policy front does Sri Lanka need to put in place?

Srinath: On the privacy and monetizing front the legality is not very clear. Clarity on that may improve things. There may be a risk of data fraud and identity theft. So we need strict guidelines and policy and these are on-going debates worldwide and Sri Lanka will take time to settle.

Sankha: Government should mandate data collection in certain industries. For example, there is a tangible benefit if you can identify good drivers. Credit risk is another example; bringing down the cost of credit and making credit more accessible to people will be possible with Big Data. The government needs to be proactive. Around 50% of insurance companies don’t have national ID numbers for all their insurance policy holders. All they have is the vehicle number, so we are all contributing to bad driving.

Ranjiva: It is a very fine line. When you share your data and get sms and email blasts, you can deal with them, but where there is personal and confidential data there can be problems. There has to be some government intervention to ensure data protection parameters are in place.

Sohan: Policy to make data secure is crucial because if there is one slip-up, everybody suffers.