The Blog

Big data is having a profound social impact

Big data is having a profound social impact

If you’ve ever interacted with me you’ll know that I’m passionate about how big data can deliver competitive advantage to organisations.  I believe that organisations that leverage big data analytics and adopt a data driven culture will outperform their peers.

However in this post, I’d like to share how big data is having a profound social impact in our community.  Here are a couple examples that I’ve found interesting.

Aadhaar Project in India

In India, there is a problem in that a sizeable proportion of aid does not reach the intended recipients due to corruption and fraud.  Claims made by “ghost citizens” deny the target recipients from receiving much needed assistance.  With the majority of the aid recipients having limited forms of identification, this problem needs to be solved.

The Aadhaar project operated by the Indian government is in the process of issuing unique identifiers to all its citizens.  The system stores biometric data that include iris scans of both eyes and all fingerprints.  When making a claim, a person can be authenticated in real time that ensures aid is reaching the intended recipients.  The project is making excellent progress with over 925 million identification numbers issued representing more than 76 percent of the population.  This is the largest citizen authentication system in the world and it has been facilitated by modern big data technologies.

Hadoop is the big data technology that underpins this project.  It has allowed this project to be delivered in a far more cost effective manner than using traditional technologies.  I’d like to congratulate our team members in India for the pivotal role they played on this project.

Crime Prevention

When a violent crime occurs, we learn in hindsight factors that may have indicated the likelihood of the offender committing a crime.  Surely with big data analytics we should be able to predict the likelihood of an offender committing a crime?  I’m pleased to report that there has been significant progress in this area.

One example is the Harvard Medical School working with the US military to analyse 975,057 soldiers who served between 2004 and 2009.  Sourcing data from 38 different data sources, they identified a ‘high risk’ group that represented only 5 percent of the population.

This ‘high risk’ group accounted for 36% of the crimes committed by men and 33% of the crimes committed by women.  The study identified 24 different factors that may indicate a crime being committed in the future.  This insight could direct resources in the form of counselling or training to this ‘high risk’ group.  Even if a small proportion of these crimes can be avoided, the return on investment is significant.

These are just a couple of examples of how big data is having a profound social impact in our community.  If you have some ideas on how big data could be applied to social causes, please comment below.  I know many people in the big data space that want to be involved in these types of projects.


Why Big Data rocks!

Why Big Data rocks!

In this post, I’d just like to share with you why I’m really excited about Big Data solutions. In a previous role, I was the head of an IT division for a successful finance company. This was a few years ago but even back then, we knew that analytics was critical to increase revenue and profit for the company. We needed to consolidate our existing data warehousing systems and add further analytic capability to our technology stack.

I got a couple of vendors in to quote a solution. After I received the quotes back, I was surprised at how expensive the ‘Phase One’ solutions were. The solutions required expensive, high performant and resilient hardware to operate on. Even though the licensing costs were expensive, it was industry standard at the time. But what caught my attention was the professional services and what that entailed.

Legacy business intelligence solutions required developers to code a ‘extract, transform and load’ (ETL) process. This translated data from various sources into a structured format that business intelligence technologies could work with. This ETL phase in the project was expensive. However, this ‘infrastructure’ was required before any real value could be delivered to the business. If business insight and evidence based decisions are critical to modern organisations, why was was this so expensive?

The solution is with Big Data technologies that were developed by leading Silicon Valley companies such as Google, Yahoo and Amazon. Although these technologies can scale to the incredible requirements of these companies, they also provide an extremely cost effective solution to many Small to Medium Enterprises.

Business insight can now be delivered to all organisations regardless of their size and budget. These solutions scale extremely well from one commodity server either on premise or cloud to the thousands of servers. Licensing of the technology stack ranges from open source to cost effective vendor licensing that includes enterprise level support. What I find really exciting is the ability to work with all types of data in its raw form that avoids the expensive ‘ETL’ process mentioned above.

We are excited about how these solutions can deliver real competitive advantage to your organisations for a fraction of the cost that you may expect.

Please do contact us to discuss your particular requirements.

Thanks for reading the post.


Welcome to Ember!

Welcome to Ember!

Welcome to Ember!  We are excited about how we can help your organisation gain or sustain competitive advantage in your industry.

As outlined on our website, we focus on two key areas being digital transformation and big data solutions.  We believe that these areas will be critical in ensuring organisations are successful.  I’ll leave it up to you to have a look at our website to see how we can add value to your organisation.

We look forward to working with you and your organisations in the near future!


Gaj Varma