We’re in the midst of the most ambitious rebuilding of our networks in the past decade. The continued demand for data-intensive video applications combined with a need for seamless connectivity across mobile and fixed devices is driving the redevelopment of our core networks. This rebuilding is focused on flexible architectures that can scale to meet bandwidth requirements as and when required. Yet while video may be the key driver here, people often overlook the raft of applications that these new networks are enabling. In fact, there are some applications that have the potential to radically alter the very way we live.

It isn’t simply the growing availability of bandwidth that’s driving the development of new applications here, but the growing understanding of how collective data can be used to add greater efficiency to our lives. Often referred to as big data, we’re now seeing companies and organisations invest millions of dollars in the necessary tools to ensure they can effectively seize new opportunities to collect, transport and analyse the enormous amounts of data that’s continually being generated.

A great example of this is a company called EcoFactor. Based in California, EcoFactor is using big data to reduce customer’s energy bills and improve their energy efficiency. The company does this by building unique profiles for its customers that focus on their individual energy habits, housing conditions, immediate climate and local weather patterns. In fact, EcoFactor collects thousands of data points per customer to understand what the optimum temperature should be for their house. By building such a tailored profile, EcoFactor can reduce average energy bills by 17%. This equates to annual savings of over $600.

It’s incredible to consider the amount of data that EcoFactor needs to collate, process and then act upon. Indeed, EcoFactor has found that it makes over 1,000 adjustments to a customer’s thermostat per month in response to the data it collects. As EcoFactor rolls out across the US, it’s fascinating to consider the volume of data that will be transported. In an earlier post, I discussed the challenges of big data, specifically in regards to data transport. This is a trend that is growing dramatically each month. More and more data is coming online and needs to be accessed and transported.

In many respects, the growth of big data is matched by the rapid development of the Internet of Things. Indeed, these two technologies are somewhat dependant upon each other to truly succeed. With the Internet of Things, we’re seeing an increasing number of objects come online. These objects range from domestic appliances, such as thermostats and fridges, to vehicles and a wealth of industrial machinery. Once online, these devices can be controlled remotely, talk to each other, share data and, perhaps most importantly, respond to data.

As EcoFactor has demonstrated, it’s the ability of devices to respond to data that makes the Internet of Things such a tantalising opportunity. With this dialogue in place, we’re effectively blurring the line between bits and atoms. We’re moving to an automated world where our connected devices will be able to respond to thousands of data points and assess which course of action to follow given certain parameters. On a domestic level this is exciting, but the impact on businesses could be phenomenal.

Research from McKinsey Global Institute suggests that US healthcare could save $300 billion per year if it was able to harness these new technologies and effectively use the data that it already has. McKinsey also predicts that American retailers could increase their profit margins by 60% if they could respond to big data opportunities. With such clear financial incentives it’s curious to know why companies are so slow to adopt.

Until now, the network was certainly one issue restricting development. Securely transporting enormous amounts of data was a challenge, but this is increasingly becoming less of a concern as we migrate to new core networks. Connectivity was also an issue. Providing thousands upon thousands of devices with access to the network is a complex and costly endeavour. However, low-power WiFi chips are helping here, as is the development of greater wireless connectivity. Putting the technology aside, it appears that the biggest hurdle to overcome is a staffing issue. Big data requires staff who understand how to effectively mine data and transform it into understandable and actionable information. McKinsey estimates that the US alone will need in excess of 1.5 million people to take advantage of the opportunities that big data offers.

Clearly there are lots of challenges ahead, but there are a growing number of companies that are starting to harness the opportunities presented by big data and the Internet of Things. Earlier this week, OPower announced that they could save US homes over one terawatt of energy per year, equal to over $100 billion. Announcements such as this are now happening frequently. This space is moving incredibly fast and I find the opportunities here exciting. We’re on the brink of a new wave of innovation that could have far reaching implications on every aspect of our lives.

What are you seeing in this space? What potential does big data have and what are the challenges that you’re seeing? Also, what opportunities does the Internet of Things present? How close are we to automating many business applications and domestic activities? What do you think? I look forward to hearing your thoughts.