Big Data and the Internet of Things

Big Data and the Internet of Things

A working mom finds a recipe for her two infant sons, while browsing on her smartphone. She sends a request to her fridge, which compiles a list of ingredients it then pre-orders from the supermarket. Another request, to preheat the oven.

Because her sons hate shopping in the car after kindergarten, she sends a request to her home entertainment system, which selects some favorite cartoons and downloads them to her car's multimedia system. With help from RFID chips sewn into their clothing, she knows to pick the boys up from the playground.

This is The Internet of Things...

Also known as the IoT; a world where everyday objects are connected to the Internet, capable of identifying themselves and communicating with other objects on the network.

The basic IoT infrastructure consists of tracking technology like RFID or bar codes, sensors, embedded software, and wireless Internet connectivity. Transponder nodes attached to physical objects uniquely identify themselves to the Internet, allowing information about these objects to be captured.

The result is a network of "smart objects" that can actively participate in a variety of business processes. And it's building up, already.

...with Big Data Involved

That much connectivity will generate masses of data. The volumes involved won't fit into a standard database, and will require new or augmented methods. IoT will demand the best analytics platforms and tools, to let business users acquire the data they need, analyze its meaning, and act on it quickly.

Information will come from a variety of sensors, attached to a wide range of devices and products. Organizations will seek to use that knowledge for any number of purposes: resource monitoring, tracking usage patterns, delivery of goods and services, etc.

Big Opportunities

IoT technologies will allow for real-time data sensing, with wireless transmission to web applications and Net-connected servers. This should enable more precise monitoring and control of physical systems.

Agricultural companies are already monitoring crops in real time to improve yield quality, and to conserve resources like pesticides, fertilizers, and water. Utility companies have set up smart meters to monitor energy, gas, and water consumption.

Authorities are launching "smart city" projects to help ease traffic congestion, improve waste management, monitor radiation from cellphone towers, and control street lights.

The IoT may play a huge role in areas like supply chain management. Using Internet-connected smart goods, a physical supply chain could be augmented by a digital one, in which monitoring services, content, updates, and other aspects could be provided.

With customer preferences or needs tracked in real time, businesses have the opportunity to react immediately, with options like dynamic messaging, special offers, pricing, or service delivery. Tracking usage patterns could let businesses plan ahead for surges in demand, or quiet periods.

Big Challenges, too

Building the IoT will entail bringing together disparate devices and carrier networks, multiple communication protocols, and a wide variety of applications. At minimum, there's the need to inventory, bar-code, and cross-check every physical object to be brought online. 

All the sensors, smart phones, smart factories, smart grids, smart vehicles, controllers, meters, and other devices will require IPv6 addresses; the IPv4 addressing scheme can't accommodate the data volume. And each technology component will have its own issues. So expertise will be required from various parts of an organization - or from outside resources.

The sheer volume of data from human users and machine-to-machine (M2M) applications will require advanced analytics capable of exploiting Big Data and the computing power of the cloud. A simple sensing and monitoring application for a site with 100 sensors installed and collecting telemetry data might produce raw data totaling more than 4 petabytes (PB) a year.

System design will need to ensure effective, data-driven decision making, and deal with new levels of data granularity.

Ownership is an issue: Who owns the sensor data these Net-enabled things produce? Who can control the device actuators - and under what circumstances? And who does information from the corresponding services belong to?

Unauthorized access in the IoT could not only lead to confidential information being leaked, the owner or authorized user might also lose control of their things in the physical world - so security is very much an issue.

Strategies to Cope

Processing all the IoT data requires data ingestion (the harvesting of data), data storage, and analytics.

Database as a service (DBaaS) solutions and managed service providers (MSPs) may be increasingly called upon to take away much of the underlying complexity, so that engineers can focus on the data. 

For storage, technologies like Hadoop and Map Reduce will be essential in providing enough disk, network, and compute capacity to cope with the influx of new data.

In the analytics phase, it will be necessary to integrate the different technologies underlying an organization's data warehouse, and the IoT data itself. Demand for real time analytics and storing many petabytes of data will require a different server, disk, and network infrastructure than what exists in most data centers today. Budgets will need to factor for this - and additional floor space.

A Final Word...

For now, the computing and software community is focused on exploiting cloud computing, M2M applications, and Big Data solutions. The convergence of these technologies will hopefully provide the capabilities necessary for building next-generation systems, and expanding the Internet of Things.