Unlocking the potential of IoT with the intelligent edge

Stephen Goldberg
© Shutterstock / Dario Lo Presti

Often folks working on IoT projects, whether at new startups or at more mature companies, do not feel empowered to tackle Big Data architectures from day one. In this article, Stephen Goldberg explains how to unlock the potential of IoT.

Big Data: A back-burner discussion?

One of the things we encounter most when speaking to IoT developers is, while they are often focused on getting their product to market, they aren’t thinking long-term about Big Data strategy.

This is for two primary reasons. Often the lead technical resource at new IoT companies has a background in hardware, electrical engineering, robotics, etc. This makes a lot of sense as getting an IoT product, like a robot that picks strawberries or a sensor that monitors vaccine temperature, up and running requires a strong knowledge of hardware and electrical engineering. That said, while often able to code, these folks don’t generally have a background in Big Data architectures and aren’t knowledgeable enough to ensure they have a future-proof, intelligent edge strategy.

Secondly, time to market is very important for startups. They are racing the clock on funding, want to capture their market before their competitors, and have limited resources. As a result, when discussing Big Data architectures with new IoT companies or companies tackling new IoT initiatives, it often becomes a back-burner discussion. “We will tackle that later after we get to market.” In fact, I have even had conversations with CTO’s at IoT companies who are well aware their data infrastructure simply will not support their scale. They are experts in hardware and highly focused on getting their product to market, however, it’s apparent to most outside observers that the data they collect may be more valuable than the function their IoT product enables.

SEE ALSO: IoT for web developers

Often folks working on IoT projects, whether at new startups or at more mature companies, do not feel empowered to tackle Big Data architectures from day one. This isn’t surprising as the accepted status quo for Big Data architectures is incredibly daunting and unnecessarily complex.

Eventually, these products mature, gain user adoption, and are deployed widely. At the edge, often a caching layer or a “lite” version of a database product is deployed. This results in a less than “intelligent” edge and an industry-wide problem where 99 percent of IoT data goes unused.

The true promise of IoT

Currently, the true promise of IoT, allowing for real-time actionability of IoT sensor feedback is not in place. This is due to a lack of feedback loop between IoT sensor data and machine learning and predictive analytics. Imagine a facility that monitors quality control of baked goods using image recognition. They analyze these images historically and look for quality issues. When an issue is identified, they check the historical data and reactively fix the problem, however this could be days, weeks, or months later.

We are seeing this trend because lite versions of products, or caching mechanisms, require data to be moved from the edge to the cloud for analysis which results in time delays, data loss,  complexity, and massive cloud infrastructure costs.

Now, imagine the same facility reacting to quality issues in real-time by leveraging video recognition technology, but combined with an intelligent edge and a Big Data strategy that harnesses the power of the edge. This saves a lot of time and money and ultimately results in happier customers.

Cloud services can be excellent resources and have a place and purpose. Providers like Amazon, Microsoft, Google, Alibaba Cloud and many more are rolling out incredible services for IoT, however real-time reaction to real-world problems needs to happen on the edge, not in the cloud. Cloud services are targeted at DevOps and highly experienced back-end developers while IoT is being built by a different group of people. As mentioned above, these are electrical engineers and hardware folks whose primary expertise lies elsewhere.

In order to move the needle from 99 percent of IoT data waste, data management solutions need to be targeted at folks building IoT solutions. They need to empower them to tackle Big Data challenges without requiring massive budgets or overly complex solutions. These solutions also need to include enterprise-grade database capabilities directly on the edge allowing for things like real-time distributed querying, peer-to-peer clustering and replication of IoT devices at the database level, and HTAP models which can span from the edge to the cloud.

SEE ALSO: IoT for web developers: From zero to firmware

These features are important because, while cloud services are perfect for historical analysis of large data sets, they are too far removed from the edge to enable real-time actionability on IoT data – the real promise of IoT. The lite versions of products lack the enterprise-grade capability that we rely on to make sense of data. Without peer-to-peer device replication, it is impossible to make decisions across a fleet of IoT devices without first centralizing that data into the cloud.

This need, forced by lite versions of products and caching mechanisms, creates time delays and data loss as synchronizing millions or billions of rows of data into a centralized cloud repository is restrained by intermittent connectivity, throughput constraints, cost, and network latency. Additionally, the need to use one product on the edge and a different product in the cloud creates complexity, delay, and multi-points of failure.

Why a more hybrid cloud strategy makes sense

In the future, companies that win the war for IoT market share will adopt a more hybrid cloud strategy. These pioneers will leverage products that can span from the edge to the cloud holistically without sacrificing features or capability. These investments in IoT devices will allow for distributed computing and real-time reaction, while utilizing cloud services to enable machine learning, predictive analytics and AI. That analysis can then be supplied to an intelligent edge to continue to make better, faster, and more valuable decisions directly on the edge while leveraging the cloud for complex, deeper solutions.


Stephen Goldberg

Stephen Goldberg is the founder and CEO of HarperDB. Stephen previously founded two startups, and was most recently CTO at Phizzle, Inc. managing product, engineering, product marketing, and support. He has worked at companies large and small including Red Hat, Inc. where he led Infrastructure for their Global Support Services division. Stephen has four pending patents, and has been a speaker at both’s Dreamforce as well as SAP’s Sapphire

Inline Feedbacks
View all comments