In a January 2017 TechTarget article, Executive Editor Lauren Horwitz wrote that companies are struggling with working with and managing data generated from IoT (Internet of Things) devices. Ms. Horwitz writes:
“While verticals like manufacturing are more business process-driven and have been able to integrate IoT devices and data into their operations, other industries are still struggling with the volume and velocity of the data and how to bring meaning to it.”
The Challenges With IoT Data
Truthfully, Ms. Horwitz is not wrong. The amount of data being produced by the Internet of Things is mind boggling. Business Insider’s BI Intelligence research team released a report in this past August in which they revealed that in 2015, there were roughly 10 billion devices connected to the internet. Granted, that number appears to include traditional smart devices like tablets and phones.
But chew on this: In that same report, BI Intelligence predicts that by 2020 there will be a total of 34 billion devices connected, with 24 Billion of those devices being what we would call IoT devices – the remaining 10 billion being our trusty mobile devices and computers.
Think about that for a moment – at the time of this writing, the current global population is estimated to be a little under 7.5 billion people. So that means by 2020, there will be about three IoT devices for every man, woman, and child on the planet. And every single one of these devices will be pumping out data in some form.
There Is No Standard For IoT Data
One of the inherent problems facing anyone wishing to work with data generated by these devices is that at present, there isn’t a definitive standard to IoT data. It’s all ad hoc. It’s like the Tower of Babel myth but with data instead of languages. The data, at least in it’s native form, is messy.
In Horwitz’s article, she quotes Brent Leary, a principal at CRM Essentials. He says:
“There is a lot of data coming at these companies, from multiple places. They have to figure out, ‘How do we get it all, aggregate it, analyze it — and what are we looking for?’ And you’re trying to do that in as near real time as possible. The technology may be there, but the culture may not be; the processes may not be in place. And that is just as critical to the success of IoT as the technology itself.”
Leary hits the nail on the head. The real value in IoT isn’t just the data, it’s being able to DO something with the data – ideally in real-time. After all, let’s think of a logistics company with a fleet of refrigerated truck which are IoT capable. It wouldn’t do much good to learn that the temperature in the trucks exceeded safe norms a week after the fact. By then, the data is useless, and the loads in question would be losses.
That’s hardly an isolated scenario. A manufacturer would be interested in data which could indicate that a component on an assembly line is nearing failure. An aviation outfit would be wise to monitor critical items on their fleet of aircraft. The potential uses for IoT span these industries as well as healthcare, military, utilities and more. But again, the problem isn’t the hardware – it’s managing the data generated by the Internet of Things.
The data management problem isn’t limited to any specific use case or industry. The problem really is being able to acquire the data, make sense of the data, and then being able to act on what these devices are telling us in real-time. But the 800 pound gorilla in this room remains: “How can we make sense of IoT data?”
The Cinchapi Data Platform
From the moment that Cinchapi founder Jeff Nelson first came up with the concept of Cinchapi, he was keenly aware that working with disparate, or decentralized, data was a growing problem.
Leaving aside IoT for just a moment, as a developer himself, Jeff was constantly spending time doing the tedious data prep and cleanup required in order to understand what aspects of the data in question was relevant, and to learn what relationships might be hidden when working with multiple data sources.
Jeff knew that there had to be a better way, so he began working on developing a platform which could do a number critical things. He wanted a data platform which could work with any source, regardless of schema or structure. He also wanted to find a method to use technology to do the heavy lifting when it came to doing data prep and clean up. Next, was the desire to make the ways of querying data more intuitive.
The result was what would become the core pieces of the Cinchapi Data Platform (CDP). With it, developers can connect, stream, and store any available data source. It doesn’t matter a whit if the data is structured or not. It can work with traditional relational databases, of course, but it isn’t limited to such.
By using machine learning, once the data sources are connected, either directly or with the CDP’s API “Sponge” component, the platform begins to understand what each source is presenting. It’s also uncovering and establishing relationships between these sources. In other words, it’s doing the data prep..
With the data and relationships beginning to take shape, the next piece of the desired functionality was to make data conversational. To that end, the Cinchapi Data Platform features a natural language processing (NLP) interface. Instead of creating a series of cryptic queries in an effort to effectively “solve for X”, Jeff knew it would be much easier and far more intuitive if the developer or user could just ask questions with common phrases.
Jeff also knew that he needed a strongly consistent database for all of this, ideally one capable of providing ad hoc analytics in real-time, but which could also allow the ability to “rewind time” once relationships had been identified. Unable to find a solution to suit his needs, he began work on the open source Concourse Database.
Concourse is Strongly Consistent, which allows developers to work with definitive data. By that, we mean data that has to be accurate at all times – be it in real-time, or in the past. Jeff likens the ability to rewind time as a “DVR for Data”. By that, he means that much like how someone might be watching a hockey or basketball game in real time, they also have the ability to pause and rewind any play to see more clearly how a goal was scored or a basket was made.
To carry that metaphor to data, imagine that you have just uncovered a relationship between multiple data sources – one wholly new to you, but absolutely interesting. With your “Data DVR”, you could go back in time and see what was happening in the context of this newly discovered relationship.
If you want to kick the tires of Concourse, have at it. It is freely available at ConcouseDB.com. Heck, we won’t even ask you to fill out a form. We’re big advocates of Open Source, and we do want folks to both use the database and we invite those interested to become contributors to the project.
That said, while Concourse is a fantastic operational database with ad hoc analytics, do be aware that it’s only the full CDP adds all of that extra goodness: The machine learning, the natural language interface, the visualization engine and assorted other goodies which you won’t be getting with Concourse solo.
The Internet of Things and the Cinchapi Data Platform
Now let’s circle back to IoT and the data produced by it. As we mentioned earlier, there is no standard for IoT data. Any manufacturer of a device may deliver data in virtually any fashion they deem desirable. There isn’t set way of producing the data. Sure, some devices may be easier to work with, and there might even be documentation to explain how the manufacturer suggests how to leverage it.
But with 20 Billion devices coming online within the next three years, can you imagine trying to master the data produced from all of them? Yeah. That’s why aspirin and antacids always seem to be found in the break room.
All kidding aside, there is a better way. Just as how the Cinchapi Data Platform can make short work of traditional data sources, it is ideally suited to work with IoT data. Remember, the CDP doesn’t impose any schema requirements on the developer. As long as data can be connected to it, the CDP streams and stores the data while machine learning makes sense of it all. That absolutely includes IoT data.
If your organization is looking at IoT as a must have, but cannot figure out how to work with the data generated from IoT (as well as all of your other data sources – even those proprietary databases that have been in production since the dawn of time), we’d love to show you what the Cinchapi Data Platform can do.
Click here, and you can watch a 60 second overview video, and then, if you want to get a full-on demonstration, fill out the form and we can set something up.