Golioth Pipelines let users manage and route IoT data

  • June 4, 2024
  • Steve Rogerson

California-based IoT development platform Golioth has launched Pipelines, a set of features that transforms the way users manage and route IoT data to any cloud, data store or platform.

Pipelines replaces the now-deprecated Output Streams, offering better functionality, scalability and user control.

Golioth developed Pipelines to offer more versatility for data routing. Unlike Output Streams, Pipelines allows data to be directed to its LightDB Stream time series storage, user databases or any other destination, in any format. This flexibility addresses regulatory and data residency requirements, eliminating previous restrictions.

Pipelines lets users build IoT infrastructure as code; developers can version pipeline configurations alongside application code. Developers can filter and transform streaming data, transforming compressed device-native data into a more cloud-friendly format. They can send data from a fleet to multiple clouds, data stores, applications or all of these. And they can enable Pipelines in their firmware to capture and route data from device to cloud with low effort.

Developers can also create custom webhooks, sending data to any destination.

“Pipelines is a huge leap towards realising the full potential of device lakes,” said CEO Jonathan Beri. “Our new features allow users to filter, transform and route IoT data with unmatched flexibility and control. This means every piece of valuable data generated by devices can be tapped into and directed where they are needed most, creating new opportunities for innovation and efficiency.”

Pipelines filtering and transformation features are intended to simplify backend services through low-code configuration. Configurations are managed as YAML files, ensuring easy deployment across multiple projects. This version-controlled approach aligns data routing configurations with application code.

Eli Thomas, head of technology at Method Recycling, a waste management company on the path to divert a billion kilograms of waste from landfills, explained: “With Pipelines, we can modify requests and data directly before sending it to our database, eliminating the need for our custom edge functions. This simplifies our workflow significantly. Previously, we had to strip and clean the data manually before it went into our database, but now we can handle different data formats directly and apply custom logic only when needed. It’s a game-changer for us, removing the middle layer and streamlining the entire process.”

Golioth says Pipelines (docs.golioth.io/data-routing) decouple data storage from routing. This means users can route data to their own compliant storage, meeting regulatory requirements for sensitive applications such as medical devices. Users who do not need data persistence can route data directly to other destinations, avoiding costs associated with streaming data to multiple storage options.

Golioth maintains usage-based pricing (golioth.io/pricing) and has introduced volume discounts to accommodate various data usage patterns. Large-volume fleets can stream sensor data for as low as $0.001 per device, per month. The first 3Mbyte of usage for Pipelines is free, allowing users to prototype without a credit card.

Golioth’s free tier enables Pipelines to try without cost to new users with generous data caps. For existing accounts, the transition from Output Streams to Pipelines is seamless, with no interruptions to data streaming. Existing users can opt-in to migrate, while new users will see Pipelines as the default in the console.

Golioth (www.golioth.io) provides IoT data management. Its IoT platform helps engineers rapidly build IoT, manage fleets, and transform and route their IoT data, ensuring compliance, efficiency and scalability.