Session
Software Best Practices for Data Pipelines: Form Energy's Multi-Day Storage R&D Pipeline
Software best practices from apps and services are familiar: scalability, flexibility, observability, etc. Mark will describe what these mean in terms of Form Energy's data pipeline for experiment data from its multi-day storage batteries, and how Form has met these software needs with specific practices and frameworks/services implemented in Python including Dagster, AWS, and OpenTelemetry/Honeycomb.
Fossil-fuel energy is the 2nd largest source of greenhouse gasses in the US [EPA]. As we transition to a renewable grid and increasing electrification, the variability of renewable production requires multi-day storage batteries. Form is developing 100-hour iron-air batteries, a drastically cheaper alternative to 8-hour lithium batteries. Internally, our data pipeline must supply research data so we can evaluate thousands of experiments and develop these novel batteries quickly -- and the data always has to be available and correct.