Killing containers to make weather beautiful
A SkillsCast for this session is not available.
We built a Kubernetes cluster in an AWS group which auto-scales based on load. We used replication controllers to process the data. Every three hours ingestion jobs are added to a queue and the number of ingestion containers are set in proportion to the queue length. Each worker completes exactly one ingestion job from the queue and then exits, at which point Kubernetes creates a new one to process the next message. This has allowed us to remove the lifespan logic from the containers and keep them light, fast and massively scalable. We are now in the process of using this in our production systems.
Thank you to our sponsors and partners
Platinum
Gold
Silver
Bronze
Killing containers to make weather beautiful
Jacob Tomlinson
The Met Office is a world leading weather forecasting and climate research organisation. It owns one of Europe’s largest supercomputers and generates data on the exabyte scale. Jacob Tomlinson is a developer and systems engineer at the Informatics Lab - the innovation unit within the Met Office. He leads their R&D efforts on new infrastructure technologies. Jacob uses Kubernetes and Docker to transform how the Met Office makes data accessible to the world.