Pelmorex Corp., a Canadian weather and media company headquartered in Oakville, Ontario, owns The Weather Network and MétéoMédia, delivering vital weather information to millions across Canada.
Through its B2B division, WeatherSource, Pelmorex provides data-driven environmental insights to enterprises in sectors like agriculture, logistics, and utilities. To support the processing of complex weather model data, Pelmorex needed a solution that was scalable, automated, and cost-effective.
The challenge: high volume, low reliability
Pelmorex faced several key challenges with its previous data warehouse, including:
- Managing high volumes of weather data required minimal manual intervention.
- There was a need for seamless integration with Google Cloud Marketplace for easy data access.
- Operational costs associated with their existing usage needed to be reduced.
- They sought to enhance performance and reliability across their data pipelines and deployments.
The solution: modern, scalable, automated
To address these challenges, Pelmorex partnered with Onix, which designed and implemented a comprehensive Google BigQuery-based data processing architecture. This new architecture replaced the existing data warehouse with a modern, scalable, and automated solution.
The key components of the solution delivered by Onix included:
- Migration of weather models from the previous data warehouse to BigQuery.
- Zarr file ingestion and parallel processing pipelines for efficient handling of large datasets.
- Automated Terraform templates to ensure reproducible and scalable deployments.
- Data archival mechanisms for cost-effective management of historical datasets.
- Seamless integration to support any legacy interoperability needs.
- A real-time monitoring dashboard to track pipeline health and performance.
- An automated model decommissioning utility, which significantly reduced manual cleanup efforts.
Pelmorex chose Onix as their transformation partner due to several key factors:
- Proven expertise in BigQuery, data engineering, and cloud-native automation.
- Ability to deliver an end-to-end automated architecture aligned with Pelmorex’s goals.
- Demonstrated cost efficiency and performance improvements early in the engagement.
- Strong partnership with Google Cloud, enabling tight coordination with GCP engineers.
The impact: lower cost, less time, higher scalability
The implementation delivered transformational value to Pelmorex across multiple dimensions. The results included:
- A remarkable cost reduction of approximately 30% compared to their previous operations.
- Performance improvement by over 6 hours for large pipelines, enabling near real-time availability of weather insights.
- Automated deployments dramatically reduced manual work.
- A flexible architecture was established, which is well-aligned with Pelmorex’s long-term goals for scalability and automation.
- The solution also aligned with their Google Cloud Marketplace goals for scalable data distribution.
“I am extremely satisfied with the Onix team. They understood that our primary goal was automation and ease of use, and they delivered a fully automated end-to-end solution that does exactly what we need it to. Work that used to take weeks can now be done in a day thanks to the new architecture they delivered.”
– Jonathan Weisbaum,
Director of Software Development
for the B2B WeatherSource
division at Pelmorex
What’s next?
Following the success of this initiative, Onix and Pelmorex plan to continue their partnership. Future plans include migrating additional weather models to BigQuery, extending the architecture to support new data formats and business use cases, and exploring opportunities for AI/ML integrations for predictive weather insights and advanced analytics.
