Data Engineer (Weather Data)tags.new

PALO IT is a global technology consultancy that crafts tech as a force for good. We design, develop and scale digital and sustainable products and services to unlock value across the triple bottom line: people, planet, profit. We do the right thing, and we do it right. We're proud to be a World Economic Forum New Champion, and a B Corp-certified company. 

• We are small enough to care locally, big enough to deliver globally (5 continents, 18 offices, +650 experts from +50 nationalities)

• We are robust and resilient (100% independent and 0 debt)

• We are entrepreneurs and passionate experts: we invest in what we believe genuinely and work as a collective intelligence

• We are positive, courageous, caring, doers and committed to excellence

While the market is still largely AI-augmenting delivery, we have reinvented the SDLC to be AI First. Our approach is a game-changer in productivity and quality, with a strong collaboration between generative AI and our best Talents:

• We now generate 95% of the entire product — code, documentation, infrastructure as code, and even design — with GitHub Copilot.

• The quality consistently exceeds the output of our best traditional engineering teams.

• A product repository houses all product artefacts, giving AI full project context for higher-quality generation.

• A library of rules and prompts defines coding standards, design principles, and security guidelines, ensuring enterprise-grade quality and scalability.

With Gen-e2, we deliver end-to-end products 2–3× faster than traditional approaches, while raising the bar for engineering excellence.

We're looking for a Data Engineer to support our partner in managing the  influx of weather-related datasets that are critical for their business transformation in the insurance domain. You’ll be the go-to expert for ingesting, organizing, and preparing weather-based data pipelines that drive insights and automation.

• Design, develop and manage scalable and secure data pipelines to ingest and process large volumes of weather data from various external sources.
• Leverage APIs, batch, and streaming ingestion mechanisms for real-time and historical weather datasets.
• Collaborate with data scientists, solution architects, and engineering teams to integrate weather data into products and machine learning workflows.
• Work with stakeholders to define data models and ensure weather data is cleaned, structured, and usable for downstream applications.
• Implement data quality checks, validation routines, and anomaly detection specific to weather information (e.g., temperature, precipitation, wind patterns).
• Optimize storage and compute costs for high-volume weather datasets, ensuring data pipelines are cost-effective and resilient.
• Maintain documentation and metadata to provide transparency and governance over data usage.

• Proven experience working as a Data Engineer, ideally within insurance, climate tech, or industries where weather data plays a pivotal role.
• Strong proficiency in Python, SQL, and at least one major cloud platform (AWS preferred; GCP or Azure also welcome).
• Experience working with weather data APIs, such as NOAA, OpenWeatherMap, Weather Company, Meteomatics, or similar.
• Proficient in ETL tools, data orchestration frameworks (Airflow, Prefect, Dagster), and streaming platforms (Kafka, Kinesis).
• Familiar with geospatial data formats and processing tools (GeoJSON, raster data, GDAL, PostGIS).
• Solid understanding of data lakes, data warehouses (Snowflake, BigQuery, Redshift), and data modeling best practices.
• Strong problem-solving skills with attention to detail and a proactive mindset.
• B2-C1 level in English 

We’re eager to adapt to change, learn from our experiences and move to meet our planet’s urgent needs. We are continuously taking action to:

• Become a climate net-zero company

• Deliver projects with a positive impact

• Train 100% of our workforce on impact

• Achieve B Corp certification among all our offices across the globe

• Continuously measure & improve employee happiness

Our clients are amongst the world's most successful companies. We innovate with both established Fortune 1000s, SMEs and start-ups who aim to make an impact, become global leaders and address the world's most complex challenges.

• Stimulating working environments

• Unique career path

• International mobility

• Internal R&D projects

• Knowledge sharing

• Personalized training

• Entrepreneurship & intrapreneurship opportunities

For more on our team culture and benefits,
👉 Check out our careers page
🎙️ And our PALOCast with direct testimonies from our Palowans!