---

Meta- and Transfer Learning

Workshop Papers

Venue	Title
NeurIPS 2024	Using Convolutional Neural Processes to Produce High-Resolution Weather Datasets Over New Zealand (Papers Track) Abstract and authors: (click to expand) Abstract: In recent years, there has been a surge in the development and success of artificial intelligence methods for global weather forecasting and climate modelling. For regional models, however, there is a lack of high-resolution datasets that can be used to train data-driven forecasting models. In this work, we showcase the use of convolutional neural processes (ConvNP) to generate hourly high-resolution (1km) weather datasets over New Zealand for temperature and precipitation. ConvNP models allow us to produce datasets that are enhanced by station observations and provide uncertainty estimates in their predictions. The generated datasets have applications in AI weather and climate models, model verification, and broader environmental research. Authors: Emily O'Riordan (Bodeker Scientific)
NeurIPS 2024	AtmosArena: Benchmarking Foundation Models for Atmospheric Sciences (Papers Track) Abstract and authors: (click to expand) Abstract: Deep learning has emerged as a powerful tool for atmospheric sciences, showing significant utility across various tasks in weather and climate modeling. In line with recent progress in language and vision foundation models, there are growing efforts to scale and finetune such models for multi-task spatiotemporal reasoning. Despite promising results, existing works often evaluate their model on a small set of non-uniform tasks, which makes it hard to quantify broad generalization across diverse tasks and domains. To address this challenge, we introduce AtmosArena, the first multi-task benchmark dedicated to foundation models in atmospheric sciences. AtmosArena comprises a suite of tasks that cover a broad spectrum of applications in atmospheric physics and atmospheric chemistry. To showcase the capabilities and key features of our benchmark, we conducted extensive experiments to evaluate two state-of-the-art deep learning models, ClimaX and Stormer on AtmosArena, and compare their performance with other deep learning and traditional baselines. By providing a standardized, open-source benchmark, we aim to facilitate further advancements in the field, much like open-source benchmarks have driven the development of foundation models for language and vision. Authors: Tung Nguyen (University of California, Los Angeles); Prateik Sinha (UCLA); Advit Deepak (University of California, Los Angeles); Karen A. McKinnon (University of California, Los Angeles); Aditya Grover (UCLA)
NeurIPS 2024	Meta-Learned Bayesian Optimization for Energy Yield in Inertial Confinement Fusion (Papers Track) Abstract and authors: (click to expand) Abstract: With the growing demand for clean energy, fusion presents a promising path to sustainable power generation. Inertial confinement fusion (ICF) experiments trigger nuclear reactions by firing lasers at a fuel target, typically composed of deuterium and tritium. These experiments are costly and require complex optimization of the laser pulse shape across multiple shots to maximize energy yield. Even though Bayesian Optimization (BO) has been commonly used to optimize such expensive scientific experiments, vanilla BO methods do not leverage prior knowledge of the function from simulations or past experiments and fail to achieve high sample efficiency. In this work, we adapted and explored BO meta-learning techniques for ICF that either meta-learn the BO surrogate model, the acquisition function, or both from simulations. Our results demonstrate that the three meta-learning techniques we investigated, Meta-Learning Acquisition Functions for BO (MetaBO), Rank-Weighted Gaussian Process Ensemble (RGPE), and Neural Acquisition Processes (NAP), drastically reduce the number of experiments needed to achieve a satisfactory yield in ICF simulations. Authors: Vineet Gundecha (Hewlett Packard Enterpise); Ricardo Luna Gutierrez (Hewlett Packard Enterprise); Sahand Ghorbanpour (Hewlett Packard Enterprise); Rahman Ejaz (University of Rochester); Varchas Gopalaswamy (University of Rochester); Riccardo Betti (University of Rochester); Avisek Naug (Hewlett Packard Enterprise); Paolo Faraboschi (Hewlett Packard Enterprise); Soumyendu Sarkar (Hewlett Packard Enterprise)
NeurIPS 2024	Wildflower Monitoring with Expert-annotated Images and Flowering Phenology (Papers Track) Abstract and authors: (click to expand) Abstract: Understanding biodiversity trends is essential for preservation policy planning, and advanced computer vision solutions now enable large-scale automated monitoring for many biodiversity use cases. Wildflower monitoring, in particular, presents unique challenges. Visual similarities in shape and color may exist between different species, while flowers within a species may have significant visual differences. Moreover, flowers follow a growth cycle and look distinctly different over the year, while different species flower at different times of the year. Having access to flowering phenology, more accurate predictions may be made. We propose a novel multi-modal wildflower monitoring task to better identify species, levering both expert-annotated wildflower images and flowering phenology estimates. Moreover, we benchmark several state-of-the-art models using two groups of common wildflower species that have high inter-class similarity, and show that this multi-modal approach significantly outperforms image-only baselines. With this work, we aim to encourage the development of standards for automated wildflower monitoring as a step towards bending the curve of biodiversity loss. The data and the code are publicly available https://georgianagmanolache.github.io/wildflowerpower/ Authors: Georgiana Manolache (Fontys University of Applied Science); Gerard Schouten (Fontys University of Applied Sciences)
NeurIPS 2024	HVAC-DPT: A Decision Pretrained Transformer for HVAC Control (Papers Track) Abstract and authors: (click to expand) Abstract: Building operations consume approximately 40% of global energy, with Heating, Ventilation, and Air Conditioning (HVAC) systems responsible for up to 50% of this consumption [1, 2]. As HVAC energy demands are expected to rise, optimising system efficiency is crucial for reducing future energy use and mitigating climate change [3]. Existing control strategies lack generalisation and require extensive training and data, limiting their rapid deployment across diverse buildings. This paper introduces HVAC-DPT, a Decision-Pretrained Transformer using in-context Reinforcement Learning (RL) for multi-zone HVAC control. HVAC-DPT frames HVAC control as a sequential prediction task, training a causal transformer on inter- action histories generated by diverse RL agents. This approach enables HVAC-DPT to refine its policy in-context, without modifying network parameters, allowing for deployment across different buildings without the need for additional training or data collection. HVAC-DPT reduces energy consumption in unseen buildings by 45% compared to the baseline controller, offering a scalable and effective approach to mitigating the increasing environmental impact of HVAC systems. Authors: Anaïs Berkes (University of Cambridge)
NeurIPS 2024	Mamba MethaneMapper: State Space Model for Methane Detection from Hyperspectral Imagery (Proposals Track) Abstract and authors: (click to expand) Abstract: Methane (CH4) is the chief contributor to global climate change. Recent advancements in AI-based image processing have paved the way for innovative approaches for the detection of methane using hyper-spectral imagery. Existing methods, while effective, often come with high computational demands and associated costs that can limit their practical applications. Addressing these limitations, we propose the Mamba MethaneMapper (MMM), a cost-effective and efficient AI-driven solution designed to enhance methane detection capabilities in hyper-spectral images. MMM will incorporate two key innovations that collectively improve performance while managing costs. First, we will utilize a gpu-aware state-space encoder, which optimizes the computational resources and efficiency of the system. Second, MMM will use an environment-sensitive module to prioritize image regions likely containing methane emissions, which are then analyzed by our efficient Mamba algorithm. This selective approach not only improves the accuracy of methane detection but also significantly reduces unnecessary computations and memory consumption. Authors: Satish Kumar (University of California, Santa Barbara); ASM Iftekhar (Microsoft); Kaikai Liu (University of California, Santa Barbara); Bowen Zhang (University Of California, Santa Barbara); Mehan Jayasuriya (Mozilla)
NeurIPS 2024	Multimodal AI framework for predicting candidate high temperature superconductors (Proposals Track) Abstract and authors: (click to expand) Abstract: Materials science is at the forefront of addressing some of the most pressing challenges of our era, particularly in enhancing energy efficiency and sustainability. One of the most promising avenues in this field is the study of superconductors—materials that, when cooled below a critical temperature (Tc), exhibit zero electrical resistance. This unique property not only eliminates energy loss due to resistance but also enables a wide range of advanced technologies, such as MRI machines, magnetically levitating trains, and other high-efficiency systems. Superconductors can significantly reduce the carbon footprint of power transmission and other industrial applications. Given the complexity and importance of predicting candidate and practical high-temperature superconductors, we propose to develop a multimodal AI framework to predict new high-Tc superconducting materials. By integrating various material properties, including structural and compositional data, we seek to study patterns and relationships that could guide the discovery of new high-temperature superconductors. Success in this endeavor could significantly reduce energy losses in electrical systems, contributing to the fight against climate change. Authors: Nidhish Sagar (Massachusetts Institute of Technology); Eslam G. Al-Sakkari (Polytechnique Montréal); Ahmed Ragab (Polytechnique Montréal)
NeurIPS 2024	Towards more efficient agricultural practices via transformer-based crop type classification (Proposals Track) Abstract and authors: (click to expand) Abstract: Machine learning has great potential to increase crop production and resilience to climate change. Accurate maps of where crops are grown are a key input to several downstream policy and research applications. This proposal presents preliminary work showing that it is possible to accurately classify crops from time series derived from Sentinel 1 and 2 satellite imagery in Mexico using a pixel-based binary crop/non-crop time series transformer model. We also find preliminary evidence that meta-learning approaches supplemented with data from similar agro-ecological zones may improve model performance. Due to these promising results, we propose further development of this method with the goal of accurate multi-class crop classification in Jalisco, Mexico via meta-learning with a dataset comprising similar agro-ecological zones. Authors: Isabella Smythe (Columbia University); Eduardo Ulises Moya (Gobierno de Jalisco); Michael Smith (Aspia Space); Yazid Mikail (Climate Change AI); Daisy Ondwari (Kabarak University)
ICLR 2024	Neural Processes for Short-Term Forecasting of Weather Attributes (Papers Track) Abstract and authors: (click to expand) Abstract: Traditional weather prediction models rely on solving complex physical equations, with long computation time. Machine learning models can process large amount of data more quickly. We propose to use neural processes (NPs) for short-term weather attributes forecasting. This is a novel avenue of research, as previous work has focused on NPs for long-term forecasting. We compare a multi-task neural process (MTNP) to an ensemble of independent single-task NPs (STNP) and to an ensemble of Gaussian processes (GPs). We use time series data for multiple weather attributes from Chichester Harbour over a one-week period. We evaluate performance in terms of NLL and MSE with 2-hours and 6-hours time horizons. When limited context information is provided, the MTNP leverages inter-task knowledge and outperforms the STNP. The STNP outperforms both the MTNP and the GPs ensemble when a sufficient, but not exceeding, amount of context information is provided. Authors: Benedetta L Mussati (University of Oxford); Helen McKay (Mind Foundry); Stephen Roberts (University of Oxford)
ICLR 2024	SkyImageNet: Towards a large-scale sky image dataset for solar power forecasting (Proposals Track) Abstract and authors: (click to expand) Abstract: The variability of solar photovoltaic (PV) output, particularly that caused by rapidly changing cloud dynamics, challenges the reliability of renewable energy systems. Solar forecasting based on cloud observations collected by ground-level sky cameras shows promising performance in anticipating short-term solar power fluctuations. However, current deep learning methods often rely on a single dataset with limited sample diversity for training, and thus generalize poorly to new locations and different sky conditions. Moreover, the lack of a standardized dataset hinders the consistent comparison of existing solar forecasting methods. To close these gaps, we propose to build a large-scale standardized sky image dataset --- SkyImageNet --- by assembling, harmonizing, and processing suitable open-source datasets collected in various geographical locations. An accompanying python package will be developed to streamline the process of utilizing SkyImageNet in a machine learning framework. We hope that the outcomes of this project will foster the development of more robust forecasting systems, advance the comparability of short-term solar forecasting model performances, and further facilitate the transition to the next generation of sustainable energy systems. Authors: Yuhao Nie (Massachusetts Institute of Technology); Quentin Paletta (European Space Research Institute); Sherrie Wang (MIT)
NeurIPS 2023	Zero shot microclimate prediction with deep learning (Papers Track) Abstract and authors: (click to expand) Abstract: While weather station data is a valuable resource for climate prediction, its reliability can be limited in remote locations. Furthermore, making local predictions often relies on sensor data that may not be accessible for a new, unmonitored location. In response to these challenges, we introduce a novel zero-shot learning approach designed to forecast various climate measurements at new and unmonitored locations. Our method surpasses conventional weather forecasting techniques in predicting microclimate variables by leveraging knowledge extracted from other geographic locations. Authors: Iman Deznabi (UMass); Peeyush Kumar (Microsoft Research); Madalina Fiterau (University of Massachusetts Amherst)
NeurIPS 2023	Sim2Real for Environmental Neural Processes (Papers Track) Abstract and authors: (click to expand) Abstract: Machine learning (ML)-based weather models have recently undergone rapid improvements.These models are typically trained on gridded reanalysis data from numerical data assimilation systems. However, reanalysis data comes with limitations, such as assumptions about physical laws and low spatiotemporal resolution. The gap between reanalysis and reality has sparked growing interest in training ML models directly on observations such as weather stations. Modelling scattered and sparse environmental observations requires scalable and flexible ML architectures, one of which is the convolutional conditional neural process (ConvCNP). ConvCNPs can learn to condition on both gridded and off-the-grid context data to make uncertainty-aware predictions at target locations. However, the sparsity of real observations presents a challenge for data-hungry deep learning models like the ConvCNP. One potential solution is `Sim2Real': pre-training on reanalysis and fine-tuning on observational data. We analyse Sim2Real with a ConvCNP trained to interpolate surface air temperature over Germany, using varying numbers of weather stations for fine-tuning. On held-out weather stations, Sim2Real training substantially outperforms the same model trained only with reanalysis data or only with station data, showing that reanalysis data can serve as a stepping stone for learning from real observations. Sim2Real could enable more accurate models for climate change monitoring and adaptation. Authors: Jonas Scholz (University of Cambridge)
NeurIPS 2023	Resource Efficient and Generalizable Representation Learning of High-Dimensional Weather and Climate Data (Papers Track) Abstract and authors: (click to expand) Abstract: We study self-supervised representation learning on high-dimensional data under resource constraints. Our work is motivated by applications of vision transformers to weather and climate data. Such data frequently comes in the form of tensors that are both higher dimensional and of larger size than the RGB imagery one encounters in many computer vision experiments. This raises scaling issues and brings up the need to leverage available compute resources efficiently. Motivated by results on masked autoencoders, we show that it is possible to use sampling of subtensors as the sole augmentation strategy for contrastive learning with a sampling ratio of $\sim$1\%. This is to be compared to typical masking ratios of $75\%$ or $90\%$ for image and video data respectively. In an ablation study, we explore extreme sampling ratios and find comparable skill for ratios as low as $\sim$0.0625\%. Pursuing efficiencies, we are finally investigating whether it is possible to generate robust embeddings on dimension values which were not present at training time. We answer this question to the positive by using learnable position encoders which have continuous dependence on dimension values. Authors: Juan Nathaniel (Columbia University); Marcus Freitag (IBM); Patrick Curran (Environment and Climate Change Canada); Isabel Ruddick (Environment and Climate Change Canada); Johannes Schmude (IBM)
NeurIPS 2023	PressureML: Modelling Pressure Waves to Generate Large-Scale Water-Usage Insights in Buildings (Papers Track) Abstract and authors: (click to expand) Abstract: Several studies have indicated that delivering insights and feedback on water usage has been effective in curbing water consumption, making it a pivotal component in achieving long-term sustainability objectives. Despite a significant proportion of water consumption originating from large residential and commercial buildings, there is a scarcity of cost-effective and easy-to-integrate solutions that provide water usage insights in such structures. Furthermore, existing methods for disaggregating water usage necessitate training data and rely on frequent data sampling to capture patterns, both of which pose challenges when scaling up and adapting to new environments. In this work, we aim to solve these challenges through a novel end-to-end approach which records data from pressure sensors and uses time-series classification by DNN models to determine room-wise water consumption in a building. This consumption data is then fed to a novel water disaggregation algorithm which can suggest a set of water-usage events, and has a flexible requirement of training data and sampling granularity. We conduct experiments using our approach and demonstrate its potential as a promising avenue for in-depth exploration, offering valuable insights into water usage on a large scale. Authors: Tanmaey Gupta (Microsoft Research India); Anupam Sobti (IIT Delhi); Akshay Nambi (Microsoft Research)
NeurIPS 2023	Predicting Adsorption Energies for Catalyst Screening with Transfer Learning Using Crystal Hamiltonian Graph Neural Network (Proposals Track) Abstract and authors: (click to expand) Abstract: As the world moves towards a clean energy future to mitigate the risks of climate change, the discovery of new catalyst materials plays a significant role in enabling the sustainable production and transformation of energy [2]. The development and verification of fast, accurate, and efficient artificial intelligence and machine learning techniques is critical to shortening time-intensive calculations, reducing costs, and improving computational feasibility. We propose applying the Crystal Hamiltonian Graph Neural Network (CHGNet) on the OC20 dataset in order to iteratively perform structure-to-energy and forces calculations and identify the lowest energy across relaxed structures for a given adsorbate-surface combination. CHGNet's predictions will be compared and benchmarked to corresponding values calculated by density functional theory (DFT) [7] and other models to determine its efficacy. Authors: Angelina Chen (Foothill College/Lawrence Berkeley National Lab); Hui Zheng (Lawrence Berkeley National Lab); Paula Harder (Mila)
ICLR 2023	Attention-based Domain Adaptation Forecasting of Streamflow in Data-Sparse Regions (Papers Track) Abstract and authors: (click to expand) Abstract: Streamflow forecasts are critical to guide water resource management, mitigate drought and flood effects, and develop climate-smart infrastructure and governance. Many global regions, however, have limited streamflow observations to guide evidence-based management strategies. In this paper, we propose an attention-based domain adaptation streamflow forecaster for data-sparse regions. Our approach leverages the hydrological characteristics of a data-rich source domain to induce effective 24hr lead-time streamflow prediction in a data-constrained target domain. Specifically, we employ a deep-learning framework leveraging domain adaptation techniques to simultaneously train streamflow predictions and discern between both domains using an adversarial method. Experiments against baseline cross-domain forecasting models show improved performance for 24hr lead-time streamflow forecasting. Authors: Roland R Oruche (University of Missouri-Columbia); Fearghal O'Donncha (IBM Research)
ICLR 2023	ClimaX: A foundation model for weather and climate (Papers Track) Abstract and authors: (click to expand) Abstract: Recent data-driven approaches based on machine learning aim to directly solve a downstream forecasting or projection task by learning a data-driven functional mapping using deep neural networks. However, these networks are trained using curated and homogeneous climate datasets for specific spatiotemporal tasks, and thus lack the generality of currently used physics-informed numerical models for weather and climate modeling. We develop and demonstrate ClimaX, a flexible and generalizable deep learning model for weather and climate science that can be trained using heterogeneous datasets spanning different variables, spatiotemporal coverage, and physical groundings. ClimaX extends the Transformer architecture with novel encoding and aggregation blocks that allow effective use of available compute and data while maintaining general utility. ClimaX is pre-trained with a self-supervised learning objective on climate datasets derived from CMIP6. The pretrained ClimaX can then be fine-tuned to address a breadth of climate and weather tasks, including those that involve atmospheric variables and spatiotemporal scales unseen during pretraining. Compared to existing data-driven baselines, we show that this generality in ClimaX results in superior performance on benchmarks for weather forecasting and climate projections. Authors: Tung Nguyen (University of California, Los Angeles); Johannes Brandstetter (Microsoft Research); Ashish Kapoor (Microsoft); Jayesh Gupta (Microsoft Research); Aditya Grover (UCLA)
NeurIPS 2022	Detecting Methane Plumes using PRISMA: Deep Learning Model and Data Augmentation (Papers Track) Abstract and authors: (click to expand) Abstract: The new generation of hyperspectral imagers, such as PRISMA, has improved significantly our detection capability of methane (CH4) plumes from space at high spatial resolution (∼30m). We present here a complete framework to identify CH4 plumes using images from the PRISMA satellite mission and a deep learning technique able to automatically detect plumes over large areas. To compensate for the sparse database of PRISMA images, we trained our model by transposing high resolution plumes from Sentinel-2 to PRISMA. Our methodology avoids computationally expensive synthetic plume from Large Eddy Simulations while generating a broad and realistic training database, and paves the way for large-scale detection of methane plumes using future hyperspectral sensors (EnMAP, EMIT, CarbonMapper). Authors: Alexis Groshenry (Kayrros); Clément Giron (Kayrros); Alexandre d'Aspremont (CNRS, DI, Ecole Normale Supérieure; Kayrros); Thomas Lauvaux (University of Reims Champagne Ardenne, GSMA, UMR 7331); Thibaud Ehret (Centre Borelli)
NeurIPS 2022	Convolutional Neural Processes for Inpainting Satellite Images: Application to Water Body Segmentation (Papers Track) Abstract and authors: (click to expand) Abstract: The widespread availability of satellite images has allowed researchers to monitor the impact of climate on socio-economic and environmental issues through examples like crop and water body classification to measure food scarcity and risk of flooding. However, a common issue of satellite images is missing values due to measurement defects, which render them unusable by existing methods without data imputation. To repair the data, inpainting methods can be employed, which are based on classical PDEs or interpolation methods. Recently, deep learning approaches have shown promise in this realm, however many of these methods do not explicitly take into account the inherent spatio-temporal structure of satellite images. In this work, we cast satellite image inpainting as a meta-learning problem, and implement Convolutional Neural Processes (ConvNPs) in which we frame each satellite image as its own task or 2D regression problem. We show that ConvNPs outperform classical methods and state-of-the-art deep learning inpainting models on a scanline problem for LANDSAT 7 satellite images, assessed on a variety of in- and out-of-distribution images. Our results successfully match the performance of clean images on a downstream water body segmentation task in Canada. Authors: Alexander Pondaven (Imperial College London); Mart Bakler (Imperial College London); Donghu Guo (Imperial College London); Hamzah Hashim (Imperial College London); Martin G Ignatov (Imperial college London); Samir Bhatt (Imperial College London); Seth Flaxman (Oxford); Swapnil Mishra (Imperial College London); Elie Alhajjar (USMA); Harrison Zhu (Imperial College London)
NeurIPS 2022	Deep Climate Change: A Dataset and Adaptive domain pre-trained Language Models for Climate Change Related Tasks (Papers Track) Abstract and authors: (click to expand) Abstract: The quantity and quality of literature around climate change (CC) and its impacts are increasing yearly. Yet, this field has received limited attention in the Natural Language Processing (NLP) community. With the help of large Language Models (LMs) and transfer learning, NLP can support policymakers, researchers, and climate activists in making sense of large-scale and complex CC-related texts. CC-related texts include specific language that general language models cannot represent accurately. Therefore we collected a climate change corpus consisting of over 360 thousand abstracts of top climate scientists' articles from trustable sources covering large temporal and spatial scales. Comparison of the performance of GPT2 LM and our 'climateGPT2 models', fine-tuned on the CC-related corpus, on claim generation (text generation) and fact-checking, downstream tasks show the better performance of the climateGPT2 models compared to the GPT2. The climateGPT2 models decrease the validation loss to 1.08 for claim generation from 43.4 obtained by GPT2. We found that climateGPT2 models improved the masked language model objective for the fact-checking task by increasing the F1 score from 0.67 to 0.72. Authors: Saeid Vaghefi (University of Zürich); Veruska Muccione (University of Zürich); Christian Huggel (University of Zürich); Hamed Khashehchi (2w2e GmbH); Markus Leippold (University of Zurich)
NeurIPS 2022	Don't Waste Data: Transfer Learning to Leverage All Data for Machine-Learnt Climate Model Emulation (Papers Track) Abstract and authors: (click to expand) Abstract: How can we learn from all available data when training machine-learnt climate models, without incurring any extra cost at simulation time? Typically, the training data comprises coarse-grained high-resolution data. But only keeping this coarse-grained data means the rest of the high-resolution data is thrown out. We use a transfer learning approach, which can be applied to a range of machine learning models, to leverage all the high-resolution data. We use three chaotic systems to show it stabilises training, gives improved generalisation performance and results in better forecasting skill. Our code is at https://github.com/raghul-parthipan/dont_waste_data Authors: Raghul Parthipan (University of Cambridge); Damon Wischik (Univeristy of Cambridge)
NeurIPS 2022	Land Use Prediction using Electro-Optical to SAR Few-Shot Transfer Learning (Papers Track) Abstract and authors: (click to expand) Abstract: Satellite image analysis has important implications for land use, urbanization, and ecosystem monitoring. Deep learning methods can facilitate the analysis of different satellite modalities, such as electro-optical (EO) and synthetic aperture radar (SAR) imagery, by supporting knowledge transfer between the modalities to compensate for individual shortcomings. Recent progress has shown how distributional alignment of neural network embeddings can produce powerful transfer learning models by employing a sliced Wasserstein distance (SWD) loss. We analyze how this method can be applied to Sentinel-1 and -2 satellite imagery and develop several extensions toward making it effective in practice. In an application to few-shot Local Climate Zone (LCZ) prediction, we show that these networks outperform multiple common baselines on datasets with a large number of classes. Further, we provide evidence that instance normalization can significantly stabilize the training process and that explicitly shaping the embedding space using supervised contrastive learning can lead to improved performance. Authors: Marcel Hussing (University of Pennsylvania); Karen Li (University of Pennsylvania); Eric Eaton (University of Pennsylvania)
NeurIPS 2022	Closing the Domain Gap -- Blended Synthetic Imagery for Climate Object Detection (Papers Track) Abstract and authors: (click to expand) Abstract: Object detection models have great potential to increase both the frequency and cost-efficiency of assessing climate-relevant infrastructure in satellite imagery. However, model performance can suffer when models are applied to stylistically different geographies. We propose a technique to generate synthetic imagery using minimal labeled examples of the target object at a low computational cost. Our technique blends example objects onto unlabeled images of the target domain. We show that including these synthetic images improves the average precision of a YOLOv3 object detection model when compared to a baseline and other popular domain adaptation techniques. Authors: Caleb Kornfein (Duke University); Frank Willard (Duke University); Caroline Tang (Duke University); Yuxi Long (Duke University); Saksham Jain (Duke University); Jordan Malof (Duke University); Simiao Ren (Duke University); Kyle Bradbury (Duke University)
NeurIPS 2022	Generalized Ice Detection on Wind Turbine Rotor Blades with Neural Style Transfer (Papers Track) Abstract and authors: (click to expand) Abstract: Wind energy’s ability to liberate the world of conventional sources of energy relies on lowering the significant costs associated with the maintenance of wind turbines. Since icing events on turbine rotor blades are a leading cause of operational failures, identifying icing in advance is critical. Some recent studies focus on specific wind parks and fail to generalize to unseen scenarios (e.g. new rotor blade designs). We propose the utilisation of synthetic data augmentation via neural style transfer to improve the generalization of existing ice prediction models. We show that training models with augmented data that captures domain-invariant icing characteristics can help improve predictive performance across multiple wind parks. Through efficient identification of icing, this study can support preventive maintenance of wind energy sources by making them more reliable towards tackling climate change. Authors: Joyjit Chatterjee (University of Hull); Maria Teresa Alvela Nieto (University of Bremen); Hannes Gelbhardt (University of Bremen); Nina Dethlefs (University of Hull); Jan Ohlendorf (University of Bremen); Klaus-Dieter Thoben (University of Bremen)
NeurIPS 2022	Towards Global Crop Maps with Transfer Learning (Papers Track) Abstract and authors: (click to expand) Abstract: The continuous increase in global population and the impact of climate change on crop production are expected to affect the food sector significantly. In this context, there is need for timely, large-scale and precise mapping of crops for evidence-based decision making. A key enabler towards this direction are new satellite missions that freely offer big remote sensing data of high spatio-temporal resolution and global coverage. During the previous decade and because of this surge of big Earth observations, deep learning methods have dominated the remote sensing and crop mapping literature. Nevertheless, deep learning models require large amounts of annotated data that are scarce and hard-to-acquire. To address this problem, transfer learning methods can be used to exploit available annotations and enable crop mapping for other regions, crop types and years of inspection. In this work, we have developed and trained a deep learning model for paddy rice detection in South Korea using Sentinel-1 VH time-series. We then fine-tune the model for i) paddy rice detection in France and Spain and ii) barley detection in the Netherlands. Additionally, we propose a modification in the pre-trained weights in order to incorporate extra input features (Sentinel-1 VV). Our approach shows excellent performance when transferring in different areas for the same crop type and rather promising results when transferring in a different area and crop type. Authors: Hyun-Woo Jo (Korea University); Alkiviadis Marios Koukos (National Observatory of Athens); Vasileios Sitokonstantinou (National Observatory of Athens); Woo-Kyun Lee (Korea University); Charalampos Kontoes (National Observatory of Athens)
NeurIPS 2022	Disaster Risk Monitoring Using Satellite Imagery (Tutorials Track) Abstract and authors: (click to expand) Abstract: Natural disasters such as flood, wildfire, drought, and severe storms wreak havoc throughout the world, causing billions of dollars in damages, and uprooting communities, ecosystems, and economies. Unfortunately, flooding events are on the rise due to climate change and sea level rise. The ability to detect and quantify them can help us minimize their adverse impacts on the economy and human lives. Using satellites to study flood is advantageous since physical access to flooded areas is limited and deploying instruments in potential flood zones can be dangerous. We are proposing a hands-on tutorial to highlight the use of satellite imagery and computer vision to study natural disasters. Specifically, we aim to demonstrate the development and deployment of a flood detection model using Sentinel-1 satellite data. The tutorial will cover relevant fundamental concepts as well as the full development workflow of a deep learning-based application. We will include important considerations such as common pitfalls, data scarcity, augmentation, transfer learning, fine-tuning, and details of each step in the workflow. Importantly, the tutorial will also include a case study on how the application was used by authorities in response to a flood event. We believe this tutorial will enable machine learning practitioners of all levels to develop new technologies that tackle the risks posed by climate change. We expect to deliver the below learning outcomes: • Develop various deep learning-based computer vision solutions using hardware-accelerated open-source tools that are optimized for real-time deployment • Create an optimized pipeline for the machine learning development workflow • Understand different performance metrics for model evaluation that are relevant for real world datasets and data imbalances • Understand the public sector’s efforts to support climate action initiatives and point out where the audience can contribute Authors: Kevin Lee (NVIDIA); Siddha Ganju (NVIDIA); Edoardo Nemni (UNOSAT)
NeurIPS 2021	Meta-Learned Bayesian Optimization for Calibrating Building Simulation Models with Multi-Source Data (Papers Track) Abstract and authors: (click to expand) Abstract: Well-calibrated building simulation models are key to reducing greenhouse gas emissions and optimizing building performance. Current calibration algorithms do not leverage data collected during previous calibration tasks. In this paper, we employ attentive neural processes (ANP) to meta-learn a distribution using multi-source data acquired during previously seen calibration tasks. The ANP informs a meta-learned Bayesian optimizer to accelerate calibration of new, unseen tasks. The few-shot nature of our proposed algorithm is demonstrated on a library of residential buildings validated by the United States Department of Energy (USDoE). Authors: Sicheng Zhan (NUS); Gordon Wichern (Mitsubishi Electric Research Laboratories (MERL)); Christopher Laughman (Mitsubishi Electric Research Laboratories); Ankush Chakrabarty (Mitsubishi Electric Research Labs)
NeurIPS 2021	Evaluating Pretraining Methods for Deep Learning on Geophysical Imaging Datasets (Papers Track) Abstract and authors: (click to expand) Abstract: Machine learning has the potential to automate the analysis of vast amounts of raw geophysical data, allowing scientists to monitor changes in key aspects of our climate such as cloud cover in real-time and at fine spatiotemporal scales. However, the lack of large labeled training datasets poses a significant barrier for effectively applying machine learning to these applications. Transfer learning, which involves first pretraining a neural network on an auxiliary “source” dataset and then finetuning on the “target” dataset, has been shown to improve accuracy for machine learning models trained on small datasets. Across prior work on machine learning for geophysical imaging, different choices are made about what data to pretrain on, and the impact of these choices on model performance is unclear. To address this, we systematically explore various settings of transfer learning for cloud classification, cloud segmentation, and aurora classification. We pretrain on different source datasets, including the large ImageNet dataset as well as smaller geophysical datasets that are more similar to the target datasets. We also experiment with multiple transfer learning steps where we pretrain on more than one source dataset. Despite the smaller source datasets’ similarity to the target datasets, we find that pretraining on the large, general-purpose ImageNet dataset yields significantly better results across all of our experiments. Transfer learning is especially effective for smaller target datasets, and in these cases, using multiple source datasets can give a marginal added benefit. Authors: James Chen (Kirby School)
NeurIPS 2021	Semi-Supervised Classification and Segmentation on High Resolution Aerial Images (Papers Track) Abstract and authors: (click to expand) Abstract: FloodNet is a high-resolution image dataset acquired by a small UAV platform, DJI Mavic Pro quadcopters, after Hurricane Harvey. The dataset presents a unique challenge of advancing the damage assessment process for post-disaster scenarios using unlabeled and limited labeled dataset. We propose a solution to address their classification and semantic segmentation challenge. We approach this problem by generating pseudo labels for both classification and segmentation during training and slowly incrementing the amount by which the pseudo label loss affects the final loss. Using this semi-supervised method of training helped us improve our baseline supervised loss by a huge margin for classification, allowing the model to generalize and perform better on the validation and test splits of the dataset. In this paper, we compare and contrast the various methods and models for image classification and semantic segmentation on the FloodNet dataset. Authors: Sahil S Khose (Manipal Institute of Technology); Abhiraj Tiwari (Manipal Institute of Technology); Ankita Ghosh (Manipal Institute of Technology)
NeurIPS 2021	Toward Foundation Models for Earth Monitoring: Proposal for a Climate Change Benchmark (Proposals Track) Abstract and authors: (click to expand) Abstract: Recent progress in self-supervision shows that pre-training large neural networks on vast amounts of unsupervised data can lead to impressive increases in generalisation for downstream tasks. Such models, recently coined as foundation models, have been transformational to the field of natural language processing. While similar models have also been trained on large corpuses of images, they are not well suited for remote sensing data. To stimulate the development of foundation models for Earth monitoring, we propose to develop a new benchmark comprised of a variety of downstream tasks related to climate change. We believe that this can lead to substantial improvements in many existing applications and facilitate the development of new applications. This proposal is also a call for collaboration with the aim of developing a better evaluation process to mitigate potential downsides of foundation models for Earth monitoring. Authors: Alexandre Lacoste (ServiceNow); Evan D Sherwin (Stanford University, Energy and Resources Engineering); Hannah R Kerner (University of Maryland); Hamed Alemohammad (Radiant Earth Foundation); Björn Lütjens (MIT); Jeremy A Irvin (Stanford); David Dao (ETH Zurich); Alex Chang (Service Now); Mehmet Gunturkun (Element Ai); Alexandre Drouin (ServiceNow); Pau Rodriguez (Element AI); David Vazquez (ServiceNow)
ICML 2021	Examining the nexus of environmental policy, climate physics, and maritime shipping with deep learning models and space-borne data (Papers Track) Abstract and authors: (click to expand) Abstract: Ship-tracks are produced by ship exhaust interacting with marine low clouds. They provide an ideal lab for constraining a critical climate forcing. However, no global survey of ship ship-tracks has been made since its discovery 55 years ago, which limits research progress. Here we present the first global map of ship-tracks produced by applying deep segmentation models to large satellite data. Our model generalizes well and is validated against independent data. Large-scale ship-track data are at the nexus of environmental policy, climate physics, and maritime shipping industry: they can be used to study aerosol-cloud interactions, the largest uncertainty source in climate forcing; to evaluate compliance and impacts of environmental policies; and to study the impact of significant socioeconomic events on maritime shipping. Based on twenty years of global data, we show cloud physics responses in ship-tracks strongly depend on the cloud regime. Inter-annual fluctuation in ship-track frequency clearly reflects international trade/economic trends. Emission policies strongly affect the pattern of shipping routes and ship-track occurrence. The combination of stricter fuel standard and the COVID-19 pandemic pushed global ship-track frequency to the lowest level in the record. More applications of our technique and data are envisioned such as detecting illicit shipping activity and checking policy compliance of individual ships. Authors: Tianle Yuan (University of Maryland, NASA); Hua Song (NASA, SSAI); Chenxi Wang (University of Maryland, NASA); Kerry Meyer (NASA); Siobhan Light (University of Maryland); Sophia von Hippel (University of Arizona); Steven Platnick (NASA); Lazaros Oreopoulos (NASA); Robert Wood (University of Washington); Hans Mohrmann (University of Washington)
ICML 2021	DroughtED: A dataset and methodology for drought forecasting spanning multiple climate zones (Papers Track) Abstract and authors: (click to expand) Abstract: Climate change exacerbates the frequency, duration and extent of extreme weather events such as drought. Previous attempts to forecast drought conditions using machine learning have focused on regional models which have two major limitations for national drought management: (i) they are trained on localised climate data and (ii) their architectures prevent them from being applied to new heterogeneous regions. In this work, we present a new large-scale dataset for training machine learning models to forecast national drought conditions, named DroughtED. The dataset consists of globally available meteorological features widely used for drought prediction, paired with location meta-data which has not previously been utilised for drought forecasting. Here we also establish a baseline on DroughtED and present the first research to apply deep learning models - Long Short-Term Memory (LSTMs) and Transformers - to predict county-level drought conditions across the full extent of the United States. Our results indicate that DroughtED enables deep learning models to learn cross-region patterns in climate data that contribute to drought conditions and models trained on DroughtED compare favourably to state-of-the-art drought prediction models trained on individual regions. Authors: Christoph D Minixhofer (The University of Edinburgh); Mark Swan (The University of Edinburgh); Calum McMeekin (The University of Edinburgh); Pavlos Andreadis (The University of Edinburgh)
ICML 2021	Self-supervised Contrastive Learning for Irrigation Detection in Satellite Imagery (Papers Track) Abstract and authors: (click to expand) Abstract: Climate change has caused reductions in river runoffs and aquifer recharge resulting in an increasingly unsustainable crop water demand from reduced freshwater availability. Achieving food security while deploying water in a sustainable manner will continue to be a major challenge necessitating careful monitoring and tracking of agricultural water usage. Historically, monitoring water usage has been a slow and expensive manual process with many imperfections and abuses. Ma-chine learning and remote sensing developments have increased the ability to automatically monitor irrigation patterns, but existing techniques often require curated and labelled irrigation data, which are expensive and time consuming to obtain and may not exist for impactful areas such as developing countries. In this paper, we explore an end-to-end real world application of irrigation detection with uncurated and unlabeled satellite imagery. We apply state-of-the-art self-supervised deep learning techniques to optical remote sensing data, and find that we are able to detect irrigation with up to nine times better precision, 90% better recall and 40% more generalization ability than the traditional supervised learning methods. Authors: Chitra Agastya (UC Berkeley, IBM); Sirak Ghebremusse (UC Berkeley); Ian Anderson (UC Berkeley); Colorado Reed (UC Berkeley); Hossein Vahabi (University California Berkeley); Alberto Todeschini (UC Berkeley)
NeurIPS 2020	Storing Energy with Organic Molecules: Towards a Metric for Improving Molecular Performance for Redox Flow Batteries (Papers Track) Abstract and authors: (click to expand) Abstract: Energy storage is an important tool in the decarbonization of energy systems, particularly when coupled with intermittent forms of energy. However, storage technologies are still not commercially competitive to garner mainstream adoption. This work focuses on the cost reduction of organic redox flow batteries (ORFBs) via materials discovery. We identify macroscopic metrics of interest to optimize for lowering their cost and relate them to the molecular properties of the materials involved. Furthermore, we consolidate a benchmark set of experimental data for building predictive models for these materials properties. Building more accurate models will aid practitioners in the rational design of new ORFB. Authors: Luis M Mejia Mendoza (University of Toronto); Alan Aspuru-Guzik (Harvard University); Martha Flores Leonar (University of Toronto)
NeurIPS 2020	The Peruvian Amazon Forestry Dataset: A Leaf Image Classification Corpus (Papers Track) Abstract and authors: (click to expand) Abstract: This paper introduces the Peruvian Amazon Forestry Dataset, which includes 59,441 leaves samples from ten of the most profitable and endangered Amazon timber-tree species. Besides, the proposal includes a background removal algorithm to feed a fine-tuned CNN. We evaluate the quantitative (accuracy metric) and qualitative (visual interpretation) impacts of each stage by ablation experiments. The results show a 96.64 % training accuracy and 96.52 % testing accuracy on the VGG-19 model. Furthermore, the visual interpretation of the model evidences that leaf venations have the highest correlation in the plant recognition task. Authors: Gerson Waldyr Vizcarra Aguilar (San Pablo Catholic University); Danitza Bermejo (Universidad Nacional del Altiplano); Manasses A. Mauricio (Universidad Católica San Pablo); Ricardo Zarate (Instituto de Investigaciones de la Amazonía Peruana); Erwin Dianderas (Instituto de Investigaciones de la Amazonía Peruana)
NeurIPS 2020	In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustness (Papers Track) Abstract and authors: (click to expand) Abstract: Many machine learning applications used to tackle climate change involve lots of unlabeled data (such as satellite imagery) along with auxiliary information such as climate data. In this work, we show how to use auxiliary information in a semi-supervised setting to improve both in-distribution and out-of-distribution (OOD) accuracies (e.g. for countries in Africa where we have very little labeled data). We show that 1) on real-world datasets, the common practice of using auxiliary information as additional input features improves in-distribution error but can hurt OOD. Oppositely, we find that 2) using auxiliary information as outputs of auxiliary tasks to pre-train a model improves OOD error. 3) To get the best of both worlds, we introduce In-N-Out, which first trains a model with auxiliary inputs and uses it to pseudolabel all the in-distribution inputs, then pre-trains a model on OOD auxiliary outputs and fine-tunes this model with the pseudolabels (self-training). We show both theoretically and empirically on remote sensing datasets for land cover prediction and cropland prediction that In-N-Out outperforms auxiliary inputs or outputs alone on both in-distribution and OOD error. Authors: Robbie M Jones (Stanford University); Sang Michael Xie (Stanford University); Ananya Kumar (Stanford University); Fereshte Khani (Stanford); Tengyu Ma (Stanford University); Percy Liang (Stanford University)
NeurIPS 2020	Expert-in-the-loop Systems Towards Safety-critical Machine Learning Technology in Wildfire Intelligence (Proposals Track) Abstract and authors: (click to expand) Abstract: With the advent of climate change, wildfires are becoming more frequent and severe across several regions worldwide. To prevent and mitigate its effects, wildfire intelligence plays a pivotal role, e.g. to monitor the evolution of wildfires and for early detection in high-risk areas such as wildland-urban-interface regions. Recent works have proposed deep learning solutions for fire detection tasks, however the current limited databases prevent reliable real-world deployments. We propose the development of expert-in-the-loop systems that combine the benefits of semi-automated data annotation with relevant domain knowledge expertise. Through this approach we aim to improve the data curation process and contribute to the generation of large-scale image databases for relevant wildfire tasks and empower the application of machine learning techniques in wildfire intelligence in real scenarios. Authors: Maria João Sousa (IDMEC, Instituto Superior Técnico, Universidade de Lisboa); Alexandra Moutinho (IDMEC, Instituto Superior Técnico, Universidade de Lisboa); Miguel Almeida (ADAI, University of Coimbra)
NeurIPS 2020	Artificial Intelligence, Machine Learning and Modeling for Understanding the Oceans and Climate Change (Proposals Track) Abstract and authors: (click to expand) Abstract: These changes will have a drastic impact on almost all forms of life in the ocean with further consequences on food security, ecosystem services in coastal and inland communities. Despite these impacts, scientific data and infrastructures are still lacking to understand and quantify the consequences of these perturbations on the marine ecosystem. Understanding this phenomenon is not only an urgent but also a scientifically demanding task. Consequently, it is a problem that must be addressed with a scientific cohort approach, where multi-disciplinary teams collaborate to bring the best of different scientific areas. In this proposal paper, we describe our newly launched four-years project focused on developing new artificial intelligence, machine learning, and mathematical modeling tools to contribute to the understanding of the structure, functioning, and underlying mechanisms and dynamics of the global ocean symbiome and its relation with climate change. These actions should enable the understanding of our oceans and predict and mitigate the consequences of climate change. Authors: Nayat Sánchez Pi (Inria); Luis Martí (Inria); André Abreu (Fountation Tara Océans); Olivier Bernard (Inria); Colomban de Vargas (CNRS); Damien Eveillard (Univ. Nantes); Alejandro Maass (CMM, U. Chile); Pablo Marquet (PUC); Jacques Sainte-Marie (Inria); Julien Salomin (Inria); Marc Schoenauer (INRIA); Michele Sebag (LRI, CNRS, France)