Computer Vision & Remote Sensing

Tutorials

Blog Posts

Discussion Seminars and Webinars

Innovation Grants

Talks

Workshop Papers

Venue Title
NeurIPS 2024 Satellite Sunroof: High-res Digital Surface Models and Roof Segmentation for Global Solar Mapping (Papers Track)
Abstract and authors: (click to expand)

Abstract: The transition to renewable energy, particularly solar, is key to mitigating climate change. Google's Solar API aids this transition by estimating solar potential from aerial imagery, but its impact is constrained by geographical coverage. This paper proposes expanding the API's reach using satellite imagery, enabling global solar potential assessment. We tackle challenges involved in building a Digital Surface Model (DSM) and roof instance segmentation from lower resolution and single oblique views using deep learning models. Our models, trained on aligned satellite and aerial datasets, produce 25cm DSMs and roof segments. With ~1m DSM MAE on buildings, ~5deg roof pitch error and ~56% IOU on roof segmentation, they significantly enhance the Solar API's potential to promote solar adoption.

Authors: Vishal Batchu (Google Research); Alex Wilson (Google); Betty Peng (Google); Carl Elkin (Google); Umangi Jain (University of Toronto); Christopher Arsdale (Google Research); Ross Goroshin (Google); Varun Gulshan (Google Research)
NeurIPS 2024 AI-Driven Predictive Modeling of PFAS Contamination in Aquatic Ecosystems: Exploring A Geospatial Approach (Papers Track)
Abstract and authors: (click to expand)

Abstract: Per- and polyfluoroalkyl substances (PFAS), a class of synthetic fluorinated compounds termed “forever chemicals”, have garnered significant attention due to their persistence, widespread environmental presence, bioaccumulative properties, and associated risks for human health. Their presence in aquatic ecosystems highlights the link between human activity and the hydrological cycle. They also disrupt aquatic life, interfere with gas exchange, and disturb the carbon cycle, contributing to greenhouse gas emissions and exacerbating climate change. Federal agencies, state governments and non-government research and public interest organizations have emphasized the need for documenting the sites and the extent of PFAS contamination. However, the time-consuming and expensive nature of data collection and analysis poses challenges. It hinders the rapid identification of locations at high risk of PFAS contamination, which may then require further sampling or remediation. To address this data limitation, our study leverages a novel geospatial dataset, machine learning models including frameworks such as Random Forest, IBM-NASA's Prithvi and UNet, and geospatial analysis to predict regions with high PFAS concentrations in surface water. Using fish data from the National Rivers and Streams Assessment (NRSA) dataset by the Environmental Protection Agency (EPA), our analysis suggests the potential value of machine learning based models for targeted deployment of sampling investigations and remediation efforts.

Authors: Jowaria Khan (University of Michigan); David Andrews (Environmental Working Group); Kaley Beins (Environmental Working Group); Sydney Evans (Environmental Working Group); Alexa Friedman (Environmental Working Group); Elizabeth Bondi-Kelly (MIT)
NeurIPS 2024 Harnessing AI for Wildfire Defense: An approach to Predict and Mitigate Global Fire Risk (Papers Track)
Abstract and authors: (click to expand)

Abstract: Wildfires pose a critical threat to wildlife, economies, properties, and human lives globally, making accurate risk assessment essential for effective management and mitigation. This study introduces a novel machine learning-based approach utilizing a Convolutional Neural Network (CNN) to evaluate wildfire risks across diverse ecosystems. Leveraging a comprehensive dataset of remote-sensed variables—including topography, vegetation health indicators, and climatic conditions—our model operates at a spatial resolution of 1000 meters per pixel, providing enhanced precision in predicting wildfire occurrences. The CNN outperforms state-of-the-art models, achieving a fire detection ratio of 0.82 and a no-fire detection ratio of 0.87. The results demonstrate that most dataset variables are crucial for accurate risk assessment, although some are non-essential. By integrating data from regions around the globe, this study underscores the feasibility and effectiveness of implementing globally scalable wildfire prediction tools.

Authors: Hassan Ashfaq (Ghulam Ishaq Khan Institute of Engineering Sciences and Technology)
NeurIPS 2024 Improving Power Plant CO2 Emission Estimation with Deep Learning and Satellite/Simulated Data (Papers Track)
Abstract and authors: (click to expand)

Abstract: CO2 emissions from power plants, as significant super emitters, contribute substantially to global warming. Accurate quantification of these emissions is crucial for effective climate mitigation strategies. While satellite-based plume inversion offers a promising approach, challenges arise from data limitations and the complexity of atmospheric conditions. This study addresses these challenges by (a) expanding the available dataset through the integration of NO2 data from Sentinel-5P, generating continuous XCO2 maps, and incorporating real satellite observations from OCO-2/3 for over 71 power plants in data-scarce regions; and (b) employing a customized U-Net model capable of handling diverse spatio-temporal resolutions for emission rate estimation. Our results demonstrate significant improvements in emission rate accuracy compared to previous methods [11]. By leveraging this enhanced approach, we can enable near real-time, precise quantification of major CO2 emission sources, supporting environmental protection initiatives and informing regulatory frameworks.

Authors: Dibyabha Deb (Manipal Institute of Technology); Kamal Das (IBM Research)
NeurIPS 2024 Methane SatMapper: Methane Detection from Satellite Imagery Using Hyperspectral Transformer (Papers Track)
Abstract and authors: (click to expand)

Abstract: Methane (CH4) plays a critical role in accelerating global climate change, and recent advancements using Sentinel-2 satellite imagery have demonstrated potential in detecting and quantifying significant methane emissions. However, existing approaches often rely on temporal analysis of shortwave-infrared spectra, assuming consistent ground conditions and prior knowledge of methane-free periods, which can lead to errors and limit scalability. To overcome these challenges, we present Methane SatMapper, an innovative end-to-end spectral transformer model specifically designed to accurately identify and quantify methane plumes. Our model introduces two novel modules: one that identifies potential methane emission sites by analyzing solar radiation absorption in the spectral domain and another that localizes and quantifies methane plumes without the need for temporal data. By utilizing all 12 spectral channels of Sentinel-2 imagery, our architecture effectively estimates ground terrain and detects methane emissions, providing enhanced robustness to variable ground conditions and increased computational efficiency by eliminating the need for historical time-series data. Primary evaluations confirm that Methane SatMapper delivers precise and reliable methane detection, addressing key limitations in scalability and temporal dependence.

Authors: Satish Kumar (University of California, Santa Barbara); ASM Iftekhar (Microsoft); Bowen Zhang (University Of California, Santa Barbara); Richard Sserunjogi (Makerere University); Mehan Jayasuriya (Mozilla Technology Foundation)
NeurIPS 2024 DivShift: Exploring Domain-Specific Distribution Shift in Large-Scale, Volunteer-Collected Biodiversity Datasets (Papers Track)
Abstract and authors: (click to expand)

Abstract: Climate change is negatively impacting the world's biodiversity. To build automated systems to monitor these negative biodiversity impacts, large-scale, volunteer-collected datasets like iNaturalist are built from community-identified, natural imagery. However, such volunteer-based data are opportunistic and lack a structured sampling strategy, resulting in geographic, temporal, observation quality, and socioeconomic, biases that stymie uptake of these models for downstream biodiversity monitoring tasks. Here we introduce DivShift North American West Coast (DivShift-NAWC), a curated dataset of almost 8 million iNaturalist plant images across the western coast of North America, for exploring the effects of these biases on deep learning model performance. We compare model performance across four known biases and observe that they indeed confound model performance. We suggest practical strategies for curating datasets to train deep learning models for monitoring climate change's impacts on the world's biodiversity.

Authors: Elena Sierra (Stanford University); Lauren Gillespie (Stanford University); Salim Soltani (University of Freiburg); Moisés Expósito-Alonso (University of California, Berkeley); Teja Kattenborn (University of Freiburg)
NeurIPS 2024 Wildflower Monitoring with Expert-annotated Images and Flowering Phenology (Papers Track)
Abstract and authors: (click to expand)

Abstract: Understanding biodiversity trends is essential for preservation policy planning, and advanced computer vision solutions now enable large-scale automated monitoring for many biodiversity use cases. Wildflower monitoring, in particular, presents unique challenges. Visual similarities in shape and color may exist between different species, while flowers within a species may have significant visual differences. Moreover, flowers follow a growth cycle and look distinctly different over the year, while different species flower at different times of the year. Having access to flowering phenology, more accurate predictions may be made. We propose a novel multi-modal wildflower monitoring task to better identify species, levering both expert-annotated wildflower images and flowering phenology estimates. Moreover, we benchmark several state-of-the-art models using two groups of common wildflower species that have high inter-class similarity, and show that this multi-modal approach significantly outperforms image-only baselines. With this work, we aim to encourage the development of standards for automated wildflower monitoring as a step towards bending the curve of biodiversity loss. The data and the code are publicly available https://georgianagmanolache.github.io/wildflowerpower/

Authors: Georgiana Manolache (Fontys University of Applied Science); Gerard Schouten (Fontys University of Applied Sciences)
NeurIPS 2024 Light-weight geospatial model for global deforestation attribution (Papers Track)
Abstract and authors: (click to expand)

Abstract: Forests are in decline worldwide and it is critical to attribute forest cover loss to its causes. We gathered a curated global dataset of all forest loss drivers and developed a neural network model to recognize the main drivers of deforestation or forest degradation at 1-km scale. Using remote sensing satellite data together with ancillary biophysical and socioeconomic data the model estimates the dominant drivers of forest loss from 2001 to 2022. Using a relatively light-weight geospatial model allowed us to to train a single world-wide model. We generated a global map of drivers of forest loss that is being validated, and present the first insights such data can provide.

Authors: Anton Raichuk (Google); Michelle Sims (WRI); Radost Stanimirova (WRI); Maxim Neumann (Google)
NeurIPS 2024 Multi-Source Temporal Attention Network for Precipitation Nowcasting (Papers Track) Best Pathway to Impact
Abstract and authors: (click to expand)

Abstract: Precipitation nowcasting is crucial across various industries and plays a significant role in mitigating and adapting to climate change. We introduce an efficient deep learning model for precipitation nowcasting, capable of predicting rainfall up to 8 hours in advance with greater accuracy than existing operational physics-based and extrapolation-based models. Our model leverages multi-source meteorological data and physics-based forecasts to deliver high-resolution predictions in both time and space. It captures complex spatio-temporal dynamics through temporal attention networks and is optimized using data quality maps and dynamic thresholds. Experiments demonstrate that our model outperforms state-of-the-art, and highlight its potential for fast reliable responses to evolving weather conditions.

Authors: Rafael Pablos Sarabia (Aarhus University & Cordulus); Joachim Nyborg (Cordulus); Morten Birk (Cordulus); Jeppe Liborius Sjørup (Cordulus); Anders Lillevang Vesterholt (Cordulus); Ira Assent (Aarhus University)
NeurIPS 2024 Multi-scale decomposition of sea surface height snapshots using machine learning (Papers Track)
Abstract and authors: (click to expand)

Abstract: Knowledge of ocean circulation is important for understanding and predicting weather and climate, and managing the blue economy. This circulation can be estimated through Sea Surface Height (SSH) observations, but requires decomposing the SSH into contributions from balanced and unbalanced motions (BMs and UBMs). This decomposition is particularly pertinent for the novel SWOT satellite, which measures SSH at an unprecedented spatial resolution. Specifically, the requirement, and the goal of this work, is to decompose instantaneous SSH into BMs and UBMs. While a few studies using deep learning (DL) approaches have shown promise in framing this decomposition as an image-to-image translation task, these models struggle to work well across a wide range of spatial scales and require extensive training data, which is scarce in this domain. These challenges are not unique to our task, and pervade many problems requiring multi-scale fidelity. We show that these challenges can be addressed by using zero-phase component analysis (ZCA) whitening and data augmentation; making this a viable option for SSH decomposition across scales.

Authors: Yue Wang (Columbia University); Jingwen Lyu (Columbia University); Chris Pedersen (NYU); Spencer Jones (Texas A&M University); Dhruv Balwada (Columbia University)
NeurIPS 2024 CanadaFire2023: Burned Area Mapping Datasets and Benchmarks for Canadian Wildfires in 2023 (Papers Track)
Abstract and authors: (click to expand)

Abstract: In 2023, wildfires burned record-breaking areas in Canada, resulting in significant carbon loss, exacerbating climate change, and underscoring the need for relevant datasets and machine learning methods for effective and efficient analysis. To understand the fire development processes and assess the climate impact of this natural disaster, burned area mapping datasets are essential for generating high-quality burned scar maps, enabling a comprehensive analysis of the 2023 wildfires, particularly given the vast expanse of Canada. To this end, we propose the CanadaFire2023 dataset, which includes burned area mapping data collected from multiple satellite platforms, namely, Landsat-8, Landsat-9, and Sentinel-2, specifically focused on these wildfires in the recorded history of Canada. To our knowledge, this is the first dataset specifically focused on burned area detection related to the unprecedented 2023 Canadian wildfires, using individual satellite imagery. We also trained four deep learning models—FCN, U-Net, multiscale ResNet, and SegFormer—for burned area mapping and evaluated the mapping performance using binary segmentation metrics, demonstrating that these datasets can serve as benchmarks for the research community studying wildfires and their environmental consequences. The CanadaFire2023 dataset could facilitate downstream applications such as disaster management, carbon emission estimation, and climate change mitigation.

Authors: Zilong Zhong (McMaster University); Alemu Gonsamo (McMaster University)
NeurIPS 2024 Scalable and interpretable deforestation detection in the Amazon rainforest (Papers Track)
Abstract and authors: (click to expand)

Abstract: Deforestation of the Amazon rainforest is a major contributor to climate change, as it is a crucial precipitation regulator, as well as a large natural carbon reserve. While there have been efforts to create real-time algorithms for deforestation detection, they are oftentimes not accurate or interpretable. We leverage multiple input signals, such as satellite imagery, time-series of deforestation indices and scalar measures, to create a single deep learning model that is both interpretable and accurate. We employ a novel dataset with millions of annotated images of the Brazilian Amazon to train our model, as well as class activation mappings to investigate the added value of interpretability in this context.

Authors: Rodrigo Schuller (IMPA); Francisco Ganacim (IMPA); Paulo Orenstein (IMPA)
NeurIPS 2024 Estimating atmospheric variables from Digital Typhoon Satellite Images via Conditional Denoising Diffusion Models (Papers Track)
Abstract and authors: (click to expand)

Abstract: This study explores the application of diffusion models in the field of typhoons, predicting multiple ERA5 meteorological variables simultaneously from Digital Typhoon satellite images. The focus of this study is taken to be Taiwan, an area very vulnerable to typhoons. By comparing the performance of Conditional Denoising Diffusion Probability Model (CDDPM) with Convolutional Neural Networks (CNN) and Squeeze-and-Excitation Networks (SENet), results suggest that the CDDPM performs best in generating accurate and realistic meteorological data. Specifically, CDDPM achieved a PSNR of 32.807, which is approximately 7.9% higher than CNN and 5.5% higher than SENet. Furthermore, CDDPM recorded an RMSE of 0.032, showing a 11.1% improvement over CNN and 8.6% improvement over SENet. A key application of this research can be for imputation purposes in missing meteorological datasets and generate additional high-quality meteorological data using satellite images. It is hoped that the results of this analysis will enable more robust and detailed forecasting, reducing the impact of severe weather events on vulnerable regions

Authors: Zhangyue Ling (Imperial College London); Pritthijit Nath (University Of Cambridge); Cesar Quilodran-Casas (Imperial College London)
NeurIPS 2024 Multi-branch Spatio-Temporal Graph Neural Network For Efficient Ice Layer Thickness Prediction (Papers Track) Overall Best Paper
Abstract and authors: (click to expand)

Abstract: Understanding spatio-temporal patterns in polar ice layers is essential for tracking changes in ice sheet balance and assessing ice dynamics. While convolutional neural networks are widely used in learning ice layer patterns from raw echogram images captured by airborne snow radar sensors, noise in the echogram images prevents researchers from getting high-quality results. Instead, we focus on geometric deep learning using graph neural networks, aiming to build a spatio-temporal graph neural network that learns from thickness information of the top ice layers and predicts for deeper layers. In this paper, we developed a novel multi-branch spatio-temporal graph neural network that used the GraphSAGE framework for spatio features learning and a temporal convolution operation to capture temporal changes, enabling different branches of the network to be more specialized and focusing on a single learning task. We found that our proposed multi-branch network can consistently outperform the current fused spatio-temporal graph neural network in both accuracy and efficiency.

Authors: Zesheng Liu (Lehigh University); Maryam Rahnemoonfar (Lehigh University)
NeurIPS 2024 Resolution-Agnostic Transformer-based Climate Downscaling (Papers Track)
Abstract and authors: (click to expand)

Abstract: Understanding future weather changes at regional and local scales is crucial for planning and decision-making, particularly in the context of extreme weather events, as well as for broader applications in agriculture, insurance, and infrastructure development. However, the computational cost of downscaling Global Climate Models (GCMs) to the fine resolutions needed for such applications presents a significant barrier. Drawing on advancements in weather forecasting models, this study introduces a cost-efficient downscaling method using a pretrained Earth Vision Transformer (Earth ViT) model. Initially trained on ERA5 data to downscale from 50 km to 25 km resolution, the model is then tested on the higher resolution BARRA-SY dataset at 3 km resolutions. Remarkably, it performs well without additional training, demonstrating its ability to generalize across different resolutions. This approach holds promise for generating large ensembles of regional climate simulations by downscaling GCMs with varying input resolutions without incurring additional training costs. Ultimately, this method could provide more comprehensive estimates of potential future changes in key climate variables, aiding in effective planning for extreme weather events and climate change adaptation strategies.

Authors: Declan Curran (UNSW); Hira Saleem (UNSW); Sanaa Hobeichi (The University of New South Wales); Flora Salim (University of New South Wales)
NeurIPS 2024 Scalable Satellite Imagery Analysis: A Cascade Framework for Sparse Target Detection (Papers Track)
Abstract and authors: (click to expand)

Abstract: Remote sensing is a crucial tool for monitoring events affecting climate change, such as tracking forest loss, identifying pollution sources, and monitoring the deployment of renewable energy infrastructure. However, applying state-of-the-art deep learning models to monitor the entire Earth is expensive. In this paper, we propose a cascade framework to reduce this cost: we apply a small MLP on precomputed embeddings of each image patch to serve as a preliminary filter, identifying key patches that warrant further examination by more resource-intensive deep models. Our approach reduces per-task inference runtime by 5x with a <1% impact on accuracy. By reducing inference cost, our method enables nonprofits and other organizations with limited resources to scale monitoring efforts to more environmental and conservation applications.

Authors: Arvind Manivannan (University of Washington); Tarun Narayanan Venkatachalam (Allen Institute for AI); Yanlin Huang (University of Washington); Favyen Bastani (Allen Institute for AI)
NeurIPS 2024 Estimating Greenland Ice Sheet Dynamics Using Neural Operators (Papers Track)
Abstract and authors: (click to expand)

Abstract: This study investigates the application of neural operators, particularly the Fourier Neural Operator (FNO), to enhance the modeling accuracy of the Greenland ice sheet's internal layers. Traditional computational methods often struggle to capture the complex spatial patterns of ice sheets adequately. Our approach leverages a radar image dataset to evaluate the effectiveness of neural operators in comparison to traditional neural network models. The FNO outperformed other models in key performance metrics, such as pixel error and layer accuracy. The results highlight the potential of neural operators to significantly improve the computational efficiency and accuracy of environmental models, paving the way for their broader application in cryospheric science and climate change research.

Authors: Maryam Rahnemoonfar (Lehigh University); Heling Wang (Lehigh University)
NeurIPS 2024 Mamba MethaneMapper: State Space Model for Methane Detection from Hyperspectral Imagery (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Methane (CH4) is the chief contributor to global climate change. Recent advancements in AI-based image processing have paved the way for innovative approaches for the detection of methane using hyper-spectral imagery. Existing methods, while effective, often come with high computational demands and associated costs that can limit their practical applications. Addressing these limitations, we propose the Mamba MethaneMapper (MMM), a cost-effective and efficient AI-driven solution designed to enhance methane detection capabilities in hyper-spectral images. MMM will incorporate two key innovations that collectively improve performance while managing costs. First, we will utilize a gpu-aware state-space encoder, which optimizes the computational resources and efficiency of the system. Second, MMM will use an environment-sensitive module to prioritize image regions likely containing methane emissions, which are then analyzed by our efficient Mamba algorithm. This selective approach not only improves the accuracy of methane detection but also significantly reduces unnecessary computations and memory consumption.

Authors: Satish Kumar (University of California, Santa Barbara); ASM Iftekhar (Microsoft); Kaikai Liu (University of California, Santa Barbara); Bowen Zhang (University Of California, Santa Barbara); Mehan Jayasuriya (Mozilla)
NeurIPS 2024 A Novel Approach to Assessing the Efficacy of Policy-driven Urban Energy Demand Reduction via Proximal Infrared Remote Sensing (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Byproducts of energy use are among the leading drivers of emissions that can lead to climate change. Heating and cooling represent the largest source of energy use in urban infrastructure; thus, significant policy efforts have been implemented to reduce energy demand. Estimates of the efficacy of these policies are often difficult as they rely on self-reporting or detailed utility data, which is not available broadly and challenging to collect in areas where utility infrastructures are unavailable. Buildings' heating, ventilation, and air conditioning (HVAC) can be used to measure energy consumption from the demand side to evaluate end use and potential energy policy-induced reductions. Therefore, we are proposing a novel, non-intrusive method of energy use monitoring that uses proximal infrared remote sensing of building envelopes to find patterns of heating and cooling use. Our results show that the technique can discern on-and-off patterns of externally facing cooling units (ACs) in a dense urban scene at distances up to ~0.5 miles. We have tested multiple cases of residential buildings in New York City. We collected ~337,000 infrared images at 10-second intervals of the buildings’ facades taken from June 11 to July 18, 2018, between 500 m and 1 km. We have applied computer vision techniques to that sequence of images to identify exterior-venting HVAC units and generate their infrared time series, which correlates directly with their temperature. We determine each AC unit's on/off transitions using a one-dimensional edge detection algorithm and identify aggregated and disaggregated patterns of end-user behavior that are direct proxies for total heating and cooling use. Finally, we demonstrate that this new methodology is sufficiently sensitive to assess policy interventions to reduce daily peak demand effectively.

Authors: Kyungmin Lee (University of Delaware); Gregory Dobler (University of Delaware)
NeurIPS 2024 Composing Open-domain Vision with RAG for Ocean Monitoring and Conservation (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Climate change's destruction of marine biodiversity is threatening communities and economies around the world which rely on healthy oceans for their livelihoods. The challenge of applying computer vision to niche, real-world domains such as ocean conservation lies in the dynamic and diverse environments where traditional top-down learning struggle with long-tailed distributions, generalization, and domain transfer. Scalable species identification for ocean monitoring is particularly difficult due to the need to adapt models to new environments and identify rare or unseen species. To overcome these limitations, we propose leveraging bottom-up, open-domain learning frameworks–specifically vision-language models (VLMs) combined with retrieval-augmented generation (RAG)–as a resilient, scalable solution for image and video analysis in marine applications. We validate this approach through a preliminary application in classifying fish from video onboard fishing vessels, demonstrating impressive emergent retrieval and prediction capabilities without domain-specific training or knowledge of the task itself.

Authors: Sepand Dyanatkar (OnDeck Fisheries AI); Angran Li (OnDeck Fisheries AI); Alexander Dungate (OnDeck Fisheries AI)
NeurIPS 2024 Seeing Inside Buildings: Leveraging Generative AI and Multimodal Data to Automate Building Material Audits (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Building retrofits and deconstruction projects are expected to surpass new construction jobs globally, putting pressure on architects and developers to sustainably reuse buildings and their materials (World Economic Forum). However, building reuse and material upcycling is hindered by the complexity of building material audits, which require expensive tests, site visits, and detailed data that is often missing for old structures. We present a Generative AI approach to predict the structural and material make-up of existing buildings from multimodal geospatial, technical, and cadaster data. Leveraging a dataset of 100 buildings across the United States with corresponding building 3D scans, geolocation, and construction data, we demonstrate the capability of a stable diffusion model to reliably predict structural diagrams for subsequent estimation of material contents. This process also offers designers actionable potential material reuse data to streamline and accelerate circularity for existing building design.

Authors: Nikita Klimenko (MIT); James Stoddart (Autodesk, Inc.); Lorenzo Villaggi (Autodesk, Inc.); Dale Zhao (Autodesk, Inc.)
NeurIPS 2024 Flood Prediction in Kenya - Leveraging Pre-Trained Models to Generate More Validation Data in Sparse Observation Settings (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Kenya has lacked a national flood risk management framework and also has sparse flood observation data, which makes developing deep learning flood prediction models on a national scale challenging. Flood prediction models are critical to operationalize Early Warning Systems (EWS). We propose two different models to feed into an EWS. The first model will leverage statistical machine learning approaches to predict flood or no flood events on a 0.25 x 0.25 degree scale (approximately 30 km x 30 km in Kenya) using ERA5 features as well as land cover and Digital Terrain Model (DTM) data. This first model will also be used to create a baseline prediction benchmark across the entire country of Kenya. The second model will leverage pre-trained remote sensing based models to generate segmented flood or no flood data on a fine spatial scale. This will increase the number of validation points by a factor of 1000, which opens the door to deep learning approaches to predict flood or no-flood events on a 30 meter x 30 meter spatial scale. We hope that this approach of leveraging pre-trained models to generate fine scale validation data can eventually be used widely in other extreme climate event forecasting scenarios given the scarcity of historical extreme climate events compared to normal weather events.

Authors: Alim Karimi (Self / Purdue University); David Quispe (University of Toronto); Hammed Akande (Concordia University); Nicole Mong'are (Athi Water Works Development Agency); Valerie Brosnan (Mitga Solutions); Asbina Baral (Ministry of Education, Science and Technology)
NeurIPS 2024 Towards more efficient agricultural practices via transformer-based crop type classification (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Machine learning has great potential to increase crop production and resilience to climate change. Accurate maps of where crops are grown are a key input to several downstream policy and research applications. This proposal presents preliminary work showing that it is possible to accurately classify crops from time series derived from Sentinel 1 and 2 satellite imagery in Mexico using a pixel-based binary crop/non-crop time series transformer model. We also find preliminary evidence that meta-learning approaches supplemented with data from similar agro-ecological zones may improve model performance. Due to these promising results, we propose further development of this method with the goal of accurate multi-class crop classification in Jalisco, Mexico via meta-learning with a dataset comprising similar agro-ecological zones.

Authors: Isabella Smythe (Columbia University); Eduardo Ulises Moya (Gobierno de Jalisco); Michael Smith (Aspia Space); Yazid Mikail (Climate Change AI); Daisy Ondwari (Kabarak University)
NeurIPS 2024 A Multimodal Causal Framework for Large-Scale Ecosystem Valuation: Application to Wetland Benefits for Flood Mitigation (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Climate change is poised to alter wetland ecosystems through changes in temperature and precipitation patterns, compounding the already pronounced influence of human-driven wetland development. In this context, policymakers and environmental managers would benefit from accurate wetland valuations to guide their decision-making, as their choices regarding this critical natural resource directly impact flood mitigation efforts, biodiversity conservation, and economic activity. This paper introduces a novel multimodal causal framework for producing location-specific ecosystem valuations at a national scale to be used in cost-benefit policy analysis. It leverages recent advances in estimating heterogeneous treatment effects to flexibly determine how the expected impact of ecosystem-level changes---such as wetland loss via development---varies conditional on high-dimensional and multimodal measures that characterize the complex interactions between human and natural systems such as aerial satellite imagery, weather sequence data, land cover classifications, and water surface networks. From this effort, we aim to create a national database of location-specific wetland valuations in an approach that can be readily extended in estimating the effect of other interventions on ecosystems. We also plan to generate open-source feature embeddings for each U.S. wetland, embeddings that can be used to address other climate-related causal questions as well.

Authors: Hannah Druckenmiller (Caltech); Georgia Gkioxari (Caltech); Connor Jerzak (University of Texas at Austin); SayedMorteza Malaekeh (University of Texas at Austin)
NeurIPS 2024 Stubble (Crop Residue) Burning Detection Through Satellite Images Using Geospatial Foundation Model: A Case Study in Punjab, India (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Stubble burning is a significant environmental challenge globally, with widespread implications for air quality, greenhouse gases emission, soil degradation and health issues. This practice is particularly prevalent in agricultural regions across the world, though its impacts are notably severe in the northern India. This proposed work focuses on improving the detection of stubble (crop residue) burning in Punjab (India), through using the geospatial foundation model. This study leverages a series of satellite images where stubble burning incidents have been documented. By refining the model to incorporate local environmental factors, this study aims to improve the accuracy of stubble burning detection, thereby contributing to a scalable solution for real-time monitoring and intervention in crop residue burning practices worldwide.

Authors: Rajiv Ranjan (Plaksha University); Ying-Jung Chen (Georgia Institute of Technology); Shashank Tamaskar (Plaksha University); Anupam Sobti (IIT Delhi)
ICLR 2024 Structured spectral reconstruction for scalable soil organic carbon inference (Papers Track)
Abstract and authors: (click to expand)

Abstract: Measuring soil organic carbon (SOC) inexpensively and accurately is crucial for soil health monitoring and agricultural decarbonization. Hyperspectral imaging is commonly evaluated as an inexpensive alternative to dry combustion for SOC measurement, but existing end-to-end approaches trained to predict SOC content from spectral data frequently fail to generalize when applied outside of their ground-truth geographic sampling distributions. Using stratified data from the USDA Rapid Carbon Assessment (RaCA), we demonstrate a method to improve model generalization out-of-distribution by training SOC regression alongside models that reconstruct input spectra. Because hyperspectra can be collected from remote platforms such as drones and satellites, this approach raises the possibility of using large hyperspectral Earth observation datasets to transfer SOC inference models to remote geographies where geographically-dense ground-truth data collection may be expensive or impossible. By replacing the decoder with a simple physics-informed model, we also learn an interpretable spectral signature of SOC, confirming its dark hue and expected reflectance troughs. Finally, we show that catastrophic generalization failures can be better addressed with these architectures by fine-tuning on large quantities of hyperspectral data.

Authors: Evan A Coleman (MIT); Sujay Nair (Georgia Institute of Technology); Xinyi Zeng (Coho Climate Advisors); Elsa Olivetti (MIT Department of Materials Science & Engineering)
ICLR 2024 Categorization of Meteorological Data by Contrastive Clustering (Papers Track)
Abstract and authors: (click to expand)

Abstract: Visualized ceilometer backscattering data, displaying meteorological phenomena like clouds, precipitation, and aerosols, is mostly analyzed manually by meteorology experts. In this work, we present an approach for the categorization of backscattering data using a contrastive clustering approach, incorporating image and spatiotemporal information into the model. We show that our approach leads to meteorologically meaningful clusters, opening the door to the automatic categorization of ceilometer data, and how our work could potentially create insights in the field of climate science.

Authors: Michael Dammann (HAW Hamburg); Ina Mattis (DWD); Michael Neitzke (HAW Hamburg); Ralf Möller (University of Lübeck)
ICLR 2024 Extreme Precipitation Nowcasting using Transformer-based generative models (Papers Track)
Abstract and authors: (click to expand)

Abstract: This paper presents an innovative approach to extreme precipitation nowcasting by employing Transformer-based generative models, specifically VideoGPT with Extreme Value Loss (EVL) regularization. Leveraging a comprehensive dataset from the Royal Netherlands Meteorological Institute (KNMI), our study focuses on predicting short-term precipitation with high accuracy. We introduce a novel method for computing EVL without assuming fixed extreme representations, addressing the limitations of current models in capturing extreme weather events. We present both qualitative and quantitative analyses, demonstrating the superior performance of the proposed VideoGPT-EVL in generating accurate precipitation forecasts, especially when dealing with extreme precipitation events.

Authors: Cristian Meo (TUDelft); Mircea T Lica (Delft University of Technology); Ankush Roy (TUDelft); Zeina Boucher (TUDelft); Junzhe Yin (TUDelft); Yanbo Wang (Delft University of Technology); Ruben Imhoff (Deltares); Remko Uijlenhoet (TUDelft); Justin Dauwels (TU Delft)
ICLR 2024 GeoFormer: A Vision and Sequence Transformer-based Approach for Greenhouse Gas Monitoring (Papers Track)
Abstract and authors: (click to expand)

Abstract: Air pollution represents a pivotal environmental challenge globally, playing a major role in climate change via greenhouse gas emissions and negatively affecting the health of billions. However, predicting the spatial and temporal patterns of pollutants remains challenging. The scarcity of ground-based monitoring facilities and the dependency of air pollution modeling on comprehensive datasets, often inaccessible for numerous areas, complicate this issue. In this work, we introduce GeoFormer, a compact model that combines a vision transformer module with a highly efficient time-series transformer module to predict surface-level nitrogen dioxide (NO2) concentrations from Sentinel-5P satellite imagery. We train the proposed model to predict surface-level NO2 measurements using a dataset we constructed with Sentinel-5P images of ground-level monitoring stations, and their corresponding NO2concentration readings. The proposed model attains high accuracy (MAE 5.65), demonstrating the efficacy of combining vision and time-series transformer architectures to harness satellite-derived data for enhanced GHG emission insights, proving instrumental in advancing climate change monitoring and emission regulation efforts globally.

Authors: Madhav Khirwar (Independent)
ICLR 2024 Black carbon plumes from gas flaring in North Africa identified from multi-spectral imagery with deep learning (Papers Track)
Abstract and authors: (click to expand)

Abstract: Black carbon (BC) is an important pollutant aerosol emitted by numerous human activities, including gas flaring. Improper combustion in flaring activities can release large amounts of BC, which is harmful to human health and has a strong climate warming effect. To our knowledge, no study has ever directly monitored BC emissions from satellite imagery. Previous works quantified BC emissions indirectly, by applying emission coefficients to flaring volumes estimated from satellite imagery. Here, we develop a deep learning framework and apply it to Sentinel-2 imagery over North Africa during 2022 to detect and quantify BC emissions from gas flaring. We find that BC emissions in this region amount to about 1 million tCO2eq, or 1 million passenger cars, more than a quarter of which are due to 10 sites alone. This work demonstrates the operational monitoring of BC emissions from flaring, a key step in implementing effective mitigation policies to reduce the climate impact of oil and gas operations.

Authors: Alexandre Tuel (Galeio); Thomas Kerdreux (INRIA/ ENS); Louis THIRY (ENS Paris)
ICLR 2024 Towards Downscaling Global AOD with Machine Learning (Papers Track)
Abstract and authors: (click to expand)

Abstract: Poor air quality represents a significant threat to human health, especially in urban areas. To improve forecasts of air pollutant mass concentrations, there is a need for high-resolution Aerosol Optical Depth (AOD) forecasts as proxy. However, current General Circulation Model (GCM) forecasts of AOD suffer from limited spatial resolution, making it difficult to accurately represent the substantial variability exhibited by AOD at the local scale. To address this, a deep residual convolutional neural network (ResNet) is evaluated for the GCM to local scale downscaling of low-resolution global AOD retrievals, outperforming a non-trainable interpolation baseline. We explore the bias correction potential of our ResNet using global reanalysis data, evaluating it against in-situ AOD observations. The improved resolution from our ResNet can assist in the study of local AOD variations.

Authors: Josh Millar (Imperial College London); Paula Harder (Mila); Lilli J Freischem (University of Oxford); Philipp Weiss (University of Oxford); Philip Stier (University of Oxford)
ICLR 2024 Analyzing the secondary wastewater-treatment process using Faster R-CNN and YOLOv5 object detection algorithms (Papers Track)
Abstract and authors: (click to expand)

Abstract: The activated sludge (AS) process is the most common type of secondary wastewater treatment, applied worldwide. Due to the complexity of microbial communities, imbalances between the different types of bacteria may occur and disturb the process, with pronounced economical and environmental consequences. Microscopic inspection of the morphology of flocs and microorganisms provides key information on AS properties and function. This is a time-consuming, highly skilled, and expensive process that is not readily available in all locations. Thus, most wastewater-treatment plants do not carry out this essential analysis, resulting in frequent operational faults. In this study, we develop a novel deep learning (DL) object detection algorithm to analyze and monitor the AS process based on a unique microscopic image database of flocs and microorganisms. Specifically, we applied YOLOv5 and Faster R-CNN algorithms as tools for segmentation and object detection to analyze the wastewater. The mean average precision (mAP) of the YOLOv5 was 0.67, outperforming the Faster R-CNN by 15%. Histogram equalization preprocessing of both bright-field and phase-contrast images significantly improved the results of the algorithm in all classes. In the case of YOLOv5, the mAP increased by 16.67%, to 0.77, where the AP of protozoa, filaments, and open floc classes outperformed the previous model by over 20%. These results demonstrate the potential of leveraging DL algorithms to enhance the analysis and monitoring of WWTPs in an affordable manner, consequently reducing environmental pollution caused by contaminated effluent. The fundamental challenge addressed herein has important global relevance, especially in an era in which the demand for high-quality wastewater reuse is expected to increase dramatically.

Authors: Offir Inbar (Tel-Aviv University); Moni Shahar (Tel Aviv University); Jacob Gidron (Tel-Aviv University); Ido Cohen (Tel-Aviv University); Dror Avisar (Tel-Aviv University)
ICLR 2024 Verifying Practices of Regenerative Agriculture: African Smallholder Farmer Dataset for Remote Sensing and Machine Learning (Papers Track)
Abstract and authors: (click to expand)

Abstract: Despite Africa’s contribution to global greenhouse gas (GHG) emissions being only a few %, the continent experiences the harshest impacts, particularly within its food production systems. Regenerative agriculture is receiving a large amount of attention as a method to strengthen both food security and climate change resilience in Africa. For practicing regenerative agriculture, carbon credits are issued, but verifying the methodologies on a large scale is one of the challenging points in popularizing it. In this paper, we provide a comprehensive dataset on regenerative agriculture in sub-Saharan Africa. The dataset has field polygon information and is labeled with several types of regenerative agriculture methodologies. The dataset can be applied to local site analysis, classification, and detection of regenerative agriculture with remote sensing and machine learning. We also highlight several machine learning models and summarize the baseline results on our dataset. We believe that by providing this dataset, we can contribute to the establishment of verification methods for regenerative agriculture. The dataset can be downloaded from https://osf.io/xgp9m/.

Authors: Yohei Nakayama (Degas Ltd); Grace Antwi (Degas Ltd); Seiko Shirasaka (Keio University)
ICLR 2024 Semi-Supervised Domain Adaptation for Wildfire Detection (Papers Track)
Abstract and authors: (click to expand)

Abstract: Recently, both the frequency and intensity of wildfires have increased worldwide, primarily due to climate change. In this paper, we propose a novel protocol for wildfire detection, leveraging semi-supervised Domain Adaptation for object detection, accompanied by a corresponding dataset designed for use by both academics and industries. Our dataset encompasses 30 times more diverse labeled scenes than the current largest benchmark wildfire dataset, HPWREN, and introduces a new labeling policy for wildfire detection. Inspired by coordconv., we propose a robust baseline, Location-Aware Object Detection for Semi-Supervised Domain Adaptation (LADA), utilizing a teacher-student based self-supervised Domain Adaptation framework capable of extracting translational variance features characteristic of wildfires. Our framework significantly outperforms the existing baseline by a notable margin of 3.8\%p in mean Average Precision on the HPWREN wildfire dataset.

Authors: Joo Young Jang (Alchera); Youngseo Cha (Alchera); Jisu Kim (Alchera); SooHyung Lee (Alchera); Geonu Lee (Alchera); Minkook Cho (Alchera); Young Hwang (Alchera); Nojun Kwak (Seoul National University)
ICLR 2024 PostRainBench: A comprehensive benchmark and a new model for precipitation forecasting (Papers Track)
Abstract and authors: (click to expand)

Abstract: Accurate precipitation forecasting is a vital challenge of societal importance. Though data-driven approaches have emerged as a widely used solution, solely relying on data-driven approaches has limitations in modeling the underlying physics, making accurate predictions difficult. We focus on the Numerical Weather Prediction (NWP) post-processing based precipitation forecasting task to couple Machine Learning techniques with traditional NWP. This task remains challenging due to the imbalanced precipitation data and complex relationships between multiple meteorological variables. To address these limitations, we introduce the PostRainBench, a comprehensive multi-variable NWP post-processing benchmark, and CAMT, a simple yet effective Channel Attention Enhanced Multi-task Learning framework with a specially designed weighted loss function. Extensive experimental results on the proposed benchmark show that our method outperforms state-of-the-art methods by 6.3%, 4.7%, and 26.8% in rain CSI and improvements of 15.6%, 17.4%, and 31.8% over NWP predictions in heavy rain CSI on respective datasets. Most notably, our model is the first deep learning-based method to outperform NWP approaches in heavy rain conditions. These results highlight the potential impact of our model in reducing the severe consequences of extreme rainfall events. Our datasets and code are available at https://github.com/yyyujintang/PostRainBench.

Authors: Yujin Tang (The Hong Kong University of Science and Technology (Guangzhou)); Jiaming Zhou (Hong Kong University of Science and Technology (Guangzhou)); Xiang Pan (Nanjing University); Zeying Gong (Hong Kong University of Science and Technology (Guangzhou)); Junwei Liang (The Hong Kong University of Science and Technology (Guangzhou))
ICLR 2024 Forecasting regional PV power in Great Britain with a multi-modal late fusion network (Papers Track)
Abstract and authors: (click to expand)

Abstract: The ability to forecast solar photovoltaic (PV) power is important for grid balancing and reducing the CO2 intensity of electricity globally. The use of multi-modal data such as numerical weather predictions (NWPs) and satellite imagery can be harnessed to make more accurate PV forecasts. In this work, we propose a late fusion model which integrates two different NWP sources alongside satellite images to make 0-8 hour lead time forecasts for grid regions across Great Britain. We limit the model inputs to be reflective of those available in a live production system. We show how the different input data sources contribute to average error at each time horizon and compare against a simple baseline.

Authors: James Fulton (Open Climate Fix); Jacob Bieker (Open Climate Fix); Peter Dudfield (Open Climate Fix); Solomon Cotton (Open Climate Fix); Zakari Watts (Open Climate Fix); Jack Kelly (Open Climate Fix)
ICLR 2024 Global High Resolution CO2 monitoring using Super Resolution (Papers Track)
Abstract and authors: (click to expand)

Abstract: Monitoring Greenhouse Gases (GHGs) concentrations and emissions is essential to mitigate climate change. Thanks to the large amount of satellite data available, it is now possible to understand GHGs' behaviours at a broad scale. However, due to remote sensing devices technological limitations, the task of global high resolution (HR) monitoring remains an open problem. To avoid waiting for new missions and better data to be generated, it is therefore relevant to experiment with processing methods able to improve existing datasets. Our paper proposes to apply Super Resolution (SR), a Deep Learning (DL) approach commonly used in Computer Vision (CV), on global L3 satellite data. We produce a daily high resolution global CO2 dataset that opens the door for globally consistent point source monitoring.

Authors: Andrianirina Rakotoharisoa (Imperial College London); Rossella Arcucci (Imperial College London)
ICLR 2024 DiffObs: Generative Diffusion for Global Forecasting of Satellite Observations (Papers Track)
Abstract and authors: (click to expand)

Abstract: This work presents an autoregressive generative diffusion model (DiffObs) to predict the global evolution of daily precipitation, trained on a satellite observational product, and assessed with domain-specific diagnostics. The model is trained to probabilistically forecast day-ahead precipitation. Nonetheless, it is stable for multi-month rollouts, which reveal a qualitatively realistic superposition of convectively coupled wave modes in the tropics. Cross-spectral analysis confirms successful generation of low frequency variations associated with the Madden--Julian oscillation, which regulates most subseasonal to seasonal predictability in the observed atmosphere, and convectively coupled moist Kelvin waves with approximately correct dispersion relationships. Despite secondary issues and biases, the results affirm the potential for a next generation of global diffusion models trained on increasingly sparse, and increasingly direct and differentiated observations of the world, for practical applications in subseasonal and climate prediction.

Authors: Jason Stock (Colorado State University); Jaideep Pathak (NVIDIA Corporation); Yair Cohen (NVIDIA Corporation); Mike Pritchard (NVIDIA Corporation); Piyush Garg (NVIDIA); Dale Durran (NVIDIA Corporation); Morteza Mardani (NVIDIA Corporation); Noah D Brenowitz (NVIDIA)
ICLR 2024 Machine Learning for the Detection of Arctic Melt Ponds from Infrared Imagery (Papers Track)
Abstract and authors: (click to expand)

Abstract: Melt ponds are pools of water on Arctic summer sea ice that play an important role in the Arctic climate system. Retrieving their coverage is essential to better understand and predict the rapidly changing Arctic, but current data are limited. The goal of this study is to enhance melt pond data by developing a method that segments thermal infrared (TIR) airborne imagery into melt pond, sea ice, and ocean classes. Due to temporally and spatially varying surface temperatures, we use a data-driven deep learning approach to solve this task. We adapt and fine-tune AutoSAM, a Segment Anything-based segmentation model. We make the code, data, and models available online.

Authors: Marlena Reil (University of Osnabrück and University of Bremen, Institute of Environmental Physics); Gunnar Spreen (University of Bremen, Institute of Environmental Physics); Marcus Huntemann (University of Bremen, Institute of Environmental Physics); Lena Buth (Alfred Wegener Institute); Dennis Wilson (University of Toulouse, ISAE-Supaero)
ICLR 2024 Predicting Species Occurrence Patterns from Partial Observations (Papers Track)
Abstract and authors: (click to expand)

Abstract: To address the interlinked biodiversity and climate crises, we need an understanding of where species occur and how these patterns are changing. However, observational data on most species remains very limited, and the amount of data available varies greatly between taxonomic groups. We introduce the problem of predicting species occurrence patterns given (a) satellite imagery, and (b) known information on the occurrence of other species. To evaluate algorithms on this task, we introduce SatButterfly, a dataset of satellite images, environmental data and observational data for butterflies, which is designed to pair with the existing SatBird dataset of bird observational data. To address this task, we propose a general model, R-Tran, for predicting species occurrence patterns that enables the use of partial observational data wherever found. We find that R-Tran outperforms other methods in predicting species encounter rates with partial information both within a taxon (birds) and across taxa (birds and butterflies). Our approach opens new perspectives to leveraging insights from species with abundant data to other species with scarce data, by modelling the ecosystems in which they co-occur.

Authors: Hager Radi Abdelwahed (Mila: Quebec AI Institute); Mélisande Teng (Mila, Université de Montréal); David Rolnick (MIT)
ICLR 2024 Towards Scalable Deep Species Distribution Modelling using Global Remote Sensing (Papers Track)
Abstract and authors: (click to expand)

Abstract: Destruction of natural habitats and anthropogenic climate change are threatening biodiversity globally. Addressing this loss necessitates enhanced monitoring techniques to assess the impact of environmental shifts and to guide policy-making efforts. Species distribution models are crucial tools that predict species locations by interpolating observed field data with environmental information. We develop an improved, scalable method for species distribution modelling by proposing a dataset pipeline that incorporates global remote sensing imagery, land use classification data, environmental variables, and observation data, and utilising this with convolutional neural network (CNN) models to predict species presence at higher spatial and temporal resolutions than well-established species distribution modelling methods. We apply our approach to modelling Protea species distributions in the Cape Floristic Region of South Africa, demonstrating its performance in a region of high biodiversity. We train two CNN models and compare their performance to Maxent, a popular conventional species distribution modelling method. We find that the CNN models trained with remote sensing data outperform Maxent, underscoring the potential of our method as an effective and scalable solution for modelling species distribution.

Authors: Emily Morris (University of Cambridge); Anil Madhavapeddy (University of Cambridge); Sadiq Jaffer (University of Cambridge); David Coomes (University of Cambridge)
ICLR 2024 SkyImageNet: Towards a large-scale sky image dataset for solar power forecasting (Proposals Track)
Abstract and authors: (click to expand)

Abstract: The variability of solar photovoltaic (PV) output, particularly that caused by rapidly changing cloud dynamics, challenges the reliability of renewable energy systems. Solar forecasting based on cloud observations collected by ground-level sky cameras shows promising performance in anticipating short-term solar power fluctuations. However, current deep learning methods often rely on a single dataset with limited sample diversity for training, and thus generalize poorly to new locations and different sky conditions. Moreover, the lack of a standardized dataset hinders the consistent comparison of existing solar forecasting methods. To close these gaps, we propose to build a large-scale standardized sky image dataset --- SkyImageNet --- by assembling, harmonizing, and processing suitable open-source datasets collected in various geographical locations. An accompanying python package will be developed to streamline the process of utilizing SkyImageNet in a machine learning framework. We hope that the outcomes of this project will foster the development of more robust forecasting systems, advance the comparability of short-term solar forecasting model performances, and further facilitate the transition to the next generation of sustainable energy systems.

Authors: Yuhao Nie (Massachusetts Institute of Technology); Quentin Paletta (European Space Research Institute); Sherrie Wang (MIT)
ICLR 2024 Enabling Clean Energy Resilience with Machine Learning-Empowered Underground Hydrogen Storage (Proposals Track)
Abstract and authors: (click to expand)

Abstract: To address the urgent challenge of climate change, there is a critical need to transition away from fossil fuels towards sustainable energy systems, with renewable energy sources playing a pivotal role. However, the inherent variability of renewable energy, without effective storage solutions, often leads to imbalances between energy supply and demand. Underground Hydrogen Storage (UHS) emerges as a promising long-term storage solution to bridge this gap, yet its widespread implementation is impeded by the high computational costs associated with high fidelity UHS simulations. This paper introduces UHS from a data-driven perspective and outlines a roadmap for integrating machine learning into UHS, thereby facilitating the large-scale deployment of UHS.

Authors: Alvaro R Carbonero Gonzales (Los Alamos National Lab); Shaowen Mao (Los Alamos National Laboratory); Mohamed Mehana (Los Alamos National Lab)
ICLR 2024 One Prompt Fits All: Visual Prompt-Tuning for Remote Sensing Segmentation (Tutorials Track) Audience Choice
Abstract and authors: (click to expand)

Abstract: Image segmentation is crucial in climate change research for analyzing satellite imagery. This technique is vital for ecosystems mapping, natural disasters assessment, and urban and agricultural planning. The advent of vision-based foundational models like the Segment Anything Model (SAM) opens new avenues in climate research and remote sensing (RS). SAM can perform segmentation tasks on any object from manually-crafted prompts. However, the efficacy of SAM largely depends on the quality of these prompts. This issue is particularly pronounced with RS data, which are inherently complex. To use SAM for accurate segmentation at scale for RS, one would need to create complex prompts for each image, which typically involves selecting dozens of points. To address this, we introduce Prompt-Tuned SAM (PT-SAM), a method that minimizes the need for manual input through a trainable, lightweight prompt embedding. This embedding captures key semantic information for specific objects of interest that would be applicable to unseen images. Our approach merges the zero-shot generalization capabilities of the pre-trained SAM model with supervised learning. Importantly, the training process for the prompt embedding not only has minimal hardware requirements, allowing it to be conducted on a CPU, but it also requires only a small dataset. With PT-SAM, image segmentation on RS data can be performed at scale without human intervention, achieving accuracies comparable to those of human-designed prompts with SAM. For example, PT-SAM can be used for analyzing forest cover across vast areas, a key factor in understanding the impact of human activities on forests. Its capability to segment a multitude of images makes it ideal for monitoring widespread land-cover changes, providing deeper insights into urbanization. This tutorial will explore how to train and utilize PT-SAM for large-scale segmentation tasks, specifically focusing on training embeddings that capture forests, and buildings.

Authors: Marshall Wang (Vector Institute); John Willes (Vector Institute); Deval Pandya (Vector Institute)
NeurIPS 2023 Augmenting Ground-Level PM2.5 Prediction via Kriging-Based Pseudo-Label Generation (Papers Track)
Abstract and authors: (click to expand)

Abstract: Fusing abundant satellite data with sparse ground measurements constitutes a major challenge in climate modeling. To address this, we propose a strategy to augment the training dataset by introducing unlabeled satellite images paired with pseudo-labels generated through a spatial interpolation technique known as ordinary kriging, thereby making full use of the available satellite data resources. We show that the proposed data augmentation strategy helps enhance the performance of the state-of-the-art convolutional neural network-random forest (CNN-RF) model by a reasonable amount, resulting in a noteworthy improvement in spatial correlation and a reduction in prediction error.

Authors: lei Duan (Duke University); Ziyang Jiang (Duke University); David Carlson (Duke University)
NeurIPS 2023 Can We Reliably Improve the Robustness to Image Acquisition of Remote Sensing of PV Systems? (Papers Track)
Abstract and authors: (click to expand)

Abstract: Photovoltaic (PV) energy is crucial for the decarbonization of energy systems. Due to the lack of centralized data, remote sensing of rooftop PV installations is the best option to monitor the evolution of the rooftop PV installed fleet at a regional scale. However, current techniques lack reliability and are notably sensitive to shifts in the acquisition conditions. To overcome this, we leverage the wavelet scale attribution method (WCAM), which decomposes a model's prediction in the space-scale domain. The WCAM enables us to assess on which scales the representation of a PV model rests and provides insights to derive methods that improve the robustness to acquisition conditions, thus increasing trust in deep learning systems to encourage their use for the safe integration of clean energy in electric systems.

Authors: Gabriel Kasmi (Mines Paris - PSL); Laurent Dubus (RTE France); Yves-Marie Saint-Drenan (Mines Paris - PSL); Philippe Blanc (Mines Paris - PSL)
NeurIPS 2023 Enhancing Data Center Sustainability with a 3D CNN-Based CFD Surrogate Model (Papers Track)
Abstract and authors: (click to expand)

Abstract: Thermal Computational Fluid Dynamics (CFD) models analyze airflow and heat distribution in data centers, but their complex computations hinder efficient energy-saving optimizations for sustainability. We introduce a new method to acquire data and model 3D Convolutional Neural Network (CNN) based surrogates for CFDs, which predict a data center's temperature distribution based on server workload, HVAC airflow rate, and temperature set points. The surrogate model's predictions are highly accurate, with a mean absolute error of 0.31°C compared to CFD-based ground truth temperatures. The surrogate model is three orders of magnitude faster than CFDs in generating the temperature maps for similar-sized data centers, enabling real-time applications. It helps to quickly identify and reduce temperature hot spots($7.7%) by redistributing workloads and saving cooling energy($2.5%). It also aids in optimizing server placement during installation, preventing issues, and increasing equipment lifespan. These optimizations boost sustainability by reducing energy use, improving server performance, and lowering environmental impact.

Authors: Soumyendu Sarkar (Hewlett Packard Enterprise); Antonio Guillen (Hewlett Packard Enterprise); Zachariah Carmichael (University of Notre Dame); Vineet Gundecha (Hewlett Packard Enterpise); Avisek Naug (Hewlett Packard Enterprise); Ashwin Ramesh Babu (Hewlett Packard Enterprise Labs); Ricardo Luna Gutierrez (Hewlett Packard Enterprise)
NeurIPS 2023 Flowering Onset Detection: Traditional Learning vs. Deep Learning Performance in a Sparse Label Context (Papers Track)
Abstract and authors: (click to expand)

Abstract: Detecting temporal shifts in plant flowering times is of increasing importance in a context of climate change, with applications in plant ecology, but also health, agriculture, and ecosystem management. However, scaling up plant-level monitoring is cost prohibitive, and flowering transitions are complex and difficult to model. We develop two sets of approaches to detect the onset of flowering at large-scale and high-resolution. Using fine grain temperature data with domain knowledge based features and traditional machine learning models provides the best performance. Using satellite data, with deep learning to deal with high dimensionality and transfer learning to overcome ground truth label sparsity, is a challenging but promising approach, as it reaches good performance with more systematically available data.

Authors: Mauricio Soroco (University of British Columbia); Joel Hempel (University of British Columbia); Xinze Xiong (University of British Columbia); Mathias Lécuyer (University of British Columbia); Joséphine Gantois (University of British Columbia)
NeurIPS 2023 Glacier Movement Prediction with Attention-based Recurrent Neural Networks and Satellite Data (Papers Track)
Abstract and authors: (click to expand)

Abstract: Studying glacier movements is crucial because of their indications for global climate change and its effects on local land masses. Building on established methods for glacier movement prediction from Landsat-8 satellite imaging data, we develop an attention-based deep learning model for time series data prediction of glacier movements. In our approach, the Normalized Difference Snow Index is calculated from the Landsat-8 spectral reflectance bands for data of the Parvati Glacier (India) to quantify snow and ice in the scene images, which is then used for time series prediction. Based on this data, a newly developed Long-Short Term Memory Encoder-decoder neural network model is trained, incorporating a Multi-head Self Attention mechanism in the decoder. The model shows promising results, making the prediction of optical flow vectors from pure model predictions possible.

Authors: Jonas Müller (University of Tübingen); Raphael Braun (University of Tübingen); Hendrik P. A. Lensch (University of Tübingen); Nicole Ludwig (University of Tübingen)
NeurIPS 2023 Detailed Glacier Area Change Analysis in the European Alps with Deep Learning (Papers Track)
Abstract and authors: (click to expand)

Abstract: Glacier retreat is a key indicator of climate change and requires regular updates of the glacier area. Recently, the release of a new inventory for the European Alps showed that glaciers continued to retreat at about 1.3% per year from 2003 to 2015. The outlines were produced by manually correcting the results of a semi-automatic method applied to Sentinel-2 imagery. In this work we develop a fully-automatic pipeline based on Deep Learning to investigate the evolution of the glaciers in the Alps from 2015 to present (2023). After outlier filtering, we provide individual estimates for around 1300 glaciers, representing 87% of the glacierized area. Regionally we estimate an area loss of -1.8% per year, with large variations between glaciers. Code and data are available at https://github.com/dcodrut/glacier_mapping_alps_tccml.

Authors: Codrut-Andrei Diaconu (DLR); Jonathan Bamber (Technical University of Munich)
NeurIPS 2023 A machine learning pipeline for automated insect monitoring (Papers Track)
Abstract and authors: (click to expand)

Abstract: Climate change and other anthropogenic factors have led to a catastrophic decline in insects, endangering both biodiversity and the ecosystem services on which human society depends. Data on insect abundance, however, remains woefully inadequate. Camera traps, conventionally used for monitoring terrestrial vertebrates, are now being modified for insects, especially moths. We describe a complete, open-source machine learning-based software pipeline for automated monitoring of moths via camera traps, including object detection, moth/non-moth classification, fine-grained identification of moth species, and tracking individuals. We believe that our tools, which are already in use across three continents, represent the future of massively scalable data collection in entomology.

Authors: Aditya Jain (Mila); Fagner Cunha (Federal University of Amazonas); Michael Bunsen (Mila, eButterfly); Léonard Pasi (EPFL); Anna Viklund (Daresay); Maxim Larrivee (Montreal Insectarium); David Rolnick (McGill University, Mila)
NeurIPS 2023 Global Coastline Evolution Forecasting from Satellite Imagery using Deep Learning (Papers Track)
Abstract and authors: (click to expand)

Abstract: Coastal zones are under increasing pressures due to climate change and the increasing population densities in coastal areas around the globe. Our ability to accurately forecast the evolution of the coastal zone is of critical importance to coastal managers in the context of risk assessment and mitigation. Recent advances in artificial intelligence and remote sensing enable the development of automatic large-scale analysis methodologies based on observation data. In this work, we make use of a novel satellite-derived shoreline forecasting dataset and a variant of the common Encoder-Decoder neural network, UNet, in order to predict shoreline change based on spatio-temporal data. We analyze the importance of including the spatial context at the prediction step and we find that it greatly enhances model performance. Overall, the model presented here demonstrates significant shoreline forecasting skill around the globe, achieving a global correlation of 0.77.

Authors: Guillaume RIU (Laboratory of Spatial Geophysics and Oceanography Studies); Mahmoud AL NAJAR (Laboratory of Spatial Geophysics and Oceanography Studies); Gregoire THOUMYRE (Laboratory of Spatial Geophysics and Oceanography Studies); Rafael ALMAR (Laboratory of Spatial Geophysics and Oceanography Studies); Dennis Wilson (ISAE)
NeurIPS 2023 IceCloudNet: Cirrus and mixed-phase cloud prediction from SEVIRI input learned from sparse supervision (Papers Track)
Abstract and authors: (click to expand)

Abstract: Clouds containing ice particles play a crucial role in the climate system. Yet they remain a source of great uncertainty in climate models and future climate projections. In this work, we create a new observational constraint of regime-dependent ice microphysical properties at the spatio-temporal coverage of geostationary satellite instruments and the quality of active satellite retrievals. We achieve this by training a convolutional neural network on three years of SEVIRI and DARDAR data sets. This work will enable novel research to improve ice cloud process understanding and hence, reduce uncertainties in a changing climate and help assess geoengineering methods for cirrus clouds.

Authors: Kai Jeggle (ETH Zurich); Mikolaj Czerkawski (ESA); Federico Serva (European Space Agency, Italian Space Agency); Bertrand Le Saux (European Space Agency (ESA)); David Neubauer (ETH Zurich); Ulrike Lohmann (ETH Zurich)
NeurIPS 2023 SAM-CD: Change Detection in Remote Sensing Using Segment Anything Model (Papers Track)
Abstract and authors: (click to expand)

Abstract: In remote sensing, Change Detection (CD) refers to locating surface changes in the same area over time. Changes can occur due to man-made or natural activities, and CD is important for analyzing climate changes. The recent advancements in satellite imagery and deep learning allow the development of affordable and powerful CD solutions. The breakthroughs in computer vision Foundation Models (FMs) bring new opportunities for better and more flexible remote sensing solutions. However, solving CD using FMs has not been explored before and this work presents the first FM-based deep learning model, SAM-CD. We propose a novel model that adapts the Segment Anything Model (SAM) for solving CD. The experimental results show that the proposed approach achieves the state of the art when evaluated on two challenging benchmark public datasets LEVIR-CD and DSIFN-CD.

Authors: Faroq ALTam (Elm Company); Thariq Khalid (Elm Company); Athul Mathew (Elm Company); Andrew Carnell (Elm Company); Riad Souissi (Elm Company)
NeurIPS 2023 Segment-then-Classify: Few-shot instance segmentation for environmental remote sensing (Papers Track)
Abstract and authors: (click to expand)

Abstract: Instance segmentation is pivotal for environmental sciences and climate change research, facilitating important tasks from land cover classification to glacier monitoring. This paper addresses the prevailing challenges associated with data scarcity when using traditional models like YOLOv8 by introducing a novel, data-efficient workflow for instance segmentation. The proposed Segment-then-Classify (STC) strategy leverages the zero-shot capabilities of the novel Segment Anything Model (SAM) to segment all objects in an image and then uses a simple classifier such as the Vision Transformer (ViT) to identify objects of interest thereafter. Evaluated on the VHR-10 dataset, our approach demonstrated convergence with merely 40 examples per class. YOLOv8 requires 3 times as much data to achieve the STC's peak performance. The highest performing class in the VHR-10 dataset achieved a near-perfect mAP@0.5 of 0.99 using the STC strategy. However, performance varied greatly across other classes due to the SAM model’s occasional inability to recognize all relevant objects, indicating a need for refining the zero-shot segmentation step. The STC workflow therefore holds promise for advancing few-shot learning for instance segmentation in environmental science.

Authors: Yang Hu (University of California, Santa Barbara); Kelly Caylor (UCSB); Anna S Boser (UCSB)
NeurIPS 2023 Top-down Green-ups: Satellite Sensing and Deep Models to Predict Buffelgrass Phenology (Papers Track)
Abstract and authors: (click to expand)

Abstract: An invasive species of grass known as "buffelgrass" contributes to severe wildfires and biodiversity loss in the Southwest United States. We tackle the problem of predicting buffelgrass "green-ups" (i.e. readiness for herbicidal treatment). To make our predictions, we explore temporal, visual and multi-modal models that combine satellite sensing and deep learning. We find that all of our neural-based approaches improve over conventional buffelgrass green-up models, and discuss how neural model deployment promises significant resource savings.

Authors: Lucas Rosenblatt (NYU); Bin Han (University of Washington); Erin Posthumus (USA NPN); Theresa Crimmins (USA NPN); Bill G Howe (University of Washington)
NeurIPS 2023 Typhoon Intensity Prediction with Vision Transformer (Papers Track)
Abstract and authors: (click to expand)

Abstract: Predicting typhoon intensity accurately across space and time is crucial for issuing timely disaster warnings and facilitating emergency response. This has vast potential for minimizing life losses and property damages as well as reducing economic and environmental impacts. Leveraging satellite imagery for scenario analysis is effective but also introduces additional challenges due to the complex relations among clouds and the highly dynamic context. Existing deep learning methods in this domain rely on convolutional neural networks (CNNs), which suffer from limited per-layer receptive fields. This limitation hinders their ability to capture long-range dependencies and global contextual knowledge during inference. In response, we introduce a novel approach, namely "Typhoon Intensity Transformer" (Tint), which leverages self-attention mechanisms with global receptive fields per layer. Tint adopts a sequence-to-sequence feature representation learning perspective. It begins by cutting a given satellite image into a sequence of patches and recursively employs self-attention operations to extract both local and global contextual relations between all patch pairs simultaneously, thereby enhancing per-patch feature representation learning. Extensive experiments on a publicly available typhoon benchmark validate the efficacy of Tint in comparison with both state-of-the-art deep learning and conventional meteorological methods. Our code is available at https://github.com/chen-huanxin/Tint.

Authors: Huanxin Chen (South China University of Technology); Pengshuai Yin (South China University of Technology); Huichou Huang (City University of Hong Kong); Qingyao Wu (South China University of Technology); Ruirui Liu (Brunel University London); Xiatian Zhu (University of Surrey)
NeurIPS 2023 Large Scale Masked Autoencoding for Reducing Label Requirements on SAR Data (Papers Track) Overall Best Paper
Abstract and authors: (click to expand)

Abstract: Satellite-based remote sensing is instrumental in the monitoring and mitigation of the effects of anthropogenic climate change. Large scale, high resolution data derived from these sensors can be used to inform intervention and policy decision making, but the timeliness and accuracy of these interventions is limited by use of optical data, which cannot operate at night and is affected by adverse weather conditions. Synthetic Aperture Radar (SAR) offers a robust alternative to optical data, but its associated complexities limit the scope of labelled data generation for traditional deep learning. In this work, we apply a self-supervised pretraining scheme, masked autoencoding, to SAR amplitude data covering 8.7\% of the Earth's land surface area, and tune the pretrained weights on two downstream tasks crucial to monitoring climate change - vegetation cover prediction and land cover classification. We show that the use of this pretraining scheme reduces labelling requirements for the downstream tasks by more than an order of magnitude, and that this pretraining generalises geographically, with the performance gain increasing when tuned downstream on regions outside the pretraining set. Our findings significantly advance climate change mitigation by facilitating the development of task and region-specific SAR models, allowing local communities and organizations to deploy tailored solutions for rapid, accurate monitoring of climate change effects.

Authors: Matthew J Allen (University of Cambridge); Francisco Dorr (Independent); Joseph A Gallego (National University Of Colombia); Laura Martínez-Ferrer (University of Valencia); Freddie Kalaitzis (University of Oxford); Raul Ramos-Pollan (Universidad de Antioquia); Anna Jungbluth (European Space Agency)
NeurIPS 2023 Methane Plume Detection with U-Net Segmentation on Sentinel-2 Image Data (Papers Track)
Abstract and authors: (click to expand)

Abstract: Methane emissions have a significant impact on increasing global warming. Satellite-based methane detection methods can help mitigate methane emissions, as they provide a constant and global detection. The Sentinel-2 constellation, in particular, offers frequent and publicly accessible images on a global scale. We propose a deep learning approach to detect methane plumes from Sentinel-2 images. We construct a dataset of 5200 satellite images with identified methane plumes, on which we train a U-Net model. Preliminary results demonstrate that the model is able to correctly identify methane plumes on training data, although generalization to new methane plumes remains challenging. All code, data, and models are made available online.

Authors: Berenice du Baret (ISAE-Supaero); Simon Finos (ISAE-Supaero); Hugo Guiglion (ISAE-Supaero); Dennis Wilson (ISAE)
NeurIPS 2023 Soil Organic Carbon Estimation from Climate-related Features with Graph Neural Network (Papers Track)
Abstract and authors: (click to expand)

Abstract: Soil organic carbon (SOC) plays a pivotal role in the global carbon cycle, impacting climate dynamics and necessitating accurate estimation for sustainable land and agricultural management. While traditional methods of SOC estimation face resolution and accuracy challenges, recent advancements harness remote sensing, machine learning, and high-resolution satellite mapping. Graph Neural Networks (GNNs), especially when integrated with positional encoders, offer promise in capturing intricate relationships between soil and climate. Using the LUCAS database, this study compared four GNN operators in the positional encoder framework. Results revealed that the PESAGE and PETransformer models outperformed others in SOC estimation, indicating their potential in capturing the complex interplay between SOC and climate features. Our findings underscore the potential of GNN architectures in advancing SOC prediction, paving the way for future explorations with more advanced GNN models.

Authors: Weiying Zhao (Deep Planet); Natalia Efremova (Queen Mary University London)
NeurIPS 2023 Deep Glacier Image Velocimetry: Mapping glacier velocities from Sentinel-2 imagery with deep learning (Papers Track)
Abstract and authors: (click to expand)

Abstract: Glacier systems are highly sensitive to climate change and play a pivotal role in global mean sea level rise. As such, it is important to monitor how glacier velocities and ice dynamics evolve under a changing climate. The growing wealth of satellite observations has facilitated the inference of glacier velocities from remote sensing imagery through feature tracking algorithms. At present, these rely on sparse cross-correlation estimates as well as computationally expensive optical flow solutions. Here we present a novel use of deep-learning for estimating annual glacier velocities, utilizing the recurrent optical-flow based architecture, RAFT, on consecutive pairs of optical Sentinel-2 imagery. Our results highlight that deep learning can generate dense per-pixel velocity estimates within an automated framework that utilizes Sentinel-2 images over the French Alps.

Authors: James B Tlhomole (Imperial College London); Matthew Piggott (Imperial College London); Graham Hughes (Imperial College London)
NeurIPS 2023 Towards autonomous large-scale monitoring the health of urban trees using mobile sensing (Papers Track)
Abstract and authors: (click to expand)

Abstract: Healthy urban greenery is a fundamental asset to mitigate climate change phenomenons such as extreme heat and air pollution. However, urban trees are often affected by abiotic and biotic stressors that hamper their functionality, and whenever not timely managed, even their survival. The current visual or instrumented inspection techniques often require a high amount of human labor making frequent assessments infeasible at a city-wide scale. In this work, we present the GreenScan Project, a ground-based sensing system designed to provide health assessment of urban trees at high space-time resolutions, with low costs. The system utilises thermal and multi-spectral imaging sensors, fused using computer vision models to estimate two tree health indexes, namely NDVI and CTD. Preliminary evaluation of the system was performed through data collection experiments in Cambridge, USA. Overall, this work illustrates the potential of autonomous mobile ground-based tree health monitoring on city-wide scales at high temporal resolutions with low-costs.

Authors: Akshit Gupta (Delft University of Technology); Martine Rutten (Delft University of Technology); RANGA RAO VENKATESHA PRASAD (TUDelft); Remko Uijlenhoet (Delft University of Technology)
NeurIPS 2023 Towards Global, General-Purpose Pretrained Geographic Location Encoders (Papers Track)
Abstract and authors: (click to expand)

Abstract: Location information is essential for modeling tasks in climate-related fields ranging from ecology to the Earth system sciences. However, obtaining meaningful location representation is challenging and requires a model to distill semantic location information from available data, such as remote sensing imagery. To address this challenge, we introduce SatCLIP, a global, general-purpose geographic location encoder that provides vector embeddings summarizing the characteristics of a given location for convenient usage in diverse downstream tasks. We show that SatCLIP embeddings, pretrained on multi-spectral Sentinel-2 satellite data, can be used for various predictive out-of-domain tasks, including temperature prediction and animal recognition in imagery, and outperform existing competing approaches. SatCLIP embeddings also prove helpful in overcoming geographic domain shift. This demonstrates the potential of general-purpose location encoders and opens the door to learning meaningful representations of our planet from the vast, varied, and largely untapped modalities of geospatial data.

Authors: Konstantin Klemmer (Microsoft Research); ESTHER ROLF (Google Research); Caleb Robinson (Microsoft AI for Good Research Lab); Lester Mackey (Microsoft Research); Marc Rußwurm (École Polytechnique Fédérale de Lausanne)
NeurIPS 2023 Antarctic Bed Topography Super-Resolution via Transfer Learning (Papers Track)
Abstract and authors: (click to expand)

Abstract: High-fidelity topography models of the bedrock underneath the thick Antarctic ice sheet can improve scientists' understanding of ice flow and its contributions to sea level rise. However, the bed topography of Antarctica is one of the most challenging surfaces on earth to map, requiring aircrafts with ice penetrating radars to survey the vast and remote continent. We propose FROST, Fusion Regression for Optimal Subglacial Topography, a method that leverages readily available surface topography data from satellites as an auxiliary input modality for bed topography super-resolution. FROST uses a non-parametric Gaussian Process model to transfer local, non-stationary covariance patterns from the ice surface to the bedrock. In a controlled topography reconstruction experiment over complex East Antarctic terrain, our proposed method outperforms bicubic interpolation at all five tested magnification factors, reducing RMSE by 67% at x2, and 25% at x6 magnification. This work demonstrates the opportunity for data fusion methods to advance downstream climate modelling and steward climate change adaptation.

Authors: Kim Bente (The University of Sydney); Roman Marchant (University of Technology Sydney); Fabio Ramos (NVIDIA, The University of Sydney)
NeurIPS 2023 Assessing data limitations in ML-based LCLU (Proposals Track)
Abstract and authors: (click to expand)

Abstract: This study addresses the accuracy challenge in Global Land Use and Land Cover (LULC) maps, crucial for policy making towards climate change mitigation. We evaluate two LULC products based on advanced machine learning techniques across two representative nations, Ecuador and Germany, employing a novel accuracy metric. The analysis unveils a notable accuracy enhancement in the convolutional neural network-based approach against the random forest model used for comparison. Our findings emphasize the potential of sophisticated machine learning methodologies in advancing LULC mapping accuracy, an essential stride towards data-driven, climate-relevant land management and policy decisions.

Authors: Angel Encalada-Davila (ESPOL); Christian Tutiven (ESPOL University); Jose E Cordova-Garcia (ESPOL)
NeurIPS 2023 High-resolution Global Building Emissions Estimation using Satellite Imagery (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Globally, buildings account for 30% of end-use energy consumption and 27% of energy sector emissions, and yet the building sector is lacking in low-temporal-latency, high-spatial-resolution data on energy consumption and resulting emissions. Existing methods tend to either have low resolution, high latency (often a year or more), or rely on data typically unavailable at scale (such as self-reported energy consumption). We propose a machine learning based bottom-up model that combines satellite-imagery-derived features to compute Scope 1 global emissions estimates both for residential and commercial buildings at a 1 square km resolution with monthly global updates.

Authors: Paul J Markakis (Duke University); Jordan Malof (University of Montana); Trey Gowdy (Duke University); Leslie Collins (Duke University); Aaron Davitt (WattTime); Gabriela Volpato (WattTime); Kyle Bradbury (Duke University)
NeurIPS 2023 Understanding Insect Range Shifts with Out-of-Distribution Detection (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Climate change is inducing significant range shifts in insects and other organisms. Large-scale temporal data on populations and distributions are essential for quantifying the effects of climate change on biodiversity and ecosystem services, providing valuable insights for both conservation and pest management. With images from camera traps, we aim to use Mahalanobis distance-based confidence scores to automatically detect new moth species in a region. We intend to make out-of-distribution detection interpretable by identifying morphological characteristics of different species using Grad-CAM. We hope this algorithm will be a useful tool for entomologists to study range shifts and inform climate change adaptation.

Authors: Yuyan Chen (McGill University, Mila); David Rolnick (McGill University, Mila)
NeurIPS 2023 Aquaculture Mapping: Detecting and Classifying Aquaculture Ponds using Deep Learning (Tutorials Track)
Abstract and authors: (click to expand)

Abstract: Mapping aquaculture ponds is critical for restoration, conservation, and climate adaptation efforts. Aquaculture can contribute to high levels of water pollution from untreated effluent and negatively impact coastal ecosystems. Large-scale aquaculture is also a significant driver in mangrove deforestation, thus reducing the world’s carbon sinks and exacerbating the effects of climate change. However, finding and mapping these ponds on the ground can be highly labor and time-intensive. Most existing automated techniques are focused only on spatial location and do not consider production intensification, which is also crucial to understanding their impact on the surrounding ecosystem. We can classify them into two main types: a) Extensive ponds, which are large, irregularly-shaped ponds that rely on natural productivity, and b) intensive ponds which are smaller and regularly shaped. Intensive ponds use machinery such as aerators that maximize production and also result in the characteristic presence of air bubbles on the pond’s surface. The features of these two types of ponds make them distinguishable and detectable from satellite imagery. In this tutorial, we will discuss types of aquaculture ponds in detail and demonstrate how they can be detected and classified using satellite imagery. The tutorial will introduce an open dataset of human-labeled aquaculture ponds in the Philippines and Indonesia. Using this dataset, the tutorial will use semantic segmentation to map out similar ponds over an entire country and classify them as either extensive or intensive, going through the entire process of i) satellite imagery retrieval, ii) preprocessing these images into a training-ready dataset, iii) model training, and iv) finally model rollout on a sample area. Throughout, the tutorial will leverage PyTorch Lightning, a machine learning framework that provides a simplified and streamlined interface for model experimentation and deployment. This tutorial aims to discuss the relevance of aquaculture ponds in climate adaptation and equip users with the necessary inputs and tools to perform their own ML-powered earth observation projects.

Authors: John Christian G Nacpil (Thinking Machines Data Science, Inc.); Joshua Cortez (Thinking Machines Data Science)
ICLR 2023 Improving global high-resolution Earth system model simulations of precipitation with generative adversarial networks (Papers Track)
Abstract and authors: (click to expand)

Abstract: Precipitation extremes are expected to become stronger and more frequent in response to anthropogenic global warming. Accurately projecting the ecological and socioeconomic impacts is an urgent task. Impact models are developed and calibrated with observation-based data but rely on Earth system model (ESM) output for future scenarios. ESMs, however, exhibit significant biases in their output because they cannot fully resolve complex cross-scale interactions of processes that produce precipitation cannot. State-of-the-art bias correction methods only address errors in the simulated frequency distributions, locally at every individual grid cell. Improving unrealistic spatial patterns of the ESM output, which would require spatial context, has not been possible so far. Here we show that a post-processing method based on physically constrained generative adversarial networks (GANs) can correct biases of a state-of-the-art global ESM both in local frequency distributions and in the spatial patterns at once. While our method improves local frequency distributions similarly well to a gold-standard ESM bias-adjustment framework, it strongly outperforms existing methods in correcting spatial patterns. Our study highlights the importance of physical constraints in neural networks for out-of-sample predictions in the context of climate change.

Authors: Philipp Hess (Technical University of Munich)
ICLR 2023 Coregistration of Satellite Image Time Series Through Alignment of Road Networks (Papers Track)
Abstract and authors: (click to expand)

Abstract: Due to climate change, thawing permafrost affects transportation infrastructure in northern regions. Tracking deformations over time of these structures can allow identifying the most vulnerable sections to permafrost degradation and implement climate adaptation strategies. The Sentinel-2 mission provides data well-suited for multitemporal analysis due to its high temporal resolution and multispectral coverage. However, the geometrical misalignment of Sentinel-2 imagery makes this analysis challenging. Towards the goal of estimating the deformation of linear infrastructure in northern Canada, we propose an automatic subpixel coregistration algorithm for satellite image time series based on the matching of binary masks of roads produced by a deep learning model. We demonstrate the feasibility of achieving subpixel coregistration through alignment of roads on a small dataset of high-resolution Sentinel-2 images from the region of Gillam in northern Canada. This is the first step towards training a road deformation prediction model.

Authors: Andres Felipe Perez Murcia (University of Manitoba); Pooneh Maghoul (University of Manitoba); Ahmed Ashraf (University of Manitoba)
ICLR 2023 Estimating Residential Solar Potential using Aerial Data (Papers Track)
Abstract and authors: (click to expand)

Abstract: Project Sunroof estimates the solar potential of residential buildings using high quality aerial data. That is, it estimates the potential solar energy (and associated financial savings) that can be captured by buildings if solar panels were to be installed on their roofs. Unfortunately its coverage is limited by the lack of high resolution digital surface map (DSM) data. We present a deep learning approach that bridges this gap by enhancing widely available low-resolution data, thereby dramatically increasing the coverage of Sunroof. We also present some ongoing efforts to potentially improve accuracy even further by replacing certain algorithmic components of Sunroof’s processing pipeline with deep learning.

Authors: Ross Goroshin (Google); Carl Elkin (Google)
ICLR 2023 Improving extreme weather events detection with light-weight neural networks (Papers Track)
Abstract and authors: (click to expand)

Abstract: To advance automated detection of extreme weather events, which are increasing in frequency and intensity with climate change, we explore modifications to a novel light-weight Context Guided convolutional neural network architecture trained for semantic segmentation of tropical cyclones and atmospheric rivers in climate data. Our primary focus is on tropical cyclones, the most destructive weather events, for which current models show limited performance. We investigate feature engineering, data augmentation, learning rate modifications, alternative loss functions, and architectural changes. In contrast to previous approaches optimizing for intersection over union, we specifically seek to improve recall to penalize under-counting and prioritize identification of tropical cyclones. We report success through the use of weighted loss functions to counter class imbalance for these rare events. We conclude with directions for future research on extreme weather events detection, a crucial task for prediction, mitigation, and equitable adaptation to the impacts of climate change.

Authors: Romain Lacombe (Stanford University); Hannah Grossman (Stanford); Lucas P Hendren (Stanford University); David Ludeke (Stanford University)
ICLR 2023 An automatic mobile approach for Tree DBH Estimation Using a Depth Map and a Regression Convolutional Neural Network (Papers Track)
Abstract and authors: (click to expand)

Abstract: Carbon credit programs finance projects to reduce emissions, remove pollutants, improve livelihoods, and protect natural ecosystems. Ensuring the quality and integrity of such projects is essential to their success. One of the most important variables used in nature-based solutions to measure carbon sequestration is the diameter at breast height (DBH) of trees. In this paper, we propose an automatic mobile computer vision method to estimate the DBH of a tree using a single depth map on a smartphone, along with our created dataset DepthMapDBH2023. We successfully demonstrated that this dataset paired with a lightweight regression convolutional neural network is able to accurately estimate the DBH of trees distinct in appearance, shape, number of tree forks, tree density and crowding, and vine presence. Automation of these measurements will help crews in the field who are collecting data for forest inventories. Gathering as much on-the-ground data as possible is required to ensure the transparency of carbon credit projects. Access to high-quality datasets of manual measurements helps improve biomass models which are widely used in the field of ecological simulation. The code used in this paper will be publicly available on Github and the dataset on Kaggle.

Authors: Margaux Masson-Forsythe (Earthshot Labs); Margaux Masson-Forsythe (Earthshot Labs)
ICLR 2023 BurnMD: A Fire Projection and Mitigation Modeling Dataset (Papers Track)
Abstract and authors: (click to expand)

Abstract: Today's fire projection modeling tools struggle to keep up with the rapid rate and increasing severity of climate change, leaving disaster managers dependent on tools which are increasingly unrepresentative of complex interactions between fire behavior, environmental conditions, and various mitigation options. This has consequences for equitably minimizing wildfire risks to life, property, ecology, cultural heritage, and public health. Fortunately, decades of data exist for fuel populations, weather conditions, and outcomes of significant fires in the American West and globally. The fire management community faces a lack of data standardization and validation among many competing fire models. Likewise, the machine learning community lacks curated datasets and benchmarks to develop solutions necessary to generate impact in this space. We present a novel dataset composed of 308 medium sized fires from the years 2018-2021, complete with both time series airborne based inference and ground operational estimation of fire extent, and operational mitigation data such as control line construction. As the first large wildfire dataset with mitigation information, Burn Mitigation Dataset (BurnMD) will help advance fire projection modeling, fire risk modeling, and AI generated land management policies.

Authors: Marissa Dotter (MITRE Corporation)
ICLR 2023 Remote Control: Debiasing Remote Sensing Predictions for Causal Inference (Papers Track)
Abstract and authors: (click to expand)

Abstract: Understanding and properly estimating the impacts of environmental interventions is of critical importance as we work towards achieving global climate goals. Remote sensing has become an essential tool for evaluating when and where climate policies have positive impacts on factors like greenhouse gas emissions and carbon sequestration. However, when machine learning models trained to predict outcomes using remotely sensed data simply minimize a standard loss function, the predictions that they generate can produce biased estimates in downstream causal inference. If prediction error in the outcome variable is correlated with policy variables or important confounders, as is the case for many widely used remote sensing data sets, estimates of the causal impacts of policies can be biased. In this paper, we demonstrate how this bias can arise, and we propose the use of an adversarial debiasing model (Zhang, Lemoine, and Mitchell 2018) in order to correct the issue when using satellite data to generate machine learning predictions for use in causal inference. We apply this method to a case study of the relationship between roads and tree cover in West Africa, where our results indicate that adversarial debiasing can recover a much more accurate estimate of the parameter of interest compared to when the standard approach is used.

Authors: Matthew Gordon (Yale); Megan Ayers (Yale University); Eliana Stone (Yale School of the Environment); Luke C Sanford (Yale School of the Environment)
ICLR 2023 SOLAR PANEL MAPPING VIA ORIENTED OBJECT DETECTION (Papers Track)
Abstract and authors: (click to expand)

Abstract: Maintaining the integrity of solar power plants is a vital component in dealing with the current climate crisis. This process begins with analysts creating a de- tailed map of a plant with the coordinates of every solar panel, making it possible to quickly locate and mitigate potential faulty solar panels. However, this task is extremely tedious and is not scalable for the ever increasing capacity of so- lar power across the globe. Therefore, we propose an end-to-end deep learning framework for detecting individual solar panels using a rotated object detection architecture. We evaluate our approach on a diverse dataset of solar power plants collected from across the United States and report a mAP score of 83.3%.

Authors: Conor Wallace (DroneBase); Isaac Corley (University of Texas at San Antonio); Jonathan Lwowski (DroneBase)
ICLR 2023 Disentangling observation biases to monitor spatio-temporal shifts in species distributions (Proposals Track)
Abstract and authors: (click to expand)

Abstract: The accelerated pace of environmental change due to anthropogenic activities makes it more important than ever to understand current and future ecosystem dynamics at a global scale. Species observations stemming from citizen science platforms are increasingly leveraged to gather information about the geographic distributions of many species. However, their usability is limited by the strong biases inherent to these community-driven efforts. These biases in the sampling effort are often treated as noise that has to be compensated for. In this project, we posit that better modelling the sampling effort (including the usage of the different platforms across countries, local accessibility, attractiveness of the location for platform users, affinity of different user groups for different species, etc.) is the key towards improving Species Distribution Models (SDM) using observations from citizen science platforms, thus opening up the possibility of leveraging them to monitor changes in species distributions and population densities.

Authors: Diego Marcos (Inria); Christophe Botella (); Ilan Havinga (Wageningen University); Dino Ienco (INRAE); Cassio F. Dantas (TETIS, INRAE, Univ Montpellier); Pierre Alliez (INRIA Sophie-Antipolis, France); Alexis Joly (INRIA, FR)
ICLR 2023 Improving the spatial accuracy of extreme tropical cyclone rainfall in ERA5 using deep learning (Papers Track)
Abstract and authors: (click to expand)

Abstract: We propose a novel method for the bias adjustment and post-processing of gridded rainfall data products. Our method uses U-Net (a deep convolutional neural network) as a backbone, and a novel loss function given by the combination of a pixelwise bias component (Mean Absolute Error) and a spatial accuracy component (Fractions Skill Score). We evaluate the proposed approach by adjusting extreme rainfall from the popular ERA5 reanalysis dataset, using the multi-source observational dataset MSWEP as a target. We focus on a sample of extreme rainfall events induced by tropical cyclones and show that the proposed method significantly reduces both the MAE (by 16\%) and FSS (by 53\%) of ERA5.

Authors: Guido Ascenso (Politecnico di Milano); Andrea Ficchì (Politecnico di Milano); Matteo Giuliani (Politecnico di Milano); Leone Cavicchia (Fondazione Centro Euro-Mediterraneo sui Cambiamenti Climatici (CMCC)); Enrico Scoccimarro (Fondazione Centro Euro-Mediterraneo sui Cambiamenti Climatici (CMCC)); Andrea Castelletti (Politecnico di Milano)
ICLR 2023 Green AutoML for Plastic Litter Detection (Papers Track)
Abstract and authors: (click to expand)

Abstract: The world’s oceans are polluted with plastic waste and the detection of it is an important step toward removing it. Wolf et al. (2020) created a plastic waste dataset to develop a plastic detection system. Our work aims to improve the machine learning model by using Green Automated Machine Learning (AutoML). One aspect of Green-AutoML is to search for a machine learning pipeline, while also minimizing the carbon footprint. In this work, we train five standard neural architectures for image classification on the aforementioned plastic waste dataset. Subsequently, their performance and carbon footprints are compared to an Efficient Neural Architecture Search as a well-known AutoML approach. We show the potential of Green-AutoML by outperforming the original plastic detection system by 1.1% in accuracy and using 33 times fewer floating point operations at inference, and only 29% of the carbon emissions of the best-known baseline. This shows the large potential of AutoML on climate-change relevant applications and at the same time contributes to more efficient modern Deep Learning systems, saving substantial resources and reducing the carbon footprint.

Authors: Daphne Theodorakopoulos (German Research Centre for Artificial Intelligence (DFKI), Marine Perception Department and Leibniz University Hannover, Institute of Artificial Intelligence); Christoph Manß (German Research Centre for Artificial Intelligence (DFKI), Marine Perception Department); Frederic Stahl (German Research Centre for Artificial Intelligence (DFKI), Marine Perception Department); Marius Lindauer (Leibniz University Hannover)
ICLR 2023 EfficientTempNet: Temporal Super-Resolution of Radar Rainfall (Papers Track)
Abstract and authors: (click to expand)

Abstract: Rainfall data collected by various remote sensing instruments such as radars or satellites has different space-time resolutions. This study aims to improve the temporal resolution of radar rainfall products to help with more accurate climate change modeling and studies. In this direction, we introduce a solution based on EfficientNetV2, namely EfficientTempNet, to increase the temporal resolution of radar-based rainfall products from 10 minutes to 5 minutes. We tested EfficientRainNet over a dataset for the state of Iowa, US, and compared its performance to three different baselines to show that EfficientTempNet presents a viable option for better climate change monitoring.

Authors: Bekir Z Demiray (University of Iowa); Muhammed A Sit (The University of Iowa); Ibrahim Demir (University of Iowa)
ICLR 2023 Bird Distribution Modelling using Remote Sensing and Citizen Science data (Papers Track) Overall Best Paper
Abstract and authors: (click to expand)

Abstract: Climate change is a major driver of biodiversity loss, changing the geographic range and abundance of many species. However, there remain significant knowl- edge gaps about the distribution of species, due principally to the amount of effort and expertise required for traditional field monitoring. We propose an approach leveraging computer vision to improve species distribution modelling, combining the wide availability of remote sensing data with sparse on-ground citizen science data from .We introduce a novel task and dataset for mapping US bird species to their habitats by predicting species encounter rates from satellite images, along with baseline models which demonstrate the power of our approach. Our methods open up possibilities for scalably modelling ecosystems properties worldwide.

Authors: Mélisande Teng (Mila, Université de Montréal); Amna Elmustafa (African Institute for Mathematical Science); Benjamin Akera (McGill University); Hugo Larochelle (UdeS); David Rolnick (McGill University, Mila)
ICLR 2023 Fourier Neural Operators for Arbitrary Resolution Climate Data Downscaling (Papers Track)
Abstract and authors: (click to expand)

Abstract: Running climate simulations informs us of future climate change. However, it is computationally expensive to resolve complex climate processes numerically. As one way to speed up climate simulations, neural networks have been used to downscale climate variables from fast-running low-resolution simulations. So far, all neural network downscaling models can only downscale input samples with a pre-defined upsampling factor. In this work, we propose a Fourier neural operator downscaling model. It trains with data of a small upsampling factor and then can zero-shot downscale its input to arbitrary unseen high-resolutions. Evaluated on Navier-Stokes equation solution data and ERA5 water content data, our downscaling model demonstrates better performance than widely used convolutional and adversarial generative super-resolution models in both learned and zero-shot downscaling. Our model's performance is further boosted when a constraint layer is applied. In the end, we show that by combining our downscaling model with a low-resolution numerical PDE solver, the downscaled solution outperforms the solution of the state-of-the-art high-resolution data-driven solver. Our model can be used to cheaply and accurately generate arbitrarily high-resolution climate simulation data with fast-running low-resolution simulation as input.

Authors: Qidong Yang (New York University); Paula Harder (Fraunhofer ITWM); Venkatesh Ramesh (University of Montreal, Mila); Alex Hernandez-Garcia (Mila - Quebec AI Institute); Daniela Szwarcman (IBM Research); Prasanna Sattigeri (IBM Research); Campbell D Watson (IBM Reserch); David Rolnick (McGill University, Mila)
ICLR 2023 Projecting the climate penalty on pm2.5 pollution with spatial deep learning (Proposals Track)
Abstract and authors: (click to expand)

Abstract: The climate penalty measures the effects of a changing climate on air quality due to the interaction of pollution with climate factors, independently of future changes in emissions. This work introduces a statistical framework for estimating the climate penalty on soot pollution (PM 2.5), which has been linked to respiratory and cardiovascular diseases and premature mortality. The framework evaluates the disparities in future PM 2.5 exposure across racial/ethnic and income groups---an important step towards informing mitigation public health policy and promoting environmental equity in addressing the effects of climate change. The proposed methodology aims to improve existing statistical-based methods for estimating the climate penalty using an expressive and scalable predictive model based on spatial deep learning with spatiotemporal trend estimation. The proposed approach will (1) use higher-resolution climate inputs, which current statistical methods to estimate the climate penalty approaches cannot accommodate; (2) integrate additional predictive data sources such as demographics, geology, and land use; (3) consider regional dependencies and synoptic weather patterns influencing PM 2.5, deconvolving the effects of climate change from increasing air quality regulations and other sources of unmeasured spatial heterogeneity.

Authors: Mauricio Tec (Harvard University); Riccardo Cadei (Harvard University); Francesca Dominici (Harvard University); Corwin Zigler (University of Texas at Austin)
NeurIPS 2022 Image-Based Soil Organic Carbon Estimation from Multispectral Satellite Images with Fourier Neural Operator and Structural Similarity (Papers Track)
Abstract and authors: (click to expand)

Abstract: Soil organic carbon (SOC) sequestration is the transfer and storage of atmospheric carbon dioxide in soils, which plays an important role in climate change mitigation. SOC concentration can be improved by proper land use, thus it is beneficial if SOC can be estimated at a regional or global scale. As multispectral satellite data can provide SOC-related information such as vegetation and soil properties at a global scale, estimation of SOC through satellite data has been explored as an alternative to manual soil sampling. Although existing works show promising results, most studies are based on pixel-based approaches with traditional machine learning methods, and convolutional neural networks (CNNs) are seldom used. To study the advantages of using CNNs on SOC remote sensing, in this paper, we propose the FNO-DenseNet based on the state-of-the-art Fourier neural operator (FNO). By combining the advantages of the FNO and DenseNet, the FNO-DenseNet outperformed the FNO in our experiments with hundreds of times fewer parameters. The FNO-DenseNet also outperformed a pixel-based random forest by 18% in the mean absolute percentage error. To the best of our knowledge, this is the first work of applying the FNO on SOC remote sensing.

Authors: Ken C. L. Wong (IBM Research – Almaden Research Center); Levente Klein (IBM Research); Ademir Ferreira da Silva (IBM Research); Hongzhi Wang (IBM Almaden Research Center); Jitendra Singh (IBM Research - India); Tanveer Syeda-Mahmood (IBM Research)
NeurIPS 2022 SolarDK: A high-resolution urban solar panel image classification and localization dataset (Papers Track)
Abstract and authors: (click to expand)

Abstract: The body of research on classification of solar panel arrays from aerial imagery is increasing, yet there are still not many public benchmark datasets. This paper introduces two novel benchmark datasets for classifying and localizing solar panel arrays in Denmark: A human annotated dataset for classification and segmentation, as well as a classification dataset acquired using self-reported data from the Danish national building registry. We explore the performance of prior works on the new benchmark dataset, and present results after fine-tuning models using a similar approach as recent works. Furthermore, we train models of newer architectures and provide benchmark baselines to our datasets in several scenarios. We believe the release of these datasets may improve future research in both local and global geospatial domains for identifying and mapping of solar panel arrays from aerial imagery. The data is accessible at https://osf.io/aj539/.

Authors: Maxim MK Khomiakov (DTU); Julius Radzikowski (DTU); Carl Schmidt (DTU); Mathias Sørensen (DTU); Mads Andersen (DTU); Michael Andersen (Technical University of Denmark); Jes Frellsen (Technical University of Denmark)
NeurIPS 2022 Optimizing toward efficiency for SAR image ship detection (Papers Track)
Abstract and authors: (click to expand)

Abstract: The detection and prevention of illegal fishing is critical to maintaining a healthy and functional ecosystem. Recent research on ship detection in satellite imagery has focused exclusively on performance improvements, disregarding detection efficiency. However, the speed and compute cost of vessel detection are essential for a timely intervention to prevent illegal fishing. Therefore, we investigated optimization methods that lower detection time and cost with minimal performance loss. We trained an object detection model based on a convolutional neural network (CNN) using a dataset of satellite images. Then, we designed two efficiency optimizations that can be applied to the base CNN or any other base model. The optimizations consist of a fast, cheap classification model and a statistical algorithm. The integration of the optimizations with the object detection model leads to a trade-off between speed and performance. We studied the trade-off using metrics that give different weight to execution time and performance. We show that by using a classification model the average precision of the detection model can be approximated to 99.5% in 44% of the time or to 92.7% in 25% of the time.

Authors: Arthur Van Meerbeeck (KULeuven); Ruben Cartuyvels (KULeuven); Jordy Van Landeghem (KULeuven); Sien Moens (KU Leuven)
NeurIPS 2022 Aboveground carbon biomass estimate with Physics-informed deep network (Papers Track)
Abstract and authors: (click to expand)

Abstract: The global carbon cycle is a key process to understand how our climate is changing. However, monitoring the dynamics is difficult because a high-resolution robust measurement of key state parameters including the aboveground carbon biomass (AGB) is required. We use deep neural network to generate a wall-to-wall map of AGB within the Continental USA (CONUS) with 30-meter spatial resolution for the year 2021. We combine radar and optical hyperspectral imagery, with a physical climate parameter of solar-induced chlorophyll fluorescence (SIF)-based gross primary productivity (GPP). Validation results show that a masked variation of UNet has the lowest validation RMSE of 37.93 ± 1.36 Mg C/ha, as compared to 52.30 ± 0.03 Mg C/ha for random forest algorithm. Furthermore, models that learn from SIF-based GPP in addition to radar and optical imagery reduce validation RMSE by almost 10% and the standard deviation by 40%. Finally, we apply our model to measure losses in AGB from the recent 2021 Caldor wildfire in California, and validate our analysis with Sentinel-based burn index.

Authors: Juan Nathaniel (Columbia University); Levente Klein (IBM Research); Campbell D Watson (IBM Reserch); Gabrielle Nyirjesy (Columbia University); Conrad M Albrecht (IBM Research)
NeurIPS 2022 Detecting Methane Plumes using PRISMA: Deep Learning Model and Data Augmentation (Papers Track)
Abstract and authors: (click to expand)

Abstract: The new generation of hyperspectral imagers, such as PRISMA, has improved significantly our detection capability of methane (CH4) plumes from space at high spatial resolution (∼30m). We present here a complete framework to identify CH4 plumes using images from the PRISMA satellite mission and a deep learning technique able to automatically detect plumes over large areas. To compensate for the sparse database of PRISMA images, we trained our model by transposing high resolution plumes from Sentinel-2 to PRISMA. Our methodology avoids computationally expensive synthetic plume from Large Eddy Simulations while generating a broad and realistic training database, and paves the way for large-scale detection of methane plumes using future hyperspectral sensors (EnMAP, EMIT, CarbonMapper).

Authors: Alexis Groshenry (Kayrros); Clément Giron (Kayrros); Alexandre d'Aspremont (CNRS, DI, Ecole Normale Supérieure; Kayrros); Thomas Lauvaux (University of Reims Champagne Ardenne, GSMA, UMR 7331); Thibaud Ehret (Centre Borelli)
NeurIPS 2022 Probabilistic forecasting of regional photovoltaic power production based on satellite-derived cloud motion (Papers Track)
Abstract and authors: (click to expand)

Abstract: Solar energy generation drastically increased in the last years, and it is expected to grow even more in the next decades. So, accurate intra-day forecasts are needed to improve the predictability of the photovoltaic power production and associated balancing measures to increase the shares of renewable energy in the power grid. Most forecasting methods require numerical weather predictions, which are slow to compute, or long-term datasets to run the forecast. These issues make the models difficult to implement in an operational setting. To overcome these problems, we propose a novel regional intraday probabilistic PV power forecasting model able to exploit only 2 hours of satellite-derived cloudiness maps to produce the ensemble forecast. The model is easy to implement in an operational setting as it is based on Pysteps, an already-operational Python library for precipitation nowcasting. With few adaptations of the Steps algorithm, we reached state-of-the-art performance, reaching a 71% lower RMSE than the Persistence model and a 50% lower CRPS than the Persistence Ensemble model for forecast lead times up to 4 hours.

Authors: Alberto Carpentieri (Bern University of Applied Science); Doris Folini (Institute for Atmospheric and Climate Science, ETH Zurich); Martin Wild (Institute for Atmospheric and Climate Science, ETH Zurich); Angela Meyer (Bern University of Applied Science)
NeurIPS 2022 Deep Hydrology: Hourly, Gap-Free Flood Maps Through Joint Satellite and Hydrologic Modelling (Papers Track)
Abstract and authors: (click to expand)

Abstract: Climate change-driven weather disasters are rapidly increasing in both frequency and magnitude. Floods are the most damaging of these disasters, with approximately 1.46 billion people exposed to inundation depths of over 0.15m, a significant life and livelihood risk. Accurate knowledge of flood-extent for ongoing and historical events facilitates climate adaptation in flood-prone communities by enabling near real-time disaster monitoring to support planning, response, and relief during these extreme events. Satellite observations can be used to derive flood-extent maps directly; however, these observations are impeded by cloud and canopy cover, and can be very infrequent and hence miss the flood completely. In contrast, physically-based inundation models can produce spatially complete event maps but suffer from high uncertainty if not frequently calibrated with expensive land and infrastructure surveys. In this study, we propose a deep learning approach to reproduce satellite-observed fractional flood-extent maps given dynamic state variables from hydrologic models, fusing information contained within the states with direct observations from satellites. Our model has an hourly temporal resolution, contains no cloud-gaps, and generalizes to watersheds across the continental United States with a 6% error on held-out areas that never flooded before. We further demonstrate through a case study in Houston, Texas that our model can distinguish tropical cyclones that caused flooding from those that did not within two days of landfall, thereby providing a reliable source for flood-extent maps that can be used by disaster monitoring services.

Authors: Tanya Nair (Cloud To Street); Veda Sunkara (Cloud to Street); Jonathan Frame (Cloud to Street); Philip Popien (Cloud to Street); Subit Chakrabarti (Cloud To Street)
NeurIPS 2022 Convolutional Neural Processes for Inpainting Satellite Images: Application to Water Body Segmentation (Papers Track)
Abstract and authors: (click to expand)

Abstract: The widespread availability of satellite images has allowed researchers to monitor the impact of climate on socio-economic and environmental issues through examples like crop and water body classification to measure food scarcity and risk of flooding. However, a common issue of satellite images is missing values due to measurement defects, which render them unusable by existing methods without data imputation. To repair the data, inpainting methods can be employed, which are based on classical PDEs or interpolation methods. Recently, deep learning approaches have shown promise in this realm, however many of these methods do not explicitly take into account the inherent spatio-temporal structure of satellite images. In this work, we cast satellite image inpainting as a meta-learning problem, and implement Convolutional Neural Processes (ConvNPs) in which we frame each satellite image as its own task or 2D regression problem. We show that ConvNPs outperform classical methods and state-of-the-art deep learning inpainting models on a scanline problem for LANDSAT 7 satellite images, assessed on a variety of in- and out-of-distribution images. Our results successfully match the performance of clean images on a downstream water body segmentation task in Canada.

Authors: Alexander Pondaven (Imperial College London); Mart Bakler (Imperial College London); Donghu Guo (Imperial College London); Hamzah Hashim (Imperial College London); Martin G Ignatov (Imperial college London); Samir Bhatt (Imperial College London); Seth Flaxman (Oxford); Swapnil Mishra (Imperial College London); Elie Alhajjar (USMA); Harrison Zhu (Imperial College London)
NeurIPS 2022 Towards a spatially transferable super resolution model for downscaling Antarctic surface melt (Papers Track)
Abstract and authors: (click to expand)

Abstract: Surface melt on the Antarctic Ice Sheet is an important climate indicator, yet the spatial scale of modeling and observing surface melt is insufficient to capture crucial details and understand local processes. High-resolution climate models could provide a solution, but they are computationally expensive and require finetuning for some model parameters. An alternative method, pioneering in geophysics, is single-image super resolution (SR) applied on lower-resolution model output. However, often input and output of such SR models are available on the same, fixed spatial domain. High-resolution model simulations over Antarctica are available only in some regions. To be able to apply an SR model elsewhere, we propose to make the single-image SR model physics-aware, using surface albedo and elevation as additional input. Our results show a great improvement in the spatial transferability of the conventional SR model. Although issues with the input satellite-derived albedo remain, adding physics awareness paves a way toward a spatially transferable SR model for downscaling Antarctic surface melt.

Authors: Zhongyang Hu (IMAU); Yao Sun (TUM); Peter Kuipers Munneke (IMAU); Stef Lhermitte (TU Delft); Xiaoxiang Zhu (Technical University of Munich,Germany)
NeurIPS 2022 Exploring Randomly Wired Neural Networks for Climate Model Emulation (Papers Track)
Abstract and authors: (click to expand)

Abstract: Exploring the climate impacts of various anthropogenic emissions scenarios is key to making informed decisions for climate change mitigation and adaptation. State-of-the-art Earth system models can provide detailed insight into these impacts, but have a large associated computational cost on a per-scenario basis. This large computational burden has driven recent interest in developing cheap machine learning models for the task of climate model emulation. In this manuscript, we explore the efficacy of randomly wired neural networks for this task. We describe how they can be constructed and compare them to their standard feedforward counterparts using the ClimateBench dataset. Specifically, we replace the dense layers in multilayer perceptrons, convolutional neural networks, and convolutional long short-term memory networks with randomly wired ones and assess the impact on model performance for models with 1 million and 10 million parameters. We find average performance improvements of 4.2% across model complexities and prediction tasks, with substantial performance improvements of up to 16.4% in some cases. Furthermore, we find no significant difference in prediction speed between networks with standard feedforward dense layers and those with randomly wired layers. These findings indicate that randomly wired neural networks may be suitable direct replacements for traditional dense layers in many standard models.

Authors: William J Yik (Harvey Mudd College); Sam J Silva (The University of Southern California); Andrew Geiss (Pacific Northwest National Laboratory); Duncan Watson-Parris (University of Oxford)
NeurIPS 2022 Closing the Domain Gap -- Blended Synthetic Imagery for Climate Object Detection (Papers Track)
Abstract and authors: (click to expand)

Abstract: Object detection models have great potential to increase both the frequency and cost-efficiency of assessing climate-relevant infrastructure in satellite imagery. However, model performance can suffer when models are applied to stylistically different geographies. We propose a technique to generate synthetic imagery using minimal labeled examples of the target object at a low computational cost. Our technique blends example objects onto unlabeled images of the target domain. We show that including these synthetic images improves the average precision of a YOLOv3 object detection model when compared to a baseline and other popular domain adaptation techniques.

Authors: Caleb Kornfein (Duke University); Frank Willard (Duke University); Caroline Tang (Duke University); Yuxi Long (Duke University); Saksham Jain (Duke University); Jordan Malof (Duke University); Simiao Ren (Duke University); Kyle Bradbury (Duke University)
NeurIPS 2022 Cross Modal Distillation for Flood Extent Mapping (Papers Track)
Abstract and authors: (click to expand)

Abstract: The increasing intensity and frequency of floods is one of the many consequences of our changing climate. In this work, we explore ML techniques that improve the flood detection module of an operational early flood warning system. Our method exploits an unlabelled dataset of paired multi-spectral and Synthetic Aperture Radar (SAR) imagery to reduce the labeling requirements of a purely supervised learning method. Past attempts have used such unlabelled data by creating weak labels out of them, but end up learning the label mistakes in those weak labels. Motivated by knowledge distillation and semi supervised learning, we explore the use of a teacher to train a student with the help of a small hand labeled dataset and a large unlabelled dataset. Unlike the conventional self distillation setup, we propose a cross modal distillation framework that transfers supervision from a teacher trained on richer modality (multi-spectral images) to a student model trained on SAR imagery. The trained models are then tested on the Sen1Floods11 dataset. Our model outperforms the Sen1Floods11 SAR baselines by an absolute margin of 4.15% pixel wise Intersection-over-Union (IoU) on the test split.

Authors: Shubhika Garg (Google); Ben Feinstein (Google); Shahar Timnat (Google); Vishal V Batchu (Google); gideon dror (The Academic College of Tel-Aviv-Yaffo); Adi Gerzi Rosenthal (Google); Varun Gulshan (Google Research)
NeurIPS 2022 Estimating Chicago’s tree cover and canopy height using multi-spectral satellite imagery (Papers Track)
Abstract and authors: (click to expand)

Abstract: Information on urban tree canopies is fundamental to mitigating climate change as well as improving quality of life. Urban tree planting initiatives face a lack of up-to-date data about the horizontal and vertical dimensions of the tree canopy in cities. We present a pipeline that utilizes LiDAR data as ground-truth and then trains a multi-task machine learning model to generate reliable estimates of tree cover and canopy height in urban areas using multi-source multi-spectral satellite imagery for the case study of Chicago.

Authors: John Francis (University College London)
NeurIPS 2022 Generalized Ice Detection on Wind Turbine Rotor Blades with Neural Style Transfer (Papers Track)
Abstract and authors: (click to expand)

Abstract: Wind energy’s ability to liberate the world of conventional sources of energy relies on lowering the significant costs associated with the maintenance of wind turbines. Since icing events on turbine rotor blades are a leading cause of operational failures, identifying icing in advance is critical. Some recent studies focus on specific wind parks and fail to generalize to unseen scenarios (e.g. new rotor blade designs). We propose the utilisation of synthetic data augmentation via neural style transfer to improve the generalization of existing ice prediction models. We show that training models with augmented data that captures domain-invariant icing characteristics can help improve predictive performance across multiple wind parks. Through efficient identification of icing, this study can support preventive maintenance of wind energy sources by making them more reliable towards tackling climate change.

Authors: Joyjit Chatterjee (University of Hull); Maria Teresa Alvela Nieto (University of Bremen); Hannes Gelbhardt (University of Bremen); Nina Dethlefs (University of Hull); Jan Ohlendorf (University of Bremen); Klaus-Dieter Thoben (University of Bremen)
NeurIPS 2022 Deep Learning for Rapid Landslide Detection using Synthetic Aperture Radar (SAR) Datacubes (Papers Track)
Abstract and authors: (click to expand)

Abstract: With climate change predicted to increase the likelihood of landslide events, there is a growing need for rapid landslide detection technologies that help inform emergency responses. Synthetic Aperture Radar (SAR) is a remote sensing technique that can provide measurements of affected areas independent from weather or lighting conditions. Usage of SAR, however, is hindered by domain knowledge that is necessary for the pre-processing steps and its interpretation requires expert knowledge. We provide simplified, pre-processed, machine-learning ready SAR datacubes for four globally located landslide events obtained from several Sentinel-1 satellite passes before and after a landslide triggering event together with segmentation maps of the landslides. From this dataset, using the Hokkaido, Japan datacube, we study the feasibility of SAR-based landslide detection with supervised deep learning (DL). Our results demonstrate that DL models can be used to detect landslides from SAR data, achieving an Area under the Precision-Recall curve exceeding 0.7. We find that additional satellite visits enhance detection performance, but that early detection is possible when SAR data is combined with terrain information from a digital elevation model. This can be especially useful for time-critical emergency interventions.

Authors: Vanessa Boehm (UC Berkeley); Wei Ji Leong (The Ohio State University); Ragini Bal Mahesh (German Aerospace Center DLR); Ioannis Prapas (National Observatory of Athens); Siddha Ganju (Nvidia); Freddie Kalaitzis (University of Oxford); Edoardo Nemni (United Nations Satellite Centre (UNOSAT)); Raul Ramos-Pollan (Universidad de Antioquia)
NeurIPS 2022 Hybrid Recurrent Neural Network for Drought Monitoring (Papers Track)
Abstract and authors: (click to expand)

Abstract: Droughts are pervasive hydrometeorological phenomena and global hazards, whose frequency and intensity are expected to increase in the context of climate change. Drought monitoring is of paramount relevance. Here we propose a hybrid model for drought detection that integrates both climatic indices and data-driven models in a hybrid deep learning approach. We exploit time-series of multi-scale Standardized Precipitation Evapotranspiration Index together with precipitation and temperature as inputs. We introduce a dual-branch recurrent neural network with convolutional lateral connections for blending the data. Experimental and ablative results show that the proposed system outperforms both the considered drought index and purely data-driven deep learning models. Our results suggest the potential of hybrid models for drought monitoring and open the door to synergistic systems that learn from data and domain knowledge altogether.

Authors: Mengxue Zhang (Universitat de València); Miguel-Ángel Fernández-Torres (Universitat de València); Gustau Camps-Valls (Universitat de València)
NeurIPS 2022 Deep Learning for Global Wildfire Forecasting (Papers Track)
Abstract and authors: (click to expand)

Abstract: Climate change is expected to aggravate wildfire activity through the exacerbation of fire weather. Improving our capabilities to anticipate wildfires on a global scale is of uttermost importance for mitigating their negative effects. In this work, we create a global fire dataset and demonstrate a prototype for predicting the presence of global burned areas on a sub-seasonal scale with the use of segmentation deep learning models. Particularly, we present an open-access global analysis-ready datacube, which contains a variety of variables related to the seasonal and sub-seasonal fire drivers (climate, vegetation, oceanic indices, human-related variables), as well as the historical burned areas and wildfire emissions for 2001-2021. We train a deep learning model, which treats global wildfire forecasting as an image segmentation task and skillfully predicts the presence of burned areas 8, 16, 32 and 64 days ahead of time. Our work motivates the use of deep learning for global burned area forecasting and paves the way towards improved anticipation of global wildfire patterns.

Authors: Ioannis Prapas (National Observatory of Athens); Akanksha Ahuja (NOA); Spyros Kondylatos (National Observatory of Athens); Ilektra Karasante (National Observatory of Athens); Lazaro Alonso (Max Planck Institute for Biogeochemistry); Eleanna Panagiotou (Harokopio University of Athens); Charalampos Davalas (Harokopio University of Athens); Dimitrios Michail (Harokopio University of Athens); Nuno Carvalhais (Max Planck Institute for Biogeochemistry); Ioannis Papoutsis (National Observatory of Athens)
NeurIPS 2022 Machine Learning for Activity-Based Road Transportation Emissions Estimation (Papers Track)
Abstract and authors: (click to expand)

Abstract: Measuring and attributing greenhouse gas (GHG) emissions remains a challenging problem as the world strives towards meeting emissions reductions targets. As a significant portion of total global emissions, the road transportation sector represents an enormous challenge for estimating and tracking emissions at a global scale. To meet this challenge, we have developed a hybrid approach for estimating road transportation emissions that combines the strengths of machine learning and satellite imagery with localized emissions factors data to create an accurate, globally scalable, and easily configurable GHG monitoring framework.

Authors: Derek Rollend (JHU); Kevin Foster (JHU); Tomek Kott (JHU); Rohita Mocharla (JHU); Rodrigo Rene Rai Munoz Abujder (Johns Hopkins Applied Physics Laboratory); Neil Fendley (JHU/APL); Chace Ashcraft (JHU/APL); Frank Willard (JHU); Marisa Hughes (JHU)
NeurIPS 2022 Comparing the carbon costs and benefits of low-resource solar nowcasting (Papers Track)
Abstract and authors: (click to expand)

Abstract: Mitigating emissions in line with climate goals requires the rapid integration of low carbon energy sources, such as solar photovoltaics (PV) into the electricity grid. However, the energy produced from solar PV fluctuates due to clouds obscuring the sun's energy. Solar PV yield nowcasting is used to help anticipate peaks and troughs in demand to support grid integration. This paper compares multiple low-resource approaches to nowcasting solar PV yield. To do so, we use a dataset of UK satellite imagery and solar PV energy readings over a 1 to 4-hour time range. Our work investigates the performance of multiple nowcasting models. The paper also estimates the carbon emissions generated and averted by deploying models such as these, and finds that short-term PV forecasting may have a benefit several orders of magnitude greater than its carbon cost and that this benefit holds for small models that could be deployable in low-resource settings around the globe.

Authors: Ben Dixon (UCL); Jacob Bieker (Open Climate Fix); Maria Perez-Ortiz (University College London)
NeurIPS 2022 EnhancedSD: Downscaling Solar Irradiance from Climate Model Projections (Papers Track)
Abstract and authors: (click to expand)

Abstract: Renewable energy-based electricity systems are seen as a keystone of future decarbonization efforts. However, power system planning does not currently consider the impacts of climate change on renewable energy resources such as solar energy, chiefly due to a paucity of climate-impacted high-resolution solar power data. Existing statistical downscaling (SD) methods that learn to map coarse-resolution versions of historical reanalysis data to generate finer resolution outputs are of limited use when applied to future climate model projections due to the domain gap between climate models and reanalysis data. In contrast, we present EnhancedSD, a deep learning-based framework for downscaling coarse-scale climate model outputs to high-resolution observational (reanalysis) data. Our proposed ML based downscaling allows for future reanalysis projections, which can be pivotal for mitigating climate change’s impacts on power systems planning.

Authors: Nidhin Harilal (University of Colorado, Boulder); Bri-Mathias S Hodge (University of Colorado Boulder); Claire Monteleoni (University of Colorado Boulder); Aneesh Subramanian (University of California, San Diego)
NeurIPS 2022 Image-based Early Detection System for Wildfires (Papers Track)
Abstract and authors: (click to expand)

Abstract: Wildfires are a disastrous phenomenon which cause damage to land, loss of property, air pollution, and even loss of human life. Due to the warmer and drier conditions created by climate change, more severe and uncontrollable wildfires are expected to occur in the coming years. This could lead to a global wildfire crisis and have dire consequences on our planet. Hence, it has become imperative to use technology to help prevent the spread of wildfires. One way to prevent the spread of wildfires before they become too large is to perform early detection i.e, detecting the smoke before the actual fire starts. In this paper, we present our Wildfire Detection and Alert System which use machine learning to detect wildfire smoke with a high degree of accuracy and can send immediate alerts to users. Our technology is currently being used in the USA to monitor data coming in from hundreds of cameras daily. We show that our system has a high true detection rate and a low false detection rate. Our performance evaluation study also shows that on an average our system detects wildfire smoke faster than an actual person.

Authors: Omkar Ranadive (Alchera X); Jisu Kim (Alchera); Serin Lee (Alchera X); Youngseo Cha (Alchera); Heechan Park (Alchera); Minkook Cho (Alchera); Young Hwang (Alchera)
NeurIPS 2022 Towards Global Crop Maps with Transfer Learning (Papers Track)
Abstract and authors: (click to expand)

Abstract: The continuous increase in global population and the impact of climate change on crop production are expected to affect the food sector significantly. In this context, there is need for timely, large-scale and precise mapping of crops for evidence-based decision making. A key enabler towards this direction are new satellite missions that freely offer big remote sensing data of high spatio-temporal resolution and global coverage. During the previous decade and because of this surge of big Earth observations, deep learning methods have dominated the remote sensing and crop mapping literature. Nevertheless, deep learning models require large amounts of annotated data that are scarce and hard-to-acquire. To address this problem, transfer learning methods can be used to exploit available annotations and enable crop mapping for other regions, crop types and years of inspection. In this work, we have developed and trained a deep learning model for paddy rice detection in South Korea using Sentinel-1 VH time-series. We then fine-tune the model for i) paddy rice detection in France and Spain and ii) barley detection in the Netherlands. Additionally, we propose a modification in the pre-trained weights in order to incorporate extra input features (Sentinel-1 VV). Our approach shows excellent performance when transferring in different areas for the same crop type and rather promising results when transferring in a different area and crop type.

Authors: Hyun-Woo Jo (Korea University); Alkiviadis Marios Koukos (National Observatory of Athens); Vasileios Sitokonstantinou (National Observatory of Athens); Woo-Kyun Lee (Korea University); Charalampos Kontoes (National Observatory of Athens)
NeurIPS 2022 Pyrocast: a Machine Learning Pipeline to Forecast Pyrocumulonimbus (PyroCb) Clouds (Papers Track)
Abstract and authors: (click to expand)

Abstract: Pyrocumulonimbus (pyroCb) clouds are storm clouds generated by extreme wildfires. PyroCbs are associated with unpredictable, and therefore dangerous, wildfire spread. They can also inject smoke particles and trace gases into the upper troposphere and lower stratosphere, affecting the Earth's climate. As global temperatures increase, these previously rare events are becoming more common. Being able to predict which fires are likely to generate pyroCb is therefore key to climate adaptation in wildfire-prone areas. This paper introduces Pyrocast, a pipeline for pyroCb analysis and forecasting. The pipeline's first two components, a pyroCb database and a pyroCb forecast model, are presented. The database brings together geostationary imagery and environmental data for over 148 pyroCb events across North America, Australia, and Russia between 2018 and 2022. Random Forests, Convolutional Neural Networks (CNNs), and CNNs pretrained with Auto-Encoders were tested to predict the generation of pyroCb for a given fire six hours in advance. The best model predicted pyroCb with an AUC of 0.90±0.04.

Authors: Kenza Tazi (University of Cambridge); Emiliano Díaz Salas-Porras (University of Valencia); Ashwin Braude (Institut Pierre-Simon Laplace); Daniel Okoh (National Space Research and Development Agency); Kara D. Lamb (Columbia University); Duncan Watson-Parris (University of Oxford); Paula Harder (Fraunhofer ITWM); Nis Meinert (Pasteur Labs)
NeurIPS 2022 Nowformer : A Locally Enhanced Temporal Learner for Precipitation Nowcasting (Papers Track)
Abstract and authors: (click to expand)

Abstract: The precipitation video datasets have distinctive meteorological patterns where a mass of fluid moves in a particular direction across the entire frames, and each local area of the fluid has an individual life cycle from initiation to maturation to decay. This paper proposes a novel transformer-based model for precipitation nowcasting that can extract global and local dynamics within meteorological characteristics. The experimental results show our model achieves state-of-the-art performances on the precipitation nowcasting benchmark.

Authors: Jinyoung Park (KAIST); Inyoung Lee (KAIST); Minseok Son (KAIST); Seungju Cho (KAIST); Changick Kim (KAIST)
NeurIPS 2022 Multimodal Wildland Fire Smoke Detection (Papers Track)
Abstract and authors: (click to expand)

Abstract: Research has shown that climate change creates warmer temperatures and drier conditions, leading to longer wildfire seasons and increased wildfire risks in the United States. These factors have in turn led to increases in the frequency, extent, and severity of wildfires in recent years. Given the danger posed by wildland fires to people, property, wildlife, and the environment, there is an urgency to provide tools for effective wildfire management. Early detection of wildfires is essential to minimizing potentially catastrophic destruction. In this paper, we present our work on integrating multiple data sources in SmokeyNet, a deep learning model using spatio-temporal information to detect smoke from wildland fires. Camera image data is integrated with weather sensor measurements and processed by SmokeyNet to create a multimodal wildland fire smoke detection system. Our results show that incorporating multimodal data in SmokeyNet improves performance in terms of both F1 and time-to-detection over the baseline with a single data source. With a time-to-detection of only a few minutes, SmokeyNet can serve as an automated early notification system, providing a useful tool in the fight against destructive wildfires.

Authors: Mai Nguyen (University of California San Diego); Shreyas Anantha Ramaprasad (University of California San Diego); Jaspreet Kaur Bhamra (University of California San Diego); Siddhant Baldota (University of California San Diego); Garrison Cottrell (UC San Diego)
NeurIPS 2022 Accessible Large-Scale Plant Pathology Recognition (Papers Track)
Abstract and authors: (click to expand)

Abstract: Plant diseases are costly and threaten agricultural production and food security worldwide. Climate change is increasing the frequency and severity of plant diseases and pests. Therefore, detection and early remediation can have a significant impact, especially in developing countries. However, AI solutions are yet far from being in production. The current process for plant disease diagnostic consists of manual identification and scoring by humans, which is time-consuming, low-supply, and expensive. Although computer vision models have shown promise for efficient and automated plant disease identification, there are limitations for real-world applications: a notable variation in visual symptoms of a single disease, different light and weather conditions, and the complexity of the models. In this work, we study the performance of efficient classification models and training "tricks" to solve this problem. Our analysis represents a plausible solution for these ecological disasters and might help to assist producers worldwide. More information available at: https://github.com/mv-lab/mlplants

Authors: Marcos V. Conde (University of Würzburg); Dmitry Gordeev (H2O.ai)
NeurIPS 2022 Learning to forecast vegetation greenness at fine resolution over Africa with ConvLSTMs (Papers Track)
Abstract and authors: (click to expand)

Abstract: Forecasting the state of vegetation in response to climate and weather events is a major challenge. Its implementation will prove crucial in predicting crop yield, forest damage, or more generally the impact on ecosystems services relevant for socio-economic functioning, which if absent can lead to humanitarian disasters. Vegetation status depends on weather and environmental conditions that modulate complex ecological processes taking place at several timescales. Interactions between vegetation and different environmental drivers express responses at instantaneous but also time-lagged effects, often showing an emerging spatial context at landscape and regional scales. We formulate the land surface forecasting task as a strongly guided video prediction task where the objective is to forecast the vegetation developing at very fine resolution using topography and weather variables to guide the prediction. We use a Convolutional LSTM (ConvLSTM) architecture to address this task and predict changes in the vegetation state in Africa using Sentinel-2 satellite NDVI, having ERA5 weather reanalysis, SMAP satellite measurements, and topography (DEM of SRTMv4.1) as variables to guide the prediction. Ours results highlight how ConvLSTM models can not only forecast the seasonal evolution of NDVI at high resolution, but also the differential impacts of weather anomalies over the baselines. The model is able to predict different vegetation types, even those with very high NDVI variability during target length.

Authors: Claire Robin (Biogeochemical Integration, Max-Planck-Institute for Biogeochemistry, Jena, Germany); Christian Requena-Mesa (Computer Vision Group, Friedrich Schiller University Jena; DLR Institute of Data Science, Jena; Max Planck Institute for Biogeochemistry, Jena); Vitus Benson (Max-Planck-Institute for Biogeochemistry); Jeran Poehls (Max-Planck-Institute for Biogeochemistry); Lazaro Alonzo (Max-Planck-Institute for Biogeochemistry Max-Planck-Institute for Biogeochemistry); Nuno Carvalhais (Max-Planck-Institute for Biogeochemistry); Markus Reichstein (Max Planck Institute for Biogeochemistry, Jena; Michael Stifel Center Jena for Data-Driven and Simulation Science, Jena)
NeurIPS 2022 Generative Modeling of High-resolution Global Precipitation Forecasts (Papers Track)
Abstract and authors: (click to expand)

Abstract: Forecasting global precipitation patterns and, in particular, extreme precipitation events is of critical importance to preparing for and adapting to climate change. Making accurate high-resolution precipitation forecasts using traditional physical models remains a major challenge in operational weather forecasting as they incur substantial computational costs and struggle to achieve sufficient forecast skill. Recently, deep-learning-based models have shown great promise in closing the gap with numerical weather prediction (NWP) models in terms of precipitation forecast skill, opening up exciting new avenues for precipitation modeling. However, it is challenging for these deep learning models to fully resolve the fine-scale structures of precipitation phenomena and adequately characterize the extremes of the long-tailed precipitation distribution. In this work, we present several improvements to the architecture and training process of a current state-of-the art deep learning precipitation model (FourCastNet) using a novel generative adversarial network (GAN) to better capture fine scales and extremes. Our improvements achieve superior performance in capturing the extreme percentiles of global precipitation, while comparable to state-of-the-art NWP models in terms of forecast skill at 1--2 day lead times. Together, these improvements set a new state-of-the-art in global precipitation forecasting.

Authors: James Duncan (University of California, Berkeley); Peter Harrington (Lawrence Berkeley National Laboratory (Berkeley Lab)); Shashank Subramanian (Lawrence Berkeley National Laboratory)
NeurIPS 2022 Performance evaluation of deep segmentation models on Landsat-8 imagery (Papers Track)
Abstract and authors: (click to expand)

Abstract: Contrails, short for condensation trails, are line-shaped ice clouds produced by aircraft engine exhaust when they fly through the cold and humid air. They generate a greenhouse effect by absorbing or directing back to Earth approximately 33% of emitted outgoing longwave radiation. They account for over half of the climate change resulting from aviation activities. Avoiding contrails and adjusting flight routes could be an inexpensive and effective way to reduce their impact. An accurate, automated, and reliable detection algorithm is required to develop and evaluate contrail avoidance strategies. Advancement in contrail detection has been severely limited due to several factors, primarily due to a lack of quality-labelled data. Recently, McCloskey et al. proposed a large human-labelled Landsat-8. Each contrail is carefully labelled with various inputs in various scenes of Landsat-8 satellite imagery. In this work, we benchmark several popular segmentation models with combinations of different loss functions and encoder backbones. This work is the first to apply state-of-the-art segmentation techniques to detect contrails in low-orbit satellite imagery. Our work can also be used as an open benchmark for contrail segmentation.

Authors: Akshat Bhandari (Manipal Institute of Technology, Manipal); Pratinav Seth (Manipal Institute of Technology); Sriya Rallabandi (Manipal Institute of Technology); Aditya Kasliwal (Manipal Institute of Technology); Sanchit Singhal (Manipal Institute of Technology)
NeurIPS 2022 Guided Transformer Network for Detecting Methane Emissions in Sentinel-2 Satellite Imagery (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Methane (CH4) is the chief contributor to global climate change and its mitigation is targeted by the EU, US and jurisdictions worldwide [2]. Recent studies have shown that imagery from the multi-spectral instrument on Sentinel-2 satellites is capable of detecting and estimating large methane emissions. However, most of the current methods rely on temporal relations between a ratio of shortwave-infrared spectra and assume relatively constant ground conditions, and availability of ground information on when there was no methane emission on site. To address such limitations we propose a guided query-based transformer neural network architecture, that will detect and quantify methane emissions without dependence on temporal information. The guided query aspect of our architecture is driven by a Sentinel Enhanced Matched Filter (SEMF) approach, also discussed in this paper. Our network uses all 12 spectral channels of Sentinel-2 imagery to estimate ground terrain and detect methane emissions. No dependence on temporal data makes it more robust to changing ground and terrain conditions and more computationally efficient as it reduces the need to process historical time-series imagery to compute a single date emissions analysis.

Authors: Satish Kumar (University of California, Santa Barbara); William Kingwill (Orbio Earth); Rozanne Mouton (Orbio Earth); Wojciech Adamczyk (ETH, Zurich); Robert Huppertz (Orbio Earth); Evan D Sherwin (Stanford University, Energy and Resources Engineering)
NeurIPS 2022 Towards Low Cost Automated Monitoring of Life Below Water to De-risk Ocean-Based Carbon Dioxide Removal and Clean Power (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Oceans will play a crucial role in our efforts to combat the growing climate emergency. Researchers have proposed several strategies to harness greener energy through oceans and use oceans as carbon sinks. However, the risks these strategies might pose to the ocean and marine ecosystem are not well understood. It is imperative that we quickly develop a range of tools to monitor ocean processes and marine ecosystems alongside the technology to deploy these solutions on a large scale into the oceans. Large arrays of inexpensive cameras placed deep underwater coupled with machine learning pipelines to automatically detect, classify, count and estimate fish populations have the potential to continuously monitor marine ecosystems and help study the impacts of these solutions on the ocean. In this proposal, we discuss the challenges presented by a dark artificially lit underwater video dataset captured 500m below the surface, propose potential solutions to address these challenges, and present preliminary results from detecting and classifying 6 species of fish in deep underwater camera data.

Authors: Kameswari Devi Ayyagari (Dalhousie University); Christopher Whidden (Dalhousie University); Corey Morris (Department of Fisheries and Oceans); Joshua Barnes (National Research Council Canada)
NeurIPS 2022 Towards the Automatic Analysis of Ceilometer Backscattering Profiles using Unsupervised Learning (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Ceilometers use a laser beam to capture certain phenomena in the atmosphere like clouds, precipitation, or aerosol layers. These measurements can be visualized in so-called quick looks that at the moment are mostly analyzed manually by meteorology experts. In this work, we illustrate the path towards the automatic analysis of quick looks by using a hybrid approach combining an image segmentation algorithm with unsupervised representation learning and clustering. We present a first proof of concept and give an outlook on possible future work.

Authors: Michael Dammann (HAW Hamburg); Ina Mattis (Deutscher Wetterdienst); Michael Neitzke (HAW Hamburg); Ralf Möller (University of Lübeck)
NeurIPS 2022 Detecting Floods from Cloudy Scenes: A Fusion Approach Using Sentinel-1 and Sentinel-2 Imagery (Proposals Track)
Abstract and authors: (click to expand)

Abstract: As the result of climate change, extreme flood events are becoming more frequent. To better respond to such disasters, and to test and calibrate flood models, we need accurate real-world data on flooding extent. Detection of floods from remote sensed imagery suffers from a widespread problem: clouds block flood scenes in images, leading to degraded and fragmented flood datasets. To address this challenge, we propose a workflow based on U-Net, and a dataset that detects flood in cloud-prone areas by fusing information from the Sentinel-1 and Sentinel-2 satellites. The expected result will be a reliable and detailed catalogue of flood extents and how they change through time, allowing us to better understand flooding in different morphological settings and climates.

Authors: Qiuyang Chen (University of Edinburgh); Xenofon Karagiannis (Earth-i Ltd.); Simon M. Mudd (University of Edinburgh)
NeurIPS 2022 Urban Heat Island Detection and Causal Inference Using Convolutional Neural Networks (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Compared to rural areas, urban areas experience higher temperatures for longer periods of time because of the urban heat island (UHI) effect. This increased heat stress leads to greater mortality, increased energy demand, regional changes to precipitation patterns, and increased air pollution. Urban developers can minimize the UHI effect by incorporating features that promote air flow and heat dispersion (e.g., increasing green space). However, understanding which urban features to implement is complex, as local meteorology strongly dictates how the environment responds to changes in urban form. In this proposal we describe a methodology for estimating the causal relationship between changes in urban form and changes in the UHI effect. Changes in urban form and temperature changes are measured using convolutional neural networks, and a causal inference matching approach is proposed to estimate causal relationships. The success of this methodology will enable urban developers to implement city-specific interventions to mitigate the warming planet's impact on cities.

Authors: Zachary D Calhoun (Duke University); Ziyang Jiang (Duke University); Mike Bergin (Duke University); David Carlson (Duke University)
NeurIPS 2022 ForestBench: Equitable Benchmarks for Monitoring, Reporting, and Verification of Nature-Based Solutions with Machine Learning (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Restoring ecosystems and reducing deforestation are necessary tools to mitigate the anthropogenic climate crisis. Current measurements of forest carbon stock can be inaccurate, in particular for underrepresented and small-scale forests in the Global South, hindering transparency and accountability in the Monitoring, Reporting, and Verification (MRV) of these ecosystems. There is thus need for high quality datasets to properly validate ML-based solutions. To this end, we present ForestBench, which aims to collect and curate geographically-balanced gold-standard datasets of small-scale forest plots in the Global South, by collecting ground-level measurements and visual drone imagery of individual trees. These equitable validation datasets for ML-based MRV of nature-based solutions shall enable assessing the progress of ML models for estimating above-ground biomass, ground cover, and tree species diversity.

Authors: Lucas Czech (Carnegie Institution for Science); Björn Lütjens (MIT); David Dao (ETH Zurich)
NeurIPS 2022 Estimating Heating Loads in Alaska using Remote Sensing and Machine Learning Methods (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Alaska and the larger Arctic region are in much greater need of decarbonization than the rest of the globe as a result of the accelerated consequences of climate change over the past ten years. Heating for homes and businesses accounts for over 75% of the energy used in the Arctic region. However, the lack of thorough and precise heating load estimations in these regions poses a significant obstacle to the transition to renewable energy. In order to accurately measure the massive heating demands in Alaska, this research pioneers a geospatial-first methodology that integrates remote sensing and machine learning techniques. Building characteristics such as height, size, year of construction, thawing degree days, and freezing degree days are extracted using open-source geospatial information in Google Earth Engine (GEE). These variables coupled with heating load forecasts from the AK Warm simulation program are used to train models that forecast heating loads on Alaska’s Railbelt utility grid. Our research greatly advances geospatial capability in this area and considerably informs the decarbonization activities currently in progress in Alaska.

Authors: Madelyn Gaumer (University of Washington); Nick Bolten (Paul G. Allen School of Computer Science and Engineering, University of Washington); Vidisha Chowdhury (Heinz College of Information Systems and Public Policy, Carnegie Mellon University); Philippe Schicker (Heinz College of Information Systems and Public Policy, Carnegie Mellon University); Shamsi Soltani (Department of Epidemiology and Population Health, Stanford University School of Medicine); Erin D Trochim (University of Alaska Fairbanks)
NeurIPS 2022 Interpretable Spatiotemporal Forecasting of Arctic Sea Ice Concentration at Seasonal Lead Times (Proposals Track)
Abstract and authors: (click to expand)

Abstract: There are many benefits from the accurate forecasting of Arctic sea ice, however existing models struggle to reliably predict sea ice concentration at long lead times. Many numerical models exist but can be sensitive to initial conditions, and while recent deep learning-based methods improve overall robustness, they either do not utilize temporal trends or rely on architectures that are not performant at learning long-term sequential dependencies. We propose a method of forecasting sea ice concentration using neural circuit policies, a form of continuous time recurrent neural architecture, which improve the learning of long-term sequential dependencies compared to existing techniques and offer the added benefits of adaptability to irregular sequence intervals and high interpretability.

Authors: Matthew Beveridge (Independent Researcher); Lucas Pereira (ITI, LARSyS, Técnico Lisboa)
NeurIPS 2022 Disaster Risk Monitoring Using Satellite Imagery (Tutorials Track)
Abstract and authors: (click to expand)

Abstract: Natural disasters such as flood, wildfire, drought, and severe storms wreak havoc throughout the world, causing billions of dollars in damages, and uprooting communities, ecosystems, and economies. Unfortunately, flooding events are on the rise due to climate change and sea level rise. The ability to detect and quantify them can help us minimize their adverse impacts on the economy and human lives. Using satellites to study flood is advantageous since physical access to flooded areas is limited and deploying instruments in potential flood zones can be dangerous. We are proposing a hands-on tutorial to highlight the use of satellite imagery and computer vision to study natural disasters. Specifically, we aim to demonstrate the development and deployment of a flood detection model using Sentinel-1 satellite data. The tutorial will cover relevant fundamental concepts as well as the full development workflow of a deep learning-based application. We will include important considerations such as common pitfalls, data scarcity, augmentation, transfer learning, fine-tuning, and details of each step in the workflow. Importantly, the tutorial will also include a case study on how the application was used by authorities in response to a flood event. We believe this tutorial will enable machine learning practitioners of all levels to develop new technologies that tackle the risks posed by climate change. We expect to deliver the below learning outcomes: • Develop various deep learning-based computer vision solutions using hardware-accelerated open-source tools that are optimized for real-time deployment • Create an optimized pipeline for the machine learning development workflow • Understand different performance metrics for model evaluation that are relevant for real world datasets and data imbalances • Understand the public sector’s efforts to support climate action initiatives and point out where the audience can contribute

Authors: Kevin Lee (NVIDIA); Siddha Ganju (NVIDIA); Edoardo Nemni (UNOSAT)
NeurIPS 2022 Automating the creation of LULC datasets for semantic segmentation (Tutorials Track)
Abstract and authors: (click to expand)

Abstract: High resolution and accurate Land Use and Land Cover mapping (LULC) datasets are increasingly important and can be widely used in monitoring climate change impacts in agriculture, deforestation, and the carbon cycle. These datasets represent physical classifications of land types and spatial information over the surface of the Earth. These LULC datasets can be leveraged in a plethora of research topics and industries to mitigate and adapt to environmental changes. High resolution urban mappings can be used to better monitor and estimate building albedo and urban heat island impacts, and accurate representation of forests and vegetation can even be leveraged to better monitor the carbon cycle and climate change through improved land surface modelling. The advent of machine learning (ML) based CV techniques over the past decade provides a viable option to automate LULC mapping. One impediment to this has been the lack of large ML datasets. Large vector datasets for LULC are available, but can’t be used directly by ML practitioners due to a knowledge gap in transforming the input into a dataset of paired satellite images and segmentation masks. We demonstrate a novel end-to-end pipeline for LULC dataset creation that takes vector land cover data and provides a training-ready dataset. We will use Sentinel-2 satellite imagery and the European Urban Atlas LULC data. The pipeline manages everything from downloading satellite data, to creating and storing encoded segmentation masks and automating data checks. We then use the resulting dataset to train a semantic segmentation model. The aim of the pipeline is to provide a way for users to create their own custom datasets using various combinations of multispectral satellite and vector data. In addition to presenting the pipeline, we aim to provide an introduction to multispectral imagery, geospatial data and some of the challenges in using it for ML.

Authors: Sambhav S Rohatgi (Spacesense.ai); Anthony Mucia (Spacesense.ai)
AAAI FSS 2022 Towards Generating Large Synthetic Phytoplankton Datasets for Efficient Monitoring of Harmful Algal Blooms
Abstract and authors: (click to expand)

Abstract: Climate change is increasing the frequency and severity of harmful algal blooms (HABs), which cause significant fish deaths in aquaculture farms. This contributes to ocean pollution and greenhouse gas (GHG) emissions since dead fish are either dumped into the ocean or taken to landfills, which in turn negatively impacts the climate. Currently, the standard method to enumerate harmful algae and other phytoplankton is to manually observe and count them under a microscope. This is a time-consuming, tedious and error-prone process, resulting in compromised management decisions by farmers. Hence, automating this process for quick and accurate HAB monitoring is extremely helpful. However, this requires large and diverse datasets of phytoplankton images, and such datasets are hard to produce quickly. In this work, we explore the feasibility of generating novel high-resolution photorealistic synthetic phytoplankton images, containing multiple species in the same image, given a small dataset of real images. To this end, we employ Generative Adversarial Networks (GANs) to generate synthetic images. We evaluate three different GAN architectures: ProjectedGAN, FastGAN, and StyleGANv2 using standard image quality metrics. We empirically show the generation of high-fidelity synthetic phytoplankton images using a training dataset of only 961 real images. Thus, this work demonstrates the ability of GANs to create large synthetic datasets of phytoplankton from small training datasets, accomplishing a key step towards sustainable systematic monitoring of harmful algal blooms.

Authors: Nitpreet Bamra (University of Waterloo), Vikram Voleti (Mila, University of Montreal), Alexander Wong (University of Waterloo) and Jason Deglint (University of Waterloo)
AAAI FSS 2022 Discovering Transition Pathways Towards Coviability with Machine Learning
Abstract and authors: (click to expand)

Abstract: This paper presents our ongoing French-Brazilian collaborative project which aims at: (1) establishing a diagnosis of socio-ecological coviability for several sites of interest in Nordeste, the North-East region of Brazil (in the states of Paraiba, Ceara, Pernambuco, and Rio Grande do Norte known for their biodiversity hotspots and vulnerabilities to climate change) using advanced data science techniques for multisource and multimodal data fusion and (2) finding transition pathways towards coviability equilibrium using machine learning techniques. Data collected in the field by scientists, ecologists, local actors combined with volunteered information, pictures from smart-phones, and data available on-line from satellite imagery, social media, surveys, etc. can be used to compute various coviability indicators of interest for the local actors. These indicators are useful to characterize and monitor the socio-ecological coviability status along various dimensions of anthropization, human welfare, ecological and biodiversity balance, and ecosystem intactness and vulnerabilities.

Authors: Laure Berti-Equille (IRD) and Rafael Raimundo (UFPB)
AAAI FSS 2022 Wildfire Forecasting with Satellite Images and Deep Generative Model
Abstract and authors: (click to expand)

Abstract: Wildfire prediction has been one of the most critical tasks that humanities want to thrive at. While it plays a vital role in protecting human life, it is also difficult because of its stochastic and chaotic properties. We tackled the problem by interpreting a series of wildfire images into a video and used it to anticipate how the fire would behave in the future. However, creating video prediction models that account for the inherent uncertainty of the future is challenging. The bulk of published attempts are based on stochastic image-autoregressive recurrent networks, which raise various performance and application difficulties such as computational cost and limited efficiency on massive datasets. Another possibility is to use entirely latent temporal models that combine frame synthesis with temporal dynamics. However, due to design and training issues, no such model for stochastic video prediction has yet been proposed in the literature. This paper addresses these issues by introducing a novel stochastic temporal model whose dynamics are driven in a latent space. It naturally predicts video dynamics by allowing our lighter, more interpretable latent model to beat previous state-of-the-art approaches on the GOES-16 dataset. Results are compared using various benchmarking models.

Authors: Thai-Nam Hoang (University of Wisconsin - Madison), Sang Truong (Stanford University) and Chris Schmidt (University of Wisconsin - Madison)
AAAI FSS 2022 Employing Deep Learning to Quantify Power Plant Greenhouse Gas Emissions via Remote Sensing Data
Abstract and authors: (click to expand)

Abstract: Greenhouse gasses (GHG) emitted from fossil-fuel-burning power plants pose a global threat to climate and public health. GHG emissions degrade air quality and increase the frequency of natural disasters five-fold, causing 8.7 million deaths per year. Quantifying GHG emissions is crucial for the success of the planet. However, current methods to track emissions cost upwards of $520,000/plant. These methods are cost prohibitive for developing countries, and are not globally standardized, leading to inaccurate emissions reports from nations and companies. I developed a low-cost solution via an end-to-end deep learning pipeline that utilizes observations of emitted smoke plumes in satellite imagery to provide an accurate, precise system for quantifying power plant GHG emissions by segmentation of power plant smoke plumes, classification of the plant fossil fuels, and algorithmic prediction of power generation and CO2 emissions. The pipeline was able to achieve a segmentation Intersection Over Union (IoU) score of 0.841, fuel classification accuracy of 92%, and quantify power generation and CO2 emission rates with R2 values of 0.852 and 0.824 respectively. The results of this work serve as a step toward the low-cost monitoring and detection of major sources of GHG emissions, helping limit their catastrophic impacts on climate and our planet.

Authors: Aryan Jain (Amador Valley High School)
AAAI FSS 2022 Curator: Creating Large-Scale Curated Labelled Datasets using Self-Supervised Learning
Abstract and authors: (click to expand)

Abstract: Applying Machine learning to domains like Earth Sciences is impeded by the lack of labeled data, despite a large corpus of raw data available in such domains. For instance, training a wildfire classifier on satellite imagery requires curating a massive and diverse dataset, which is an expensive and time-consuming process that can span from weeks to months. Searching for relevant examples in over 40 petabytes of unlabelled data requires researchers to manually hunt for such images, much like finding a needle in a haystack. We present a no-code end-to-end pipeline, Curator, which dramatically minimizes the time taken to curate an exhaustive labeled dataset. Curator is able to search massive amounts of unlabelled data by combining self-supervision, scalable nearest neighbor search, and active learning to learn and differentiate image representations. The pipeline can also be readily applied to solve problems across different domains. Overall, the pipeline makes it practical for researchers to go from just one reference image to a comprehensive dataset in a diminutive span of time.

Authors: Tarun Narayanan (SpaceML), Ajay Krishnan (SpaceML), Anirudh Koul (Pinterest, SpaceML, FDL) and Siddha Ganju (NVIDIA, SpaceML, FDL)
AAAI FSS 2022 De-risking Carbon Capture and Sequestration with Explainable CO2 Leakage Detection in Time-lapse Seismic Monitoring Images
Abstract and authors: (click to expand)

Abstract: With the growing global deployment of carbon capture and sequestration technology to combat climate change, monitoring and detection of potential CO2 leakage through existing or storage induced faults are critical to the safe and long-term viability of the technology. Recent work on time-lapse seismic monitoring of CO2 storage has shown promising results in its ability to monitor the growth of the CO2 plume from surface recorded seismic data. However, due to the low sensitivity of seismic imaging to CO2 concentration, additional developments are required to efficiently interpret the seismic images for leakage. In this work, we introduce a binary classification of time-lapse seismic images to delineate CO2 plumes (leakage) using state-of-the-art deep learning models. Additionally, we localize the leakage region of CO2 plumes by leveraging Class Activation Mapping (CAM) methods.

Authors: Huseyin Tuna Erdinc (Georgia Institute of Technology), Abhinav Prakash Gahlot (Georgia Institute of Technology), Ziyi Yin (Georgia Institute of Technology), Mathias Louboutin (Georgia Institute of Technology) and Felix J. Herrmann (Georgia Institute of Technology)
AAAI FSS 2022 Intermediate and Future Frame Prediction of Geostationary Satellite Imagery With Warp and Refine Network
Abstract and authors: (click to expand)

Abstract: Geostationary satellite imagery has applications in climate and weather forecasting, planning natural energy resources, and predicting extreme weather events. For precise and accurate prediction, higher spatial and temporal resolution of geostationary satellite imagery is important. Although recent geostationary satellite resolution has improved, the long-term analysis of climate applications is limited to using multiple satellites from the past to the present due to the different resolutions. To solve this problem, we proposed warp and refine network (WR-Net). WR-Net is divided into an optical flow warp component and a warp image refinement component.We used the TV-L1 algorithm instead of deep learning-based approaches to extract the optical flow warp component. The deep-learning-based model is trained on the human-centric view of the RGB channel and does not work on geostationary satellites, which is gray-scale one-channel imagery.The refinement network refines the warped image through a multi-temporal fusion layer. We evaluated WR-Net by interpolation of temporal resolution at 4 min intervals to 2 min intervals in large-scale GK2A geostationary meteorological satellite imagery. Furthermore, we applied WR-Net to the future frame prediction task and showed that the explicit use of optical flow can help future frame prediction.

Authors: Minseok Seo (SI Analytics), Yeji Choi (SI Analytics), Hyungon Ryu (NVIDIA), Heesun Park (National Institute of Meteorological Science), Hyungkun Bae (SI Analytics), Hyesook Lee (National Institute of Meteorological Science) and Wanseok Seo (NVIDIA)
NeurIPS 2021 Flood Segmentation on Sentinel-1 SAR Imagery with Semi-Supervised Learning (Papers Track)
Abstract and authors: (click to expand)

Abstract: Floods wreak havoc throughout the world, causing billions of dollars in damages, and uprooting communities, ecosystems and economies. The NASA Impact Emerging Techniques in Computational Intelligence (ETCI) competition on Flood Detection tasked participants with predicting flooded pixels after training with synthetic aperture radar (SAR) images in a supervised setting. We propose a semi-supervised learning pseudo-labeling scheme that derives confidence estimates from U-Net ensembles, thereby progressively improving accuracy. Concretely, we use a cyclical approach involving multiple stages (1) training an ensemble model of multiple U-Net architectures with the provided high confidence hand-labeled data and, generated pseudo labels or low confidence labels on the entire unlabeled test dataset, and then, (2) filter out quality generated labels and, (3) combine the generated labels with the previously available high confidence hand-labeled dataset. This assimilated dataset is used for the next round of training ensemble models. This cyclical process is repeated until the performance improvement plateaus. Additionally, we post process our results with Conditional Random Fields. Our approach sets a high score, and a new state-of-the-art on the Sentinel-1 dataset for the ETCI competition with 0.7654 IoU, an impressive improvement over the 0.60 IOU baseline. Our method, which we release with all the code including trained models, can also be used as an open science benchmark for the Sentinel-1 released dataset.

Authors: Siddha Ganju (Nvidia Corporation); Sayak Paul (Carted)
NeurIPS 2021 Short-term Solar Irradiance Prediction from Sky Images (Papers Track)
Abstract and authors: (click to expand)

Abstract: Solar irradiance forecasting is essential for the integration of the solar power into the power grid system while maintaining its stability. This paper focuses on short-term solar irradiance forecasting (upto 4-hour ahead-of-time prediction) from a past sky image sequence. Most existing work aims for the prediction of the most likely future of the solar irradiance. While it is likely deterministic for intra-hourly prediction, the future solar irradiance is naturally diverse over a relatively long-term horizon (>1h). We therefore introduce approaches to deterministic and stochastic predictions to capture the most likely as well as the diverse future of the solar irradiance. To enable the autoregressive prediction capability of the model, we proposed deep neural networks to predict the future sky images in a deterministic as well as stochastic manner. We evaluate our approaches on benchmark datasets and demonstrate that our approaches achieve superior performance.

Authors: Hoang Chuong Nguyen (Australia National University); Miaomiao Liu (The Australian National University)
NeurIPS 2021 Memory to Map: Improving Radar Flood Maps With Temporal Context and Semantic Segmentation (Papers Track)
Abstract and authors: (click to expand)

Abstract: Global flood risk has increased due to worsening extreme weather events and human migration into growing flood-prone areas. Accurate, high-resolution, and near-real time flood maps can address flood risk by reducing financial loss and damage. We propose Model to Map, a novel machine learning approach that utilizes bi-temporal context to improve flood water segmentation performance for Sentinel-1 imagery. We show that the inclusion of unflooded context for the area, or "memory," allows the model to tap into a "prior state" of pre-flood conditions, increasing performance in geographic regions in which single-image radar-based flood mapping methods typically underperform (e.g. deserts). We focus on accuracy across different biomes to ensure global performance. Our experiments and novel data processing technique show that the confluence of pre-flood and permanent water context provides a 21% increase in mIoU over the baseline overall, and over 87% increase in deserts.

Authors: Veda Sunkara (Cloud to Street); Nicholas Leach (Cloud to Street); Siddha Ganju (Nvidia)
NeurIPS 2021 Hurricane Forecasting: A Novel Multimodal Machine Learning Framework (Papers Track)
Abstract and authors: (click to expand)

Abstract: This paper describes a machine learning (ML) framework for tropical cyclone intensity and track forecasting, combining multiple distinct ML techniques and utilizing diverse data sources. Our framework, which we refer to as Hurricast (HURR), is built upon the combination of distinct data processing techniques using gradient-boosted trees and novel encoder-decoder architectures, including CNN, GRU and Transformers components. We propose a deep-learning feature extractor methodology to mix spatial-temporal data with statistical data efficiently. Our multimodal framework unleashes the potential of making forecasts based on a wide range of data sources, including historical storm data, and visual data such as reanalysis atmospheric images. We evaluate our models with current operational forecasts in North Atlantic (NA) and Eastern Pacific (EP) basins on 2016-2019 for 24-hour lead time, and show our models consistently outperform statistical-dynamical models and compete with the best dynamical models. Furthermore, the inclusion of Hurricast into an operational forecast consensus model leads to a significant improvement of 5% - 15% over NHC's official forecast, thus highlighting the complementary properties with existing approaches.

Authors: Léonard Boussioux (MIT, CentraleSupélec); Cynthia Zeng (MIT); Dimitris Bertsimas (MIT); Théo J Guenais (Harvard University)
NeurIPS 2021 Global ocean wind speed estimation with CyGNSSnet (Papers Track)
Abstract and authors: (click to expand)

Abstract: The CyGNSS (Cyclone Global Navigation Satellite System) satellite system measures GNSS signals reflected off the Earth's surface. A global ocean wind speed dataset is derived, which fills a gap in Earth observation data, will improve cyclone forecasting, and could be used to mitigate effects of climate change. We propose CyGNSSnet, a deep learning model for predicting wind speed from CyGNSS observables, and evaluate its potential for operational use. With CyGNSSnet, performance improves by 29\% over the current operational model. We further introduce a hierarchical model, that combines an extreme value classifier and a specialized CyGNSSnet and slightly improves predictions for high winds.

Authors: Caroline Arnold (German Climate Computing Center); Milad Asgarimehr (German Research Centre for Geosciences)
NeurIPS 2021 Rotation Equivariant Deforestation Segmentation and Driver Classification (Papers Track)
Abstract and authors: (click to expand)

Abstract: Deforestation has become a significant contributing factor to climate change and, due to this, both classifying the drivers and predicting segmentation maps of deforestation has attracted significant interest. In this work, we develop a rotation equivariant convolutional neural network model to predict the drivers and generate segmentation maps of deforestation events from Landsat 8 satellite images. This outperforms previous methods in classifying the drivers and predicting the segmentation map of deforestation, offering a 9% improvement in classification accuracy and a 7% improvement in segmentation map accuracy. In addition, this method predicts stable segmentation maps under rotation of the input image, which ensures that predicted regions of deforestation are not dependent upon the rotational orientation of the satellite.

Authors: Joshua Mitton (University of Glasgow); Roderick Murray-Smith (University of Glasgow)
NeurIPS 2021 WiSoSuper: Benchmarking Super-Resolution Methods on Wind and Solar Data (Papers Track)
Abstract and authors: (click to expand)

Abstract: The transition to green energy grids depends on detailed wind and solar forecasts to optimize the siting and scheduling of renewable energy generation. Operational forecasts from numerical weather prediction models, however, only have a spatial resolution of 10 to 20-km, which leads to sub-optimal usage and development of renewable energy farms. Weather scientists have been developing super-resolution methods to increase the resolution, but often rely on simple interpolation techniques or computationally expensive differential equation-based models. Recently, machine learning-based models, specifically the physics-informed resolution-enhancing generative adversarial network (PhIREGAN), have outperformed traditional downscaling methods. We provide a thorough and extensible benchmark of leading deep learning-based super-resolution techniques, including the enhanced super-resolution generative adversarial network (ESRGAN) and an enhanced deep super-resolution (EDSR) network, on wind and solar data. We accompany the benchmark with a novel public, processed, and machine learning-ready dataset for benchmarking super-resolution methods on wind and solar data.

Authors: Rupa Kurinchi-Vendhan (Caltech); Björn Lütjens (MIT); Ritwik Gupta (University of California, Berkeley); Lucien D Werner (California Institute of Technology); Dava Newman (MIT); Steven Low (California Institute of Technology)
NeurIPS 2021 MS-nowcasting: Operational Precipitation Nowcasting with Convolutional LSTMs at Microsoft Weather (Papers Track)
Abstract and authors: (click to expand)

Abstract: We present the encoder-forecaster convolutional long short-term memory (LSTM) deep-learning model that powers Microsoft Weather's operational precipitation nowcasting product. This model takes as input a sequence of weather radar mosaics and deterministically predicts future radar reflectivity at lead times up to 6 hours. By stacking a large input receptive field along the feature dimension and conditioning the model's forecaster with predictions from the physics-based High Resolution Rapid Refresh (HRRR) model, we are able to outperform optical flow and HRRR baselines by 20-25% on multiple metrics averaged over all lead times.

Authors: Sylwester Klocek (Microsoft Corporation); Haiyu Dong (Microsoft); Matthew Dixon (Microsoft Corporation); Panashe Kanengoni (Microsoft Corporation); Najeeb Kazmi (Microsoft); Pete Luferenko (Microsoft Corporation); Zhongjian Lv (Microsoft Corporation); Shikhar Sharma (); Jonathan Weyn (Microsoft); Siqi Xiang (Microsoft Corporation)
NeurIPS 2021 SunCast: Solar Irradiance Nowcasting from Geosynchronous Satellite Data (Papers Track)
Abstract and authors: (click to expand)

Abstract: When cloud layers cover photovoltaic (PV) panels, the amount of power the panels produce fluctuates rapidly. Therefore, to maintain enough energy on a power grid to match demand, utilities companies rely on reserve power sources that typically come from fossil fuels and therefore pollute the environment. Accurate short-term PV power prediction enables operators to maximize the amount of power obtained from PV panels and safely reduce the reserve energy needed from fossil fuel sources. While several studies have developed machine learning models to predict solar irradiance at specific PV generation facilities, little work has been done to model short-term solar irradiance on a global scale. Furthermore, models that have been developed are proprietary and have architectures that are not publicly available or rely on computationally demanding Numerical Weather Prediction (NWP) models. Here, we propose a Convolutional Long Short-Term Memory Network model that treats solar nowcasting as a next frame prediction problem, is more efficient than NWP models and has a straightforward, reproducible architecture. Our models can predict solar irradiance for entire North America for up to 3 hours in under 60 seconds on a single machine without a GPU and has a RMSE of 120 W/m^2 when evaluated on 2 months of data.

Authors: Dhileeban Kumaresan (UC Berkeley); Richard Wang (UC Berkeley); Ernesto A Martinez (UC Berkeley); Richard Cziva (UC Berkeley); Alberto Todeschini (UC Berkeley); Colorado J Reed (University of California, Berkeley); Puya Vahabi (UC Berkeley)
NeurIPS 2021 Synthetic Imagery Aided Geographic Domain Adaptation for Rare Energy Infrastructure Detection in Remotely Sensed Imagery (Papers Track)
Abstract and authors: (click to expand)

Abstract: Object detection in remotely sensed data is frequently stymied by applications in geographies that are different from that of the training data. When objects are rare, the problem is exacerbated further. This is true of assessments of energy infrastructure such as generation, transmission, and end-use consumption; key to electrification planning as well as for effective assessment of natural disaster impacts which are varying in frequency and intensity due to climate change. We propose an approach to domain adaptation that requires only unlabeled samples from the target domain and generates synthetic data to augment training data for targeted domain adaptation. This approach is shown to work consistently across four geographically diverse domains, improving object detection average precision by 15.5\% on average for small sample sizes.

Authors: Wei Hu (Duke University); Tyler Feldman (Duke University); Eddy Lin (Duke University); Jose Luis Moscoso (Duke); Yanchen J Ou (Duke University); Natalie Tarn (Duke University); Baoyan Ye (Duke University); Wendy Zhang (Duke University); Jordan Malof (Duke University); Kyle Bradbury (Duke University)
NeurIPS 2021 Being the Fire: A CNN-Based Reinforcement Learning Method to Learn How Fires Behave Beyond the Limits of Physics-Based Empirical Models (Papers Track)
Abstract and authors: (click to expand)

Abstract: Wildland fires pose an increasing threat in light of anthropogenic climate change. Fire-spread models play an underpinning role in many areas of research across this domain, from emergency evacuation to insurance analysis. We study paths towards advancing such models through deep reinforcement learning. Aggregating 21 fire perimeters from the Western United States in 2017, we construct 11-layer raster images representing the state of the fire area. A convolution neural network based agent is trained offline on one million sub-images to create a generalizable baseline for predicting the best action - burn or not burn - given the then-current state on a particular fire edge. A series of online, TD(0) Monte Carlo Q-Learning based improvements are made with final evaluation conducted on a subset of holdout fire perimeters. We examine the performance of the learned agent/model against the FARSITE fire-spread model. We also make available a novel data set and propose more informative evaluation metrics for future progress.

Authors: William L Ross (Stanford)
NeurIPS 2021 A Transfer Learning-Based Surrogate Model for Geological Carbon Storage with Multi-Fidelity Training Data (Papers Track)
Abstract and authors: (click to expand)

Abstract: Geologic carbon storage (GCS) entails injecting large volumes of carbon dioxide (CO2) in deep geologic formations to prevent its release to the atmosphere. Reservoir simulation is widely used in GCS applications to predict subsurface pressure and CO2 saturation. High fidelity numerical models are prohibitively expensive for data assimilation and uncertainty quantification, which require a large number of simulation runs. Deep learning-based surrogate models have shown a great promise to alleviate the high computational cost. However, the training cost is high as thousands of high-fidelity simulations are often necessary for generating the training data. In this work, we explore the use of a transfer learning approach to reduce the training cost. Compared with the surrogate model trained with high-fidelity simulations, our new transfer learning-based model shows comparable accuracy but reduces the training cost by 80%.

Authors: Su Jiang (Stanford University)
NeurIPS 2021 National Cropland Classification with Agriculture Census Information and EO Datasets (Papers Track)
Abstract and authors: (click to expand)

Abstract: National cropland classification is critical to monitor food security, comprehend environmental circumstances and climate change, and participate in agricultural policy development. The increasing earth observation datasets, especially the free available Sentinel and Landsat, open unprecedented large-scale mapping opportunities. However, most applied machine learning techniques have relied on substantial training datasets, which are not always available and may be expensive to create or collect. Focusing on Japan, this work indicates what kinds of information can be extracted from agriculture census information then used for mapping different crop types. Different classification approaches of pixel-based and parcel-based are compared. Then, the efficient method is used to generate Japan's first national cropland classification with Sentinel-1 C-band and Landsat-8 time series. For 2015, the overall accuracies for the prefectures range between 71\% and 94\%. This national cropland classification map, which particularly succeeds in extracting high-precision rice products for the whole of Japan and other classes for different prefectures, can be treated as the base map of Japan for future studies related to agriculture, environment, and climate change.

Authors: Junshi Xia (RIKEN); Naoto Yokoya (The University of Tokyo); Bruno Adriano (RIKEN Center for Advanced Intelligence Project (AIP))
NeurIPS 2021 FIgLib & SmokeyNet: Dataset and Deep Learning Model for Real-Time Wildland Fire Smoke Detection (Papers Track)
Abstract and authors: (click to expand)

Abstract: The size and frequency of wildland fires in the western United States have dramatically increased in recent years. On high fire-risk days, a small fire ignition can rapidly grow and get out of control. Early detection of fire ignitions from initial smoke can assist response to such fires before they become difficult to manage. Past deep learning approaches for wildfire smoke detection have suffered from small or unreliable datasets that make it difficult to extrapolate performance to real-world scenarios. In this work, we present the Fire Ignition Library (FIgLib), a publicly-available dataset of nearly 25,000 labeled wildfire smoke images as seen from fixed-view cameras deployed in Southern California. We also introduce SmokeyNet, a novel deep learning architecture using spatio-temporal information from camera imagery for real-time wildfire smoke detection. When trained on the FIgLib dataset, SmokeyNet outperforms comparable baselines. We hope that the availability of the FIgLib dataset and the SmokeyNet architecture will inspire further research into deep learning methods for wildfire smoke detection, leading to automated notification systems to reduce the time to wildfire response.

Authors: Anshuman Dewangan (University of California, San Diego); Mai Nguyen (University of California, San Diego); Garrison Cottrell (UC San Diego)
NeurIPS 2021 Multitask Learning for Estimating Power Plant Greenhouse Gas Emissions from Satellite Imagery (Papers Track)
Abstract and authors: (click to expand)

Abstract: The burning of fossil fuels produces large amounts of carbon dioxide (CO2), a major Greenhouse Gas (GHG) and a main driver of Climate Change. Quantifying GHG emissions is crucial for accurate predictions of climate effects and to enforce emission trading schemes. The reporting of such emissions is only required in some countries, resulting in insufficient global coverage. In this work, we propose an end-to-end method to predict power generation rates for fossil fuel power plants from satellite images based on which we estimate GHG emission rates. We present a multitask deep learning approach able to simultaneously predict: (i) the pixel-area covered by plumes from a single satellite image of a power plant, (ii) the type of fired fuel, and (iii) the power generation rate. We then convert the predicted power generation rate into estimates for the rate at which CO2 is being emitted. Experimental results show that our model approach allows us to estimate the power generation rate of a power plant to within 139 MW (MAE, for a mean sample power plant capacity of 1177 MW) from a single satellite image and CO2 emission rates to within 311 t/h. This multitask learning approach improves the power generation estimation MAE by 39 % compared to a single-task network trained on the same dataset.

Authors: Joëlle Hanna (University of St. Gallen); Michael Mommert (University of St. Gallen); Linus M. Scheibenreif (University of St. Gallen); Damian Borth (University of St. Gallen)
NeurIPS 2021 Learning to identify cracks on wind turbine blade surfaces using drone-based inspection images (Papers Track)
Abstract and authors: (click to expand)

Abstract: Wind energy is expected to be one of the leading ways to achieve the goals of the Paris Agreement but it in turn heavily depends on effective management of its operations and maintenance (O&M) costs. Blade failures account for one-third of all O&M costs thus making accurate detection of blade damages, especially cracks, very important for sustained operations and cost savings. Traditionally, damage inspection has been a completely manual process thus making it subjective, error-prone, and time-consuming. Hence in this work, we bring more objectivity, scalability, and repeatability in our damage inspection process, using deep learning, to miss fewer cracks. We build a deep learning model trained on a large dataset of blade damages, collected by our drone-based inspection, to correctly detect cracks. Our model is already in production and has processed more than a million damages with a recall of 0.96. We also focus on model interpretability using class activation maps to get a peek into the model workings. The model not only performs as good as human experts but also better in certain tricky cases. Thus, in this work, we aim to increase wind energy adoption by decreasing one of its major hurdles - the O&M costs resulting from missing blade failures like cracks.

Authors: Akshay B Iyer (SkySpecs, Inc.); Linh V Nguyen (SkySpecs Inc); Shweta Khushu (SkySpecs Inc.)
NeurIPS 2021 Evaluating Pretraining Methods for Deep Learning on Geophysical Imaging Datasets (Papers Track)
Abstract and authors: (click to expand)

Abstract: Machine learning has the potential to automate the analysis of vast amounts of raw geophysical data, allowing scientists to monitor changes in key aspects of our climate such as cloud cover in real-time and at fine spatiotemporal scales. However, the lack of large labeled training datasets poses a significant barrier for effectively applying machine learning to these applications. Transfer learning, which involves first pretraining a neural network on an auxiliary “source” dataset and then finetuning on the “target” dataset, has been shown to improve accuracy for machine learning models trained on small datasets. Across prior work on machine learning for geophysical imaging, different choices are made about what data to pretrain on, and the impact of these choices on model performance is unclear. To address this, we systematically explore various settings of transfer learning for cloud classification, cloud segmentation, and aurora classification. We pretrain on different source datasets, including the large ImageNet dataset as well as smaller geophysical datasets that are more similar to the target datasets. We also experiment with multiple transfer learning steps where we pretrain on more than one source dataset. Despite the smaller source datasets’ similarity to the target datasets, we find that pretraining on the large, general-purpose ImageNet dataset yields significantly better results across all of our experiments. Transfer learning is especially effective for smaller target datasets, and in these cases, using multiple source datasets can give a marginal added benefit.

Authors: James Chen (Kirby School)
NeurIPS 2021 An Automated System for Detecting Visual Damages of Wind Turbine Blades (Papers Track)
Abstract and authors: (click to expand)

Abstract: Wind energy’s ability to compete with fossil fuels on a market level depends on lowering wind’s high operational costs. Since damages on wind turbine blades are the leading cause for these operational problems, identifying blade damages is critical. However, recent works in visual identification of blade damages are still experimental and focus on optimizing the traditional machine learning metrics such as IoU. In this paper, we argue that pushing models to production long before achieving the "optimal" model performance can still generate real value for this use case. We discuss the performance of Blue, our damage’s suggestion model in production and how this system works in coordination with humans as part of a commercialized product and how it can contribute towards lowering wind energy’s operational costs.

Authors: Linh V Nguyen (SkySpecs Inc); Akshay B Iyer (SkySpecs, Inc.); Shweta Khushu (SkySpecs Inc.)
NeurIPS 2021 HyperionSolarNet: Solar Panel Detection from Aerial Images (Papers Track)
Abstract and authors: (click to expand)

Abstract: With the effects of global climate change impacting the world, collective efforts are needed to reduce greenhouse gas emissions. The energy sector is the single largest contributor to climate change and many efforts are focused on reducing dependence on carbon-emitting power plants and moving to renewable energy sources, such as solar power. A comprehensive database of the location of solar panels is important to assist analysts and policymakers in defining strategies for further expansion of solar energy. In this paper we focus on creating a world map of solar panels. We identify locations and total surface area of solar panels within a given geographic area. We use deep learning methods for automated detection of solar panel locations and their surface area using aerial imagery. The framework, which consists of a two-branch model using an image classifier in tandem with a semantic segmentation model, is trained on our created dataset of satellite images. Our work provides an efficient and scalable method for detecting solar panels, achieving an accuracy of 0.96 for classification and an IoU score of 0.82 for segmentation performance.

Authors: Poonam Parhar (UCBerkeley); Ryan Sawasaki (UCBerkeley); Alberto Todeschini (UC Berkeley); Colorado Reed (UC Berkeley); Hossein Vahabi (University California Berkeley); Nathan Nusaputra (UC Berkeley); Felipe Vergara (UC Berkeley)
NeurIPS 2021 NoFADE: Analyzing Diminishing Returns on CO2 Investment (Papers Track)
Abstract and authors: (click to expand)

Abstract: Climate change continues to be a pressing issue that currently affects society at-large. It is important that we as a society, including the Computer Vision (CV) community take steps to limit our impact on the environment. In this paper, we (a) analyze the effect of diminishing returns on CV methods, and (b) propose a \textit{``NoFADE''}: a novel entropy-based metric to quantify model--dataset--complexity relationships. We show that some CV tasks are reaching saturation, while others are almost fully saturated. In this light, NoFADE allows the CV community to compare models and datasets on a similar basis, establishing an agnostic platform.

Authors: Andre Fu (University of Toronto); Justin B Tran (University of Toronto); Andy Xie (University of Toronto); Jonathan T Spraggett (University of Toronto); Elisa Ding (University of Toronto); Chang-Won Lee (University of Toronto); Kanav Singla (University of Toronto); Mahdi S. Hosseini (University of New Brunswick); Konstantinos N Plataniotis (UofT)
NeurIPS 2021 DEM Super-Resolution with EfficientNetV2 (Papers Track)
Abstract and authors: (click to expand)

Abstract: Efficient climate change monitoring and modeling rely on high-quality geospatial and environmental datasets. Due to limitations in technical capabilities or resources, the acquisition of high-quality data for many environmental disciplines is costly. Digital Elevation Model (DEM) datasets are such examples whereas their low-resolution versions are widely available, high-resolution ones are scarce. In an effort to rectify this problem, we propose and assess an EfficientNetV2 based model. The proposed model increases the spatial resolution of DEMs up to 16 times without additional information.

Authors: Bekir Z Demiray (University of Iowa); Muhammed A Sit (The University of Iowa); Ibrahim Demir (The University of Iowa)
NeurIPS 2021 Resolving Super Fine-Resolution SIF via Coarsely-Supervised U-Net Regression (Papers Track)
Abstract and authors: (click to expand)

Abstract: Climate change presents challenges to crop productivity, such as increasing the likelihood of heat stress and drought. Solar-Induced Chlorophyll Fluorescence (SIF) is a powerful way to monitor how crop productivity and photosynthesis are affected by changing climatic conditions. However, satellite SIF observations are only available at a coarse spatial resolution (e.g. 3-5km) in most places, making it difficult to determine how individual crop types or farms are doing. This poses a challenging coarsely-supervised regression task; at training time, we only have access to SIF labels at a coarse resolution (3 km), yet we want to predict SIF at a very fine spatial resolution (30 meters), a 100x increase. We do have some fine-resolution input features (such as Landsat reflectance) that are correlated with SIF, but the nature of the correlation is unknown. To address this, we propose Coarsely-Supervised Regression U-Net (CSR-U-Net), a novel approach to train a U-Net for this coarse supervision setting. CSR-U-Net takes in a fine-resolution input image, and outputs a SIF prediction for each pixel; the average of the pixel predictions is trained to equal the true coarse-resolution SIF for the entire image. Even though this is a very weak form of supervision, CSR-U-Net can still learn to predict accurately, due to its inherent localization abilities, plus additional enhancements that facilitate the incorporation of scientific prior knowledge. CSR-U-Net can resolve fine-grained variations in SIF more accurately than existing averaging-based approaches, which ignore fine-resolution spatial variation during training. CSR-U-Net could also be useful for a wide range of "downscaling'" problems in climate science, such as increasing the resolution of global climate models.

Authors: Joshua Fan (Cornell University); Di Chen (Cornell University); Jiaming Wen (Cornell University); Ying Sun (Cornell University); Carla P Gomes (Cornell University)
NeurIPS 2021 PreDisM: Pre-Disaster Modelling With CNN Ensembles for At-Risk Communities (Papers Track)
Abstract and authors: (click to expand)

Abstract: The machine learning community has recently had increased interest in the climate and disaster damage domain due to a marked increased occurrences of natural hazards (e.g., hurricanes, forest fires, floods, earthquakes). However, not enough attention has been devoted to mitigating probable destruction from impending natural hazards. We explore this crucial space by predicting building-level damages on a before-the-fact basis that would allow state actors and non-governmental organizations to be best equipped with resource distribution to minimize or preempt losses. We introduce PreDisM that employs an ensemble of ResNets and fully connected layers over decision trees to capture image-level and meta-level information to accurately estimate weakness of man-made structures to disaster-occurrences. Our model performs well and is responsive to tuning across types of disasters and highlights the space of preemptive hazard damage modelling.

Authors: Vishal Anand (Columbia University); Yuki Miura (Columbia University)
NeurIPS 2021 A hybrid convolutional neural network/active contour approach to segmenting dead trees in aerial imagery (Papers Track)
Abstract and authors: (click to expand)

Abstract: The stability and ability of an ecosystem to withstand climate change is directly linked to its biodiversity. Dead trees are a key indicator of overall forest health, housing one-third of forest ecosystem biodiversity, and constitute 8% of the global carbon stocks. They are decomposed by several natural factors, e.g. climate, insects and fungi. Accurate detection and modeling of dead wood mass is paramount to understanding forest ecology, the carbon cycle and decomposers. We present a novel method to construct precise shape contours of dead trees from aerial photographs by combining established convolutional neural networks with a novel active contour model in an energy minimization framework. Our approach yields superior performance accuracy over state-of-the-art in terms of precision, recall, and intersection over union of detected dead trees. This improved performance is essential to meet emerging challenges caused by climate change (and other man-made perturbations to the systems), particularly to monitor and estimate carbon stock decay rates, monitor forest health and biodiversity, and the overall effects of dead wood on and from climate change.

Authors: Jacquelyn Shelton (Hong Kong Polytechnic University); Przemyslaw Polewski (TomTom Location Technology Germany GmbH); Wei Yao (The Hong Kong Polytechnic University); Marco Heurich (Bavarian Forest National Park)
NeurIPS 2021 Reducing the Barriers of Acquiring Ground-truth from Biodiversity Rich Audio Datasets Using Intelligent Sampling Techniques (Papers Track)
Abstract and authors: (click to expand)

Abstract: The potential of passive acoustic monitoring (PAM) as a method to reveal the consequences of climate change on the biodiversity that make up natural soundscapes can be undermined by the discrepancy between the low barrier of entry to acquire large field audio datasets and the higher barrier of acquiring reliable species level training, validation, and test subsets from the field audio. These subsets from a deployment are often required to verify any machine learning models used to assist researchers in understanding the local biodiversity. Especially as many models convey promising results from various sources that may not translate to the collected field audio. Labeling such datasets is a resource intensive process due to the lack of experts capable of identifying bioacoustics at a species level as well as the overwhelming size of many PAM audiosets. To address this challenge, we have tested different sampling techniques on an audio dataset collected over a two-week long August audio array deployment on the Scripps Coastal Reserve (SCR) Biodiversity Trail in La Jolla, California. These sampling techniques involve creating four subsets using stratified random sampling, limiting samples to the daily bird vocalization peaks, and using a hybrid convolutional neural network (CNN) and recurrent neural network (RNN) trained for bird presence/absence audio classification. We found that a stratified random sample baseline only achieved a bird presence rate of 44% in contrast with a sample that randomly selected clips with high hybrid CNN-RNN predictions that were collected during bird activity peaks at dawn and dusk yielding a bird presence rate of 95%. The significantly higher bird presence rate demonstrates how intelligent, machine learning-assisted selection of audio data can significantly reduce the amount of time that domain experts listen to audio without vocalizations of interest while building a ground truth for machine learning models.

Authors: Jacob G Ayers (UC San Diego); Sean Perry (UC San Diego); Vaibhav Tiwari (UC San Diego); Mugen Blue (Cal Poly San Luis Obispo); Nishant Balaji (UC San Diego); Curt Schurgers (UC San Diego); Ryan Kastner (University of California San Diego); Mathias Tobler (San Diego Zoo Wildlife Alliance); Ian Ingram (San Diego Zoo Wildlife Alliance)
NeurIPS 2021 Two-phase training mitigates class imbalance for camera trap image classification with CNNs (Papers Track)
Abstract and authors: (click to expand)

Abstract: By leveraging deep learning to automatically classify camera trap images, ecologists can monitor biodiversity conservation efforts and the effects of climate change on ecosystems more efficiently. Due to the imbalanced class-distribution of camera trap datasets, current models are biased towards the majority classes. As a result, they obtain good performance for a few majority classes but poor performance for many minority classes. We used two-phase training to increase the performance for these minority classes. We trained, next to a baseline model, four models that implemented a different versions of two-phase training on a subset of the highly imbalanced Snapshot Serengeti dataset. Our results suggest that two-phase training can improve performance for many minority classes, with limited loss in performance for the other classes. We find that two-phase training based on majority undersampling increases class-specific F1-scores up to 3.0%. We also find that two-phase training outperforms using only oversampling or undersampling by 6.1% in F1-score on average. Finally, we find that a combination of over- and undersampling leads to a better performance than using them individually.

Authors: Farjad Malik (KU Leuven); Simon Wouters (KU Leuven); Ruben Cartuyvels (KULeuven); Erfan Ghadery (KU Leuven); Sien Moens (KU Leuven)
NeurIPS 2021 Leveraging Machine Learning to Predict the Autoconversion Rates from Satellite Data (Papers Track)
Abstract and authors: (click to expand)

Abstract: One way of reducing the uncertainty involved in determining the radiative forcing of climate change is by understanding the interaction between aerosols, clouds, and precipitation processes. This can be studied using high-resolution simulations such as the ICOsahedral Non-hydrostatic Large-Eddy Model (ICON-LEM). However, due to the extremely high computational cost required, this simulation-based approach can only be run for a limited amount of time within a limited area. To address this, we developed new models using emerging machine learning approaches that leverage a plethora of satellite observations providing long-term global spatial coverage. In particular, our machine learning models are capable of capturing the key process of precipitation formation which greatly control cloud lifetime, namely autoconversion rates -- the term used to describe the collision and coalescence of cloud droplets responsible for raindrop formation. We validate the performance of our models against simulation data, showing that our models are capable of predicting the autoconversion rates fairly well.

Authors: Maria C Novitasari (University College London); Johannes Quaas (University of Leipzig); Miguel Rodrigues (University College London)
NeurIPS 2021 Towards Automatic Transformer-based Cloud Classification and Segmentation (Papers Track)
Abstract and authors: (click to expand)

Abstract: Clouds have been demonstrated to have a huge impact on the energy balance, temperature, and weather of the Earth. Classification and segmentation of clouds and coverage factors is crucial for climate modelling, meteorological studies, solar energy industry, and satellite communication. For example, clouds have a tremendous impact on short-term predictions or 'nowcasts' of solar irradiance and can be used to optimize solar power plants and effectively exploit solar energy. However even today, cloud observation requires the intervention of highly-trained professionals to document their findings, which introduces bias. To overcome these issues and contribute to climate change technology, we propose, to the best of our knowledge, the first two transformer-based models applied to cloud data tasks. We use the CCSD Cloud classification dataset and achieve 90.06% accuracy, outperforming all other methods. To demonstrate the robustness of transformers in this domain, we perform Cloud segmentation on SWIMSWG dataset and achieve 83.2% IoU, also outperforming other methods. With this, we signal a potential shift away from pure CNN networks.

Authors: Roshan Roy (Birla Institute of Technology and Science, Pilani); Ahan M R (BITS Pilani); Vaibhav Soni (MANIT Bhopal); Ashish Chittora (BITS Pilani)
NeurIPS 2021 Machine Learning in Automating Carbon Sequestration Site Assessment (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Carbon capture and sequestration are viewed as an indispensable component to achieve the Paris Agreement climate goal, i.e., keep the global warming within 2 degrees Celsius from pre-industrial levels. Once captured, most CO2 needs to be stored securely for at least decades, preferably in deep underground geological formations. It is economical to inject and store CO2 near/around a depleted gas/oil reservoir or well, where a geological trap for CO2 with good sealing properties and some minimum infrastructure exist. In this proposal, with our preliminary work, it is shown that Machine Learning tools like Optical Character Recognition and Natural Language Processing can aid in screening and selection of injection sites for CO2 storage, facilitate identification of possible CO2 leakage paths in the subsurface, and assist in locating a depleted gas/oil well suitable for CO2 injection and long-term storage. The automated process based on ML tools can also drastically decrease the decision-making cycle time in site selection and assessment phase by reducing human effort. In the longer term, we expect ML tools like Deep Neural Networks to be utilized in CO2 storage monitoring, injection optimization etc. By injecting CO2 into a trapping geological underground formation in a safe and sustainable manner, the Energy industry can contribute substantially to reducing global warming and achieving the goals of the Paris Agreement by the end of this century.

Authors: Jay Chen (Shell); Ligang Lu (Shell); Mohamed Sidahmed (Shell); Taixu Bai (Shell); Ilyana Folmar (Shell); Puneet Seth (Shell); Manoj Sarfare (Shell); Duane Mikulencak (Shell); Ihab Akil (Shell)
NeurIPS 2021 Detecting Abandoned Oil And Gas Wells Using Machine Learning And Semantic Segmentation (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Around the world, there are millions of unplugged abandoned oil and gas wells, leaking methane into the atmosphere. The locations of many of these wells, as well as their greenhouse gas emissions impacts, are unknown. Machine learning methods in computer vision and remote sensing, such as semantic segmentation, have made it possible to quickly analyze large amounts of satellite imagery to detect salient information. This project aims to automatically identify undocumented oil and gas wells in the province of Alberta, Canada to aid in documentation, estimation of emissions and maintenance of high-emitting wells.

Authors: Michelle Lin (McGill University); David Rolnick (McGill University, Mila)
NeurIPS 2021 Toward Foundation Models for Earth Monitoring: Proposal for a Climate Change Benchmark (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Recent progress in self-supervision shows that pre-training large neural networks on vast amounts of unsupervised data can lead to impressive increases in generalisation for downstream tasks. Such models, recently coined as foundation models, have been transformational to the field of natural language processing. While similar models have also been trained on large corpuses of images, they are not well suited for remote sensing data. To stimulate the development of foundation models for Earth monitoring, we propose to develop a new benchmark comprised of a variety of downstream tasks related to climate change. We believe that this can lead to substantial improvements in many existing applications and facilitate the development of new applications. This proposal is also a call for collaboration with the aim of developing a better evaluation process to mitigate potential downsides of foundation models for Earth monitoring.

Authors: Alexandre Lacoste (ServiceNow); Evan D Sherwin (Stanford University, Energy and Resources Engineering); Hannah R Kerner (University of Maryland); Hamed Alemohammad (Radiant Earth Foundation); Björn Lütjens (MIT); Jeremy A Irvin (Stanford); David Dao (ETH Zurich); Alex Chang (Service Now); Mehmet Gunturkun (Element Ai); Alexandre Drouin (ServiceNow); Pau Rodriguez (Element AI); David Vazquez (ServiceNow)
ICML 2021 Examining the nexus of environmental policy, climate physics, and maritime shipping with deep learning models and space-borne data (Papers Track)
Abstract and authors: (click to expand)

Abstract: Ship-tracks are produced by ship exhaust interacting with marine low clouds. They provide an ideal lab for constraining a critical climate forcing. However, no global survey of ship ship-tracks has been made since its discovery 55 years ago, which limits research progress. Here we present the first global map of ship-tracks produced by applying deep segmentation models to large satellite data. Our model generalizes well and is validated against independent data. Large-scale ship-track data are at the nexus of environmental policy, climate physics, and maritime shipping industry: they can be used to study aerosol-cloud interactions, the largest uncertainty source in climate forcing; to evaluate compliance and impacts of environmental policies; and to study the impact of significant socioeconomic events on maritime shipping. Based on twenty years of global data, we show cloud physics responses in ship-tracks strongly depend on the cloud regime. Inter-annual fluctuation in ship-track frequency clearly reflects international trade/economic trends. Emission policies strongly affect the pattern of shipping routes and ship-track occurrence. The combination of stricter fuel standard and the COVID-19 pandemic pushed global ship-track frequency to the lowest level in the record. More applications of our technique and data are envisioned such as detecting illicit shipping activity and checking policy compliance of individual ships.

Authors: Tianle Yuan (University of Maryland, NASA); Hua Song (NASA, SSAI); Chenxi Wang (University of Maryland, NASA); Kerry Meyer (NASA); Siobhan Light (University of Maryland); Sophia von Hippel (University of Arizona); Steven Platnick (NASA); Lazaros Oreopoulos (NASA); Robert Wood (University of Washington); Hans Mohrmann (University of Washington)
ICML 2021 A human-labeled Landsat-8 contrails dataset (Papers Track)
Abstract and authors: (click to expand)

Abstract: Contrails (condensation trails) are the ice clouds that trail behind aircraft as they fly through cold and moist regions of the atmosphere. Avoiding these regions could potentially be an inexpensive way to reduce over half of aviation's impact on global warming. Development and evaluation of these avoidance strategies greatly benefits from the ability to detect contrails on satellite imagery. Since little to no public data is available to develop such contrail detectors, we construct and release a dataset of several thousand Landsat-8 scenes with pixel-level annotations of contrails. The dataset will continue to grow, but currently contains 4289 scenes (of which 47% have at least one contrail) representing 950+ person-hours of labeling time.

Authors: Kevin McCloskey (Google); Scott Geraedts (Google); Brendan Jackman (Google); Vincent R. Meijer (Laboratory for Aviation and the Environment, Massachusetts Institute of Technology); Erica Brand (Google); Dave Fork (Google); John C. Platt (Google); Carl Elkin (Google); Christopher Van Arsdale (Google)
ICML 2021 Urban Tree Species Classification Using Aerial Imagery (Papers Track)
Abstract and authors: (click to expand)

Abstract: Urban trees help regulate temperature, reduce energy consumption, improve urban air quality, reduce wind speeds, and mitigating the urban heat island effect. Urban trees also play a key role in climate change mitigation and global warming by capturing and storing atmospheric carbon-dioxide which is the largest contributor to greenhouse gases. Automated tree detection and species classification using aerial imagery can be a powerful tool for sustainable forest and urban tree management. Hence, This study first offers a pipeline for generating labelled dataset of urban trees using Google Map's aerial images and then investigates how state of the art deep Convolutional Neural Network models such as VGG and ResNet handle the classification problem of urban tree aerial images under different parameters. Experimental results show our best model achieves an average accuracy of 60% over 6 tree species.

Authors: Emily Waters (Anglia Ruskin University); Mahdi Maktabdar Oghaz (Anglia Ruskin University); Lakshmi Babu Saheer (Anglia Ruskin University)
ICML 2021 Seasonal Sea Ice Presence Forecasting of Hudson Bay using Seq2Seq Learning (Papers Track)
Abstract and authors: (click to expand)

Abstract: Accurate and timely forecasts of sea ice conditions are crucial for safe shipping operations in the Canadian Arctic and other ice-infested waters. Given the advancement of machine-learning methods and the recent observations on the declining trend of Arctic sea ice extent over the past decades due to global warming, new machine learning approaches are deployed to provide additional sea ice forecasting products. This study is novel in comparison with previous machine learning (ML) approaches in the sea-ice forecasting domain as it provides a daily spatial map of probability of ice presence in the domain up to 90 days. The predictions are further used to predict freeze-up/breakup dates and show their capability to capture both the variability and the increasing trend of open water season in the domain over the past decades.

Authors: Nazanin Asadi (University of Waterloo); K Andrea Scott (University of Waterloo); Philippe Lamontagne (National Research Council Canada)
ICML 2021 Semantic Segmentation on Unbalanced Remote Sensing Classes for Active Fire (Papers Track)
Abstract and authors: (click to expand)

Abstract: Wildfires generate considerable research interest due to their high frequency of occurrence along with global climate change. Future wildfire detection sensors would equip an on-orbit processing module that filters the useless raw images before data transmission. To efficiently detect heat anomalies from the single large scene, we need to handle the unbalanced sample sets between small active fire pixels and large-size complex background information. In this study, we contribute to solving this problem by enhancing the target feature representation in three ways. We first preprocess training images by constraining sampling ranges and removing background patches. Then we use the object-contextual representation (OCR) module to strengthen the active fire pixel representation based on the self-attention unit. The HRNet backbone provides multi-scale pixel representation as input to the OCR module. Finally, the combined loss of weighted cross-entropy loss and Lovasz hinge loss improve the segmentation accuracy further by optimizing the IoU of the foreground class. The performance is tested on the aerial FLAME dataset, whose ratio between labeled active fire and background pixels is 5.6%. The proposed framework improves the mIoU from 83.10% (baseline U-Net) to 90.81%. Future research will expand the technique for active fire detection using satellite images.

Authors: Xikun Hu (KTH Royal Institute of Technology); Alberto Costa Nogueira Junior (IBM Research Brazil); Tian Jin (College of Electronic Science, National University of Defense Technology)
ICML 2021 Forest Terrain Identification using Semantic Segmentation on UAV Images (Papers Track)
Abstract and authors: (click to expand)

Abstract: Beavers' habitat is known to alter the terrain, providing biodiversity in the area, and recently their lifestyle is linked to climatic changes by reducing greenhouse gases levels in the region. To analyse the impact of beavers’ habitat on the region, it is, therefore, necessary to estimate the terrain alterations caused by beaver actions. Furthermore, such terrain analysis can also play an important role in domains like wildlife ecology, deforestation, land-cover estimations, and geological mapping. Deep learning models are known to provide better estimates on automatic feature identification and classification of a terrain. However, such models require significant training data. Pre-existing terrain datasets (both real and synthetic) like CityScapes, PASCAL, UAVID, etc, are mostly concentrated on urban areas and include roads, pathways, buildings, etc. Such datasets, therefore, are unsuitable for forest terrain analysis. This paper contributes, by providing a finely labelled novel dataset of forest imagery around beavers’ habitat, captured from a high-resolution camera on an aerial drone. The dataset consists of 100 such images labelled and classified based on 9 different classes. Furthermore, a baseline is established on this dataset using state-of-the-art semantic segmentation models based on performance metrics including Intersection Over Union (IoU), Overall Accuracy (OA), and F1 score.

Authors: Muhammad Umar (Anglia Ruskin University); Lakshmi Babu Saheer (Anglia Ruskin University); Javad Zarrin (Anglia Ruskin University)
ICML 2021 Commercial Vehicle Traffic Detection from Satellite Imagery with Deep Learning (Papers Track)
Abstract and authors: (click to expand)

Abstract: Road freight traffic is a major greenhouse gas emitter: commercial vehicles (CVs) contribute ∼7% to the global CO 2 emission budget, a fraction that is likely to increase in the future. The quantitative monitoring of CV traffic rates, while essential for the implementation of targeted road emission regulations, is costly and as such only available in developed regions. In this work, we investigate the feasibility of estimating hourly CV traffic rates from freely available Sentinel-2 satellite imagery. We train a modified Faster R-CNN object detection model to detect individual CVs in satellite images and feed the resulting counts into a regression model to predict hourly CV traffic rates. This architecture, when trained on ground-truth data for Switzerland, is able to estimate hourly CV traffic rates for any freeway section within 58% (MAPE) of the actual value; for freeway sections with historic information on CV traffic rates, we can predict hourly CV traffic rates up to within 4% (MAPE). We successfully apply our model to freeway sections in other coun tries and show-case its utility by quantifying the change in traffic patterns as a result of the first CoVID-19 lockdown in Switzerland. Our results show that it is possible to estimate hourly CV traffic rates from satellite images, which can guide civil engineers and policy makers, especially in developing countries, in monitoring and reducing greenhouse gas emissions from CV traffic.

Authors: Moritz Blattner (University of St. Gallen); Michael Mommert (University of St. Gallen); Damian Borth (University of St. Gallen)
ICML 2021 Estimation of Air Pollution with Remote Sensing Data: Revealing Greenhouse Gas Emissions from Space (Papers Track)
Abstract and authors: (click to expand)

Abstract: Air pollution is a major driver of climate change. Anthropogenic emissions from the burning of fossil fuels for transportation and power generation emit large amounts of problematic air pollutants, including Greenhouse Gases (GHGs). Despite the importance of limiting GHG emissions to mitigate climate change, detailed information about the spatial and temporal distribution of GHG and other air pollutants is difficult to obtain. Existing models for surface-level air pollution rely on extensive land-use datasets which are often locally restricted and temporally static. This work proposes a deep learning approach for the prediction of ambient air pollution that only relies on remote sensing data that is globally available and frequently updated. Combining optical satellite imagery with satellite-based atmospheric column density air pollution measurements enables the scaling of air pollution estimates (in this case NO2) to high spatial resolution (up to ~10m) at arbitrary locations and adds a temporal component to these estimates. The proposed model performs with high accuracy when evaluated against air quality measurements from ground stations (mean absolute error <6 microgram/m^3). Our results enable the identification and temporal monitoring of major sources of air pollution and GHGs.

Authors: Linus M. Scheibenreif (University of St. Gallen); Michael Mommert (University of St. Gallen); Damian Borth (University of St. Gallen)
ICML 2021 Attention For Damage Assessment (Papers Track)
Abstract and authors: (click to expand)

Abstract: Due to climate change the hurricanes are getting stronger and having longer impacts. To reduce the detrimental effects of these hurricanes faster and accurate assessments of damages are essential to the rescue teams. Like other computer vision techniques semantic segmentation can identify the damages and help in proper and prompt damage assessment. Current segmentation methods can be classified into attention and non-attention based methods. Existing non-attention based methods suffers from low accuracy and therefore attention based methods are becoming popular. Self-attention based methods can map the mutual relationship and dependencies among pixels of an image and thus improve semantic segmentation accuracy. In this paper, we present a self-attention semantic segmentation method on UAV imageries to assess the damages inflicted by a natural disaster. The proposed method outperforms four state-of-art segmentation methods both quantitatively and qualitatively with a mean IoU score of 84.03 %.

Authors: Tashnim Chowdhury (University of Maryland Baltimore County); Maryam Rahnemoonfar (University of Maryland Baltimore County)
ICML 2021 ForestViT: A Vision Transformer Network for Convolution-free Multi-label Image Classification in Deforestation Analysis (Papers Track)
Abstract and authors: (click to expand)

Abstract: Understanding the dynamics of deforestation as well as land uses of neighboring areas is of vital importance for the design and development of appropriate forest conservation and management policies. In this paper, we approach deforestation as a multi-label classification problem in an endeavor to capture the various relevant land uses from satellite images. To this end, we propose a multi-label vision transformer model, ForestViT, which leverages the benefits of self-attention mechanism, obviating any convolution operations involved in commonly used deep learning models utilized for deforestation detection.

Authors: Maria Kaselimi (National Technical University of Athens); Athanasios Voulodimos (University of West Attica); Ioannis Daskalopoulos (University of West Attica); Nikolaos Doulamis (National Technical University of Athens); Anastasios Doulamis (Technical University of Crete)
ICML 2021 Self-supervised Contrastive Learning for Irrigation Detection in Satellite Imagery (Papers Track)
Abstract and authors: (click to expand)

Abstract: Climate change has caused reductions in river runoffs and aquifer recharge resulting in an increasingly unsustainable crop water demand from reduced freshwater availability. Achieving food security while deploying water in a sustainable manner will continue to be a major challenge necessitating careful monitoring and tracking of agricultural water usage. Historically, monitoring water usage has been a slow and expensive manual process with many imperfections and abuses. Ma-chine learning and remote sensing developments have increased the ability to automatically monitor irrigation patterns, but existing techniques often require curated and labelled irrigation data, which are expensive and time consuming to obtain and may not exist for impactful areas such as developing countries. In this paper, we explore an end-to-end real world application of irrigation detection with uncurated and unlabeled satellite imagery. We apply state-of-the-art self-supervised deep learning techniques to optical remote sensing data, and find that we are able to detect irrigation with up to nine times better precision, 90% better recall and 40% more generalization ability than the traditional supervised learning methods.

Authors: Chitra Agastya (UC Berkeley, IBM); Sirak Ghebremusse (UC Berkeley); Ian Anderson (UC Berkeley); Colorado Reed (UC Berkeley); Hossein Vahabi (University California Berkeley); Alberto Todeschini (UC Berkeley)
ICML 2021 Refining Ice Layer Tracking through Wavelet combined Neural Networks (Papers Track)
Abstract and authors: (click to expand)

Abstract: Rise in global temperatures is resulting in polar ice caps to melt away, which can lead to drastic sea level rise and coastal floods. Accurate calculation of the ice cap reduction is necessary in order to project its climatic impact. Ice sheets are monitored through Snow Radar sensors which give noisy profiles of subsurface ice layers. The sensors take snapshots of the entire ice sheet regularly, and thus result in large datasets. In this work, we use convolutional neural networks (CNNs) for their property of feature extraction and generalizability on large datasets. We also use wavelet transforms and embed them as a layer in the architecture to help in denoising the radar images and refine ice layer detection. Our results show that incorporating wavelets in CNNs helps in detecting the position of deep subsurface ice layers, which can be used to analyse their change overtime.

Authors: Debvrat Varshney (University of Maryland Baltimore County); Masoud Yari (College of Engineering and Information Technology, University of Maryland Balitimore County); Tashnim Chowdhury (University of Maryland Baltimore County); Maryam Rahnemoonfar (University of Maryland Baltimore County)
ICML 2021 Visual Question Answering: A Deep Interactive Framework for Post-Disaster Management and Damage Assessment (Papers Track)
Abstract and authors: (click to expand)

Abstract: Each natural disaster has left a trail of destruction and damage, which must be managed very effectively to reduce the disaster's impact. Lack of proper decision making in post-disaster managerial level can increase human suffering and waste a great amount of money. Our objective is to incorporate a deep interactive approach in the decision-making system especially in a rescue mission after any natural disaster for the systematic distribution of the limited resources and accelerating the recovery process. We believe that Visual Question Answering (VQA) is the finest way to address this issue. In visual question answering, a query-based answer regarding the situation of the affected area can add value to the decision-making system. Our main purpose of this study is to develop a Visual Question Answering model for post-disaster damage assessment purposes. With this aim, we collect the images by UAV (Unmanned Aerial Vehicle) after Hurricane Harvey and develop a dataset that includes the questions that are very important in the decision support system after a natural disaster. In addition, We propose a supervised attention-based approach in the modeling segment. We compare our approach with the two other baseline attention-based VQA algorithms namely Multimodal Factorized Bilinear (MFB) and Stacked Attention Network (SAN). Our approach outperforms in providing answers for several types of queries including simple counting, complex counting compares to the baseline models.

Authors: Argho Sarkar (University of Maryland, Baltimore County); Maryam Rahnemoonfar (University of Maryland Baltimore County)
ICML 2021 Sky Image Prediction Using Generative Adversarial Networks for Solar Forecasting (Papers Track)
Abstract and authors: (click to expand)

Abstract: Large-scale integration of solar photovoltaics (PV) is challenged by high variability in its power output, mainly due to local and short-term cloud events. To achieve accurate solar forecasting, it is paramount to accurately predict the movement of clouds. Here, we use generative adversarial networks (GANs) to predict future sky images based on past sky image sequences and show that our trained model can generate realistic future sky images and capture the dynamics of clouds in the context frames. The generated images are then evaluated for a downstream solar forecasting task; results show promising performance.

Authors: Yuhao Nie (Stanford University); Andea Scott (Stanford University); Eric Zelikman (Stanford University); Adam Brandt (Stanford University)
ICML 2021 FIRE-ML: A Remotely-sensed Daily Wildfire Forecasting Dataset for the Contiguous United States (Papers Track)
Abstract and authors: (click to expand)

Abstract: Wildfires are natural phenomena that can have devastating effects on ecosystems, urban developments, and the environment. Improving the scientific understanding of these events and the ability to forecast how they will evolve in the short- and long-term are ongoing multi-decadal challenges. We present a large-scale dataset, well-suited to machine learning, that aggregates and aligns multiple remotely-sensed and forecasted data products to provide a holistic set of features for forecasting wildfires on daily timescales. This dataset includes 4.2 million unique active fire detections, covers the majority of the contiguous United States from 2012 to 2020, and includes active fire detections, land cover, topography, and meteorology.

Authors: Casey A Graff (UC Irvine)
ICML 2021 Short-term Hourly Streamflow Prediction with Graph Convolutional GRU Networks (Papers Track)
Abstract and authors: (click to expand)

Abstract: The frequency and impact of floods are expected to increase due to climate change. It is crucial to predict streamflow, consequently flooding, in order to prepare and mitigate its consequences in terms of property damage and fatalities. This paper presents a Graph Convolutional GRUs based model to predict the next 36 hours of streamflow for a sensor location using the upstream river network. As shown in experiment results, the model presented in this study provides better performance than the persistence baseline and a Convolutional Bidirectional GRU network for the selected study area in short-term streamflow prediction.

Authors: Muhammed A Sit (The University of Iowa); Bekir Demiray (The University of Iowa); Ibrahim Demir (The University of Iowa)
ICML 2021 Wildfire Smoke Plume Segmentation Using Geostationary Satellite Imagery (Papers Track)
Abstract and authors: (click to expand)

Abstract: Wildfires have increased in frequency and severity over the past two decades, especially in the Western United States. Beyond physical infrastructure damage caused by these wildfire events, researchers have increasingly identified harmful impacts of particulate matter generated by wildfire smoke on respiratory, cardiovascular, and cognitive health. This inference is difficult due to the spatial and temporal uncertainty regarding how much particulate matter is specifically attributable to wildfire smoke. One factor contributing to this challenge is the reliance on manually drawn smoke plume annotations, which are often noisy representations limited to the United States. This work uses deep convolutional neural networks to segment smoke plumes from geostationary satellite imagery. We compare the performance of predicted plume segmentations versus the noisy annotations using causal inference methods to estimate the amount of variation each explains in Environmental Protection Agency (EPA) measured surface level particulate matter <2.5μm in diameter (PM2.5).

Authors: Jeffrey L Wen (Stanford University); Marshall Burke (Stanford University)
ICML 2021 Deep learning applied to sea surface semantic segmentation: Filtering sunglint from aerial imagery (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Water waves are an ubiquitous feature of the oceans, which serve as a pathway for interactions with the atmosphere. Wave breaking in particular is crucial in developing better understanding of the exchange of momentum, heat, and gas fluxes between the ocean and the atmosphere. Characterizing the properties of wave breaking using orbital or suborbital imagery of the surface of the ocean can be challenging, due to contamination from sunglint, a persistent feature in certain lighting conditions. Here we propose a supervised learning approach to accurately detect whitecaps from airborne imagery obtained under a broad range of lighting conditions. Finally, we discuss potential applications for improving ocean and climate models.

Authors: Teodor Vrecica (UCSD); Quentin Paletta (University of Cambridge); Luc Lenain (UCSD)
ICML 2021 Long-term Burned Area Reconstruction through Deep Learning (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Wildfire impact studies are significantly hampered by the absence of a global long-term burned area dataset. This prevents conclusive statements on the role of anthropogenic activity on wildfire impacts over the last century. Here, we propose a workflow to construct a 1901-2014 reanalysis of monthly global burned area at a 0.5° by 0.5° scale. A neural network will be trained with weather-related, vegetational, societal and economic input parameters, and burned area as output label for the 1982-2014 time period. This model can then be applied to the whole 1901-2014 time period to create a data-driven, long-term burned area reanalysis. This reconstruction will allow to investigate the long-term effect of anthropogenic activity on wildfire impacts, will be used as basis for detection and attribution studies and could help to reduce the uncertainties in future predictions.

Authors: Seppe Lampe (Vrije Universiteit Brussel); Bertrand Le Saux (European Space Agency (ESA)); Inne Vanderkelen (Vrije Universiteit Brussel); Wim Thiery (Vrije Universiteit Brussel)
ICML 2021 Street2Sat: A Machine Learning Pipeline for Generating Ground-truth Geo-referenced Labeled Datasets from Street-Level Images (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Ground-truth labels on crop type and other variables are critically needed to develop machine learning methods that use satellite observations to combat climate change and food insecurity. These labels difficult and costly to obtain over large areas, particularly in Sub-Saharan Africa where they are most scarce. We propose Street2Sat, a new framework for obtaining large data sets of geo-referenced crop type labels obtained from vehicle-mounted cameras that can be extended to other applications. Using preliminary data from Kenya, we present promising results from this approach and identify future to improve the method before operational use in 5 countries.

Authors: Madhava Paliyam (University of Maryland); Catherine L Nakalembe (University of Maryland); Kevin Liu (University of Maryland); Richard Nyiawung (University of Guelph); Hannah R Kerner (University of Maryland)
ICML 2021 MethaNet - an AI-driven approach to quantifying methane point-source emission from high-resolution 2-D plume imagery (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Methane (CH4) is one of the most powerful anthropogenic greenhouse gases with a significant impact on global warming trajectory and tropospheric air quality. Quantifying an emission rate of observed CH4 plumes from aerial or satellite images is a critical step for understanding the local distributions and subsequently prioritizing mitigation target sites. However, there exists no method that can reliably predict emission rates from detected plumes in real-time without ancillary data. Here, we trained a convolutional neural network model, called MethaNet, to predict methane point-source emission directly from high-resolution 2-D plume images without relying on other local measurements such as background wind speeds. Our results support the basis for the applicability of using deep learning techniques to quantify CH4 point sources in an automated manner over large geographical areas. MethaNet opens the way for real-time monitoring systems, not only for present and future airborne field campaigns but also for upcoming space-based observations in this decade.

Authors: Siraput Jongaramrungruang (Caltech)
ICML 2021 Tackling the Overestimation of Forest Carbon with Deep Learning and Aerial Imagery (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Forest carbon offsets are increasingly popular and can play a significant role in financing climate mitigation, forest conservation, and reforestation. Measuring how much carbon is stored in forests is, however, still largely done via expensive, time-consuming, and sometimes unaccountable field measurements. To overcome these limitations, many verification bodies are leveraging machine learning (ML) algorithms to estimate forest carbon from satellite or aerial imagery. Aerial imagery allows for tree species or family classification, which improves on the satellite imagery-based forest type classification. However, aerial imagery is significantly more expensive to collect and it is unclear by how much the higher resolution improves the forest carbon estimation. This proposal paper describes the first systematic comparison of forest carbon estimation from aerial imagery, satellite imagery, and “ground-truth“ field measurements via deep learning-based algorithms for a tropical reforestation project. Our initial results show that forest carbon estimates from satellite imagery can overestimate above-ground biomass by more than 10-times for tropical reforestation projects. The significant difference between aerial and satellite-derived forest carbon measurements shows the potential for aerial imagery-based ML algorithms and raises the importance to extend this study to a global benchmark between options for carbon measurements.

Authors: Gyri Reiersen (TUM); David Dao (ETH Zurich); Björn Lütjens (MIT); Konstantin Klemmer (University of Warwick); Xiaoxiang Zhu (Technical University of Munich,Germany); Ce Zhang (ETH)
ICML 2021 Enhancing Laboratory-scale Flow Imaging of Fractured Geological Media with Deep Learning Super Resolution (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Injection into deep geological formations is a promising approach for the utilization, sequestration, and removal from the atmosphere of CO2 emissions. Laboratory experiments are essential to characterize how CO2 flows and reacts in various types of geological media. We reproduce such dynamic injection processes while imaging using Computed Tomography (CT) at sufficient temporal resolution to visualize changes in the flow field. The resolution of CT, however, is on the order of 100's of micrometers and insufficient to characterize fine-scale reaction-induced alterations to micro-fractures. Super resolution deep learning is, therefore, an essential tool to improve spatial resolution of dynamic CT images. We acquired and processed pairs of multi-scale low- and high-resolution CT rock images. We also show the performance of our baseline model on fractured rock images using peak signal to noise ratio and structural similarity index. Coupling dynamic CT imaging with deep learning results in visualization with enhanced spatial resolution of about a factor of 4 thereby enabling improved interpretation.

Authors: Manju Pharkavi Murugesu (Stanford University); Timothy Anderson (Stanford University); Niccolo Dal Santo (MathWorks, Inc.); Vignesh Krishnan (The MathWorks Ltd); Anthony Kovscek (Stanford University)
ICML 2021 Deep learning network to project future Arctic ocean waves (Proposals Track)
Abstract and authors: (click to expand)

Abstract: The Arctic Ocean is warming at an alarming rate and will likely become ice-free in summer by mid-century. This will be accompanied by higher ocean surface waves, which pose a risk to coastal communities and marine operations. In order to develop climate change adaptation strategies, it is imperative to robustly assess the future changes in the Arctic ocean wave climate. This requires a large ensemble of regional ocean wave projections to properly capture the range of climate modeling uncertainty in the Arctic region. This has been proven challenging, as ocean wave information is typically not provided by climate models, ocean wave numerical modeling is computationally expensive, and most global wave climate ensembles exclude the Arctic region. Here we present a framework to develop a deep learning network based on CNN and LSTM which could be potentially used to obtain such a large ensemble of Arctic wave projections with an affordable cost.

Authors: Merce Casas Prat (Environment and Climate Change Canada); Lluis Castrejon (Mila, Université de Montréal, Facebook AI Research); Shady Moahmmed (University of Ottawa)
ICML 2021 Leveraging Domain Adaptation for Low-Resource Geospatial Machine Learning (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Machine learning in remote sensing has matured alongside a proliferation in availability and resolution of geospatial imagery, but its utility is bottlenecked by the need for labeled data. What's more, many labeled geospatial datasets are specific to certain regions, instruments, or extreme weather events. We investigate the application of modern domain-adaptation to multiple proposed geospatial benchmarks, uncovering unique challenges and proposing solutions to them.

Authors: John M Lynch (NC State University); Sam Wookey (Masterful AI)
NeurIPS 2020 A Temporally Consistent Image-based Sun Tracking Algorithm for Solar Energy Forecasting Applications (Papers Track)
Abstract and authors: (click to expand)

Abstract: Improving irradiance forecasting is critical to further increase the share of solar in the energy mix. On a short time scale, fish-eye cameras on the ground are used to capture cloud displacements causing the local variability of the electricity production. As most of the solar radiation comes directly from the Sun, current forecasting approaches use its position in the image as a reference to interpret the cloud cover dynamics. However, existing Sun tracking methods rely on external data and a calibration of the camera, which requires access to the device. To address these limitations, this study introduces an image-based Sun tracking algorithm to localise the Sun in the image when it is visible and interpolate its daily trajectory from past observations. We validate the method on a set of sky images collected over a year at SIRTA's lab. Experimental results show that the proposed method provides robust smooth Sun trajectories with a mean absolute error below 1% of the image size.

Authors: Quentin Paletta (University of Cambridge); Joan Lasenby (University of Cambridge)
NeurIPS 2020 Characterization of Industrial Smoke Plumes from Remote Sensing Data (Papers Track)
Abstract and authors: (click to expand)

Abstract: The major driver of global warming has been identified as the anthropogenic release of greenhouse gas (GHG) emissions from industrial activities. The quantitative monitoring of these emissions is mandatory to fully understand their effect on the Earth’s climate and to enforce emission regulations on a large scale. In this work, we investigate the possibility to detect and quantify industrial smoke plumes from globally and freely available multi-band image data from ESA’s Sentinel-2 satellites. Using a modified ResNet-50, we can detect smoke plumes of different sizes with an accuracy of 94.3%. The model correctly ignores natural clouds and focuses on those imaging channels that are related to the spectral absorption from aerosols and water vapor, enabling the localization of smoke. We exploit this localization ability and train a U-Net segmentation model on a labeled sub-sample of our data, resulting in an Intersection-over-Union (IoU) metric of 0.608 and an overall accuracy for the detection of any smoke plume of 94.0%; on average, our model can reproduce the area covered by smoke in an image to within 5.6%. The performance of our model is mostly limited by occasional confusion with surface objects, the inability to identify semi-transparent smoke, and human limitations to properly identify smoke based on RGB-only images. Nevertheless, our results enable us to reliably detect and qualitatively estimate the level of smoke activity in order to monitor activity in industrial plants across the globe. Our data set and code base are publicly available.

Authors: Michael Mommert (University of St. Gallen); Mario Sigel (Sociovestix Labs Ltd.); Marcel Neuhausler (ISS Technology Innovation Lab); Linus M. Scheibenreif (University of St. Gallen); Damian Borth (University of St. Gallen)
NeurIPS 2020 Towards Tracking the Emissions of Every Power Plant on the Planet (Papers Track) Best Pathway to Impact
Abstract and authors: (click to expand)

Abstract: Greenhouse gases emitted from fossil-fuel-burning power plants are a major contributor to climate change. Current methods to track emissions from individual sources are expensive and only used in a few countries. While carbon dioxide concentrations can be measured globally using remote sensing, background fluctuations and low spatial resolution make it difficult to attribute emissions to individual sources. We use machine learning to infer power generation and emissions from visible and thermal power plant signatures in satellite images. By training on a data set of power plants for which we know the generation or emissions, we are able to apply our models globally. This paper demonstrates initial progress on this project by predicting whether a power plant is on or off from a single satellite image.

Authors: Heather D Couture (Pixel Scientia Labs); Joseph O'Connor (Carbon Tracker); Grace Mitchell (WattTime); Isabella Söldner-Rembold (Carbon Tracker); Durand D'souza (Carbon Tracker); Krishna Karra (WattTime); Keto Zhang (WattTime); Ali Rouzbeh Kargar (WattTime); Thomas Kassel (WattTime); Brian Goldman (Google); Daniel Tyrrell (Google); Wanda Czerwinski (Google); Alok Talekar (Google); Colin McCormick (Georgetown University)
NeurIPS 2020 Formatting the Landscape: Spatial conditional GAN for varying population in satellite imagery (Papers Track)
Abstract and authors: (click to expand)

Abstract: Climate change is expected to reshuffle the settlement landscape: forcing people in affected areas to migrate, to change their lifeways, and continuing to affect demographic change throughout the world. Changes to the geographic distribution of population will have dramatic impacts on land use and land cover and thus constitute one of the major challenges of planning for climate change scenarios. In this paper, we explore a generative model framework for generating satellite imagery conditional on gridded population distributions. We make additions to the existing ALAE [30] architecture, creating a spatially conditional version: SCALAE. This method allows us to explicitly disentangle population from the model’s latent space and thus input custom population forecasts into the generated imagery. We postulate that such imagery could then be directly used for land cover and land use change estimation using existing frameworks, as well as for realistic visualisation of expected local change. We evaluate the model by comparing pixel and semantic reconstructions, as well as calculate the standard FID metric. The results suggest the model captures population distributions accurately and delivers a controllable method to generate realistic satellite imagery.

Authors: Tomas Langer (Intuition Machines); Natalia Fedorova (Max Planck Institute for Evolutionary Anthropology); Ron Hagensieker (Osir.io)
NeurIPS 2020 Predicting Landsat Reflectance with Deep Generative Fusion (Papers Track)
Abstract and authors: (click to expand)

Abstract: Public satellite missions are commonly bound to a trade-off between spatial and temporal resolution as no single sensor provides fine-grained acquisitions with frequent coverage. This hinders their potential to assist vegetation monitoring or humanitarian actions, which require detecting rapid and detailed terrestrial surface changes. In this work, we probe the potential of deep generative models to produce high-resolution optical imagery by fusing products with different spatial and temporal characteristics. We introduce a dataset of co-registered Moderate Resolution Imaging Spectroradiometer (MODIS) and Landsat surface reflectance time series and demonstrate the ability of our generative model to blend coarse daily reflectance information into low-paced finer acquisitions. We benchmark our proposed model against state-of-the-art reflectance fusion algorithms.

Authors: Shahine Bouabid (University of Oxford); Jevgenij Gamper (Cervest Ltd.)
NeurIPS 2020 Monitoring the Impact of Wildfires on Tree Species with Deep Learning (Papers Track)
Abstract and authors: (click to expand)

Abstract: One of the impacts of climate change is the difficulty of tree regrowth after wildfires over areas that traditionally were covered by certain tree species. Here a deep learning model is customized to classify land covers from four-band aerial imagery before and after wildfires to study the prolonged consequences of wildfires on tree species. The tree species labels are generated from manually delineated maps for five land cover classes: Conifer, Hardwood, Shrub, ReforestedTree, and Barren land. With an accuracy of 92% on the test split, the model is applied to three wildfires on data from 2009 to 2018. The model accurately delineates areas damaged by wildfires, changes in tree species, and regrowth in burned areas. The result shows clear evidence of wildfires impacting the local ecosystem and the outlined approach can help monitor reforested areas, observe changes in forest composition, and track wildfire impact on tree species.

Authors: Wang Zhou (IBM Research); Levente Klein (IBM Research)
NeurIPS 2020 ForestNet: Classifying Drivers of Deforestation in Indonesia using Deep Learning on Satellite Imagery (Papers Track)
Abstract and authors: (click to expand)

Abstract: Characterizing the processes leading to deforestation is critical to the development and implementation of targeted forest conservation and management policies. In this work, we develop a deep learning model called ForestNet to classify the drivers of primary forest loss in Indonesia, a country with one of the highest deforestation rates in the world. Using satellite imagery, ForestNet identifies the direct drivers of deforestation in forest loss patches of any size. We curate a dataset of Landsat 8 satellite images of known forest loss events paired with driver annotations from expert interpreters. We use the dataset to train and validate the models and demonstrate that ForestNet substantially outperforms other standard driver classification approaches. In order to support future research on automated approaches to deforestation driver classification, the dataset curated in this study is publicly available at https://stanfordmlgroup.github.io/projects/forestnet .

Authors: Jeremy A Irvin (Stanford); Hao Sheng (Stanford University); Neel Ramachandran (Stanford University); Sonja Johnson-Yu (Stanford University); Sharon Zhou (Stanford University); Kyle Story (Descartes Labs); Rose Rustowicz (Descartes Labs); Cooper Elsworth (Descartes Labs); Kemen Austin (RTI International); Andrew Ng (Stanford University)
NeurIPS 2020 Mangrove Ecosystem Detection using Mixed-Resolution Imagery with a Hybrid-Convolutional Neural Network (Papers Track)
Abstract and authors: (click to expand)

Abstract: Mangrove forests are rich in biodiversity and are a large contributor to carbon sequestration critical in the fight against climate change. However, they are currently under threat from anthropogenic activities, so monitoring their health, extent, and productivity is vital to our ability to protect these important ecosystems. Traditionally, lower resolution satellite imagery or high resolution unmanned air vehicle (UAV) imagery has been used independently to monitor mangrove extent, both offering helpful features to predict mangrove extent. To take advantage of both of these data sources, we propose the use of a hybrid neural network, which combines a Convolutional Neural Network (CNN) feature extractor with a Multilayer-Perceptron (MLP), to accurately detect mangrove areas using both medium resolution satellite and high resolution drone imagery. We present a comparison of our novel Hybrid CNN with algorithms previously applied to mangrove image classification on a data set we collected of dwarf mangroves from consumer UAVs in Baja California Sur, Mexico, and show a 95\% intersection over union (IOU) score for mangrove image classification, outperforming all our baselines.

Authors: Dillon Hicks (Engineers for Exploration); Ryan Kastner (University of California San Diego); Curt Schurgers (University of California San Diego); Astrid Hsu (University of California San Diego); Octavio Aburto (University of California San Diego)
NeurIPS 2020 Predicting the Solar Potential of Rooftops using Image Segmentation and Structured Data (Papers Track)
Abstract and authors: (click to expand)

Abstract: Estimating the amount of electricity that can be produced by rooftop photovoltaic systems is a time-consuming process that requires on-site measurements, a difficult task to achieve on a large scale. In this paper, we present an approach to estimate the solar potential of rooftops based on their location and architectural characteristics, as well as the amount of solar radiation they receive annually. Our technique uses computer vision to achieve semantic segmentation of roof sections and roof objects on the one hand, and a machine learning model based on structured building features to predict roof pitch on the other hand. We then compute the azimuth and maximum number of solar panels that can be installed on a rooftop with geometric approaches. Finally, we compute precise shading masks and combine them with solar irradiation data that enables us to estimate the yearly solar potential of a rooftop.

Authors: Daniel de Barros Soares (nam.R); François ANDRIEUX (nam.R); Bastien HELL (nam.R); Julien LENHARDT (nam.R; ENSTA); JORDI BADOSA (Ecole Polytechnique); Sylvain GAVOILLE (nam.R); Stéphane GAIFFAS (nam.R; LPSM (Université de Paris)); Emmanuel BACRY (nam.R; CEREMADE (Université Paris Dauphine, PSL))
NeurIPS 2020 Annual and in-season mapping of cropland at field scale with sparse labels (Papers Track)
Abstract and authors: (click to expand)

Abstract: Spatial information about where crops are being grown, known as cropland maps, are critical inputs for analyses and decision-making related to food security and climate change. Despite a widespread need for readily-updated annual and in-season cropland maps at the management (field) scale, these maps are unavailable for most regions at risk of food insecurity. This is largely due to lack of in-situ labels for training and validating machine learning classifiers. Previously, we developed a method for binary classification of cropland that learns from sparse local labels and abundant global labels using a multi-headed LSTM and time-series multispectral satellite inputs over one year. In this work, we present a new method that uses an autoregressive LSTM to classify cropland during the growing season (i.e., partially-observed time series). We used these methods to produce publicly-available 10m-resolution cropland maps in Kenya for the 2019-2020 and 2020-2021 growing seasons. These are the highest-resolution and most recent cropland maps publicly available for Kenya. These methods and associated maps are critical for scientific studies and decision-making at the intersection of food security and climate change.

Authors: Gabriel Tseng (NASA Harvest); Hannah R Kerner (University of Maryland); Catherine L Nakalembe (University of Maryland); Inbal Becker-Reshef (University of Maryland)
NeurIPS 2020 NightVision: Generating Nighttime Satellite Imagery from Infra-Red Observations (Papers Track)
Abstract and authors: (click to expand)

Abstract: The recent explosion in applications of machine learning to satellite imagery often rely on visible images and therefore suffer from a lack of data during the night. The gap can be filled by employing available infra-red observations to generate visible images. This work presents how Deep Learning can be applied successfully to create those images by using U-Net based architectures. The proposed methods show promising results, achieving a structural similarity index (SSIM) up to 86\% on an independent test set and providing visually convincing output images, generated from infra-red observations.

Authors: Paula Harder (Fraunhofer ITWM); William Jones (University of Oxford); Redouane Lguensat (LSCE-IPSL); Shahine Bouabid (University of Oxford); James Fulton (University of Edinburgh); Dánnell Quesada-Chacón (Technische Universität Dresden); Aris Marcolongo (University of Bern); Sofija Stefanovic (University of Oxford); Yuhan Rao (North Carolina Institute for Climate Studies); Peter Manshausen (University of Oxford); Duncan Watson-Parris (University of Oxford)
NeurIPS 2020 Automated Identification of Oil Field Features using CNNs (Papers Track)
Abstract and authors: (click to expand)

Abstract: Oil and gas production sites have been identified as a major source of anthropogenic methane emissions. Emissions studies utilize counts of equipment to estimate emissions from production facilities. However these counts are poorly documented, including both information about well pad locations and major equipment on each well pad. While these data can be found by manually reviewing satellite imagery, it is prohibitively difficult and time consuming. This work, part of a larger study of methane emission studies in Colorado, US, adapted a machine learning (ML) algorithm to detect well pads and associated equipment. Our initial model showed an average well pad detection accuracy of 95% on the Denver-Julesburg (DJ) basin in northeastern Colorado. Our example demonstrates the potential for this type of automated detection from satellite imagery, leading to more accurate and complete models of production emissions.

Authors: SONU DILEEP (Colorado State University); Daniel Zimmerle (Colorado State University); Ross Beveridge (CSU); Timothy Vaughn (Colorado State University)
NeurIPS 2020 Street to Cloud: Improving Flood Maps With Crowdsourcing and Semantic Segmentation (Papers Track)
Abstract and authors: (click to expand)

Abstract: To address the mounting destruction caused by floods in climate-vulnerable regions, we propose Street to Cloud, a machine learning pipeline for incorporating crowdsourced ground truth data into the segmentation of satellite imagery of floods. We propose this approach as a solution to the labor-intensive task of generating high-quality, hand-labeled training data, and demonstrate successes and failures of different plausible crowdsourcing approaches in our model. Street to Cloud leverages community reporting and machine learning to generate novel, near-real time insights into the extent of floods to be used for emergency response.

Authors: Veda Sunkara (Cloud to Street); Matthew Purri (Rutgers University); Bertrand Le Saux (European Space Agency (ESA)); Jennifer Adams (European Space Agency (ESA))
NeurIPS 2020 Movement Tracks for the Automatic Detection of Fish Behavior in Videos (Papers Track)
Abstract and authors: (click to expand)

Abstract: Global warming is predicted to profoundly impact ocean ecosystems. Fish behavior is an important indicator of changes in such marine environments. Thus, the automatic identification of key fish behavior in videos represents a much needed tool for marine researchers, enabling them to study climate change-related phenomena. We offer a dataset of sablefish (Anoplopoma fimbria) startle behaviors in underwater videos, and investigate the use of deep learning (DL) methods for behavior detection on it. Our proposed detection system identifies fish instances using DL-based frameworks, determines trajectory tracks, derives novel behavior-specific features, and employs Long Short-Term Memory (LSTM) networks to identify startle behavior in sablefish. Its performance is studied by comparing it with a state-of-the-art DL-based video event detector.

Authors: Declan GD McIntosh (University Of Victoria); Tunai Porto Marques (University of Victoria); Alexandra Branzan Albu (University of Victoria); Rodney Rountree (University of Victoria); Fabio De Leo Cabrera (Ocean Networks Canada)
NeurIPS 2020 Counting Cows: Tracking Illegal Cattle Ranching From High-Resolution Satellite Imagery (Papers Track)
Abstract and authors: (click to expand)

Abstract: Cattle farming is responsible for 8.8\% of greenhouse gas emissions worldwide. In addition to the methane emitted due to their digestive process, the growing need for grazing areas is an important driver of deforestation. While some regulations are in place for preserving the Amazon against deforestation, these are being flouted in various ways. Hence the need to scale and automate the monitoring of cattle ranching activities. Through a partnership with \textit{Anonymous under review}, we explore the feasibility of tracking and counting cattle at the continental scale from satellite imagery. With a license from Maxar Technologies, we obtained satellite imagery of the Amazon at 40cm resolution, and compiled a dataset of 903 images containing a total of 28498 cattle. Our experiments show promising results and highlight important directions for the next steps on both counting algorithms and the data collection processes for solving such challenges.

Authors: Issam Hadj Laradji (Element AI); Pau Rodriguez (Element AI); Alfredo Kalaitzis (University of Oxford); David Vazquez (Element AI); Ross Young (Element AI); Ed Davey (Global Witness); Alexandre Lacoste (Element AI)
NeurIPS 2020 Machine learning for advanced solar cell production: adversarial denoising, sub-pixel alignment and the digital twin (Papers Track)
Abstract and authors: (click to expand)

Abstract: Photovoltaic is a main pillar to achieve the transition towards a renewable energy supply. In order to continue the tremendous cost decrease of the last decades, novel cell techologies and production processes are implemented into mass production to improve cell efficiency. Raising their full potential requires novel techniques of quality assurance and data analysis. We present three use-cases along the value chain where machine learning techniques are investigated for quality inspection and process optimization: Adversarial learning to denoise wafer images, alignment of surface structuring processes via sub-pixel coordinate regression, and the development of a digital twin for wafers and solar cells for material and process analysis.

Authors: Matthias Demant (Fraunhofer ISE); Leslie Kurumundayil (Fraunhofer ISE); Philipp Kunze (Fraunhofer ISE); Aditya Kovvali (Fraunhofer ISE); Alexandra Woernhoer (Fraunhofer ISE); Stefan Rein (Fraunhofer ISE)
NeurIPS 2020 FireSRnet: Geoscience-driven super-resolution of future fire risk from climate change (Papers Track)
Abstract and authors: (click to expand)

Abstract: With fires becoming increasingly frequent and severe across the globe in recent years, understanding climate change’s role in fire behavior is critical for quantifying current and future fire risk. However, global climate models typically simulate fire behavior at spatial scales too coarse for local risk assessments. Therefore, we propose a novel approach towards super-resolution (SR) enhancement of fire risk exposure maps that incorporates not only 2000 to 2020 monthly satellite observations of active fires but also local information on land cover and temperature. Inspired by SR architectures, we propose an efficient deep learning model trained for SR on fire risk exposure maps. We evaluate this model on resolution enhancement and find it outperforms standard image interpolation techniques at both 4x and 8x enhancement while having comparable performance at 2x enhancement. We then demonstrate the generalizability of this SR model over northern California and New South Wales, Australia. We conclude with a discussion and application of our proposed model to climate model simulations of fire risk in 2040 and 2100, illustrating the potential for SR enhancement of fire risk maps from the latest state-of-the-art climate models.

Authors: Tristan C Ballard (Sust Global, Stanford University); Gopal Erinjippurath (Sust Global)
NeurIPS 2020 An Enriched Automated PV Registry: Combining Image Recognition and 3D Building Data (Papers Track)
Abstract and authors: (click to expand)

Abstract: While photovoltaic (PV) systems are installed at an unprecedented rate, reliable information on an installation level remains scarce. As a result, automatically created PV registries are a timely contribution to optimize grid planning and operations. This paper demonstrates how aerial imagery and three-dimensional building data can be combined to create an address-level PV registry, specifying area, tilt, and orientation angles. We demonstrate the benefits of this approach for PV capacity estimation. In addition, this work presents, for the first time, a comparison between automated and officially-created PV registries. Our results indicate that our enriched automated registry proves to be useful to validate, update, and complement official registries.

Authors: Benjamin Rausch (Stanford); Kevin Mayer (Stanford); Marie-Louise Arlt (Stanford); Gunther Gust (University of Freiburg); Philipp Staudt (KIT); Christof Weinhardt (Karlsruhe Institute of Technology); Dirk Neumann (Universität Freiburg); Ram Rajagopal (Stanford University)
NeurIPS 2020 Satellite imagery analysis for Land Use, Land Use Change and Forestry: A pilot study in Kigali, Rwanda (Papers Track)
Abstract and authors: (click to expand)

Abstract: Estimating greenhouse gases from the Agriculture, Forestry and Other land Use (AFOLU) sector is very challenging partly due to the unavailability of data (particularly for land use and land use change sectors) and inadequate experts to analyze this data in case it is available. We used Collect Earth together with Machine Learning techniques to be able to predict and classify all the land use types based on some few points collected using Collect Earth. We investigated the adoption of this tool and technology in Rwanda to help its national and sub-national inventories. The use of Collect Earth and the Machine Learning (ML) implementation will help Rwanda monitor and predict its Land Use, Land Use Change, and Forestry in a cost-effective manner whiles enhancing the quality of reports submitted to national and international bodies whiles introducing a new approach. Among the classification algorithms we tested, we had an overall classification accuracy of 97% using the Classification and Regression Trees (CART) algorithm to to predict the six land Use classes across the country.

Authors: Bright Aboh (African Institute for Mathematical Sciences); Alphonse Mutabazi (UN Environment Program)
NeurIPS 2020 EarthNet2021: A novel large-scale dataset and challenge for forecasting localized climate impacts (Papers Track)
Abstract and authors: (click to expand)

Abstract: Climate change is global, yet its concrete impacts can strongly vary between different locations in the same region. Seasonal weather forecasts currently operate at the mesoscale (> 1 km). For more targeted mitigation and adaptation, modelling impacts to < 100 m is needed. Yet, the relationship between driving variables and Earth’s surface at such local scales remains unresolved by current physical models. Large Earth observation datasets now enable us to create machine learning models capable of translating coarse weather information into high-resolution Earth surface forecasts encompassing localized climate impacts. Here, we define high-resolution Earth surface forecasting as video prediction of satellite imagery conditional on mesoscale weather forecasts. Video prediction has been tackled with deep learning models. Developing such models requires analysis-ready datasets. We introduce EarthNet2021, a new, curated dataset containing target spatio-temporal Sentinel 2 satellite imagery at 20 m resolution, matched with high-resolution topography and mesoscale (1.28 km) weather variables. With over 32000 samples it is suitable for training deep neural networks. Comparing multiple Earth surface forecasts is not trivial. Hence, we define the EarthNetScore, a novel ranking criterion for models forecasting Earth surface reflectance. For model intercomparison we frame EarthNet2021 as a challenge with four tracks based on different test sets. These allow evaluation of model validity and robustness as well as model applicability to extreme events and the complete annual vegetation cycle. In addition to forecasting directly observable weather impacts through satellite-derived vegetation indices, capable Earth surface models will enable downstream applications such as crop yield prediction, forest health assessments, coastline management, or biodiversity monitoring. Find data, code, and how to participate at www.earthnet.tech .

Authors: Christian Requena-Mesa (Computer Vision Group, Friedrich Schiller University Jena; DLR Institute of Data Science, Jena; Max Planck Institute for Biogeochemistry, Jena); Vitus Benson (Max-Planck-Institute for Biogeochemistry); Jakob Runge (Institute of Data Science, German Aerospace Center (DLR)); Joachim Denzler (Computer Vision Group, Friedrich Schiller University Jena, Germany); Markus Reichstein (Max Planck Institute for Biogeochemistry, Jena; Michael Stifel Center Jena for Data-Driven and Simulation Science, Jena)
NeurIPS 2020 OGNet: Towards a Global Oil and Gas Infrastructure Database using Deep Learning on Remotely Sensed Imagery (Papers Track)
Abstract and authors: (click to expand)

Abstract: At least a quarter of the warming that the Earth is experiencing today is due to anthropogenic methane emissions. There are multiple satellites in orbit and planned for launch in the next few years which can detect and quantify these emissions; however, to attribute methane emissions to their sources on the ground, a comprehensive database of the locations and characteristics of emission sources worldwide is essential. In this work, we develop deep learning algorithms that leverage freely available high-resolution aerial imagery to automatically detect oil and gas infrastructure, one of the largest contributors to global methane emissions. We use the best algorithm, which we call OGNet, together with expert review to identify the locations of oil refineries and petroleum terminals in the U.S. We show that OGNet detects many facilities which are not present in four standard public datasets of oil and gas infrastructure. All detected facilities are associated with characteristics critical to quantifying and attributing methane emissions, including the types of infrastructure and number of storage tanks. The data curated and produced in this study is freely available at https://link/provided/in/camera/ready/version.

Authors: Hao Sheng (Stanford University); Jeremy A Irvin (Stanford); Sasankh Munukutla (Stanford University); Shawn Zhang (Stanford University); Christopher Cross (Stanford University); Zutao Yang (Stanford University); Kyle Story (Descartes Labs); Rose Rustowicz (Descartes Labs); Cooper Elsworth (Descartes Labs); Mark Omara (Environmental Defense Fund); Ritesh Gautam (Environmental Defense Fund); Rob Jackson (Stanford University); Andrew Ng (Stanford University)
NeurIPS 2020 VConstruct: Filling Gaps in Chl-a Data Using a Variational Autoencoder (Papers Track)
Abstract and authors: (click to expand)

Abstract: Remote sensing of Chlorophyll-a is vital in monitoring climate change. Chlorphylla measurements give us an idea of the algae concentrations in the ocean, which lets us monitor ocean health. However, a common problem is that the satellites used to gather the data are commonly obstructed by clouds and other artifacts. This means that time series data from satellites can suffer from spatial data loss. There are a number of algorithms that are able to reconstruct the missing parts of these images to varying degrees of accuracy, with Data INterpolating Empirical Orthogonal Functions (DINEOF) being the current standard. However, DINEOF is slow, suffers from accuracy loss in temporally homogenous waters, reliant on temporal data, and only able to generate a single potential reconstruction. We propose a machine learning approach to reconstruction of Chlorophyll-a data using a Variational Autoencoder (VAE). Our accuracy results to date are competitive with but slightly less accurate than DINEOF. We show the benefits of our method including vastly decreased computation time and ability to generate multiple potential reconstructions. Lastly, we outline our planned improvements and future work.

Authors: Matthew Ehrler (University of Victoria); Neil Ernst (University of Victoria)
NeurIPS 2020 Automated Salmonid Counting in Sonar Data (Papers Track)
Abstract and authors: (click to expand)

Abstract: The prosperity of salmonids is crucial for several ecological and economic functions. Accurately counting spawning salmonids during their seasonal migration is essential in monitoring threatened populations, assessing the efficacy of recovery strategies, guiding fishing season regulations, and supporting the management of commercial and recreational fisheries. While several different methods exist for counting river fish, they all rely heavily on human involvement, introducing a hefty financial and time burden. In this paper we present an automated fish counting method that utilizes data captured from ARIS sonar cameras to detect and track salmonids migrating in rivers. Our results show that our fully automated system has a 19.3% per-clip error when compared to human counting performance. There is room to improve, but our system can already decrease the amount of time field biologists and fishery managers need to spend manually watching ARIS clips.

Authors: Peter Kulits (Caltech); Angelina Pan (Caltech); Sara M Beery (Caltech); Erik Young (Trout Unlimited); Pietro Perona (California Institute of Technology); Grant Van Horn (Cornell University)
NeurIPS 2020 Short-term PV output prediction using convolutional neural network: learning from an imbalanced sky images dataset via sampling and data augmentation (Papers Track)
Abstract and authors: (click to expand)

Abstract: Integrating photovoltaics (PV) into electricity grids is one of the major pathways towards a low-carbon energy system. However, the biggest challenge is the strong fluctuation in PV power generation. In recent years, sky-image-based PV output prediction using deep neural networks has emerged as a promising approach to alleviate the uncertainty. Despite the research surge in exploring different model architectures, there is currently no study addressing the issue of learning from an imbalanced sky images dataset, the outcome of which would be highly beneficial for improving the reliability of existing and new solar forecasting models. In this study, we train convolutional neural network (CNN) models from an imbalanced sky images dataset for two disparate PV output prediction tasks, i.e., nowcast and forecast. We empirically examine the efficacy of using different sampling and data augmentation approaches to create synthesized dataset for model development. We further apply a three-stage selection approach to determine the optimal sampling approach, data augmentation technique and oversampling rate.

Authors: Yuhao Nie (Stanford University); Ahmed S Zamzam (The National Renewable Energy Laboratory); Adam Brandt (Stanford University)
NeurIPS 2020 Machine Learning for Glacier Monitoring in the Hindu Kush Himalaya (Papers Track)
Abstract and authors: (click to expand)

Abstract: Glacier mapping is key to ecological monitoring in the Hindu Kush Himalaya region. Climate change poses a risk to individuals whose livelihoods depend on the health of glacier ecosystems. In this work, we present a machine learning based approach to support ecological monitoring, with a focus on glaciers. Our approach is based on semi-automated mapping from satellite images. We utilize readily available remote sensing data to create a model to identify and outline both clean ice and debris-covered glaciers from satellite imagery. We also release data and develop a web tool that allows experts to visualize and correct model predictions, with the ultimate aim of accelerating the glacier mapping process.

Authors: Shimaa Baraka (Mila); Benjamin Akera (Makerere University); Bibek Aryal (The University of Texas at El Paso); Tenzing Sherpa (International Centre for Integrated Mountain Development); Finu Shrestha (International Centre for Integrated Mountain Development); Anthony Ortiz (Microsoft); Kris Sankaran (University of Wisconsin-Madison); Juan M Lavista Ferres (Microsoft); Mir A Matin (International Center for Integrated Mountain Development); Yoshua Bengio (Mila)
NeurIPS 2020 Spatiotemporal Features Improve Fine-Grained Butterfly Image Classification (Papers Track)
Abstract and authors: (click to expand)

Abstract: Understanding the changing distributions of butterflies gives insight into the impacts of climate change across ecosystems and is a prerequisite for conservation efforts. eButterfly is a citizen science website created to allow people to track the butterfly species around them and use these observations to contribute to research. However, correctly identifying butterfly species is a challenging task for non-specialists and currently requires the involvement of entomologists to verify the labels of novice users on the website. We have developed a computer vision model to label butterfly images from eButterfly automatically, decreasing the need for human experts. We employ a model that incorporates geographic and temporal information of where and when the image was taken, in addition to the image itself. We show that we can successfully apply this spatiotemporal model for fine-grained image recognition, significantly improving the accuracy of our classification model compared to a baseline image recognition system trained on the same dataset.

Authors: Marta Skreta (University of Toronto); Sasha Luccioni (Mila); David Rolnick (McGill University, Mila)
NeurIPS 2020 In-N-Out: Pre-Training and Self-Training using Auxiliary Information for Out-of-Distribution Robustness (Papers Track)
Abstract and authors: (click to expand)

Abstract: Many machine learning applications used to tackle climate change involve lots of unlabeled data (such as satellite imagery) along with auxiliary information such as climate data. In this work, we show how to use auxiliary information in a semi-supervised setting to improve both in-distribution and out-of-distribution (OOD) accuracies (e.g. for countries in Africa where we have very little labeled data). We show that 1) on real-world datasets, the common practice of using auxiliary information as additional input features improves in-distribution error but can hurt OOD. Oppositely, we find that 2) using auxiliary information as outputs of auxiliary tasks to pre-train a model improves OOD error. 3) To get the best of both worlds, we introduce In-N-Out, which first trains a model with auxiliary inputs and uses it to pseudolabel all the in-distribution inputs, then pre-trains a model on OOD auxiliary outputs and fine-tunes this model with the pseudolabels (self-training). We show both theoretically and empirically on remote sensing datasets for land cover prediction and cropland prediction that In-N-Out outperforms auxiliary inputs or outputs alone on both in-distribution and OOD error.

Authors: Robbie M Jones (Stanford University); Sang Michael Xie (Stanford University); Ananya Kumar (Stanford University); Fereshte Khani (Stanford); Tengyu Ma (Stanford University); Percy Liang (Stanford University)
NeurIPS 2020 Towards Data-Driven Physics-Informed Global Precipitation Forecasting from Satellite Imagery (Papers Track)
Abstract and authors: (click to expand)

Abstract: Under the effects of global warming, extreme events such as floods and droughts are increasing in frequency and intensity. This trend directly affects communities and make all the more urgent widening the access to accurate precipitation forecasting systems for disaster preparedness. Nowadays, weather forecasting relies on numerical models necessitating massive computing resources that most developing countries cannot afford. Machine learning approaches are still in their infancy but already show the promise for democratizing weather predictions, by leveraging any data source and requiring less compute. In this work, we propose a methodology for data-driven and physics-aware global precipitation forecasting from satellite imagery. To fully take advantage of the available data, we design the system as three elements: 1. The atmospheric state is estimated from recent satellite data. 2. The atmospheric state is propagated forward in time. 3. The atmospheric state is used to derive the precipitation intensity within a nearby time interval. In particular, our use of stochastic methods for forecasting the atmospheric state represents a novel application in this domain.

Authors: Valentina Zantedeschi (GE Global Research); Daniele De Martini (University of Oxford); Catherine Tong (University of Oxford); Christian A Schroeder de Witt (University of Oxford); Piotr Bilinski (University of Warsaw / University of Oxford); Alfredo Kalaitzis (University of Oxford); Matthew Chantry (University of Oxford); Duncan Watson-Parris (University of Oxford)
NeurIPS 2020 Deep Fire Topology: Understanding the role of landscape spatial patterns in wildfire susceptibility (Papers Track)
Abstract and authors: (click to expand)

Abstract: Increasing wildfire activity across the globe has become an urgent issue with enormous ecological and social impacts. While there is evidence that landscape topology affects fire growth, no study has yet reported its potential influence on fire ignition. This study proposes a deep learning framework focused on understanding the impact of different landscape topologies on the ignition of a wildfire and the rationale behind these results. Our model achieves an accuracy of above 90\% in fire occurrence prediction, detection, and classification of risky areas by only exploiting topological pattern information from 17,579 landscapes. This study reveals the strong potential of landscape topology in wildfire occurrence prediction and its implications for similar research. The proposed methodology can be applied to multiple fields/studies to understand and capture the role and impact of different topological features and their interactions.

Authors: Cristobal Pais (University of California Berkeley); Alejandro Miranda (University of Chile); Jaime Carrasco (University of Chile); Zuo-Jun Shen (University of California, Berkeley)
NeurIPS 2020 Interpretability in Convolutional Neural Networks for Building Damage Classification in Satellite Imagery (Papers Track)
Abstract and authors: (click to expand)

Abstract: Natural disasters ravage the world's cities, valleys, and shores on a monthly basis. Having precise and efficient mechanisms for assessing infrastructure damage is essential to channel resources and minimize the loss of life. Using a dataset that includes labeled pre- and post- disaster satellite imagery, we train multiple convolutional neural networks to assess building damage on a per-building basis. In order to investigate how to best classify building damage, we present a highly interpretable deep-learning methodology that seeks to explicitly convey the most useful information required to train an accurate classification model. We also delve into which loss functions best optimize these models. Our findings include that ordinal-cross entropy loss is the most optimal loss function to use and that including the type of disaster that caused the damage in combination with a pre- and post-disaster image best predicts the level of damage caused. Our research seeks to computationally contribute to aiding in this ongoing and growing humanitarian crisis, heightened by climate change.

Authors: Thomas Y Chen (The Academy for Mathematics, Science, and Engineering)
NeurIPS 2020 Towards DeepSentinel: An extensible corpus of labelled Sentinel-1 and -2 imagery and a proposed general purpose sensor-fusion semantic embedding model (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Earth observation offers new insight into anthropogenic changes to nature, and how these changes are effecting (and are effected by) the built environment and the real economy. With the global availability of medium-resolution (10-30m) synthetic aperature radar (SAR) Sentinel-1 and multispectral Sentinel-2 imagery, machine learning can be employed to offer these insights at scale, unbiased to company- and country-level reporting. In this proposal, we document the development of an extensible corpus of labelled and unlabelled Sentinel-1 and Sentinel-2 imagery for the purposes of sensor fusion research. We make a large corpus and supporting code publicly available. We propose our own experiment design for the development of \emph{DeepSentinel}, a general-purpose semantic embedding model. Our aspiration is to provide pretrained models for transfer learning applications, significantly accelerating the impact of machine learning-enhanced earth observation on climate change mitigation.

Authors: Lucas Kruitwagen (University of Oxford)
NeurIPS 2020 Monitoring Shorelines via High-Resolution Satellite Imagery and Deep Learning (Proposals Track)
Abstract and authors: (click to expand)

Abstract: Twentieth-century has seen an overall sea-level rise of 0.5m [7, 11] and the studies for the twenty-first-century project the overall increment within a range of 0.5m to 2m, considering high emission scenarios and rapid melting of major Antarctic glaciers. Naturally, this has a severe impact on a major percentage of the population inhabiting coastal land zones], with a recent study placing 110million people living below the local high tide line. Of all the different coastline types, sandy shores, forming 31% of the world’s beaches, undergo major erosion and accretion changes and hence are of special focus in this paper. Because of these reasons, it is paramount to regularly monitor the coastline changes across the world for better understanding and to create necessary preparation and mitigation strategies.

Authors: Venkatesh Ramesh (HyperVerge); Digvijay Singh (HyperVerge)
NeurIPS 2020 Residue Density Segmentation for Monitoring and Optimizing Tillage Practices (Proposals Track)
Abstract and authors: (click to expand)

Abstract: "No-till" and cover cropping are often identified as the leading simple, best management practices for carbon sequestration in agriculture. However, the root of the problem is more complex, with the potential benefits of these approaches depending on numerous factors including a field's soil type(s), topography, and management history. Instead of using computer vision approaches to simply classify a field as till vs. no-till, we instead seek to identify the degree of residue coverage across a field through a probabilistic deep learning segmentation approach to enable more accurate analysis of carbon holding potential and realization. This approach will not only provide more precise insights into currently implemented practices, but also enable a more accurate identification process of fields with the greatest potential for adopting new practices to significantly impact carbon sequestration in agriculture.

Authors: Jennifer Hobbs (IntelinAir); Ivan A Dozier (IntelinAir); Naira Hovakimyan (UIUC)
NeurIPS 2020 A Multi-source, End-to-End Solution for Tracking Climate Change Adaptation in Agriculture (Proposals Track)
Abstract and authors: (click to expand)

Abstract: The impact of climate change on tropical agri-food systems will depend on both the direction and magnitude of climate change, and the agricultural sector’s adaptive capacity, the latter being affected by the chosen adaptation strategies. By extending SEIRS, a Satellite Remote Sensing (SRS) based system originally developed by the International Center for Tropical Agriculture - CIAT for monitoring U.S. Government-funded development programs across cropping areas in Africa, this research proposes the development and deployment of a scalable AI-based platform exploiting free-of-charge SRS data that will enable the agri-food sector to monitor a wide range of climate change adaptation (CCA) interventions in a timely, evidence-driven and comparable manner. The main contributions of the platform are i) ingesting and processing variety sources of SRS data with a considerable record (> 5 years) of vegetation greenness and precipitation (input data); ii) operating an end-to-end system by exploiting AI-based models suited to time series analysis such as Seq2Seq and Transformers; iii) providing customised proxies informing the success or failure of a given local CCA intervention(s).

Authors: Alejandro Coca-Castro (Kings College London); Aaron Golden (NUI Galway); Louis Reymondin (The Alliance of Bioversity International and the International Center for Tropical Agriculture)
NeurIPS 2020 Expert-in-the-loop Systems Towards Safety-critical Machine Learning Technology in Wildfire Intelligence (Proposals Track)
Abstract and authors: (click to expand)

Abstract: With the advent of climate change, wildfires are becoming more frequent and severe across several regions worldwide. To prevent and mitigate its effects, wildfire intelligence plays a pivotal role, e.g. to monitor the evolution of wildfires and for early detection in high-risk areas such as wildland-urban-interface regions. Recent works have proposed deep learning solutions for fire detection tasks, however the current limited databases prevent reliable real-world deployments. We propose the development of expert-in-the-loop systems that combine the benefits of semi-automated data annotation with relevant domain knowledge expertise. Through this approach we aim to improve the data curation process and contribute to the generation of large-scale image databases for relevant wildfire tasks and empower the application of machine learning techniques in wildfire intelligence in real scenarios.

Authors: Maria João Sousa (IDMEC, Instituto Superior Técnico, Universidade de Lisboa); Alexandra Moutinho (IDMEC, Instituto Superior Técnico, Universidade de Lisboa); Miguel Almeida (ADAI, University of Coimbra)
NeurIPS 2020 Artificial Intelligence, Machine Learning and Modeling for Understanding the Oceans and Climate Change (Proposals Track)
Abstract and authors: (click to expand)

Abstract: These changes will have a drastic impact on almost all forms of life in the ocean with further consequences on food security, ecosystem services in coastal and inland communities. Despite these impacts, scientific data and infrastructures are still lacking to understand and quantify the consequences of these perturbations on the marine ecosystem. Understanding this phenomenon is not only an urgent but also a scientifically demanding task. Consequently, it is a problem that must be addressed with a scientific cohort approach, where multi-disciplinary teams collaborate to bring the best of different scientific areas. In this proposal paper, we describe our newly launched four-years project focused on developing new artificial intelligence, machine learning, and mathematical modeling tools to contribute to the understanding of the structure, functioning, and underlying mechanisms and dynamics of the global ocean symbiome and its relation with climate change. These actions should enable the understanding of our oceans and predict and mitigate the consequences of climate change.

Authors: Nayat Sánchez Pi (Inria); Luis Martí (Inria); André Abreu (Fountation Tara Océans); Olivier Bernard (Inria); Colomban de Vargas (CNRS); Damien Eveillard (Univ. Nantes); Alejandro Maass (CMM, U. Chile); Pablo Marquet (PUC); Jacques Sainte-Marie (Inria); Julien Salomin (Inria); Marc Schoenauer (INRIA); Michele Sebag (LRI, CNRS, France)
ICML 2019 Detecting anthropogenic cloud perturbations with deep learning (Research Track) Best Paper Award
Abstract and authors: (click to expand)

Abstract: One of the most pressing questions in climate science is that of the effect of anthropogenic aerosol on the Earth's energy balance. Aerosols provide the `seeds' on which cloud droplets form, and changes in the amount of aerosol available to a cloud can change its brightness and other physical properties such as optical thickness and spatial extent. Clouds play a critical role in moderating global temperatures and small perturbations can lead to significant amounts of cooling or warming. Uncertainty in this effect is so large it is not currently known if it is negligible, or provides a large enough cooling to largely negate present-day warming by CO2. This work uses deep convolutional neural networks to look for two particular perturbations in clouds due to anthropogenic aerosol and assess their properties and prevalence, providing valuable insights into their climatic effects.

Authors: Duncan Watson-Parris (University of Oxford); Sam Sutherland (University of Oxford); Matthew Christensen (University of Oxford); Anthony Caterini (University of Oxford); Dino Sejdinovic (University of Oxford); Philip Stier (University of Oxford)
ICML 2019 Learning Radiative Transfer Models for Climate Change Applications in Imaging Spectroscopy (Research Track)
Abstract and authors: (click to expand)

Abstract: According to a recent investigation, an estimated 33-50% of the world's coral reefs have undergone degradation, believed to be as a result of climate change. A strong driver of climate change and the subsequent environmental impact are greenhouse gases such as methane. However, the exact relation climate change has to the environmental condition cannot be easily established. Remote sensing methods are increasingly being used to quantify and draw connections between rapidly changing climatic conditions and environmental impact. A crucial part of this analysis is processing spectroscopy data using radiative transfer models (RTMs) which is a computationally expensive process and limits their use with high volume imaging spectrometers. This work presents an algorithm that can efficiently emulate RTMs using neural networks leading to a multifold speedup in processing time, and yielding multiple downstream benefits.

Authors: Shubhankar V Deshpande (Carnegie Mellon University), Brian D Bue (NASA JPL/Caltech), David R Thompson (NASA JPL/Caltech), Vijay Natraj (NASA JPL/Caltech), Mario Parente (UMass Amherst)