set the channel description: Share new conservation-related papers with the community
Interesting recent review paper: Tackling Climate Change with Machine Learning (https://arxiv.org/abs/1906.05433)
Bonnie and I are making a shorter memo summarizing the content. Will share if it turns out well
@Jennifer Marsman has joined the channel
@Elijah Cole (Deactivated) has joined the channel
@Lily Xu - OLD ACCOUNT, DO NOT MESSAGE has joined the channel
@Ștefan Istrate has joined the channel
@Ben Weinstein has joined the channel
@Riccardo Pressiani has joined the channel
@Thomas Starnes has joined the channel
and kenyan mammal population counts. https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210X.13277
We just published a paper about measuring bird migration in weather radar using ML. With @Subhransu Maji, @Garrett Bernstein, @Kevin Winner
https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210X.13280
Computational Sustainability faculty recently published a paper in the Communications of the ACM which was highlighted as the cover article:
-- Link to paper: https://cacm.acm.org/magazines/2019/9/238970-computational-sustainability/fulltext
-- Accompanying video: https://vimeo.com/351182289
New paper on Chimp facial recognition: https://advances.sciencemag.org/content/5/9/eaaw0736
This is the paper we recently published at KDD2019 in the Data Mining and AI for Conservation Workshop
A Framework for Identifying Group Behavior of Wild Animals: https://arxiv.org/abs/1907.00932
Activity recognition and, more generally, behavior inference tasks are gaining a lot of interest. Much of it is work in the context of human behavior. New available tracking technologies for wild animals are generating datasets that indirectly may provide information about animal behavior. In this work, we propose a method for classifying these data into behavioral annotation, particularly collective behavior of a social group. Our method is based on sequence analysis with a direct encoding of the interactions of a group of wild animals. We evaluate our approach on a real world dataset, showing significant accuracy improvements over baseline methods.
Our new paper is out this week on using aerial imagery to train a CNN to detect whales in high-res satellite imagery https://journals.plos.org/plosone/article?id=10.1371%2Fjournal.pone.0212532
The paper I presented at KDD is up on biorxiv now, https://www.biorxiv.org/content/10.1101/790071v1
dataset available for benchmarking (paper to come soon) https://github.com/weecology/NeonTreeEvaluation
High level review of deep learning in conservation.
https://www.cell.com/current-biology/fulltext/S0960-9822(19)31032-2
Our recent work "Long Term Temporal Context for Per-Camera Object Detection" is now up on arxiv! https://arxiv.org/abs/1912.03538
just downloaded it, reading it tonight.
https://www.sciencedirect.com/science/article/pii/S157495412030011X?via%3Dihub
https://www.nature.com/articles/s41598-020-57549-y
on contextual metadata for species classification
First NAS paper I've seen for wildlife applications: https://www.sciencedirect.com/science/article/abs/pii/S092523122030388X
We’re thrilled to announce the GeoLifeCLEF 2020 dataset!
- 1.9 million species observations, split between France and the US.
- Each species observation is paired with high-resolution remote sensing imagery, land cover, and altitude.
- Also includes traditional climate and soil covariates.
Paper: https://arxiv.org/abs/2004.04192
Competition: https://www.aicrowd.com/challenges/lifeclef-2020-geo
Interesting project from CMU's CREATE lab using computer vision to identify individual industrial smoke emissions.
project: https://smoke.createlab.org/
paper: https://arxiv.org/abs/2005.06111
https://ai.facebook.com/blog/dense-pose-for-animal-classes-using-transfer-learning
@Hannah Kerner @Tony Chang @John Brandt I emailed the authors to inquire what it would take to transform their annotations (raster segmentations) into bounding boxes to train a satellite tree model for that ecotype. In general I’m incredibly suspect on individual tree modeling at 50cm resolution, but that ecotype might be more effective. Ignore all the proclamations in the paper, they overpuffed it a touch to get into nature?
Hey Ben! Sorry for the late response. We work with Compton Tucker - the PI for this project - rather closely. I agree I think a bit of the results are overpuffed and selected in areas where the model performs well. They have said for almost 2 years that they'd share the lat, longs and shape files of tree segmentations with us, but it hasn't happened - and we're an official partner with their lab -- so I'm doubtful that you'll get much
yeah, it was quite untasteful I can be honest, I asked and they ignored me, responding to other parts of the email as if they hadn’t read the whole thing…
sigh - reaching out to Compton tucker this AM will let you know if we get any updates
Have you started using the planet imagery? Would love to chat about ideas. Been working on sentinel 1/2/planet fusion for like 2 days
The tucker data is here: https://daac.ornl.gov/cgi-bin/dsviewer.pl?ds_id=1832
what’s the fusion resolution, can you paste an image of what that ends up looking like?
i always feel like i download planet data, but it never looks quite as good as I hoped> i think they do alot of nice pan sharpening and postprocessing.
Here is planet, sentinel, and 10-m labels (white = tree, black = background) for 2 random areas in Kenya
its 4.77 meter imagery resampled to 5 meters for planet. Thinking of running my normal convLSTM on the sentinel time series, and then fusing in the planet imagery after a 2-stride conv layer to make it the same dimensions before it goes to the normal backbone network
Thank you so much for posting the link to polygon vectors! This is pretty cool. Just to be clear this isn't the ground truth annotations though. It's rather a producted as derived from their algorithm, right?
Yes that's true. Their ground truth data is from another paper... Martin Brandt 2018 in nature I think. hand annotations of 20,000 trees in the Sahel
Our first paper from the BearID Project: https://onlinelibrary.wiley.com/doi/10.1002/ece3.6840. We even got some coverage in the New York Times!
Just reread this paper, pretty cool work in adapting human pose to animals: https://gdude.de/densepose-evolution/
not new, but @Casey Youngflesh https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=9174973
super interesting paper on self-supervision, might be of use to @gvanhorn @Sarra Alqahtani https://arxiv.org/pdf/2011.09980.pdf
@Sarra Alqahtani has joined the channel
deep learning for visual pollen analysis, really useful for plant-pollinator work. https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.13575
I invited the author to join the channel and asked if they wanted to upload to lila. I don't know if that is something happening elsewhere for pollen images, but you can imagine a global database kinda thing elsewhere.
https://arxiv.org/abs/2104.05107
"Towards a Collective Agenda on AI for Earth Science Data Analysis"
Awesome new paper by Devis Tuia et al from EPFL. Takeaways are that the majority of work needed isn't in deep learning but instead in reasoning, human-centric and hybrid ML models, interpretability, and causality
https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210X.13608
Happy to announce our new paper,"Automated retrieval of information on threatened species from online sources using machine learning", which details the methodology to extract news articles on threatened species. An end‐to‐end pipeline that begins from searching and downloading news articles about species listed in Appendix I of (CITES) along with news articles from specific Twitter handles and proceeds with implementing natural language processing and machine learning methods to filter and retain only relevant articles while extracting named entities. We create a database of over 15k articles and adding more with time..
We released a short paper on the iWildCam competition for the FGVC Workshop: https://arxiv.org/abs/2105.03494
New paper - using vacation photos from Flickr to fill data gaps and understand the distribution of Antarctic seals
https://www.frontiersin.org/articles/10.3389/fmars.2021.645288/full
@Ben Weinstein @Elijah Cole (Deactivated) @Ben Augustine nice paper on model calibration: https://arxiv.org/abs/2106.07998
particularly section 4.3 "Accuracy and Calibration Under Distribution Shift"
does anyone have experience with this http://proceedings.mlr.press/v48/gal16.pdf I am reviewing a paper that leans hard on the idea that ensembling a bunch of randomly initialized models with dropout is a form of bayesian approximation. This seems fringe to me.
I haven’t read the paper, but it’s a popular idea and I hear that claim about bayesian approximation repeated often
it feels somewhat a stretch? No priors, no bayes rule, the resulting distributions are not posteriors in the sense that they are not well calibrated. I guess in terms of the way ecologists engage with bayesian models, the connection isn't obvious. But good to know others have seen it. I'll dial back my language in the review.
The appendix seems to have a lot of the details:
https://arxiv.org/abs/1506.02157
MC dropout is definitely closer to parametric bootstrapping than proper Bayes. I think the idea is that the weight initialization scheme functions as some kind of improper prior by biasing the optimizer toward different modes in the posterior (see https://arxiv.org/abs/1912.02757) However, the end result is definitely prone to miscalibration and strongly depends on the size of the network and dropout rate, which I think have been the subjects of quite a few follow up papers (e.g. https://arxiv.org/abs/2006.11584)
Thanks for the papers, this fits my expectation.
Can our datasets be of value to you? A few project updates
The University of Hong Kong has just developed an app called ‘Saving Face’ for i-id of napoleon wrasse:
https://onlinelibrary.wiley.com/doi/abs/10.1002/aqc.3199
https://www.scmp.com/magazines/post-magazine/short-reads/article/3134123/facial-recognition-app-used-protect-endangered
The team has developed other ML applications in conservation such as:
https://www.clearbot.org/
https://openoceancam.com/
Awesome new comprehensive review of sensors used for wildlife monitoring: https://academic.oup.com/bioscience/advance-article/doi/10.1093/biosci/biab073/6322306
Such an awesome paper!! I’ve been waiting for something like this to be published for years!
New paper "Becoming Good at AI for Good" by the Microsoft AI for Good team, including @Caleb Robinson, @Bistra Dilkina and @Siyu Yang!
https://arxiv.org/abs/2104.11757
hey channel! I'm going to the tropics for a bit and will have plenty of afternoons for reading. Please drop literally all possibly interesting papers in here! I'm sure i'm not alone in having some summer reading to do, so if everyone drops one paper, we can get a decent reading list going. Your own work counts too! Bird detector preprint is up. https://www.biorxiv.org/content/10.1101/2021.08.05.455311v1
Does anyone know of a paper studying temporal identification of individual objects in remote sensing. Given a set of images, performing object detection and then specifying that this object is the same among images given small changes in acquisition and appearance? The tracking literature (Like https://ieeexplore-ieee-org.lp.hscl.ufl.edu/abstract/document/9492311) all makes assumptions about that the object is actually moving and use things like optical flow. The image registration literature is concerned with shifting the entire image (like https://ieeexplore-ieee-org.lp.hscl.ufl.edu/abstract/document/8404075).
I don't but we've been thinking about this in the context of SSL for urban trees. Would be excited if someone knew of/could point to the right literature :)
if i feels like either i'm not using the right words, or its surprisingly understudied compared to change detection
sort of https://ietresearch.onlinelibrary.wiley.com/doi/10.1049/iet-cvi.2017.0261, but not really.
In terms of the comparison portion, similarity comparison networks might help.
https://arxiv.org/abs/1902.09324
So for clarification Ben, are you trying to determine whether objects in the same known position are the same object at different time points or two different objects?
which is distinguished from registration because the alignment is already known?
here is a picture. Here are two years of tree predictions shown at the same time. Its easy to do heuristically if you have just two years, but imagine we have 10 years. An ideal method using information from multiple years (like your camera trap paper), but also allows for new objects to appear. So the network would take in a series of images, and return object detections AND a 2nd head that had a unique code that represents an individual, similar to what @Stefan Schneider just shared.
it feels like this should have been done in the building detection world. Buildings change over time. This building in these 5 images is the same building.
Trees might change more over time than buildings?
You would also want to catch deaths, yeah? So IDs would also be able to dissapear
we think that 99% of the change among images is actually changes in angle/illumination
Basically sounds like an adaptation of multi-object tracking
certainly all that happens and is important, but if you look at naive predictions, the variance among years is almost entirely driven by superficial appearance.
right, with strong changes in superficial appearance, the signal is mostly in...trunk position? and positive instead of negative growth?
When does a simple IoU-based matching approach fail? Assuming the images are well-aligned?
ah, see that works for a small number of years, but the number of permutations is insane. Imagine 5 years, there are 5 choose 2 = 10 combinations of potential matches. Very quickly it becomes weird when tree A matches in year 4, 3, 1, but Tree B matches in year 5, 2.
it just starts to chain together trees to make super trees
i'm going to go read the animal identification paper.
hmmmm interesting. I think one thing that's different in this case from the animal re-ID is that here the stronger signal is the position of the object as opposed to the specific appearance. But possibly position, size, and time could be encoded into the similarity score in a cool way.
How good of a registration can you compute, year over year, with a proper orthorectification and features derived from the surrounding scenery? It seems like you should be able to get the reg good enough that the trees should almost completely overlay in their connection to the ground plane, so the trunks
For Python users & bioacoustics folks --
AP-10K: A Benchmark for Animal Pose Estimation in the Wild
Accurate animal pose estimation is an essential step towards understanding animal behavior, and can potentially benefit many downstream applications, such as wildlife conservation. Previous works only focus on specific animals while ignoring the diversity of animal species, limiting the generalization ability. In this paper, we propose AP-10K, the first large-scale benchmark for general animal pose estimation, to facilitate the research in animal pose estimation. AP-10K consists of 10,015 images collected and filtered from 23 animal families and 60 species following the taxonomic rank and high-quality keypoint annotations labeled and checked manually. Based on AP-10K, we benchmark representative pose estimation models on the following three tracks: (1) supervised learning for animal pose estimation, (2) cross-domain transfer learning from human pose estimation to animal pose estimation, and (3) intra- and inter-family domain generalization for unseen animals. The experimental results provide sound empirical evidence on the superiority of learning from diverse animals species in terms of both accuracy and generalization ability. It opens new directions for facilitating future research in animal pose estimation.
https://github.com/AlexTheBad/AP-10K
@Sara Beery https://www.corp.at/archive/CORP2021_143.pdf
Hello everyone! I am a sixth-year PhD student from UC Berkeley. We have just published a paper on Nature Machine Intelligence about deployable wildlife AI recognition systems with efficient humans in the loop and imperfect models! Please check it out if you are interested! This project is moving towards practical deployment in Africa. We hope this work can become an important step to the AI solutions for real-world conservation problems with imperfect AI models. I am very happy to answer any questions you have! Here is the link to the journal webpage (with a preprint link incorporated in case you don't have subscriptions): https://www.nature.com/articles/s42256-021-00393-0 Thank you very much!
in the open long tailed framework (cited in your paper), how does one decide what is too far away in the clustered feature space? Looking at this figure, which relates to your work. How to decide when to call in a human? How sensitive is the performance to this threshold, seems like a really tough call about when a sample is novel enough.
Hello @Ben Weinstein. Thanks for asking! For OLTR, it is set up with a threshold as a hyperparameter to determine known v.s. novel. In terms of sensitivity, it was measured by a modified F1 score (originally proposed in ODIN) reported in the results section with the same threshold across all compared baselines. It is the same process in the iterative wildlife recognition framework. The only difference is we now don't focus to much on novel samples, but difficult samples as well because the purpose is not solely out of distribution detection, but efficient human intervention. The sensitivity is measured as percentage of high confidence and low confidence prediction in both known and unknown classes on validation or second period training data, which we think is more intuitive. Does that make sense?
ya. it does make sense, i'm just trying to think about how to come up with a first estimate for what the value of that hyperparameter should be. There are no units in that dimensional space. I get how to evaluate it. In practice did you just search for an optimal value by hand?
Oh, I see. Yes, we did some search with validation data for the iterative model. But for OLTR, it was just based on past experience where 0.2 gave a relatively good performance. And it is definitely not the optimal value. We only use it to compare with other methods.
thanks, this is something I will be investigating, we have a similar use case with trees. 192 species with atleast 1 sample, but only maybe 80 species with more than 10 samples. Are there other foundational papers on the meta-learning/metric learning/joint learning + clustering that you find helpful?
Yes, if you are looking for long-tail related papers. LDAM (https://arxiv.org/pdf/1906.07413.pdf), DecouplingNetwork (https://arxiv.org/abs/1910.09217), BBN (https://arxiv.org/abs/1912.02413), and RIDE (https://arxiv.org/abs/2010.01809) are the four highly discussed follow ups after our OLTR. RIDE is currently the state of the art method but relatively more complicated.
Great thanks. I'll let you know, we have plenty of data if this the kind of thing you/others want to get involved with. Collaborative group here at UF.
Great! I can also answer any question you have!
"A Comprehensive Overview of Technologies for Species and Habitat Monitoring and Conservation"
https://academic.oup.com/bioscience/article/71/10/1038/6322306 October 2021
“Perspectives in Machine learning for Wildlife Conservation”, fresh from the press in Nature Communication. In our co-authors’ words “our call to action” 🙂 https://www.nature.com/articles/s41467-022-27980-y Many of the people in the group have contributed to this paper, for instance @Sara Beery @Benjamin Kellenberger @Silvia Zuffi @Holger Klinck @gvanhorn @Tanya Berger-Wolf and many others !
"Coding for Life: Designing a Platform for Projecting and Protecting Global Biodiversity" https://doi.org/10.1093/biosci/biab099 (including (brief) mention of deep RL methods). I wasn't involved in this, but I thought this was a compelling pitch, notably coming from a mostly or entirely ecology-based team, led by a field ecologist.
What's the name of that platform they're coding? do they have a github repo?
Thanks for pointing this; looks useful; but it is behind a paywall .. surprisingly not in OA.
Interesting use of 3D depth from stereo pair camera traps, combined with optic flow: generates 3D bounding boxes/detection of animals in the wild (May 2022 issue): https://doi.org/10.1016/j.ecoinf.2021.101535
https://www.nature.com/articles/s41893-022-00851-6
https://www.nature.com/articles/s41559-022-01702-5
Which mammals can be identified from camera traps and crowdsourced photographs? (Roland Kays et al)
While museum voucher specimens continue to be the standard for species identifications, biodiversity data are increasingly represented by photographic records from camera traps and amateur naturalists. Some species are easily recognized in these pictures, others are impossible to distinguish. Here we quantify the extent to which 335 terrestrial nonvolant North American mammals can be identified in typical photographs, with and without considering species range maps. We evaluated all pairwise comparisons of species and judged, based on professional opinion, whether they are visually distinguishable in typical pictures from camera traps or the iNaturalist crowdsourced platform on a 4-point scale: (1) always, (2) usually, (3) rarely, or (4) never. Most (96.5%) of the 55,944 pairwise comparisons were ranked as always or usually distinguishable in a photograph, leaving exactly 2,000 pairs of species that can rarely or never be distinguished from typical pictures, primarily within clades such as shrews and small-bodied rodents. Accounting for a species geographic range eliminates many problematic comparisons, such that the average number of difficult or impossible-to-distinguish species pairs from any location was 7.3 when considering all species, or 0.37 when considering only those typically surveyed with camera traps. The greatest diversity of difficult-to-distinguish species was in Arizona and New Mexico, with 57 difficult pairs of species, suggesting the problem scales with overall species diversity. Our results show which species are most readily differentiated by photographic data and which taxa should be identified only to higher taxonomic levels (e.g., genus). Our results are relevant to ecologists, as well as those using artificial intelligence to identify species in photographs, but also serve as a reminder that continued study of mammals through museum vouchers is critical since it is the only way to accurately identify many smaller species, provides a wealth of data unattainable from photographs, and constrains photographic records via accurate range maps. Ongoing specimen voucher collection, in addition to photographs, will become even more important as species ranges change, and photographic evidence alone will not be sufficient to document these dynamics for many species.
https://academic.oup.com/jmammal/advance-article-abstract/doi/10.1093/jmammal/gyac021/6564439?redirectedFrom=fulltext
New paper from the BearID Project: https://doi.org/10.1007/s42991-021-00168-5
Awesome collective effort led by @Noa Rigoudy @Simon Chamaillé & @Vincent Miele CNRS The DeepFaune initiative: a collaborative effort towards the automatic identification of the French fauna in camera-trap images https://www.biorxiv.org/content/10.1101/2022.03.15.484324v1
Thanks for the pointer @Nico Lang ! https://doi.org/10.1016/j.ecoinf.2022.101641
Shameless self-promotion 😇
"Trade-off between deep learning for species identification and inference about predator-prey co-occurrence: Reproducible R workflow integrating models in computer vision and ecological statistics"
published in Computo a new diamond open access journal in statistics and machine learning 🤩
https://computo.sfds.asso.fr/published-202204-deeplearning-occupancy-lynx/
Just spotted in PNAS, Machine learning in evolutionary studies comes of age, attached.
Hi all! I’m currently an undergraduate student working on landscape connectivity research using Rad seq data from the Northern Red Legged Frog. I don’t have a computer science background but I came across some interesting papers and talks that lead me here. Specifically this paper and one of the papers that it sites about Hyla squirella. The work I’m doing currently does not use machine learning but I am extremely interested in how it could be applied. Not only to this current project but more broadly in ecological research. I’m hoping someone here with more experience and insight might be able to shed some light on these papers or at least give an opinion about the methods used, and their soundness or lack thereof.
https://arxiv.org/abs/2205.11324
"Towards automatic detection of wildlife trade using machine vision models"
excited to share our new paper coming out soon.. we compare popular machine vision models on a new dataset to identify if a mammal is in a wild surrounding or is a captive individual. the aim is to develop image analysis techniques to monitor wildlife trade (sale of exotic species). we found interesting results by training specifically for negative features and testing on out of distribution test data.
@Sara Beery a very late follow up on an old thread where i asked if watermarks affect results.. which i cant find on slack now.. (does the model use watermarks as features).. the short answer is yes, as expected.. i put an example in figure S3 of the supplementary doc.. so i guess important to mitigate watermark effects in classification..
(also, congratulations on your Ph.D. ! )
Ah cool!! "Yes, as expected"... yeah it really does cheat however it can huh
“Smart Oceans: Artificial intelligence and marine protected area governance”
sciencedirect.com/science/article/pii/S2589811622000106
“Towards Continuous Streamflow Monitoring with Time-Lapse Cameras and Deep Learning”, some recent work with my inimitable co-authors @Tony Chang, @Jeff Walker, and @Ben Letcher! Thanks also to @Caleb Robinson and @Bistra Dilkina for feedback on early drafts and to @Dan Morris for championing this effort!
I will be presenting this paper at COMPASS in Seattle next week--if any of you will be at the conference, please come say hello!! 🙂
Hi Carly, there is a tentative DOI in the PDF, but the online version of the paper will become available a few weeks (months?) after the conference. We might also put this on arXiv, in which case I'll DM you back with a link.
The online version came out before I expected! Here's a working link, for any who are interested: https://doi.org/10.1145/3530190.3534805
Recently published in OA: a solution for camera traps not relying on PIR, developed for an RPi base, using available motion vectors from the video standard format. Code base is available. https://www.sciencedirect.com/science/article/pii/S1574954122001066
Hello, we've proposed a very simple pipeline for re-identification of Giant Sunfish. It's based on keypoint matching using SuperPoint and SuperGlue and it works quite well without the need of any training or finetuning. We have not tried this on other animals, but we expect it to work on animals with distinct and unique patterns.
Example of usecase: we have helped marine researchers to annotate a large database of Sunfish images by providing them the top-X ranked matches simply based on the number of matched keypoints.
Pedersen, M., Haurum, J.B., Moeslund, T.B. and Nyegaard, M., 2022, March. Re-Identification of Giant Sunfish using Keypoint Matching. In Proceedings of the Northern Lights Deep Learning Workshop (Vol. 3).
https://www.eludamos.org/index.php/nldl/article/view/6234
Very nice. I would like to ask whether the dataset is public.
Hi Lukas, unfortunately not. We have only used a small portion (~90 images and ~30 IDs) of the Match My Mola dataset in the paper (simply because it was the only images that had been manually re-identified by the marine researchers at the time).
We are working on making the entire dataset (several thousand images) publicly available, but we are not done manually verifying the annotations yet and we also need to gather licenses to the images, which are most often provided by tourist divers.
I will drop a post if we get the dataset online (I cross my fingers), but it will probably not be until the end of this year or beginning of the next.
Perfect. Please write me a message if you manage to make it public. Thanks 🙂
Hi everyone, we're excited to introduce The Caltech Fish Counting Dataset (CFC): A Benchmark for Multiple-Object Tracking and Counting, to appear at ECCV 2022!
CFC is a large-scale dataset for detecting, tracking, and counting fish in sonar videos, an important conservation application and a rich data source for advancing low signal-to-noise computer vision applications and tackling domain generalization in tracking and counting.
Sonar video is used to monitor salmon during their seasonal migration, informing sustainable fisheries management. Manually counting fish in these videos requires significant effort. Automation using computer vision could reduce this burden and help scale up monitoring efforts.
To enable this, we collected over half a million annotations in over 1,500 videos sourced from seven different sonar cameras, allowing researchers to train multiple-object tracking and counting algorithms and evaluate generalization performance at unseen test locations.
We encourage others to check out the dataset (linked below) and join us in tackling these challenges! Work done with an amazing team: @Peter Kulits @Suzanne Stathatos @Tiffany Deng @Erik Young @Sara Beery @gvanhorn @Pietro Perona
Data and code: https://github.com/visipedia/caltech-fish-counting
Paper: https://arxiv.org/abs/2207.09295
@Dan Morris @Sara Beery https://arxiv.org/pdf/2208.10607.pdf with data https://github.com/jonathanventura/urban-tree-detection-data to be added to lila.science
Added to the list here:
https://lila.science/otherdatasets#images-plants
LMK if you know of other datasets that belong on that list.
also @Dan Morris, old dataset, but actually a really interesting paper http://bird.nae-lab.org/cattle/
Thanks! Added to:
https://lila.science/otherdatasets#images-domestic-animals
https://www.deepmind.com/blog/advancing-conservation-with-ai-based-facial-recognition-of-turtles
Just because they are too polite to post it themselves, really interesting paper from @Devis Tuia group out this week. Trying to build explicit biological knowledge into black box models. Out of curiosity, are there tree stems (x,y point locations) associated with the NFI plot data, or just land-cover classifications?
Hi Ben, thanks for sharing the paper, and glad that you like it! Regarding the NFI plot data, the Swiss NFI team provided me with land-cover classification data only. I've heard that internally, they have a dataset of geo-located trees, but to my knowledge it is not openly available. More info on the Swiss NFI here: https://www.lfi.ch/publikationen/publ/ergebnisberichte/lfi4.php (some documents are only in French or German unfortunately).
Fish dataset for object detection http://metadata.nmdc.no/metadata-api/landingpage/01d102345aef4639f063a13ea20cd3f3 https://rmets.onlinelibrary.wiley.com/doi/10.1002/gdj3.114
Thanks! Added to:
https://lila.science/otherdatasets#images-marine
The community is getting really close to a critical mass of public data for training a general-purpose fish detector, and I'm seeing a decent amount of demand for that.
Now, I know someone would still have to actually train that detector, and validate that it's useful for anyone's real scenario, yadda yadda, deep learning, yadda yadda.
But can we get to the important stuff in advance? Specifically, can we get the great minds at DrivenData - legendary pun-creators of the AI for Conservation community - to start naming the hypothetical fish detector that no one has built yet?
@Greg Lipstein @Peter Bull
I feel like I've read 10 papers titled FishNet
Including my personal favorite, this one that has nothing to do with wildlife it just kinda looks like a fish
Image generators that design image generators that look like images is the next frontier in deep learning.
This seems like a good place to bring up that a long term goal for our lab in Florida is a unified animal detector across remote imagery, fish/bird/mammal. I'm very interested in opinions whether we think there are inherent tradeoffs in trying to bring a diverse group of imagery together. Whether camera trap and airborne imagery can live in the same model. Whether training on birds means we are likely to get poorer features for fish, etc. Its one of the reasons I keep sending datasets to LILA is that along with @Benjamin Kellenberger I see the future as running though a system like AIDE for general screening and then using these features as a starting point for species classification models. We took a baby step https://www.sciencedirect.com/science/article/pii/S157495412030011X, looking at training models across different forests and did not find any tradeoffs, i.e learning trees in california did not make a new york tree model worse.
I would bet $20 that [camera trap data] and [fish data from camera-trap-like-angles] will get the best overall results from living in separate models, and I would bet $50 that either of those will get the best overall results from living in separate models from airborne imagery, but I wouldn't bet $100 on any of those hypotheses.
interesting. I think the question becomes what kind of tradeoffs are involved. I think it would be a huge benefit to the community to have such a starting point, and might be worth a few percentage accuracy. If its 20% accuracy then no, that's not worth it.
or whether an airborne only perspective is useful just as a separate model
the other piece to this, that i've wanted to talk to @gvanhorn is whether highly curated images, such as the millions at the Macaulay Library have role in training these broad models, atleast within birds. Either through synthetically creating airborne data from these images, or in teaching the model what a side on/high quality view looks like. I've been looking at airborne images from @Brad Pickens and others from fish and wildlife and thinking about those side-on data from Cornell.
Specifically, can we get the great minds at DrivenData - legendary pun-creators of the AI for Conservation community - to start naming the hypothetical fish detector that no one has built yet?@Dan Morris Was pretty proud of this one! That said, I started thinking something like “icth-ference” but that is maybe a little unwieldy. We’ll need to workshop it….
@Ben Weinstein Due to the huge differences between the aquatic and terrestrial domains I think a single model to "solve" it all is not feasible (or not the best option, at least). The attenuation of light, backscatter, refraction, and turbidity (when you are under water) combined with different camera sensors, image compression, a ton of different species etc. etc. is just a huge mouthful. Another solution could be to have a single model that analyses the environment of the image/video and classifies whether it is, e.g., 1) clear underwater 2) air-borne terrestrial 3) turbid underwater 4) bla bla... and then you can use that to automatically choose which model is best suited to use. Like a cascade of small models, which can be individually fine-tuned and adjusted. IMO this could lead to easier maintenance and debugging and probably much better results. Just my thoughts 👍
@Malte Pedersen I think this is where foundational models should be useful. Train an unsupervised backbone model on a huge corpus of remotely sensed imagery (e.g. camera trap, aerial, etc) - views/images that are not generally in common large image datasets that CV researchers use. And then use those models as strong feature extractors for few-shot learning on target tasks.
This is especially a problem for satellite data because of domain shift between instruments. Models often don't transfer well even between similar satellites e.g. Sentinel 2 -> Landsat. Terrestrial based models seem to do better, but maybe we're taking advantage of millions of individual cameras in training corpora like OpenImages/ImageNet.
Problem is usually these models are expensive to train and curate - you need huge amounts of data and compute, but at least once someone's done it once it should be reusable. And you'd probably need a suite of models for generic applications like 'horizontal' camera trap, airborne, satellite, etc. That ought to be a better approach than everyone fine-tuning networks that are at some point derived from ImageNet and experimentally it seems to work well.
https://arxiv.org/abs/2108.07258
There is some critical mass of data and modalities where throwing everything into a pot does produce some incredible results. The extreme example of this are the large foundational models like CLIP, etc... which actually work incredibly well on a huge number of domains. We've played with CLIP for ground imagery, aerial, satellite, multiple modalities (EO, IR, SAR), etc... and it just works. This is trained on something like a billion text-image pairs though. Im not sure anyone has a great heuristic (definitely not a formula) for saying where that magic line is, above which you can throw everything into a pot and outperform models trained on very specific tasks, e.g. a small set of animals in a single domain. Practically all the successful applications Ive seen try to limit the ontology of what the network has to learn to be as narrow as it can be and still add value
@Josh Veitch-Michaelis as you also mention there are many pitfalls and uncertainties regarding the behavior and nature of foundation models. You loose a lot of control and it is extremely difficult to explain the decisions made by such large and diverse models (even more difficult than regular DL models, which are already unexplainable for most (all?) experts). In my opinion, explainability is extremely important when it comes to decisions that actively affect humans and/or animals. That said, I completely agree that foundation models could potentially be extremely useful for this application. Very interesting!
@Benjamin Kellenberger has joined the channel
New paper from @Matt Allen - "Tree species classification from complex laser scanning data in Mediterranean forests using deep learning" https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210X.13981
https://www.sciencedirect.com/science/article/pii/S2352938522001446 not an AI paper, but about 7% difference between elephant seals in satellite and drone in the Antarctic, @Heather Lynch @Casey Youngflesh @Devis Tuia and anyone else working on seals/pinnipeds
not necessarily AI-specific, but cool none the less with lots of potential applications for AI - https://www.nature.com/articles/s41467-022-33223-x
This is an interesting application of super resolution https://openaccess.thecvf.com/content/WACV2022W/RWS/papers/Xue_Small_or_Far_Away_Exploi[…]ep_Super-Resolution_and_Altitude_Data_WACVW_2022_paper.pdf
https://arxiv.org/abs/2210.00889
@Vienna Saccomanno has a cool paper on drones and kelp. https://zslpublications.onlinelibrary.wiley.com/doi/10.1002/rse2.295
@Vienna Saccomanno has joined the channel
Good morning! Peter Bermant, @Leah Brickson, and I just released a new paper on bioRxiv that we'd like to share https://doi.org/10.1101/2022.10.12.511740. We'd love your feedback!
Looking at this now, can you make the abstract clearer about the performance gain versus supervised learning
respectively, and an area under the Precision-Recall curve (PR-AUC) of 0.917, while a threshold detector based on signal energy amplitude returns a maximal R-value and F1-score of 0.620 and 0.576, respectively, and a PR-AUC of 0.571I think the 'threshold detector' is supervised? or is that unsupervised? Only read the abstract.
I just skimmed the paper, I think this kind of work is really important and provides massive potential benefits over the very old and outdated stuff I published a few years ago https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.13011. Couple thoughts on very first glance. Can you provide a concrete comparison to supervised data? Basically how many sperm whale calls would you need to annotate to get a similar performance? It's great that unsupervised works at all, which is welcome news, just wondering about its efficiency versus supervised. Second thought, is how universal do you think those feature backbones are? Imagine we collected petabytes of biodiversity video data and wanted to make a general animal detector. Could a single backbone be used, or do you think the contrastive approach is not generalizable? We have similar goals in mining background features for airborne object detection and hundreds of TB of imagery from National Ecological Observatory Network. @Alexandre Lacoste
@Ben Weinstein to your second question, we think they will be quite generalizable, which is why we show preliminary data in different modalities. A full characterization of generalizability was outside the scope of this one paper, but we found that any discontinuity is possible to generalize to. Of course we'd need to do the experiments, but would love to know if you are applying it in your modality, or are interested in chatting about how it would apply.
To your first point, again great feedback and I'll take it back to my team.
@Alexandre Lacoste has joined the channel
Big shoutout to @Sara Beery for this wonderful paper - Species Distribution Modeling for Machine Learning Practitioners: A Review. As a computer scientist newly working on species distribution modeling (SDM), this was such a helpful review of topics and it elegantly framed some the challenges in this field like “what the heck is ground truth here?“, etc. Not a new paper, but highly recommended reading for those new to SDMs 🥳
Shout out to @Elijah Cole (Deactivated) as well!! This was very much a joint effort
Nice paper, Sara! SDM's also have a history of tackling issues directly relevant to deep learning, such as problems with the random split https://onlinelibrary.wiley.com/doi/abs/10.1111/geb.12684 ; I'm going in the opposite direction...moving from SDM to deep learning, but always happy to discuss SDMs.
On the counting of migratory Sandhill Cranes with thermal imagery and deep learning: huge thanks to William and Mary and @Emilio Luz-Ricca for this research regarding a globally important stopover area for this species https://zslpublications.onlinelibrary.wiley.com/doi/full/10.1002/rse2.301
This is a somewhat random tidbit, but might be useful to people here!
https://twitter.com/eli_cole_/status/1583119979804704768
Twitter
https://ieeexplore.ieee.org/abstract/document/9918242
Today at ECCV we’re presenting our paper on using label hierarchies to improve object localization. We also released a new iNat dataset with manually verified bounding boxes. Drop by if you’re at the conference! 🙂
https://twitter.com/eli_cole_/status/1584846750787174400
Twitter
https://arxiv.org/abs/2210.12300
I’m excited to introduce AVES, aka “BERT for animals”, a self-supervised audio encoder model which beats strong baselines including VGGish etc. across the board in bioacoustics tasks. https://twitter.com/mhagiwara/status/1585496748298219520
Twitter
Twitter
https://www.sciencedirect.com/science/article/pii/S1574954122003260
https://www.sciencedirect.com/science/article/pii/S1569843222002473
https://www.mdpi.com/2313-433X/8/4/96
https://www.biorxiv.org/content/10.1101/2022.10.28.513761v1
Virtual scene learning for tree detection. https://arxiv.org/pdf/2210.17424.pdf
Very cool! Looks like the dataset is here:
https://github.com/norlab-ulaval/PercepTreeV1
I'll add a link on the LILA "other data sets" page.
@Zhongqi Miao https://www.sciencedirect.com/science/article/pii/S1574954122003260
https://www.sciencedirect.com/science/article/pii/S1470160X22010949
🌊🐬 🚤 📷➡️ 🖥️ 🔍
Excited to present our recent work, Towards Automatic Cetacean Photo-Identification: A Framework for Fine-Grain, Few-Shot Learning in Marine Ecology, which highlights a system for automated individual cetacean photo-id through the use of a pipeline of computer vision models and post-processing methodologies. The work will be presented later this month at the IEEE Big Data 2022 Conference in Osaka, Japan though the pre-print is now available on arXiv
Hi all. We have just uploaded a pre-print discussing our new deep-learning based method for bat echolocation call detection and species classification. Feel free to drop me a message if you have any thoughts/questions.
🦇🔊💻
https://www.biorxiv.org/content/10.1101/2022.12.14.520490v1
Hi All,
Preprint on learning Great Ape behavioural actions using camera-trap footage with @Otto Brookes, @Tilo Burghardt and Hjalmar Kuhl:
https://deepai.org/publication/triple-stream-deep-metric-learning-of-great-ape-behavioural-actions
Open Access to:
Dynamic Curriculum Learning for Great Ape Detection in the Wild,
Yang, X., Burghardt, T. & Mirmehdi, M.
International Journal of Computer Vision (2023).
https://link.springer.com/article/10.1007/s11263-023-01748-3
Hey Majid! A bit late to the show, but I've been reading this paper preparing to talk about it in a small group meeting. It's very cool!! I was wondering if I could steal maybe half an hour at some point to discuss about it shortly?
Fuentes, M., Van Doren, B. M., Fink, D., & Sheldon, D. (2023). BirdFlow: Learning seasonal bird movements from eBird data. Methods in Ecology and Evolution, 00, 1– 16. https://doi.org/10.1111/2041-210X.14052
https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.14052
hi all, introducing our recently published paper, where we tested a bunch or popular architectures on newly created data to differentiate images of captive mammals from wild ones, aimed at monitoring illegal wildlife trade. Some insights on specifically including negative features while training, and the importance of testing on out of distribution data.
https://www.sciencedirect.com/science/article/pii/S0006320723000241
Has anyone else tried drones + bird microphone (@gvanhorn @Tessa Rhinehart @Dan Stowell)https://cdnsciencepub.com/doi/pdf/10.1139/dsa-2022-0015
Hi folks,
I get asked to review dozens of papers using machine learning for camera traps, or other wildlife-related studies, every year and I cannot do them all. If you are in this field and you are interested in (and qualified for) reviewing this type of paper, please let me know. I’d appreciate your name and contact details. Peer review is a great service to the scientific community and it can look really good on your CV.
Thanks,
Mikey
I would love this list as well, for the same reasons.
(i think the answer is that everyone is)
Beware: If you post your information here, you will get all of us passing reviews to you!
question @Mikey Tabak about this (a challenge I've faced with reviews), do they allow senior PhD students/early postdocs to be reviewers, and if so does the review have to be signed by their PI?
Andrew, grad students are able to review papers. If you are unexperienced, it might be good to get some guidance from someone with more experience. It does not need to be signed by the PI, though. If you are concerned and want to involve your PI, it could be something to bring up with the editor when the contact you about reviewing
so I'm more of speaking from experience with where I commonly review - exp: Journal of Experimental Biology there is a ~soft requirement to be at least a postdoc to review.
there is a list in behavioral biology for this exact reason when folks need people that have some time to review, maybe could make like a google doc with those that have time in the coming weeks/months to be a reviewer and could fill in how many papers they have the availability for? Could allow more early career follks to sign up as well?
Good idea! Some journals can have specific requirements, but it is not essential in general. I think, at this point, editors are so desperate they would not be too restrictive in their searches for reviewers
Excited to announce our new Research Methods Guide on using drones and deep learning to observe the movement and behavior of animal groups in the context of their natural environments! https://besjournals.onlinelibrary.wiley.com/doi/10.1111/1365-2656.13904
See Twitter for more info! https://mobile.twitter.com/ben_koger/status/1638635520816476160
Twitter
Are the training data available there? @Dan Morris. It would nice for us to check how we are doing at doing object detection for novel classes like zebras and other large mammals.
Yes, it should be here: https://edmond.mpdl.mpg.de/dataset.xhtml?persistentId=doi:10.17617/3.EMRZGH (Sorry for the giant download size. If you have recommendation for a better place to store just the annotated dataset let me know and I can put them there.)
yeah, that's why i pinged @Dan Morris before, its the first place is on the archive he maintains.
What would you say is a maintain pain points for the object detection + drone parts of the workflow, not the tracking. We are always trying to gauge what we should be working on. 1) active learning/integrated annotation frameworks, 2) managing large geospatial tiles, for example cutting them, storing them and retrieving them, 3) getting enough data, starting from relevant backbones, 4) prediction at scale, 5) visualizing results, 6) anything else.
I think especially for the ecologists who want to start using this stuff, the biggest hurdle is still mostly around point 1: helping people as painlessly as possible go from "I have a bunch of drone videos where the animals are pretty easy to see" to a usable trained model. I think a lot of initial projects are still pretty small scale footage-wise so some of the other stuff is less of a bottleneck.
so that might look like, take in a video, process into images, use a baseline model to identify potential crops, show those crops to user for annotation, maybe look them up against any data we have, or potentially resources like inaturalist, use those annotations to train a model, and save it somewhere online for them to point from a url (@Sara Beery for our meeting).
This looks like a really interesting dataset! I downloaded it and explored a little, the videos are pretty mesmerizing.
I'd like to add it to LILA's list of other data sets: https://lila.science/otherdatasets. @Ben Koger, is there a summary of the dataset, i.e., how many annotated bounding boxes on how many species? Basically: is there a README?
In terms of giving it a (new) permanent home, I don't usually re-host something on LILA when it already has a definitive place to live, and the mpg.de page looks like it's a good place. The download isn't too big a deal, although it would be a lot faster if it were hosted directly on a cloud bucket. So I'm not opposed to re-hosting on LILA. Ben K, if that's of interest to you, drop me an email at info@lila.science . These are the most analogous datasets hosted on LILA right now:
https://lila.science/datasets/aerial-seabirds-west-africa/
https://lila.science/datasets/conservationdrones
@Ben Koger @Blair Costelloe I'm also very interested in more of a breakdown of the data! What human detection and keypoint annotations were collected, across what number of flights, what locations/regions, what species, what timestamps...? I don't see information like this in the supplementary, though you do have a great discussion of how careful this selection process should be, let me know if I've missed it!
Right now the best info we have in the text is Table 1 in the supplement which breaks down the annotated data by species and has info about bounding box size and our trained model performance on the different species classes in our test set. @Sara Beery we currently don't list the additional information you mention which I agree would be very useful. I'm working now to write down some of that additional information. Ultimately I think it will make sense to include the stand alone annotated dataset with clearer and more comprehensive metadata on LILA (@Dan Morris, I'll email info@lila.science when I get things put together)
Very cool, and I'm especially interested in the dataset! I downloaded it and tinkered a little, the annotations are straightforward and - most importantly - the videos are really fun to stare at. I added to the list of marine image datasets here:
https://lila.science/otherdatasets#images-marine
New segmentation dataset/model/paper from meta that looks very helpful at a minimum for annotation but probably for much more: https://ai.facebook.com/blog/segment-anything-foundation-model-image-segmentation/
This is super cool. It seems like a nicely packaged assist tool for annotation (I've not tried text-guided segmentation yet). The big model does work on my M1 mac CPU (I didn't try mps) but it's pretty heavy so you definitely want a gpu to play with it. Seems to work "ok" on satellite images and at least as good as any commercial auto-segment too I've tried.
https://www.biorxiv.org/content/10.1101/2023.04.07.535991v1
provocative but fun paper about some of the possible risks to AI-assisted governance, potentially relevant to AI conservation: https://journals.sagepub.com/doi/full/10.1177/20539517231164119
neat, thanks! Anyone know if there’s an ai for conservation reading club or something similar?
This sounds interesting:
Empowering Wildlife Guardians: An Equitable Digital Stewardship and Reward System for Biodiversity Conservation using Deep Learning and 3/4G Camera Traps (https://arxiv.org/abs/2304.12703).
From the abstract:
"""
This paper proposes a radical new solution based on "Interspecies Money," where animals own their own money. Creating a digital twin for each species allows animals to dispense funds to their guardians for the services they provide. For example, a rhinoceros may release a payment to its guardian each time it is detected in a camera trap as long as it remains alive and well. To test the efficacy of this approach 27 camera traps were deployed over a 400km2 area in Welgevonden Game Reserve in Limpopo Province in South Africa. The motion-triggered camera traps were operational for ten months and, using deep learning, we managed to capture images of 12 distinct animal species. For each species, a makeshift bank account was set up and credited with £100. Each time an animal was captured in a camera and successfully classified, 1 penny (an arbitrary amount - mechanisms still need to be developed to determine the real value of species) was transferred from the animal account to its associated guardian.
"""
Thank you for sharing. It's interesting.
It reminds me of a conversation with Ale (the cofounder) from LAMAVE https://www.lamave.org/ about how there was this one specific family that was selling turtle eggs on the market and they'd get 4 pesos for it. So then the turtle nurseries offered them 6 pesos, to make sure they'd drop the eggs at the nursery instead of the market. But if the market paid 8 pesos ...
He mentioned that it's important to create some kind of inherent value about the biodiveristy around us (often traditionally it comes in the form of myths and stories), because if there is a clear monetary value, then it's very "easy "to shift to whoever who will pay more.
I am curious about where biodiversity and money intersect.
And curious to hear about what you think about the paper too.
This seems like a useful tool for bio-mass estimation. The pipeline includes the DINOv2 and segment-anything model. RGB airborne images have been trained along with LiDAR images of the same geo-referenced location for identifying the canopy height with sub-meter resolution.
https://research.facebook.com/blog/2023/4/every-tree-counts-large-scale-mapping-of-canopy-height-at-the-resolution-of-individual-trees/
Check out our new paper on a system analysis and review of the challenges and flexibility of using deep learning techniques to real-world imagery datasets, featuring a case study on overhead bird recognition:
Challenges and solutions for automated avian recognition in aerial imagery
https://zslpublications.onlinelibrary.wiley.com/doi/full/10.1002/rse2.318
You know how into aerial bird survey data I am right now... 🙂 Is any of the training data from this paper available?
@Dan Morris we did publish some data for this project, but only cropped and resized instances. The full overhead imagery are still not published and I think it is determined by USGS. If you are still interested in the cropped instances, there is a link to the data in the paper!
@Dan Morris, i'm working with these teams, if you need data, let me know too.
I don't "need" data, I'm just sort of in the mode of hoarding data. 🙂 I'll let you know if I want to dig deeper, and @Zhongqi Miao, I will check out the crops. Thanks!
Jokes apart, great work @Zhongqi Miao! if someone needs some more data, we had this paper out some time ago and the images we got were a good starting point for us in avian recognition: https://zslpublications.onlinelibrary.wiley.com/doi/pdf/10.1002/rse2.200
@Zhongqi Miao Thanks, I downloaded the crops; the dataset is very clear and well-organized. I added the dataset to this list:
https://lila.science/otherdatasets#images-terrestrial-animals
Thank you very much for adding it to lila!
I enjoyed https://proceedings.neurips.cc/paper_files/paper/2022/file/43119db5d59f07cc08fca7ba6820179a-Paper-Datasets_and_Benchmarks.pdf. Existing computer vision methods have a hard time capturing temporal shifts in classes and predictors. This is really important for wildlife surveys that may capture animal migration or phenology. Does anyone have experience on trying to influence predictors with image metadata. Somewhat connection to older paper, but time instead of space, as paper by @Oisin Mac Aodha https://openaccess.thecvf.com/content_ICCV_2019/html/Aodha_Presence-Only_Geographical_P[…]for_Fine-Grained_Image_Classification_ICCV_2019_paper.html).
We've been looking at some of these issues in fish detection and tracking in sonar with @Justin Kay @Suzanne Stathatos and @gvanhorn
A followup paper is here https://openaccess.thecvf.com/content/CVPR2022/papers/Yang_Dynamic_MLP_for_Fine-Grained[…]ication_by_Leveraging_Geographical_and_CVPR_2022_paper.pdf. Reviewing different adaptive strategies for additional data temporal integration. Gets us closer to the seasonality question, since similar timestamps may be non-linear and non-consecutive, as with migrant birds returning in spring and fall. Bias from temporal imbalance in training data is lurking in all of this.
https://arxiv.org/pdf/2304.06306.pdf is a nice model on fusing large language model text and vision features. One thing i've been brewing on for a long time is how to do we merge the context that many ecologists/conservation managers have of a set of images with the computer vision classifier. For example, if you were piloting a UAV over a forest, you might give the prompt, "This survey took place in a dense hardwood forest in Northern Vermont. Many small red maples are present". Could we fuse that knowledge with the typical computer vision classifier to help push the model to make better predictions of tree species. The alternative is what we do now, try to embed structured information like geography and environment into models and concat features (https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.13335) or the kind of rule-based knowledge distillation with student/teacher methods. I've never had much success with either kind of embeddings. They overfit quickly and are awkward. Thoughts?
Our new paper on tree species prediction for one NEON site is out at Remote Sensing in Ecology and Conservation. Tree species predictions for 100 million NEON trees will be out soon for ~25 sites.
Data will look like this, released as 1km shapefiles.
Helpful.
https://arxiv.org/abs/2304.12210
regardless of what you think of the evidence (for the record i'm not convinced), I think its incredibly important to highlight how image and sensor based technologies are the being actively used and relied on during the fieldwork surrounding Ivory-billed Woodpeckers. Camera traps, UAVs and sound recorders. When you step back, these pieces of evidence are the only convincing way to document the species, should it exist. I think we as a community should be proud of that. https://onlinelibrary.wiley.com/doi/10.1002/ece3.10017
New preprint from us! "Learning to detect an animal sound from five examples" https://arxiv.org/abs/2305.13210
Very cool! Is there something like simclr in the field of acoustics? Would be near to see the equivalent of a large foundational model for all things bird sounds
I'm deeply interested in the 'foundation model' idea, when and where it is possible/useful/worth time as a community. Check out @gvanhorn paper https://openaccess.thecvf.com/content/CVPR2021/html/Van_Horn_Benchmarking_Representatio[…]g_for_Natural_World_Image_Collections_CVPR_2021_paper.html, that didn't find a ton of downstream improvement and @Elijah Cole (Deactivated) https://openaccess.thecvf.com/content/CVPR2022/papers/Cole_When_Does_Contrastive_Visual_Representation_Learning_Work_CVPR_2022_paper.pdf. which also didn't find a ton of promise. I defer to these two on additional insights, but I will say that at UF we are really working towards the question 'when do foundational models work in ecological monitoring' looking at airborne data for both trees and birds. If there are CS students out there looking for additional data or projects, let me know.
@Daniel Davila @Ben Weinstein Yes! The idea of a foundation model for wildlife sound is something I've been thinking about a lot, and IMHO the recent methods that seem to build towards it are (a) our few-shot work and (b) van Horn's work that you cite (specifically, its general "meta-learning for Wildlife QA" approach is neat).
I have some Opinions about self-supervised learning for animal sound, too long to type out here, but suffice to say that I'm doubtful it will be the solution in itself, some algorithm guidance is needed. Thus, keep the idea of foundation models distinct from the idea of SSL, and let's try and work out how we can build them...
@Masato Hagiwara has done some work in this direction, see https://arxiv.org/pdf/2210.14493.pdf.
New preprint, which is joint with @Maddie Cusimano. We worked with eight other research groups to create a multi-species benchmark for ML applications to fine-scale bio-logger data. We also tested a lot of methods (both previously proposed and new) for analyzing this type of data. All the code, data, and annotations are available for others to use 🦊. https://arxiv.org/abs/2305.10740
Ace. Thanks for putting this together! I will look into using this data in teaching. Would you say it's appropriate for that?
Thank you! I think it would be. We formatted all the data in a consistent way, and most of the datasets are modestly sized by modern standards. So it would be easy for students to get started with the data and to run multiple experiments quickly.
If some of you visit CVPR next week, do check out our new paper 3D-POP : An automated approach annotation approach to facilitate markerless 2D-3D tracking of freely moving birds with marker-based motion capture
Highlights:
A method to use mo-cap with birds for 3D posture
Dataset with 6Hrs of videos in different group sizes, 3D Posture with ground truth
Ground truth on Identity of each individual , other annotations: 2D locations, trajectory, 3D trajectory
Scene : Indoor with 4 views
Applications :
- Full 3D markerless tracking (position, trajectory, identitifaction) of pigeons in indoor environment
- Hyrid tracking (Mocap + Markerless)
- Geometry of birds wont change so potentially same model should work for birds in outdoor environment for lifting pose from 2D to 3D.
Happy to chat more and collaborate.
Video : https://www.youtube.com/watch?v=uGMsJ0qQZrA
Link:
https://openaccess.thecvf.com/content/CVPR2023/papers/Naik_3D-POP_-_An_Automated_Annota[…]pproach_to_Facilitate_Markerless_2D-3D_CVPR_2023_paper.pdf
Not new, but topical with the discussion on few shot learning and how we move the community towards foundational models that adapt quickly.
https://openaccess.thecvf.com/content/ICCV2021/papers/Phoo_Coarsely-Labeled_Data_for_Better_Few-Shot_Transfer_ICCV_2021_paper.pdf
Shared areas in Africa and Asia where humans and big cats live together can sometimes lead to rare but notable cases of people being harmed or killed by lions, tigers, and leopards. Among these big cats, leopards are the most widespread and can be found even in areas with high human populations. This increases the chances of encounters between leopards and humans, resulting in attacks. India is a country where such attacks are common, with 50% of the states reporting injuries and deaths caused by leopards.
Himachal Pradesh (HP) state, for example, reported 30 fatal and 287 non-fatal attacks by leopards on humans each year between 2004 and 2015. By understanding the patterns of big cat attacks on people, we can develop targeted interventions to reduce these incidents. This study aims to determine if leopards are the main cause of injuries and deaths among humans by examining the following questions:
(a) do leopard attacks on humans happen in specific areas and at certain times?
(b) among these leopard attacks, do unprovoked attacks on humans cluster together in terms of location and time? and
(c) what environmental factors are associated with these clustered leopard attacks on humans?
The authors have used a statistical method commonly used in epidemiology called space-time permutation scan statistic to test for clustering of leopard attacks in specific areas and times. The attacks were scattered across 75% of HP, covering an area of approximately 42,000 square kilometers, in 11 out of 12 districts. We found that 23% of the attacks clustered into 12 significant spatio-temporal clusters. Around 14% of the leopard attacks were unprovoked, but these attacks did not form significant clusters.
The authors used statistical models to examine the relationship between eight environmental factors and the clustered attacks. They discovered that leopard attacks that occurred farther away from protected areas and closer to district boundaries had a higher likelihood of clustering. The approach developed by the authors helps identify outbreaks of unprovoked leopard attacks and confirms that there are no dedicated "man-eating" leopards in the study region. This approach can be useful in managing conflicts between humans and wildlife, and it demonstrates the usefulness of the scan statistic in ecological research.
https://www.frontiersin.org/articles/10.3389/fcosc.2023.1157067/full
Hi everyone! I wanted to share the review we released today - “Elephants and Algorithms: A Review of the Current and Future Role of AI in Elephant Monitoring”
We’d love your thoughts and feedback!
https://arxiv.org/abs/2306.13803
Abstract:
“Artificial intelligence (AI) and machine learning (ML) present revolutionary opportunities to enhance our understanding of animal behavior and conservation strategies. Using elephants, a crucial species in Africa's protected areas, as our focal point, we delve into the role of AI and ML in their conservation. Given the increasing amounts of data gathered from a variety of sensors like cameras, microphones, geophones, drones, and satellites, the challenge lies in managing and interpreting this vast data. New AI and ML techniques offer solutions to streamline this process, helping us extract vital information that might otherwise be overlooked. This paper focuses on the different AI-driven monitoring methods and their potential for improving elephant conservation. Collaborative efforts between AI experts and ecological researchers are essential in leveraging these innovative technologies for enhanced wildlife conservation, setting a precedent for numerous other species.”
Hi everyone! This may not fit (exactly) with this group. But I wanted to share a pre-print with everyone. This work aimed to show how important this field is and...it works. The big takeaway is that we surveyed and interviewed participants in an intradisciplinary setting and through a conservation technology course...Computer Scientists identify as stronger engineers, environmentalists, conservationists,....and even programmers & biologists! It's awesome( not all were statistically significant...but still pretty impressive). https://www.biorxiv.org/content/10.1101/2023.07.03.546429v1
4 pieces of recent research from my amazing students and postdocs http://mcld.co.uk/blog/2023/recent-research-from-my-amazing-students-and-postdocs.html A new crop of preprints 🙂
Hi everyone, sharing our latest work on 3D tracking animals in a large environment. Tracking insects, birds and mammals with high precision 3D (marker based, markerless and acoustic tracking) for open and closed loop experiments that can be designed for multiple questions with wide range of species.
A short youtube introduction can be found here : https://www.youtube.com/watch?v=X9RjxY2AZWs, and the paper is in Science Advances can be downloaded from here : https://www.science.org/doi/10.1126/sciadv.adf8068
Hi all!
Our new paper on reinforcement learning (RL) and complex ecological control problems is published! Check it out here:
Pretty darn good control: When are are Approximate Solutions Better than Approximate Models.
F. Montealegre-Mora, M. Lapeyrolerie, M. Chapman, A. G. Keller, C. Boettiger
Link to publication, link to arXiv pre-print, and link to source code.
We study the adaptive fishery management problem: given estimates of stock sizes for several species in a food web, decide the intensity with which to fish each species. We show that RL-based policies are more responsive to the system's dynamics compared to standard alternatives (MSY and constant escapement strategies). This leads to a higher long-term profitability and a lower chance of events such as near-extinctions in our simulations.
There is much to be said and done still before RL can be incorporated into adaptive management decisions, of course. We'd love to generate some discussion around the topic, and all feedback is super welcome!
We have released species labels for 81 species for the 100 million crown dataset in the National Ecological Observatory Network. Accuracy varies by site and length of local species list, but is generally around 70% for most sites using a hierarchical multi-temporal mixture of experts model with spectral attention (described previously here https://zslpublications.onlinelibrary.wiley.com/doi/full/10.1002/rse2.335). The preprint for the tree maps is now up on biorxiv: https://www.biorxiv.org/content/10.1101/2023.10.25.563626v1.full.pdf. Dataset is here: https://zenodo.org/records/10067302 and on Earth Engine (updating to v2 today). The data release also contains pre-cropped HSI numpy arrays for about 40,000 trees. I will be making a more formal machine learning ready dataset soon. The model code is not fully reproducible due to the immense size of the underlying data, but is still open here: https://github.com/weecology/DeepTreeAttention/tree/species_release. The tree maps on earth engine can also be overlayed over other lower resolution remote sensing products , like NAIP on earth engine, and be used for pretraining across wide range of tree ecosystems. If you are a expert in earth engine and want to make this happen, let me know. Paper is being submitted to PLOS Biology. Remaining interesting machine learning problems include domain transfer among years and sensor calibration settings, handling class imbalance in a non-hierarchical model, and self-cleaning for incorrect label detection in both train and test. We welcome all uses of the data and models for machine learning and ecological applications.
Our review of AI for elephant monitoring came out today!
Declaration of AI use😉
Yes, we have used AI-assisted technologies in creating this article.
“Elephants and algorithms: a review of the current and future role of AI in elephant monitoring
Leandra Brickson , Libby Zhang , Fritz Vollrath , Iain Douglas-Hamilton and Alexander J. Titus”
Our first paper working with species observation data will be presented at WACV 2024: BirdSAT: Cross-View Contrastive Masked Autoencoders for Bird Species Classification and Mapping
arXiv: https://arxiv.org/pdf/2310.19168v1.pdf
Code: https://github.com/mvrl/BirdSAT
Lead author: https://sites.wustl.edu/srikumarsastry/
Social media hype: https://www.linkedin.com/posts/jacobsn_ive-been-wanting-to-do-something-with-inatural[…]632219754496-0QM8?utm_source=share&utm_medium=member_desktop
Happy to discuss.
BioCLIP Tree of life project...
https://imageomics.github.io/bioclip/
We have released a new preprint and would like feedback from users who want to train a customised AI model for wildlife-image classification!
A user-friendly AI workflow for customised wildlife-image classification
Monitoring wildlife is crucial for making informed conservation and land-management decisions. Remotely triggered cameras are widely used for this, but the resulting 'big data' are laborious to process. Artificial intelligence (AI) offers a solution to this bottleneck, but it has been challenging for ecologists and practitioners to tailor current approaches to their specific use cases. Generic, online offerings also have issues of ongoing costs and data privacy. Here we present an open-source, scalable, modular, cross-platform workflow, deployed using Docker, which leverages deep learning for wildlife-image classification. Run via a user-friendly command-line interface, our workflow democratises the implementation of AI for wildlife-image classification enabling end-users without specialised technical expertise to execute a full range of tasks—from animal detection to species prediction—on local or cloud GPU-accelerated machines. It integrates seamlessly with the widely used open-source camera-trapping software ‘Camelot’, writing AI-classification data directly to image metadata and to CSV files, ready for either expert verification or direct data analysis. The end result is an advanced but accessible pipeline for wildlife-image classification. A case study with Tasmanian wildlife demonstrates the utility of our end-to-end pipeline, from classifier training to inference.
https://doi.org/10.32942/X2ZW3D
Camtrap DP: an open standard for the FAIR exchange and archiving of camera trap data (2023)
https://camtrap-dp.tdwg.org/
New paper out from the Bayne lab and the Alberta Biodiversity Monitoring Institute on the effects of audio file compression! Likely of interest to folks who use deep learning on large acoustic datasets. We examined CNN recognizers in our exploration of MP3 compression and sample rate, with some interesting and complex effects.
OA paper here: https://www.tandfonline.com/doi/full/10.1080/09524622.2023.2290718?src=exp-la
Twitter synthesis here: https://twitter.com/ellycknight/status/1743374262407450775
Really happy to see this Policy Forum piece out this week in Science on "Biodiversity monitoring for a just planetary future" -- led by @Millie Chapman and bringing together co-authors from across ecology, computer science, sociology, geography and other disciplines. https://www.science.org/doi/10.1126/science.adh8874
For people looking to apply computer vision classification to their work but don't know where to start, we published a methods-guide paper for you! "A gentle introduction to computer vision-based specimen classification in ecological datasets" http://doi.org/10.1111/1365-2656.14042
Hi all - wanted to share our new paper on using hyperspectral data to detect animals based on their spectral signature rather than their shape. There is very little existing research about the application of hyperspectral data for wildlife and this is a first of a kind assessment of the utility of very high resolution drone-derived spectral data and ai models for terrestrial wildlife detection.
https://www.mdpi.com/2072-4292/16/2/406
New paper on ecoacoustics + ML pipeline for detecting lemurs in Madagascar, and also comparisons to in-person obs & manual analysis (time, cost, etc.)! With @Emmanuel Dufourq @Lorène Jeantet & more 🙂
https://onlinelibrary.wiley.com/doi/10.1002/ajp.23599
This is the first paper I've seen to put a dollar value not only on the benefits of AI for camera trap image processing, but also on the benefits of using connected cameras (and demonstrating the cost + carbon benefits of not constantly visiting cameras):
https://www.researchsquare.com/article/rs-3637685/v1
The platform being evaluated was eVorta:
https://web.evorta.com/
I don't know whether any of the authors or any of the eVorta devs are on this Slack, but if you are, great work!
Food for thought, we hope, for this community: https://www.nature.com/articles/s41561-023-01369-y
https://www.mdpi.com/2076-3417/13/18/10397
looks like a cool dataset, but mAP of 0.983 seems really high
ya, I wasn't particularly happy with the paper, just posting so we can keep track of data. @Dan Morris
Oh, you know me too well, @Ben Weinstein, new datasets I can tinker with always make my day. Added to the Big List O' Aerial Datsets, with sample code:
https://github.com/agentmorris/agentmorrispublic/blob/main/drone-datasets.md#waid-wildlife-aerial-images-from-drone
Caveat: even when I paged through just a few images, to pick a thumbnail, I noticed that (a) not all images are completely annotated, and (b) many of the objects labeled "camelus" are... definitely not (but the boxes were still spatially accurate). Also images seem to have been forced to square aspect ratios, so animals are generally squished in one dimension. And if I looked at 0.01% of the dataset and noticed all that, the issues are likely somewhat widespread. But data is never perfect, and I still think it's good enough to be useful overall, and it was very straightforward to work with, so, a net positive.
It seems they did uniform random train/val/test splits, which makes it certain that some frames from the same video would be in both train and test
Plus this one is about goats. https://www.nature.com/articles/s41597-023-02555-8
3D reconstruction and semantic segmentation of coral reefs! Fresh from the press at ME&E: https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210X.14307
Mapping forests in the Swiss Alps since the 1940s: semantic segmentation and knowledge-guided deep learning help us building the time machine and show how much the treeline moves upwards in time: https://www.sciencedirect.com/science/article/pii/S0034425724001202 Super work by @Thiên-Anh Nguyen
Whenever I give a talk, I always stress that we are looking to complement traditional forest surveys with airborne information. I often offhand say there will never be a time in which robots will walk through forests identifying trees....https://ora.ox.ac.uk/objects/uuid:d29224bc-f133-484f-b8eb-f99fb34b6eb3/files/sx920fz468
Excited to share our paper on individual identification of sloth bears (Melursus ursinus) using SSIM and Deep Learning !!
https://link.springer.com/article/10.1007/s42991-023-00396-x
PanAf20K: A Large Video Dataset for Wild Ape Detection and Behaviour Recognition
Paper: https://link.springer.com/article/10.1007/s11263-024-02003-z
Project Website: https://obrookes.github.io/panaf.github.io/
Dataset: PanAf20K AI-enabled Video Dataset of Great Apes in the Wild - Datasets - data.bris
New paper from us - training efficient small neural nets for @Benjamin Cretois’s "ecoVAD" task https://www.mdpi.com/1424-8220/24/7/2046
The conservation value of forests can be predicted at the scale of 1 hectare (2024)
The positive impact of conservation action
https://www.science.org/doi/10.1126/science.adj6598
https://phys.org/news/2024-04-kind-actions-effective-halting-reversing.html
Our paper on mapping California’s urban forest:
https://www.sciencedirect.com/science/article/pii/S1569843224002024
Explore our state-wide map of trees here:
https://jventu09.users.earthengine.app/view/urban-tree-detector
Great paper! really glad to see both zero-shot and fine-tuned performance for DeepForest, let us know (https://github.com/weecology/DeepForest/issues), if there is anything about the workflow that could make it easier to serve as a baseline.
Thanks, Ben! Was pretty straightforward to test out DeepForest on our data — only issue was that we had point annotations so we had to devise some strategies to figure out how to train and test with a bounding box model. I don’t know if others would fine it useful but I would be happy to share our code for that.
Great stuff! Is the annotated training data publicly available?
Thanks! It is, you can find the data here: https://github.com/jonathanventura/urban-tree-detection-data
and the code here: https://github.com/jonathanventura/urban-tree-detection
Are you looking for species distribution maps for trees? Look no further! We have 10K of those for you (and estimates of distribution in 2100 as well!)
Check out our latest paper, with @Nina van Tiel in the lead: https://www.nature.com/articles/s41467-024-48276-3
🌟 Introducing BirdAVES: Our Latest Bird Sound Foundation Model 🌟
We are thrilled to announce the release of our latest series of self-supervised animal vocalization encoder (AVES) models, named BirdAVES, specifically scaled and trained for bird sounds.
🤖Foundation models like GPTs have played a pivotal role in the AI revolution over the past few years. ESP has been heavily investing in the development of foundation models, and we released the first-ever discriminative foundation model, AVES, for animal vocalization last year. Trained through self-supervision, AVES can encode a wide range of animal vocalizations, achieving significant performance gains in many bioacoustic datasets and tasks.
🦜In this iteration, we focused on bird vocalizations, as they are one of the best-studied and well-resourced types of animal vocalization in bioacoustics. Bird sounds have a wide range of ecological, behavioral, and conservation implications through behavioral studies, population monitoring, and habitat analysis.
🚀We significantly scaled up the training of AVES in terms of training data, model size, and compute power, as it has been shown that the scaling law of foundation models apply across many domains of machine learning.
📈The newly trained BirdAVES models have achieved over a 20% performance improvement on bird datasets and have shown enhanced performance across various other datasets as well.
Check out our pretrained models here: https://github.com/earthspecies/aves?tab=readme-ov-file#birdaves
Can the confidence scores produced by your species classification model be interpreted as probabilities?
Check out my first paper to find out! https://zslpublications.onlinelibrary.wiley.com/doi/full/10.1002/rse2.412
(with @Simon Chamaillé and @Vincent Miele CNRS)
Quick skim was super interesting! Great stuff. I didn't see any analysis of how calibration breaks down by class imbalance.
Furthermore, we have focused our work on the calibration of the top-1 label on a multiclass classification task, and with a relatively small number of classes. But in future work, it would also be interesting to look at the calibration of the predictions on other tasks of interest for models applied to ecology. For instance, calibration could be explored in multi-label classification, hierarchical classification, object detection or classification over a very large number of classes.Do you anticipate that the calibration of the rarest of the 22nd classes would be worse? We have found decreasing reasonableness of confidence scores as classes is rarer in the dataset.
Thank you very much! I didn't look in detail at the calibration by class, and in particular for the under-represented classes because I ended up with too few predictions in each bin to correctly estimate the ECE. So unfortunately I don't have an answer to your question!
But perhaps I could try grouping together the rare classes to calculate the ECE
Review & recommendations for individual ID in acoustic recordings! We argue this field needs a lot of attention from you fine folks in the deep learning world to truly open doors. Check out the twitter thread here (https://x.com/ellycknight/status/1802719544261165206) for a plain-language summary and the paper here (https://authors.elsevier.com/a/1jEmtcZ3X04sY) for the technical recommendations! This was a big multidisciplinary effort with many folks including @Tessa Rhinehart @Justin Kitzes @Matt Weldy
I have just finished my PhD in the applications of deep learning to detect and classify dolphin signature whistles 🐬. I'm hoping to get some papers published from my research chapters and a dataset containing signature whistles to go with it!
Amazing! Looking forward to it @Georgia Atkinson! Check out the recommendations in the paper around storing datasets with the Environmental Data Initiative if you get a chance
Our ICIP2024 paper OpenAnimalTracks has been released !!
Identify animal species by their footprints👣
arXiv: https://arxiv.org/abs/2406.09647
Our latest research paper is published: a first semi-automated survey of African mammals using oblique imagery and deep learning 📷💻🐘 Click here to access the full paper!
🔎 In this study, we explored the synergy of a semi-automated deep learning model and oblique cameras for better estimation of large African mammal populations. Our research aimed to enhance the efficiency and accuracy of traditional systematic reconnaissance flights with rear-seat observers, which are widely used to monitor these populations.
📈 The proposed approach provided higher population estimates for small and static species and comparable estimates for larger ones. It required only 111 hours of human effort to process more than 190,000 images - a significant reduction compared to manual interpretation - making wildlife camera surveys more efficient and cost-effective.
🌟 This study marks a significant step towards integrating AI in wildlife monitoring, suggesting a future shift to AI-assisted aerial surveys for greater efficiency and safety, especially with microlight aircraft.
Is there any hope for the data, model weights, to be made available? Anything to contribute to the community to avoid future users to need repeat the annotation effort?
Thanks for the request, indeed it would be great but unfortunately the data is held by the park managers and cannot be shared with third parties.
🌳 Happy to report our global tree crown delineation dataset + paper (pre-print) is now live: https://arxiv.org/abs/2407.11743
💾 You can find our code here: https://github.com/Restor-Foundation/tcd
🤗 Our models and dataset are hosted on HuggingFace: https://huggingface.co/restor; data are mirrored on Zenodo in MS-COCO JSON format: https://zenodo.org/records/11617167
📚 Documentation: https://restor-foundation.github.io/tcd/
📔 Colab example: https://colab.research.google.com/drive/1N_rWko6jzGji3j_ayDR7ngT5lf4P8at_
There's still a lot to do and I'm actively working to improve the usability of the pipeline, but hopefully it's a useful resource for the community and after 2 and a bit years working on it, I'm excited (and exhausted) that it's finally public. Keen to get feedback if it's useful for anyone here and to answer questions 🙂
⚠️ Limitations: see the supplementary info in the paper, but the main one is that we did not attempt (by design) to label every tree in dense/closed canopy, so in those situations you're better off using the semantic segmentation models (which agree well with LIDAR comparision). Instance segmentation will give you good image coverage, but if you need e.g. every tree in a rainforest you should look at packages like DetectTree2. The motivation for the work was to support restoration monitoring where we often care about (a) early stage projects pre- canopy closure and (b) later stage projects where often coverage is a more useful metric than counts. You can probably get a good first-order estimate for density estimation if you have a prior on crown size distribution for a particular site + the coverage %.
TL;DR: we're releasing an instance/semantic segmentation dataset containing over 280k individually delineated tree polygons and over 50k tree groups. Our images are sampled globally from high resolution (10 cm/px) orthomosaics from Open Aerial Map; over 90% are CC-BY 4.0 licensed with a small portion -NC or -SA. Code (Apache 2.0) and models are also public.
Many of know about/are already using the data, but the official publication for the first NEON tree species prediction dataset is out in PLOS Biology. We released crown maps for 24 US forests for a total of 100 million crowns, classified to 81 canopy tree species. Predictions and training data are available on zenodo. https://journals.plos.org/plosbiology/article?id=10.1371/journal.pbio.3002700. We use a hierarchical mixture of exports + temporal averaging + spectral attention for hyperspectral classification for each site. For those interested in challenging computer vision problems, the next steps are to find a single model across all sites that dynamically handles class imbalance, more sophisticated temporal ensembling, and better data augmentation and self-correction for misattributed samples.
Introducing “PetFace: A Large-Scale Dataset and Benchmark for Animal Identification” (ECCV2024) !! 🐦
✅250k IDs (The largest ever!)
✅13 species
✅Fine-grained annotations
Website: https://dahlian00.github.io/PetFacePage/
Paper: https://arxiv.org/abs/2407.13555
I forgot to post here these last two works on horses: “The Poses for Equine Research Dataset (PFERD)” (https://www.nature.com/articles/s41597-024-03312-1), where we do dense motion capture on horses, placing more than 100 markers, also capturing a wide range of poses (i.e. a sitting horse)
We have a new PhD student starting today(!) on the topic of detecting pain in horses from their facial expressions. Thanks for sharing this dataset, may come in handy at some point 🦄
You’re welcome! All this work has been done with people from SLU and KTH and they are also interested in horse pain detection
What a great name for the dataset! (:flag_de: language learner here)
And this one “VAREN: Very Accurate and Realistic Equine Network”, where we create an articulated 3D shape model of horses from 4D scans of real animals (https://varen.is.tue.mpg.de/).
really interesting. How accurate/designed for this purpose do the scans need to be? I've been toying with the idea of using museum specimen scans to try to make synthetic data for species detection.
We tried that a long time ago and it did not work for us. The specimens we scanned in the museums had over smoothed and not realistic shapes.
Please check out our new paper on multimodal language models in biodiversity and conservation: New frontiers in AI for biodiversity research and conservation with multimodal language models ! Our paper provides a thorough yet accessible introduction to the capabilities and transformative potential of MML models in biodiversity
and conservation applications. Furthermore, we outline existing hurdles to implementing these techniques
in real-world scenarios and propose directions for future research to overcome these challenges. We aim to foster robust discussion with this paper and provide information for future research into the sustainable and equitable development of MML models as a potential paradigm shift in AI for biodiversity monitoring and conservation. It is currently in submission but any comments are welcome! Thank you very much!
Nice work! Am working on multi-agent systems in natural history collections space so will take a close look. 🙂 Love the Koala and Antechinus examples! 🐨
I'm excited to share this research paper:
"Bird song comparison using deep learning trained from avian perceptual judgments"
Zandberg et al (2024), Plos Computational Biology
https://doi.org/10.1371/journal.pcbi.1012329
We attempted something quite difficult: instead of using human labels, or unsupervised learning, to train machine learning on animal sounds -- can we train an algorithm directly from the birds' own perceptual judgments?
(Social media summary is here on my mastodon)
📢 Excited to share the release of our new papar, dataset and DL benchmarking competition led by Stefano Puliti. #FORspecies20K is a new dataset 🚀of 20,000+ trees across 33 species 🤯 for benchmarking tree species classification using point clouds from TLS, MLS, and ULS 🌲🌳
📰pre-print: arxiv.org/abs/2408.06507
📊data: zenodo.org/records/13255198
💻code: github.com/stefp/FOR-species
📈 benchmark: codabench.org/competitions/3667/
More details here: https://x.com/stefanopuliti/status/1825467026028052914?t=SOwcBO5K_kCqiqBk4n4l8Q&s=19
I am happy to share that our latest work "From Weak to Strong Sound Event Labels using Adaptive Change-Point Detection and Active Learning" has been accepted at EUSIPCO 2024 and nominated to the best student paper category.
- code: https://github.com/johnmartinsson/adaptive-change-point-detection
- paper: https://eurasip.org/Proceedings/Eusipco/Eusipco2024/pdfs/0000902.pdf
- data: https://zenodo.org/records/10811797
We propose a method to collect strong sound event annotations in a cost efficient way, while still keeping the annotation task very simple by only asking the annotator to provide class labels for given audio segments. The method is demonstrated on bioacoustic sound data (🐦), but should in principle work for any sound data (and potentially many other types of time series data) with fairly minor changes.
Please come by and listen to the presentation on Wednesday 28 if you are at EUSIPCO and you are interested in annotation of bioacoustic data!
📸 🦌 🐘 Happy to share with this community our recent camera trap paper “Deep learning-based ecological analysis of camera trap images is impacted by training data quality and size”
Link: https://arxiv.org/abs/2408.14348
⚖️ In this work we compared key ecological metrics derived from expert annotations with those generated from model predictions across two diverse ecosystems. Moreover, we studied the impact of deep learning training decisions (model choice, dataset size, label noise) on downstream ecological analysis.
🕵️♀️ We show that the model choice does not influence the outcome and further demonstrate the resilience of deep learning-derived ecological analysis, tolerating up to 10% error in species labels and 50% data reduction. Despite this robustness, noisy or small datasets can have a significant impact on the accurate calculation of ecological metrics and thus practitioners should take mitigation actions to ensure reliability of the findings.
I know many of you are out in the field, but I’m sure some also deal with animals in human care. We’ve developed a system for the automated detection of stereotypic behaviour, specifically focusing on polar bears. Using this system, we gained fascinating insights into the seasonal patterns of stereotypic behaviour by analysing data collected over three years.
Paper: https://doi.org/10.1016/j.ecoinf.2024.102840
Video: https://www.youtube.com/watch?v=HMebXKtyFKI
Code: https://github.com/team-vera/stereotypy-detector
Website: https://team-vera.github.io/
I'm excited to share that "TaxaBind: A Unified Embedding Space for Ecological Applications" has been accepted to WACV 2025!
"TaxaBind is a multimodal embedding space across six modalities: ground-level images of species, geographic location, satellite image, text, audio, and environmental features, useful for solving ecological problems."
Project: https://vishu26.github.io/taxabind/
Paper: https://arxiv.org/abs/2411.00683
Code & Models: https://github.com/mvrl/TaxaBind
I encourage you to test the TaxaBind pre-trained models if you are considering using BioCLIP, ArborCLIP, GeoCLIP, SatCLIP, ScaleMAE, SatMAE, or other similar models for an ecological mapping task!
Can you describe more about
We release iSatNat, containing 2.7M pairs of satellite images and ground-level species images.what does it mean to have a satellite view of a species image given the resolution? Would there be a way to generate airborne views (probably not satellite) of a 'species' in this embedding? I'd love to see an example.
Is it basically a habitat map, or is it more than that? I've always wanted a way to try to port ground based images into airborne representations. @Zhongqi Miao and I were emailing about this morning.
It's just a satellite image of the location of the species observation.
🌍 We are excited to introduce NatureLM-audio — the first large audio-language model tailored for understanding animal sounds. 🐦🐳
Trained on millions of bioacoustic recordings and enriched with speech and music data, NatureLM-audio can:
- 🌱 Perform zero-shot classification across thousands of species without additional training
- 🔍 Generalize to identify species it’s never seen before
- 🧠 Respond to natural language prompts and generate text descriptions of animal sounds
Plus, it goes beyond typical tasks to classify life stages, identify call types, and caption bioacoustic recordings with rich detail.
We’re thrilled about these early results and what they could mean for biodiversity research! You can check out the preprint and demo here:
📄 Preprint: https://arxiv.org/abs/2411.07186
🎥 Demo: https://earthspecies.github.io/naturelm-audio-demo/
Coming soon: open-source code, a UI, and plans to incorporate video, motion, and more. 🌱
🌎 Hey team! For those of you tackling nature-based climate solutions, I partnered with The Nature Conservancy to build an “expert-in-the-loop” AI pipeline that synthesized evidence across 2M scientific studies.
We focused on two key questions:
1️⃣ How much research exists on different climate solutions?
2️⃣ What does the evidence say about their impacts—benefits and potentially, trade-offs—on people and nature?
🚨 Key takeaway: Some systems with huge mitigation potential (like wetlands or methane emissions in cultivated landscapes like rice) or regions of the world (e.g. parts of SE Asia, W/C Africa) have a paucity of evidence--and thus should be a priority for future work.
✅ On the flip side, if you’re working in well-studied systems (e.g. forests) or regions (e.g. Brazil, Indonesia, USA), our findings strengthen the case that climate solutions that call on nature for help are not only shovel-ready but also offer co-benefits for people, biodiversity, and ecosystem services.
Keen to connect if this resonates or if you’d like to learn more 🌱✨
Link to the study: https://rdcu.be/d10Nh
Backstory on our ML/NLP methods
Commentary situating this work wrt global carbon markets (now in excess of $100B in revenue!)
🎉 We're happy to share the release of Whombat, an open-source annotation tool designed to facilitate your bioacoustics AI workflows! 🐸🦇🦉
Whombat helps you:
- Organize audio datasets and metadata
- Manage and track annotation projects
- Annotate with precision and verify your work
- Export data for training and import predictions for analysis
- And more!
Available for Linux, macOS, and Windows as a standalone executable (no dependencies!) or host it on a shared server.
🤝 Help us improve by providing feedback, reporting bugs, contributing code, or suggesting features.
Done work in collaboration with @Oisin Mac Aodha, @Kate Jones, @Matt Weldy, Ella Browning and Holly Pringle.
Links:
- Paper: http://doi.org/10.1111/2041-210X.14468
- Code: https://github.com/mbsantiago/whombat
- Docs: https://mbsantiago.github.io/whombat/
Congratulations Santiago! 🎉 Outstanding work. The emphasis on open-source, collaborative and machine guided annotation is really promising. Well deserved!
🚨 New paper 🚨
The automated analysis of imagery is slowly becoming the standard method for benthic ecologists to perform biodiversity monitoring, though most lack the knowledge of computer vision to make full use of these new tools. At the same time, computer scientists are often missing the ecological domain knowledge to know where their skills can be of the most help.
To combat this, we compiled a review which bridges the gap between benthic ecology and computer vision. Born out of everything we had to learn for our own benthic-CV project at the British Antarctic Survey, the paper is designed as a starter-pack for those wanting to do similar in the future.
https://doi.org/10.1016/j.ecoinf.2024.102989
"Evaluating the method reproducibility of deep learning models in biodiversity research" https://peerj.com/articles/cs-2618/ - a paper I handled as editor. It's a useful examination of how reproducible the published research actually is.
Ever wondered how well large language models (LLMs) perform on ecology-related tasks? 🌍🦉
In our latest preprint, we evaluate Gemini 1.5 Pro and GPT-4o on a suite of ecological challenges: predicting species presence, generating range maps, listing critically endangered species, classifying threats, and estimating species traits. Using a new benchmark dataset, we compare their outputs to expert-derived data to assess their accuracy and reliability.
Check out our work at https://www.biorxiv.org/content/10.1101/2025.02.10.637097v1
Consider including DeepSeek, as it is an open-source model that can be run locally without requiring an API (paywall).
Great suggestion. The code for recreating our experiments with different models is available online:
https://github.com/filipgdorm/eco-llm
We are hoping that others can apply it to different LLMs.
Hello everyone,
This is one I am really excited to share with all of you…
After several months of efforts (and waiting), our perspective paper of AI to fill ecological shortfalls is out in Nature reviews Biodiversity:
https://www.nature.com/articles/s44358-025-00022-3
It’s a joint effort of the ABC biodiversity center and a joint think tank of a number of great people, including @Tanya Berger-Wolf @Sara Beery @Laura Pollock @David Rolnick @Justin Kitzes @Kaitlyn Gaynor @Oisin Mac Aodha
I hope it will be an inspiring read!!!
Hello everyone, check out our new paper on using multi-modal language models in zero-shot bioacoustics recognition! It is a case study, but we hope this can spark more discussion on the zero-shot potential of multi-modal language models in bioacoustics and conservation in general! https://www.nature.com/articles/s41598-025-89153-3?utm_source=rct_congratemailt&utm_[…]m_campaign=oa_20250228&utm_content=10.1038/s41598-025-89153-3
This is super interesting, thanks for sharing, we are trying to do the same except with a image foundation model (DinoV2)!!
CVPR2025 paper:
The PanAf-FGBG Dataset: Understanding the Impact of Backgrounds in Wildlife Behaviour Recognition
https://arxiv.org/abs/2502.21201
Is the dataset available? I love making lists of datasets, but I couldn't find a link to the dataset in the paper.
Hi Dan, it will be ready imminently. I will let you know.
@Dan Morris Is your list of datasets online? I know of your list of UAV based datasets but is there another one with all type of image based ecological datasets. I can also contribute to this effort. I am publishing and producing datasets myself and happy to support open dataset publishing.
This page is where I track things like this:
https://lila.science/otherdatasets
It includes all the datasets I'm aware of - including the ones on the aerial/UAV datasets list - except for the data that's already on LILA.
Thanks, I can offer three of our own datasets, I do not see in your list. Thanks for keeping such a list.
( https://edmond.mpg.de/dataset.xhtml?persistentId=doi:10.17617/3.HPBBC7)
(https://edmond.mpg.de/dataset.xhtml?persistentId=doi:10.17617/3.JCZ9WK)
(https://edmond.mpg.de/dataset.xhtml?persistentId=doi:10.17617/3.ENDMTI)
Cool paper (not my own!) using yolo on coral reef fish sounds from the US Virgin Islands:
https://doi.org/10.1121/10.0035829
Might interest this group. Our new preprint (in review) testing deep learning tree crown segmentation on rigourous ground truth TLS-cut high resolution RGB imagery.
tldr: performance is much worse than on (our) manual annotations https://arxiv.org/abs/2503.14273
This is a great paper and I sent some comments to Matt (who I know via Frontier Development Lab). Looking forward to the data release, as expert-reviewed/annotated TLS is remarkably hard to come by!
@Josh Veitch-Michaelis its incredibly hard to create! Matt has done amazing work to even generate these data
Yeah we did some work with a Leica TLS and just stitching the scans took forever in a small patch of forest. Also I remember the user experience getting the data off the scanner was... interesting.
you'll be pleased to hear that TLS pre-processing has come a long way so that isn't an issue any more. but segmenting trees is still a very manual process, and linking to the drone data is very challenging because of unrealiable GPS and distortions in the orthomosaicking process
New paper from us! "Performance of Computer Vision Algorithms for Fine-Grained Classification Using Crowdsourced Insect Images" https://doi.org/10.1049/cvi2.70006 This work led by @Rita Pucci investigated how various DL architectures respond to fine-grained differences between insect species.
🔵 𝐂𝐎𝐏-𝐆𝐄𝐍-𝐁𝐞𝐭𝐚: 𝐔𝐧𝐢𝐟𝐢𝐞𝐝 𝐆𝐞𝐧𝐞𝐫𝐚𝐭𝐢𝐯𝐞 𝐌𝐨𝐝𝐞𝐥𝐥𝐢𝐧𝐠 𝐨𝐟 𝐂𝐎𝐏𝐞𝐫𝐧𝐢𝐜𝐮𝐬 𝐈𝐦𝐚𝐠𝐞𝐫𝐲 𝐓𝐡𝐮𝐦𝐛𝐧𝐚𝐢𝐥𝐬
Today we release a prototype of COP-GEN - a universal generative model for Copernicus data. 𝐂𝐎𝐏-𝐆𝐄𝐍-𝐁𝐞𝐭𝐚 is a model trained globally on the thumbnails of the Major TOM Core datasets, including Sentinel-2 L1C, Sentinel-2 L2A, Sentinel-1 RTC, and COP-DEM GLO-30.
How is it universal? COP-GEN learns a joint generative process of all modalities, which means that it can reconstruct data from any subset of present observations. 𝐖𝐢𝐭𝐡𝐨𝐮𝐭 𝐭𝐫𝐚𝐢𝐧𝐢𝐧𝐠 𝐬𝐩𝐞𝐜𝐢𝐟𝐢𝐜𝐚𝐥𝐥𝐲 to perform any of these tasks it can be used to approximate:
✅ Sentinel-1 to Sentinel-2 translation
✅ Elevation estimation from Sentinel-2 or Sentinel-1
✅ Atmospheric Correction (L1C to L2A pipeline)
✅ Atmospheric Generation (L2A to L1C)
✅ ...and any other task involving translation between the supported modalities
On its own, the model can be used as a useful prior for estimating the data likelihood distribution for Copernicus data. COP-GEN-Beta learns joint, conditional, and marginal distributions within a single unified backbone, allowing to flexibly sample any modality given any condition.
Why is it Beta? Because thumbnails are a low-cost representation of the data that scales well and we managed to develop this prototype quite fast. We are currently developing the more costly COP-GEN model that supports the original data. For now, we wanted to showcase the prototype and make it available to the community for a test!
📡 Stay tuned for the full model announcement and if pass through CVPR, COP-GEN-Beta has been accepted to the 𝐌𝐎𝐑𝐒𝐄 𝟐𝟎𝟐𝟓 𝐰𝐨𝐫𝐤𝐬𝐡𝐨𝐩.
🌐 𝐖𝐞𝐛𝐬𝐢𝐭𝐞 https://huggingface.co/spaces/mikonvergence/COP-GEN-Beta
💻 𝐂𝐨𝐝𝐞 https://github.com/miquel-espinosa/COP-GEN-Beta
⚖️ 𝐌𝐨𝐝𝐞𝐥 https://huggingface.co/mespinosami/COP-GEN-Beta
📱 𝐃𝐞𝐦𝐨 https://huggingface.co/spaces/mikonvergence/COP-GEN-Beta
📄 𝐏𝐚𝐩𝐞𝐫 https://arxiv.org/abs/2504.08548
This is some recent work we have been doing on dolphins for 3D reconstruction from drone videos. https://arxiv.org/abs/2504.15782
Have you come across this work before? https://www.yousufsoliman.com/projects/going-with-the-flow.html
I'm a phd student in computer graphics (graduating soon). This was a recent graphics paper, that was a follow up to this one: https://olligross.github.io/projects/MotionFromShapeChange/MotionFromShapeChange_project.html
Perhaps the mathematical machinery they propose would be useful for the sort of work you are doing!!!
Wow, that is super nice, thank you for the pointer. We use a much simpler model, but probably having a more complex (=realistic) model of the media could be a way to set implicit constraints to the type of motion that we aim at recovering.
of course! I really liked this project, very elegant formulation! It's a little formidable to wrap your mind around, but looks like just what is needed for your line of work 🙂
This looks really interesting multi-model inputs for zero-shot bioacoustics classification https://github.com/earthspecies/NatureLM-audio. @Benjamin Hoffman I would love to discuss love to discuss https://arxiv.org/pdf/2503.00296? I think is a really great paper that I encourage others to read as we move towards multi-modal data synthesis for pretraining, I know i've had this conversation with many of you thinking along similar lines.
Hey, thanks! Yeah let’s set up a time to chat
this paper just came out, might be of interest to some of you!
"Towards Application-Specific Evaluation of Vision Models: Case Studies in Ecology and Biology"
https://arxiv.org/pdf/2505.02825
About why models should be evaluated using application-specific metrics that directly represent model performance in the context of its final use case.
I have been looking into SSL and foundation models in ecology. Found this while exploring the space. Thought to share. https://arxiv.org/pdf/2504.18256
A work on multimodal dataset for biodiversity research https://arxiv.org/pdf/2505.11568
Hi folks! In case of interest, I wanted to share my thesis from the mid-career MIT Sloan Fellows MBA program. Title: “The Net Climate Impact of AI: Balancing Current Costs with Future Climate Benefits”. It was an honor to work with John Sterman and Harvey Michaels on this research, which explores the dual role of AI as both a climate challenge and a potential climate solution. Details and recordings.
I’m now looking to apply my expertise in climate, energy and AI within a mission-driven organization making a real impact. My background spans product development, strategy, partnerships, business development, leadership, and education - and I’d love to bring that to a team building meaningful solutions. Linkedin profile. If you have any questions or want to connect, feel free to reach out at jturliuk@mit.edu.
Hello folks,
if you are at CVPR (and I know many of you will be, due to the birding walk), please make sure to drop to our poster at the main conf, where @Valentin Gabeff will present our new animal behaviour understanding dataset Mammalps (also presented at the main conf and cv4animals). Wanna know more? Have a look here: https://arxiv.org/abs/2503.18223 https://eceo-epfl.github.io/MammAlps/ (codes + tutorials, + benchmarks). It’s a wonderful work with @Alexander MathisEPFL!
Besides cv4animals, @Valerie will also present EcoWikiRS at the Earthvision workshop, an image/text/GBIF dataset and study about ecosystem mapping from Wikipedia and GBIF! More info here: https://arxiv.org/abs/2504.19742
🎉 I'm excited to share that our new paper is out in Methods in Ecology and Evolution!
"Simulated soundscapes and transfer learning boost the performance of acoustic classifiers under data scarcity" makes progress on a big challenge in bioacoustics: building accurate classifiers when labeled data are scarce.
📣 By combining transfer learning with soundscape simulation, we were able to train high-performing acoustic classifiers with as few as two labeled examples per class.
🌎 This work supports more accessible, scalable tools for conservation, especially where resources for manual annotation are limited.
Huge thanks to my co-authors and collaborators in the US and Europe!
📄 Weldy et al. 2025
🔗 https://doi.org/10.1111/2041-210X.70089
New paper alert from my lab:
Global and Local Entailment Learning for Natural World Imagery (by Srikumar, Aayush, Eric, Subash) has been accepted to the International Conference on Computer Vision (ICCV 2025)
TL;DR: if you are using BioCLIP for something, give this a shot. We make the learned embeddings pay more attention to the taxonomic hierarchy and it often works better.
Abstract:
Learning the hierarchical structure of data in vision-language models is a significant challenge. Previous works have attempted to address this challenge by employing entailment learning. However, these approaches fail to model the transitive nature of entailment explicitly, which establishes the relationship between order and semantics within a representation space. In this work, we introduce Radial Cross-Modal Embeddings (RCME), a framework that enables the explicit modeling of transitivity-enforced entailment. Our proposed framework optimizes for the partial order of concepts within vision-language models. By leveraging our framework, we develop a hierarchical vision-language foundation model capable of representing the hierarchy in the Tree of Life. Our experiments on hierarchical species classification and hierarchical retrieval tasks demonstrate the enhanced performance of our models compared to the existing state-of-the-art models.
Webpage: https://vishu26.github.io/RCME/index.html
Paper: https://arxiv.org/abs/2506.21476
Model: https://huggingface.co/MVRL/rcme-tol-vit-base-patch16
Code: https://github.com/mvrl/RCME
Hi 👋, super excited to share that our new paper: Towards scalable insect monitoring: Ultra-lightweight CNNs as on-device triggers for insect camera traps, has now been published in Methods in Ecology and Evolution, for its special issue covering AI for conservation and ecology 🥳
Here, we explore the challenging problem of developing an alternative to the camera trap PIR trigger which is suited to ectothermic insect detection. We opt to use ultra lightweight AI models (less than 500k parameters) and optimise these to detect insects on deployed on microcontroller hardware to minimise power consumption. As well as achieving impressive results both in and out of training domain tasks, our little models consume <300mW of power, paving the way for future ectotherm camera trap designs which can leverage efficiency for scalability benefits. We also implement an "explainable AI" evaluation pipeline using saliency maps to quantify prediction reliance on spurious background features and evaluate models based on this in addition to specificity and recall.
A toolkit to help develop future variations is provided. The Ecto-Trigger project is now live on GitHub.
https://rossgardiner.github.io/ecto-trigger/html/
https://github.com/rossGardiner/ecto-trigger
Happy to chat comments/questions, you can find me here or over email: rg654@exeter.ac.uk
Cheers!
Ross
Here is the direct link to the paper: https://besjournals.onlinelibrary.wiley.com/doi/full/10.1111/2041-210X.70098
Great work!
Ah thank you for the catch! I updated the link now 🙂
Hi everybody,
We are happy to announce the release of our preprint “A Roadmap for Climate-Relevant Robotics Research” (https://arxiv.org/abs/2507.11623). I realize this is tangentially related, but I think many topics coincide.
This paper aims to foster collaboration between robotics researchers and climate experts by identifying actionable, high-impact problems where their combined expertise can address urgent climate challenges. We include opportunities not only to deploy physical robots but also apply the broader robotics toolkit, such as planning, perception, control, and estimation algorithms to climate-relevant problems.
More info and contact here: https://www.climate-robotics.org/
Thanks!
Thought to share here https://arxiv.org/abs/2507.09080
Hi everyone! Our new open-access paper just came out in PNAS where we talk about the next generation of animal telemetry devices (including smart on board processing) 🎉 Additionally, we document a global lack of animal movement studies in urban areas, biodiversity hotspots, and areas experiencing deforestation (among other findings).
Check it out here: https://www.pnas.org/doi/10.1073/pnas.2410947122
Hope you find it interesting!
I figure many people here are relying on satellite imagery (links to dataset and paper within): https://deepmind.google/discover/blog/alphaearth-foundations-helps-map-our-planet-in-unprecedented-detail/
How can we track animals like sharks in aerial imagery, without training? We’re excited to share our new paper Zero-shot shark tracking and biometrics from aerial imagery has recently been published in Methods in Ecology and Evolution 🎉. We introduce FLAIR - an improved way to leverage video segmentation models for zero-shot animal tracking!
FLAIR uses simple text prompts to effortlessly track animals—like sharks, zebras, or birds—without any additional training or fine-tuning. We show FLAIR masks are useful for computing a variety of biometrics including measuring shark tail beat frequency in several species across very different habitats.
Check out our paper, blogpost, or code... and try it out yourself!
To people interested in fishery science, we just published this paper on Fish and Fisheries! 🙂
Using machine learning to inform harvest control rule design in complex fishery scenarios
Felipe Montealegre-Mora, @Carl Boettiger, Carl Walters, Chris Cahill
https://doi.org/10.1111/faf.70013
We looked at using techniques from Reinforcement Learning and Bayesian Optimization to optimize multi-observation harvest control rules for fisheries with highly variable "spasmodic" recruitment. (We focused on the case of Walleye fisheries.)
Hope it can be interesting to some folks here!
How much effort is to annotate animals with bounding boxes on aerial images? Is it worthy when you detect and count with YOLO-like architectures? with @Giacomo May , @Benjamin Kellenberger and @Alex Delplanque we have gave a thought (and quite some GPU hours) to the question….
Have a look: https://doi.org/10.1016/j.ecoinf.2025.103387 Freshly printed in Ecological Informatics!
Hi! We released a new paper 📣
AnimalClue: Recognizing Animals by their Traces [ICCV25 Highlight✨]
- 968 animal species
- 159,605 bounding boxes
- 22 fine-grained annotations
ProjectPage: https://dahlian00.github.io/AnimalCluePage/
Paper: https://arxiv.org/abs/2507.20240
Demo: https://huggingface.co/spaces/risashinoda/animalclue_yolo_det
X: https://x.com/dahlian0/status/1950204855597092895
Hello everyone, check out our new perspective paper on the multimodal language models in the context of conservation and biodiversity:
https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210X.70120?af=R
It is an updated version of our preprints. This paper provides a general overview of multimodal language models and their differences from conventional machine learning methods in biodiversity studies. It explores potential contributions to the fields and the impact on practitioners. We also discuss key challenges associated with the practical application of multimodal language models. We hope this paper will spark further discussion and research into the use of this new and wonderful tool in conservation and biodiversity.
The summary report of the 2025 US-UK Forum on Measuring Biodiversity for Addressing the Global Biodiversity Crisis has just been released and is available online:
https://nap.nationalacademies.org/catalog/29211/measuring-biodiversity-for-addressing-the-global-biodiversity-crisis-summary-of
(you can download as guest, but need to share an email address)
I’m excited to share our new paper, Developing custom computer vision models with Njobvu-AI: A collaborative, user-friendly platform for ecological research, published in Ecological Applications. We introduce a software tool for image annotation and training CV models and present the development of a multiclass detector model for processing camera trap images from Malawi. As this is geared towards an ecology audience, we focus on useful interpretations of model performance and discuss different approaches for model development.
🖥️ There has been lots of great discussion about re-inventing the wheel with software tools. We hope Njobvu-AI is complementary to other options out there and provides a gentle introduction to CV models for ecologists. While our example uses camera trap data, this workflow could be particularly useful for other image modalities that don’t already have generalizable detectors available like MegaDetector.
🐘 Our dataset of annotated images (Nkhotakota Camera Traps) is available on LILA BC thanks to @Dan Morris
🔗 paper: https://esajournals.onlinelibrary.wiley.com/doi/10.1002/eap.70096
🔗 GitHub: https://github.com/sullichrosu/Njobvu-AI
🔗 dataset: https://lila.science/datasets/nkhotakota-camera-traps/
Hello everyone, please check out our new technical report on using AI to detect lost marine fishing gears and nets: GhostNetZero: AI for Detecting Marine Ghost Nets . This is a first step of our ghost net project. Our next step is to scale it up and add more types of ghost gears. Please let us know what you think about it. Thank you very much!
Hey @Zhongqi Miao super cool. We had a similar project at Ai.Fish in collaboration with DFO Canada focused on detecting ghost lobster traps and ropes in side-scan sonar. Would be happy to chat more if it would be useful to share results!
@Justin Kay Totally! We would love to talk with you! What time would work for you? I can send you an invite.
There are lots of publicly-available models out there (e.g. HuggingFace Models or Pytorch-Wildlife model zoos) -- potentially, this means you can now use AI to analyze your conservation data without having to train your own model. But which pre-existing model should you choose?
Today we are introducing CODA, a method for active model selection that makes this question interactive, iterative, and very efficient. Instead of needing to curate and label a large test dataset to find the best model for your data, CODA tells you which data points to label to find the best model for you as quickly as possible (often with as few as 25 labels required).
CODA will appear at ICCV next week - come find us at Poster Session 1 and Demo Session 6!
Paper: https://www.arxiv.org/abs/2507.23771
Github: https://github.com/justinkay/coda
MIT News coverage: https://www.csail.mit.edu/news/3-questions-how-ai-helping-us-monitor-and-support-vulnerable-ecosystems
And importantly! We built a fun demo for finding the best species classifier for camera trap imagery: https://huggingface.co/spaces/justinkay/coda
Give it a try 🙂
i'd like to give this a try for our airborne data. Does it make sense to also use it in iterative active learning manner, or just among competing models?
Do you mean combining it with active learning - updating/re-training the models as you go?
are the data points for model selection different than the data points for finetuning.
In general yes they will be different. Typically the best data points for model selection are the ones that have high disagreement between candidate models. These may or may not be similar to the ones that active learning would pick, depending on the AL method you're using.
Something we've talked about a lot (cc @Sara Beery) is the more general version of this that combines active learning, testing, model selection, etc -- given some labeling budget, which data points should you label, and what should you use them for?
@Ben Weinstein I know you've thought about this a lot too, particularly some of the statistical messiness you can get into in the worst case scenario
Would love to hear what your current thinking around the problem is!
we've tried lots of the workflows. almost all suffer from the same problem. It is very hard to develop features for fine-grained differences. So if you are studying birds in the gulf of mexico, and you want to mine your data for unseen/rare species that appear in 1 in a million images, its extremely difficult to use any clustering, feature manifolds, or decision boundary to unearth these valuable images. What almost every method does is return weird, not-useful, things that the model hasn't been trained or conceived of. Cones, boats, drifting objects, the active learning needs to be, we want things that are different, but in this specific way, not just novel outliers in feature space. I actually think VLMs with their strong general knowledge are successful here. We've seen this in the trees in hyperspectral as well. The outlier/active learning idea for finding rare trees mostly just returns houses, swimming pools and other open-set structures because it doesn't have enough data to train for these really rare classes, which of course is the point, if the features were robust to these differences already, you wouldn't need to mine for them!
@Rupa Kurinchi-Vendhan also relevant to some of the conversations we've been having
Could synthetic data be generated in a targeted way for these rare cases?
We've written proposals for this kind of thing but not done too much more than theorize.
@Matt Weldy and I have had similar thoughts
This adds an interesting dimension to workflow thoughts. We did put in a proposal similar to the one Ben mentioned and spoke with a few other researchers using synthetic data along the way. I am hopeful that synthetic data can play a role in this, because, as Ben pointed to, it is often the really rare classes that we want to surface.
I've had success working through the AL workflow for a number of targets (in a bioacoustic context). But I recently ran into a failure case, where the class is sufficiently rare that the AL workflow just continues surfacing FP failures. It is definitely important to find those, but at some point, with rare classes we need to find a few more positives as well. In this specific case, I suspect we may have surfaced all the examples from this dataset. But, it is tough to say that with certainty, given it is a pretty large collection of acoustic monitoring data.
Another distinction I will throw in the mix here, is that I've spent most of my time thinking about the use of synthetic data in the context of feature discrimination (e.g., using transfer learning approaches over frozen features). The use of synthetic examples and routines for choosing among base models may vary for feature learning (i.e., training embedding models) and training classifiers.
Now flagging @Julia Chae re:synthetic data for rare stuff
Sara, you need a Bat-Signal to call in the full lab!
Next group meeting I expect everyone to decide which DC character they are
Hey Ben! We’ve been thinking about this a lot from the bioacoustics perspective—why foundation models may fail for underrepresented or novel taxa/domains and how we can address that to better represent rare/unseen classes in the embedding space. Here’s an active labeling paper you might find interesting: https://arxiv.org/abs/2505.03071 … Of course this is a space where synthetic data generation is underexplored and isn’t fully reliable yet (especially we don’t have ways of evaluating how realistic outputs are and we have so little data to train on)
Imageomics crew is also thinking about synthetic data generation for rare taxa using BioCLIP2 as it has internal taxonomic hierarchy representation at every taxonomic level so the generative version can also work at any taxonomic level. We should all talk about this
Hi, for those at ICCV this year, come and check out our poster: Bridging Domain Gaps for Fine-Grained Moth Classification Through Expert-Informed Adaptation and Foundation Model Priors
Here, we explore practical methods to reduce the domain-gap for lightweight classification models trained on moth images from GBIF and images collected in the field from automated insect monitoring (AMI) systems. We investigate the impact of mixing precious expert-labelled examples in the training set alongside knowledge distillation from the extensively pretrained BIOCLIP2 model. These strategies are complementary.
This is in the short paper archival track for the Computer Vision for Ecology workshop. Full text: https://arxiv.org/abs/2508.20089
This work was largely completed through a two-week short-term scientific mission (STSM) through the InsectAI COST action. My visit to Aarhus University was practical, loads of fun and very informative; many thanks to Toke Høye for hosting me. If you get the chance to visit, do it!
For coral enthusiasts (and beyond), and fresh from the press!
We stress tested our dear DeepReefMap on 365 video transects on 45 diving sites and found remarkable consistency wrt time and acquisition conditions! This is a wonderful achievement of @Jonathan Sauder and shows that the model is robust!
https://www.nature.com/articles/s41598-025-20795-z
Don't know how many people have poked around with SAM1 or SAM2, but FAIR just a couple hours ago posted SAM3. It's still not quite perfect, but it's definitely closing the performance gap on my groups in-house models.
- whitepaper: https://ai.meta.com/research/publications/sam-3-segment-anything-with-concepts/
- playground: https://www.aidemos.meta.com/segment-anything
And finally with underwater data! Thanks to MBARI/Fathomnet!
And they used WCS Camera Traps (from LILA) too!
plus a bunch of camera trap videos: https://arxiv.org/pdf/2511.15622
https://www.conservationxlabs.com/sa-fari
(with very meticulous annotations!)
Related to the SAM3 release yesterday, we also just released a comparably large camera trap video dataset (SA-FARI - yes, the acronym is forced) that was used to train SAM3 plus an accompanying short paper. There are other large video datasets @Otto Brookes @Valentin Gabeff @Jenna Kline @Hemal Naik etc. But this one particularly focuses on extremely meticulous annotations for multi-animal tracking + a wide range of species (and camera traps of course). It is VERY 'jungle leaning' but hopefully we can grow this to other biomes! Special shoutout to @Otto Brookes @Maximilian Schall @Christopher Beirne and many others that (might be) in this channel!
Hi all,
Here is the arXiv paper of our new earth-observation foundation model. We have already applied this model for a variety of applications this group may be interested in including high precision ecosystem mapping, deforestation cause mapping in the Amazon, wildfire risk assessments, crop-type mapping, and more. We have also written tutorials for producing embeddings (a la Google/AlphaEarth), and published a variety of fine-tuning models/recipes as well. Many details/more results in the paper. Let us know if you have any feedback!
- arXiv: https://www.arxiv.org/abs/2511.13655
- Code: https://github.com/allenai/olmoearth_pretrain
- HuggingFace (weights): https://huggingface.co/collections/allenai/olmoearth
- More info at Ai2 website: https://allenai.org/olmoearth
Hi Patrick, minor comment in case you missed it, the abstract links are not well formatted in arXiv (the paper is fine):
...OlmoEarth source code, training data, and pre-trained weights are available at $\href{this https URL}{\text{this https URL}}$."
Hey @Gerard thank you so much! we are fixing it now.
Hello all,
I’m pleased to share my latest research, "A Framework for Detecting and Tracking Elephants in Drone Videos," just published in Drone Systems and Applications.
The Problem: Manual processing of drone footage for large mammal monitoring is often unscalable.
Our Solution: We introduce an automated pipeline integrating YOLOv11 for detection and BoT-SORT for tracking, enhanced by a custom re-identification algorithm to handle temporal dynamics.
Key Outputs: Automated counts, individual speed estimation, and cluster statistics.
We hope this framework helps streamline data-driven decision-making for wildlife conservation. You can read the full paper here: https://cdnsciencepub.com/doi/10.1139/dsa-2025-0032
Hi all - my mentee, Rohan Gowda Thanh Quang (senior analyst at Aurora Energy), just published a paper showing that $128B in climate tech venture capital is systematically overlooking high-impact solutions, especially nature-based solutions which can benefit biodiversity conservation and human well-being.
📄 Read the full paper here (no paywall!): https://doi.org/10.1088/2753-3751/ae1b08
If you are on BlueSky or LinkedIn, please consider amplifying Rohan’s work. Happy winter holidays in advance everyone!
📰 Excited to share that our work has been published in Methods in Ecology and Evolution!
✨ MaskSDM with Shapley values to improve flexibility, robustness, and explainability in species distribution modelling
As biodiversity declines, deep learning can help us understand where species occur and why. In this work, we introduce MaskSDM, a new multimodal masked-modeling approach that enables us to model the distributions of 12,738 plant species worldwide 🌱
MaskSDM naturally handles missing environmental data and allows us to generate predictions using any subset of variables or modalities. We also introduce a new method for computing Shapley values, making it possible to identify more precisely which environmental factors drive each species’ distribution. Using the transformer architecture, MaskSDM is designed as a general and scalable tool for ecologists and anyone working with biodiversity data 🗺️
📄 Paper: http://doi.org/10.1111/2041-210x.70200
💻 GitHub: https://github.com/zbirobin/MaskSDM-MEE
Huge thanks to @Nina van Tiel, Gencer Sumbul, @Chiara Vanalli, @Benjamin Kellenberger, @Devis Tuia for making this project possible 🙌
Deep learning and SNR for bird sound detection in Ecological Indicators !! 🐦🐦
𝗔𝗰𝗼𝘂𝘀𝘁𝗶𝗰 𝗱𝗲𝘁𝗲𝗰𝘁𝗶𝗼𝗻 𝗼𝗳 𝗮 𝗻𝗼𝗰𝘁𝘂𝗿𝗻𝗮𝗹 𝗯𝗶𝗿𝗱 𝘄𝗶𝘁𝗵 𝗱𝗲𝗲𝗽 𝗹𝗲𝗮𝗿𝗻𝗶𝗻𝗴: 𝘁𝗵𝗲 𝗰𝗵𝗮𝗹𝗹𝗲𝗻𝗴𝗲 𝗼𝗳 𝗹𝗼𝘄 𝘀𝗶𝗴𝗻𝗮𝗹-𝘁𝗼-𝗻𝗼𝗶𝘀𝗲 𝗿𝗮𝘁𝗶𝗼
With Jérôme Sueur, Frédéric Sèbe, Maxime Le Cesne, Sylvain Haupert, we investigated the impact of the signal-to-noise ratio (SNR) on the convergence and performance of 𝗖𝗡𝗡𝘀 for bird sound detection.
The goal of this work is 𝘁𝗼 𝗾𝘂𝗲𝘀𝘁𝗶𝗼𝗻 𝘁𝗵𝗲 𝗮𝗰𝗰𝘂𝗺𝘂𝗹𝗮𝘁𝗶𝗼𝗻 𝗼𝗳 𝗲𝘃𝗲𝗿 𝗰𝗼𝗺𝗽𝗹𝗲𝘅 𝗽𝗿𝗼𝗰𝗲𝘀𝘀𝗲𝘀 to try to counter insurmountable physical limits to on the contrary encourage 𝘀𝗶𝗺𝗽𝗹𝗶𝗰𝗶𝘁𝘆, 𝗿𝗼𝗯𝘂𝘀𝘁𝗻𝗲𝘀𝘀 and 𝘀𝘁𝗮𝗯𝗶𝗹𝗶𝘇𝗮𝘁𝗶𝗼𝗻 of 𝗖𝗡𝗡𝘀 because despite being used for almost 40 years, they remain cryptic objects that can fail very much.
article: https://www.sciencedirect.com/science/article/pii/S1470160X25014074
Dataset: https://zenodo.org/records/15528666
github: https://github.com/ear-team/MICHAUD_CNN_SNR_BOREAL_OWL
[duplicate from general thread]
🚀 New Publication –Automated species recognition for tropical forests monitoring!
Hi all! Very happy to share the release of 𝗗𝗲𝗲𝗽𝗙𝗼𝗿𝗲𝘀𝘁𝗩𝗶𝘀𝗶𝗼𝗻, an open-source algorithm for camera trap photos and videos from African tropical forests. Team work at the French Museum of Natural History and the One Forest Vision initiative with my primatologist colleague Sabrina Krief and 16 co-authors.
📚 Research article accessible in open-access in the BES Ecological Solutions and Evidence journal
https://besjournals.onlinelibrary.wiley.com/doi/10.1002/2688-8319.70167
💻 Model weights and Python inference code available on the project GitHub page
https://github.com/MNHN-OFVI/DeepForestVision
DeepForestVision is also acccessible in the very handy AddaxAI interface from @Peter van Lunteren
Thanks also to @Dan Morris, @Sara Beery and the Microsoft team for making MegaDetector available for the detection step 🙏
We have been the using DeepForestVision in Kibale national park, Uganda for a year, with overall convincing performance, and identified some biaises/confusions that I will try to improve in an upcoming V2. Any feedback will be very useful!
I'll also be happy to demo the model if needed, with the hope that it can be useful to research and conservation projects 🙂
New paper: Visual-textual Dermatoglyphic Animal Biometrics: A First Case Study on Panthera tigris
https://arxiv.org/abs/2512.14878
Code+data: https://wenshuoli001.github.io/vttigerbio/
look forward to the code release. But some really interesting points are shared here.
@Aakash Gupta Many thanks. Code and project page, now in main announcement.
🌍 New publication in Methods in Ecology and Evolution — with the great @Hager Radi and @Mélisande Teng
CISO: Species Distribution Modelling Conditioned on Incomplete Species Observations
We introduce CISO, a deep learning SDM that can condition predictions on any available species observations alongside environmental variables, capturing complex and asymmetric species interactions using a transformer.
Tested on plants (sPlotOpen), birds (SatBird), and a new butterfly dataset (SatButterfly), CISO shows clear performance gains when partial biotic information is available, and can even benefit from combining multiple datasets.
Overall, CISO offers a flexible and scalable way to incorporate biotic interactions into SDMs, without requiring complete species inventories 🌱🐦🦋
📄 Paper: https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210x.70238
💻 Code: https://github.com/RolnickLab/CISO-SDM
🔗 Website: https://zbirobin.github.io/publications/ciso/
🙏 This was a large joint effort across MILA, Université de Montréal, McGill, and EPFL, including @Laura Pollock, @Hugo Larochelle, @Devis Tuia, and @David Rolnick
Hi everyone!
New publication - How many dollars are in the sea? Estimating sand dollar (Echinarachnius parma) abundance using an iteratively trained convolutional neural network and generalized additive models
I'm happy to share my first publication on building a human-in-the-loop program to increase the accuracy of our machine learning model and speed up the manual annotation process! Big thanks to my advisors Dr. Hart and Dr. Chang.
Paper: https://www.sciencedirect.com/science/article/pii/S1574954125003206
Hi everyone 👋
ICTC is kicking off tomorrow with pre-workshop today, so we are very pleased with the timing of this!
While I won't be there in person, I wanted to contribute to the discussion by sharing a piece of work I’ve been developing in my PhD. The preprint has just been uploaded to bioRxiv. 💥👀
📄 Preprint "Future of Edge AI in Biodiversity Monitoring." : https://arxiv.org/abs/2602.13496
In this review, we attempt to build a shared vocabulary between ecologists and engineers, and to synthesise the challenges and opportunities shaping edge AI systems for biodiversity and conservation. 🐙:female_technologist:
Hopefully, it brings some additional context for the ICTC conversations, especially those focused on edge computing.
I would very much welcome your thoughts, critiques, and ideas. Wish you a happy reading and/or happy ICTC !! 🌎🌵🤖
🚀 New paper out ➡️ Introducing TropiCam‐AI: A taxonomically flexible automated classifier of Neotropical arboreal mammals and birds from camera‐trap data
📄 Paper: https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210x.70213
🔗 Project page: https://github.com/andrewzamp/TropiCam-AI
💻 Demo: https://huggingface.co/spaces/andrewzamp/TropiCam-AI-demo
We trained TropiCam-AI on several camera-trapping dataset spanning multiple countries in the Neotropics, and tested if including iNaturalist images for targeted species can fill training gaps for data-deficient classes and increase overall model performance (spoiler: it can!). We also included in our evaluation the taxonomic aggregation strategy presented in the SpeciesNet paper, showing how great it is in enhancing generalizability and usefulness of predictions.
📷 🐒 🐦 TropiCam-AI is a model specifically designed for arboreal camera-trap deployment in the Neotropics, which is an amazing (and very challenging) means of surveying arboreal vertebrate communities. Our model is open-source, and already available on the mighty AddaxAI platform.
🙏 A huge thanks to all the collaborators that made this work possible, and a special acknowledgement goes to @Peter van Lunteren for its incredible patience and efficience in integrating TropiCam-AI in AddaxAI!
@Yu-Cheng Wei joined #new_papers. They’re also new to AI for Conservation.
ProM3E: Probabilistic Masked MultiModal Embedding Model for Ecology was recently accepted to CVPR 2026!
We introduce ProM3E, a probabilistic masked multimodal embedding model for any-to-any generation of multimodal representations for ecology. ProM3E is based on masked modality reconstruction in the embedding space, learning to infer missing modalities given a few context modalities. By design, our model supports modality inversion in the embedding space. The probabilistic nature of our model allows us to analyse the feasibility of fusing various modalities for given downstream tasks, essentially learning what to fuse. Using these features of our model, we propose a novel cross-modal retrieval approach that mixes inter-modal and intra-modal similarities to achieve superior performance across all retrieval tasks. We further leverage the hidden representation from our model to perform linear probing tasks and demonstrate the superior representation learning capability of our model.
New paper at ICLR 2026 :flag_br:
MIAM: Modality Imbalance-Aware Masking for Multimodal Ecological Applications 😋
We propose a new masking strategy that improves robustness to both entire missing modalities (e.g., cloud-covered satellite imagery 📡) and partial missing data within modalities (e.g., gaps in time series 📈). Compared to existing masking approaches (e.g., 4M), MIAM explores the full space of input combinations while prioritizing important subsets. It also explicitly addresses modality imbalance by dynamically adjusting masking probabilities based on per-modality performance and learning dynamics.
Models trained with MIAM can be evaluated under any combination of available inputs — enabling fine-grained analysis of which data sources, variables, time segments, or image patches drive predictions.
Discover more:
📄 Paper: https://openreview.net/forum?id=oljjAkgZN4
🌐 Webpage (with interactive visualizations): https://zbirobin.github.io/publications/miam/
💻 Code: https://github.com/zbirobin/MIAM
Excited to share this collaborative work with Wesley Monteith-Finas, Gencer Sumbul, @Nina van Tiel, @Chiara Vanalli, and @Devis Tuia 🙌
@Kate Reed joined #new_papers. They’re also new to AI for Conservation.
@Taneea S Agrawaal joined #new_papers. They’re also new to AI for Conservation.
@Ronan Wallace joined #new_papers. They’re also new to AI for Conservation.
This is my semi-regular reminder for everyone to post papers here. Finding literature is getting harder and harder. I think some of us might be shy but don't see it as self promotion, I know I'm not the only one who has nearly given up with staying up to date with scholar, it's too much.
@Ben Weinstein here you go https://arxiv.org/abs/2602.08131 https://arxiv.org/abs/2603.02386
@Maria Camila joined #new_papers. They’re also new to AI for Conservation.
@Lucas Mateo Waldburger has joined the channel
@Edwin Arkel Rios joined #new_papers. They’re also new to AI for Conservation.
@Jason Holmberg (Wild Me) has joined the channel
@Moeri Okuda joined #new_papers. They’re also new to AI for Conservation.
New paper out! 🎉
Large language models possess some ecological knowledge, but how much? Now published in Ecological Informatics.
We probe the ecological capabilities of LLMs across a range of tasks, from predicting species presence to classifying threats and estimating traits.
Big thanks to my collaborators Joe Millard, @Drew Purves, Michael Harfoot, and @Oisin Mac Aodha.
📄 https://www.sciencedirect.com/science/article/pii/S1574954126001056
Super cool! I'm wondering how you protect the benchmark from becoming training data, so that it can continue to serve as evaluation
Great question! Our ground truth comes from existing public databases (IUCN, AVONET ...), so we can't rule out that some of this data has been seen by models during training. We do think the low performance across many tasks is at least suggestive that models aren't simply recalling answers. The main thing working in our favour is that the benchmark is grounded in structured, updateable data sources, so in principle it could be extended or refreshed over time. But you're right that contamination is an open problem here, as it is for most static benchmarks.
We introduce BioVITA for biological visual-textual-acoustic alignment at CVPR 2026 !
https://arxiv.org/abs/2603.23883
✔︎1.3M Audio, 2.3M Images for 14k Species
✔︎BioVITA Tri-Modal model
✔︎Multimodal retrieval benchmark
Let’s talk about migrating Monarchs 🦋 https://onlinelibrary.wiley.com/doi/10.1111/gcb.70805
In this work our great @Chiara Vanalli models distributions of Monarchs across seasons in Northern and Central America and studies niches projecting them to the end of the Century!
Great work by the DeepSDM team, with @Robin Zbinden and @Nina van Tiel
Fresh from the press at Global Change Biology!
(slightly out of scope)
Hello everyone,
As an independent researcher, I just had my first (tool) paper accepted at ICST2026 with a focus on automated debugging and fixing of machine learning workflows. This paper is a first step in my research towards an automated system for ML model development in the light of Karpathy's autoresearch (https://github.com/karpathy/autoresearch).
I'd very much appreciate additional feedbacks as I prepare for the camera-ready version. Preprint here: https://arxiv.org/abs/2603.14099
Cheers 😄
hi - in this paper we demonstrate our CAPTAIN reinforcement learning approach to optimize conservation planning strategies in Germany: https://doi.org/10.1016/j.ecoinf.2026.103794
Ecologically-Constrained Task Arithmetic for Multi-Taxa Bioacoustic Classifiers Without Shared Data, accepted at Interspeech 2026
Bioacoustic datasets sit in separate institutions that sometimes can't share raw recordings, whether because of data agreements, sensitive species locations, or sheer volume. We show that classifiers trained independently on birds, whales, and amphibians can be merged into a single 661-species model with no shared data. It works because different taxa occupy different frequency bands, so their learned weight updates come out nearly orthogonal.
𝗣𝗿𝗲𝗽𝗿𝗶𝗻𝘁: arxiv.org/pdf/2605.03914
Interesting work from the DINO team using metadata as weak supervision that shows some results on camera traps
https://arxiv.org/pdf/2606.05107
Code is out for SAM3D Animal for 3D shape and pose estimation of multiple animals from a single image. Project page: georgehux.com/SAM3D-Animal-project-page
The link is broken if you try clicking on it but copy-pasting works
How much are tropical forests suffering from extended periods of heat? This super work led by @Nina van Tiel and @gaston lenczner maps out exceeding thermal thresholds in the tropics! pnas.org/doi/10.1073/pnas.2528622123
Our work on finding evidence for scientific hypotheses in iNaturalist was just published!! Led by Edward Vendrow, @Rupa Kurinchi-Vendhan and @Julia Chae ❤️❤️
Check it out: https://www.sciencedirect.com/science/article/pii/S1574954126002864?via%3Dihub
very cool, is there any place where i can try the tool?
Yes! Our live demo typically lives at inat-search-demo.csail.mit.edu--it's currently down but we are actively troubleshooting. In the meantime, our repo hows how to spin up an offline instance of the tool on your laptop! github.com/Beery-Lab/INQUIRE-Search/tree/main
hi rupa, i vibe-coded a quick colab notebook to fetch your model and run the model. Worked well but it cant access the data on S3 - colab.research.google.com/drive/…?usp=sharing , the app is made visible though and i export the right csvs from the temporary page, i just cant see the original file because its on your s3 i reckon. Pretty neat, if you remember please let me know when the trial version is availabe
this is the temp link i'll keep live for a little bit longer: payment-mice-paint-fat.trycloudflare.com
This is super neat, thanks for sharing!! I'll definitely follow-up once our link is live again
@Nuno Sa The demo is live again at inat-search-demo.csail.mit.edu! Sorry for the delay
What could the next 20 years of wildlife tracking look like, and why should we care?
Our new horizon scan, out today in Proceedings of the Royal Society B, is the first of its kind for movement ecology. Working with 38 experts across six continents, we identified 28 priority innovations across three areas: engineering breakthroughs like next-generation bio-loggers, analytical tools like AI to help predict where animals will move next, and coordination efforts like shared data and tracking infrastructure.
We also mapped these innovations against the global policy frameworks shaping conservation today, including the Kunming-Montreal Global Biodiversity Framework and the UN Sustainable Development Goals, showing where movement data could do more to inform real decisions.
Led by WILDLABS’ Talia Speaker alongside co-authors Sarah Davidson, Roland Kays, Christian Rutz, Jennifer Solomon, Carly Batist, and Stephanie O'Donnell (and me). A huge thank you to the horizon scan participants and broader community whose ideas and expertise shaped this work, and to those who are already taking these ideas forward.
Read more on WILDLABS: wildlabs.net/en/article/out-today-emerging-innovations-movement-ecology-address-planetary-challenges
Access the full paper: doi.org/10.1098/…
new perspective piece from Forest Landscape Ecology lab on the potential impacts of GenAI on scientific workflows and ecological research
onlinelibrary.wiley.com/doi/10.1002/…
We’ve released a new model called BirdCODE which is focused on zero-shot bird sound detection (i.e. it predicts time boundaries, not just species ID!). It’s trained on >9000 bird species. We also release detections on >1M citizen science recordings. To demonstrate applications, we use these detections to look at vocal behavior across species, seasons, geography, and mixed-species interactions. Model weights, detections, and inference code are all open.
Preprint: biorxiv.org/content/10.64898/…
Github Repo: github.com/earthspecies/sound-event-detection
Blog post: earthspecies.org/2026/08/20/birdcode-scaling-zero-shot-sound-event-detection-for-bioacoustics
