This sounds really interesting, but I haven't worked much underwater. What methods are you using currently?

Amazing!! Great work as always ☺️

And one more:

http://lila.science/datasets/snapshot-karoo

LILA BC

Snapshot Karoo - LILA BC

This data set contains 14889 sequences of camera trap images, totaling 38074 images, from the Snapshot Karoo project, part of the Snapshot Safari network. Using the same camera trapping protocols at every site, Snapshot Safari members are collecting standardized data from many protected areas in Africa, which allows for cross-site comparisons to assess the efficacy...

Original URL: http://lila.science/datasets/snapshot-karoo

Incidentally, the best part of my job is picking thumbnail images for camera trap data sets.

And a special shout-out to @Sara Beery, who - aside from kicking this whole workstream off for us ~1.5 years ago - created the video that Slack chose as the thumbnail. 🙂

Hi @Frederic, I’m here, and good find! Just corrected this link and others pointing to GitHub pages that have moved. Thanks :)

Yeah it's been on my todo list for a while too

@Sara Beery I am just about to start transitioning my focus from photos to videos for the BearID Project. I'm going to take advantage of the #AIforEarth grant this year!

Very cool!!! What are you planning to try first?

I plan to compare an OD trained for bears (and other species) with MegaDetector followed by classifier. I will also look to add object tracking to try to maintain individuals across a clip.

I think different smart ways of combining information across the clip to get better IDs is really cool.

I think this fits with this weak attention idea that we are all really exploring.

ecology has this inherent connection in time and space

but due to differing positions and angles we expect classifications/detections to be similar, but not identical.

Exactly. We've been seeing good results (3% boost, on a pretty hard dataset) with Context R-CNN on AVA btw, which is a video object detection dataset

so you would use the features from frame x to inform proceeding and following frames.

I’m moving towards similiar idea with tree detections across years.

we expect that detections in year 1 to inform year 2 to do joint multi-temporal learning.

Ben I was JUST talking about that exact thing with Pietro yesterday

following sara’s general idea, but one-shot detector, etc.

Can we meet up sometime next week and talk about trees?

I've got trees on the brain lately 😉

for sure.

i’m on baby duty all week, let’s do next week.

I'm not planning to do any cutting edge research, just hoping to get something working reliably. My main aim with object tracking is to boost accuracy in individual identification as per Chimpanzee face recognition from videos in the wild using deep learning.

Getting something to work well on a new species IS cutting edge research! Screw all the theorists who would tell you otherwise 😂

@Ed Miller i’m happy to look it over once you have a working pipeline. The idea of cross-imagery feature learning is the focus of alteast 3 projects i’m running, so it’ll be good to share ideas.

Same 🙂

Sounds good. The existing bearid pipeline (for photos) is built around dlib (for historical reasons). For the moment I'm just coming up to speed on Azure. I've run some experiments on OD and classification using Custom Vision. I'll look at some of the existing AI for Earth APIs. I had a link to some other networks from Microsoft (including tracking), but I can't seem to find it at the moment.

Feel free to send a links to models I should be looking at. 😀

Oh, and did I mention I also want efficient models I can eventually run on the edge? As part of my day job (at Arm), I will be testing some models on smartphones, Raspberry Pi's, camera SoCs, microcontrollers, etc.

I found the link for for other networks from Microsoft that I had "misplaced": https://github.com/microsoft/computervision-recipes

GitHub

microsoft/computervision-recipes

Best Practices, code samples, and documentation for Computer Vision. - microsoft/computervision-recipes

Original URL: https://github.com/microsoft/computervision-recipes

yeah, behind the scenes it is basically some ffmpeg and handwaving, then business as usual as if they were just photos from the start. 🙂

Exactly

This is amazing @Dan Morris and @Siyu Yang!!! Really useful :)

I don't, but would also be interested! I relied heavily on Trail Cam Pro when deciding what cameras to buy for my study, but wished there was a better resource for ecology specifically.

Awesome!!

Exited about this!

This worked great -- easy to follow, I was detecting animals from a private data set very quickly!

If you want to play with it, we set up a colab with our Snapshot Serengeti trained models: https://colab.sandbox.google.com/github/tensorflow/models/blob/master/research/object_detection/colab_tutorials/context_rcnn_tutorial.ipynb

colab.research.google.com

Google Colaboratory

Original URL: https://colab.sandbox.google.com/github/tensorflow/models/blob/master/research/object_detection/colab_tutorials/context_rcnn_tutorial.ipynb

These models (both Faster R-CNN and Context R-CNN) were trained on the 1st 6 seasons, using the training locations recommended on LILA.science: http://lila.science/datasets/snapshot-serengeti

LILA BC

Snapshot Serengeti - LILA BC

Overview This data set contains approximately 2.65M sequences of camera trap images, totaling 7.1M images, from seasons one through eleven of the Snapshot Serengeti project, the flagship project of the Snapshot Safari network. Using the same camera trapping protocols at every site, Snapshot Safari members are collecting standardized data from many protected areas in Africa,...

Original URL: http://lila.science/datasets/snapshot-serengeti

They were both trained on TPUs, and all hyperparameters are specified in the config files of the models, locations of the downloadable models can be found in the TFODAPI model zoo, or in the colab 🙂

If you have other Snapshot Serengeti data saved locally, you can use the colab to play with the effect of context. It will use whatever group of images you point to to build up the contextual memory bank, and then use that context when running inference on each image with Context R-CNN.

The demo images we included are from one of the LILA.science test locations (unseen during training), you can see that our baseline model isn't handling this case very well.

can you point to where the model is actually defined?

like the layers, not the colab create_model

Yep! It's here: https://github.com/tensorflow/models/blob/master/research/object_detection/meta_architectures/context_rcnn_meta_arch.py

GitHub

tensorflow/models

Models and examples built with TensorFlow. Contribute to tensorflow/models development by creating an account on GitHub.

Original URL: https://github.com/tensorflow/models/blob/master/research/object_detection/meta_architectures/context_rcnn_meta_arch.py

And attention fns are defined here: https://github.com/tensorflow/models/blob/master/research/object_detection/meta_architectures/context_rcnn_lib.py

GitHub

tensorflow/models

Models and examples built with TensorFlow. Contribute to tensorflow/models development by creating an account on GitHub.

Original URL: https://github.com/tensorflow/models/blob/master/research/object_detection/meta_architectures/context_rcnn_lib.py

Oh, and we took a distillation approach to training, so any images without bbox labels got weakly-supervised bboxes from MegaDetector v3 (any boxes over 0.9 score), matched to the image-level label. We only report performance on human-bbox-labeled (from the bbox json file on LILA) images from the recommended test locations (so we don't end up evaluating on weak labels). Looking forward to testing with MDV4 to see if that boosts performance further, as several iWildCam teams reported increased performance with the new version.

awesome work, Sara!

Hi Sara, is there a PyTorch implementation of this currently?

Nope, sorry! Haha that's what happens when you do work at Google.

If you want to make one please do!!!

Haha I understand. I’ll try to give it a go

Happy to answer implementation questions 🙂

I’m forwarding to colleagues. Long term thought, but i’ll probably push we start from this as a baseline for our new grant for video processing. We are replacing DeepMeerkat (http://benweinstein.weebly.com/deepmeerkat.html) with more recent tools. Hopefully a human-in-loop web component + portal for people to download models to apply to local data. We’ll probably need to think about pruning and reducing the backbone size, but that’s a idea for like december. We haven’t even hired the postdoc on this grant yet.

Dr. Ben Weinstein

DeepMeerkat

Machine learning for video-based biodiversity surveys.

Original URL: http://benweinstein.weebly.com/deepmeerkat.html

Very cool. We'll also want to think about when to store a context feature, because the high frame rate videos will be different than the low frame rate bursts 🙂

yup. I have this vision of combining with the ladder type networks (https://dawn.cs.stanford.edu//2017/06/22/noscope/) I like this idea since especially in wildlife time-lapse the vast majority of frames are almost identical and don’t need a deep learning model.

dawn.cs.stanford.edu

NoScope: 1000x Faster Deep Learning Queries over Video · Stanford DAWN

Original URL: https://dawn.cs.stanford.edu//2017/06/22/noscope/

Awesome.

Yannic Kilcher just released a GREAT walkthrough video for the paper: https://www.youtube.com/watch?v=eI8xTdcZ6VY&feature=youtu.be

YouTube

Yannic Kilcher (https://www.youtube.com/channel/UCZHmQk67mSJgfCCTn7xBfew)

Context R-CNN: Long Term Temporal Context for Per-Camera Object Detection (Paper Explained)

Original URL: https://www.youtube.com/watch?v=eI8xTdcZ6VY&feature=youtu.be

Just watched this now. Amazingly good explanation. Working on my hyperspec attention network. I added in the comments that the code was out.

Not that I'm aware of. Maybe the animal categories in openimages might be useful? https://storage.googleapis.com/openimages/web/visualizer/index.html?set=train&type=segmentation&r=false&c=%2Fm%2F01dws

Yeah, maybe I can bootstrap something from these 🙂

Perhaps, some of the old-school background modeling approaches could be used to extract masks?

This was actually the subject of my first ever paper back in undergrad (https://ieeexplore.ieee.org/abstract/document/7532575). Because the frame rate is really low (no greater than one frame per second), because the background plants can also move, and because sometimes the animals don't move, the masks aren't great. I think it's reasonable to use as weak supervision, but I was hoping that someone had already gathered labels to accurately evaluate methods against. Looks like I'm going to need to label some data, at least for eval :)

I see. Might be an interesting research topic to use all the sequence data + bounding box to learn a camera specific background prior and then do use something like GrabCut to extract the segmentation.

Yeah, that sounds like a great idea. I'm mentoring an incoming PhD student of Pietro's (Laure Delisle) and she's looking at weakly-supervised segmentation for camera traps. We were thinking of seeing if we can adapt the attention-based approach that worked well for detection, but we're in really early stages. Would you have time to meet up and brainstorm with us sometime?

Yeah, happy to!

Cool! She's on vacation this week in Corsica (which I'm very jealous of lol) but I'll chat with her when she gets back and we can find a time to meet 🙂

@Ben Seleb wants to work on this for his PhD!

Conservation X-Labs has an open challenge:
https://conservationx.com/challenge/cameratrap/camera

The issue is cost and scale. It would be great to have an open-source camera trap that allows for customized software, but you need to work out manufacturing at scale to make it work for researchers. Most researchers just want a camera that works and don't want to assemble, program etc. their camera before deployment. If I need 100 new camera traps three weeks from now that are ready to deploy for a project that just got funded, can you have a company manufacture and deliver them for <$150 including hardware and housing?

conservationx.com

Camera Trap 3.0

Conservation X - a site for great ideas, and exponential answers to environmental problems.

Original URL: https://conservationx.com/challenge/cameratrap/camera

Currently trying to lay the ground work for this — happy to discuss!

Hey @Ben Seleb - I'm also interest in learning about what you're up to. I work for the Nature Conservancy and we're very interested in trying out modular, open-source camera trapping hardware.

@Mathias Tobler Thanks for the link to the Camera Trap challenge. I agree cost and scale are always an issue, but so is fit for purpose. AudioMoth is a good example of a working open source hardware model. Custom software doesn't have to be the primary focus, it should just work out of the box for a core set of applications.

openacousticdevices

AudioMoth | Open Acoustic Devices

Home for all information on AudioMoth, a low-cost, open-source acoustic monitoring device.

Original URL: https://www.openacousticdevices.info/

@Ben Seleb I would definitely like to discuss this with you!

@Nathaniel Rindlaub @Ben Seleb - we make Open Source marine camera traps. http://openoceancam.com/

Would be cool to chat!

Open Ocean Camera (OOCAM)

Open Ocean Camera (OOCAM): Underwater Research Camera

The Open Ocean Camera or OOCAM is a modern &amp; open-source underwater camera ideal for marine ocean research and AI powered post-processing!

Original URL: http://openoceancam.com/

@Ed Miller we have projects like that too. happy to chat!

@Maxime Cauchoix would love to check yours out. Any link?

WII has contributed training data to the MegaDetector training data set too!

that is awesome, Saket! well done!

Thanks Bistra! The credit really goes to WII to pull it off. I'm excited to be involved though 🙂.
@Siyu Yang That's wonderful! Thanks for this information. I didn't know that.

It sounds really interesting, but I haven't done anything with PIR.

I did some experimentation with using a PIR sensor as a human detector for a wearable camera. It ended up not being a good solution for a device that is not in a fixed location. I never got to the point of "smart PIR". I suggest you start with a platform like Edge Impulse which makes it easy to experiment with various ML models for sensors (Dan Situnayaki from Edge Impulse did a Tech Tutors talk on WildLabs). If you search for "tinyml pir" you can also find some interesting references.

Thanks for these pointers. I will definitely check these out!

one of the things we looked at a while back was background subtraction from one of these low-power, always-on cameras (and one of these https://www.adafruit.com/product/4469) - @Vivian Shen did most of this work. it had mixed results, but many of the problems that face PIRs (false positives, range etc.) were still an issue. Could definitely be something interesting here though... and happy to help out with anything on this!!

cool. I had looked at the lower resolution 8x8 panasonic GridEye part (adafruit AMG8833). Even without the post processing, the power consumption (4 mA typical ** 3 volt = 12 mW) was an issue. As one one expect, the higher resolution Melexis part is even "hotter" -- something like 60 mW @Vivian Shen how did you evaluate the performance (accuracy) of MLX90640 solution?

@Bob Zak actually sorry I think it actually may have been the grideye.

@Sam Kelly Just found a newer IR sensor coupled with an ULP AI inference chip. See: https://greenwaves-technologies.com/thermal-ir-based-gappoc-b-board-for-people-detection-and-counting/ Development board includes 80x80 IR sensor. Unfortunately, I still think this about 10x the power it needs to be (when operated continously, vs. every 10's of seconds in human counting application). I am making slow progress on an AI inference backend for the standard two (or 4 element) PIR sensor aimed at reducing number of missed animals. Taking a long time to put together a methodology and tools to generate training data and compare to current state of art (which does seem to be improving)

thanks for sharing this! very cool hardware/software combination you have going on here. i have shared it with our team at wild me ... we work with lamave via whaleshark.org and mantamatcher.org, etc.

@Jon Van Oast awesome! We’ve deployed a cam with LAMAVE in the Bohol Strait and are cooking up more deployment plans with Alessandro.

Actually, it would if OOCAM Founder Sidhant and I could jump on the phone with you to chat hardware. We want to know what people with wealth of field experience like you are currently using!

+852 94413850 Whatsapp
mike@mikecee.solutions email 🙏

@Mathias Tobler perhaps the guys at https://medium.com/konvergen might be able to chip in advice. If not, they should be able to put you in touch with someone with practical experience

Medium

Konvergen.AI – Medium

The sharing platform of <a href="http://Konvergen.ai">Konvergen.ai</a>. Visit our homepage at <https://konvergen.ai>.

Original URL: https://medium.com/konvergen

Regarding how many units to use in your final layer, I think that would depend on how distinctive the 50 classes you have are. e.g. imagine if your dataset had only 2 classes, say automobiles and deer from CIFAR10. Samples from those classes could most likely be condensed down to 2d embeddings for the purpose of classification. The more visually similar/complex the classes, the more features might be needed to distinguish between them. If you're getting good performance with 1024 final-layer features, you could reduce it and see if the test set performance falls (indicating you maybe do want to use a larger dense layer to learn enough features), improves (you might have been overfitting the feature extractor to the train set), or doesn't change?

I've seen really nice results recently with EfficientNet, as to learning rate my go-to starting point is a 2K step warm-up from 0 to my initial learning rate, which is usually best around 0.002 or 0.004, then drop the learning rate by 10x after one-two epochs (I do this manually, once the model appears to have converged at the first learning rate) I don't freeze any part of the network during training, I haven't found it to make a difference.

Then I'll do a bit of a sweep over initial learning rates, going up and down by factors of 2, to make sure that I'm not missing something

Once I've got a good set of parameters for this dataset and a good idea of convergence time, I'll switch to a cosine learning rate which usually will give a bit of a boost.

@Sara Beery thanks for the input! I just saw your comments now, but it's good timing. I finished updating my code to use EfficientNet, started some first training runs yesterday but I am still playing around with learning rates. Will try your suggestions. What optimizer are you using?

Just saw this, sorry!! Adam, usually, but I tend to go with what the original code for that architecture used so sometimes I use SGD. In my experience SGD can be a little more sensitive to hyperparameters.

That's awesome! Curious which islands? The webpage just says 7 from 6 countries

We obfuscated the specific island names away prior to release at the request of Island Conservation; they are considered sensitive information. If you have a specific project that would benefit from access to one or more of the specific locations, I would contact David Will (his email is on the dataset page).

But it's only the specific islands that were obfuscated; the countries (or regions in some cases) are in the metadata... they are: Ecuador, Chile, Dominican Republic, Puerto Rico, Micronesia, and Palau.

Hey @Dan Morris, do you know whether a location id (not specific lat/lon coordinates need) and datetime are available anywhere? They do not seem to be part of the json file as with other camera trap datasets. I sent an email to David as suggested but asking here as well in case you know

Hi Peter. You can add wpsWatch (https://wildlifeprotectionsolutions.org/wpswatch/) to the list. It's a web platform for managing connected camera traps and associated data, and it uses Megadetector, a model hosted by silverpond (https://silverpond.com.au/2020/08/14/wps-model-optimization-a-case-study/), and additional AI integrations coming soon. I'd be happy to answer any questions you have about it.

wildlifeprotectionsolutions.org

wpsWatch | Wildlife Protection Solutions

MONITORING WILDLIFE AND DETECTING THREATS IN REAL TIME wpsWatch is our core mobile and desktop app solution for fighting wildlife crime in real time. Using a suite of custom camera traps, sensors, and software, we’ve developed a system capable of providing 24/7 monitoring for any given conservat ...

Original URL: https://wildlifeprotectionsolutions.org/wpswatch/

Silverpond

WPS model optimization, a case study - Silverpond

Report by Bhavesh Sharma. WPS and Silverpond’s HighLighter Wildlife Protection Solutions (WPS) is a non-profit organisation that provides solutions for […]

Original URL: https://silverpond.com.au/2020/08/14/wps-model-optimization-a-case-study/

Awesome, I will add it to the list. Thanks.

FYI I try to keep this up to date:

http://aka.ms/cameratrapsurvey

Feel free to grab content, and let me know what I'm missing!

@Petar Gyurov just seeing this, seems like a great list. Here's an addition if you're including camera trap videos (also mentioned in Dan's list above): Zamba is the open source python project, ZambaCloud is the web application built on top that handles execution on the cloud. Will be doing a round of additional development in the next six months, e.g. to facilitate retraining

Thanks @Greg Lipstein, I will add it to the list 👍

@Dan Morris @Siyu Yang I know we talked about canopy data at one point. Are any of the datasets on LILA captured from a similar angle?

Unfortunately no, but the camera angle question may be moot, since I'm not aware of any public datasets with any humans in them at all. :(

I'd like it to happen too! I'll bring it up with Dan and Siyu 🙂

yes please!

Yes, we are working on MegaDetector version 5 now, and with extremely high probability, it will be based on TF2. Still a few months out, though; lots of training data to curate before we get to the model-training part. But moving along!

great! hopefully is tf2x

I confess that I parsed "tf 2x" in your question as "TF 2.x", when now I realize that I think you intended it as "TFX in TF2.x". On that I have no comment, since I just learned from this thread that TFX is a thing. :)

I also thought you meant tf 2.x, which is what I meant 🙂

yes, I meant tf 2.x sorry for the confusion :)

(Also in case this applies, people tell us that you can load the model in a TF2 environment by using the compatibility package in the import statements - see this issue tensorflow-gpu version · Issue #246 · microsoft/CameraTraps (github.com) )

pbick (https://github.com/pbick)

#246 tensorflow-gpu version

<p>Hello, just a quick question about the supported tensorflow-gpu version. I am having installation issues with version 1.13.1 and was wondering if there is a more recent version that works, or if it must necessarily be 1.13.1. Thanks!</p>

Comments

2

<a href="https://github.com/microsoft/CameraTraps">microsoft/CameraTraps</a>

Thanks I'll give it a try

Hi, just one word to mention that we (here in CNRS France, with @Noa Rigoudy) would be very happy to test MegaDetector version 5 when it will be available. Super usefull work @Dan Morris @Sara Beery @Siyu Yang If we can help...

That was a long time ago, so I don't remember specifically, but I can tell you the most common reason this happens in general, and it's because the only species that occur at those locations have so few occurrences at other locations that meaningful classifier training and/or validation would be impossible for those species. Does that compute here? Are these locations with either very few images or bespoke species?

Not really. I think some locations could be considered as having few images (if ~100 images counts as "few"), but most have thousands and for quite frequent species. Take for example this missing location:

"4334": {
"human": 1226,
"motorcycle": 40,
"meleagris ocellata": 180,
"leopardus pardalis": 100,
"empty": 95,
"procyon lotor": 590,
"odocoileus virginianus": 80,
"didelphis sp": 370,
"dasyprocta punctata": 170,
"urocyon cinereoargenteus": 170,
"unknown": 40,
"claravis pretiosa": 30,
"agouti paca": 119,
"leptotila plumbeiceps": 10,
"equus ferus": 10,
"puma yagoroundi": 20,
"panthera onca": 10,
"crax rubra": 10,
"dasypus novemcinctus": 20
},

Out of these 19 classes, 16 are from top 100 in WCS, i.e. classes with at least 800 images in the entire dataset.

I'm attaching a file with more (but not all) missing locations from "recommended train/val/test splits".

@Dan Morris I didn't make the recommended splits for these so I'm not sure, but one possibility is that more data was added to the dataset later on and the split wasn't changed?

The WCS dataset was not updated anything other than minor bug fixes... Stefan, I'm sorry to say this one might just remain a mystery. Did you happen to compute the total number of images that are excluded from the splits? If it's <10%, I recommend we add all the missing locations to "train" and call it a day. If it's more substantial, we might consider a v2 of the splits (leaving the old one available as well); to my knowledge no one has published using the existing splits. Sorry we don't have a clear answer here!

FWIW since we posted those splits, I've become a little less enthusiastic about including a "test" split in public splits like this; the concept of "test" doesn't really make sense when everything is public, or at the very least is a little misleading. So maybe it's a blessing in disguise if we decide to re-split into just train/val.

agreed re: train/val

According to my calculations, the proportion of images excluded from the splits is around 35%. That being said I wasn't planning on relying on the recommended splits anyway. I was just investigating the data a bit. :-)

can we see a sample image?

All of the images will be available online in the next 6 months under CC license at https://library.big-bee.net/portal. Here are some prototypes (https://ucsb.box.com/s/m0bzpme4xhg0stbx99e6a5kkjcgf85sn)

Not camera trap data, but this work comes to mind:
https://github.com/vladan-stojnic/Detection-of-Small-Flying-Objects-in-UAV-Videos

Also not camera trap data, but close to what you're asking about, and labeled with pretty rich annotations:

https://lila.science/datasets/boxes-on-bees-and-pollen

LILA BC

Boxes on Bees and Pollen - LILA BC

Overview The goal of the BeeLivingSensor project is to non-invasively track honey bees at hive entrances, and to track the type and volume of pollen they bring into the hive. By analyzing the color of the pollen and aggregating it with other data, the project aims to to determine the plant biodiversity around the beehive....

Written by

lilawp

Est. reading time

1 minute

Original URL: https://lila.science/datasets/boxes-on-bees-and-pollen

This paper is awesome. It's so useful to see a case study of what the different platforms and models translate to in terms of practical use!!

^Totally agree!!! Very cool paper

Thank you @Sara Beery and @Carly Batist! 🙂

@Juliana Velez i'm forwarding this to my friend/colleague Juan Parra at Antoquia who is also doing biodiversity surveys, but I think in eastern colombia.

Also a word of enthusiasm for this data set: if someone is looking for a dataset to demonstrate that there's a ceiling on the recall we can achieve without looking at sequence information, this is a perfect dataset. I looked at all the MegaDetector misses on the whole dataset, and they are more or less the same cases where a human can't see the images in an individual image either: you have to see the fuzzy blob moving behind the bushes. I also think that a sequence-based system doesn't have to "beat" an image-based system to be super-useful for cases like this, it could complement it to improve overall recall. So I'll buy anyone a granola bar who can run something like Sara's Context R-CNN work against this dataset and find a bunch of little fuzzy moving blobs that MegaDetector misses, even if the precision you need to get to that recall is a lot lower, and even if the sequence-based system misses a bunch of non-moving things that you can see perfectly well in single images.

Awesome! Would it be possible to get a quantized model so I could run this on a raspberry pi? Or is that a totally bonkers idea? 😇

MDv5 was exported at the end of training using the standard Yolov5 export process, which quantizes to FP16. Depending on how many triggers you typically get (e.g. per minute) on your camera, it's definitely plausible that this will run fine on a Pi device. A few folks even did this with MDv4, which was much heavier. So if you're up for it, give it a try and let us know what happens in terms of processing times. You may be able to run the full .pt file, but Yolov5 also supports exporting to TFLite, and then you might follow a tutorial like this one to run the TFLite model on your device:

https://www.codeproject.com/Articles/5293079/Deploying-YOLOv5-Model-on-Raspberry-Pi-with-Coral

All of that said, MDv5 - just like MDv4 - is trained to favor accuracy over speed and is not designed for embedded systems, so really what we recommend for embedded environments is using MD as an oracle, i.e. to generate bounding boxes on unlabeled images that match the ecosystem where you'll be deploying your embedded device, and using those boxes to train whatever detector best matches your resource constraints.

@Dan Morris Just a word to say that I tested MDv5 on very hard cases (those that MDv4 was enable to handle) and big success ! I'm actually enable to find cases in my big database were MDv5 is failing... Congrats

@Vincent Miele CNRS... that's great! We're always curious about these cases, so if you are able to share a description of the kinds of images where MDv5 does better, or a couple representative images (either here or by email to cameratraps@lila.science), that would satiate my curiosity. 🙂 But in any case, glad to see that you saw improvement!

Sure, I'll do that asap. We stay in touch

Very cool. Do you know if this maps directly to the taxonomy used on iNat?

Yes, the mapping is to the most recent version (at least as of last week) of the iNat taxonomy.

May I ask how you implemented the inference on AWS? I am currently trying to optimize MegaDetector v5 for embedded devices (and ease of deployment) using yolort (see https://github.com/timmh/MegaDetectorLite), that's why I'm interested

Unfortunately, I'm not entirely sure what information to give you since I think most of it won't be relevent to implementation on embedded devices. The best option would probably be to actually dig around in our implementation on our repository:

https://github.com/WildEyeConservation/TrapTagger

We run inference on a number of temporary workers that consist of GPU-equipped AWS EC2 servers. They run what's in the megadetectorworker folder - essentially a launch script that kicks off a docker container that runs inference based in the functions in megaDetector.py (which in turn just pulls functions directly from MegaDetector). I haven't gotten around to really trying to optimise anything too heavily.

Let me know if you'd like me to expand on anything.

Sorry for being imprecise, I was primarily interested in the implementation you used and whether you performed any quantization. Thank you for clarifying!

WILDLABS may also be a good place to post 🙂

Maybe AIDE? @Benjamin Kellenberger

Though AIDE has some internal model training functionality that you wouldn't need

Not OSS AFAIK, but I think meets your other requirements:

https://www.wildid.app/

@Sara Beery I don't know how I didn't find AIDE in my search. On my way to try it out, thanks!
@Dan Morris Thanks as well!

I'm always hawking it here, but you should check put VIAME. One of my colleagues supports NOAA fisheries with almost this exact same workflow. It's completely FOSS, we're just always looking for new users and use cases. Happy to put you in contact with lead

https://github.com/VIAME/VIAME

@Daniel Davila This tool is amazing, it might be my silver bullet!

Hi Valentin, seems like https://github.com/PetervanLunteren/EcoAssist might also suit your needs.

@Peter van Lunteren This looks great, I'd be happy to be a test user for Linux

Thanks for spending time to review all of the projects out there! We are working on an app for reviewing images from light-based moth traps. But it would be ideal to not reinvent the wheel. Here are a couple other relevant open source desktop apps I have come across. These don't have integrated ML (yet), but have more mature graphical interfaces for reviewing and annotating -- which I find to be the more time consuming part to develop.

Timelapse Image Analyzer
https://saul.cpsc.ucalgary.ca/timelapse/

Naturtag
https://github.com/pyinat/naturtag

The MegaDector readme has a longer list of web-based platforms here as well:
https://github.com/microsoft/CameraTraps/blob/main/megadetector.md#is-there-a-gui

If we relax the constraint of editing bounding boxes and ask the more general question "what tools do people use to review camera trap images?", I try to keep updated lists of ML-accelerated tools:

https://agentmorris.github.io/camera-trap-ml-survey/#camera-trap-platforms-using-ml-or-at-least-thinking-about-ml

...and non-ML-accelerated tools:

https://agentmorris.github.io/camera-trap-ml-survey/#manual-labeling-tools-people-use-for-camera-traps

@Michael Bunsen Timelapse has the drawback of not working on Linux AFAIK. Didnt know about Naturtag, thanks!
And as always thanks @Dan Morris for curating this amazing page. It was indeed my starting point in my search.

After reading this thread, I made a very half-hearted effort to run Timelapse on Linux via WINE. I think I got... 70% of the way there? I saw an actual functioning Timelapse window, but then got a stream of incomprehensible .net errors, which seems to be par for the course if you don't know WINE, which I don't. But Timelapse is just a well-behaved .net application, and Photoshop runs in WINE, so I know there's a way. If there are any WINE experts lurking on this Slack and looking for a project...

Thanks for giving that a try @Dan Morris! The Wine HQ forum still looks active if you want to give that a try. Otherwise I'd like to get to know Timelapse better, the features that folks like, and then contribute to porting those to another cross-platform app.

@Dan Morris I was wondering, Is there a step-by-step guide on how to deploy the Megadetector batch API on a virtual machine? The readme of the batch API assumes a little more knowledge than I have

@Valentin Lucet No, sorry, there's not a step-by-step guide. We never documented it to the degree that we document the standard script-based approach to running MD, since it was meant for a relatively small number of deployments that were either internal to Microsoft or internal to collaborators working closely with us (when Siyu and I were at Microsoft).

But... more often than not, the batch API is overkill, and any time you spin up lots of GPUs, you are taking on a high risk of accidental overspend, and you have to really be on top of things to make sure you're not firing off dozens of expensive jobs left and right... what do you anticipate your daily image load will be? If you're not turning around tens of millions of images per day, it's likely far easier to do everything on a single VM, which will get you close to 1M images per day with MDv5 (assuming a V100 GPU). That's a lot of images!

It's also very Azure-specific (unlike the real-time API, which is quite generic)... are you working on Azure?

@Dan Morris Oh np, yes it would definitely be an overkill, I see I misjudged the use case. We have an initial need to process 2-3TB of images before the end of the year.
I was starting to consider the implications of using Azure for our project... But what would be much easier for us would be run MD on a Linux VM of the type described here (with vGPUs).

Indeed, IMO that falls into the category of "more than you want to do on a laptop without a GPU, but easy to churn through on a single machine anywhere you can get a decent GPU". If you're on Azure, assuming that's around 2M-3M images, that's probably ~3-4 days on a V100 GPU.

Hi @Valentin Lucet, I'm curious if VIAME or any of the software you discovered ended up working for.

@Michael Bunsen we went with Label Studio and overall it's going okay but I'm not sure I would recommend it. It seemed the best for our use case (CVAT looks good too but I had lots of issues setting this one up) but I dont think its great. I dont believe there is a solution that tick all thoses boxes and is also not too complicated to set up.

we also ran MD on our own

Thanks for the update @Valentin Lucet! I may give Label Studio a try and customize it for our use case. I like the tech stack they are using. Which aspects is it missing?

Hi Valentin, thanks for sparking the conversation. I am working on an insect monitoring project, partly in collaboration with Kim Bjerge from the 2nd paper you cited. Our current project is focused on machine learning for moth monitoring using images captured by the Easy RIDER system. I see you already found our discussion thread on Wildlabs!
https://wildlabs.net/groups/autonomous-camera-traps-insects

Thank you for finding those other recent papers. I am curious to know more details about your mobile phone setup. I am experimenting with my own versions of that for all pollinators as well. Did you write a custom app for running YOLO on a phone? How may frames per second roughly are you able to process?

Hi @Michael Bunsen, we do not run any edge AI on the phones. They just act as a camera and try to capture one image every second. You can imagine that this produces a lot of images without insects. Therefore, I am curious if someone has developed already some affordable custom camera system that we could use. I also tried to attach some more powerful lenses to the phones to gain optical zoom, but the tests were not that successful, though there is potential. The biggest issue is to have a trigger sensitive to small objects like insects landing on the target flower and can run without the need of a nano GPU (which are almost impossible to order at the moment).

I told @Malika Nisal Ratnayake about this slack, I think he's in the main channel. He may have some thoughts.

@Sachith Seneviratne, thank you very much. These days both I and @Valentin Ștefan are working together on this problem (image capturing and species classification) at iDiv.

Ah I see, wasn't aware, just saw your name on the first message 🙂

Thanks for describing what you have tried @Valentin Ștefan. I think finding the right camera and/or lens will be a good collective challenge. We have the camera pointed at a flat insect sheet and the motion detection works okay for that (although still produces more images than needed). When pointed at pollinator plants the motion picks up spiders (and lots of wind movement!) but doesn't get triggered on most insects. Close focus is a an issue too.

I guess natural flower backgrounds, with wind movements, insects that do not trigger usual IR sensors of wildlife camera traps, insects that are generally tiny and also can vary a lot in dimensions (think of small pollinator flies vs large butterflies) gather to form a complex challenge. Also what works in the arctic conditions where flowers are small and in patches, doesn't work well in temperate regions (imagine a grassland with diverse flowers, of diverse heights and dimensions, especially towards eastern Europe). This year we had plants of almost 2 m height and the flowers were on top. We had to fix the phones on high tripods fixed on some boxes 🙂

Hi!
I look after an open-sourced AI camera trap project based on Raspberry Pi. We call it DynAikonTrap: https://dynaikon.com/trap/
We use a motion trigger based on the vectors created from H264 encoding, suitable for small and fast-moving things without the use of PIR. An insect CNN model could also be easily trained and fit into our detection pipelines. I'm very interested to try out DynAikonTrap for insect detection. Give me an email if you'd like to collab: ross.gardiner@dynaikon.com
We also have one published paper and two masters theses outlining details of our design: https://dynaikon.com/trap-docs/about.html

@Dan Morris @Nicholas Osner

I've mentioned the egregious, bizarre Snapshot Serengeti failures:

https://github.com/ultralytics/yolov5/issues/9294

This is closer to a singularity than an out-of-domain issue. Basically this is what MDv5 does when you run it in the Upside-Down. And some cameras from early seasons of Snapshot Serengeti appear to be the Upside-Down. Still a good example, if the general theme is "machine learning can be unpredictable, so always verify on your own data".

Other than Snapshot Serengeti, 99% of failures I've seen fall into more expected, edge-of-the-supported-domain categories:

• Reptiles (especially snakes)


• Very very far away mammals (particularly prevalent in tundra surveys)

might be a silly question, but are MDv5 (?) and MD5 (the hashing algorithm) the same thing?

also, love the Upside-Down reference 🙂

Sorry, MDv5 refers to MegaDetector version 5:

https://github.com/microsoft/CameraTraps/releases/tag/v5.0

ah got it - thank you!

Hi Sara, I do have some nice examples of where MDv5 fails. I am just awaiting permission from the data owners.

@Sara Beery are you looking for false positives or false negatives? Can probably find some false positives.

Both!!

@Sara Beery I don't know if that counts.. but just came across this on twitter https://twitter.com/matbourbonnais/status/1574870630746722304?t=fJCFHtekIQ5pFGphgl97yA&s=19

twitter

Mathieu Bourbonnais (https://twitter.com/matbourbonnais/status/1574870630746722304)

Bear is such a chonker <b>#megadetector</b> thinks it’s two bears! <a href="https://twitter.com/ShaneyShaneface">@ShaneyShaneface</a>

Original URL: https://twitter.com/matbourbonnais/status/1574870630746722304?t=fJCFHtekIQ5pFGphgl97yA&amp;s=19

Twitter

@Sara Beery Here you go. A selection of false positives. Wonder what confidence people use. I found that with MDv5 one has to go pretty low to avoid false negatives, 0.1 or 0.2. I would rather have a small percentage of FP than having FN, which would require sorting through 100s of thousands of empty images. https://1drv.ms/u/s!Amz2mMAWHLF_5QDqzSppU0CYd37d?e=YVocmC

Good point, I realize that I don't even think of "false positives" as "failure", but there is definitely a false positive rate that can reasonably be called a "failure". I don't see that a lot, but I'm still running MDv5 on all of LILA, and I think there is at least one Snapshot Safari camera that "fails" by this metric.

Hoping to release those results in a few weeks.

Does it "fail" if you use that simple box-location-based filtering as a postprocessing step?

I don't have exact number, but we see FP rates of 1-2%, which means several 1000 images that need to be manually changed to Empty. Not a bit deal given that there are 100s of thousands of TN. We had some datasets from Kenya where MD (v4 at the time) regularly picked up rocks as animals in some cameras, that was an issue. We will check those again with MDv5.

@Sara Beery We have not implemented box location based filtering, but in the examples I sent you the box is no always consistent. Might need to look at box location filtering again at some point.

FWIW I would consider a FP rate of 1-2% a wild success. 🙂. I.e., if you see much less than that, IMO you may be cutting it too close, assuming essentially zero tolerance for FNs.

@Sara Beery for this particular Snapshot Safari camera that I'm referring to, yes, I'm about 99% sure that the typical repeat detection elimination procedure would clean this up without issue. @Mathias Tobler when we do this, we don't require boxes to overlap exactly, but we set a pretty high IoU threshold to consider two boxes the same:

https://github.com/microsoft/CameraTraps/blob/main/api/batch_processing/postprocessing/repeat_detection_elimination/README.md

But it's not perfect, and in particular it can treat sleeping animals as repeat detections, so we always treat it as a very fast but still only semi-automated process.

@Dan Morris I totally agree, 1-2% is a huge success! The performance of MD is amazing! Will take a look at the box post-processing if we find this still to be an issue with MDv5.

@Sara Beery We're using camera traps for research on increased recreational use (people in boats) on rivers in Montana. Camera traps are labor intensive on waterways, as everyone knows, due to reflectance false positives. Our longer term solution is a PIR AI model we use in our own camera traps. In the meantime, we use MD to get rid of empty images. I've noticed that boats get classified as vehicles as an example of a false positive (boxed version is mdv5). Not a big deal for us, but an example nonetheless.

FYI boats were not part of the "vehicle" class for MDv4, but they are for MDv5, so I would argue that image is a slight false negative (re: the trailing smaller rafts), depending on the threshold you're using.

Good point Dan...I didn't know that boats are now vehicles in MDv5...so I guess the trailer tubes would be a false negative example...thanks for all of your work. On a positive note, lots of true positives from MD and elimination of a ton of empty frames, cattle walking by, and the occasional bird...since we have to run these camera traps in 1 minute time lapse intervals (at 15 locations for 5 months out of the year). Whew! You saved us a lot of time. But I'm still hopeful for our Smart Trigger which will decrease labor significantly.

The EXIF data should store this... are you seeing a temperature reading in the frame around your images? I believe it should be in the top right corner. e.g.:

I'm about 97% sure this is stored in the EXIF "makernotes" field for Reconyx cameras, as fields called "Ambient Temperature" and "Ambient Temperature Farenheit". exiftool (https://exiftool.org/) knows about lots of binary makernotes fields, so if you run exiftool my_image.jpg, it will print out all those fields as if they were standard EXIF. exiv2, for example, does not have this magical property. Try exiftool and see if that shows you the temperature properties?

If exiftool shows you the temperature values, we have a script to run exiftool on a folder of images and write the results (all available properties) to .json or .csv:

https://github.com/microsoft/CameraTraps/blob/main/data_management/read_exif.py

YMMV, this has been run a total of 1.5 times.

Yup, I get the makernotes in exiftool. Was using the exifread package in python but for some reason it doesn't output many of the tags...

Thanks for your help :)

FYI as I understand it, exifread misses this because "makernotes" is by definition binary gibberish unless you have a library of manufacturer-specific decoders. exiftool is the only thing I've found that does this well, so even when I need to read EXIF data from a million images, I fork a million invocations of exiftool.exe.

Woah this is great @Sasha Luccioni!

I'm not familiar with this model, but in general this is usually indicative of a mis configuration. Either your model is not set up correctly, or you may not be preprocessing your imagery correctly and therefore the data you are giving it is out of distribution from how it was trained. Does it give halfway decent results (confidence wise and bbox localization accuracy) when you give it an image with a know positive target?

What is "a lot"? FWIW 0.4 is actually a pretty high threshold for MDv5; we usually use 0.15 or 0.2. So if you're seeing, e.g., 10% false positives, I'm going to talk you into trying to get more false positives. 🙂

If you are using the recommended MDv5 inference code, and the false positives are in sensible places on rocks and sticks that maybe kind of look like part of an animal, I would say this is just the cost of doing business with any AI model, and - as I think Michael Jordan said - "if you're not missing, you're not shooting enough" (or maybe he never said that, but he could have). The recommend approach to those false positives is just good-old-fashioned human review, but you can also experiment with automatically or semi-automatically removing detections that occur over and over in the same place on a given camera, a la:

https://github.com/microsoft/CameraTraps/tree/main/api/batch_processing/postprocessing/repeat_detection_elimination

If the false positives are in totally random places in the middle of the sky, something is probably up, and I'm happy to take a look at some of those images.

Blame it on all the animals that are literally trying to look like rocks and sticks.

Fantastic! Thank you for sharing the recording.

thanks so much for this tour of oddities. these edge cases and failure cases are always good for some head scratching.

i am sorta curious about the "moved" annotation cases - i wonder if it is something off in the image metadata. we once saw this where the rotation/orientation metadata (exif or xmp or whatever) was incorrect. (we actually have had a history of huge headaches with orientation! see: phones) ... also, there has been some weird math when the pixel size (or aspect ratio) is funky in the metadata. it would not be all that surprising (depending on which tool you used) if rescaling it simple did away with the questionable metadata, or fixed it in some way.

(though, if these same cameras have produced images that worked fine in other cases, it would be very weird that only some of the exif would be off.)

My first instinct was something about rotation or metadata as well, but I've ruled this out in every way I know how... if I literally screenshot the pixels and save to a new image, the problem persists on these pathological images. Ditto if I convert to .png, or resample from the original size (which is larger than 1280px on the long side) to 1280 px on the long side. It seems to be something about the images themselves, not a metadata issue. This was also a good reminder to check whether Magical New Fixes in more recent versions of YOLOv5 and PyTorch have fixed this, but alas, no luck. (The official MD release doesn't support newer versions of PyTorch, but I did that test using the strategy discussed here, using PyTorch 2.0+ and a current YOLOv5).

The only good news is that in running MegaDetector on literally around 300M images, I have never seen this issue on any other dataset. The closest analogy I can think of is the examples we see in talks about adversarial attacks where imperceptible pixel changes create radical changes in output; these images seem to have the perfect adversarial attack for YOLOv5.

I'm working on classification for reptile camera traps and am new to this slack. I have not played with megadector yet, but would love to connect!

@Toryn Schafer, what reptiles are you studying? Specifically, where do your reptiles land along the axis of "big things you can easily see them in camera trap images" to "tiny things that could land on your head and you wouldn't notice". Asking because I'm sort of ambiently curating training data to hypothetically fine-tune MD to work better on reptiles. Tiny geckos are always going to be out of domain for MegaDetector, but, e.g., the tegus I show on that page should be in-domain, we just need training data to fine-tune a little. In theory. Message me if you have a pile of big-reptile pictures you can share!

There are two questions here... (1) will MD work in foggy conditions? and (2) is there work on detecting foggy conditions?

To (1), there's not a simple answer; I would go with "better than you might think, but fog doesn't help". Overall we haven't seen a significant number of cases where a human could see animals through fog/snow that MD misses. MD has plenty of failure cases that I'm always happy to rant about (I just added a new one to the challenges page, I'll link below!), but this hasn't been a huge issue.

To (2), AFAIK the answer is "no", but this has been something we've been thinking about on the backburner for a long time. The goal would be to count operational days automatically, and generally to differentiate "no animals detected because nothing is there" from "no animals detected because the lens is obscured". My first instinct was that detecting fog/rain/snow (maybe to the degree of "none", "partially obscuring", "totally obscured") might not even require ML, but we weren't able to do this reliably with simple CV heuristics. Idaho F&G provided a snow dataset with rain/snow labels a few years ago, and it was enough to show that our heuristics didn't work, but not really enough to train a model. I just re-started this discussion a few weeks ago with @Mitch Fennell to see if we could assemble a dataset.

If folks have images labeled with rain/fog/snow, and someone wants to take a crack at this, I think it would be a useful and relatively straightforward model! Data is the hard part.

Oh, and I promised a new MD failure case... courtesy of skyislandalliance.org:

https://github.com/agentmorris/MegaDetector/blob/main/megadetector-challenges.md#mega-occlusion-that-humans-are-smart-enough-to-parse-but-ai-is-not

tl;dr: animals are sometimes behind a fence and about 75% obscured by vertical bars, but in a way that human eyes are really good at ignoring.

Thank you Dan, this is really helpful! Agree that simple CV heuristics might be sufficient in this case. I'll check we can share our dataset (though note that it is currently unlabeled...it's also animal-free images of skyline in SF since the camera is trying to document when an air sensor is being disrupted by fog)

@Catherine Breen was doing weather detection models at CV4E last year and might also have additional insight on the obstacles/areas of promise.

At Conservation X Labs, we've been working on a few downstream tasks that tend to depend on image quality. @Dan Morris We also initially tried CV heuristics but didn't have much luck. We spun up a quick web app to annotate image quality but we could always add/modify labels. https://cct-image-annotation.streamlit.app/

@Peter Bermant Very cool! I think I just gave you three annotations. 🙂 That's super-relevant, let me know if at some point you are able to share those annotations publicly, I think a lot of folks here would be interested in playing with that dataset.

I think the ideal dataset for image occlusion due to weather probably looks a little different, i.e. I don't think for that dataset there's a need to crop animals, it's more about the gestalt of the image and pretty specific to weather (or other whole-image camera malfunctions). The one animal-related thing that the hypothetical ideal perfect unicorn of a dataset would really need is a decent distribution of animals smushed right up against the camera lens; that is a common phenomenon in places where snow is an issue, and one of the major reasons that simple heuristics weren't enough in the images I was looking at. Grrrr, curious elk.

Good question... I'm not aware of anything... model zoos are hard! Everyone's environment is different, even the structure of what's being classified is fundamentally different across classifiers (images, crops, sequences, videos), so it's hard to have something super-standard. Plus I think a slight majority of regional classifiers that exist are available as part of a platform, but not for direct download, and therefore are not quite model-zoo-compatible.

But (a) I'd love to try your classifiers, please share links! and (b) I do keep a list of repos in this area, which I think is almost a superset of a hypothetical list of models one could include in a hypothetical model zoo:

https://agentmorris.github.io/camera-trap-ml-survey/#oss-repos-about-ml-for-camera-traps

It wouldn't be too much work to click through those and see which repos include model weights, then either create a separate list or just add an "includes models" tag to that list.

Thanks! Yeah, the variation in everyone's systems does make it tough to serve drop-in models. But having some kind of place to share models where folks know to look for them seems worthwhile...and then maybe in a backwards way find the system that they were designed for. Maybe that would more aptly be described as a "model wilderness" 🙂

The list you have put together is pretty great.

We'll figure out where to put our models in the short-term and we'll share links so that you can have a look.

Did you just invent "model wilderness"? That's good. I'm going to steal that.

That's a great resource Dan, thanks for putting that together. As for the usefulness of a "model zoo", I would assume that most people who work on camera trap image classification still train standard image classifiers either on cropped (usually with MegaDetector) or non-cropped images. If those are based on the standard models (Resnet, ImageNet, EfficientNet etc.) then they could be integrated into different frameworks pretty easily. So having a "model zoo" might actually encourage people to somewhat standardize their tools, share models and make them more accessible to practitioners. Maybe?

@Ian Ingram, I believe Conservation X Labs are working on a model zoo of sorts. The aim in mainly for models compatible with their Sentinel smart cameras, but perhaps for general download as well. @Henrik Cox (Sentinel)?

I am also interested in object detectors and species classifiers for BearID Project. Currently we assume images contain only bears. Eventually we will need to add a detector and/or classifier before hand. We are mainly focused on brown bears in western Canada and Alaska, but are also looking into other bears species like Andean bears.

Thanks @Ed Miller, hi @Ian Ingram, yes we're working on a model marketplace as part of the Sentinel system we're building out to have pre-trained models ready for download, and also as a hub for third party models to be hosted. Right now it's in MVP and only found on the Model Marketplace page if you have an account on our dashboard: mysentinel.info, so there's a lot of work we need to do to populate the zoo with more of our own models. But we do want to make this it's own platform (no need to log into a dashboard) once we've done a few more major deployments of Sentinel hardware devices.

As for hosting third party models, it's able to do that now - we're currently hosting MegaDetector v5a and v5b on there as a proof of concept to it.

MBARI recently started putting their detectors and models for ocean life on Hugging Face. I am considering doing the same! It is becoming a familiar home and interface for practitioners on the ML implementation side of things https://huggingface.co/FathomNet

And you get a free basic page for inference! which is very helpful to get a quick idea of a model's performance on your own images.
https://huggingface.co/spaces/FathomNet/MBARI_Monterey_Bay_Benthic

huggingface.co

MBARI Monterey Bay Benthic - a Hugging Face Space by FathomNet

Discover amazing ML apps made by the community

Original URL: https://huggingface.co/spaces/FathomNet/MBARI_Monterey_Bay_Benthic

Even if they're large enough to be in the size range of what MegaDetector can do, it's definitely hit or miss for both reptiles and amphibians. They're so infrequently studied with camera traps that they're significantly under-represented in MD's training data, so... hit or miss. See:

https://github.com/agentmorris/MegaDetector/blob/main/megadetector-challenges.md#reptiles-and-other-under-represented-species

I should say that sometimes it totally just works, and lots of users have used MD for reptile or amphibian studies. But I'd say for whatever "good enough to be useful" is, if someone really cares about reptiles or amphibians, MD is "good enough" maybe 50% of the time. So we try to make sure to really thoroughly review results on some unlabeled data before recommending that someone should use MD for those use cases.

But this is something I've been really trying to improve lately, not by fixing MD per se, but by making it pretty efficient to train bespoke one-off fine-tuned versions of MD. Do you have a labeled dataset (not boxes, presumably, just image-level labels) with amphibians from your ecosystem? If so, around how many examples do you think you have? And how big are your amphibians?

As Dan suggested you can run MDv5 on some sample images, just to see if it identifies the amphibians in it.
Try running other detectors like Yolov5 or higher versions to see if they can help.
In case you have an annotated dataset. Here is a list of open-source tools, in case you wish to annotate the dataset.

I have a team working on classification of timed images for snakes. We went away from megadetector and are training our own model. We are still in preliminary stages, but are trying to characterize the potential for a CNN in the backend to reduce human workload

Thanks for the info all! @Dan Morris we’re actually still waiting to receive the data, and will be doing some initial testing/playing around once we have it in hand. I’ll shoot you a message to let you know how it works, as well as a couple of sample images. We don’t have anything labelled at this point, but might be interested in training up something bespoke after data is processed from this year!

Not a toad but ive seen MD able to do australian reptiles! Dont have an accuracy score yet though

Hi @Valentin Ștefan, sorry for the late reply! For some reason it didn't notify me of this message.... But to answer your question, no I haven't tried the SAM method. I did try to incorporate Facebook's segment anything model to the automated camera trap classification pipeline (localise, crop, segment, remove background, stretch to fill square, classify), but that didn't yield very promising results because the SAM model wan't very good in segmenting blurry, black-black-and-white camera trap animals... But if you help the model find the object of interest it's actually pretty good.

Really excited to try this! Your documentation is excellent, including the fantastic logo.

First random question... the main page refers to using the TF object detection API, i.e., under "mewc-train", it says:

"This Docker image is used to train an EfficientNetV2 model using the TensorFlow Object Detection API."

...but it also says (consistent with what you wrote in your message here) that you're using EfficientNet for classification. How does the TFODAPI fit in to the training process?

Second random question... you've jumped through some hoops to support both MDv4 and MDv5. Have you found cases where MDv4 performs better, that make it worth all those hoops (and slower inference)?

Thanks Dan, I'll ask Zach (who maintains the GitHub repo) about that inconsistency. We're definitely using EfficientNet in the classifier training, although it would be straightforward to plug-in other CNN. I'm currently working on the next version of MEWC which will use JAX as a back-end, and include option for a ViT classifier, but what we have up right now, using Tensorflow 2.11, works, so we froze it at that during development.

We kept the v4 option in there as more of a legacy, as we initially had problems with v5 on newer GPUs due to some PyTorch incompatibilities. We've since fixed that, but the option to use v4 remains. We default to v5a though, and that's all I use now.

Yep, that was a residual. Fixed on GitHub now, thanks Dan!

Have you explored leveraging an end-to-end solution like the TRAPPER web application? It offers the flexibility to be installed on your local machine or server, providing customizable access for your stakeholders. TRAPPER boasts the capability to detect various objects, including humans, animals, vehicles, and blank images.. Additionally, it extends beyond detection, offering classification capabilities for over 20 European mammal species.

The Citizen Science UI is designed for user-friendly human annotation and expert review processes, making it particularly useful for non-technical and non-ecology-oriented users.

With its robust functionality, open source TRAPPER stands out as a comprehensive tool to optimize your camera trap initiatives


Paper (2016)
https://besjournals.onlinelibrary.wiley.com/doi/10.1111/2041-210X.12571

Documentation
https://trapper-project.readthedocs.io/

Gitlab repository
https://gitlab.com/trapper-project/trapper

Hey @Piotr Tynecki - do you know how i can access the demo?

@Casey Clifton https://demo.trapper-project.org/ register a new account, I will active it

Hi @Piotr Tynecki, is it possible to activate my account as well? I've been very curious to try the demo. notbot@gmail.com

@Michael Bunsen sure, I shared data access on PW

Not exactly an answer to your question, but I’ve heard of some labs allowing people to “adopt” a camera to support grad research. When someone adopts a camera, they pay some flat rate and all the images taken by that camera (minus human photos for privacy purposes) are delivered to the person who adopted the camera. Makes it a little more exciting when you can email someone saying “you’ll never guess what YOUR camera just caught!!”

ooh, thanks! This is a great idea

I actually built an open source tool that does something like this! Uses a text model to generate a dataset and then applies a smaller DINO model for inference. It works pretty well, and I’m able to get it running in real time on an iPhone. Classification not object detection though

I’m late to the party but do you just need to know if a shark is in frame or is the bounding box important?

Hi Nate! bbox is important because we need to extract the patch for a downstream classifier. How come?

@Filippo Varini Did you try to look at the Roboflow / CVAT software? How they resolve this kind of situation could inspire you.

I know how it was resolved in TRAPPER software but the old UI is not so fancy and robus as the new one "Citizen Science" (not published yet).

Hi Piotr, thanks for the suggestion, I will have a look at it!

I use labelme for this:

https://github.com/labelmeai/labelme

I found labelme to be very easy to modify for minor workflow customizations, and it was easy to get data into the format it likes. IMO the goal in making a review workflow efficient is always to get your hands off the mouse entirely, and for what you're describing, I think you could easily get everything down to keyboard shortcuts with minor changes to labelme. I did this for a few camera trap data sets, though I expect the specific changes I made are particular to the fact that I had a small number of objects in each image, and if you have lots of objects in each image, you might need slightly different tweaks:

https://github.com/agentmorris/labelme

I also really like BoundingBoxEditor, though I mostly only use it as a viewer, and my lack of Java-ness makes it hard for me to modify, but it's quite slick:

https://github.com/mfl28/BoundingBoxEditor

Robo-flow and labelme are both great solutions.

Here's a longer list of freely available annotation tools:
https://www.thinkevolveconsulting.com/list-of-open-source-annotation-tools-for-machine-learning-research/

Given how drastically different fire looks from most other things in the photos, I would imagine running a classification algorithm (something akin to the ct_classifier we used this summer) would suffice. Just a binary classifier to say fire vs. no fire?

Hi Cara, try Yolo World — zero shot object detector. It should work well on your data. Here is a link --> https://github.com/AILab-CVC/YOLO-World

thank you both 🙂

Hi Cara. We've thought about doing something similar in adding another class for 'fire' to our YOLO multiclass model for species ID but haven't tried it yet due to limited resources, wanting to build a bigger training dataset (we have about 200 images with fire or smoke), and uncertainty whether it would be a useful approach. Our use case is real-time detection, and I'd be very interested if you come up with something. I'd be happy to contribute our data for training, and I'll let you know if/when we try it out.

I used a different approach in one of our projects. I use a small ViT Q&A model. You need to prompt it with the image an a simple query on whether there is a fire in the image. And it will give you a relevant output.

This is what you will call as a zero-shot approach

There are some companies and NGOs doing this already. Here is one example: https://pyronear.org/en/

@Arthur Caillau thank you!

Added a HF space using a VQA model

https://huggingface.co/spaces/skylord/FireDetector

huggingface.co

FireDetector - a Hugging Face Space by skylord

Discover amazing ML apps made by the community

Original URL: https://huggingface.co/spaces/skylord/FireDetector

If you’re using Python, I recommend trying https://github.com/PyAV-Org/PyAV which is much more flexible compared to OpenCVs video support

Thank you Timm!

@Ed Miller I'm available to discuss TRAPPER and TrapperAI with you (as a Trapper core developer). Our platform now supports object detection, anonymization of humans/vehicles, and species classification (european mammals and birds) for both images and videos as well as Citizen Science mode. TrapperAI module is able to execute any kind of AI model for wildlife in the cloud/standalone, so it should not be a problem to dockerize your models and include them in the pipeline.

Happy to talk 🙂.

One other option I would take a look at, at least for workflow inspiration, even if you don't use the actual code... TrapTagger is also open-source, and their workflow incorporates an individual ID stage (based on HotSpotter that last time I looked, I don't know how modular it is):

https://wildeyeconservation.org/traptagger/
https://github.com/WildEyeConservation/TrapTagger

I can't really speak to the ease of deployment, but the workflow is nice. That said, Piotr's quick response is a good reason to focus on TRAPPER first. 🙂

@Piotr Tynecki Thank you for the quick response! I will follow up with you via DM.

@Dan Morris Thanks for the link to WildEye. I didn't realize they had open source. I'll check it out.

Hi @Ed Miller, I have used TrapTagger a little and incorporate a fancy workflow for the re-id, and also a early integration with Wildbook. But I think only support images, not videos.

Hi @Ed Miller we are also developing a pipeline at SDZWA

https://github.com/conservationtechlab/animl-r
https://github.com/conservationtechlab/animl-py

with integration with CameraBase and eventually other forms of ReID. We process videos by extracting a small number of frames first. It currently uses MegaDetector for bounding boxes and our own trained models for classification, but they are swappable.

Hi @Kyra Swanson. I spoke with @Ian Ingram a while back (via Russ V.). I am very famliar with your great work! In fact, we have used animl-py to test some of your existing models on some of our data. I was not familiar with CameraBase. Is this still under active development?

I am looking to potentially integrate AI packages with TRAPPER, using the TRAPPER AI package they will release soon. TRAPPER already has some integration with MegaDetector and species classifiers like their own model for Europe. What I like about TRAPPER is it is platform independent (runs on Docker), uses a web browser interface, and can be run locally or in the cloud.

It would be great if we can collaborate to build out more models for the Americas and integrate them with relevant software packages like TRAPPER or CameraBase. I believe you are already extending your S. America models, which would be beneficial for work we are starting in Ecuador. We also have data for Western Canada we could use for training.

Happy to discuss here, via email and/or on a call!

Ed

GitLab

trapper-project / trapper-ai · GitLab

Module responsible for the integration and configuration of AI models for object detection and species classification. This component is organized using the Trapper AI Manager web-based component and...

Original URL: https://gitlab.com/trapper-project/trapper-ai

huggingface.co

OSCF/TrapperAI-v02.2024 · Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Original URL: https://huggingface.co/OSCF/TrapperAI-v02.2024

Yeah we're a little behind in terms of gui, but I'm more than happy to talk to you about a specially trained model that might incorporate species from the other models we have. If you'd like to chat directly im at tswanson@sdzwa.org.
Camera Base was made by our own Mathias Tobler about a decade ago and while it is still technically under development, we have long term plans to completely redo it and fully integrate aniML.

I'll follow up by email soon!

@Ed Miller @Kyra Swanson From my experience, joining forces is always the best and optimal solution. TRAPPER team is always open to meeting with a larger group of researchers and engineers. For your information, we are working on a next version of TrapperAI Model and have also added initial support for MegaDetector v6 (beta). Happy to share more details during the meeting.

Hi Alba, the Labeled Information Library of Alexandria: Biology and Conservation would be perfect for that! It is maintained by @Dan Morris, but you can also send an email to info@lila.science for upload instructions or more info.

LILA BC

LILA BC (Labeled Image Library of Alexandria: Biology and Conservation)

LILA BC is a repository for archival data sets related to biology and conservation. Our intention is to create a valuable resource for the scientific community, including for machine learning researchers and/or those that want to harness machine learning for biology and conservation. Machine learning depends on labeled data, but getting access to such data in biology and conservation is a challenge. Consequently, everyone benefits when more labeled data is made available.

Original URL: https://lila.science/

great! thank you ☺️

Place it on HuggingFace dataset for greater visibility!

@Dan Morris loves to hear about cases where megadetector fails! And he likely will have advice for other options for ya

Hi Carly! Yes Dan has previously weighed in on this and is actually the one who steered me down the classifier path

Can you show a sample photo?

Sample:

This is a ‘middle of the road’ example in terms of clarity. Some are more clear (more contrast/not submerged or muddy), others are more obstructed or partially out of frame

There are a few more examples from this data set on the "MegaDetector challenges" page; they are the ones credited to Michael Knoerr:

https://github.com/agentmorris/MegaDetector/blob/main/megadetector-challenges.md#reptiles-and-other-under-represented-species

@Eric Cunningham, it may be helpful to share the rough data set size and class distribution. The dataset is relatively small, but not silly-small, so IMO ResNet-18 is about as much as one could train without freezing anything, but I don't have a great sense for the tradeoffs wrt training about the same number of parameters in a larger model. Inference time is not a big issue.

Here is the class breakdown for the dataset:

other_animal    3015
turtle          5227
no_animal       41195

total           49437

Do you need to classify these images, count the turtles in it, or localize them precisely in the frame? These are different CV tasks. A very good off the shelf solution is just to go with YOLO (v10 is out now but v8 is still a great pick) from ultralytics and prepare your dataset for training. It can do image classification and object detection (even bytetracking if you need to count animals using video streams).

Its not clear if you are trying to train a classifier the task or trying to train a model that can localize it (generate bounding boxes around the detection) If you are looking at the later (generating bboxes). You can check the Yolo family.
Also try out zero-shot segmentation engines. Like the recent segment-anything-2 model. Should work for such usecases. Just run the images which have turtles in them first and mark the area of interest (the turtles)

For now I’m am looking to classify the images. Thanks for the suggestions, will check these out

Just fyi this is the output of a zero-shot model. If that helps.

Thread Reply:

I tried out YOLOv8 over the weekend and I’m seeing some pretty promising results! Thanks for the recommendation @Aakash Gupta and @Arthur Caillau!

Just chiming in here, might be late to the party but you may find success in using BIOCLIP with zero-shot text inputs for your classes. https://huggingface.co/spaces/imageomics/bioclip-demo

huggingface.co

Bioclip Demo - a Hugging Face Space by imageomics

Discover amazing ML apps made by the community

Original URL: https://huggingface.co/spaces/imageomics/bioclip-demo

Hi @Mathias Tobler, I’m using square crops with padding. This makes sure that the animals are not distorted. I’ve also heard some people just squeezing it in a square, which also works well since an animal is generally squeezed in the same manner. I never compared the two though…

With regards to the padding, I use a fixed padding for ‘normal’ sized animals and a larger padding for small animals like rodents etc. This to make sure a mouse doesn’t look like a bear when resized 😁

Let me know if you want the Python code for this.

Thanks @Peter van Lunteren. Interesting about the padding size, first time I hear that.

Hi @Mathias Tobler, I'm a bit late on this thread, but if it helps: I've tried both square crops and resizing on a large dataset (2M images, 25 classes) and there was no significant difference on the overall performance. Of course this would require finer analysis. Would love to know your conclusions if you look into it.

Hi @Hugo Magaldi thanks for sharing those results. We have used letterboxing in earlier model and just standard resize more recently. There have been many other changes (training data size, model architecture) so a direct comparison is not possible. We might need to run some experiments to compare the two.

@Dan Morris @Tanya Birch

Can you clarify what you mean by "get grouped in as 'empty' AND 'human'"?

For example if its 3 photos: animal, animal, empty, the AI has tagged all as an animal. However, the cameras aren't set to take bursts, pictures are at least 30s apart, but the grouping on WI makes it into a 'burst'.

Ah, thanks, I understand now. When you upload images to a sequence-based project, sequence boundaries are based on a fixed 60-second timeout. The image content isn't used when creating sequences, so AI isn't a factor here.

The relevant bit of documentation is here:
https://www.wildlifeinsights.org/get-started/basics/projects

Hope that helps!

ohhh that explains it all! thank you!!!

Hi Adrien, its a timelapse camera right? does it not capture passerby wildlife too?

can it be attached to a drone ? Or it requires to be on a stable platform?

not really, it weight about 2kg. It is designed to be installed on a stable support. It has various fixation accessories .

Before we found the special issue where we published the paper that Stefan mentioned one message earlier on this chat, we put together this list of journals with relevant examples:

• Methods in Ecology and Evolution (journal) (example recent papers on AI for camera traps 1,2)


• Ecological Informatics (journal) (example recent papers on AI for camera traps 1,2,3,4)


• Ecology and Evolution (journal) (example recent papers on AI for camera traps 1,2)


• Remote Sensing in Ecology and Conservation (journal) (example recent papers on AI for camera traps 1,2,3,4)

British Ecological Society

Methods in Ecology and Evolution

Methods in Ecology and Evolution promotes the development of new methods in ecology and evolution, and facilitates their dissemination and uptake.

Est. reading time

1 minute

Original URL: https://www.britishecologicalsociety.org/publications/journals/methods-in-ecology-and-evolution/

Journal of Applied Ecology is another good one, that's where the original random encounter model (REM) for camera trapping was published, so novel methods might be of interest. Not sure about UW but my uni has agreements with some publishers where we can publish there for free (!!), and we've been selecting from that list, which covers some good ones.

Thank you both! Looks like I was on the right track 🙂

Okay, I came across this publication first, I was just confused because it doesn't explicitly include the term ‘MegaDetector’. But now I know, I'm glad I asked, thanks for the quick help! 🙂

We never wrote more than this short workshop paper for it, but this used to be the listed citation on the codebase. I think it got lost in the migration, but we'll try to make it more clear :)

I try to track publicly-available models here:

https://agentmorris.github.io/camera-trap-ml-survey/#publicly-available-ml-models-for-camera-traps

Other models specific to Africa are:

https://huggingface.co/Addax-Data-Science/Namib-Desert-v1
https://github.com/Appsilon/mbaza/releases/tag/v2.1.1
https://sandiegozoo.app.box.com/s/9f3xuqldvg9ysaix9c9ug8tdcrmc2eqx/folder/196933085878

Thank you very much Dan! I'll be looking into those 😄

I like this a lot! It would be nice to display brief summaries for each filtered search as well (e.g., number of sites, number of photos)

ho yeah that makes sense! The panel with pictures could indeed show more informations. I had a similar observation with the timeline: showing marker for pictures on the timeline to get a sense of the distribution

This is cool @Jeremy Vuillermet. Is the code available on GitHub somewhere?

thanks! No it's mostly a design prototype so far with hardcoded data

Wow, super cool visualisation! Thanks for sharing

MegaDetector is still called MegaDetector! Pytorch Wildlife is intended to be infrastructure that supports the use of many such models, including but not limited to MegaDetector

Ah i see, thanks!

There are newer depth estimation models out there, e.g. DepthAnythignv2. You can try a demo here:
https://huggingface.co/spaces/depth-anything/Depth-Anything-V2

huggingface.co

Depth Anything V2 - a Hugging Face Space by depth-anything

Discover amazing ML apps made by the community

Original URL: https://huggingface.co/spaces/depth-anything/Depth-Anything-V2

Thanks! That actually works quite well (on the very limited tests I did).

thanks for sharing. I'm currently working on this area and it's a valuable source to compare what's existing!

Saket Anand at IIIT Delhi is working building a strong camera trap model for India in partnership with WRI, you can reach out to him at anands@iiitd.ac.in

The first thing I would recommend is putting a very fine point on what you mean by "automated". There are close to zero cases where camera trap image processing is fully automated in the sense of every image being classified to species level without human intervention, and the cases where that happens tend to be pretty simple cases with a small number of very stable cameras, or a very small number of species (e.g. in semi-captive environments). That doesn't mean automation is impossible, but total automation with 100% accuracy is probably impossible, so you have to pick the compromises you're comfortable with. I recommend thinking about what you care about most and how much human time you can afford to get there... e.g., maybe you want to make sure you achieve 90% recall on species x/y/z, and you don't care what happens to the other species, and you can have humans spending k hours per month on image review.

Once you have goals that are as quantitative as possible and you're focused on efficiency, rather than total automation, it becomes easier to evaluate existing systems, whether or not they are even aware of your specific species (this is not always a requirement for an AI system to help you meet your goals, as long as that AI system has seen species that are visually similar to your species).

To help people get pointed in the right direction wrt choosing a system, I usually start with this series of questions:

http://lila.science/camera-trap-questions

Maybe those questions are useful just to get you thinking about your system, but if you want to talk through the implications of some of the answers, feel free to reply here or to email me at agentmorris@gmail.com .

Hi Vinay - interesting concept of IPO a forest. and I have a lot of questions regarding this concept. There are a number of individuals who are buying up large tracts of forest land in India. Whats the best way of connecting with you?

Please do checkout VanNetra that we have developed

Think Evolve Consultancy

Green AI Products — VanNetra - Think Evolve Consultancy

Green AI Products — VanNeTra VanNetra is an AI-based platform for assessing and simulating the biodiversity of forested regions. It provides the tool for realtime monitoring of the forest for any kind of unauthorized activities. This includes – poaching, illegal logging and wildfires.  The platform provides the framework for integrating realtime data from multi-modal sources […]

Est. reading time

2 minutes

Original URL: https://www.thinkevolveconsulting.com/products/green-ai-products-vannetra/

definitely following (as someone who lives in new mexico). i can consult the local lora/meshtasic group, which i know has talked a lot about solar/batteries here. (anecdotally, i have not heard of solar failing; but definitely remember threads about batteries vs heat/cold)

Might defeat the purpose of having the solar panel models, but sun shade could help reduce surface temps

Similar to the DIY rain covers you see

Can you clarify your scenario a bit? Camtrap DP is (mostly) a format used to archive and transfer labeled data.

If you have a Camtrap DP package of labeled images, what is your goal with Wildlife Insights?

If you have a Camtrap DP package of unlabeled images... (a) that would be unusual, so mostly out of curiosity, where did it originate? and (b) how would uploading it to Wildlife Insights be different than just letting Wildlife Insights read the EXIF information directly from the images?

Hey Dan 🙂
The scenario is:

1. Deploying a low tech camera trap that save images in the CamTrap DP format without any labels but with EXIF metadata


2. Drag and drop the whole folder (with media folder, datapackage, deployments, media and observations files) from the SD card to a tool for annotations

• Automatically reading EXIF info from media.csv


• Export them and populate metadata of the project's deployment 🙂

All clear.

No, Wildlife Insights doesn't support this (at least not that I'm aware of, I'll confirm, but if I don't reply to this thread to say otherwise, assume I confirmed that).

Wildlife Insights has a beta version of what you describe, but it still requires some manual steps and the intervention of one of the project administrators. However, Jorge (the Executive Director at WI) indicated that he is happy to help folks with this if they send him data in the WI bulk upload format.

Outside of WI... I think CamtrapDP is not the important bit here, since without labels, it's relatively easy to move metadata between formats; I think the important part is "are there Web-based annotation platforms that allow bulk upload of unlabeled data with deployment information"? This is a good question! I haven't tried this personally, but TRAPPER appears to support exactly what you're looking for:

https://trapper-project.readthedocs.io/en/latest/tutorial.html#add-deployments-to-trapper

I'm interested in whether folks know of other tools with similar functionality; this is a level below the depth of my familiarity with most Web-based systems.

Also if you are working locally in AddaxAI, and you're happy with that, and you are happy to keep working locally, it's straightforward to import both AI results and deployment information into Timelapse (https://timelapse.ucalgary.ca/). It doesn't literally support CamtrapDP import, but it's straightforward to import arbitrary metadata from a .csv file.

This dataset might be a good start: https://lila.science/datasets/seattleish-camera-traps/

LILA BC

Seattle(ish) Camera Traps - LILA BC

Overview I’m breaking the fourth wall right from the start here: this dataset comes from my house, and is intended to fill some specific gaps in public camera trap data that are difficult to fill from “real” camera trap data. In particular, the following things are important to developing and testing AI models for camera...

Written by

lilawp

Est. reading time

4 minutes

Original URL: https://lila.science/datasets/seattleish-camera-traps/

There are also some people in Caltech Camera Traps, but they are generally not labeled as people, since annotators were mostly only reviewing animals:

https://lila.science/datasets/caltech-camera-traps

Timm is correct that "Seattle-ish Camera Traps" is the only dataset I'm aware of that has people... no, not "people", just "person", since I removed everyone who wasn't me... that are labeled correctly.

You can extensively evaluate whether your model can detect specifically Dan 🙂

Thanks guys! Seems like an awkward space with the permissions. Dan may very well become our first guinea pig!

I feel like there's an opportunity for researchers to contribute pictures of themselves on their camera traps consensually. I wouldn't mind people using pictures of me, maybe this could be something we build as a community? I know this comes up all the time

Hello Piotr, we used ByteTrack to create MammAlps (from MegaDetector detections). Had to make few adjustments to the association process for it to work well, and still needed quite some manual correction afterwards. We have not evaluate it yet on the corrected dataset though. Happy to talk about it if needed.

Hi Hugo, we've been working on similar things - not a deterrent system but an edge-AI device. Happy to chat anytime. I believe there are some people that specifically work on edge-AI powered deterrent systems @Thijs at https://www.hack-the-planet.io/ and I'm sure @Peter van Lunteren knows some more people working on things like that. Also worth checking out this group on Wildlabs. Always useful to check out @Dan Morris lists, among other things he's very good at making useful lists. Oh and maybe @Ed Miller

hack-the-planet.io

Hack the Planet – Solving global challenges using technology

Accelerating efforts by creating and implementing smart technologies in collaborative projects.

Original URL: https://www.hack-the-planet.io/

Thank you! Happy to chat, I'll reach out in pm

There are some Wildlabs groups, that you can check out

👉 Full announcement: https://wildlabs.net/article/new-group-announcement-edge-computing
👉 Join the group: https://wildlabs.net/groups/edge-computing

Thanks @Aakash Gupta!

Also working on edge devices. Happy to chat, pm me!

I can’t think of any datasets that provide the cameraHeading but some of the timelapse cameras + snow poles datasets have the lat/long of the camera and the lat/long of the pole. You could deduce the cameraHeading by looking at those two locations in relation to each other. One of those datasets is available here: https://nsidc.org/data/snex20_tli/versions/1

Thanks Catherine, I love that idea! Hadn't thought about snow cameras

Yes, no problem! This is the website we use to track some of the various snow pole datasets: https://catherine-m-breen.github.io/snowpolesmap.html. It's still in progress, so lmk if any of it is confusing! I just scanned it, and none of the other datasets besides the one I shared jumped out as having cameraHeading information (some people track the lat/long of the pole, and some don't unfortunately). However, cameraHeading would be really nice information because it would give an indication of aspect and radiation which is a key driver for snowmelt and other snow processes!

What are ballpark estimates for (a) number of images (including blanks), (b) number of images with animals, and (c) number of animals? If you don't have lots of big groups, (b) and (c) are likely about the same.

(a) About 40k images currently, not including blanks, although we have more we could use. We haven't added blanks to training yet. (b/c) This project does get lots of groups, so estimated 1/3 of the images have >1 animal in them

At that (relatively low) number of total crops, I would personally review them all, at least to remove obvious outliers. Specifically, I would render all the crops to species-specific folders (with flat files in each species folder, i.e. no subfolders) with maybe 100px of padding on each side for each crop (that's not what I would use for training, but it makes it easier for your eyes to review), then I would use any fast image viewer to page through them very quickly and delete anything that doesn't fit. You should be able to check at least 4-5 crops per second this way, and you can do it while you listen to a podcast, and in a couple of hours you've removed outliers and you know your dataset really well (and you can't put a price on knowing your dataset really well!). Ergonomics matter a lot here, e.g. I wouldn't use a browser-based tool for this. Personally I use IrfanView.

If you had said a larger number, I would have recommended the following (both of which will likely take longer than manually reviewing your crops if you're only doing this once). YMMV, if you don't have a high tolerance for paging through images, or if you are going to do this frequently and want to invest the time to add some new processes that will pay dividends later, you can use these approaches instead:

1. Do aggressive repeat detection elimination (http://lila.science/repeat-detection-elimination) to get rid of a bunch of false positives. This is why I asked about the blanks; you really need to start with all the images from the original cameras to do this effectively, because you are getting meaningful information about false positives from the blanks. This process takes several hours to really understand, then every time you do it after that, it's very fast (I typically spend around 5 minutes doing this per million images in a dataset). I always do this when I run MD/SpeciesNet, but I've also done it a zillion times, so I've long since paid down the overhead of learning to do it. I'm not sure I would do this if I were only going to process a single 40k-image dataset.


2. Run SpeciesNet on the crops, remove (or manually review) crops that are classified as "blank" (SpeciesNet is quite good at classifying junk as "blank"), and possibly remove (or manually review) crops that are classified as something that is wildly inconsistent with the true class. E.g. bobcats classified as elephants.

Also I would consider at least keeping a record of all of the false detections you review and remove, and putting them in a "blank" class in your classifier. This is another good reason to review everything above a relatively low threshold. Having an ecosystem-specific "blank" class might turn out to save you more time in the long run than the species classification part of your species classifier.

@Dan Morris we should think about how to codify this with whatever tooling is needed to get this off the ground, I sense a vibe coding session in the works.

@Dan Morris Extremely helpful as always, thanks Dan! I agree that manual review is good if we keep the dataset this size. I'll bookmark your other suggestions for down the road. We are using 0.2 as a threshold which I think makes sense, especially given your point about wanting more difficult examples for training also.
If this review process doesn't generate many "blank" crops, would you generate some from the empty images to add to training?

Re: generating blank crops from empty images...

I probably would, yes. You might not end up using them, but it's pretty close to free training data. A couple other thoughts about sampling blanks:

1. False positives tend to occur over and over on the same pixels, so if you sample randomly, you'll get a lot of redundancy in your blank crops. Consider doing something like round-robin sampling across cameras, and maybe even trying to preferentially sample crops that have different pixel locations within a camera. The latter might be overkill, and it does require a bit more code to factor in the actual locations of the detections within the images, but it's not a lot more code.


2. However you end up sampling blank crops, consider writing the crops out to disk into a flat folder, with the confidence at the beginning of the filenames (e.g. blankconf0p81camera004image008crop005.jpg), so you can quickly page through at least the highest-confidence blanks, if not most of them. You never know, you may find some animals that were incorrectly labeled. I'm sure there are image viewers that will let you sort by some other metadata field in a more elegant way, but when I want to view images in a particular order, I find it easiest to just make the filenames represent what I want, and use IrfanView or similar to page through them.

Great ideas. For (2), what is the confidence in this case? If MD was used to separate animals/empties first?

Ooooh, sorry, I didn't get that you used MD to do the blank labeling in the first place. That does make this less "free". If you get enough false detections when you review the labeled images, I would probably stop there in terms of mining blank examples. You could also use the detections between, e.g., 0.1 and 0.2 confidence, but you would now have to review those carefully, because a small but non-negligible percentage will be animals, and no one has reviewed those yet.

No, you were right, I think I made this confusing. We did not use MD to do blank labeling; imagine we just have a folder of manually sorted empty images and want to create some blank crops from them. Maybe I'm not following your suggestion in (2) above?

Oh yes, in that case I would run MD on the folder of known blanks and use crops over maybe 0.1 or 0.2 as a training example for a "blank" class (balancing across cameras as per (1) above). If you are 100% sure they are really blank, no further review required. My point in (2) above is that... how sure are you they are really blank? Sometimes MD finds things humans miss, so additional review of your "blank" class doesn't hurt if it's just a couple thousand crops.

Where 0.1 or 0.2 confidence are for the "animal" class?

I would probably include all MD detection categories. If your backgrounds are natural scenes, MD false positives are almost always the "animal" category. With human-made objects in the background, you are more likely to get person/vehicle FPs. I would probably use all of them if your "empty" images aren't expect to have people or vehicles in them.

Ok, now I am following. Thanks!

I think this thread is the most the universe has ever thought about these questions, so, let us know what you decide and how it goes!

😅 I figured it was a well-worn path but I have only worked with manual boxes before.

For the sake of discussion, then, what is the rationale for not including empty examples with MD predictions <0.1? These very-low-confidence ones are more likely to be actual background and not objects, right? But therefore less likely to be things that MD would confuse for animals and so less useful for training?

Correct, my assumption is that when you run this classifier, you'll still be running MD and accepting detections over some threshold, so you want the animal-i-est of the false detections. There are an essentially infinite number of very-low-confidence detections, so you have to threshold somewhere, and I would set it at whatever number gives you a useful number of examples (i.e., around the same order of magnitude as your other classes), and I'm guessing that's maybe 0.1 or 0.2. But there's nothing wrong with using, e.g., 0.05. At 0.001, for example (the default threshold applied at inference time if you use rundetectorbatch), there's a lot of junk that wouldn't be very useful.

Gotcha, and yes, we would be running MD prior to the classifier for our inference runs

Very helpful insight here, I appreciate it!

Also don't forget to tell us what podcast you listened to while you pressed "page down" >40,000 times.

For sure, although I may need to battle an undergrad for the honor

I have a pretty high tolerance for that kind of clicking, up to a point. Theoretically I find value in getting to know the data, in practice I think maybe I just really like podcasts.

Totally agree. The amount of camera trap images I have looked through, while working on projects with the goal of NOT having to look through camera trap images, is much higher than my advisors would have liked, I'm sure 😅

this is actually quite amazing Arthur!

We just shipped a manual to explain how Biowatch works: https://biowatch.earthtoolsmaker.org/

Amazing Arthur 🙂 I might reach out in the near future to know more about what you guys are doing!

Anytime. You can gain some insights from our website here: https://www.earthtoolsmaker.org/

earthtoolsmaker.org

earthtoolsmaker

Open Technology for Wildlife and the Planet

Original URL: https://www.earthtoolsmaker.org/

Yes i was already checking your site, we are also working with individual identifications from images. I really enjoyed the elephant example also 🙂

Hi Arthur, ive been going deeper into your tool and it has so much great potential to help our teams on the ground. There are some QoL features it would be nice to have i reckon soon i'll be "adding more work" to you on the github repo. Its really a lean and clean solution for CT deployment and analysis 🙂

Hey Nuno 👋,
Thanks for your kind word. Feel free to file any ticket/features you would like added to Biowatch. We evaluate these carefully to not turn Biowatch into a big bowl of features so I can't promise we'll add them all.
Also, we are always looking for sponsors and partners to keep the work going on Biowatch. If you know anyone that would be interested, let us know 🤗