Sorry i missed these. Is there anything that i can help with?

Hi Jason! Is there a Google Hangouts link for the make-up session?

I think @Jason Parham still needs help with the Grevy's Zebra data! It's a good way to get familiar with the types of tasks we're asking models to do, and their associated challenges.

This paper that just came out seems kinda relevant? https://www.pnas.org/content/early/2020/07/02/1922097117

PNAS

The ecology of human–carnivore coexistence

The persistence of large carnivores in human-dominated landscapes will become increasingly challenging as the human footprint expands. Here, we bring together long-term demographic and behavioral data on one of the worlds’ most conflict-prone species, the brown bear, to quantify the mechanisms facilitating human–carnivore coexistence. We found that human-dominated landscapes are highly lethal, especially to young bears, until they learn to adapt to people. As bears age, they avoid times when people are most active but do not strongly avoid where people live. To sustain human–carnivore coexistence under high rates of mortality requires the influx of animals from areas with low human presence (i.e., demographic rescue). Paradoxically, our work demonstrates that connectivity leads to both coexistence and conflict. With a shrinking supply of wilderness and growing recognition that top predators can have a profound influence on ecosystems, the persistence of large carnivores in human-dominated landscapes has emerged as one of the greatest conservation challenges of our time. Carnivores fascinate society, yet these animals pose threats to people living near them, resulting in high rates of carnivore death near human settlements. We used 41 y of demographic data for more than 2,500 brown bears—one of the world’s most widely distributed and conflict-prone carnivores—to understand the behavioral and demographic mechanisms promoting carnivore coexistence in human-dominated landscapes. Bear mortality was high and unsustainable near people, but a human-induced shift to nocturnality facilitated lower risks of bear mortality and rates of conflict with people. Despite these behavioral shifts, projected population growth rates for bears in human-dominated areas revealed a source-sink dynamic. Despite some female bears successfully reproducing in the sink areas, bear persistence was reliant on a supply of immigrants from areas with minimal human influence (i.e., wilderness). Such mechanisms of coexistence reveal a striking paradox: Connectivity to wilderness areas supplies bears that likely will die from people, but these bears are essential to avert local extirpation. These insights suggest carnivores contribute to human–carnivore coexistence through behavioral and demographic mechanisms, and that connected wilderness is critical to sustain coexistence landscapes.

Original URL: https://www.pnas.org/content/early/2020/07/02/1922097117

I'll check this out, thanks! This has more focus on human activity than I was thinking but it's the same problem.

Hey. Having to use excessively small batches is somewhat common when training large models (which the original authors often trained on huge servers). Having said that, here are a few quick thoughts:

• the number of images in your dataset doesn't affect the memory demands on your GPU


• 640x640 is quite a large patch size. At least for semantic segmentation tasks which I'm usually working on. Maybe that's different here since you're doing object detection, right? However, as long as the network is fully convolutional, the training patch size should be variable. Just try something smaller (maybe in the ballpark of 256^2). But I'm quite sure there are some object detection people here that can prove me wrong 😄


• 33 h for training a large network from scratch is not a lot


• How do you determine when training is "done"? Seems like your only guidance is some maximum number of iterations/epochs in someones code (who apparently used a different set of hyperparameters)


• Take care to adjust other hyperparameters accordingly if you use a different batch size (most prominently the LR). Also you may get better results when turning off batch norm when training with small batches.


• I don't have any real experience with cloud services for DL but maybe a Google Collab instance with a GPU backend could be a starting point? Not sure how suitable that is for training though. Again, I'm sure there are people around here with first-hand experience that may be able to provide further info 🙂

Hi Lukas. Thanks for the response -- it's very useful!

Seems like your only guidance is some maximum number of iterations

I was under the impression that I'd be doing "fine tuning" and not training from scratch...

Well now I am not sure 😆 My LR config is:

learning_rate {
        cosine_decay_learning_rate {
          learning_rate_base: 0.03999999910593033
          total_steps: 25000
          warmup_learning_rate: 0.013333000242710114
          warmup_steps: 2000
        }
      }
      momentum_optimizer_value: 0.8999999761581421

Considering that one of the steps in this tutorial reads: "Download Pre-Trained Model" I guess you were on the right track with your guess that you are merely fine tuning a network. There's also this part in your config:

  fine_tune_checkpoint: "pre-trained-models/ssd_resnet50_v1_fpn_640x640_coco17_tpu-8/checkpoint/ckpt-0" # Path to checkpoint of pre-trained model
  num_steps: 25000
  startup_delay_steps: 0.0
  replicas_to_aggregate: 8
  max_number_of_boxes: 100
  unpad_groundtruth_tensors: false
  fine_tune_checkpoint_type: "detection" # Set this to "detection" since we want to be training the full detection model
  use_bfloat16: false # Set this to false if you are not training on a TPU
  fine_tune_checkpoint_version: V2
}

I guess I am a little confused because in one of the official TF tutorials they take a different approach -- what stands out to me here is that they explicitly "restore all but the classification layer at the top", where as the tutorial above doesn't really mention doing anything of the sort.

Cutting off the top layer implies that you want to keep the "deep features" learned by the lower layers but want to re-purpose the net for a different classification problem. You'll often see them freezing the weights of the lower layers to keep them intact while only really training the classifier head. The other tutorial may follow a slightly different strategy. Yet, if you start off with pre-trained weights and only wiggle them slightly to adapt to your new dataset you're still fine tuning (which will require far less training data and training time). Just follow along some of the mentioned tutorials and you'll get the hang of it quickly 🙂 If you even intend to learn more about it. If this is just a tool you want to use once, maybe you can find someone here who can help you with training and deploying the network! 🙂

Cutting off the top layer implies that you want to keep the "deep features" learned by the lower layers but want to re-purpose the net for a different classification problem. You'll often see them freezing the weights of the lower layers to keep them intact while only really training the classifier head

You're very welcome 🙂 Good luck with your project. Let us know what you came up with!

Will do 👍 🙂

Super late to this, but in case it might be useful in the future, what was the data type of your input? Float64? Sometimes you can reduce it to float16 or even float8, although last time I did this it required some tinkering to get smaller data types to work

@Isaac Griswold-Steiner I don't quite remember; I think it was float64. Will keep this in mind, thanks 🙂

I reviewed a paper on something similar for a conference a year or so ago, let me see if I can find it.

Thanks Sara! That would be super appreciated

Ok, I can't find the paper I reviewed (not sure if it's been published yet) but while searching I found some papers looking at general water body identification if you're interested? I'm not sure of these have the change over time component you're interested in though.

https://ieeexplore.ieee.org/abstract/document/8013683?casa_token=CZ78y-Tsz-sAAAAA:z5JN[…]5q04nqbkaz_YYsb5Wjh7PVsDYQ5ma6EwsJUng9BLOQWILKXVWVNTBsOur4w

ieeexplore.ieee.org

Surface Water Mapping by Deep Learning

Mapping of surface water is useful in a variety of remote sensing applications, such as estimating the availability of water, measuring its change in time, and predicting droughts and floods. Using the imagery acquired by currently active Landsat missions, a surface water map can be generated from any selected region as often as every 8 days. Traditional Landsat water indices require carefully selected threshold values that vary depending on the region being imaged and on the atmospheric conditions. They also suffer from many false positives, arising mainly from snow and ice, and from terrain and cloud shadows being mistaken for water. Systems that produce high-quality water maps usually rely on ancillary data and complex rule-based expert systems to overcome these problems. Here, we instead adopt a data-driven, deep-learning-based approach to surface water mapping. We propose a fully convolutional neural network that is trained to segment water on Landsat imagery. Our proposed model, named Deep-WaterMap, learns the characteristics of water bodies from data drawn from across the globe. The trained model separates water from land, snow, ice, clouds, and shadows using only Landsat bands as input. Our code and trained models are publicly available at <http://live.ece.utexas.edu/research/deepwatermap/>.

Original URL: https://ieeexplore.ieee.org/abstract/document/8013683?casa_token=CZ78y-Tsz-sAAAAA:z5JNUBtKhWEl-od5q04nqbkaz_YYsb5Wjh7PVsDYQ5ma6EwsJUng9BLOQWILKXVWVNTBsOur4w

This one is looking at floods, which is kinda the reverse of what you described https://openaccess.thecvf.com/content_CVPRW_2020/html/w11/Bonafilia_Sen1Floods11_A_Geor[…]aset_to_Train_and_Test_Deep_Learning_CVPRW_2020_paper.html

https://link.springer.com/article/10.1007/s40899-020-00425-4

Sustainable Water Resources Management

Surface water detection and delineation using remote sensing images: a review of methods and algorithms

Multispectral and hyperspectral images captured by remote sensing satellites or airborne sensors contain abundant information that can be used to study and analyze objects of interest on the surface of earth and their properties. The potential of remotely sensed images for studying natural resources like water has been studied by researchers over the past many years. As water is an important natural resource that needs to be conserved, such studies have been of great interest to the scientific community. By employing appropriate digital image processing techniques on images taken from remote sensing satellites or airborne sensors, an effective system can be developed to study the quantitative and qualitative changes happening to surface water bodies over a period of time. Surface water detection and mapping is a crucial and necessary step in such studies and different automated and semi-automated methods have been developed over the years for mapping water in remotely sensed images. Remote sensing sensors capture images at multiple bands corresponding to different wavelength ranges in the EM spectrum. Digital image processing based techniques for water mapping falls predominantly into four categories; (i) single band based methods, (ii) spectral index based methods, (iii) machine learning based methods and (iv) spectral mixture analysis based methods. This paper presents a review of techniques, methods, algorithms and the sensors/satellites that have been developed and experimented with to perform surface water body detection and delineation from remote sensing images.

Original URL: https://link.springer.com/article/10.1007/s40899-020-00425-4

This one looks at surface water change https://www.tandfonline.com/doi/abs/10.1080/01431161.2020.1723817

https://meridian.allenpress.com/jcr/article-abstract/91/SI/426/428359

Dude thanks!

This has multi-source/multi-temporal which sounds neat, but is urban-focused https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7014233/

PubMed Central (PMC)

Intelligent Object Recognition of Urban Water Bodies Based on Deep Learning for Multi-Source and Multi-Temporal High Spatial Resolution Remote Sensing Imagery

High spatial resolution remote sensing image (HSRRSI) data provide rich texture, geometric structure, and spatial distribution information for surface water bodies. The rich detail information provides better representation of the internal components ...

Original URL: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC7014233/

What are you working on? Seems like a cool project 🙂

ouf thanks for your patience on this reply - this week was a blackhole. I’m living and working remote in an ML+art residency near Bombay Beach on the coast of the Salton Sea and I’ve become fascinated with the bizarre-ness of this place. Last summer, myself, @Björn Lütjens and a few others used pix2pixHD and SLOSH to generate physics-informed satellite images “of the future” in the eastern parts of the US. I want to do something similar for the Salton Sea with styleGAN and try to find some “drought vector” ? We’ll see! I’ve attached a proof-of-concept whipped up in illustrator. The moonshot goal would be to have this on a timescale UI.

COOL!!

Not sure about the other stuff, but ImageMagick is something I've needed to install in the past on my Mac!

Thank you!

If you’re doing Edge stuff, the folks at Edge Impulse may be able to help! I’d also perhaps post about this in WILDLABS (wildlabs.net), an online conservation tech community full of people doing all sorts of CV, ML & tech for conservation 🙂

So this is a RasPi camera trap type thing then? If so, this paper may also be of use/interest if you haven’t already ready it: https://doi.org/10.1111/2041-210X.13610 (“A Raspberry Pi-based camera system & image processing procedure for low-cost and long-term monitoring of forest canopy dynamics”)

hi @Carly Batist, thanks for the paper and advice. My main issue is not edge stuff for now but a evaluation of the SSD mobinet trained on a GPU

@Maxime Cauchoix, I'm no expert but my first impression is: would it be helpful to include a background/other class? That is if you don't already have one. Since it is failing when predicting the most frequent class, I wonder if the model has not just learned to chuck the kitchen sink into the most frequent class?

I'm not an R dev, so I may be way off, but it looks like it may be easier to call an external program or script from R to do this.

1. R write EXIF data to JPEG file


2. Call Python from R

Mac OS Big Sur 11.5.2

It'd be useful to know what size image you're training on (and also how many images are in your dataset). You can train a Yolov5 medium model (for example) on a fairly small desktop GPU (8-10 GB). With that many images I would consider looking at some sort of selection strategy to reduce your dataset size, though with augmentation and a huge dataset epochs are less meaningful and you can just keep training until your loss flattens.

Basically your VRAM requirement is dependent on model size, batch size and input image size, but I think something like 4-8GB x batch_size is a fair estimate for a medium model (last time I did this with 640x480px images). CPU requirements aren't that high, 4-8 cores per GPU should be OK to keep it fed. Most cloud platforms will enforce reasonable defaults for the number of vCPUs per GPU per instance.

e.g. https://github.com/ultralytics/yolov5/blob/master/data/hyps/hyp.scratch-high.yaml is the config for YOLOv5 on COCO. They use a 4xA100 (40 GB?) instance I think, with batch size 32 and 1280px images (I guess even for the XL models). But it's unlikely you'd need that unless you were training from scratch, that thing would burn ~250 bucks a day.

Thanks for this! The plan is indeed not to train from scratch but to do transfer learning. It will happen on compute canada's HPCs.
Yes, now that you mention it my training set will of course be much smaller, I just don't know yet how much. I'll be cropping out detections from megadetector, so I will have to resize the cropped images.

Ah so is this just a classifier, not an object detector? (you mentioned YOLO). If that's the case then you need even less. Object detectors are pretty heavy.

Oh my bad, I don't know why I wrote YOLO

For classification you could even try a quick test with a library like fastai (just run it in Colab), and see how well that does before throwing it at a cluster 🙂

Ok, but basically you would expect the needs to be rather modest?
I've just done a quick estimate, but the whole set should be around 300-400k detections. For sure below 500k.

Yeah, it really depends on what the data looks like and if your species are easy to differentiate. I would try something very simple like a resnet50 (or even a resnet18/34) and see how well it does.

If you're looking for someone to throw out a number to help you estimate budget, I will throw out a number, then duck while people caution you about all the caveats... on a V100, training a model like what you're describing is probably on the order of a week. Longer than a day, shorter than a month. YMMV.

I also wouldn't think about it in terms of "needs". You can train anything on anything (within reason) if you have enough time to wait. I would think of it as a tradeoff between up-front cost and time.

Yeah I can see how it's a bit of an unfair question to ask when I haven't done any benchmarks... The issue is that the deadline for applying for resources is way earlier than when I'll be able to start benchmarking. So I have to guess a bit

Also remember the first model you train almost never works the way you hope it will; I'm answering (as are others) wrt the requirements for training one model. I would assume you will train ~3 times, so if I were budgeting cloud costs, for example, I would ask for a couple months of V100 time.

Yes that makes sense. Thanks for the ballpark, that is helpful. In my case, it's better to undershoot and ask for more later than to overshoot.

You could generate indices for samples, then random.shuffle and then partitioning the list into train/test/split etc. Is that still not repeatable if you call random.seed just before?

Safest would just be to generate a split once and then upload it or store it in your repo.

Hi, I am sorry for not answering your question, but I would like to add a comment, which I hope you find useful:

depending on the type, amount, and variation of your data, you might want to make the splits yourself to ensure approximately equal distribution of classes, number of objects per sequence, or what else you can use as a complexity measure for your data. Using a random function to generate your train/test splits is (in my opinion) often not a good idea. E.g., you could end up having complex and varied data in the train split and the opposite in the test split, which would most likely provide you with a non-representative high performance of your model.

We published a paper at ECCV this year aimed at handling this problem within the multi-object tracking domain https://arxiv.org/pdf/2207.10031.pdf

@Josh Veitch-Michaelis Thanks for the suggestions. I will try the index shuffling method and if it doesn't work I'll likely resort to the offline split. @Malte Pedersen Thanks for your note as well. I have this implemented in a stratified manner such that the class distributions of the training/validation/testing sets are the same, so I don't think there should be a problem

That's great, it's often not taken into consideration

Hi Valentin, this might be related to the non-deterministic computations on GPUs (even on single ones). Just quickly found the following link: https://github.com/NVIDIA/framework-determinism
You probably need to search for "GPU and determinism" to find out more in this direction, but there are various known issues, e.g., with respect to the cudnn library. Hope this pointer helps.

Yes, it does help. At least I know what key words to use and that helps a lot already. Thanks!

Just a note that we've also had trouble with this in Detectron2.

IMO the main impact of randomisation should be the order in which your model sees new training data, and what augmentation is applied. For an "ideal" dataset that's sufficiently representative, etc, that order shouldn't matter too much. I would argue that if you're seeing significant differences due to framework determinism (things like rounding errors) then that's a "smell" that you should investigate your training data and try to figure out if the model is really generalising. In the talk the difference between the deterministic and non-deterministic models is pretty small (like 5-6% I think?).

You could also look at doing a bunch of runs and seeing what the variance is as a function of epoch. Perhaps you just need to train for longer, or lower the learning rate a bit if the model is jumping around? I'm not sure I'd trust the results if you can get a 10-20% improvement just by tweaking a random seed (is it a fluke on your validation set?). A more computationally intensive workaround for this is to ensemble a bunch of models. I agree it's tricky, especially if you want to try and reproduce a run with an identical seed.

Thanks for sharing your thoughts Josh.
I trained for 300 epochs each time and it looked like enough. The two identical runs give so different results that makes me think that I implemented something wrong, but I can't see what and I need to see how they perform on the identical test dataset. I have 100 images for training, 50 for validation and 50 for testing (a total of 200 images per species). I run it the 3rd time and I also get again different results. If this is inherently a problem in the YOLOv5 or v7 implementation when using multiple GPUs (as mentioned for example here https://github.com/ultralytics/yolov5/discussions/9288#discussioncomment-3560835), then how am I supposed to even optimize a model? Should I bite the bullet and just train on a single GPU every time? It is very hard to get a sense of direction when things feel random.
It is fascinating that detectron2 has the same issue. Is this a current huge problem in the field?

see here for example: https://github.com/facebookresearch/detectron2/issues/4260

I'm not sure if you have seen this already, but since it seems you are working with pytorch, this may be helpful as well: https://pytorch.org/docs/stable/notes/randomness.html

Yes, I saw that. I realize that it was mentioned by Glen Jocher at the link above too https://github.com/ultralytics/yolov5/discussions/9288#discussioncomment-3560835 I have no idea how to solve this. I am more of a person who wants to learn how to drive the car, not how to fix the engine 🙂 I mean, I am not at the level of building architectures and fixing bugs within them, but more on the side of making use of them for our research questions

I'll try to collaborate with the Cluster support team and see what I find out. But any other suggestions are welcomed as I cannot solve this by myself 🙂 It looks like this is an open issue when using multiple GPUs both for YOLO and detectron2.

Yeah this was from a colleague - I'm also using Detectron but I've not tried to measure the effect on my own data yet.

I'll try to investigate more and share what I get. As an alternative we could run a model N times, so that we get a distribution of performance metrics and we would always compare distributions to decide how to optimize models or to compare different approaches.

But that is easily overkill and not so affordable if a model takes days to run. Perhaps running it several times with less epochs might be also a compromise (like 50 epochs or so, or maybe even less, but then the results might not be stable / variance is really high and not informative)

Updates:
Indeed YOLOv5, starting with the release v6.2 (as mentione above), implemented reproducibility only for using a single GPU. With YOLOv7 I couldn't get reproducible results on a single GPU. Reproducibility with parallel processing, using multiple GPUs at once, seems to remain an open issue both for YOLOv5 and v7.
Perhaps someone who can edit their code might implement this, but it is out of my reach for now. If anyone else makes progress with this please let me know as I find this extremely crucial for open science, reproducibility & transparency.

If you’re totally new to computer science, I strongly recommend starting with these lectures. These will, in theory, give you a good foundation on computers in general, and help you use tools like AWS/Azure/GCP if you’d like! Afterward, I’d focus on honing your python skills

https://missing.csail.mit.edu/

Would also love to chat with you as you go more about your process and learning experience.

Also, if your job would allow/if your drone work is vision-focused, I’d highly recommend applying for the CV4 ecology summer workshop. More info here

Hi Remi, I just want to say that, generally, the ecology/behavior focused PhD projects in WildDrone do not require a computer science background, and would provide lots of opportunities for gaining such skills and collaborating with CS folks to apply their skills to wildlife/behavior problems. So if any of the projects (particularly those in Theme 1) appeal to you I would encourage you to apply!

In fact, I would say that the Theme 1 projects are generally aimed at people such as yourself - who have a strong background in the wildlife side of things, and are motivated to combine these skills with drone technology and computer vision through skills development, but most importantly through collaboration. A grounding principle of the WildDrone project is that no one can be an expert in all three areas (wildlife, drone technology and computer vision), so collaboration is the necessary and best way forward. Please apply!

I was looking at the theme one projects! This is really reassuring, I'll definitely plan to apply ☺️

Great!

Have you looked into transformers? There is a lot of work coming out where people are exploiting additional metadata by appending them to the patches, in a similar way to how you append positional encodings on the vision transformer

Interesting, do you have a reference? At the moment we're using Mask-RCNN because it's pretty stable, but should be simple enough to swap out.

Is there a reason you need to use a NN for the reconstruction and not something like photogrammetry? I would explore whether you can simply take a bunch of photos of the tree from various angles and stitch them together.

For example, do you need "understanding" or just a good, consistent 3D model?

I would suggest that you split this into two stages - one would be to generate a model, and second would be to segment/classify. NeRFs look good, but they require a lot of compute power and the results may be sensitive to the scene. Depending on the conditions and scale, you might be able to use things like structured light scanning, time of flight, etc.

I've used TLS for this and it's quite effective up to 20-30m, but very expensive (the scanners are, anyway).

I second that computer vision approaches for photogrammetry (e.g. Structure From Motion) might be a good first step to build a 3D model prior to implementing your NN.

Wikipedia (https://en.wikipedia.org/)

Structure from motion

Structure from motion (SfM) is a photogrammetric range imaging technique for estimating three-dimensional structures from two-dimensional image sequences that may be coupled with local motion signals. It is studied in the fields of computer vision and visual perception. In biological vision, SfM refers to the phenomenon by which humans (and other living creatures) can recover 3D structure from the projected 2D (retinal) motion field of a moving object or scene.

Original URL: https://en.wikipedia.org/wiki/Structure_from_motion

Yes, a two stage approach is likely what I need…

Photogrammetry is a good shout but ultimately I don’t care about actually having a 3D model. I’m thinking I need something like instance segmentation on steroids.

I’ll give you a bit more context using the tree analogy again. For a given image I need to identify unique branches. I can train a segmentation model to do that do some degree but likely not very granular. Branches are just too similar in nature to get good results… I could probably segment the background, the tree trunk and all branches together as one entity.

Hence why I am interested in a model that can “understand” the shape/characteristics of the whole tree as opposed to just what’s present in a single image.

Also, what is TLS in this context? Thanks.

(have a look at MicMac - https://micmac.ensg.eu/index.php/Accueil)

TLS is Terrestrial Laser Scanning

I suppose I am curious about the second step… segmentation in 3D space? Similar to point cloud classification maybe?

Yes, basically. You can do 3D convolution instead of 2D, for example. Depends a little on whether you're doing voxel based classification or true point clouds (which are a lot more difficult in some ways because they're unstructured).

See https://arxiv.org/abs/1612.00593

arXiv.org

PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation

Point cloud is an important type of geometric data structure. Due to its<br>irregular format, most researchers transform such data to regular 3D voxel<br>grids or collections of images. This, however, renders data unnecessarily<br>voluminous and causes issues. In this paper, we design a novel type of neural<br>network that directly consumes point clouds and well respects the permutation<br>invariance of points in the input. Our network, named PointNet, provides a<br>unified architecture for applications ranging from object classification, part<br>segmentation, to scene semantic parsing. Though simple, PointNet is highly<br>efficient and effective. Empirically, it shows strong performance on par or<br>even better than state of the art. Theoretically, we provide analysis towards<br>understanding of what the network has learnt and why the network is robust with<br>respect to input perturbation and corruption.

Original URL: https://arxiv.org/abs/1612.00593

But the inputs to all these models is a 3D representation of the scene, so you still need to generate that first. And if you get a good reconstruction, you'll solve the problem of disambiguating branches because they're just part of the point cloud.

Makes sense, thank you! That cleared up some things in my head.

I have use makesense.ai and was good for my usecase. Here's a partial list of urls with similar toolset:

https://www.makesense.ai/
http://labelme.csail.mit.edu/Release3.0/
https://scalabel.ai/
https://github.com/tzutalin/labelImg

A larger list of open-source tools

https://www.simonwenkel.com/lists/software/list-of-annotation-tools-for-machine-learning-research.html#computer-vision:~:tex[…]D%20Pose%20Annotation

thank you very much!

Closely-related thread from a few days ago:

https://aiforconservation.slack.com/archives/CLWGQ4BJ6/p1678977264828679

But now you've piqued all of our curiosity... what are you putting boxes on?

thank you! And primate faces— happy to chat more if interested!

Neat! Let this Slack know what you end up using for annotations, this is a question that comes up a lot here.

@Felipe Parodi this is pretty doable in CVAT if you're just after boxes. Though I would suggest batching your data into subtasks for convenience (like 1k images at at time). The upload system is pedantic, but it works.

It's not super clear from your question, but you already have labels and you want to tighten the boxes? I would imagine that you'd be able to do well by fixing even 5% of the data, re-training and re-labelling.

@Josh Veitch-Michaelis yes i do have boxes to tighten-- thanks for the tip!

Here are some examples of such bounding boxes

We re considering training some model to tease apart false from true positives for our specific use case (either using characteristics of the BB or directly training a classifier.. we think a classifier will not generalise to future wind events at different sites though).

Can you show the whole images for the boxes above?

Yes! Second image is actually a bad example, its a root which could have been filtered out. The last too are more emblematic of the issue

there are so many of these windy images that even if MD's error rate on this is low, it still produces 10s of thousands of "relatively confident" false positives due to near sensor vegetation

yeah, that's definitely tough. How much motion are we talking about? Also, what species are of interest? Maybe there's a way to tweak the box location based filtering for this use case

a lot of these if I squint could maybe look similar to the tip of an ear or something

How much motion are we talking about? => its only really a pain to sort through when the vegetation is fairly mobile. Im gonna make a gif to examplify that.

what species are of interest? => we dont have targets, we are looking at everything (which of course doesnt help)

Hopefully it is not too heavy to upload its about 100MB

a smaller and shorter version probably visible without download here I think

that branch on the side is an example of what will trigger a lot of false positives and we have this on a lot of our sites due to the biome we work in and the extreme seasonality

[Also see this answer from a couple weeks ago, where I said basically the same thing I'm about to say here. 🙂 ]

What are the overall rates of true positives MD false positives at your confidence threshold? In general, if MD (or any AI tools) are substantially reducing the number of images you have to deal with, false positives are just a part of life, and the only way to reliably get rid of them is to sacrifice some recall. So the first thing I would ask is whether any steps to reduce those FP numbers are going to take more time than it would take to just review those false positives.

That said, there are a few tricks we recommend if you really want to put some time in:

• Always try both MDv5a and MDv5b on your data. Typically MDv5a performs a little better, but every once in a while, MDv5b "wins", and that's as close to a "free" improvement as you'll find.


• We sometimes combine results from both MDv5a and MDv5b (and sometimes eevn MDv4). We usually do this to improve recall, but you can do this to improve precision: the two models tend to agree on true positives more than they agree on false positives, so if you tinker with confidence thresholds appropriately, you can preferentially take detections that the models agree on.


• The #1 thing we recommend in these cases is our repeat detection elimination process, which is semi-manual but very fast when you get used to it: https://tinyurl.com/3htnrau2. It's not uncommon to get rid of 100,000 false positives in ~5 minutes this way, if you have a dataset of several million images. Unfortunately this is much less helpful when the detections are due to wind, because they are less geometrically stable.


• And of course, you can fine-tune MD or train a post-hoc classifier (animal vs. junk) to eliminate noise. That classifier wouldn't really need to generalize, it would be OK if it just works for your cameras. I don't know of anyone doing this; it's straightforward in theory, but... if you want to go down this path, happy to help by email.


• Just in the last couple of weeks, I have started experimenting with YOLOv5's test-time augmentation (https://tinyurl.com/2mcvbsxk), which requires running MD outside of our usual inference scripts. This has shown amazing promise for improving recall on really difficult data sets, though YMMV if you want to experiment with using this to improve precision.

Thanks for all this.

Concerning the rate of FP: out of the 65,338 detections (for a total of 54,677 images) we have manually verified so far, 30,621 were false positives (or 46.87 %), and 33,237 were true positives (or 50.87 %), to which 1,480 false negatives were added (manually added bounding boxes, 2.27 %). Out of the 33,237 true positives, 1,738 had to have their bounding box adjusted.

I am now doubting that I ran the model properly, which actually brings to mind another issue we ran into, where sometimes in a sequence of images where an animal was detected, there would be an image in the middle of that sequence where MD just completely missed the animal even if found it in all of the previous and subsequent images (does that happen?).
Below is the code I used to run the model, to give you an idea. I dont think I departed from the doc. I apply the threshold of 0.1 later.

python $MD_FOLDER/detection/run_detector_batch.py <br>                $MD_FOLDER/$MODEL $RUN_DIR $STORAGE_DIR/$OUTPUT_JSON <br>                --output_relative_filenames --recursive <br>                --checkpoint_frequency $CHECKPOINT_FREQ <br>                --checkpoint_path $STORAGE_DIR/$CHECKPOINT_PATH <br>                --quiet <br>                --resume_from_checkpoint $STORAGE_DIR/$CHECKPOINT_PATH <br>                --allow_checkpoint_overwrite

Also, we try to process the detections with highest confidence first, so my number above are skewed toward true positives.
I completely agree with having to think hard as to whether this is even worth solving. I want to know if my number of false positives are really above normal, and unless ive ran the model wrong, it seems to be the case.

Your code looks reasonable... I'm actually not sure what happens if you specify --resumefromcheckpoint when starting a job for the first time (that's used to resume processing if, e.g., your machine reboots), but whatever it is, I don't think it's something we need to worry about wrt accuracy.

A 50% false positive rate would be pretty high... but what was the total number of images from which those 55k detections came? If that was 30k false detections out of 100k total images, these results would be... not good. If that was 30k false detections out of 5M images, I would be thrilled with those results.

Re: missing an animal in the middle of a sequence... everything that could possibly go wrong will eventually go wrong. 🙂 Is this common? And is this a subtle case where the detections were close to the confidence boundary, and maybe the middle one just dipped to a slightly lower confidence? Sometimes in postprocessing we'll smooth detections over sequences, e.g., in a second pass, apply a lower confidence threshold to images in the same sequence as an image with a high-confidence detection. I don't usually do this until after doing whatever steps you're going to do to remove false positives.

In any case, I'm always curious to learn more about MD failures; if you want to send some images for me to take a look at, email cameratraps@lila.science.

Sorry I actually copied the chunk for after checkpoint restart. The new run code is the same minus that.
I cant say how many iamges that comes from because we are processing more certain detections first, and I dont have a proper "set" estimate at hand. But it is more along the line of 30k in 100k.

thanks again for all this. I will first try the repeat detection scripts and probably come back with more targeted estimates and discussion points

Does it make sense to make calibration to the camera trap. Reduce the number of images captured (sleep time between images) or reduce the sensitivity parameters of the sensors?

I am also a little inclined towards wondering if different geographies can impact the output of MD? This is more of an intuition, rather than based on any hard scientific analysis. But I have attached the geographic distribution of Wildlife Insights and Wildcam data. There is a gross imbalance with respect to data from the Indian subcontinent. This could be one of the reasons for MDv5 having a high number of false positives.

I have not worked with Megadetector before, but have you considered training a post-hoc filtering algorithm? You should be able to fine-tune a pre-trained off the shelf model for your sites to reduce false positives.

Variability in background can definitely impact accuracy, although I would think of it as a function of background and target taxa, more than geography. Background is slightly correlated with geography, but only a little. Dense rainforests are hard whether they're in Vietnam or Peru, and relatively open areas are easier whether they're in Tanzania or Montana. Reptiles and tiny nocturnal rodents are enormously difficult whether they're in India or Florida. That said, a few random notes:

1. MDv5 includes a pretty decent representation of training data from India, provided by the Wildlife Institute of India.


2. I agree with Cody's suggestion that if you want to reduce false positives, at the cost of some engineering and data labeling time, you would likely get very good results training a site-specific noise classifier. This would be specific to your cameras more than it would be specific to India, for example. I don't expect this to be the most efficient path, compared to alternatives like the repeat detection elimination approach (or even compared to just putting up with false positives), but I'd like to be wrong about that, i.e. I'd like to see this process get more efficient.

@Dan Morris, Ive tested the repeat detection scripts (it required to add a few packages to the conda env btw, i dont know if this is intentional?) and the default iouThreshold (0.85) doesnt yield anything, so ive lowered it to 0.5 and its looking like it will be useful, as its catching more of that wind movement. I was wondering if you had any advice on choosing the Threshold. Have you had to change it significantly in the past?

Re: conda environment... sorry, there's a separate conda environment for everything that isn't literally running the model (environment.yml, rather than environment-detector.yml).

Note to self: maybe this is a little silly at this point and we should just add the other packages to environment-detector.yml. (Self says "noted").

Re: IoU threshold... I personally wouldn't use 0.5 unless you also set a very small maximum box size, at that point you're de facto just deleting all detections in an area of the image, which is a totally reasonable thing to do, but there are easier ways to do it, that don't require looking at lots of images. I never reduce this below 0.85.

But what confidence threshold are you using? I noticed that the default threshold is set to 0.8, which was sensible for MDv4, but not sensible for MDv5. I typically set this to just a little lower than the confidence threshold I'm going to use for image review, so maybe something like 0.1. If that's still at the default, try it at 0.1?

Oh, i did not notice that the default --confidenceMin was at 0.8. I should def try to lower that.

FWIW the only parameter I usually play with is the occurrenceThreshold (the number of matching detections in the same place that are required to generate a candidate), and I typically never lower this below the default of 20. I tune this number to make sure that I'm going to spend around five minutes on the whole review process, so if I have a batch that's 50,000 images in total, I'll leave it at 20, but if I have a batch that's 2M images, I might raise this as high as 75 and just worry about the really egregious repeat detections. It's a little bit of guesswork, but I'm basically shooting for around 1000 candidate detections (which could be a couple hundred thousand of actual images).

And the other important bit is making sure that the script's concept of "camera" is accurate. If each folder is a camera, you're good to go with the defaults, but if you have a more complex folder structure, or if your camera creates overflow folders, you may have to tinker just a bit.

And the important bit is making sure that the script's concept of "camera" is accurate.

When running this at the command line, the only tool you have to get non-default camera identification (other than moving your images around) is the "nDirLevelsFromLeaf" parameter. But if sometimes a "leaf node" folder is a camera, and sometimes it's not, you can't easily catch that. It may not be worth worrying about, nothing bad happens in the case of typical RCNX100/RCNX101/etc. overflow folders, you just miss some opportunities for repeat detections to stack up. But if you need to do something more complicated, you may have to call this from Python and use the "customDirNameFunction" option:

https://github.com/microsoft/CameraTraps/blob/main/api/batch_processing/postprocessing/repeat_detection_elimination/repeat_detections_core.py#L139

I can't think of a great way to expose this at the command line.

<https://github.com/microsoft/CameraTraps/blob/main/api/batch_processing/postprocessing/repeat_detection_elimination/repeat_detections_core.py | repeat_detections_core.py>

<pre><code> customDirNameFunction = None </code></pre>

<a href="https://github.com/microsoft/CameraTraps">microsoft/CameraTraps</a>

Ok, in my case, setting nDirLevelsFromLeaf = 1 instead of 0 is just what I need. We just have overflow folders at the last level. And that is already exposed at CL. Thanks Dan!

That's fine as long as you always have that overflow folder. I would just double-check. I think this is the default on most cameras, but Bad Stuff will happen if nDirLevelsFromLeaf == 1 and that RCNX100 folder or similar isn't present for every camera.

So far, the number of folder detected by the script is my expected number of cameras in my site folder (the folder I'm running the script on). I'll double check, but I'm pretty sure we always have that (at least) RCNX100.

There is so much information here. I will save it as a reference!

Also see this thread (more stuff about repeat detection elimination):

https://aiforconservation.slack.com/archives/CM7GR4DB6/p1686105494925979

What do you want from it? How much time dd you want to deficate? Per day/in total? How much money?

I did Le wagon in person in brussels (after seeing a post from @Alessandra Sellini mentioning it). For me it was really good. It is quite expensive, but i really needed the intensity and the peer pressure (as in learning with others and being pushed), and i was satisfied with the content and pedagogy too.


I used to work in ed tech and did all kinds of courses before, which also worked, (and probably you can find the content online for cheap/free) but when i did the bootcamp i didnt have the right frame of mind to be able to succeed in such a time frame (9 weeks) on my own.

Happy to answer any questions

I want to shift my career from strictly animal behavior to conservative technology, and I'm really interested in data science and machine learning experience so I can be prepared for a PhD in that field
I'm planning to do something full time, preferably remotely, and am on the hunt for various scholarships to help with funding

I tried doing part time courses but my current job schedule is too hectic to keep up with anything on the side, so I'm using self-pressure of being a full time student again to make sure I'm really focused on finishing the programs I sign up for! I'd love to chat more about the various programs you've had experience with!!

I haven't found any useful scholarship options. Sometimes jobs give edu budget. Some courses offer discounts (for example for women in stem).
I did Le wagon and was very satisfied, a friend did founders and coders (but not the ML one, they added that one more recently, and I am also not sure if they offer an online version).

Online masters are also an option, but I think the fastest you could finish that is 1.5/2 years. There are some offered by University of London (Data and CS), Colorado via coursera, https://omscs.gatech.edu/home. But not sure how much peer pressure and peer learning comes with these options. (somewhere between 5-15k)

I went to do my bootcamp in person because I knew that that's the surest way that I'll be focused on learning for 8+ hours a day. And in the case of le wagon, it's 9 weeks, so one can deal with an unsustainable amount of pressure as well.

How much time do you want to give it?

Hi @matteo redana - this is a hot research area, where you build a 3D model, using few images. I can share some existing research which may be useful for your research.

In this paper they have created a 3D rendered model of a zebra using an open-source technique GRADE to identify them later in drone imagery. Maybe it gives you some ideas to work on!

https://arxiv.org/abs/2305.00432

arXiv.org

Synthetic Data-based Detection of Zebras in Drone Imagery

Nowadays, there is a wide availability of datasets that enable the training of common object detectors or human detectors. These come in the form of labelled real-world images and require either a significant amount of human effort, with a high probability of errors such as missing labels, or very constrained scenarios, e.g. VICON systems. On the other hand, uncommon scenarios, like aerial views, animals, like wild zebras, or difficult-to-obtain information, such as human shapes, are hardly available. To overcome this, synthetic data generation with realistic rendering technologies has recently gained traction and advanced research areas such as target tracking and human pose estimation. However, subjects such as wild animals are still usually not well represented in such datasets. In this work, we first show that a pre-trained YOLO detector can not identify zebras in real images recorded from aerial viewpoints. To solve this, we present an approach for training an animal detector using only synthetic data. We start by generating a novel synthetic zebra dataset using GRADE, a state-of-the-art framework for data generation. The dataset includes RGB, depth, skeletal joint locations, pose, shape and instance segmentations for each subject. We use this to train a YOLO detector from scratch. Through extensive evaluations of our model with real-world data from i) limited datasets available on the internet and ii) a new one collected and manually labelled by us, we show that we can detect zebras by using only synthetic data during training. The code, results, trained models, and both the generated and training data are provided as open-source at <https://eliabntt.github.io/grade-rr>.

Original URL: https://arxiv.org/abs/2305.00432

Yes there are models to count fish. You can also check this blog which uses Segment Anything for counting grains.

https://www.thinkevolveconsulting.com/2050-2-counting-grains/

Here's a version of the megadetector for fishes:
https://github.com/warplab/megafishdetector

Another interesting usecase in case you want to identify the species is run the image crop through one of the multi-modal generative models like Bard and you can even get the species classified.

Thanks Aakash. I'll look into this.

I have been maintaining a list of all the publicly-available models for finding fish in imagery that I'm aware of (including MegaFishDetector, which Aakash mentioned):

https://github.com/agentmorris/agentmorrispublic/blob/main/fish-datasets.md#publicly-available-models-for-fish-detection

@Filippo Varini and I have also been keeping track of datasets suitable for training models like this:

https://github.com/filippovarini/filippo_public/blob/master/fish-datasets.md

You could also take a look at https://lila.science/otherdatasets#:~:text=horses%20(code)-,Marine/freshwater%20images,-Also%20see%20the

LILA BC

List of other conservation data sets - LILA BC

LILA BC - List of other conservation-related data sets

Written by

lilawp

Est. reading time

38 minutes

Original URL: https://lila.science/otherdatasets#:~:text=horses%20(code)-,Marine/freshwater%20images,-Also%20see%20the

Hey @Jessica Bellworthy, I'm aware of HandwritingOCR (a subscription service) which offers structured outputs (e.g. the screenshot I attached) but if you want it to be really reliable you may want to finetune the process on your scans and the expected format (I bet you're reusing the same dataframe templates for these thousands of pages?)
I also tried gpt4vision to export JSON and it does okay, not sure how cost-effective it would be with thousands though (second picture attached).
I don't know how much effort you want to spend automating this process but I'd be happy to chat more/help out.

Oh Nice thank you. Already better than I did. I'm going to check out HandwritingOCR and get back to you with questions. Thank you for the reply.

Hi @Jessica Bellworthy I was hoping for something similar, but when I tried img2table I got hung up installing dependencies on my M2 Macbook. Might be worth a test.
https://github.com/xavctn/img2table/tree/main
I had one question about your table: Some of the cells have mixed content in them (like a number and a note). Would it be possible to add a separate Comment column? Or is there too much of a legacy? Seems it would be easier to parse if the cells always contained just numbers.

https://medium.com/@lidores98/image-table-to-dataframe-using-python-ocr-773c8afb713d

Medium

Image Table to DataFrame using Python OCR

Ever had an image of a table and wanted to get the data into your DataFrame? well, I have the article for you!

Reading time

5 min read

Original URL: https://medium.com/@lidores98/image-table-to-dataframe-using-python-ocr-773c8afb713d

Interesting, I’ve only used RAID with HDD arrays, where it provides a large speedup. What kind of connection do you have from computer to hard drives? Is it possible that you are i/o bottlenecked by the physical connection?

So far I have only used it on videos but based on the way the pipeline is set up you extract the videos into frames to do the annotations for training 🙂
So i think you would just need to skip the extraction part and i think this may be easier if you are using their jupyter notebook instead of the software...
here are some notebooks with examples : https://github.com/DeepLabCut/DeepLabCut/blob/main/examples/README.md

<https://github.com/DeepLabCut/DeepLabCut/blob/main/examples/README.md | README.md>

<p><strong>Demo Jupyter &amp; Colaboratory Notebooks:</strong><br><br><a href="https://camo.githubusercontent.com/bfa27d783cc433684c389f4c517fbb22fcb25830d8e6977ae84fa57435cb9854/68747470733a2f2f696d616765732e73717561726573706163652d63646e2e636f6d2f636f6e74656e742f76312f3537663664353163396637343536366635356563663237312f313537323239333630343338322d573642574136334c5a394a3852374e30514541352f6b6531375a77644742546f6464493870446d34386b497736596b525545796f67653438353875414a66614d55717378525571716272316d4f4a594b66495052374c6f4451396d58504f6a6f4a6f71793831533249384e5f4e34563176556235416f494949624c5a685659774c3849654467365f33422d42527546346e4e724e63516b5675415437746445726430775146454746536e483977555069493862476f582d455161646b624c494a77687a6a4970773339332d754577534b4f37565a494c39674e5f536235495f644c77765772796a65434a672f646c635f6f766572766965772d30312e706e673f666f726d61743d3130303077">DLC</a><br><br>We provide a Project Manager GUI that will walk you through the major steps and options of the DeepLabCut Toolbox. However, there are more options and features that can be accessed by running the code in an interactive environment, such as Jupyter**. Moreover, if you don't have a GPU, you can create your project on any computer, then move your project to the cloud to use GPUs. To do this, we provide you with Google Colaboratory Notebooks (see </examples#demo-deeplabcut-training-and-analysis-on-google-colaboratory-with-googles-gpus|Demo using Google Colaboratory below>).<br><br><strong>Demo 1: run DeepLabCut <JUPYTER/Demo<em>labeledexample</em>Openfield.ipynb|on our open-field data></strong><br><br>• This will give you a feel of the workflow for DeepLabCut. Follow the instructions inside the notebook!<br><br>Note, the notebooks with labeled data: <JUPYTER/Demo<em>labeledexample</em>MouseReaching.ipynb|reaching data>, or <JUPYTER/Demo<em>labeledexample</em>Openfield.ipynb|open-field data> can be run on a CPU, GPU, etc. The one with the open-field data even achieves good/okay results, when trained for half an hour on a GPU! (To note, this is NOT the full dataset that was used in Mathis et al, 2018)<br><br><strong>Demo 2: Set up DeepLabCut on <JUPYTER/Demo_yourowndata.ipynb|your own data></strong><br><br>• Now that you're a master of the demos, this Notebook walks you through how to build your own pipeline: <br> • Create a new project <br> • Label new data <br> • Then, either use your CPU, or your GPU (the Notebook will guide you at this junction), to train, analyze and perform some basic analysis of your data.<br><br>For GPU-based training and analysis you will need to switch to either our <a href="https://deeplabcut.github.io/DeepLabCut/docs/docker.html">supplied Docker container</a>, or you need to <a href="https://deeplabcut.github.io/DeepLabCut/docs/recipes/installTips.html?highlight=gpu#how-to-confirm-that-your-gpu-is-being-used-by-deeplabcut">install your local GPU</a> in an Anaconda Env, or use Google Colab, more below: <a href="https://colab.research.google.com/github/DeepLabCut/DeepLabCut/blob/master/examples/COLAB/COLAB_YOURDATA_TrainNetwork_VideoAnalysis.ipynb">Open in Colab</a><br><br><strong>Demo 3: DeepLabCut training and analysis on Google Colaboratory (with Google's GPUs!):</strong><br><br>We suggest making a "Fork" of this repo, git clone or download the folder into your google drive, then linking your google account to your GitHub (you'll see how to do this in the Notebook below). Then you can edit the Notebooks for your own data too (just put <a href="https://colab.research.google.com/">https://colab.research.google.com/</a> in front of the web address of your own repo).<br><br>• You can use Google <a href="https://colab.research.google.com">Colaboratory</a> to demo running DeepLabCut on our data. Here is an example colab-ready Jupyter Notebook for the open field data, which you can launch by clicking the badge below: <a href="https://colab.research.google.com/github/DeepLabCut/DeepLabCut/blob/master/examples/COLAB/COLAB_DEMO_mouse_openfield.ipynb">Open in Colab</a><br>• Using Colab on your data for the training and analysis of new videos, i.e. the parts that need a GPU! <br><a href="https://colab.research.google.com/github/DeepLabCut/DeepLabCut/blob/master/examples/COLAB/COLAB_YOURDATA_TrainNetwork_VideoAnalysis.ipynb">Open in Colab</a><br>1. Click Open in Colab to launch the notebook.<br>2. Make the notebook live by clicking 'Connect' in the Colab toolbar, and then click "Runtime &gt; Change Runtime Type &gt; and select Python3 and GPU as your hardware. Follow the instructions in the Notebook.<br>3. Be aware, they often don't let you run on their GPUs for very long (&gt;6 hrs) without a Pro account, so make sure your <code>save_inters</code> variable is lower for this setting.<br><br>Here is a demo of us using the Colab Notebooks: <a href="https://www.youtube.com/watch?v=qJGs8nxx80A">https://www.youtube.com/watch?v=qJGs8nxx80A</a> &amp; <a href="https://www.youtube.com/watch?v=j13aXxysI2E">https://www.youtube.com/watch?v=j13aXxysI2E</a><br><br><em>Warning: Colab updates their CUDA/TensorFlow likely faster than we can keep up, so this may not work at all future points in time (and, as a reminder, this whole package is released with a </LICENSE|LICENSE> that implies no Liability and no Warranty).</em><br><br><strong>Using 3D DeepLabCut:</strong><br><br>Ready to take your pose estimation to a new dimension? As of 2.0.7+ we support 3D within the package. Please check out the dedicated 3D<em>Demo</em>DeepLabCut.ipynb above for more details!<br><br><strong>Using the DLC Model Zoo:</strong><br><br>We provide a COLAB notebook to use the growing number of networks that are trained on specific animals/scenarios. Read more here: <a href="http://www.mousemotorlab.org/dlc-modelzoo">http://www.mousemotorlab.org/dlc-modelzoo</a>. This code will also create a new project folder so you can refine, add new bodyparts or label other objects, and re-train. Launch COLAB here: <a href="https://colab.research.google.com/github/DeepLabCut/DeepLabCut/blob/master/examples/COLAB/COLAB_DLC_ModelZoo.ipynb">Open in Colab</a><br><br><strong>Using Python/iPython:</strong><br><br>All of DeepLabCut can be run from an ipython console in the program <strong>terminal</strong>! Go </docs/UseOverviewGuide.md|here> for detailed instructions!<br><br>We also have some video tutorials to demonstrate how we use Anaconda and Docker via the terminal:<br><br><a href="https://www.youtube.com/watch?v=7xwOhUcIGio">https://www.youtube.com/watch?v=7xwOhUcIGio</a> &amp; <a href="https://www.youtube.com/watch?v=bgfnz1wtlpo">https://www.youtube.com/watch?v=bgfnz1wtlpo</a><br><br>**** You can download DeepLabCut &amp; associated files:**<br><br>To have a copy of DeepLabCut on your own computer, we recommend using <strong>Anaconda to install Python and Jupyter Notebooks, see the </docs/installation.md|Installation> page</strong>. Then, on your local machine using these notebooks to guide you, you can (1) demo our labeled data (or create your own), (2) create a project, extract frames to label, use the GUI to label, and create a training set for the neural network(s).<br><br>We suggest making a "Fork" of this repo and/or then place DeepLabCut files in a folder: <br><code>git clone &lt;https://github.com/DeepLabCut/DeepLabCut&gt;</code> <br>so you can access it locally with <strong>Anaconda.</strong> You can also click the "download" button, rather than using <code>git</code>. Then you can edit the Notebooks as you like!</p>

<a href="https://github.com/DeepLabCut/DeepLabCut">DeepLabCut/DeepLabCut</a>

I'm happy to help if I can 🙂

Thanks heaps! I got a helpful response from the DLC team on GitHub and there is now a direct implementation for image inference with SuperAnimal (here: https://github.com/DeepLabCut/DeepLabCut/issues/2789). Although the notebook wasn't in the main branch which explains why it was so hard to find!

#2789 Inferencing images with SuperAnimal

<p>Hi, I've read the docs but couldn't find anything on inferencing on images instead of videos. Just wondering if anyone has any pointers? I'd like to get SuperAnimal running, but it seems like SuperAnimal hasn't been properly implemented yet?<br><br>As an example, I tried running the example code from the notebook <code>DeepLabCut/examples/testscript_superanimal_inference.py</code> and the docs as follows:<br><br>``` video<em>path = 'demo-video.mp4' superanimal</em>name = 'superanimal<em>quadruped</em>hrnetw32'</p> <p>deeplabcut.video<em>inference</em>superanimal([video], superanimal<em>name, model</em>name='resnet<em>50', video</em>adapt = True)</p> <p>``<code>&lt;br&gt;&lt;br&gt;However, all returned this error:&lt;br&gt;&lt;br&gt;ValueError:</code>modelname=superanimal<em>quadruped</em>None<code>should be one of: full_human, full_cat, full_dog, primate_face, mouse_pupil_vclose, horse_sideview, full_macaque, superanimal_topviewmouse_dlcrnet, superanimal_quadruped_dlcrnet, superanimal_topviewmouse_hrnetw32, superanimal_quadruped_hrnetw32, superanimal_topviewmouse, superanimal_quadruped, superanimal_bird_fasterrcnn_mobilenet_v3_large_fpn, superanimal_bird_ssdlite, superanimal_bird_resnet_50, superanimal_topviewmouse_fasterrcnn_mobilenet_v3_large_fpn, superanimal_topviewmouse_fasterrcnn_resnet50_fpn_v2, superanimal_topviewmouse_hrnet_w32, superanimal_topviewmouse_resnet_50, superanimal_quadruped_fasterrcnn_mobilenet_v3_large_fpn, superanimal_quadruped_fasterrcnn_resnet50_fpn_v2, superanimal_quadruped_hrnet_w32, superanimal_quadruped_resnet_50.&lt;br&gt;&lt;br&gt;I tried adding</code>modelname` as a parameter but it seems like it is not actually a parameter?<br><br>Either way, ideally I'd like to know if it is possible to use the models on some camera trap images that I have. Thanks!</p>

Assignees

<a href="https://github.com/n-poulsen">@n-poulsen</a>

Labels

question, DLC3.0🔥

<a href="https://github.com/DeepLabCut/DeepLabCut">DeepLabCut/DeepLabCut</a>

Hey Omer! I'm a ML practitioner and I'd be happy to have a look at your dataset (no need for compensation). You can DM me if you'd like

Just confirming: you are trying to identify individuals, right, not just the number of individuals present in a given image?

Assuming I've got that right, my first question for individual ID problems is always: can you tell one individual apart from another in a video? I.e., are the markings sufficiently distinct that human experts do this? It's not quite never the case that AI can do an identification task that human experts can't, but it's almost never the case.

Thanks both! @Sebastien Ouellet I'll DM you.

@Dan Morris I see these as two sides of the same coin. If I can identify individuals, I can count how many I had.
We are aware that this is a difficult task, which is why our hopes would not be to have full id of all individuals, but to be able to have a good estimate of the minimum number of individuals.

I think human experts should be able to it at least partially, but it might require an insane attention to little details like the combination of variables like the length of the tail and the distance between the ears.

Hi Omer - believe you have got some good inputs from Dan and Sebastian. But still I would like to offer my help, if its required. This will be a pro-bono.
Alternatively, you could open source the dataset on Zenodo (or other platform) and let us work on it! 🙂

If you are interested. I can share a few usecases/connects to govt officials from India.

@Quentin Geissmann ?

check out this approach that i used for counting grains. maybe it can be adopted for your usecase

https://www.thinkevolveconsulting.com/2050-2-counting-grains/

Think Evolve Consultancy

Counting Grains!

Dive into the methodologies of fine grained segmentation and counting to elevate your analytical capabilities and precision.

Est. reading time

1 minute

Original URL: https://www.thinkevolveconsulting.com/2050-2-counting-grains/

Nice, thanks for sharing, Aakash.

Did the editor express specific concerns about Zenodo? Zenodo is widely accepted and checks all the boxes editors usually like.

Also, can you describe the dataset a bit? Is it new data, or is it a composite of datasets collected from the Web or from other datasets? About how large is it?

Thanks @Dan Morris for the reply! And sorry for my late reply, I thought it is best @Josip Saric takes over as one of the authors, but then I forgot to add him to the slack last week 🙈

Hey Dan, thanks for the interest in our situation. Sorry for the late reply. No specific concerns about Zenodo were expressed. Perhaps the issue is that it does not support automatic metadata export in ISO format. My plan is to create the ISO metada xml with some other tool, and then upload it to the dataset repository. What do you think about this solution?

Dataset contains around 2000 high-res images of fish each labeled with 18 keypoints. The data has not been previously published and has been collected by one of the coauthors. It is like 12.0 GB in size.

This sounds like a perfect fit for Zenodo: the only downsides of Zenodo are that (1) there's a size limit of 50GB (a little more by request IIURC) and (2) downloads are a little slow, which only matters when you get close to 50GB. Neither of these are an issue here. I agree that there's no benefit to having the site generate metadata automatically, for datasets that are updated infrequently it's almost definitely better to just upload the metadata you want. COCO is probably the most standard format for keypoint annotations, but any standard-ish format will be fine.

I asked all these questions because I'm happy to host this dataset on lila.science, but in this case, given the small size of the dataset, I think you're better off on zenodo where you have total control and don't have to go through me as an intermediate. But if I'm missing something and there's a reason LILA is a better fit than Zenodo, reach out to me at agentmorris@gmail.com .

Thank you for your thoughts. I had a look at your project, and it looks great! I think I already found some interesting datasets that we might find useful in our future work!

I'm working on my PhD in ecology doing stats on a field study. Happy to discuss approaches. Whats your question/data?

Hey thanks so much for being in touch! Our group has a lot of different datasets & questions, and so we were looking for someone who we might be able to check in with, in a somewhat consistent way

Hi Lionel — this sounds like a really interesting direction! Could you share a bit more context?

• What org are you with, and how is your social science framework currently documented or used?


• What kinds of tasks do you imagine the AI supporting (e.g. staff training, intervention design, literature synthesis)?


• When you say “pushing internally,” do you mean supporting adoption or decision-making?


• Any constraints around data privacy given the wildlife-crime focus?


• What’s your timeline and do you have any resources/budget allocated yet?

CV4Ecology 🙂

https://cv4ecology.caltech.edu/

oh that's a great resource we should add! I'll do that now!

Added!

@Sam Lapp :)

@Inês Nolasco wrote a paper about individual id recently, not birdnet tho https://arxiv.org/pdf/2409.08673

Hi Eriberto, we actually have a tutorial notebook from today's workshop at ICTC that does exactly this!
https://colab.research.google.com/drive/1unHYpWmoFLjnErOszQyjF-7KzA9CB3N3
For more context you can also check out the workshop slides and other workshop materials
https://drive.google.com/drive/folders/1ybiRy1Wk2r5EswUeLAA2H4SqzCI0POKc
And this GitHub associated with my Ovenbird individual ID project
https://github.com/sammlapp/ovenbird-individual-recognition

Of course reach out if you have any questions or want to discuss your project

colab.research.google.com

Google Colab

Original URL: https://colab.research.google.com/drive/1unHYpWmoFLjnErOszQyjF-7KzA9CB3N3

Thanks a bunch everyone! @Sam Lapp would you potentially have any materials using the BirdNET analyzer gui? I have been attempting to use that with my recordings. My next step is using python and I will practice with the material you shared (I am very new to python).

I don't have experience with using the BirdNet gui for this, but it does provide support for generating embeddings

Thank you again! I will post my progress with BirdNet gui and try out your tutorial.

@Jason Holmberg (Wild Me)?

@Matt Weldy Happy to talk: holmbergius@gmail.com

Thanks! Email heading your way soon.

Being a total sap here, but I LOVE how much this community shows up to help each other

It really is pretty incredible!

Re-your old post, you can find it in the message archive 🙂 http://beerylab.csail.mit.edu/AIforConservationArchive/

Thank you very much!

Good reminder that we need to re-up the archive!

Hi @Eriberto, as far as I know the BirdNET analyzer GUI does not leverage GPU hardware. Therefore, having a powerful computer won’t speed up your workflow very much. Most modern laptops will work decently well. For larger throughput, the github.com/kitzeslab/bioacoustics-model-zoo provides gpu-compatible embedding models with a Python API, and the Dipper app has Beta functionality for running these workflows from a desktop gui

Hello @Sam Lapp, thanks so much for that information. I remember seeing that github from your lab and I plan to get into that as I learn more python. I will look into the dipper app as well. Thanks!

Happy to work with you on this. We just implemented PairX to explain MiewID results. Happy to share our thinking and benefit from yours. Looks like a great survey!

Nice! Here, I'm comparing CRP and HiResCAM on MiewID and MegaDescriptor (swin based). I will take a look at PairX, I didn't know it, but I just found the paper for it. Happy to be in touch and share thoughts yes, thank you!

@Daniel Thornton