Hi @Emmanuel Azuh, have you seen this.
https://www.drivendata.org/competitions/82/competition-wildlife-video-depth-estimation/

DrivenData

Deep Chimpact: Depth Estimation for Wildlife Conservation

Help conservationists monitor wildlife populations. In this challenge, you will estimate the depth of animals in camera trap videos. Better distance estimation models can rapidly accelerate the availability of critical information for wildlife monitoring and conservation.

Original URL: https://www.drivendata.org/competitions/82/competition-wildlife-video-depth-estimation/

Hi @Oisin Mac Aodha I hadn't seen it but this is a good starting point! Thanks a ton 🙂

Hello, @Emmanuel Azuh I spent a bit of time this summer exploring this with some camera trapping data I have. Overall I was really surprised how well some of the trained models worked for predicting relative depths within images (like the monodepth model). I took it a few steps further and tested some photos with known distances and found that the predictions were fairly accurate in an absolute sense within a set distance, but that the model predictions were unpredictably different outside of a set range. If you pursue this further, I would be really interested to hear how it goes.

Thread Reply:

I'll keep you posted. The use case I'll eventually apply this to is targeting something like half a mile range so i'll likely run into some challenges but it should be a fun experience :)

Hi, ecologist here. Distance definitely has application - google ‘distance sampling’

Responding here since I can’t respond on the site without creating an account (I believe)

ML algorithm can do all of these. The limitation is the data. Pre-trained algorithms can work with much less data (e.g. dataset size of 1k-10k). But the more data you have, the higher the accuracy.

Yep, you have to create an account - with is rather the point 😂

But thank you for the reply!

I will say we did purposefully build it so you could see everything without an account so that nothing was locked away! We made all the info in the community as open and accessible as possible

We’ve often discussed this as a use case for iNaturalist data:

https://www.inaturalist.org/

inaturalist.org

iNaturalist

iNaturalist is a social network for naturalists! Record your observations of plants and animals, share them with friends and researchers, and learn about the natural world.

Original URL: https://www.inaturalist.org/

ohhh! I am currently working on a project related to this actually 🙂

Awesome! In what part of the world?

The data I am working with is for the Bega Valley in Australia

Jon - check out Dendra Systems too

Another idea: I'm inspired by the homegrown national Park / Pollinator Pathways groups here.

What if we set a goal to figure out how long it takes to walk to the closest pollinator garden?
Similarly I want to count, how many people in a community are within a 15-minute walk of a pollinator garden.
Wouldn't need AI, just a lot of Google API calls :-)

Ok, wow. Dendra does what I'm looking for. Next question: is it available in the U.S., and what does it cost

Hi there, wondering If Picterra would not be also what you look for. We ve had quite some customers detecting and mapping out invasive vegetation.

Hi Jon, I was triggered by the keyword 'roadside': a Dutch based company is developing a system that is attached to mowers, the MowHawk. They collect data from mowers and aim to use it for invasive species detection and monitoring.

this is a very large commercial space. https://www.nv5.com/geospatial/

https://www.overstory.com this start up is tackling this

I think this is a useful resource if you are looking for work on animals in computer vision, maybe if you scroll along the papers you can find what you need https://github.com/anl13/animal_papers

thank you

How about DeepLabCut? https://github.com/DeepLabCut/DeepLabCut

thanks Ed. Yes we are using deep lab cut but if I am not mistaken it is only for 2D pose. I am currently more interested in 3D

plus, it seems that there many papers out there but few with any useful working source code

actually, this seems close to what I am looking for https://github.com/lambdaloop/anipose (build on top of deeplabcut)

I was going to suggest anipose. Glad you found it

we contributed to this (for chimps) https://arxiv.org/abs/2003.00080 and code is here https://paperswithcode.com/paper/transferring-dense-pose-to-proximal-animal

arXiv.org

Transferring Dense Pose to Proximal Animal Classes

Recent contributions have demonstrated that it is possible to recognize the<br>pose of humans densely and accurately given a large dataset of poses annotated<br>in detail. In principle, the same approach could be extended to any animal<br>class, but the effort required for collecting new annotations for each case<br>makes this strategy impractical, despite important applications in natural<br>conservation, science and business. We show that, at least for proximal animal<br>classes such as chimpanzees, it is possible to transfer the knowledge existing<br>in dense pose recognition for humans, as well as in more general object<br>detectors and segmenters, to the problem of dense pose recognition in other<br>classes. We do this by (1) establishing a DensePose model for the new animal<br>which is also geometrically aligned to humans (2) introducing a multi-head<br>R-CNN architecture that facilitates transfer of multiple recognition tasks<br>between classes, (3) finding which combination of known classes can be<br>transferred most effectively to the new animal and (4) using self-calibrated<br>uncertainty heads to generate pseudo-labels graded by quality for training a<br>model for this class. We also introduce two benchmark datasets labelled in the<br>manner of DensePose for the class chimpanzee and use them to evaluate our<br>approach, showing excellent transfer learning performance.

Original URL: https://arxiv.org/abs/2003.00080

paperswithcode.com

Papers with Code - Transferring Dense Pose to Proximal Animal Classes

Implemented in one code library.

Original URL: https://paperswithcode.com/paper/transferring-dense-pose-to-proximal-animal

@mimi thanks a lot for this info! looks like a great effort. I have only skimmed but already noticed that you provide some ground truth labels -- which is indeed very beneficial, in addition to the nice transfer approach.

You’re looking to separate frames of the video that have detections and frames of the video that are empty?

if it’s frame extraction, you can use ffmpeg (with GPU acceleration if you have one)

I just recently had to deal with frame-wise prediction in videos. There's generally two Python interfaces available, OpenCV and PyAV, both piggybacking on ffmpeg and both not 100% perfect. I'd recommend OpenCV; could work as follows:

Open video and read frame-by-frame, with optional frame skip:

import cv2
import torch

reader = cv2.VideoCapture('path/to/video.mp4')

# read a frame
success, frame = reader.read()

if success:
  # convert frame to PyTorch tensor
  frame = torch.from_numpy(frame).permute((2,0,1)).float()

# to skip to the nth frame
frame_num = 5   # skip to 5th frame
reader.set(cv2.CAP_PROP_POS_FRAMES, frame_num)

# always close reader at the end
reader.release()

# get total no. frames in video
num_frames = int(reader.get(cv2.CAP_PROP_FRAME_COUNT))

# get frame rate of video
fps = reader.get(cv.CAP_PROP_FPS)

# length of video in seconds
duration = num_frames / fps

This is awesome!

we also have used ffmpeg to do frame extraction and then run ML to look for objects of interest in each frame. definitely a pretty crude method, but it works. (being able to maintain time-based knowledge between frames would be an improvement; but this is very simple and generally works for us.)

If you just want to extract every nth frame from a video, you may be able to use the videotoframes() function from our MegaDetector repo:

https://github.com/microsoft/CameraTraps/blob/main/detection/video_utils.py#L103

There is also a videofolderto_frames() function that runs that over a folder full of videos, optionally using multiple threads:

https://github.com/microsoft/CameraTraps/blob/main/detection/video_utils.py#L199

It uses OpenCV under the hood. LMK if you want to give it a try but you hit any snags.

<https://github.com/microsoft/CameraTraps/blob/main/detection/video_utils.py | video_utils.py>

<a href="https://github.com/microsoft/CameraTraps">microsoft/CameraTraps</a>

<https://github.com/microsoft/CameraTraps/blob/main/detection/video_utils.py | video_utils.py>

<a href="https://github.com/microsoft/CameraTraps">microsoft/CameraTraps</a>

I typically use ffmpeg to extract the whole video into frames and then run python (using OpenCV or Pytorchs torchvision stack) on the individual frames:
I use this command to extract the frames to preserv the timestamps of the frames:

ffmpeg -i video.mp4 -q:v 2 -s -f image2 -frame_pts true frames_to_annotate/frame_%05d.jpg

running ffmpeg in a separate process is quite quick, and one can then process many frames simultaneously in parallel by passing them all through pytorch/torchvision and whatever detector is desired

ffmpeg can be told to reduce the framerate to something lower adding -r 0.5 after the -i inputvideo.mp4 would reduce it to 0.5 fps

or whatever you need

Thank you so much for the responses everyone, I’ll check these all out and reply back with what works and any tips etc

Also I would be remiss if I didn't point out that if you want to preferentially extract frames that have birds in them, MegaDetector may be able to help. 🙂. But it sounds like you're looking at very dense, busy images that are different from what MD usually sees, so I don't know exactly how it would perform. I wouldn't be surprised if generic YOLOv5 - just trained on COCO - works as well as MegaDetector in that environment. I'm curious enough that if you were able to share some sample images/videos, I would be happy to take a look; email me at agentmorris@gmail.com if you are also curious and wouldn't mind sharing some samples.

There may be another challenge since if the videos are from a marketplace, you may find them behind cages? Are they similar to these images ? As Dan points out MDv5 may find it difficult due to the density of the objects, as well as a different environment. Maybe you can have a look at dense object detection models

A simple hack is using Adobe Premiere Pro to easily convert all the videos to frames. You can set your frame rate and picture quality as well.

could the cages be removed with a simple band-filter? run a 2-dimensional spacial FFT to create real and imaginary frequency images - the cage bars should appear as relatively stong peaks -strong enough that an algorithm could detect them automatically and compute a filter-matrix. multiply with the suppressive matrix in the frequency space, and reverse FFT back into image space and the cages should be mostly gone. alternatively a simple LPF (blurring) would probably reduce the effect of the bars significantly
finally, the frequency mask above can be expressed as a convolution - which could be learned - one could solve for a convolutional filter (aka train a CNN) that removes cage bars - if that is a frequent/common issue

I believe you are talking about using an image inpainting approach, to mask and remove the cages from the images and then run a detection model.

paperswithcode.com

Papers with Code - Image Inpainting

Original URL: https://paperswithcode.com/task/image-inpainting

that would be one way to do it, but learned inpainting models sometimes have a tendency of altering the parts of the image one wants to leave unaltered, so you need a mask that enforces to keep the non-cage parts alone - so you need a cage-segmentation algorithm first to detect which pixels are cage

Hi Eric and Aakash - I tried inpainting before but with no real luck! Aakash - some of the photos do look like what you shared - but they are usually closer up than that, and I am testing a model with one species per photo to start with. There was no real way to scale up and the images were quite distorted even after testing with a good inpainting GUI. I would be really interested to explore with the cage segmentation algorithm idea but I have little knowledge of spatial FFT models. Eric do you have any suggestions on where to start?

It will be interesting to use in-segmentation techniques or FFT transformation for removing the cages. Are there any sample image (10-15) that you can share. via DM if you dont want to share in a public forum. Also whats your end objective?