heya @Mike C... better late than never? would still be interested to chat a little about this; maybe loop in @Jason Holmberg (Wild Me) and @Jason Parham as well.

(& @Jason Parham )

Should we set up a time next week? Also, @Mike C thanks for offering to share data! Annotated face data for wrasse may be helpful in our future research!

@Mike C we are jason@wildme.org, jon@wildme.org, and parham@wildme.org

yup, i think i am pretty open next week as well.

Done

Hi @Jason Holmberg (Wild Me)! Thanks a lot for sharing these! These are great!

No experience with Megadescriptor, but with Multispecies re-ID I believe @Lasha Otarashvili has seen similar, incredibly high top-1 ID prediction accuracy for Amur tigers and the Tunisian horse face dataset, which we have viewed more as a signal that the dataset is relatively small and not diverse enough in terms of the imagery available for each individual. The datasets are still useful in a cluster of real world, multispecies ID modeling, but I would be surprised in a single species context if the 98%+ ID prediction accuracy generalized well outside of the training data.

Hi @Antón Álvarez, and thank you for the shout-out! ❤️ It is highly appreciated!

Regarding performance, I partially agree with Jason that it is most likely a sign of overfitting to some characteristics of your dataset or partial train --> test data leakage. However, I’m impressed by the zero-shot performance (80%), as it never saw any bear.

Speaking about what MegaDescriptor “has seen” during training. What do you think about providing data for the future development of MegaDescriptor-V2?

Ditto - a perfect 100 often tells as much about the data and it does about the model. From a production perspective, this means ‘with real data, expect good performance, but not THAT good'. A good look at the data should tell more. A major factor contributing to data being 'easy' is the camera perspective being homogenous concerning the animal’s features. This can be due to pictures taken in burst mode, or an alignment pre-processing step. Creating images that look very similar or ‘near duplicates’.

Sometimes you might get these ‘near-duplicates’ in clusters. Think of 3x2 bursts of 2 images of the same individual captured during 3 separate encounters with some time between encounters. Each of the same encounter pairs will be much more easily matchable against its counterpart than it will be with the rest. In such cases, top-1 accuracy is misleading as the perfect score can be achieved by correctly matching just one of all valid samples. mAP is more balanced for capturing performance across all valid samples - some of which can be more diverse. Drawback is that mAP is less directly interpretable. High rank-1, low mAP is often a sign of ’near-duplicates’ being present in the data.

Thanks for sharing the work @Antón Álvarez. Also, cool to see MegaDescriptor zero-shotting this.

Thanks all for your feedback! Sadly we are not able to share the data since it is property of BearID. However I can give you some more insight on the data. We've build a very performant bear face detector to pre-process the data so there will indeed be little variation in the image, but that's kind of the point, we do not align the images though.
The dataset is quite significant containing ~5k images of ~150 individuals with several having 50+ sightings.
I agree train test leakage seems likely though we took great care in crafting the train test splits base on literature, however, testing to validate this is still in progress.
Regarding that the problem might be too easy, next to MegaDescriptor we are trying various other models like LightGlue and metric learning (with different backbones and loss functions), and none of them scored 100% where the best other performing model is a fintuend ArcFace Covnext metric learning model.

Hi @Sako Arts, Thank you for some insights. What I would suggest to prevent the data leakage. Just do the train/test split using the information about location or time. For example, use encounters from different days or locations.

I've found the bug that was causing 100% accuracy; now it's at 92% with the basic model. These results are more believable. The error was in my evaluation code, although the zero-shot accuracy is correct.

@Lasha Otarashvili Thank you for the explanation about Top-1 and mAP.

@Lukas Picek Does the MegaDescriptor have any capability to generate saliency maps, or are you considering incorporating this feature in MegaDescriptor 2? From my experience as a biologist working with the Iberian lynx, this feature is crucial for the usability of MegaDescriptor. For instance, I would prefer using SIFT (72% accuracy) over MegaDescriptor (75% accuracy) for leopards. However, this also depends on the species; for example, for the heads of sea turtles or animals that can be easily identified by a human, I wouldn't see the need for saliency maps as much. But I am biased towards the lynx, which is difficult to identify.
@Jason Holmberg (Wild Me)Wild Me employs a metric learning approach with saliency maps that I appreciate, but they are more complicated to interpret than the ones from HotSpotter.

@Antón Álvarez I agree. Hotspotter's visualizations are easy to understand and relate to. MiewID's Grad-CAM based visualizations are good for checking in on training results, but they're not very useful in a production environment when evaluating match candidates. Worse, deep learning can pick up on features that are less relatable to humans than more obvious patterning (e.g., beluga dorsal ridges versus scars), and adding visualization can significantly add to inference time and eat up disk space. Wild Me might run 2-5k ID jobs per day across our platforms on average, and we actually turn off MiewID visualization in most cases to optimize for speed. Even for Hotspotter, we try to delete visualizations within two weeks as they eat up disk space quickly. Visualization is very much an active field of research, and we're looking for solutions that are better than Grad-CAM and more relatable.

@Jess Tam We run HotSpotter in the Wildbook platform but are replacing it quickly with the MiewID algorithm (multispecies) for scalability reasons.

Which species are you working on with HotSpotter?

I'm trying to re-id an unknown population of feral cats, hence wanting to get the one-to-one model working since we have no existing database. What is your experience with MiewID?

This is our current multispecies production model for MiewID:
https://huggingface.co/conservationxlabs/miewid-msv2

Very good but feral cats are hard and has no feral cat training data (yet).

huggingface.co

conservationxlabs/miewid-msv2 · Hugging Face

We’re on a journey to advance and democratize artificial intelligence through open source and open science.

Original URL: https://huggingface.co/conservationxlabs/miewid-msv2

I see, maybe this domestic cat dataset might help?: https://wildlifedatasets.github.io/wildlife-datasets/datasets/#catindividualimages

I'll give it a go anyways since I see that it's been trained on other patterned species, thanks heaps!

Just a fair warning that we normally run a detector model to match same viewpoint and normalize orientation in advance of ID. miewid-msv2 is powerful but generally trained on relatively clean mark-recapture datasets. Advance study design on which features and viewpoints to match is important.

Thanks for the heads up, would there be a script somewhere that I can reference?

Hi @Jason Holmberg (Wild Me) , is there a way to cite miewid, eg paper or report?

@Lasha Otarashvili How would you like to cite MiewID, and should we add a citation.cff file to the repo?

Hi @Kostas Papafitsoros a writeup is underway. I have added a way to cite here

@Jess Tam Hi Jess, example notebooks are great to get going fast. They are done on a Beluga dataset that is single-viewpoint, though. For multiple distinct viewpoints (such as left vs right) for species that do not exhibit bilateral symmetry, the go-to approach is to handle each viewpoint+name combination as a separate individual. This can be done by configuring the name_keys argument. Conversely, there are cases where this is too aggressive, such as when you have viewpoints like 'frontleft', 'frontright' and are still partially matchable. In such cases you might opt to let the model learn to match cross-viewpoint, or map them to common 'left' and 'right' depending on experimentation. The model gets quite good at partial matching if the dataset allows for it.

Our general pipeline also includes orientation normalization - rotation only. This is done for legacy for compatibility with other algos we use. The CNN backbone is good at learning invariance. We have been trying to reduce that requirement by training with rotation augmentation, so that the model is not handicapped by it - so not a hard requirement. It is still not fully invariant to extreme (30deg+) rotation for most of the terrestrial species.

This is the way to go!

@Vojta Čermák - I am exploring your library. Can you point to the code blocks which can generate the above visualization.

HI, the visualization is from this notebook.

https://github.com/WildlifeDatasets/wildlife-tools/blob/main/examples/example-pairwise-sim.ipynb

The visualization function visualise_matches is from:
https://github.com/WildlifeDatasets/wildlife-tools/blob/main/wildlife_tools/similarity/pairwise/utils.py

Let me know if you are able to replicate that.

<https://github.com/WildlifeDatasets/wildlife-tools/blob/main/examples/example-pairwise-sim.ipynb | example-pairwise-sim.ipynb>

<a href="https://github.com/WildlifeDatasets/wildlife-tools">WildlifeDatasets/wildlife-tools</a>

<https://github.com/WildlifeDatasets/wildlife-tools/blob/main/wildlife_tools/similarity/pairwise/utils.py | utils.py>

<a href="https://github.com/WildlifeDatasets/wildlife-tools">WildlifeDatasets/wildlife-tools</a>