It’s easy in the digital age to amass tens of thousands of photos (or more!). Categorising these can be a challenging task, let alone searching through them to find that one happy snap from 10 years ago.
Significant advances in machine learning over the past decade have made it possible to automatically tag and categorise photos without user input (assuming a machine learning model has been pre-trained). Many social media and photo sharing platforms make this functionality available for their users — for example, Flickr’s “Magic View”. What if a user has a large number of files stored locally on a Hard Disk?
- 49,049 uncategorised digital images stored locally
- Manual categorisation
- No easy way to search (e.g. “red dress”, “mountain”, “cat on a mat”)
- Microsoft Azure subscription
- Microsoft Azure Computer Vision API
- Python scripting
- Obtain a Microsoft Azure cloud subscription (note – Azure is not free, however free trials may be available):
- Start a cognitive services account from the Azure portal and take note of one of the “Keys” (keys are interchangeable):
- Log in to your Linux machine and ensure you have python3 installed:
email@example.com:~> which python3 /usr/bin/python3
- Ensure you have these python libraries installed:
sudo su - pip3 install python-xmp-toolkit pip3 install argparse pip3 install Pillow exit
- Obtain a copy of the image-auto-tag script:
git clone https://github.com/niftimusmaximus/image-auto-tag
- Automatically tag, caption and categorise an image (e.g. image.jpg):
cd image-auto-tag./image-auto-tag.py --key XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX --captionConfidenceLevel 0.50 --tagConfidenceLevel 0.5 --categoryConfidenceLevel 0.5 image.jpg
Note – replace key with one of the ones obtained from the Azure Portal above
Script will process the image:
INFO: [image.jpg] Reading input file 1/1 INFO: [image.jpg] Temporarily resized to 800x600 INFO: [image.jpg] Uploading to Azure Computer Vision API (length: 107330 bytes) INFO: [image.jpg] Response received from Azure Computer Vision API (length: 1026 bytes) INFO: [image.jpg] Appended caption 'a river with a mountain in the background' (confidence: 0.67 >= 0.50) INFO: [image.jpg] Appended category 'outdoor_water' (confidence: 0.84 >= 0.50) INFO: [image.jpg] Appending tag 'nature' (confidence: 1.00 >= 0.50) INFO: [image.jpg] Appending tag 'outdoor' (confidence: 1.00 >= 0.50) INFO: [image.jpg] Appending tag 'water' (confidence: 0.99 >= 0.50) INFO: [image.jpg] Appending tag 'mountain' (confidence: 0.94 >= 0.50) INFO: [image.jpg] Appending tag 'river' (confidence: 0.90 >= 0.50) INFO: [image.jpg] Appending tag 'rock' (confidence: 0.89 >= 0.50) INFO: [image.jpg] Appending tag 'valley' (confidence: 0.75 >= 0.50) INFO: [image.jpg] Appending tag 'lake' (confidence: 0.60 >= 0.50) INFO: [image.jpg] Appending tag 'waterfall' (confidence: 0.60 >= 0.50) INFO: [image.jpg] Finished writing XMP data to file 1/1
- Verify the results:
Note – please see here for the API’s 86 category taxonomy
- Writes to standard XMP metadata tags within JPG images which can be read by image management applications such as XnView MP and digiKam
- Sends downsized images to Azure to improve performance
Example – only send image of width 640 pixels (original image will retain its dimensions)
./image-auto-tag.py --key XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX --azureResizeWidth 640 image.jpg
- Allows customisation of thresholds for tags, description and caption. This is useful because whilst good, the API is not perfect!
Example – only caption image if caption confidence score from API is 0.5 or above:
./image-auto-tag.py --key XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX --captionConfidenceLevel 0.5 image.jpg
One thought on “Automatically tagging, captioning and categorising locally stored images using the Azure Computer Vision API”
Hey I know you wrote this a while ago, but I just wanted to still give you a big shout out for writing this.
I’m a photographer-turned-software-developer, and I am no longer interested in any sort of manual DAM.
This is exactly one of the puzzle pieces I needed for the solution I’m trying to come up with.