It’s easy in the digital age to amass tens of thousands of photos (or more!). Categorising these can be a challenging task, let alone searching through them to find that one happy snap from 10 years ago.
Significant advances in machine learning over the past decade have made it possible to automatically tag and categorise photos without user input (assuming a machine learning model has been pre-trained). Many social media and photo sharing platforms make this functionality available for their users — for example, Flickr’s “Magic View”. What if a user has a large number of files stored locally on a Hard Disk?
The problem
- 49,049 uncategorised digital images stored locally
- Manual categorisation
- No easy way to search (e.g. “red dress”, “mountain”, “cat on a mat”)
The solution
- Microsoft Azure subscription
- Microsoft Azure Computer Vision API
- Python scripting
Steps
- Obtain a Microsoft Azure cloud subscription (note – Azure is not free, however free trials may be available):
https://azure.microsoft.com/en-us/free/
- Start a cognitive services account from the Azure portal and take note of one of the “Keys” (keys are interchangeable):
https://portal.azure.com/
- Log in to your Linux machine and ensure you have python3 installed:
user@host.site:~> which python3 /usr/bin/python3
- Ensure you have these python libraries installed:
sudo su - pip3 install python-xmp-toolkit pip3 install argparse pip3 install Pillow exit - Obtain a copy of the image-auto-tag script:
git clone https://github.com/niftimusmaximus/image-auto-tag - Automatically tag, caption and categorise an image (e.g. image.jpg):
cd image-auto-tag./image-auto-tag.py --key XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX --captionConfidenceLevel 0.50 --tagConfidenceLevel 0.5 --categoryConfidenceLevel 0.5 image.jpgNote – replace key with one of the ones obtained from the Azure Portal above
Script will process the image:
INFO: [image.jpg] Reading input file 1/1 INFO: [image.jpg] Temporarily resized to 800x600 INFO: [image.jpg] Uploading to Azure Computer Vision API (length: 107330 bytes) INFO: [image.jpg] Response received from Azure Computer Vision API (length: 1026 bytes) INFO: [image.jpg] Appended caption 'a river with a mountain in the background' (confidence: 0.67 >= 0.50) INFO: [image.jpg] Appended category 'outdoor_water' (confidence: 0.84 >= 0.50) INFO: [image.jpg] Appending tag 'nature' (confidence: 1.00 >= 0.50) INFO: [image.jpg] Appending tag 'outdoor' (confidence: 1.00 >= 0.50) INFO: [image.jpg] Appending tag 'water' (confidence: 0.99 >= 0.50) INFO: [image.jpg] Appending tag 'mountain' (confidence: 0.94 >= 0.50) INFO: [image.jpg] Appending tag 'river' (confidence: 0.90 >= 0.50) INFO: [image.jpg] Appending tag 'rock' (confidence: 0.89 >= 0.50) INFO: [image.jpg] Appending tag 'valley' (confidence: 0.75 >= 0.50) INFO: [image.jpg] Appending tag 'lake' (confidence: 0.60 >= 0.50) INFO: [image.jpg] Appending tag 'waterfall' (confidence: 0.60 >= 0.50) INFO: [image.jpg] Finished writing XMP data to file 1/1 - Verify the results:
Auto tagging
API has applied “tags” which can be searched Auto captioning
API has captioned this image as “a beach with palm trees” Auto categorisation
API has applied the category “plant_tree” to this image Note – please see here for the API’s 86 category taxonomy
Script features
- Writes to standard XMP metadata tags within JPG images which can be read by image management applications such as XnView MP and digiKam
- Sends downsized images to Azure to improve performance
Example – only send image of width 640 pixels (original image will retain its dimensions)./image-auto-tag.py --key XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX --azureResizeWidth 640 image.jpg - Allows customisation of thresholds for tags, description and caption. This is useful because whilst good, the API is not perfect!
Example – only caption image if caption confidence score from API is 0.5 or above:
./image-auto-tag.py --key XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX --captionConfidenceLevel 0.5 image.jpg
Analytics Anvil 