[Solved/share] Exiftool bash script to process image in a specific time range recursively.

N0x0n@lemmy.ml · edit-2 3 months ago

[Solved/share] Exiftool bash script to process image in a specific time range recursively.

AernaLingus [any]@hexbear.net · 4 months ago

Without looking too carefully at your script, I strongly suspect the issue is the time it takes to spin up an exiftool process once per file (not ideal for any program, but I find that exiftool is especially slow to start).

I have never done this myself, but apparently there’s an option -stay_open which allows you to start up exiftool once and have it continually read arguments from a file or pipe. If you were to write your script around this flag I suspect it would be much faster. Here’s a brief forum discussion which may prove useful.

If you decide to do that, let me know how it goes! Might use it myself should the need arise.

N0x0n@lemmy.ml · 4 months ago

Hello :)

I read your comment yesterday, but was to busy improving the script with the suggestion made above. I read and tried to understand how to use this switch but after some time roaming the web I found an answer from Phil Harvey:

You don’t need to use -stay_open if you want to do the same thing to a large number of files. Just use -@ with a file list as you mentioned. The -stay_open is useful to speed things up if you want to run exiftool a number of times with different options each time. (source)

I don’t think this Is applicable in my use case (or does it ? IDK) but even if it does… I have absolutely no idea how I should/would/could implemented it… I’m pretty noob+ in bash or any other programming language… :/

Thanks for your input and I feel sorry not being able to give it a shot.

PS: With the above advice from GenderNeutralBro I went from 16min to 11min on the same sample :) Quite an improvement for a rookie script !!

AernaLingus [any]@hexbear.net · 4 months ago

Yeah, I think the fact that you need to capture the output and then use that as input to another exiftool command complicates things a lot; if you just need to run an exiftool command on each photo and not worry about the output I think the -stay_open approach would work, but I honestly have no idea how you would juggle the input and output in your case.

Regardless, I’m glad you were able to see some improvement! Honestly, I’m the wrong person to ask about bash scripts, since I only use them for really basic stuff. There are wizards who do all kinds of crazy stuff with bash, which is incredibly useful if you’re trying to create a portable tool with no dependencies beyond any binaries it may call. But personally, if I’m just hacking myself together something good enough to solve a one-off problem for myself I’d rather reach for a more powerful tool like Python which demands less from my puny brain (forgive my sacrilege for saying this in a Bash community!). Here’s an example of how I might accomplish a similar task in Python using a wrapper around exiftool which allows me to batch process all the files in one go and gives me nice structured data (dictionaries, in this case) without having to do any text manipulation:

import exiftool
import glob

files = glob.glob(r"/path/to/photos/**/*", recursive=True)
with exiftool.ExifToolHelper() as et:
  metadata = et.get_metadata(files)
  for d in metadata:
    for tag in ["EXIF:DateTimeOriginal", "EXIF:CreateDate", "File:FileCreateDate", "File:FileModifyDate", "EXIF:DateAcquired"]:
      if tag in d.keys():
        # Per file logic goes here
        print(f'{d["File:FileName"]} {d[tag]}')
        break

This outline of a script (which grabs the metadata from all files recursively and prints the filename and first date tag found for each) ran in 4.2 s for 831 photos on my machine (so ~5 ms per photo).

Since I’m not great in bash and not well versed in exiftool’s options, I just want to check my understanding: for each photo, you want to check if it’s in the specified date range, and then if it is you want to copy/move it to a directory of the format YYYYMMDD? I didn’t actually handle that logic in the script above, but I showed where you would put any arbitrary operations on each file. If you’re interested, I’d be happy to fill in the blank if you can describe your goal in a bit more detail!

N0x0n@lemmy.ml · edit-2 3 months ago

Edit:

I’m leaving my comment in case you’re willing to read it through xD ! Another user proposed a hardware solution with parallel package to run my script with all my CPU cores. Fantastic solution ! Right now I get a processing time of 1min22s for 2200 files, that’s a huge improvement. It’s not the scripts efficiency itself that got improve so if you’re inspired and still want to fill the blanks, feel free ! Otherwise It’s alright and thank you for your help 😘 !

#!/bin/bash

if [ $# -eq 0 ]; then
    echo "Usage: $0 <filename>"
    exit 1
fi

# Concatenate all arguments into one string for the filename, so calling "./script.sh /path/with spaces.jpg" should work without quoting
filename="$*"

start_range=20170101
end_range=20201230

FIRST_DATE=$(/usr/bin/vendor_perl/exiftool -m -d '%Y%m%d' -T -DateTimeOriginal -CreateDate -FileModifyDate -DateAcquired -ModifyDate "$filename" | tr -d '-' | awk '{print $1}')
#echo $FIRST_DATE

if [[ "$FIRST_DATE" != '' ]] && [[ "$FIRST_DATE" -gt $start_range ]] && [[ "$FIRST_DATE" -lt $end_range ]]; then
        /usr/bin/vendor_perl/exiftool -api QuickTimeUTC -d %Y/%B '-directory<$DateAcquired/' '-directory<$ModifyDate/' '-directory<$FileModifyDate/' '-directory<$CreateDate/' '-directory<$DateTimeOriginal/' '-FileName=%f%-c.%e' "$filename"

else
        echo "Error"

fi

time find /home/user/Pictures/ -type f | parallel ./exif-test.bash

Hey thank you for your answer :)

Here’s an example of how I might accomplish a similar task in Python using a wrapper around exiftool

I also though maybe with python It could be faster and maybe easier? Python’s syntax is easier to understand and maybe easier to troubleshoot, but who am I to make any conclusion, I’m only putting things together while roaming the web, so I have no idea what I’m talking about xD !

Since I’m not great in bash and not well versed in exiftool’s options, I just want to check my understanding: for each photo, you want to check if it’s in the specified date range, and then if it is you want to copy/move it to a directory of the format YYYYMMDD? I didn’t actually handle that logic in the script above, but I showed where you would put any arbitrary operations on each file. If you’re interested, I’d be happy to fill in the blank if you can describe your goal in a bit more detail!

Yes, because my mother has a VERY large groupe of photos on different hard drives and I need to search through a specific date (That’s the purpose of my start_range, end_range) and order them to a directory of the format YEAR/MONTH. She’s looking for something specific and only somehow remembers the date, the problem is… A huge part of her photo metadata/folder is FUCKED UP… because of how she edited the files, extracted them into other folders (lot of duplicates…) moved them and even played arround with the metadata, tags…

After a few attempts of searching through the hard drives I just though I should make a script that automates everything, I like learning new things but I’m slowly giving up because trying to make something out without the necessary skill set and knowlege is a real pain… I already did something similar with FFMPEG (but was a 2 line bash script) so I though It might be possible for this to.

As a bullet point what I’m trying to achieve is to:

Search through all the files
Process only the files in a specific start_range end_range
Order them in a directory of the format YEAR/MONTH
Use the following date time metadata in that order (DateTimeOriginal-CreateDate-FileModifyDate-ModifyDate-AcquiredDate)

I do have an actually working script that exactly does that (Thanks to a every person here trying to give me a hand), but it’s kinda slow. exiftool alone will process the same folder sample in ±2 mintues while my actual script takes 10 mins (quite an improvement over my first script I managed to put together and took 16min !) for 2200 files.

#!/bin/bash

start_range=20160101
end_range=20221212

find /home/dany/Pictures  -type f -print0 | while IFS= read -r -d '' file_path; do
        image_path=$file_path

FIRST_DATE=$(exiftool -m -d '%Y%m%d' -T -DateTimeOriginal -CreateDate -FileModifyDate -DateAcquired -ModifyDate "$image_path" | tr -d '-' | awk '{print $1}')

if [[ "$FIRST_DATE" != '' ]] && [[ "$FIRST_DATE" -gt $start_range ]] && [[ "$FIRST_DATE" -lt $end_range ]]; then
	exiftool -api QuickTimeUTC -d %Y/%B '-directory<dateacquired' '-directory<modifydate' '-directory<filemodifydate' '-directory<createdate' '-directory<datetimeoriginal' '-FileName=%f%-c.%e' "$image_path"

else
	echo "Error"

fi

done

If you feel inspired a want to fill in the blanks, feel free :)

Thanks for your contribution !

AernaLingus [any]@hexbear.net · edit-2 3 months ago

Alright, here’s what I’ve got!

#!/usr/bin/env python3

import datetime
import glob
import os
import re
import shutil

import exiftool


files = glob.glob(r"/path/to/photos/**/*", recursive=True)
# Necessary to avoid duplicate files; if all photos have the same extension
# you could simply add that extension to the end of the glob path instead
files = [f for f in files if os.path.isfile(f)]

parent_dir = r'/path/to/sorted/photos'
start_date = datetime.datetime(2015, 1, 1)
end_date = datetime.datetime(2024, 12, 31)
date_extractor = re.compile(r'^(\d{4}):(\d{2}):(\d{2})')

with exiftool.ExifToolHelper() as et:
  metadata = et.get_metadata(files)
  for d in metadata:
    for tag in ["EXIF:DateTimeOriginal", "EXIF:CreateDate",
                "File:FileModifyDate", "EXIF:ModifyDate",
                "XMP:DateAcquired"]:
      if tag in d.keys():
        # Per file logic goes here
        year, month, day = [int(i) for i in date_extractor.match(d[tag]).group(1, 2, 3)]
        filedate = datetime.datetime(year, month, day)
        if filedate < start_date or filedate > end_date:
          break
        
        # Can uncomment below line for debugging purposes
        # print(f'{d['File:FileName']} {d[tag]} {year}/{month}')
        subdirectory = f'{parent_dir}/{year}/{month}'
        if not os.path.exists(subdirectory):
          os.makedirs(subdirectory)

        shutil.move(d['SourceFile'], subdirectory)
        
        break

Other than PyExifTool which will need to be installed using pip, all libraries used are part of the standard library. The basic flow of the script is to first grab metadata for all files using one exiftool command, then for each file to check for the existence of the desired tags in succession. If a tag is found and it’s within the specified date range, it creates the YYYY/MM subdirectory if necessary, moves the file, and then proceeds to process the next file.

In my preliminary testing, this seemed to work great! The filtering by date worked as expected, and when I ran it on my whole test set (831 files) it took ~6 seconds of wall time. My gut feeling is that once you’ve implemented the main optimization of handling everything with a single execution of exiftool, this script (regardless of programming language) is going to be heavily I/O bound because the logic itself is simple and the bulk of time is spent reading and moving files, meaning your drive’s speed will be the key limiting factor. Out of those 6 seconds, only half a second was actual CPU time. And it’s worth keeping in mind that I’m doing this on a speedy NVME SSD (6 GB/s sequential read/write, ~300K IOPS random read/write), so it’ll be slower on a traditional HDD.

There might be some unnecessary complexity for some people’s taste (e.g. using the datetime type instead of simple comparisons like in your bash script), but for something like this I’d prefer it to be brittle and break if there’s unexpected behavior because I parsed something wrong or put in nonsensical inputs rather than fail silently in a way I might not even notice.

One important caveat is that none of my photos had that XMP:DateAcquired tag, so I can’t be certain that that particular tag will work and I’m not entirely sure that will be the tag name on your photos. You may want to run this tiny script just to check the name and format of the tag to ensure that it’ll work with my script:

#!/usr/bin/env python3

import exiftool
import glob
import os


files = glob.glob(r"/path/to/photos/**/*", recursive=True)
# Necessary to avoid duplicate files; if all photos have the same extension
# you could simply add that extension to the end of the glob path instead
files = [f for f in files if os.path.isfile(f)]
with exiftool.ExifToolHelper() as et:
  metadata = et.get_metadata(files)
  for d in metadata:
    if "XMP:DateAcquired" in d.keys():
      print(f'{d['File:FileName']} {d[tag]}')

If you run this on a subset of your data which contains XMP-tagged files and it correctly spits out a list of files plus the date metadata which begins YYYY:MM:DD, you’re in the clear. If nothing shows up or the date format is different, I’d need to modify the script to account for that. In the former case, if you know of a specific file that does have the tag, it’d be helpful to get the exact tag name you see in the output from this script (I don’t need the whole output, just the name of the DateAcquired key):

#!/usr/bin/env python3

import exiftool
import json


with exiftool.ExifToolHelper() as et:
  metadata = et.get_metadata([r'path/to/dateacquired/file'])
  for d in metadata:
    print(json.dumps(d, indent=4))

If you do end up using this, I’ll be curious to know how it compares to the parallel solution! If the exiftool startup time ends up being negligible on your machine I’d expect it to be similar (since they’re both ultimately I/O bound, and parallel saves time by being able to have some threads executing while others are waiting for I/O), but if the exiftool spin-up time constitutes a significant portion of the execution time you may find it to be faster! If you don’t end up using it, no worries–it was a fun little exercise and I learned about a library that will definitely save me some time in the future if I need to do some EXIF batch processing!

N0x0n@lemmy.ml · edit-2 3 months ago

Hello again :)

Sorry for pinging you ! But I somehow figured it out!

with exiftool.ExifToolHelper(common_args=None) as et:
  metadata = et.get_metadata(files)
  for d in metadata:
    for tag in ["DateTimeOriginal", "CreateDate",
                "FileModifyDate", "ModifyDate",
                "DateAcquired"]:

This seems to work as expect with only the tag names and without the need of the tag groups ([IFD0], [ExifIFD], [QuickTime]). The processing time is impressive !! Thank you !!

However if I may ask only one last thing… Is there anyway to change the file name when the file is moved and already exists to something like

/Pictures/sorted/2018/2/IMG_0993-1.JPG
/Pictures/sorted/2018/2/IMG_0993-2.JPG
/Pictures/sorted/2018/2/IMG_0993-3.JPG
....

I tried to wrap my head arround os.move or os.rename but can’t make any sense out of it !

Thank you in advance !!

AernaLingus [any]@hexbear.net · 3 months ago

Wow, nice find! I was going to handle it by just arbitrarily picking the first tag which ended with CreateDate, FileModifyDate, etc., but this is a much better solution which relies on the native behavior of exiftool. I feel kind of silly for not looking at the documentation more carefully: I couldn’t find anything immediately useful when looking at the documentation for the class used in the script (ExifToolHelper) but with the benefit of hindsight I now see this crucial detail about its parameters:

All other parameters are passed directly to the super-class constructor: exiftool.ExifTool.__init__()

And sure enough, that’s where the common_args parameter is detailed which handles this exact use case:

common_args (list of str*, or* None.) –

Pass in additional parameters for the stay-open instance of exiftool.

Defaults to ["-G", "-n"] as this is the most common use case.

-G (groupName level 1 enabled) separates the output with groupName:tag to disambiguate same-named tags under different groups.

-n (print conversion disabled) improves the speed and consistency of output, and is more machine-parsable

Passed directly into common_args property.

As for the renaming, you could handle this by using os.path.exists as with the directory creation and using a bit of logic (along with the utility functions os.path.basename and os.path.splitext) to generate a unique name before the move operation:

# Ensure uniqueness of path
basename = os.path.basename(d['SourceFile'])
filename, ext = os.path.splitext(basename)
count = 1        
while os.path.exists(f'{subdirectory}/{basename}'):
  basename = f'{filename}-{count}{ext}'
  count += 1

shutil.move(d['SourceFile'], f'{subdirectory}/{basename}')

N0x0n@lemmy.ml · edit-2 3 months ago

Hey ha :) !!

Wow, nice find! I was going to handle it by just arbitrarily picking the first tag which ended with CreateDate, FileModifyDate, etc., but this is a much better solution which relies on the native behavior of exiftool. I feel kind of silly for not looking at the documentation more carefully

Yeah I know that feeling, I posted and add unnecessary noise to Phil Harvey’s forum about something I though was a “bug” or odd behavior with EXIF-tool, while it’s was just my lacking reading skills… I felt so dumb :/. Because I’m unable to build it up form the ground myself,like you did (great work, thanks again !!), I can only fiddle around and do my best reading the documentation to somehow find my way out. I was pretty happy and had a little surge of dopamine level :D !

#Ensure uniqueness of path
basename = os.path.basename(d['SourceFile'])
...

THAT did the trick ! Thank you. I somehow “wrote” something similar but don’t look at it, it’s nonfunctional and ugly XD but I gave it a try while roaming the web.

        try:
          shutil.move(d['SourceFile'], subdirectory)
        except:
          i = 0
          while os.path.exists(d['SourceFile']):
            i += 1
            base_name, extension = os.path.splitext(d['SourceFile'])
            new_filename = f"{base_name}-{i}{extension}"
            print (new_filename)
            os.rename(d['SourceFile'], new_filename)
            shutil.move(new_filename, subdirectory)

Final words

First, your script is a bomb ! Blazing fast and does everything I wanted ! And you were right with your first impression and the -stay_open switch. That’s what PyExifTool uses under the hood (read is somewhere in the docs)! I gave it try to implement that switch with an arg file in my old/ugly/painful bash scirpt, but didn’t worked as expected. I will give it another try sometimes in the near future. Right now I’m exhausted from reading and all the re-runs to troubleshoot and test things and more than happy with your script (thanks again for everything !!!).

Second, I hope you won’t be mad, but after a thorough re-reading of the exif-tool documentation and playing around a bit, I even managed to get exif-tool do the same thing, it looks something like this:

exiftool -P -d %Y:%m:%d -if '$DateTimeOriginal gt "2018:01:01" and $DateTimeOriginal lt "2021:01:01"' -api QuickTimeUTC -r '-directory<${DateTimeOriginal#;DateFmt("%Y/%B")}/' '-FileName=%f%-c.%e' .

In plain English this translates to:

Scan recursively current directory, in a specific time range condition formatted to %Y:%m:%d and based on DateTimeOriginal tag. Order all images that respects the condition in a reformatted Year/month directory structure with the DateTimeOriginal tag. Rename the files incrementally if duplicate exist to filename-x+1.extension.

This was the first command I was working on before starting to try a bash script, but It somehow messed up the folder creation, long story short: It was because of how my command formatted the date in the condition (-d %Y/%B): 2018/June gt "2018:01:01" (yeah this will cause some strange behavior xD). However, your script is faster !!! For the same batch:

2200 files
---
Exif-Tool: 24s
PyExifTool: 11s

Compared to my painful and ugly 11 minutes script… uuuhg !

Again, thank you very much for sharing your knowledge, your help/time and staying with me. 👍 😁 I hope we will meet again and maybe/hopefully have a proper conversation on programming/scripting !

Thanks 🙏.

hydroptic@sopuli.xyz · edit-2 4 months ago

I would assume that exiftool is where most time is spent. Spitballing here and I’m just ever so slightly drunk (shut up I know it’s Thursday) so don’t take me too seriously, but maybe you could parallelize this somehow?

Something like modifying that script so that it takes a list of files as arguments (escaping might become a problem though because fucking bash) and then munges them, and then either in the shell or in another script you use eg GNU parallel to run that script on yourdir/**/*?

edit: you’d want to check if it’s easier to use cli arguments or stdin in a script being run with parallel, been a while since I used it. Could be either one is fine

N0x0n@lemmy.ml · 4 months ago

Hey :)

Thanks for your input, but I have absolutely no idea what you’re talking about XD. Not because you were slightly drunk (cheers BTW a bit late… But I also had a couple yesterday after bagging my head with bash) but I’m really not familiar with bash or programming logic etc…

Just changing how my script loops through the variables, I went from 16min to 11min not bad at all :) There’s probably quite some room to even further reduce the processing time… But hey that’s the best I can do right now and I’m tired running the script over and over again to “troubleshoot” !

hydroptic@sopuli.xyz · edit-2 4 months ago

Ah sorry lol, I happily assumed you’re just a bit unfamiliar with bash and not programming in general – you’re definitely doing a great job if that’s the case, shell scripting isn’t exactly easy in a lot of ways like you discovered with how spaces in file names can bork things sometimes. But yeah, 11min isn’t terrible at all and if you don’t have to run that script very often there’s not much point in spending time optimizing it.

Just for future reference in case you end up scripting again, what I was thinking of was instead of processing every filename in $folder_name/**/* one at a time, you could do it in parallel with some creative rewriting, meaning you’d possibly be able to take advantage of having a multicore CPU – but whether it’ll actually be faster depends on a lot of things so it’d basically just have to be tried and timed.

Something along these lines, totally untested, the script can be further optimized and might not even run as-is, but should give you the idea. Starting off with the script, I modified it to just take a single file name as an argument:

#!/bin/bash
# exif_date_filter.sh

if [ $# -eq 0 ]; then
    echo "Usage: $0 <filename>"
    exit 1
fi

# Concatenate all arguments into one string for the filename, so calling "./script.sh /path/with spaces.jpg" should work without quoting
filename="$*"

start_range=20170101
end_range=20201230

DateTimeOriginal=$(/usr/bin/vendor_perl/exiftool -m -d '%Y%m%d' -T -DateTimeOriginal "$filename")
CreateDate=$(/usr/bin/vendor_perl/exiftool -m -d '%Y%m%d' -T -CreateDate "$filename")

if [[ "$DateTimeOriginal" =~ ^[0-9]+$ ]] || [[ "$CreateDate" =~ ^[0-9]+$ ]]; then
    if  ([ "$DateTimeOriginal" -gt $start_range ] && [ "$DateTimeOriginal" -lt $end_range ]) || 
        ([ "$CreateDate" -gt $start_range ] && [ "$CreateDate" -lt $end_range ]); then
        /usr/bin/vendor_perl/exiftool -api QuickTimeUTC -d %Y/%B '-directory<$DateTimeOriginal/' '-directory<$CreateDate/' '-FileName=%f%-c.%e' "$filename"
        echo "Found a value"
        echo "Okay its $(tput setab 22)DateTimeOriginal$(tput sgr0)"
    else
        echo "FINISH YOUR SYNTAX !!"
    fi
fi

So it’s called as ./exif_date_filter.sh /some/image.jpg

Then, assuming you have GNU parallel installed (should be easy to find on just about every distro), you could do… uh, I think just eg. find /home/user/Pictures -type f | parallel ./exif_date_filter.sh to automagically run as many “copies” of that script at the same time as you have CPU cores

N0x0n@lemmy.ml · edit-2 3 months ago

Hello :)

Thanks again for your input and spoon feeding me ! I took some time to understand but when I got it I felt excited. Because I’m unable to further improve to script itself using my hardware is such a great idea !!! I would never though of that !!

And after a single test drive on a sample, it was BLAZING FAST (3min on a 2200 sample thanks CPU cores :D)! Thank you very much for sharing your knowledge !! You brought my hope back.

Thanks 😘 👍 🙏

Edit:

I rewrote the script and combined your solution with the script

#!/bin/bash

if [ $# -eq 0 ]; then
    echo "Usage: $0 <filename>"
    exit 1
fi

# Concatenate all arguments into one string for the filename, so calling "./script.sh /path/with spaces.jpg" should work without quoting
filename="$*"

start_range=20170101
end_range=20201230

FIRST_DATE=$(/usr/bin/vendor_perl/exiftool -m -d '%Y%m%d' -T -DateTimeOriginal -CreateDate -FileModifyDate -DateAcquired -ModifyDate "$filename" | tr -d '-' | awk '{print $1}')
#echo $FIRST_DATE

if [[ "$FIRST_DATE" != '' ]] && [[ "$FIRST_DATE" -gt $start_range ]] && [[ "$FIRST_DATE" -lt $end_range ]]; then
        /usr/bin/vendor_perl/exiftool -api QuickTimeUTC -d %Y/%B '-directory<$DateAcquired/' '-directory<$ModifyDate/' '-directory<$FileModifyDate/' '-directory<$CreateDate/' '-directory<$DateTimeOriginal/' '-FileName=%f%-c.%e' "$filename"

else
        echo "Error"

fi

running time find /home/user/Pictures/ -type f | parallel ./exif-test.bash took 1min21s for 2200 files without any errors and everything is perfectly organized !!! Thanks a lot !!!

hydroptic@sopuli.xyz · 3 months ago

Hey great that it worked out. Glad I could help

GenderNeutralBro@lemmy.sdf.org · 4 months ago

I have not tested this, but I have a couple ideas off the top of my head.

#1 - Retrieve all fields with a single exiftool command. e.g. ALL_DATES=$(exiftool -m -d '%Y%m%d' -T -DateTimeOriginal -CreateDate -FileModifyDate -DateAcquired -ModifyDate "$filename")

Then retrieve individual fields from $ALL_DATES with something like awk. e.g. echo $ALL_DATES | awk '{print $1}' will return the first field (DateTimeOriginal), and changing that to ‘{print $2}’ will return the second field (CreateDate).

#2 - Perhaps process multiple files with a single exiftool call. e.g. exiftool -m -d '%Y%m%d' -T -DateTimeOriginal -CreateDate -FileModifyDate -DateAcquired -ModifyDate ~/Pictures/*. You might compare whether running just this exiftool query once vs running it in a loop takes a significantly different amount of time. If not, it’s probably simpler to use one call per file.

N0x0n@lemmy.ml · edit-2 4 months ago

Hello :) !

Thanks for the nice write up <3 ! Not only did it improve the processing speed but it also made my script way more readable <3 !

#1 - Retrieve all fields with a single exiftool command. e.g. ALL_DATES=$(exiftool -m -d '%Y%m%d' -T -DateTimeOriginal -CreateDate -FileModifyDate -DateAcquired -ModifyDate "$filename")

I went from 16min for the same sample to:

real	4m47,630s
user	4m16,868s
sys	0m32,487s

That’s like a BIG improvement !!! Thank you very much !!!

I don’t know if I did it how you had it in mind but here is a snipped of how I changed the script

#!/bin/bash

shopt -s globstar

folder_name=/home/user/Pictures
start_range=20170101
end_range=20181230

for filename in $folder_name/**/*; do
        ALL_DATES=$(/usr/bin/vendor_perl/exiftool -m -d '%Y%m%d' -T -DateTimeOriginal -CreateDate -FileModifyDate -DateAcquired -ModifyDate "$filename")
#       echo $ALL_DATES | awk '{print $1}'

        if [[ $(echo $ALL_DATES | awk '{print $1}') =~ ^[0-9]+$ ]]; then
                if  [[ $(echo $ALL_DATES | awk '{print $1}') -gt $start_range ]] && [[ $(echo $ALL_DATES | awk '{print $1}') -lt $end_range ]]; then
                        /usr/bin/vendor_perl/exiftool -api QuickTimeUTC -r -d %Y/%B '-directory<$DateTimeOriginal/' '-FileName=%f%-c.%e' "$filename"

                echo "Okay its $(tput setab 22)DateTimeOriginal$(tput sgr0)"
                fi

...

While every file gets processed successfully I get A LOT of error from EXIF tool, it seems like something is still looping through the files while they have already been moved?? I will try to investigate why it throws this error.

Error: File not found - /home/user/Pictures/2018/September/IMG_2078.JPG
Error: File not found - /home/user/Pictures/2018/September/IMG_2079.JPG
Error: File not found - /home/user/Pictures/2018/September/IMG_2080.JPG
Error: File not found - /home/user/Pictures/2018/September/IMG_2081.JPG
Error: File not found - /home/user/Pictures/2018/September/IMG_2082.JPG
Error: File not found - /home/user/Pictures/2018/September/IMG_2083.JPG
Error: File not found - /home/user/Pictures/2018/September/IMG_2084.JPG
Error: File not found - /home/user/Pictures/2018/September/IMG_2085.JPG
Error: File not found - /home/user/Pictures/2018/September/IMG_2086.JPG
Error: File not found - /home/user/Pictures/2018/September/IMG_2087.JPG

#2 - Perhaps process multiple files with a single exiftool call. e.g. exiftool -m -d '%Y%m%d' -T -DateTimeOriginal -CreateDate -FileModifyDate -DateAcquired -ModifyDate ~/Pictures/*. You might compare whether running just this exiftool query once vs running it in a loop takes a significantly different amount of time. If not, it’s probably simpler to use one call per file.

Hummm, I can’t think of where and how I could fit it in the script :/ Will probably have to do a lot of try&break.

Thank you very much for your help and insightful edit with find :).

N0x0n@lemmy.ml · edit-2 4 months ago

Hey ha :) Sorry to pin you, I just wanted to give you a little update !

I was maybe a bit to hasty to cheer up. With the last updated script, It actually behaved strangely, some if statements were skipped, looped strangely through my folder and it wrongly ordered many files with the wrong Field. Also it raised many strange errors I couldn’t make any sense of while looking at the script…

HOWEVER !!

I changed how my script loops through my files according to your find suggestions:

#!/bin/bash

start_range=20160101
end_range=20221212
folder_name=/home/user/Pictures

find ~/Pictures -type f -print0 | while IFS= read -r -d '' file_path; do
        image_path=$file_path

ALL_DATES=$(/usr/bin/vendor_perl/exiftool -m -d '%Y%m%d' -T -DateTimeOriginal -CreateDate -FileModifyDate -DateAcquired -ModifyDate "$file_path")

for images in $file_path; do

        if [[ $(echo $ALL_DATES | awk '{print $1}') =~ ^[0-9]+$ ]] && [[ $(echo $ALL_DATES | awk '{print $1}') -gt $start_range ]] && [[ $(echo $ALL_DATES | awk '{print $1}') -lt $end_range ]]; then
                /usr/bin/vendor_perl/exiftool -api QuickTimeUTC -d %Y/%B '-directory<$DateTimeOriginal/' '-FileName=%f%-c.%e' "$images"
                echo "Okay its $(tput setab 22)DateTimeOriginal$(tput sgr0)"

While it doesn’t give me the “BIG improvement” I dreamed yesterday, it does give a non-neglieable improvement ! Also there are no errors anymore and the script loops perfectly through all files… So it’s a win :D.

real	11m5,139s
user	9m47,445s
sys	1m16,224s

To further improve the script, I probably need to find a way to make only a single exiftool call, as you suggested, but that’s probably out of my league… Also I already lost enough amount of time to somehow put this together without any coding skills and just wrongly make use of a tool I have no clue about !!!

Thank you very much for your help :))))

GenderNeutralBro@lemmy.sdf.org · edit-2 4 months ago

Glad it’s working! Couple more quick ideas:

Since you’re looping through the results of find, $file_path will be a single path name, so you don’t need to loop over it with for images in $file_path; anymore.

I think you’re checking each field of the results in its own if statement, e.g. if [[ $(echo $ALL_DATES | awk '{print $1}')... then if [[ $(echo $ALL_DATES | awk '{print $2}')... etc. While I don’t think this is hurting performance significantly, it would make your code easier to read and maintain if you first found the correct date, and then did only one comparison operation on it.

For example, exiftool -m -d '%Y%m%d' -T -DateTimeOriginal -CreateDate -FileModifyDate -DateAcquired -ModifyDate "$file_path" returns five columns, which contain either a date or “-”, and it looks like you’re using the first column that contains a valid date. You can try something like this to grab the first date more easily, then just use that from then on:

FIRST_DATE=$(exiftool -m -d '%Y%m%d' -T -DateTimeOriginal -CreateDate -FileModifyDate -DateAcquired -ModifyDate "$file_path" | tr -d '-' | awk '{print $1}')

tr -d '-' will delete all occurrences of ‘-’. That means the result will only contain whitespace and valid dates, so awk '{print $1}' will print the first valid date. Then you can simply have one if statement:

if [[ "$FIRST_DATE" != '' ]] && [[ "$FIRST_DATE" -gt $start_range ]] && [[ "$FIRST_DATE" -lt $end_range ]]; then

Hope this helps!

N0x0n@lemmy.ml · edit-2 4 months ago

Update

I found something interesting ! It seems that the tag FileModifyDate is not being processed in the script! After removing all time tags except FileModifyDate the file is not even processed, it directly goes to the else statement ! I’m still digging :D

Hey again !

Thank you very much for sharing your knowledge and ELI !

I gave it a try and while I understand what it does (thanks to your concise and easy to understand examples) and how it should be implemented exiftool seems to behave very strangely (maybe a bug, but I guess skill issue).

FIRST_DATE=$(exiftool -m -d '%Y%m%d' -T -DateTimeOriginal -CreateDate -FileModifyDate -DateAcquired -ModifyDate "$file_path" | tr -d '-' | awk '{print $1}')

if [[ "$FIRST_DATE" != '' ]] && [[ "$FIRST_DATE" -gt $start_range ]] && [[ "$FIRST_DATE" -lt $end_range ]]; then

	/usr/bin/vendor_perl/exiftool -api QuickTimeUTC -d %Y/%B '-directory<$DateTimeOriginal/' '-directory<$CreateDate/' '-directory<$FileModifyDate/' '-directory<$DateAcquired/' '-directory<$ModifyDate/' '-FileName=%f%-c.%e' "$file_path"

As per the exiftool documentation (source example 12)

-directory<$DateTimeOriginal/' '-directory<$CreateDate/' '-directory<$FileModifyDate/' '-directory<$DateAcquired/' '-directory<$ModifyDate/'

In this command the Directory tag is set from multiple other date/time tags. ExifTool evaluates the command-line arguments left to right, and latter assignments to the same tag override earlier ones, so the Directory for each image is ultimately set by the rightmost copy argument that is valid for that image.

However, it sometimes skip the DateTimeOriginal field and takes FileModifyDate instead even if the first one is present. My guess is that exiftool needs more time to correctly process the file, but it’s only a guess ! Because with the for loop and all elif calls it works without any issues.

Thanks again for your insightful help :)

Side note: I gave a test run with only one time field to see If there is any time gain with calling only the first valid date, while it seems ~2ms slower per file I think it would really make the difference without all the elif calls on the long run !!

Thanks again !!

GenderNeutralBro@lemmy.sdf.org · edit-2 4 months ago

Off the top of my head I’m not sure why that would be. To troubleshoot, it might help to print the output every step of the way so you can see if there are any oddities. Something like this perhaps, in place of the FIRST_DATE= line.

EXIF_OUT=$(exiftool -m -d '%Y%m%d' -T -DateTimeOriginal -CreateDate -FileModifyDate -DateAcquired -ModifyDate "$file_path")
echo "$EXIF_OUT"
EXIF_FILTERED=$(echo "$EXIF_OUT" | tr -d '')
echo "$EXIF_FILTERED"
FIRST_DATE=$(echo "$EXIF_FILTERED" | awk '{print $1}')
echo "$FIRST_DATE"```

N0x0n@lemmy.ml · edit-2 3 months ago

Heyho :)

Back again ! Thanks you for helping troubleshoot !!! It’s was actually something else… (my reading skills) !

ExifTool evaluates the command-line arguments left to right, and latter assignments to the same tag override earlier ones.

That’s why everything was messed up, because it took the last assignment to write the directory date… I feel quite stupid/bad and added unnecessarily noise to Phil Harvey’s forum :/.

Thanks to you and another user I greatly improved my script and went from 16min to 1min21s for the exact same batch ! He/she proposed to use the parallel package alongside the script to make full use of my CPU cores ! Also, another way to loop through my files. It’s a mix of both your ideas and it looks so much better ! Thank you very much for your help ! 😘

#!/bin/bash

if [ $# -eq 0 ]; then
    echo "Usage: $0 <filename>"
    exit 1
fi

# Concatenate all arguments into one string for the filename, so calling "./script.sh /path/with spaces.jpg" should work without quoting
filename="$*"

start_range=20170101
end_range=20201230

FIRST_DATE=$(/usr/bin/vendor_perl/exiftool -m -d '%Y%m%d' -T -DateTimeOriginal -CreateDate -FileModifyDate -DateAcquired -ModifyDate "$filename" | tr -d '-' | awk '{print $1}')
#echo $FIRST_DATE

if [[ "$FIRST_DATE" != '' ]] && [[ "$FIRST_DATE" -gt $start_range ]] && [[ "$FIRST_DATE" -lt $end_range ]]; then
        /usr/bin/vendor_perl/exiftool -api QuickTimeUTC -d %Y/%B '-directory<$DateAcquired/' '-directory<$ModifyDate/' '-directory<$FileModifyDate/' '-directory<$CreateDate/' '-directory<$DateTimeOriginal/' '-FileName=%f%-c.%e' "$filename"

else
        echo "Error"

fi

time find /home/user/Pictures/ -type f | parallel ./exif-test.bash

hydroptic@sopuli.xyz · edit-2 4 months ago

Ah yeah this suggestion is much better than mine. Reducing the number of calls to exiftool is probably the easiest way to speed things up, and giving it a glob instead of individual files is definitely worth trying because it can be quite a bit faster sometimes depending on what’s being run

Shadow@lemmy.ca · edit-2 4 months ago

This is probably the answer.

Op, it’s generally not native bash that’s slow, it’s the overhead of calling a fresh copy of exiftool each time. Especially if it’s written in Perl, where it needs to be freshly interpreted / compiled each time.

[Solved/share] Exiftool bash script to process image in a specific time range recursively.

[Solved/share] Exiftool bash script to process image in a specific time range recursively.

Edit

How to use

What Am I trying to achieve ?

The script in question

Things I have tried

How to Improve the processing time ?

Edit:

Final words

Update