Python: Creating a timed image slideshow with PIL and OpenCV v2

Problem

Apparently, one of the hardest video-editing task to do with a script is to create a dynamically-timed slideshow without any fancy-drag and drop GUIs.


With Adobe After Effects, you cannot dynamically load external images using an expression (they will need to be loaded into your project beforehand, and even then, you cannot load the image into a comp with an expression).

And adding hundreds of layers of images and having to go through each and every one of them to edit the expression is a fairly tedious task.

Worse of all, every change that you make - such as adding a new image to the slideshow  - will compound towards the chore of doing things manually.

Solution

Building upon my last two posts: Python: Converting from PIL to OpenCV 2 Image Formats and Python: PIL to mp4, we've reached the end.

With "Python: PIL to mp4", a simple blending transition was created using PIL and OpenCV, But the objective of this post is to introduce timings to delay the animation for numerous/multiple images.

We can extend this idea of having a primitive transition to allow for an image to be delayed from transitioning until a certain amount of time has elapsed, and to allow the transition to occur after "x" amount of seconds, hence forming a slideshow.

Process

Initialization

So to start off with, we're going to need some data to work with.
Since it's Python, you can do whatever you want to feed data in - you could use a JSON file, CSV, Pickle, whatever you're comfortable with, or perhaps, whatever arbitrary file format that you're locked into using.

But here, a basic python array will be used to indicate the timings and image file that will be fed into the slideshow, amongst other data...

songData = [
    [390, u'Fractal', u'Itvara', 'minimix', u'image1.jpg'],
    [322, u'Case & Point', u'Error Code', 'minimix', u'image2.jpg'],
    [261, u'Excision & Pegboard Nerds', u'Bring the Madness (Noisestorm Remix) [feat. Mayor Apeshit]', 'minimix', u'image3.jpg'],
    [157, u'Nitro Fun', u'Final Boss', 'minimix', u'image4.jpg'],
    [88, u'Astronaut', u'Quantum (Virtual Riot Remix)', 'minimix', u'image5.jpg'],
    [0, u'Fractal', u'Contact', 'minimix', u'image6.jpg']]

As you can see in the data above, the most relevant data is songData[][0] and songData[][4], indicating the timings (in seconds) and the image file locations, respectively.

We're going to set the FPS of the slideshow... 60FPS is the standard nowadays, so we're going to set that and process the songData above to reflect this...

FPS = 60 # Sets the FPS of the entire video
currentFrame = 0 # The animation hasn't moved yet, so we're going to leave it as zero
startFrame = 0 # The animation of the "next" image starts at "startFrame", at most
trailingSeconds = 5 # Sets the amount of time we give our last image (in seconds)
blendingDuration = 3.0 # Sets the amount of time that each transition should last for
                       # This could be more dynamic, but for now, a constant transition period is chosen
blendingStart = 10 # Sets the time in which the image starts blending before songFile

for i in songData:
    i[0] = i[0] * FPS # Makes it so that iterating frame-by-frame will result in properly timed slideshows

Now the first image is going to be loaded in by the script - as so:

im1 = Image.open(songData[-1][4]) # Load the image in
im2 = im1 # Define a second image to force a global variable to be created

current = songData[-1][4] # We're going to let the script know the location of the current image's location
previous = current # And this is to force/declare a global variable

And next up is to create the actual OpenCV video handling capability. You can have a read-up about this here: Python: PIL to mp4

height, width, layers = np.array(im1).shape # Get some stats on the image file to create the video with
video = cv2.VideoWriter("slideshow.avi",-1,60,(width,height),True)

So that was the basic initialization routine. If you don't get how it works together yet, don't worry. Just read on - as the full code with everything combined is below.

Main loop

So the strategy behind generating this slideshow is to loop through each and every frame and continuously feed that into our output video file. Sure some corners can be cut - by which you only generate the transitions (leaving the gaps to be manually filled by an external program) - but this post is looking more into automating the entire slideshow generation process with only Python, PIL and OpenCV.

We're going to have a main while loop that sets the limit on how long our slideshow should last.

while currentFrame < songData[0][0] + FPS * 60 * trailingSeconds: # RHS defines the limit of the slideshow

And this is where the nitty gritty kicks in: the actual code that makes the transition between each image within the slideshow...

    for i in songData: # Loop through each image timing
        if currentFrame >= i[0] - (blendingStart * FPS): # If the image timing happens to be for the
                                                         # current image, the continue on...
                                                         # (Notice how songData is reversed)
                                                         
            # The print statement adds some verbosity to the program
            print str(currentFrame) + " - " + str(i[0] - (blendingStart * FPS)) + " - " + i[2]
            if not current == i[4]: # Check if the image file has changed
                previous = current # We'd want the transition to start if the file has changed
                current = i[4]
                startFrame = i[0] - (blendingStart * FPS)

                # The two images in question for the blending is loaded in
                im1 = Image.open(previous)
                im2 = Image.open(current)
            break

    # See: http://blog.extramaster.net/2015/07/python-pil-to-mp4.html for the part below
    diff = Image.blend(im1, im2, min(1.0, (currentFrame - startFrame) / float(FPS) / blendingDuration))
    video.write(cv2.cvtColor(np.array(diff), cv2.COLOR_RGB2BGR))
    
    currentFrame += 1 # Next frame

The ending to this program is pretty self-explanatory...

# At this point, we'll assume that the slideshow has completed generating, and we want to close everything off to prevent a corrupted output.
video.release()

All together now!

So here's all the code required to create a timed image slideshow with PIL and OpenCV v2!

Code:

from PIL import Image
import cv2
import numpy as np

songData = [
    [390, u'Fractal', u'Itvara', 'minimix', u'image1.jpg'],
    [322, u'Case & Point', u'Error Code', 'minimix', u'image2.jpg'],
    [261, u'Excision & Pegboard Nerds', u'Bring the Madness (Noisestorm Remix) [feat. Mayor Apeshit]', 'minimix', u'image3.jpg'],
    [157, u'Nitro Fun', u'Final Boss', 'minimix', u'image4.jpg'],
    [88, u'Astronaut', u'Quantum (Virtual Riot Remix)', 'minimix', u'image5.jpg'],
    [0, u'Fractal', u'Contact', 'minimix', u'image6.jpg']]

FPS = 60 # Sets the FPS of the entire video
currentFrame = 0 # The animation hasn't moved yet, so we're going to leave it as zero
startFrame = 0 # The animation of the "next" image starts at "startFrame", at most
trailingSeconds = 5 # Sets the amount of time we give our last image (in seconds)
blendingDuration = 3.0 # Sets the amount of time that each transition should last for
                       # This could be more dynamic, but for now, a constant transition period is chosen
blendingStart = 10 # Sets the time in which the image starts blending before songFile

for i in songData:
    i[0] = i[0] * FPS # Makes it so that iterating frame-by-frame will result in properly timed slideshows

im1 = Image.open(songData[-1][4]) # Load the image in
im2 = im1 # Define a second image to force a global variable to be created

current = songData[-1][4] # We're going to let the script know the location of the current image's location
previous = current # And this is to force/declare a global variable

height, width, layers = np.array(im1).shape # Get some stats on the image file to create the video with
video = cv2.VideoWriter("slideshow.avi",-1,60,(width,height),True)

while currentFrame < songData[0][0] + FPS * 60 * trailingSeconds: # RHS defines the limit of the slideshow
    for i in songData: # Loop through each image timing
        if currentFrame >= i[0] - (blendingStart * FPS): # If the image timing happens to be for the
                                                         # current image, the continue on...
                                                         # (Notice how songData is reversed)
                                                         
            # The print statement adds some verbosity to the program
            print str(currentFrame) + " - " + str(i[0] - (blendingStart * FPS)) + " - " + i[2]
            if not current == i[4]: # Check if the image file has changed
                previous = current # We'd want the transition to start if the file has changed
                current = i[4]
                startFrame = i[0] - (blendingStart * FPS)

                # The two images in question for the blending is loaded in
                im1 = Image.open(previous)
                im2 = Image.open(current)
            break

    # See: http://blog.extramaster.net/2015/07/python-pil-to-mp4.html for the part below
    diff = Image.blend(im1, im2, min(1.0, (currentFrame - startFrame) / float(FPS) / blendingDuration))
    video.write(cv2.cvtColor(np.array(diff), cv2.COLOR_RGB2BGR))
    
    currentFrame += 1 # Next frame

# At this point, we'll assume that the slideshow has completed generating, and we want to close everything off to prevent a corrupted output.
video.release()

Sample output

So with all the code above, it begs the question, why do I need to create a slideshow using scripts?
Well, here's a little sample of what you can do with a simple little slideshow.
Note the timings from "songData",

songData = [
    [390, u'Fractal', u'Itvara', 6:30, u'image1.jpg'],
    [322, u'Case & Point', u'Error Code', 6:22, u'image2.jpg'],
    [261, u'Excision & Pegboard Nerds', u'Bring the Madness (Noisestorm Remix) [feat. Mayor Apeshit]', 4:21, u'image3.jpg'],
    [157, u'Nitro Fun', u'Final Boss', 2:37, u'image4.jpg'],
    [88, u'Astronaut', u'Quantum (Virtual Riot Remix)', 1:28, u'image5.jpg'],
    [0, u'Fractal', u'Contact', 0, u'image6.jpg']]

With this slideshow, you can really enhance the effect of audio-react "music" YouTube videos, especially Youtube Music Mixes, like this:
Direct Link: https://www.youtube.com/watch?v=XI25k5Z-t88
Direct Link: https://www.youtube.com/watch?v=XI25k5Z-t88

Python: PIL to mp4

Why?

If you're having trouble piping image data frame-by-frame into FFMpeg (with the subprocess module), you may be interested in another way to convert python image data into a movie without having to store each individual frame as a file (whether it be .png, .jpg, .gif sequences).

However, this method is a little more complicated then attempting to get FFMpeg to work with python, so here's a little scenario to help with whether or not this tutorial is for you.

Let's say you're in a scenario, where you've encountered one of these errors:

"AttributeError: 'Popen' object has no attribute 'proc'"

"IOError: [Errno 22] Invalid argument"

and even

"IOError: [Errno 32] Broken pipe"

Either resulting from this tutorial: Read and write video frames in Python using FFMPEG - __del__( self ) or from some code elsewhere, and you are willing to use an alternative to ffmpeg to convert your PIL images to a video format like in this case: python - Piped FFMPEG won't write frames correctly - Stack Overflow, then read on.

If you're not willing to switch from FFMpeg to another alternative then press backspace and keep on searching!

How?

To do this, we're going to employ the assistance of OpenCV v2 and NumPy, which has been covered here: Python: Converting from PIL to OpenCV 2 Image Formats

Please note that the OpenCV version used is version 2, which uses

import cv2

as the import statement, as opposed to something like

import cv

There may be an update to OpenCV that breaks the code like with the answer found in the following link. image - Python JPEG to movie - Stack Overflow.
But without further ado, let's jump right into it!

Imports

We're going to be using PIL for loading and manipulating the images, NumPy for a PIL-to-OpenCV bridge, and OpenCV Version 2 for the actual image-to-movie process.
So our imports will look something like

from PIL import Image
import numpy, cv2

Manipulation

Obviously, the purpose of using python to convert from PIL to a movie is to be able to manipulate the image frame-by-frame using the power of PIL.
So here, I'm going to demonstrate some basic image blending functionality, just to provide a basis for this tutorial.

Here, we have two images, demo3_1.jpg and demo3_2.jpg...

demo3_1.jpg

demo3_2.jpg

With PIL, you can do something like

# Imports can be found in the "Imports" section above

# Load up the first and second demo images
image1 = Image.open("demo3_1.jpg")
image2 = Image.open("demo3_2.jpg")

# Create a new image which is the half-way blend of image1 and image2
# The "0.5" parameter denotes the half-way point of the blend function.
images1And2 = Image.blend(image1, image2, 0.5)

# Save the resulting blend as a file
images1And2.save("demo3_3.jpg")

In order to use PIL to blend two images together, in this case, the two images are blended at the halfway point, which means that half of each image is merged to become the resultant blended image.

demo3_3.jpg

This is a primitive version of an "additive blend", so you can think of it as an "additive frame blending using PIL" - note also that ImageChops isn't used for simplicity, but that's some additional power that you can use to manipulate images with.

Writing a video with OpenCV

With OpenCV, there's a "VideoWriter" method which you can access to create a movie with.
The method goes something like this:

video = cv2.VideoWriter(filename, codec selection, frames per second, (width, height))

Writing to the VideoWriter can be done with "video.write( numpy array as string )", and the VideoWriter can be "closed" by using "video.release()".

And that's all you need to know to convert from PIL to mp4 (or at least a movie, you need a certain codec for conversion to mp4).

All together now

With all the elements from the sections above in mind, here's the code in action

# Imports can be found in the "Imports" section above

# Load up the first and second demo images, assumed is that image1 and image2 both share the same height and width
image1 = Image.open("demo3_1.jpg")
image2 = Image.open("demo3_2.jpg")

# Grab the stats from image1 to use for the resultant video
height, width, layers =  numpy.array(image1).shape

# Create the OpenCV VideoWriter
video = cv2.VideoWriter("demo3_4.avi", # Filename
                        -1, # Negative 1 denotes manual codec selection. You can make this automatic by defining the "fourcc codec" with "cv2.VideoWriter_fourcc"
                        10, # 10 frames per second is chosen as a demo, 30FPS and 60FPS is more typical for a YouTube video
                        (width,height) # The width and height come from the stats of image1
                        )

# We'll have 30 frames be the animated transition from image1 to image2. At 10FPS, this is a whole 3 seconds
for i in xrange(0,30):
    images1And2 = Image.blend(image1, image2, i/30.0)

    # Conversion from PIL to OpenCV from: http://blog.extramaster.net/2015/07/python-converting-from-pil-to-opencv-2.html
    video.write(cv2.cvtColor(numpy.array(images1And2), cv2.COLOR_RGB2BGR))

# And back from image2 to image1...
for i in xrange(0,30):
    images2and1 = Image.blend(image2, image1, i/30.0)
    video.write(cv2.cvtColor(numpy.array(images2and1), cv2.COLOR_RGB2BGR))

# Release the video for it to be committed to a file
video.release()

Note that when you run the code above, you'll get a prompt for codec selection...
It is possible to find a codec for direct .mp4 conversions, however here, the default "Intel IYUV" codec was chosen and used.

From this stage as well, you can use FFMpeg (outside of Python) or your favourite video conversion program to convert from the codec that you selected to the format that you want, which can indeed include .mp4 files... ImageMagick was used to convert the output to the gif below:

And that's it!




Note that this post was made specifically for OpenCV v2, as documentation online were frustratingly for OpenCV v1, which has different methods and such.

As a disclaimer, things might change in a newer version of OpenCV, so this information is correct as of July 2015.



Covered Topics:
PIL to mp4 conversion
PIL to movie conversion
PIL/Python/OpenCV Video file from Images
PIL/Python/OpenCV JPEG to movie
Creating an OpenCV movie from PIL images
Converting an OpenCV movie into PIL Images

Python: Converting from PIL to OpenCV 2 Image Formats

Code: opencvImage = numpy.array(PILImage)

Suppose you have an image that has been manipulated with the Python Imaging Library, and you want to convert that image into a format that can be understood by the OpenCV Version 2 Library.

To do that, as of OpenCV v2, you can use the NumPy array as an intermediary format between the two libraries, where NumPy can convert PIL data into the NumPy array format, and OpenCV v2 can recognize the NumPy array natively.

To demonstrate this conversion, here's some code.

# First you need to import the libraries in question.
import numpy
import cv2
from PIL import Image

# And then you need a PIL image to work with, for now, an image from a local file is going to be used.
PILImage = Image.open("demo1.jpg")

demo1.jpg
# The conversion from PIL to OpenCV is done with the handy NumPy method "numpy.array" which converts the PIL image into a NumPy array.
opencvImage = numpy.array(PILImage)

# Display the OpenCV image using inbuilt methods.
cv2.imshow('Demo Image',opencvImage)
cv2.waitKey(0)
cv2.destroyAllWindows()

# Which results in:

However, as you can see in the demonstration, the output OpenCV image turned a little weird, with the colour not matching the original PIL image (in the sense that the OpenCV image having the wrong colours). You can try this out for yourself...

This is because we're dealing with a multi-channel/"RGB" format and not a single channel image file, and a conversion from PIL to OpenCV involves a little bit of additional translation, this can be solved with OpenCV's "cvtColor" method.

Code: opencvImage = cv2.cvtColor(numpy.array(PILImage), cv2.COLOR_RGB2BGR)

To demonstrate this additional translation, here's some code.

# First you need to import the libraries in question.
import numpy
import cv2
from PIL import Image

# And then you need a PIL image to work with, for now, an image from a local file is going to be used.
PILImage = Image.open("demo2.jpg")

demo2.jpg
# The conversion from PIL to OpenCV is done with the handy NumPy method "numpy.array" which converts the PIL image into a NumPy array.
# cv2.cvtColor does the trick for correcting the colour when converting between PIL and OpenCV Image formats via NumPy.
opencvImage = cv2.cvtColor(numpy.array(PILImage), cv2.COLOR_RGB2BGR)

# Display the OpenCV image using inbuilt methods.
cv2.imshow('Demo 2 Image',opencvImage)
cv2.waitKey(0)
cv2.destroyAllWindows()

# Which results in:

And that's it!
Note that this post was made specifically for OpenCV v2, as documentation online were frustratingly for OpenCV v1, which has different methods and such.

As a disclaimer, things might change in a newer version of OpenCV, so this information is correct as of July 2015.



Covered Topics:
PIL to NumPy conversion
PIL to cv2 conversion
PIL to OpenCV conversion
Adaptors.PIL2Ipl alternative/replacement/not working
Creating an OpenCV image from a PIL image
Converting an OpenCV image into a PIL Image