Image pipeline with .png retains state between runs with thumbnail!/2

[bbangert]

I have some code that stores a portion of an image pipeline, and then tacks on commands and re-runs it multiple times. My understanding was that this should be fine as its immutable, the pipeline is just a set of instructions and has no state, etc. This doesn't seem to be the case for .png files using the thumbnail operation.

Here's a simple reproduction using a .png (I will attach it here as well):

Using other image operators instead of thumbnail returns the same byte stream consistently. This also doesn't seem to happen when I send a .svg through.

Here's the image I'm using:

[bbangert]

Seems to be related to the optimization around sending the path directly to thumbnail, using Image.open!/1 first alleviates the issue:

[kipcole9]

Definitely unexpected. If I reduce your example to vix/libvips primitives I can see:

iex> pipeline = Vix.Vips.Operation.thumbnail! "/Users/kip/Desktop/400315209-9fbca590-639d-4fcc-8c8a-b89b9320fe99.png", 90
%Vix.Vips.Image{ref: #Reference<0.3787134555.533594139.29044>}
iex> i1 = Vix.Vips.Image.write_to_binary(pipeline)
{:ok,
 <<24, 188, 242, 255, 24, 188, 242, 255, 24, 188, 242, 255, 24, 188, 242, 255,
   24, 188, 242, 255, 24, 188, 242, 255, 24, 188, 242, 255, 24, 188, 242, 255,
   24, 188, 242, 255, 24, 188, 242, 255, 24, 188, 242, 255, 24, 188, 242, 255,
   ...>>}
iex> i2 = Vix.Vips.Image.write_to_binary(pipeline)
{:error, "Failed to write extracted region to memory"}

Which is basically suggesting a strange interplay between Vix.Vips.Operation.thumbnail!/2 and Vix.Vips.Image.write_to_binary/2. @akash-akya, any thoughts on this? I'll open an issue on vix if you think that's appropriate?

[kipcole9]

Seems to be related to the optimization around sending the path directly to thumbnail, using Image.open!/1 first alleviates the issue:

Image.thumbnail/2 does indeed do a lot of optimisations. From the libvips documentation:

Make a thumbnail from a file. Shrinking is done in three stages: using any shrink-on-load features available in the file import library, using a block shrink, and using a lanczos3 shrink. At least the final 200% is done with lanczos3. The output should be high quality, and the operation should be quick

And indeed the most common advice on the libvips discussions is to use thumbnail using a path argument and to avoid wherever possible doing thumbnail_buffer.

The equivalent advice in Image is to always use Image.thumbnail("path/to/image", 90) and avoid Image.open!(image) |> Image.thumbnail(90). Of course that's not always possible, with a common example being an image uploaded from a browser.

[akash-akya]

@kipcole9 please open an issue, I'll check.
It might be something to do with access: :sequential? not sure, I'll check later today.

[kipcole9]

Given John's response and comments from @bbangert and @akash-akya I think a reasonable approach is:

1. Implement an :access option to Image.thumbnail/3 for cases where the argument is a path. The values can be :sequential or :random and the default is :random as it is for Image.open/2.
2. The implementation of access: :random would be the equivalent of his example thumb.copy_memory. @akash-akya, is there a vix equivalent of this?
3. Given recent comments that some file loaders allow random access and some don't, testing will be required to only implement the memory copy for formats that need it. Since this is a bit of a heuristic at this time, the approach needs to be conservative to minimise developer surprise.

I'll do some testing of different image types over the next day or so and implement a proof-of-concept as well.

[kipcole9]

Perhaps the equivalent of thumb.copy_memory is?

image
|> Vix.Vips.Image.write_to_binary(...)
|> Vix.Vips.Image.new_from_binary(...)

[kipcole9]

Although exposing vips_image_copy_memory would probably be a better idea @akash-akya ?

[akash-akya]

Hi @kipcole9, sounds good. I am thinking of simpler solution for now in Vix, that is update the documentation and communicate the case to the user, until I have a better solution.

Image.copy_memory is already present, it is just is not made public. I'll update the documentation and make it public

[kipcole9]

Thanks Akash. I don't think we want to apply VImage.copy_memory/1 is all cases - that would seem against the principles of libvips. And thumnailing can also be use to increase image size so there is a risk of unintentionally and unnecessarily exhausting memory.

I'll work on my proof of concept over the next couple of days and report back.

@bbangert for now you could do:

iex> i = Image.thumbnail!("/path/to/image.png", 90)
%Vix.Vips.Image{ref: #Reference<0.2025189106.1617559581.43837>}
iex> {:ok, i2} = Vix.Vips.Image.copy_memory(i)
{:ok, %Vix.Vips.Image{ref: #Reference<0.2025189106.1617559581.43843>}}

And then run repeatable pipelines on i2.

[kipcole9]

I've tried a number of experiments on this and my conclusion (as late as it is it), is that Vix.Vips.Image.copy_memory(i) is the best avaialable solution (I'll add it as Image.copy_memory/1 that delegates to vix) - along with some improved documentation.

[bbangert]

Thanks @kipcole9 , that sounds great.