26 October 2012

Overlay transparent image in OpenCV

It seems OpenCV does not provide very much support for transparent images. It has images with 4 channels, which are treated as BGRA images, with the fourth channel being the Alpha channel. It also supports saving these images using the PNG format, yet it does not support properly displaying these images in highgui, nor does it support combining these images.



I have looked a while for a way to combine a (BGR) background image with a (BGRA) foreground image, with semi-transparent images, but was unable to find one. So, I wrote my own and since I couldn't find a solution on the internet, I'll bet I can save some people some time by sharing my implementation, which is as follows:

void overlayImage(const cv::Mat &background, const cv::Mat &foreground, 
  cv::Mat &output, cv::Point2i location)
{
  background.copyTo(output);


  // start at the row indicated by location, or at row 0 if location.y is negative.
  for(int y = std::max(location.y , 0); y < background.rows; ++y)
  {
    int fY = y - location.y; // because of the translation

    // we are done of we have processed all rows of the foreground image.
    if(fY >= foreground.rows)
      break;

    // start at the column indicated by location, 

    // or at column 0 if location.x is negative.
    for(int x = std::max(location.x, 0); x < background.cols; ++x)
    {
      int fX = x - location.x; // because of the translation.

      // we are done with this row if the column is outside of the foreground image.
      if(fX >= foreground.cols)
        break;

      // determine the opacity of the foregrond pixel, using its fourth (alpha) channel.
      double opacity =
        ((double)foreground.data[fY * foreground.step + fX * foreground.channels() + 3])

        / 255.;


      // and now combine the background and foreground pixel, using the opacity, 

      // but only if opacity > 0.
      for(int c = 0; opacity > 0 && c < output.channels(); ++c)
      {
        unsigned char foregroundPx =
          foreground.data[fY * foreground.step + fX * foreground.channels() + c];
        unsigned char backgroundPx =
          background.data[y * background.step + x * background.channels() + c];
        output.data[y*output.step + output.channels()*x + c] =
          backgroundPx * (1.-opacity) + foregroundPx * opacity;
      }
    }
  }
}


This code only works if:
  • The background is in BGR colour space.
  • The foreground is in BGRA colour space.
The output image will always be of the same size as the background image, in BGR colour space. The position parameter determines how the foreground is placed on top of the background. A position of (100, -50) will move the foreground 100 pixels to the right and 50 pixels up.

It might not be the best solution as it was written in a hurry, so please feel free to improve.

Explanation of the code:
We first copy the background to the output, so we can continue with only the pixels where foreground and background overlap.
We start at the first scanline where they overlap, being either the top row of the background (y=0) or the top row of the foreground (y = position.y), so we take the maximum of these two. We do the same for the column value (x). We use fX and fY (foreground-x and foreground-y) as the current coordinates in the foreground image.
We determine the opacity of the foreground image using its fourth (alpha) channel. If it is > 0, we combine the pixels of the background and foreground for all channels.

We are done if we passed all overlapping pixels and output will now contain an image of the background, overlayed with the transparent foreground image.

Example usage:


int main(int argc, char *argv[])
{
  // add the second parameter "-1" as flag, to make sure the transparancy channel is read!
  cv::Mat foreground = imread("D:/images/foreground.png", -1);
  cv::Mat background = imread("D:/images/background.jpg");
  cv::Mat result;

  overlayImage(background, foreground, result, cv::Point(0,0));
  cv::imshow("result", result);
}


40 comments :

Devendra said...

here is code which I combined to test your function http://pastebin.com/rGnV6vy3

and following is the error I am getting


http://pastebin.com/HdBC6fad

Michael Jepson said...

Hi Devendra,

Could you check if both images are read correctly by showing them in a window before creating the overlay? I have seen reports by people having problems reading png files because of some issues with libpng.

Regards,
Michael

Devendra said...

Yes its working now Michael, Thanks I check and there was problem in my files.
Sorry to bother you for that.

Mihir Das said...

Thanks a lot for your code. It works for me like a magic.
Really you saved a lot of time of mine.. Thanks once again....

Mihir Das said...

Hi Michael,
As a posted earlier that you code is working correctly but i have got an issue,
Actually i am inserting the BGRA overlay google onto the Video , in my case openCV returns BGRA Image to process, As a result after you code runs the red Overlay image becomes blue, Actually i am very new to OpenCV and after doing doing lot of changes could not figure our what exactly should i do to make it work correctly. I tried to convert the BGRA image to BGR but in that case overlay image become correct but the background image becomes blue. Can you please help me out..

Thanks and Regards,
Mihir

Michael Jepson said...

Hi Mihir,

All images in OpenCV are in BGR or BGRA space by default. If you want to show them in the viewer in OpenCV, only BGR will show the correct colours.
You can convert colour spaces using the appropriate method (use cvtColor), but these other colour spaces will not show up correctly when using imshow, as it expects BGR.

Regards,
Michael

Mihir Das said...

Hi Michael,
Actually i am using your code in my iPhone application,In iphone using native API i am opening iPhone video Camera then i am processing each of the frame of Video with you code to insert an overlay image. I am getting BGRA format image from Video Camera and the output will be displayed in BGRA colorspace. But as you clearly mentioned that to run properly we need to have BGR image as background and BGRA image as foreground and the output will be in BGR color space. Can you please suggest any link or code which will overlay BGRA background with BGRA foreground and will output BGRA color space.That will solve my problem and it will be great help for me. Waiting for your reply.

Thanks and Regards,
Mihir Das

Michael Jepson said...

Hi Mihir,

From GBRA to GBR basically just means removing the fourth (transparency) channel.
But to be honest, I can not imagine your camera producing GBRA images, as the A is the transparency channel. No camera can record transparency as such, so no camera can include it in its imagery.
My code uses a non-transparent background (BGR) and overlays it with an image that has transparency (BGRA), producing a completely filled image, in which the whole transparency channel would be wasted, so it's BGR again.

I hope you can do something with this information.

Regards,
Michael

Mihir Das said...

Hi Michael,
You were absolutely correct as i my camera is giving image in RGB colorspace. I have just change it to BGR color space and your code works perfectly now.
Thank you so much for showing me the right way...

Adrian said...

Hi, first of all thanks man, you have saved my life.

So far I have made it run successfully but I'm having an issue with the opacity. The foreground image shows a low level of it and thats not desirable in my application.

Is there anything I could change in the code to increase opacity? Does its occur to you what could I be doing wrong?

Again, thank you very much for the code, and sorry to bother.

Michael Jepson said...

Hi Adrian,

Opacity depends on the png overlay image. Every pixel with Opacity = 255 should appear completely opaque.
If it seems to be transparent, I think the A-channel is off somehow. You could split the image and show the opacity (last) channel as a grey value image. Pixels that should not appear transparent should be pure white. If not, I think there is something "wrong" with the png image. You could also try opening the png image with an image editor and saving it again, OpenCV seems to be very picky as far as transparent images go, maybe the png contains some information that is somehow misinterpreted by OpenCV.

Regards,
Michael

Adrian said...

Thanks Michael, I'll try your advise

Regards

Ronan CIMADURE said...

Thanks a lot

Regard
Ronan

Hamid Bazargani said...

Thanks for sharing
It was so helpful

irlmaks said...

Hey,
I have recently started out on OpenCV and this is a problem that i am trying to solve. This isn't much literature that I could dig up online about transparency in OpenCV apart from this post. I tried implementing. The code is getting succesfully build, but the exe is crashing once it tries to run.

This is the code that I am using

http://pastebin.com/f5bGzL4B

It would be of really awesome help if you could point out what mistake I am doing.

Sven Frank said...

Thanks this post was very helpful.

But now I have another big problem and mabye you can help me.

As you did i want to create glasses as an overlay. I do the detection of both eyes by a haarcascade file.
So i get two center points, one for each eye. But the glasses should be displayed only once.

I have no idea how to calculate the loaction point for the glasses....

Best Regards

Sven

Michael Jepson said...

Hi Sven,

Assuming your glasses are properly centered in their image, I would suggest finding the center point between the two eyes.
If you know the (approximate) locations of the eye centers, you can take the average of those two points, which would be the point exactly between those two. Now you can place your glasses with the center of the transparent image over the point between the two eyes. You will need to calculate the needed translation and pass it to the overlay function.

Regards,

Michael

Sven Frank said...

Thank you Michael,

simple solution which works. No idea, but somehow I did not get it before. Maybe I already think that only a complicated solution can help me. Hahaha - Anyway thank you!

Sven

James Michael Dupont said...

Hi there,
can you please license this for inclusion in the opencv code? thanks
mike

fku fuy said...

Thanks a lot it helps me !

dinesh meena said...

I am getting an error this error:

OpenCV Error: Unspecified error (The function is not implemented. Rebuild the library with Windows, GTK+ 2.x or Carbon support. If you are on Ubuntu or Debian, install libgtk2.0-dev and pkg-config, then re-run cmake or configure script) in cvShowImage, file /home/dinesh/vtr/opencv/modules/highgui/src/window.cpp, line 501
terminate called after throwing an instance of 'cv::Exception'
what(): /home/dinesh/vtr/opencv/modules/highgui/src/window.cpp:501: error: (-2) The function is not implemented. Rebuild the library with Windows, GTK+ 2.x or Carbon support. If you are on Ubuntu or Debian, install libgtk2.0-dev and pkg-config, then re-run cmake or configure script in function cvShowImage

Aborted (core dumped)

Michael Jepson said...

Hi Dinesh,

Have you tried the solutions as specified in the error message? It seems the problem lies not in the rotation code, but in the ShowImage call. I see you are on a Linux or BSD (or othr *nix system) and you probably need certain libraries in order for the highgui libraries to operate properly - such as Gtk+ for instance).

Regards,
Michael

Dinesh Meena said...

Hey Michael , when I try to install gtk it shows that I already have it installed.

Anonymous said...

Very helpful. Works perfectly. Thanks.

Anonymous said...

Thank you, You save me.

Mark said...

Thanks for posting this info. I just want to let you know that I just check out your site and I find it very interesting and informative. I can’t wait to read lots of your posts

image masking service provider

cliping path said...

Thanks for posting this blog.It's informative as well as there is something new to learn from here.
clipping path service

Catalina G. Obrien said...

Oh my!!! You are just too true to be good. How do you manage to write and research on such wonderful things? You have inspired me to work harder now. I shall try as much as possible to enjoy life to the fullest and be satiated with the wonderful things that are around me, which I have been unaware of until now.

clipping path bangladesh

serge mbamba said...

Hi Michael,
My name is Serge. I would like to re-use your function in one my project.
Can I have a permission via email
at 492751@students.wits.ac.za.

Regards

Anonymous said...

Thanks bro, It was very helpful.

Anonymous said...

Hi!
I am a student from Poland, would you mind letting me use your code in my project?
Please let me know via email:
mb12345@wp.pl

Michael Jepson said...

Sure you can use my code in your project, that's why it's on the internet ;)

Anonymous said...

Thank you very much Michael !
That's very usefull

Anonymous said...

Easier way:

//Get alpha channel out and use as a mask
cv::Mat split_img[4];
cv::split(temp2, split_img);
temp2.copyTo(merged_image, split_img[3]);

Michael Jepson said...

Hi Anonymous,

If you use the alpha channel as a mask, you will get very hard edges and opaque sunglasses, as it won't "mix" the two images where the overlay is semi-transparent (i.e. has an alpha >0 and < 255).

Regards,
Michael

Max5684 said...

I know this thread is a bit old but I thought I'd post a copy of the function that modifies an existing Mat instead of creating a new output Mat in case anyone finds it useful: https://gist.github.com/maximus5684/082f8939edb6aed7ba0a

programmersn said...


Thank you soo much for sharing brother !
It was really helpful !

Lajel Omar said...

hi haw i can make this code work using java

dharini said...
This comment has been removed by the author.
Dharini Prajapati said...

code working perfect for me in ios but result image color changed