26 October 2012

Overlay transparent image in OpenCV

It seems OpenCV does not provide very much support for transparent images. It has images with 4 channels, which are treated as BGRA images, with the fourth channel being the Alpha channel. It also supports saving these images using the PNG format, yet it does not support properly displaying these images in highgui, nor does it support combining these images.



I have looked a while for a way to combine a (BGR) background image with a (BGRA) foreground image, with semi-transparent images, but was unable to find one. So, I wrote my own and since I couldn't find a solution on the internet, I'll bet I can save some people some time by sharing my implementation, which is as follows:

void overlayImage(const cv::Mat &background, const cv::Mat &foreground, 
  cv::Mat &output, cv::Point2i location)
{
  background.copyTo(output);


  // start at the row indicated by location, or at row 0 if location.y is negative.
  for(int y = std::max(location.y , 0); y < background.rows; ++y)
  {
    int fY = y - location.y; // because of the translation

    // we are done of we have processed all rows of the foreground image.
    if(fY >= foreground.rows)
      break;

    // start at the column indicated by location, 

    // or at column 0 if location.x is negative.
    for(int x = std::max(location.x, 0); x < background.cols; ++x)
    {
      int fX = x - location.x; // because of the translation.

      // we are done with this row if the column is outside of the foreground image.
      if(fX >= foreground.cols)
        break;

      // determine the opacity of the foregrond pixel, using its fourth (alpha) channel.
      double opacity =
        ((double)foreground.data[fY * foreground.step + fX * foreground.channels() + 3])

        / 255.;


      // and now combine the background and foreground pixel, using the opacity, 

      // but only if opacity > 0.
      for(int c = 0; opacity > 0 && c < output.channels(); ++c)
      {
        unsigned char foregroundPx =
          foreground.data[fY * foreground.step + fX * foreground.channels() + c];
        unsigned char backgroundPx =
          background.data[y * background.step + x * background.channels() + c];
        output.data[y*output.step + output.channels()*x + c] =
          backgroundPx * (1.-opacity) + foregroundPx * opacity;
      }
    }
  }
}


This code only works if:
  • The background is in BGR colour space.
  • The foreground is in BGRA colour space.
The output image will always be of the same size as the background image, in BGR colour space. The position parameter determines how the foreground is placed on top of the background. A position of (100, -50) will move the foreground 100 pixels to the right and 50 pixels up.

It might not be the best solution as it was written in a hurry, so please feel free to improve.

Explanation of the code:
We first copy the background to the output, so we can continue with only the pixels where foreground and background overlap.
We start at the first scanline where they overlap, being either the top row of the background (y=0) or the top row of the foreground (y = position.y), so we take the maximum of these two. We do the same for the column value (x). We use fX and fY (foreground-x and foreground-y) as the current coordinates in the foreground image.
We determine the opacity of the foreground image using its fourth (alpha) channel. If it is > 0, we combine the pixels of the background and foreground for all channels.

We are done if we passed all overlapping pixels and output will now contain an image of the background, overlayed with the transparent foreground image.

Example usage:


int main(int argc, char *argv[])
{
  // add the second parameter "-1" as flag, to make sure the transparancy channel is read!
  cv::Mat foreground = imread("D:/images/foreground.png", -1);
  cv::Mat background = imread("D:/images/background.jpg");
  cv::Mat result;

  overlayImage(background, foreground, result, cv::Point(0,0));
  cv::imshow("result", result);
}


58 comments :

Devendra said...

here is code which I combined to test your function http://pastebin.com/rGnV6vy3

and following is the error I am getting


http://pastebin.com/HdBC6fad

Michael Jepson said...

Hi Devendra,

Could you check if both images are read correctly by showing them in a window before creating the overlay? I have seen reports by people having problems reading png files because of some issues with libpng.

Regards,
Michael

Devendra said...

Yes its working now Michael, Thanks I check and there was problem in my files.
Sorry to bother you for that.

Mihir Das said...

Thanks a lot for your code. It works for me like a magic.
Really you saved a lot of time of mine.. Thanks once again....

Mihir Das said...

Hi Michael,
As a posted earlier that you code is working correctly but i have got an issue,
Actually i am inserting the BGRA overlay google onto the Video , in my case openCV returns BGRA Image to process, As a result after you code runs the red Overlay image becomes blue, Actually i am very new to OpenCV and after doing doing lot of changes could not figure our what exactly should i do to make it work correctly. I tried to convert the BGRA image to BGR but in that case overlay image become correct but the background image becomes blue. Can you please help me out..

Thanks and Regards,
Mihir

Michael Jepson said...

Hi Mihir,

All images in OpenCV are in BGR or BGRA space by default. If you want to show them in the viewer in OpenCV, only BGR will show the correct colours.
You can convert colour spaces using the appropriate method (use cvtColor), but these other colour spaces will not show up correctly when using imshow, as it expects BGR.

Regards,
Michael

Mihir Das said...

Hi Michael,
Actually i am using your code in my iPhone application,In iphone using native API i am opening iPhone video Camera then i am processing each of the frame of Video with you code to insert an overlay image. I am getting BGRA format image from Video Camera and the output will be displayed in BGRA colorspace. But as you clearly mentioned that to run properly we need to have BGR image as background and BGRA image as foreground and the output will be in BGR color space. Can you please suggest any link or code which will overlay BGRA background with BGRA foreground and will output BGRA color space.That will solve my problem and it will be great help for me. Waiting for your reply.

Thanks and Regards,
Mihir Das

Michael Jepson said...

Hi Mihir,

From GBRA to GBR basically just means removing the fourth (transparency) channel.
But to be honest, I can not imagine your camera producing GBRA images, as the A is the transparency channel. No camera can record transparency as such, so no camera can include it in its imagery.
My code uses a non-transparent background (BGR) and overlays it with an image that has transparency (BGRA), producing a completely filled image, in which the whole transparency channel would be wasted, so it's BGR again.

I hope you can do something with this information.

Regards,
Michael

Mihir Das said...

Hi Michael,
You were absolutely correct as i my camera is giving image in RGB colorspace. I have just change it to BGR color space and your code works perfectly now.
Thank you so much for showing me the right way...

Adrian said...

Hi, first of all thanks man, you have saved my life.

So far I have made it run successfully but I'm having an issue with the opacity. The foreground image shows a low level of it and thats not desirable in my application.

Is there anything I could change in the code to increase opacity? Does its occur to you what could I be doing wrong?

Again, thank you very much for the code, and sorry to bother.

Michael Jepson said...

Hi Adrian,

Opacity depends on the png overlay image. Every pixel with Opacity = 255 should appear completely opaque.
If it seems to be transparent, I think the A-channel is off somehow. You could split the image and show the opacity (last) channel as a grey value image. Pixels that should not appear transparent should be pure white. If not, I think there is something "wrong" with the png image. You could also try opening the png image with an image editor and saving it again, OpenCV seems to be very picky as far as transparent images go, maybe the png contains some information that is somehow misinterpreted by OpenCV.

Regards,
Michael

Adrian said...

Thanks Michael, I'll try your advise

Regards

Unknown said...

Thanks a lot

Regard
Ronan

Unknown said...

Thanks for sharing
It was so helpful

irlmaks said...

Hey,
I have recently started out on OpenCV and this is a problem that i am trying to solve. This isn't much literature that I could dig up online about transparency in OpenCV apart from this post. I tried implementing. The code is getting succesfully build, but the exe is crashing once it tries to run.

This is the code that I am using

http://pastebin.com/f5bGzL4B

It would be of really awesome help if you could point out what mistake I am doing.

Unknown said...

Thanks this post was very helpful.

But now I have another big problem and mabye you can help me.

As you did i want to create glasses as an overlay. I do the detection of both eyes by a haarcascade file.
So i get two center points, one for each eye. But the glasses should be displayed only once.

I have no idea how to calculate the loaction point for the glasses....

Best Regards

Sven

Michael Jepson said...

Hi Sven,

Assuming your glasses are properly centered in their image, I would suggest finding the center point between the two eyes.
If you know the (approximate) locations of the eye centers, you can take the average of those two points, which would be the point exactly between those two. Now you can place your glasses with the center of the transparent image over the point between the two eyes. You will need to calculate the needed translation and pass it to the overlay function.

Regards,

Michael

Unknown said...

Thank you Michael,

simple solution which works. No idea, but somehow I did not get it before. Maybe I already think that only a complicated solution can help me. Hahaha - Anyway thank you!

Sven

H4ck3rm1k3 said...

Hi there,
can you please license this for inclusion in the opencv code? thanks
mike

Unknown said...

Thanks a lot it helps me !

Unknown said...

I am getting an error this error:

OpenCV Error: Unspecified error (The function is not implemented. Rebuild the library with Windows, GTK+ 2.x or Carbon support. If you are on Ubuntu or Debian, install libgtk2.0-dev and pkg-config, then re-run cmake or configure script) in cvShowImage, file /home/dinesh/vtr/opencv/modules/highgui/src/window.cpp, line 501
terminate called after throwing an instance of 'cv::Exception'
what(): /home/dinesh/vtr/opencv/modules/highgui/src/window.cpp:501: error: (-2) The function is not implemented. Rebuild the library with Windows, GTK+ 2.x or Carbon support. If you are on Ubuntu or Debian, install libgtk2.0-dev and pkg-config, then re-run cmake or configure script in function cvShowImage

Aborted (core dumped)

Michael Jepson said...

Hi Dinesh,

Have you tried the solutions as specified in the error message? It seems the problem lies not in the rotation code, but in the ShowImage call. I see you are on a Linux or BSD (or othr *nix system) and you probably need certain libraries in order for the highgui libraries to operate properly - such as Gtk+ for instance).

Regards,
Michael

Unknown said...

Hey Michael , when I try to install gtk it shows that I already have it installed.

Anonymous said...

Very helpful. Works perfectly. Thanks.

Anonymous said...

Thank you, You save me.

Unknown said...

Thanks for posting this blog.It's informative as well as there is something new to learn from here.
clipping path service

Unknown said...

Hi Michael,
My name is Serge. I would like to re-use your function in one my project.
Can I have a permission via email
at 492751@students.wits.ac.za.

Regards

Anonymous said...

Thanks bro, It was very helpful.

Anonymous said...

Hi!
I am a student from Poland, would you mind letting me use your code in my project?
Please let me know via email:
mb12345@wp.pl

Michael Jepson said...

Sure you can use my code in your project, that's why it's on the internet ;)

Anonymous said...

Thank you very much Michael !
That's very usefull

Anonymous said...

Easier way:

//Get alpha channel out and use as a mask
cv::Mat split_img[4];
cv::split(temp2, split_img);
temp2.copyTo(merged_image, split_img[3]);

Michael Jepson said...

Hi Anonymous,

If you use the alpha channel as a mask, you will get very hard edges and opaque sunglasses, as it won't "mix" the two images where the overlay is semi-transparent (i.e. has an alpha >0 and < 255).

Regards,
Michael

Max5684 said...

I know this thread is a bit old but I thought I'd post a copy of the function that modifies an existing Mat instead of creating a new output Mat in case anyone finds it useful: https://gist.github.com/maximus5684/082f8939edb6aed7ba0a

programmersn said...


Thank you soo much for sharing brother !
It was really helpful !

Unknown said...

hi haw i can make this code work using java

Dharini Prajapati said...
This comment has been removed by the author.
Dharini Prajapati said...

code working perfect for me in ios but result image color changed

Unknown said...

can anyone help me to get the java code of the same work?

Him said...

Thank you for your code, i am currently using this code for a proyect in which i had to develop an overlay for drone cameras but i couldn´t make it work with opencv addWeighted method because i wanted an opaque overlay.

Anonymous said...

hey can u please give the code for overlaying images in real time video

Michael Jepson said...

Hi Anonymous,

You will just have to use this code for every frame in the real-time video. So basically, grab a frame, call the function to do the overlay and then show, send or encode the frame (or whatever you need to do with the frame afterwards).

Regards,

Michael

Anonymous said...

hey tq michael .I understood but then am a newbie to opencv and iOS so could you please help me more?

Sonya said...

Really you saved a lot of time of mine, Thanks once again.. i think it's a cool tricks

Anonymous said...

Hi Michael, i got a problem. Your code runs too slow for my needs.
I need to process two images per frame in a 30 fps video
overlayImage is at worst 10ms processing time in my cpu
I need to paralelize your method.
Is there a way to paralelize it?

Michael Jepson said...

Hi Anonymous,

You could just run every scanline in a separate thread for instance. So that if you have 8 cores, you can do 8 scanlines at once. Or you could see if you could divide the whole images in 8 equal parts, so for 800 scanlines, you'd get 8 parts of 100 scanlines each and then run those in parallel.

Regards,

Michael

Him said...
This comment has been removed by the author.
Anonymous said...

Thank you. It's work.

Unknown said...

thank you for your code! It helps a lot!

SmileJimin said...

Hi Thanks for your tutorial. For my usecase, I'd like overlay an png image as only background as transparent, but main object(here in your example sunglasses as opaque). If I just do the simple copyTo, then the overlay image copied with black background. Is there any way that I can do this? Basically I just want to add the overlay png image as photoshop layer.

Udaib khan said...

Hey Micheal Jepson i am also facing such kind of problem i.e image overlay using transparency can you please help me ..
i just want to place one image on another using image overlay transparency.
here is my problem
https://www.mathworks.com/matlabcentral/answers/383715-place-one-image-on-another-image-using-image-overlay

Michael Jepson said...

Hi Udaib,

Just lower the opacity value. So if yo have a foreground image that consists of a non-transparent logo, on a transparent background, you could just alter the opacity.

You can see that before the overlay I calculate the opacity by getting the 4th channel value of a pixel of the foreground, which I divide by 255.0 to get a value between 0 and 1.
You could just add a line that says something like:
if(opacity > 0) opacity = 0.5;
and then let it continue. This will make the non-transparent appear 50% transparent. Obviously you will have to play around with the opacity to get the desired effect.

Good luck!

Best regards,

Michael

Udaib khan said...

Michael Jepson basically i have applied this already but i am getting wrong result
i only want to overlay logo over shirt image such that it looks realtics please see my required images result

clipping path associate said...

It is very Useful post to me and i am also web developer.

Unknown said...

can i get the java code of this please ?

Anonymous said...

Amazing, thank you. I had to apply an overlay to a radar video stream, and this work perfectly on the first try.

Anonymous said...

You for sure saved me some time! thank you!

Anonymous said...

Thank you bro, you're life saver