Basic Image Operations

After acquiring an image, it is first necessary to understand the basic operations for processing the image, such as separating image colors, modifying pixels, stretching and rotating the image, and even adding some basic shapes to the image and performing simple processing. Therefore, this chapter focuses on the basic image operations provided in OpenCV 4, including the introduction of color spaces, pixel operations, changing image shapes, drawing geometric figures, and generating image pyramids, etc.

1.Image color space

By mixing the three colors red, green, and blue in different proportions, images can display a rich array of colors. This model is called the RGB color model. The RGB color model is one of the most common color models and is often used to represent and display images. To represent the mixture of the three colors, images store the red component, green component, and blue component of a certain color separately in a multi-channel format. In addition to the RGB color model, image color models also include YUV, HSV, and other models, which respectively represent components such as brightness, chrominance, and saturation of the image. Understanding image color spaces is of great help for segmenting images with color-distinguishing features. For example, extracting red objects from an image can be achieved by comparing the pixel values of the image's red channel.

1.1.Color Models and Conversion

This section will introduce several common color models in OpenCV 4 that can be converted to each other, such as the RGB model, HSV model, Lab model, YUV model, and GRAY model, and will introduce the mathematical conversion relationships among these models as well as the transformation functions provided in OpenCV 4 for conversions among these models.

1. RGB color model

The RGB color model has been introduced previously. This model is named using the English initials of three colors: Red, Green, and Blue. Although the color model is named with red first, in OpenCV the order is reversed: the first channel is the Blue (B) component, the second channel is the Green (G) component, and the third channel is the Red (R) component. Depending on the storage order, OpenCV 4 provides a reverse-order format of this order for storing images in which the first channel is the red component, but the color space of these two image formats is the same. The color space model is shown in Figure 3-1. The three channels have the same range for color description; therefore, the spatial structure of the RGB color model is a cube. In the RGB color model, all colors are obtained by mixing these three colors in different proportions. If all three color components are 0, the color is represented as black; if the three color components are the same and all at their maximum values, the color is represented as white. Each channel represents the process of a certain color from 0 to 1, and images with different bit depths represent this color variation process subdivided into different levels. For example, an image in 8UC3 format quantizes this process into 256 levels per channel, represented by 0 to 255 respectively. Adding a fourth channel to this model yields the RGBA model. The fourth channel represents color transparency. When there is no need for transparency, the RGBA model degenerates into the RGB model.

2. YUV color model

The YUV model is the color encoding method used in television signal systems. These three variables respectively represent the brightness (Y) of a pixel, the difference between the red component and the brightness signal (U), and the difference between blue and brightness (V). This color model is mainly used for the transmission of video and images, and the emergence of this model is closely related to the development history of television sets. Since color televisions were invented after black-and-white televisions, video signals used for color televisions needed to be compatible with black-and-white televisions. Color televisions require data from 3 channels to display color, whereas black-and-white televisions require data from only one channel. Therefore, in order to make video signals compatible with both color televisions and black-and-white televisions, the RGB encoding method is converted into the YUV encoding method. Its Y channel is the brightness of the image, and black-and-white televisions need only use this channel to display black-and-white video images, while color televisions can display color images by converting YUV encoding into RGB encoding. This better solves the problem of the same video signal being compatible with different types of television sets. The conversion relationship between the RGB model and the YUV model is shown in Equation (3-1), where the value range of RGB is all 0 to 255.

3. HSV color model

HSV is the abbreviation for Hue, Saturation, and Value. As can be seen from the name, this model describes colors through these three attributes. Hue is the basic attribute of color, which is commonly referred to as color, such as red, blue, etc.; saturation refers to the purity of the color. The higher the saturation, the purer and more vivid the color; the lower the saturation, the more the color gradually becomes gray and dark. The value range of saturation is 0% to 100%; value is the brightness of the color, and its value range is from 0 to the maximum value allowed in the computer. Since the value ranges of hue, saturation, and value are different, its color space model is represented by a cone, as shown in Figure 3-2. Compared with the shortcoming of the RGB model that the three color components are not intuitively related to the final color, the HSV model is more in line with the way humans perceive color: color, shade, and brightness.

4. Lab color model

The Lab color model compensates for the deficiencies of the RGB model and is a device-independent color model based on physiological characteristics. In the model, L represents luminosity, and a and b are two color channels, both with a value range of -128 to 127, where the color corresponding to the a channel value from small to large changes from green to red, and the color corresponding to the b channel value from small to large changes from blue to yellow. The color space formed by the Lab color model is a sphere, as shown in Figure 3-3.

5. GRAY Color Model

The GRAY model is not a color model, but a grayscale image model; its name uses the English word gray in all uppercase letters. A grayscale image has only a single channel. Depending on the image bit depth, grayscale values from 0 to the maximum represent black to white in sequence. For example, in the 8UC1 format, black to white is quantized into 256 levels, represented by 0 to 255, where 255 represents white. Color images have the characteristics of rich colors and high information content, but grayscale images still have certain advantages in image processing. For example, grayscale images offer the advantages of small storage size for the same dimensions and compression format, ease of acquisition, and convenience of transmission. The commonly used method for converting an RGB model to a grayscale image is shown in Equation (3-2).

6. Interconversion Between Different Color Models

For mutual conversion between different color models of images, OpenCV 4 provides the cvtColor() function to implement the conversion functionality. The prototype of this function is given in code listing 3-1.

Code Listing 3-1 cvtColor() function prototype

void cv::cvtColor(InputArray src,
                  OutputArray dst,
                  int code,
                  int dstCn = 0);

• src: The original image of the color model to be converted.
• dst: The target image after color model conversion.
• code: The flag for color space conversion, such as from RGB space to HSV space. Common flags and their meanings are given in Table 3-1.
• dstCn: The number of channels in the target image. If the parameter is 0, the number of channels is automatically derived from src and code.

The function is used to convert an image from one color model to another color model. The first two parameters are used for the input image to be converted and the target image after color space conversion. The third parameter is used to specify the specific conversion model space of the function. Commonly used flags are given in Table 3-1. Readers can consult the OpenCV 4 tutorials to learn the detailed flags. The fourth parameter generally does not require special setting and can use the default parameter. It should be noted that the image value range before and after the transformation by this function: since the pixels of an 8-bit unsigned image are 0 to 255, those of a 16-bit unsigned image are 0 to 65535, and those of a 32-bit floating-point image are 0 to 1, special attention must be paid to the pixel range of the target image. In the case of linear transformation, the range issue does not need to be considered, and the pixels of the target image will not exceed the range. In the case of nonlinear transformation, the input RGB image should be normalized to an appropriate range to obtain correct results. For example, to convert an 8-bit unsigned image to a 32-bit floating-point image, the image pixels need to first be scaled to the range 0 to 1 by dividing by 255 to prevent erroneous results.

Note: If an alpha channel (the fourth channel in the RGB model, representing transparency) is added during the conversion process, its value will be set to the maximum value of the corresponding channel range: CV_8U is 255, CV_16U is 65535, and CV_32F is 1.

Table 3-1 Common Flag Parameters for Color Model Conversion in the cvtColor() Function

To intuitively experience how the same image appears in different color spaces, Code Listing 3-2 provides a program for converting among the aforementioned color models, and the running results are shown in Figure 3-4. It should be noted that the Lab color model contains negative values, and images displayed via the imshow() function cannot display negative values; therefore, the results include the appearance of the image in the Lab model as displayed using the Image Watch plugin. In the program, to prevent out-of-range values after conversion, we first convert the CV_8U type to the CV_32F type before performing the color model conversion.

Code Listing 3-2 myCvColor.cpp Mutual Conversion Between Image Color Models

#include "chapter3_1_cvclolor/inc/CvColor.hpp"
#include <cstdio>
#include <opencv2/opencv.hpp>


int opencv_function9(void)
{
  const std::string & file_name = std::string(MEDIA_PATH) + "林星阑L.jpg";

  cv::Mat img = cv::imread(file_name);
  if(img.empty() == true)
  {
      std::cout << "请确认图像文件是否正确，请检查输入格式" << std::endl;
      return 1;
  }
  else
  {
      std::cout << "图像成功读取!" << std::endl;
  }
  cv::Mat img32;
  cv::Mat gray,HSV,YUV,Lab;
  img.convertTo(img32,CV_32F,1.0/255);   //缩放因子:1.0/255指将现在的图像的范围转换为目标图像的范围需要乘以的因数
  cv::cvtColor(img32,HSV,cv::COLOR_BGR2HSV);
  cv::cvtColor(img32,YUV,cv::COLOR_BGR2YUV);
  cv::cvtColor(img32,Lab,cv::COLOR_BGR2Lab);
  cv::cvtColor(img32,gray,cv::COLOR_BGR2GRAY);

  cv::namedWindow("原图BGR",cv::WINDOW_FREERATIO);
  cv::namedWindow("HSV",cv::WINDOW_FREERATIO);
  cv::namedWindow("YUV",cv::WINDOW_FREERATIO);
  cv::namedWindow("Lab",cv::WINDOW_FREERATIO);
  cv::namedWindow("gray",cv::WINDOW_FREERATIO);
  cv::imshow("原图BGR",img32);
  cv::imshow("HSV",HSV);
  cv::imshow("YUV",YUV);
  cv::imshow("Lab",Lab);
  cv::imshow("gray",gray);
  cv::waitKey(0);

  return 0;
}

In the program, we utilize the data type conversion function convertTo() provided by the Mat class in OpenCV 4. In everyday use of image data, we often encounter issues with converting between different data types. Therefore, the following provides a detailed introduction to the usage of this conversion function. In code listing 3-3, the prototype of this function is given.

Code Listing 3-3 convertTo() function prototype

void cv::Mat::convertTo(OutputArray m,
                        int rtype,
                        double alpha = 1,
                        double beta = 0);

• m: The output image after type conversion.
• rtype: The data type of the converted image.
• alpha: The scaling factor during conversion.
• beta: The bias factor during conversion.

This function is used to convert an existing image into an image of a specified data type. The first parameter is used to output the image after data type conversion, and the second parameter is used to declare the data type of the converted image. The third and fourth parameters are used to declare the conversion relationship between the two data types. The specific conversion form is shown in Equation (3-3).

From the conversion formula, it can be seen that this conversion method performs a linear transformation on the original data and outputs it according to the specified data type. According to its conversion rules, it can be seen that this function can not only convert between different data types but also perform linear transformations within the same data type. In code listing 3-2, we provide an example of mutual conversion between the CV_8U type and the CV_32F type. Mutual conversion between other types is similar and will not be repeated here. Readers can explore on their own and experience the usage of this function through practice.

1.2.Multi-channel separation and merging

In an image color model, different components are stored in different channels. If we only need one component of the color model, for example, if we only need to process the red channel in an RGB image, then we can separate the red channel from the data of the three channels and then process it. This approach can reduce the memory occupied by the data and speed up the program's execution. At the same time, after we have processed multiple channels separately, we need to merge all the channels together to regenerate the RGB image. For the separation and merging of multiple image channels, OpenCV 4 provides the split() function and the merge() function to meet these needs.

1. Multichannel separation function split()

In OpenCV 4, there are two overloaded prototypes for the multi-channel separation function split(), and these two function prototypes are given in Code Listing 3-4.

Code Listing 3-4 Prototype of the split() Function

void cv::split(const Mat &src,
               Mat *mvbegin);

void cv::split(InputArray m,
               OutputArrayOfArrays mv);

• mvbegin: The separated single-channel images, in array form; the array size must be the same as the number of channels of the image.
• m: The multichannel image to be separated.
• mv: The separated single-channel images, in vector form.

This function is mainly used to separate a multi-channel image into several single-channel images. The difference between the two function prototypes is that the second parameter of the former outputs an array of Mat type, whose array length must be equal to the number of channels of the multi-channel image and defined in advance; the second parameter of the second function prototype outputs avector<Mat>container, and there is no need to know the number of channels of the multi-channel image. Although the types of input parameters of the two function prototypes are different, the principle of channel separation is the same.

2. Multi-channel merge function merge()

In OpenCV 4, the multi-channel merge function merge() also has two overloaded prototypes, both of which are given in Code Listing 3-5. Code Listing 3-5 Prototypes of the merge() function

void cv::merge(const Mat *mv,
               size_t count,
               OutputArray dst
                )

void cv::merge(InputArrayOfArrays mv,
               OutputArray dst);

• mv（第一种重载原型参数）：需要合并的图像数组，其中每个图像必须拥有相同的尺寸和数据类型。
• count：输入的图像数组的长度，其数值必须大于0。
• mv（第二种重载原型参数）：需要合并的图像向量(vector)，其中每个图像必须拥有相同的尺寸和数据类型。
• dst：合并后输出的图像，与mv[0]具有相同的尺寸和数据类型，通道数等于所有输入图像的通道数总和。

This function is primarily used to merge multiple images into a single multi-channel image. This function also has two different function prototypes, each corresponding to the split() function. The two prototypes respectively accept image data in array form and image data in vector form. In the prototype that accepts data in array form, the length of the array must also be provided. The output of the merge function is a multi-channel image, whose number of channels is the sum of the channel counts of all input images. It should be noted here that the images used for merging are not necessarily all single-channel; multiple images with different numbers of channels can also be merged into an image with more channels. Although these images may have different numbers of channels, all images are required to have the same size and data type.

3. Example Routine for Multi-channel Image Splitting and Merging

To make readers more familiar with the operations of image multi-channel separation and merging, and to deepen their understanding of the roles of different image channels, Code Listing 3-6 implements the functions of multi-channel separation and merging of images. In the program, two function prototypes are used to separate RGB images and HSV images respectively. To verify that the merge() function can merge multiple images with different channels, the program uses two function prototypes to merge multiple images with different channels. The merged image has 5 channels and cannot be displayed by the imshow() function, so we use the Image Watch plugin to view the merged result. Since the three separated RGB channels are all displayed in grayscale and differ little from one another, Figure 3-5 does not present their separation results; it only presents the merged image displayed in green after merging, and also presents the HSV separation results. Readers can view the other results by running the program themselves. Code Listing 3-6 mySplitAndMerge.cpp Implementing image separation and merging

#include "chapter3_1_cvclolor/inc/SplitAndMerge.hpp"
#include <cstdio>
#include <opencv2/opencv.hpp>

bool Mat_Arr_Split_merge(const std::string &file_name);
bool Vector_Split_merge(const std::string &file_name);


int opencv_function10(void)
{
  Mat_Arr_Split_merge(std::string(MEDIA_PATH) + "林星阑L.jpg");
//   Vector_Split_merge(std::string(MEDIA_PATH) + "林星阑L.jpg");

  return 0;
}

bool Mat_Arr_Split_merge(const std::string &file_name)
{
  cv::Mat img = cv::imread(file_name);
  if(img.empty() == false)
  {
    printf("成功读取图片!");
  }
  else
  {
    printf("无法读取图片，请确定图片文件是否存在，输入格式是否正确!");
    return false;
  }
  cv::Mat imgs[3];
  cv::Mat result[2];
  cv::split(img,imgs);
  cv::namedWindow("RGB-B通道",cv::WINDOW_FREERATIO);
  cv::namedWindow("RGB-G通道",cv::WINDOW_FREERATIO);
  cv::namedWindow("RGB-R通道",cv::WINDOW_FREERATIO);
  cv::imshow("RGB-B通道",imgs[0]);
  cv::imshow("RGB-G通道",imgs[1]);
  cv::imshow("RGB-R通道",imgs[2]);
  imgs[2] = img;  //改变图像通道数量
  cv::merge(imgs,3,result[0]);
  // cv::namedWindow("合并图像结果0",cv::WINDOW_FREERATIO);
  // cv::imshow("合并图像结果0",result[0]);   //imshow最多显示4个通道，需要Image Watch插件
  cv::Mat zero = cv::Mat::zeros(img.rows,img.cols,CV_8UC1);  //一个通道的0矩阵
  imgs[0] = zero;
  imgs[2] = zero;
  cv::merge(imgs,3,result[1]);
  cv::namedWindow("合并图像结果1",cv::WINDOW_FREERATIO);
  cv::imshow("合并图像结果1",result[1]);   //显示合并结果
  cv::waitKey(0);
  cv::destroyAllWindows();
  return true;
}

bool Vector_Split_merge(const std::string &file_name)
{
  cv::Mat img = cv::imread(file_name);
  if(img.empty() == false)
  {
    printf("成功读取图片!");
  }
  else
  {
    printf("无法读取图片，请确定图片文件是否存在，输入格式是否正确!");
    return false;
  }
  cv::Mat HSV;
  cv::Mat result;
  cv::cvtColor(img,HSV,cv::COLOR_RGB2HSV);
  std::vector<cv::Mat> imgv;
  cv::split(HSV,imgv);
  cv::namedWindow("HSV-H通道",cv::WINDOW_FREERATIO);
  cv::namedWindow("HSV-S通道",cv::WINDOW_FREERATIO);
  cv::namedWindow("HSV-V通道",cv::WINDOW_FREERATIO);
  cv::imshow("HSV-H通道",imgv.at(0));
  cv::imshow("HSV-S通道",imgv.at(1));
  cv::imshow("HSV-V通道",imgv.at(2));
  imgv.push_back(HSV);  //将imgv中的图像通道变成不一致
  cv::merge(imgv,result);
  // cv::namedWindow("合并图像结果2",cv::WINDOW_FREERATIO);
  // cv::imshow("合并图像结果2",result);   //imshow最多显示4个通道，需要Image Watch插件
  cv::waitKey(0);
  cv::destroyAllWindows();
  return true;
}

2.Image pixel manipulation processing

After gaining an understanding of the different channels of an image, the next step is to introduce operations related to image pixels within each channel. The relevant concepts of pixels have been introduced earlier; for example, in a CV_8U image, the pixel value range is divided into 256 levels from black to white, and the grayscale value from 0 to 255 represents this process of change. Therefore, the magnitude of a pixel's grayscale value indicates the brightness or darkness of the pixel at a certain position, and at the same time, the degree of change in grayscale value also indicates the degree of change in image texture. Thus, understanding pixel-related operations is the first step in understanding image content.

2.1.Image pixel statistics

We can understand a digital image as a matrix of a certain size, where the value of each element in the matrix represents the brightness or darkness of each pixel in the image. Therefore, finding the maximum value in the matrix is equivalent to finding the pixel with the highest grayscale value in the image, and computing the mean is equivalent to computing the average grayscale of the image pixels, which can be used to represent the overall brightness or darkness of the image. Thus, statistical operations on matrix data also have certain significance and role for image pixels. OpenCV 4 integrates many functions for statistics, such as finding the maximum value, minimum value, mean, and standard deviation of image pixels. The relevant functions for these features are described in detail below.