Data loading, display, and saving

To process an image, the first step is to obtain that image. In everyday life, we can capture photos using devices such as cameras and smartphones, and store them on a hard drive in some format. Similarly, when performing image processing in a computer program, it is also necessary to acquire image data in a specific way, store it in a corresponding container using a certain data type, and then present it to the user in some form. Therefore, this chapter will introduce the loading, storage, and output of image data, including reading images and videos, creating and using image storage containers, and saving the processed results in image or video form, among other topics.

1.Image storage container

Unlike the images we see in everyday life, digital images are stored in computers in the form of matrices. Each element in the matrix is used to describe some kind of information in the image, such as brightness, color, etc., as shown in Figure 2-1. The essence of digital image processing is the process of extracting deeper-level information from these matrix data through a series of operations. Therefore, the first step in learning image processing is to master how to operate on these matrix data. For readers who have been exposed to C++ programming, strings are usually stored as the string type, and integers are stored as the int type. Similarly, in OpenCV, a Mat class is provided for storing matrix data. This section will introduce in detail how to use the Mat class and the operations it supports. Through this learning, you will be able to flexibly use Mat-type variables in programs.

1.1.Introduction to the Mat class

In the early OpenCV 1.0 version, images were stored using a C-language structure called IplImage. Therefore, its use can still be seen in some older OpenCV tutorials.

However, the IplImage type has a clear drawback—it requires the user to manually free memory. If there are still unreleased IplImage variables when the program ends, it will cause memory leak problems.

With the continuous development of OpenCV, the C++ interface was introduced into the library, and the Mat class was provided for data storage. The Mat class adopts an automatic memory management mechanism, which effectively solves the problem of memory release. When a variable is no longer used, the memory it occupies is automatically released.

The Mat class is used to store matrix-type data, including vectors, matrices, as well as grayscale images and color images, etc.

Structurally, the Mat class consists of two parts: a matrix header and a pointer to the actual data. The matrix header contains information such as the matrix dimensions, storage method, data address, and reference count. The size of the matrix header is fixed and does not change with the size of the matrix.

In the vast majority of cases, the space occupied by the matrix header is far smaller than the matrix data itself; therefore, during image copying and transmission, the primary overhead comes from the data portion.

To solve this problem, when copying or passing an image, OpenCV does not copy the complete data, but only copies the matrix header and a pointer to the data. Therefore, when creating a Mat object, you can first create the matrix header and then assign data to it. The specific method is shown in Code Listing 2-1.

cv::Mat a;                      // 创建一个名为 a 的矩阵头
a = cv::imread("test.jpg");    // 读取图像数据，使 a 指向图像像素数据
cv::Mat b = a;                 // 复制矩阵头，并命名为 b

The above code first creates a matrix header named a, then reads an image and makes the matrix pointer in a point to the pixel data of that image. Finally, the matrix header of a is copied to b. It should be noted that although a and b each have independent matrix headers, their matrix pointers point to the same block of data. Therefore, modifying the data through either matrix header will also change the data accessed by the other matrix header. In addition, when the variable a is deleted, b does not become empty data; only after both a and b are released will the underlying matrix data actually be released. This is because a reference count is maintained in the matrix header to record how many objects currently share the same block of data. Only when the reference count decreases to 0 will that data be released.

tips: Using reference counting to free stored content is a common approach in C++. This approach can avoid the problem of deleting data while a variable still references it, which would cause the program to crash, while also greatly reducing the memory occupied when the program is running.

Next, we will explain the data types that can be stored in the Mat class. According to the official inheritance diagram of the Mat class, as shown in Figure 2-2, we find that the data types that the Mat class can store include double, float, uchar, unsigned char, as well as custom templates, etc.

We can declare a Mat class variable that stores a specified type through the method shown in Code Listing 2-2.

cv::Mat A = cv::Mat_<double>(3, 3);  // 创建一个 3×3 的 double 类型矩阵

Since the Mat class in OpenCV is primarily used to store image data, and the range of pixel values directly affects image quality, using 8-bit unsigned integers to store 16-bit image data would cause severe color distortion and even data errors.

Additionally, compilers with different word sizes may define data type lengths differently. To avoid issues in program execution caused by inconsistent variable bit widths across different platforms or environments, OpenCV has standardized its data types.

Therefore, OpenCV defines a dedicated data type system based on the number of storage bits of numeric variables. Table 2-1 lists the commonly used data types in OpenCV and their value ranges.

Data types alone are not sufficient to fully describe image data; the number of image channels (Channel) also needs to be defined. For example, grayscale images are single-channel data, while color images are typically 3-channel (RGB) or 4-channel (such as RGBA) data.

To address this requirement, OpenCV defines channel-count identifiers: C1, C2, C3, and C4, representing single-channel, two-channel, three-channel, and four-channel data, respectively. Since each data type may correspond to a different number of channels, OpenCV combines the "data type" and the "number of channels" to fully describe the image data type. For example: CV_8UC1: represents 8-bit unsigned, single-channel data, typically used for grayscale images. CV_8UC3: represents 8-bit unsigned, three-channel data, typically used for color images. We can create a Mat object with a specified data type and number of channels as shown in Code Listing 2-3.

cv::Mat a(640, 480, CV_8UC3);  // 创建一个 640×480 的 3 通道矩阵（用于彩色图像）

cv::Mat b(3, 3, CV_8UC1);      // 创建一个 3×3 的 8 位无符号单通道矩阵

cv::Mat c(3, 3, CV_8U);        // 创建单通道矩阵，C1 标识可以省略

Although in 64-bit compilers, uchar and CV_8U both represent 8-bit unsigned integers, there is a strict distinction between the two: CV_8U can only be used for data type identification within the Mat class. If you use Mat_<CV_8U>(3,3) or Mat a(3,3,uchar) an error will be prompted during creation.

1.2.Mat Class Construction and Assignment

The previous subsection introduced three methods for constructing Mat class variables, but the latter two did not initialize the variables with values. This subsection focuses on how to flexibly construct and assign values to Mat class variables. According to the OpenCV source code definitions, there are more than 20 ways to construct the Mat class. However, in simple everyday applications, many complex construction methods are not very useful. Therefore, this book focuses on the construction and assignment methods commonly used by the author in learning and project work.

1. Mat class construction (1) Using the default constructor (see code listing 2-4)

cv::Mat::Mat();

Through Code Listing 2-4, a Mat class is constructed using the default constructor. This construction method does not require any input parameters. When subsequently assigning values to the variable, it automatically determines the type and size of the matrix, enabling flexible storage. It is commonly used to store read image data and the output results of a function operation.

(2) Construct based on the input matrix size and type (see Code Listing 2-5)

cv::Mat::Mat(int rows,
             int cols,
             int type);

• rows: Number of rows to construct the matrix.
• cols: Number of columns of the matrix.
• type: In addition to CV_8UC1, CV_64FC4, etc. from 1 to 4 channels, more channel parameters are provided, and a multi-channel matrix is constructed via n in CV_8UC(n), where n can be up to 512.

We have also seen this construction method in the previous text. It constructs by inputting the number of rows and columns of the matrix, as well as the storage data type. This definition method is clear, intuitive, and easy to read, and is commonly used when the storage data dimensions and data type are explicitly required, such as the camera intrinsic matrix, the object's rotation matrix, etc. There is a variant of the method for constructing the Mat class using the input matrix size and data type. It assigns values by combining the number of rows and columns into a Size() structure. Code listing 2-6 gives the prototype of this construction method.

Code Listing 2-6 Constructing a Mat class using the Size() structure

cv::Mat::Mat(Size size(),
             int type);

• size: Size of a two-dimensional array variable, assigned via Size(cols, rows).
• type: Same as the parameters in code listing 2-5.

When constructing the Mat class in this way, special attention must be paid: in the Size() structure, the order of the matrix rows and columns is opposite to the method in Code Listing 2-5. When using Size() (see Code Listing 2-7), the columns come first and the rows come after. If this is not noticed, although the Mat class will still be constructed successfully, when we need to view a certain element, we will not know that the rows and columns are reversed, and an array out-of-bounds error may occur.

Code Listing 2-7 Example of constructing a Mat using the Size structure

cv::Mat a(Size(480, 640), CV_8UC1);   // 构造一个行为640、列为480的单通道矩阵
cv::Mat b(Size(480, 640), CV_32FC3); // 构造一个行为640、列为480的3通道矩阵

(3) Construct using an existing matrix (see code listing 2-8)

cv::Mat::Mat(const Mat &m);

• m: The already constructed Mat class matrix data.

This construction method is very simple and can construct a variable with the same stored contents as an existing Mat class variable. Note that this construction method only copies the matrix header of the Mat class, and the matrix pointer points to the same address. Therefore, if the data in the matrix is modified through one Mat class variable, the data in the other variable will also change.

Hint: If you want to copy two identical Mat classes without affecting each other, you can use the clone() function.

If the matrix to be constructed is smaller in size than an existing matrix and stores a sub-content of the existing matrix, then it can be constructed using the method in Code Listing 2-9.

Listing 2-9 Constructing a subclass of an existing Mat class

cv::Mat::Mat(const Mat &m,
             const Range &rowRange,
             const Range &colRange = Range::all());

• m: The already constructed Mat class matrix data.
• rowRange: The range of rows to be extracted from the existing matrix. It is a Range variable. For example, from row 2 to row 5 can be represented as Range(2,5).
• colRange: The range of columns to be extracted from the existing matrix. It is a Range variable. For example, from column 2 to column 5 can be represented as Range(2,5). When no value is provided, it indicates that all columns will be extracted.

This method is mainly used for taking a screenshot from the original image. However, it should be noted that a Mat object constructed in this way shares the same data with an existing Mat object, that is, if the data in one of the two Mat objects is modified, the other will change accordingly.

Code Listing 2-10 Extracting a Sub-Mat from the Original Mat

cv::Mat b(a, Range(2, 5), Range(2, 5));  // 从a中截取部分数据构造b
cv::Mat c(a, Range(2, 5));               // 默认最后一个参数构造c

2. Assignment of the Mat Class After constructing the Mat class, the variable does not contain any data, so data needs to be assigned to it. For different situations, OpenCV 4.1 provides multiple assignment methods. The following describes how to assign values to Mat class variables.

(1) Assigning values during construction (see code listing 2-11)

cv::Mat::Mat(int rows,
             int cols,
             int type,
             const Scalar &s);

• rows: the number of rows of the matrix.
• cols: the number of columns of the matrix.
• type: the type of data stored.
• s: the parameter variable that assigns a value to each pixel in the matrix, for example Scalar(0,0,255).

In this method, assignment is performed at the same time as construction (see code listing 2-12); simply place the value to be assigned to each element into the Scalar structure. Note here that using this method will assign the same numerical value to every element in the image; for example, Scalar(0,0,255) will assign the three channel values of each pixel as 0, 0, and 255 respectively.

Listing 2-12 Example of assigning values during construction

cv::Mat a(2, 2, CV_8UC3, cv::Scalar(0, 0, 255));  // 创建一个3通道矩阵，每个像素都是0,0,255
cv::Mat b(2, 2, CV_8UC2, cv::Scalar(0, 255));     // 创建一个2通道矩阵，每个像素都是0,255
cv::Mat c(2, 2, CV_8UC1, cv::Scalar(255));         // 创建一个单通道矩阵，每个像素都是255

We set a breakpoint before the program's return statement for debugging and used Image Watch to inspect the data in each Mat class variable. The result is shown in Figure 2-3, which proves that the matrix was successfully constructed and assigned values.

Note: The number of variables in the Scalar structure must correspond to the number of channels in the definition. If the number of variables in the Scalar structure is greater than the number of channels, values positioned after the number of channels will not be read. For example, after executing a(2,2,CV_8UC2,Scalar(0,0,255)), each pixel value will be (0,0), and 255 will not be read. If the number of variables in the Scalar structure is less than the number of channels, it will be supplemented with 0.

(2) Assignment by enumeration method

This assignment method enumerates all elements in the matrix one by one and assigns them to the Mat class in the form of a data stream. The specific assignment form is shown in Code Listing 2-13.

Code Listing 2-13 Example of Assignment Using Enumeration Method

cv::Mat a = (cv::Mat_<int>(3, 3) << 1, 2, 3, 4, 5, 6, 7, 8, 9);
cv::Mat b = (cv::Mat_<double>(2, 3) << 1.0, 2.1, 3.2, 4.0, 5.1, 6.2);

The first line of code above creates a 3×3 matrix, in which the 9 integers from 1 to 9 are stored. The first row of the matrix is filled first, and then the second and third rows are stored, that is, 1, 2, and 3 are stored in the first row of matrix a, 4, 5, and 6 are stored in the second row of matrix a, and 7, 8, and 9 are stored in the third row of matrix a. The second line of code creates a 2×3 matrix, and its storage method is the same as that of matrix a.

Note: When using the enumeration method, the number of input data must be equal to the number of matrix elements. For example, in Code Listing 2-13, if the first line of code inputs only the 8 numbers 1 to 8, an error will occur during the assignment process. Therefore, this method is commonly used when the matrix data is relatively small.

(3) Circular assignment method

Similar to the assignment method through enumeration, loop-based assignment also assigns values to each element in the matrix, but it does not need to be assigned when the variable is declared, and it can assign values to any part of the matrix. The specific assignment form is shown in code listing 2-14.

Code Listing 2-14 Example of Assignment Using the Loop Method

cv::Mat c = cv::Mat(3, 3, CV_8UC1);  // 定义一个3*3的矩阵
for (int i = 0; i < c.rows; i++)    // 矩阵行数循环
{
    for (int j = 0; j < c.cols; j++)  // 矩阵列数循环
    {
        c.at<uchar>(i, j) = i + j;
    }
}

The above code also creates a 3×3 matrix and assigns values to each element of the matrix through a for loop. Note that when assigning a value to each element of the matrix, the variable type declared in the assignment function must be the same as the variable type used when the matrix is defined, that is, the variable types in line 1 and line 6 of Code Listing 2-14 must be the same. If line 6 is changed toc.at<double>the program will report an error and cannot perform the assignment.

(4) Class method assignment

In the Mat class, methods for quick assignment are provided, which can initialize a specified matrix. For example, generate an identity matrix, a diagonal matrix, a matrix with all elements equal to 0 or 1, etc. The specific usage is shown in Code Listing 2-15.

Code Listing 2-15 Example of Assignment Using Class Methods

cv::Mat a = cv::Mat::eye(3, 3, CV_8UC1);
cv::Mat b = (cv::Mat_<int>(1, 3) << 1, 2, 3);
cv::Mat c = cv::Mat::diag(b);
cv::Mat d = cv::Mat::ones(3, 3, CV_8UC1);
cv::Mat e = cv::Mat::zeros(4, 2, CV_8UC3);

The role of each function in the above code and the meaning of its parameters are described as follows:

• eye(): Constructs an identity matrix. The first two parameters are the number of rows and columns of the matrix, and the third parameter is the data type and number of channels in which the matrix is stored. If the number of rows and columns is not equal, then 1 is placed at the main diagonal positions (1,1), (2,2), (3,3), etc.
• diag(): Constructs a diagonal matrix. Its parameter must be a one-dimensional variable of type Mat, used to store the values of the diagonal elements.
• ones(): Constructs a matrix all of whose elements are 1. The meanings of the parameters are the same as for eye().
• zeros(): Constructs a matrix all of whose elements are 0. The meanings of the parameters are the same as for eye().

(5) Assign values using an array

This method is similar to the enumeration method, but it can change the number of channels of the Mat class matrix according to requirements and can be regarded as an extension of the enumeration method. The assignment form of this method is given in code listing 2-16.

Code Listing 2-16 Example of Assignment Using an Array

float a[8] = {5, 6, 7, 8, 1, 2, 3, 4};
cv::Mat b = cv::Mat(2, 2, CV_32FC2, a);
cv::Mat c = cv::Mat(2, 4, CV_32FC1, a);

This assignment method first stores the variables to be stored in the Mat class into an array, and then splits the array variables into a matrix by setting the size and number of channels of the Mat class matrix. This splitting method can freely define the number of channels of the matrix. When the number of elements in the matrix is greater than the data in the array, 1.0737418e+08 will be used to fill and assign to the matrix; when the number of elements in the matrix is less than the data in the array, after the matrix assignment is completed, the remaining data in the array will no longer be assigned. The process of assigning from an array to a matrix is to first assign all channels of the first element in the matrix in sequence, and then assign the next element. To better understand this process, the b and c matrices we defined are given in Figure 2-4.

1.3.Operations supported by the Mat class

When processing data, it is necessary to perform addition, subtraction, multiplication, and division operations on the data. For example, operations such as filtering and enhancing images require addition, subtraction, multiplication, and division operations at the pixel level. To facilitate calculations, Mat class variables support addition, subtraction, multiplication, and division operations on matrices; that is, when we use Mat class variables, we can simply treat them as ordinary matrices. For example, multiplication of a Mat class variable by a constant follows the rules for multiplying a matrix by a constant. When performing operations between a Mat class and a constant, they can be implemented directly using the addition, subtraction, multiplication, and division symbols. Code Listing 2-17 provides an example program for performing addition, subtraction, multiplication, and division operations between Mat class variables and constants.

Code Listing 2-17 Addition, subtraction, multiplication, and division operations of the Mat class

cv::Mat a = (cv::Mat_<int>(3, 3) << 1, 2, 3, 4, 5, 6, 7, 8, 9);
cv::Mat b = (cv::Mat_<int>(3, 3) << 1, 2, 3, 4, 5, 6, 7, 8, 9);
cv::Mat c = (cv::Mat_<double>(3, 3) << 1.0, 2.1, 3.2, 4.0, 5.1, 6.2, 2, 2, 2);
cv::Mat d = (cv::Mat_<double>(3, 3) << 1.0, 2.1, 3.2, 4.0, 5.1, 6.2, 2, 2, 2);
cv::Mat e, f, g, h, i;
e = a + b;
f = c - d;
g = 2 * a;
h = d / 2.0;
i = a - 1;

It should be noted here that when performing addition and subtraction operations on two Mat class variables, the data types in the two matrices must be the same, that is, two Mat class variables that store int and double data types respectively cannot undergo addition and subtraction operations. Unlike conventional multiplication and division, the data type of the result of an operation between a constant and a Mat class variable retains the data type of the Mat class variable. For example, when a double-type constant is operated on with an int-type Mat class variable, the final result is still of int type. In the last line of code in code listing 2-17, subtracting a constant from a Mat class variable means that this constant must be subtracted from every element in the Mat class variable.

When performing convolution operations on images, two matrices need to be multiplied. OpenCV not only provides multiplication operations for two Mat class matrices, but also defines the inner product of two matrices and element-wise multiplication operations. Code listing 2-18 provides a code implementation for multiplication of two Mat class matrices.

Code Listing 2-18 Multiplication of Two Mat-Class Matrices

cv::Mat j, m;
double k;
j = c * d;  //乘法
k = a.dot(b);  //内积
m = a.mul(b);   //对位乘法

The matrix definition and assignment in Code Listing 2-18 are the same as in Code Listing 2-17. In the code, two Mat class variables and one double variable are defined, implementing respectively the multiplication, inner product, and element-wise multiplication of two Mat class matrices.

The "" operator in line 3 represents the mathematical product of two matrices. For example, given two matrices A₃ₓ₃ and B₃ₓ₃, the result of the "" operation is a matrix C₃ₓ₃, and each element in C₃ₓ₃ is expressed as:

需要注意的是，"*"运算要求第一个Mat类矩阵的列数必须与第二个Mat类矩阵的行数相同，而且该运算要求Mat类中的数据类型必须是CV_32FC1、CV_64FC1、CV_32FC2、CV_64FC2这4种中的一种，也就是对于一个二维的Mat类矩阵，其保存的数据类型必须是float类型或者double类型。 代码清单2-18中的第4行代码表示两个Mat类矩阵的内积。根据输出结果可以知道dot()方法结果是一个double类型的变量，该运算的目的是求取一个行向量和一个列向量点乘，例如存在两个向量d = [d₁ d₂ d₃]和e = [e₁ e₂ e₃]，经过dot()方法运算的结果为：

It should be noted that the two input Mat matrices must have the same number of elements, but regardless of the dimensions of the two input Mat matrices, the two Mat matrices will be expanded into a row vector and a column vector, so the result of the dot() operation is always a variable of type double.

Line 5 in code listing 2-18 represents the element-wise product of two Mat class matrices. According to the output results, it can be seen that the result of the mul() method operation is also a Mat class matrix. For two matrices A₃ₓ₃ and B₃ₓ₃, each element in the result C₃ₓ₃ after the mul() method operation can be expressed as:

It should be noted that, unlike the previous two multiplication operations, the data stored in the two Mat class matrices involved in the mul() method operation can be of any type as long as they are guaranteed to be the same, and the default output data type is consistent with the two Mat class matrices. In the field of image processing, the commonly used data type is CV_8U, whose range is 0 to 255. When two relatively large integers are multiplied, result overflow occurs, and the output result is 255. Therefore, when using the mul() method, it is necessary to prevent data overflow problems.

1.4.Reading of Mat-type elements

For reading and modifying Mat class matrices, we have already introduced in the loop assignment of matrices how to use the at method to assign values to each element of the matrix. This is only one of the multiple ways provided by OpenCV to read matrix elements. This section will introduce in detail how to read elements in a Mat class matrix and modify their values.

Before learning how to read elements of a Mat class matrix, it is first necessary to understand how Mat class variables are stored in a computer. A multi-channel Mat class matrix is similar to three-dimensional data, whereas a computer's storage space is a two-dimensional space. Therefore, when a Mat class matrix is stored in a computer, the three-dimensional data is converted into two-dimensional data: first the data of each channel of the first element is stored, and then the data of each channel of the second element is stored. The elements of each row are stored in this manner. Therefore, if we find the starting position of each element, we can find the data of each channel in that element. Figure 2-5 shows the storage method of a three-channel matrix, in which the consecutive blue, green, and red squares respectively represent the 3 channels of each element.

After understanding how Mat class variables are stored, let us look at the properties of the Mat class. Table 2-2 lists the commonly used properties of Mat class matrices and provides a detailed introduction to the role of each property.

Table 2-2 Common Properties of the Mat Class Matrix

By combining these attributes with one another, most attributes of Mat class matrices can be obtained. For example, by combining the step attribute with the cols attribute, the number of bytes occupied by each element can be determined. Then, by further combining this with the channels() attribute, the number of bytes per channel can be known, and thereby the type of data volume stored in the matrix can be understood. The use of each attribute is illustrated in detail with an example below: define a matrix using Mat(3, 4, CV_32FC3). In this case, the number of channels channels() is 3; the number of columns cols is 4; the number of rows rows is 3; the number of elements in the matrix is 3 × 4, which equals 12; the number of bytes per element is 32/8 × channels(), and in this example the final result is 12; the effective length step in bytes is elemSize() × cols, and in this example the result is 48.

Common methods for reading elements of the Mat class matrix include reading via the at method, reading via the pointer ptr, reading via iterators, and reading via address-based location of matrix elements. The following will describe these four reading methods in detail.

1. Read elements in a Mat class matrix using the at method

Reading matrix elements via the at method is divided into methods for single-channel matrices and methods for multi-channel matrices. Code Listing 2-19 provides code for reading single-channel matrix elements via the at method. Code Listing 2-19 Reading single-channel matrix elements of the Mat class via the at method

cv::Mat a = (cv::Mat_<uchar>(3, 3) << 1, 2, 3, 4, 5, 6, 7, 8, 9);
int value = (int)a.at<uchar>(0, 0);

To read an element via the at method, you need to append "<data type>". If the data type here differs from the data type used when the matrix was defined, an error message will occur due to the data type mismatch. This method specifies the coordinates of the element to be read in the form of (row number, column number). It should be noted that if the matrix is defined with uchar-type data, then when data needs to be input, it must be cast to int-type data for output; otherwise, the output result will not be an integer.

Since a single-channel image is a two-dimensional matrix, the corresponding element can be accessed by providing two-dimensional plane coordinates at the end of the at method. However, each element coordinate in a multi-channel matrix contains multiple data values, so a variable is introduced to represent the multiple data values of the same element. In OpenCV, for three-channel matrices, six types—cv::Vec3b, cv::Vec3s, cv::Vec3w, cv::Vec3d, cv::Vec3f, and cv::Vec3i—are defined to represent the 3-channel data of the same element. From these six data types, their naming rule can be summarized: the number indicates the number of channels, and the last character is an abbreviation of the data type—b is the abbreviation for uchar, s for short, w for ushort, d for double, f for float, and i for int. Of course, OpenCV also defines corresponding variable types for two-channel and four-channel matrices, and their naming follows the same rule; for example, the uchar types for two-channel and four-channel matrices are represented by cv::Vec2b and cv::Vec4b, respectively. Code List 2-20 provides the implementation code for reading a multi-channel matrix via the at method.

Code Listing 2-20 Using the at method to read elements of a multi-channel Mat matrix

cv::Mat b(3, 4, CV_8UC3, cv::Scalar(0, 0, 1));
cv::Vec3b vc3 = b.at<cv::Vec3b>(0, 0);
int first = (int)vc3.val[0];
int second = (int)vc3.val[1];
int third = (int)vc3.val[2];

When using the multi-channel variable type, it is likewise necessary to ensure that the data variable type in the at method corresponds to the data variable type of the matrix, and the cv::Vec3b type requires casting its variable type to int when inputting the data of each channel. However, if the data read by the at method is directly assigned to a cv::Vec3i type variable, then it is not necessary to perform data type casting when outputting the data of each channel.

2. Read the elements in the Mat class matrix through the pointer ptr

We previously analyzed how Mat class matrices are stored in memory. Each element in each row of the matrix is stored contiguously. If the starting address of each row's elements is found, then when reading elements at different positions in each row of the matrix, it suffices to move the pointer backward by several positions from the starting position. In code listing 2-21, the code implementation for reading Mat class matrix elements via the pointer ptr is given. Code Listing 2-21 Reading Mat class matrix elements via the pointer ptr

cv::Mat b(3, 4, CV_8UC3, cv::Scalar(0, 0, 1));
for (int i = 0; i < b.rows; i++)
{
    uchar* ptr = b.ptr<uchar>(i);
    for (int j = 0; j < b.cols * b.channels(); j++)
    {
        cout << (int)ptr[j] << endl;
    }
}

在程序里，首先有一个大循环用来控制矩阵中每一行，之后定义一个uchar类型的指针ptr，在定义时需要声明Mat类矩阵的变量类型，并在定义最后用小括号声明指针指向Mat类矩阵的哪一行。第二个循环控制用于输出矩阵中每一行所有通道的数据。根据图2-5中所示的存储形式，每一行中存储的数据数量为列数与通道数的乘积，即指针可以向后移动cols×channels()位，如第7行代码所示，指针向后移动的位数在中括号给出。程序中给出了循环遍历Mat类矩阵中的每一个数据的方法，当我们能够确定需要访问的数据时，可以直接通过给出行数和指针后移的位数进行访问，例如，当读取第2行数据中第3个数据时，可以直接通过ptr[2]访问。

3. Accessing Elements in a Mat Class Matrix Through Iterators

A Mat class variable is also a container variable; therefore, a Mat class variable has iterators for accessing the data in the Mat class variable. Through the iterators, traversal of each element in the matrix can be achieved. The code implementation is given in Code Listing 2-22. Code Listing 2-22 Reading Mat class matrix elements through iterators

cv::MatIterator_<uchar> it = a.begin<uchar>();
cv::MatIterator_<uchar> it_end = a.end<uchar>();
for (int i = 0; it != it_end; it++)
{
    cout << (int)(*it) << " ";
    if ((++i % a.cols) == 0)
    {
        cout << endl;
    }
}

The iterator variable type of the Mat class is cv::MatIterator_<>, and when defining it, the data variable type also needs to be declared within the angle brackets. The start of the Mat class iterator is Mat.begin<>(), and the end is Mat.end<>(). Its usage is the same as other iterators: the "++" operation is used to iterate the pointer position downward. The data reading method is to first read each channel of the first element, and then read each channel of the second element, until the last channel of the last element.

4. Access elements through address-based matrix element indexing

The previous three ways of reading elements all require knowing the data type stored in the Mat class matrix. Moreover, conceptually, we would prefer to be able to read data within a certain channel by declaring "the X-th row, the X-th column, the X-th channel". Code Listing 2-23 presents exactly this way of reading data.

Code Listing 2-23 Accessing Elements by Locating the Addresses of Matrix Elements

(int)(*(b.data + b.step[0] * row + b.step[1] * col + channel));

In the code, the row variable means the row number of the element where a certain data item is located, the col variable means the column number of the element where a certain data item is located, and the channel variable means the channel number of the element where a certain data item is located. This approach is similar to the way we read data through pointers, in that both point to the data to be read by moving the address pointer of the first data item by a certain number of positions; however, this approach can perform reading by directly specifying the row, column, and channel numbers, and does not require the user to calculate the position of a certain data item within the storage space of this row of data.

2.Image reading and display

This section will introduce in detail the relevant functions of image reading and display.

2.1.Image reading function imread

We have already introduced the calling method of the image reading function imread() earlier (see Code Listing 1-1). Here we provide the function prototype (see Code Listing 2-24). Code Listing 2-24 Prototype of the imread() function

cv::Mat cv::imread(const String &filename,
                   int flags = IMREAD_COLOR);

• filename: The name of the image file to be read, including the image path, name, and image file extension.
• flags: Flags for the image reading mode, such as reading a color image as a grayscale image. The default parameter reads the image in color format. The optional parameters are given in Table 2-3.

The function is used to read the specified image and return it to a Mat class variable. When the image file does not exist, is corrupted, or is in an unsupported format, the image cannot be read. In this case, the function returns an empty matrix. Therefore, whether the image is read successfully can be determined by checking whether the data attribute of the returned matrix is empty or whether the empty() function returns true. If reading the image fails, the data attribute returns 0, and the empty() function returns 1.

The function can read image files in multiple formats, but due to differences in the codecs used across operating systems, an image file that can be read on one system may not be readable on another. Regardless of the system, BMP files and DIB files can always be read. On Windows and macOS systems, the codecs included with OpenCV (libjpeg, libpng, libtiff, and libjasper) are used by default, so JPEG (jpg, jpeg, jpe), PNG, and TIFF (tiff, tif) files can be read. On Linux systems, these codecs need to be installed separately, and after installation, these types of files can also be read.

However, it should be noted that whether the function can read file data is independent of the file extension; instead, it determines the image type based on the file's content. For example, if a file's extension is changed from png to exe, the function can still read the image, but if the extension is changed from exe to png, the function cannot load the file. The first parameter of this function specifies the path of the image to be read as a string, and the second parameter sets the mode for reading the image. The default parameter reads the image as a color image, and the parameter can be changed according to different requirements. In OpenCV 4.1, 13 modes for reading images are provided, which can be summarized as reading in the original style, reading as a grayscale image, reading as a color image, reading with multiple bits, and reducing the image to a certain size when reading, among others. The specific selectable parameters and their effects are shown in Table 2-3. It should be pointed out here that converting a color image to a grayscale image through internal codec conversion may differ from the result of converting a color image to a grayscale image in an OpenCV program. These flag parameters can be declared simultaneously in multiple instances as long as their functions do not conflict, with different parameters separated by "|".

Table 2-3 Formal parameters of the imread() function for reading images

Note: By default, the number of pixels in an image to be read must be less than 2³⁰. This requirement is not affected in the vast majority of image processing fields, but the number of pixels in satellite remote sensing images and ultra-high-resolution images may exceed this threshold. The maximum number of pixels that can be read can be adjusted by modifying the OPENCV_IO_MAX_IMAGE_PIXELS parameter in the system variables.

2.2.Image window function namedWindow

In our previous programs, we have not introduced window functions, because when displaying an image, if no image window is actively defined, the program will automatically generate a window for displaying the image. However, sometimes it is necessary to operate on the image window before displaying the image, for example, adding a slider, and in this case, the image window needs to be created in advance. Code Listing 2-25 gives the prototype of the window creation function. Code Listing 2-25 Prototype of the namedWindow() function

void cv::namedWindow(const String &winname,
                     int flags = WINDOW_AUTOSIZE);

• winname: Window name, used as the identifier for the window.
• flags: Window attribute setting flags.

This function creates a window variable used to display images and sliders. The window is referenced by its name. If a window with the same name already exists when the window is being created, the function performs no operation. Creating a window consumes some memory resources; therefore, after a window is created by this function, the window needs to be closed to release memory resources when it is no longer needed. OpenCV provides two functions for closing window resources, namely the cv::destroyWindow() function and cv::destroyAllWindows(). As can be understood from their names, the former function is used to close a window with a specified name, that is, entering the string of the window name within the parentheses closes the corresponding window; the latter function closes all windows in the program and is generally used at the end of the program.

However, in fact, in a simple program, we do not need to call these functions, because when the program exits, all resources and windows of the application are automatically closed. Although not actively releasing windows will also release window resources when the program ends, OpenCV 4.0 will report an error about windows not being released at the end, whereas OpenCV 4.1 will not report an error.

The first parameter of this function is the name of the declared window, used for unique identification of the window. The second parameter is the attribute of the declared window, mainly used to set whether the window size is adjustable, whether the displayed image fills the window, etc. The specific selectable parameters and their meanings are given in Table 2-4. By default, the flag parameters loaded by the function are WINDOW_AUTOSIZE | WINDOW_KEEPRATIO | WINDOW_GUI_EXPANDED.

Table 2-4 namedWindow() Function Window Property Flag Parameters

2.3.Image display function imshow

We have already introduced the calling convention of the image display function imshow() earlier. Here we give the function prototype (see Code Listing 2-26). Code Listing 2-26 Prototype of the imshow() function

void cv::imshow(const String &winname,
                InputArray mat);

• winname: Name of the window where the image is to be displayed, assigned as a string.
• mat: Image matrix to be displayed.

This function displays an image in the specified window. If an image window with the same name has not been created before this function, it creates a window with the WINDOW_AUTOSIZE flag and displays the image at its original size; if an image window has been created, it scales the image to fit the window properties. This function scales the image according to its depth, and the specific scaling rules are as follows:

• 如果图像是8位无符号类型，那么按照原样显示。
• 如果图像是16位无符号类型或者32位整数类型，那么会将像素除以256，将范围由[0,255×256]映射到[0,255]。
• 如果图像是32位或64位浮点类型，那么将像素乘以255，即将范围由[0,1]映射到[0,255]。

The first parameter in the function is the name of the image display window, and the second parameter is the Mat class matrix of the image to be displayed. A special note is needed here: we can see that the second parameter is not the commonly seen Mat class, but InputArray. This is a type declaration reference defined by OpenCV, used as an identifier for input parameters. When we encounter it, we can consider it as requiring a Mat class data as input. Similarly, OpenCV also defines the OutputArray type for output, and we can likewise consider it as outputting a Mat class data.

After this function runs, the program continues executing subsequent code. If the subsequent code finishes and exits directly, the displayed image may flash and disappear. Therefore, in programs that need to display images, the imshow() function is often followed by the cv::waitKey() function to pause the program for a period of time. The waitKey() function specifies the waiting duration in milliseconds. If the parameter is default or "0", it means waiting for the user to press a key to end the function.

Code to display an image:

#include "chapter2_2_show_image/inc/show_image.hpp"
#include <iostream>
#include <opencv2/opencv.hpp>


void opencv_function1(void)
{
    cv::Mat picture_demo_mat = cv::imread(std::string(MEDIA_PATH) + "林星阑L.jpg");
    cv::imshow("xiaoshen", picture_demo_mat);

    std::cout << "成功运行OpenCV!" << std::endl; 

    cv::waitKey(0);   			// 这句确保窗口一直打开
}

Code to display an image (using NVIDIA GPU CUDA acceleration):

#include "chapter2_2_show_image/inc/show_image_CUDA.hpp"
#include <iostream>
#include <opencv2/opencv.hpp>

void opencv_function2(void)
{
    cv::Mat picture_demo_mat = cv::imread(std::string(MEDIA_PATH) + "林星阑H.jpg");
    cv::cuda::GpuMat gpuImage; 
    gpuImage.upload(picture_demo_mat); 
    cv::Mat result; 
    gpuImage.download(result);
    cv::namedWindow("林星阑",cv::WINDOW_NORMAL);
    cv::imshow("林星阑", result);
    std::cout << "CUDA成功运行!" << std::endl;
    cv::waitKey(0);   			// 这句确保窗口一直打开

}

3.Video loading and camera invocation

Previously, we introduced how to read image data programmatically. This section will introduce the VideoCapture class in OpenCV, which is designed for reading video files and accessing cameras.

3.1.Reading of video data

Although a video file is composed of multiple images, the imread() function cannot directly read video files. A dedicated video reading function is required to read the video and save each frame image into a Mat class matrix. Code listing 2-27 shows the constructor of the VideoCapture class when reading a video file. Code listing 2-27 Constructor of the VideoCapture class for reading video files

cv::VideoCapture::VideoCapture();  // 默认构造函数

cv::VideoCapture::VideoCapture(const String& filename,
                               int apiPreference = CAP_ANY);

• filename: Name of the video file or image sequence to be read.
• apiPreference: Properties set when reading data, such as encoding format, whether to call OpenNI, etc.

This function constructs a video stream capable of reading and processing video files. The first line in code listing 2-27 is the default constructor of the VideoCapture class, which merely declares a class capable of reading video data. Which specific video file to read must be specified using the open() function when in use, for example, cap.open("1.avi") is the VideoCapture class variable cap reading the 1.avi video file.

The second constructor assigns the video data to the variable while declaring the variable. The types of files that can be read include video files (e.g., video.avi), image sequences, or URLs of video streams. Reading an image sequence requires unifying the names of multiple images into the form "prefix + number", invoked in the form "prefix+%02d"; for example, when a folder contains images img_00.jpg, img_01.jpg, img_02.jpg to be loaded, the filenames are represented as img_%02d.jpg. The property-setting tag for reading video in the function defaults to automatically searching for a suitable flag; therefore, in ordinary use, it can be left at its default, with only the video name entered. As with the imread() function, the constructor may also fail to read the file, so it must be checked using the isOpened() function. If the read succeeds, the return value is true; if the read fails, the return value is false. The constructor only loads the video file into the VideoCapture class variable. When we need to use the images in the video, we also need to export the images from the VideoCapture class variable to a Mat class variable for subsequent data processing. This operation can be performed using the ">>" operator to assign images from the VideoCapture class variable to the Mat class variable in video order. When all images in the VideoCapture class variable have been assigned to the Mat class variable, the Mat class variable will become an empty matrix on the next assignment; therefore, empty() can be used to determine whether all images in the VideoCapture class variable have been read.

The VideoCapture class variable also provides a get() function that can be used to view video properties. By inputting specified flags, video properties such as the pixel dimensions, frame count, and frame rate of the video can be obtained. The commonly used flags and their meanings in the get() method of the VideoCapture class are given in Table 2-5.

Table 2-5 Flag parameters in the get() method of the VideoCapture class

To become more familiar with the VideoCapture class, Code Listing 2-28 provides a program that reads a video, outputs the video properties, and displays the video at the original frame rate. The running result is shown in Figure 2-6.

Listing 2-28 VideoCapture.cpp Reading a video file

#include "chapter2_3_video_capture/inc/read_video.hpp"
#include <cstdio>
#include <opencv2/opencv.hpp>


int opencv_function3(void)
{
    cv::VideoCapture video__(std::string(MEDIA_PATH) + "hei.mp4");
    if(video__.isOpened() == true)  //判断视频是否导入成功
    {
        printf("视频中图像的宽度=%lf",video__.get(cv::CAP_PROP_FRAME_WIDTH));
        printf("视频中图像的高度=%lf",video__.get(cv::CAP_PROP_FRAME_HEIGHT));
        printf("视频的帧率=%lf",video__.get(cv::CAP_PROP_FPS));
        printf("视频的总帧数=%lf",video__.get(cv::CAP_PROP_FRAME_COUNT));
    }
    else
    {
        printf("导入视频失败，请确认视频文件是否正确");
        return 1;
    }

    while(true)
    {
        cv::Mat frame__;
        video__ >> frame__;
        if(frame__.empty() == true) //检测图像是不是空的，如果是空的，说明视频最后一帧也已经导入完了
        {
        break;
        }
        cv::imshow("视频播放",frame__);
        cv::waitKey(1000 / video__.get(cv::CAP_PROP_FPS));   //FPS为1秒每帧（也就是帧的数量），用1000ms / 帧的数量 = 每帧所需时间
    }
    cv::waitKey(0);   			// 这句确保窗口视频播放完后一直打开，不关闭
}

Figure 2-6 Result of running the video reading program

3.2.Direct invocation of the camera

The VideoCapture class can also access a camera, with a constructor as shown in Code Listing 2-29.

Listing 2-29: VideoCapture class camera constructor call

cv::VideoCapture::VideoCapture(int index,
                               int apiPreference = CAP_ANY);

By comparing with Code Listing 2-27, calling the camera differs from reading a video file only in the first parameter. When calling the camera, the first parameter is the ID of the camera device to be opened, with IDs starting from 0. The method for reading image data from the camera is the same as that for reading image data from a video: the image captured by the camera at the current moment is read using the ">>" symbol. Moreover, when reading a video, the properties of the VideoCapture class can also be used. We change the video file in Code Listing 2-28 to the camera ID (0), run the modified program in Code Listing 2-28 again, and the result is shown in Figure 2-7.

Linux query camera ID:

ls /dev/video*

Then output

/dev/video0  /dev/video1  /dev/video2  /dev/video3

The four camera devices here are not necessarily all real cameras. On my side, 0 is the laptop's built-in camera, while 2 is my external USB camera. Just try them all.

Figure 2-7 Running result of the camera invocation program

4.Data Storage

During image processing, new images are generated, for example, making a blurry image clearer through algorithms, converting a color image into a grayscale image, and at the same time, the processing results need to be saved as files in the form of images or videos. This section will describe in detail how to save a Mat class matrix as an image or video file.

4.1.Image Saving

OpenCV provides the imwrite() function for saving Mat class matrices as image files. The prototype of this function is given in Code Listing 2-30.

Code Listing 2-30 imwrite() function prototype

bool cv::imwrite(const String& filename,
                 InputArray img,
                 const std::vector<int>& params = std::vector<int>());

• filename: The address and filename for saving the image, including the image format.
• img: The Mat class matrix variable to be saved.
• params: Flags for setting image format attributes. This function is used to save a Mat class matrix as an image file. If saved successfully, it returns true; otherwise, it returns false. The image formats that can be saved refer to the image file formats that can be read by the imread() function. Typically, this function can only save 8-bit single-channel images and 3-channel BGR color images, but images in different formats can be saved by changing the third parameter. The number of bits of images that can be saved in different image formats is as follows:
• 16-bit unsigned (CV_16U) images can be saved as PNG, JPEG, and TIFF format files;
• 32-bit floating-point (CV_32F) images can be saved as PFM, TIFF, OpenEXR, and Radiance HDR format files;
• 4-channel (Alpha channel) images can be saved as PNG format files. The third parameter of this function generally does not need to be filled in. To save as a specified file format, you only need to directly change the file extension after the first parameter. However, when the data in the Mat class matrix to be saved is relatively special (such as 16-bit depth data), the third parameter needs to be set. The setting method of the third parameter is shown in code listing 2-31, and common optional setting flags are given in Table 2-6.

Code Listing 2-31 Setting method of the third parameter in the imwrite() function

vector<int> compression_params;
compression_params.push_back(IMWRITE_PNG_COMPRESSION);
compression_params.push_back(9);
imwrite(filename, img, compression_params);

Table 2-6 Flags selectable for the third parameter of the imwrite() function and their effects

To better understand how to use the imwrite() function, Code Listing 2-32 provides a program that generates a matrix with an Alpha channel and saves it as a PNG format image. After the program runs, it generates a PNG format image that stores 4 channels. To see the image result more intuitively, we present in Figure 2-8 the image seen in the Image Watch plugin and the image saved in PNG format.

Code Listing 2-32 imgWriter.cpp Save Image

#include "chapter2_4_save_media_file/inc/save_image.hpp"
#include <cstdio>
#include <opencv2/opencv.hpp>


void AlphaMat(cv::Mat *mat);

int opencv_function6(void)
{
    cv::Mat mat__(480,640,CV_8UC4);
    AlphaMat(&mat__);
    std::vector<int> compression_params;
    compression_params.push_back(cv::IMWRITE_PNG_COMPRESSION);   //PNG图像压缩标志
    compression_params.push_back(9);   //设置最高压缩质量
    if(cv::imwrite(std::string(SRCSRC_PATH) + "chapter2_4_save_media_file/save_files/save_image_test1.png",mat__,compression_params) == false)
    {
        printf("保存成PNG图像失败");
        return 1;
    }
    printf("保存成PNG图像成功");
    return 0;
}

void AlphaMat(cv::Mat *mat)
{
  CV_Assert(mat->channels() == 4);  //如果通道不等于4，那么抛出异常
  for(int i = 0;i < mat->rows;i++)   //行
  {
    for(int j = 0;j < mat->cols;j++)    //列
    {
      /*
      cv::Vec4b bgra = mat->at<cv::Vec4b>(i,j);
      bgra.val[0] = cv::saturate_cast<uint8_t>(255);    //B:255   //该函数防止溢出，里面的数只能是0-255
      bgra.val[1] = cv::saturate_cast<uint8_t>(0);      //G:0
      bgra.val[2] = cv::saturate_cast<uint8_t>(0);      //R:0
      bgra.val[3] = cv::saturate_cast<uint8_t>(180);    //α:180
      */
      cv::Vec4b &bgra = mat->at<cv::Vec4b>(i,j);
      bgra[0] = cv::saturate_cast<uint8_t>(255);     //B:255   //该函数防止溢出，里面的数只能是0-255
      bgra[1] = cv::saturate_cast<uint8_t>(0);      //G:0   //重载运算符[]
      bgra[2] = cv::saturate_cast<uint8_t>(0);      //R:0
      bgra[3] = cv::saturate_cast<uint8_t>(90);     //α:90
    }
  }
}

Figure 2-8 4-channel image in the program and after saving (left: Image Watch, right: PNG file)

4.2.Saving of Videos

Sometimes we need to generate a video from multiple images, or directly save data captured by a camera as a video file. OpenCV provides the VideoWriter class for saving multiple images as a video file. The prototype of this class's constructor is given in code listing 2-33.

Code Listing 2-33 VideoWriter class constructor for saving video files

cv::VideoWriter::VideoWriter();  // 默认构造函数

cv::VideoWriter::VideoWriter(const String& filename,
                             int fourcc,
                             double fps,
                             Size frameSize,
                             bool isColor = true);

• filename: The address and filename for saving the video, including the video format.
• fourcc: The 4-character codec code for the compressed frames; detailed parameters are given in Table 2-7.
• fps: The frame rate for saving the video, i.e., the number of images per second in the video.
• frameSize: The size of the video frame.
• isColor: Whether the saved video is a color video.

The usage of the first default constructor in Code Listing 2-33 is the same as that of VideoCapture(), both create a data stream for saving video, and subsequently set a series of parameters such as the save file name, codec, frame count, etc. through the open() function. The second constructor requires as its first input parameter the name of the video file to be saved, and as its second parameter the codec code. The codec options that can be set are given in Table 2-7. If "-1" is assigned, an appropriate codec will be automatically searched for. It should be noted that there are some differences in the input method between OpenCV 4.0 and OpenCV 4.1, and the specific differences are given in Table 2-7. The third parameter is the frame rate for saving the video, which can be freely set according to requirements, for example, to achieve double-speed playback of the original video, slow-motion playback of the original video, etc. The fourth parameter is to set the size of the saved video file. It should be noted here that when setting it, it must be the same as the image size, otherwise the video cannot be saved. The last parameter is to set whether the saved video is in color. In the program, the default is to save as a color video.

This function is very similar to VideoCapture(), as both can determine whether a video stream was successfully created through the isOpened() function, and various properties of the video stream can be viewed through get(). When saving a video, we only need to assign the images that generate the video frame by frame to the video stream through the "<<" operator (or the write() function), and finally use release() to close the video stream.

Table 2-7 Video Coding Formats

To better understand how to use the VideoWriter() class, Code Listing 2-34 provides an example routine that uses existing video file data or directly generates a new video file via a camera. Readers should focus on understanding the similarities between the VideoWriter() class and the VideoCapture() class, as well as the precautions when using them.

Listing 2-34 VideoWriter.cpp Save Video File

#include "chapter2_4_save_media_file/inc/save_video.hpp"
#include <cstdio>
#include <opencv2/opencv.hpp>


template <typename T>
bool Save_Video(T file_opened_name_or_index,const std::string & file_saved_name);

int opencv_function5(void)
{
    // 截取视频里的一段视频并保存到本地
    Save_Video<const std::string &>(std::string(MEDIA_PATH) + "hei.mp4",std::string(SRCSRC_PATH) + "chapter2_4_save_media_file/save_files/save_video_test1.mp4");

    // 截取摄像头里的一段视频并保存到本地
    // Save_Video<int>(2,std::string(SRCSRC_PATH) + "chapter2_4_save_media_file/save_files/save_video_test2.mp4");
    
    return 0;
}


template <typename T>
bool Save_Video(T file_opened_name_or_index,const std::string & file_saved_name)
{
    cv::Mat img;
    cv::VideoCapture video(file_opened_name_or_index);
    if(video.isOpened() == true)
    {
        printf("调用摄像头或打开视频成功!");
        printf("视频中图像的宽度=%lf",video.get(cv::CAP_PROP_FRAME_WIDTH));
        printf("视频中图像的高度=%lf",video.get(cv::CAP_PROP_FRAME_HEIGHT));
        printf("视频的帧率=%lf",video.get(cv::CAP_PROP_FPS));
        printf("视频的总帧数=%lf",video.get(cv::CAP_PROP_FRAME_COUNT));
    }
    else
    {
        printf("调用摄像头或者视频失败，请检查摄像头是否连接成功或者视频文件是否存在");
        return false;
    }
    video >> img;
    if(img.empty() == true)
    {
        printf("帧图像获取失败!");
        return false;
    }
    cv::VideoWriter writer;
    auto file_name = file_saved_name;
    int codec = writer.fourcc('M','J','P','G');
    double fps = 30.0;
    auto size = img.size();
    bool isColor = (img.type() == CV_8UC3);
    writer.open(file_name,codec,fps,size,isColor);

    if(writer.isOpened() == true)  //判断视频流是否创建成功!
    {
        printf("视频流创建成功!");
    }
    else
    {
        printf("视频流创建失败，请确认是否为合法输入!");
        return false;
    }

    while(true)
    {
        if(video.read(img) == false)   //判断是否还能够继续从摄像头或者视频中读出一帧图像
        {
            printf("摄像头断开连接或者视频读取完成!");
            break;
        }
        writer << img;  //writer.write(img);
        cv::namedWindow("Live");
        cv::imshow("Live",img);
        int8_t keyborad_value = cv::waitKey(1000 / video.get(cv::CAP_PROP_FPS));   //FPS为1秒每帧（也就是帧的数量），用1000ms / 帧的数量 = 每帧所需时间
        if(keyborad_value == 27)  //ESC键的ASCII码值为27，按下ESC键退出循环
        {
            break;
        }
    }
    video.release();
    writer.release();
    return true;
}

Save image from video:

#include "chapter2_4_save_media_file/inc/save_image_in_video.hpp"
#include <cstdio>
#include <opencv2/opencv.hpp>


int opencv_function7(void)
{
    cv::VideoCapture video__(0);
    cv::Mat frame__;
    if(video__.isOpened() == true)  //判断视频是否导入成功
    {
        printf("视频中图像的宽度=%lf",video__.get(cv::CAP_PROP_FRAME_WIDTH));
        printf("视频中图像的高度=%lf",video__.get(cv::CAP_PROP_FRAME_HEIGHT));
        printf("视频的帧率=%lf",video__.get(cv::CAP_PROP_FPS));
        printf("视频的总帧数=%lf",video__.get(cv::CAP_PROP_FRAME_COUNT));
    }
    else
    {
        printf("导入摄像头视频失败，请确认摄像头是否正常打开");
        return 1;
    }
    printf("按下空格键截图!");
    printf("按下ESC结束播放!");
    while(true)
    {
        video__ >> frame__;
        cv::imshow("Live",frame__);
        int32_t key_boards_val = cv::waitKey(1000/video__.get(cv::CAP_PROP_FPS));
        if(key_boards_val==32)   //检测到按下了空格键
        {
            if(cv::imwrite(std::string(SRCSRC_PATH) + "chapter2_4_save_media_file/save_files/save_image_test2.png",frame__) == false)
            {
                printf("截图失败!");
            }
            else
            {
                printf("截图成功!");
            }
            if(frame__.empty() == true) //检测图像是不是空的，如果是空的，说明视频最后一帧也已经导入完了
            {
                printf("视频播放结束,请选择截图或是按ESC退出!");
            }
        }
        else if(key_boards_val==27)   //检测到按下了ESC按键
        {
            break;
        }
    }
    video__.release();
    printf("播放被终止或已结束!");

    return 0;
}

4.3.Save and read XML and YAML files

In addition to image data, sometimes small-sized Mat class matrices, strings, arrays, and other data in a program also need to be saved. These data are usually saved as XML files or YAML files. This section will introduce how to use functions in OpenCV 4 to save data as XML files or YAML files, and how to read data from these two types of files. XML is a metalanguage. A metalanguage means that users can define their own tags according to their needs. For example, tags such as <age> and <color> can be used to define the meaning of data, such as using <age>24</age> to indicate that the value of the age data is 24. XML is a structured language. Through the XML language, the subordinate relationships between data can be known. For example, <color><red>100</red><blue>150</blue></color> indicates that the color data contains two data items named red and blue, whose values are 100 and 150 respectively. By using tags, no matter in what form the data is saved, as long as the file conforms to the XML format, the data read out will not be confused or ambiguous. YAML is a data-centric language. It represents the value of each data item in the form of "variable: value", and uses different indentation to represent the structure and subordinate relationships between different data. YAML has high readability and is often used to express the format of data sequences. It references multiple languages, including XML, C, Python, Perl, and others. OpenCV 4 provides the FileStorage class for generating and reading XML files and YAML files. The class defines methods such as initializing the class, writing data, and reading data. When using the FileStorage class, we first need to initialize it. Initialization can be understood as declaring the file to be operated on and the operation type. OpenCV 4 provides two initialization methods: initialization without inputting any parameters (which can be understood as only defining, not initializing), and initialization by inputting the file name and operation type. The prototype of the constructor for the latter initialization method is given in code listing 2-35. Code listing 2-35 Prototype of the FileStorage() function

cv::FileStorage::FileStorage(const String &filename,
                             int flags,
                             const String &encoding = String());

• filename: Name of the file to open.
• flags: Flags for the type of operation to perform on the file. Common parameters and their meanings are given in Table 2-8.
• encoding: Encoding format. UTF-16 XML encoding is currently not supported; UTF-8 XML encoding must be used. Table 2-8 Common flags and their meanings for file operation types in the FileStorage() constructor

This function is the constructor of the FileStorage class, used to declare the name of the file to be opened and the type of operation. The first parameter of this function is the name of the file to be opened, which is of string type, and the file extension is ".xml" or ".yaml" (or ".yml"). The file to be opened may or may not already exist; however, when performing a read operation on the file, it must be an existing file. The second parameter is the flag for the type of operation to be performed on the file, such as a read operation or a write operation on the file, etc. The commonly used parameters and their meanings are given in Table 2-8. Since this flag is within the FileStorage class, the class name must be prefixed when using it, for example, "FileStorage::WRITE". The last parameter is the encoding format of the file. Currently, UTF-16 XML encoding is not supported, and UTF-8 XML encoding must be used. Normally, the default value of this parameter can be used.

After opening the file, you can determine whether the file was successfully opened using the isOpened() function in the FileStorage class. If the file is successfully opened, the function returns true; if opening the file fails, the function returns false.

Since the default constructor in the FileStorage class has no parameters, the file to be opened and the type of operation are not declared. In this case, they need to be declared separately through the open() function in the FileStorage class. The prototype of this function is given in Code Listing 2-36.

Code Listing 2-36 Prototype of the open() Function

virtual bool cv::FileStorage::open(const String &filename,
                                    int flags,
                                    const String &encoding = String());

• filename: Name of the file to open.
• flags: Operation type flags for the file; common parameters and their meanings are given in Table 2-8.
• encoding: Encoding format. Currently, UTF-16 XML encoding is not supported; UTF-8 XML encoding is required. This function solves the problem that the default constructor does not declare an opened file; the function can specify the file opened by the FileStorage class. If the file is opened successfully, the return value is true; otherwise, it is false. All parameters and their meanings in this function are the same as those in Code Listing 2-35, so they are not repeated here. Similarly, after opening a file through this function, whether the file was opened successfully can still be determined through the isOpened() function in the FileStorage class. After opening the file, similar to the data stream created in C++, data can be written to the file through the "<<" operator, or data can be read from the file through the ">>" operator. In addition, data can also be written to the file through the write() function in the FileStorage class; the prototype of this function is given in Code Listing 2-37. Code Listing 2-37 Prototype of the write() function

void cv::FileStorage::write(const String &name,
                            int val);

• name: Name of the variable written to the file.
• val: Variable value.

This function can write variable names and variable values of different data types to a file. The first parameter of this function is the variable name written to the file. The second parameter is the variable value. In code listing 2-37, the variable value is of type int, but the FileStorage class provides multiple overloaded write() functions, respectively used to implement writingdouble、String、Mat、vector<String>

使用操作符向文件中写入数据时与write()函数类似，都需要声明变量名和变量值，例如变量名为"age"，变量值为"24"，可以通过file << "age" << 24来实现。如果某个变量的数据是一个数组，可以用"[]"将属于同一个变量的变量值标记出来，例如file << "age" << "[" << 24 << 25 << "]"。如果某些变量隶属于某个变量，可以用"{}"表示变量之间的隶属关系，例如file << "age" << "{" << "Xiaoming" << 24 << "Wanghua" << 25 << "}"。

读取数据时，可以通过file["x"] >> xRead读取变量名为x的变量值。但是，当某个变量中含有多个数据或者含有子变量时，就需要通过FileNode节点类型和迭代器FileNodeIterator进行读取，例如某个变量的变量值是一个数组，首先需要定义一个形如file["age"]的FileNode节点类型变量，之后通过迭代器遍历其中的数据。另一种方法可以不使用迭代器，通过在变量后边添加"[]"（地址）的形式读取数据，例如FileNode[0]表示数组变量中的第一个数据，FileNode["Xiaoming"]表示"age"变量中的"Xiaoming"变量的数据，依次向后添加"[]"（地址）实现多节点数据的读取。

为了了解如何生成和读取XML文件和YAML文件，在代码清单2-38中给出了实现文件写入和读取的示例程序。该程序中使用write()函数和"<<"操作符两种方式向文件中写入数据，使用迭代器和"[]"（地址）两种方式从文件中读取数据。数据的写入和读取方法在前面已经介绍，在代码清单2-38中需要重点了解如何通过程序实现写入与读取。该程序生成的XML文件和YAML文件中的数据在图2-9中给出，读取文件数据的结果在图2-10中给出。

Code Listing 2-38 myXMLandYAML.cpp Saving and Reading XML and YAML Files

#include "chapter2_4_save_media_file/inc/save_XMLandYMAL.hpp"
#include <cstdio>
#include <opencv2/opencv.hpp>

using namespace std;
using namespace cv;

int opencv_function8(void)
{
system("color F0");  //修改运行程序背景和文字颜色
	// string fileName = std::string(SRCSRC_PATH) + "chapter2_4_save_media_file/save_files/datas.xml";  //文件的名称
	string fileName = std::string(SRCSRC_PATH) + "chapter2_4_save_media_file/save_files/datas.yaml"; //文件的名称
	//以写入的模式打开文件
	cv::FileStorage fwrite(fileName, cv::FileStorage::WRITE);
	
	//存入矩阵Mat类型的数据
	Mat mat = Mat::eye(3, 3, CV_8U);
	fwrite.write("mat", mat);  //使用write()函数写入数据
	//存入浮点型数据，节点名称为x
	float x = 100;
	fwrite << "x" << x;
	//存入字符串型数据，节点名称为str
	String str = "Learn OpenCV 4";
	fwrite << "str" << str;
	//存入数组,节点名称为number_array
	fwrite << "number_array" << "[" <<4<<5<<6<< "]";
	//存入多node节点数据,主名称为multi_nodes
	fwrite << "multi_nodes" << "{" << "month" << 8 << "day" << 28 << "year"
		<< 2019 << "time" << "[" << 0 << 1 << 2 << 3 << "]" << "}";

	//关闭文件
	fwrite.release();

	//以读取的模式打开文件
	cv::FileStorage fread(fileName, cv::FileStorage::READ);
	//判断是否成功打开文件
	if (!fread.isOpened())
	{
		cout << "打开文件失败，请确认文件名称是否正确！" << endl;
		return -1;
	}

	//读取文件中的数据
	float xRead;
	fread["x"] >> xRead;  //读取浮点型数据
	cout << "x=" << xRead << endl;

	//读取字符串数据
	string strRead;
	fread["str"] >> strRead;
	cout << "str=" << strRead << endl;

	//读取含多个数据的number_array节点
	FileNode fileNode = fread["number_array"];
	cout << "number_array=[";
	//循环遍历每个数据
	for (FileNodeIterator i = fileNode.begin(); i != fileNode.end(); i++)
	{
		float a;
		*i >> a;
		cout << a<<" ";
	}
	cout << "]" << endl;

	//读取Mat类型数据
	Mat matRead;
	fread["mat="] >> matRead;
	cout << "mat=" << mat << endl;

	//读取含有多个子节点的节点数据，不使用FileNode和迭代器进行读取
	FileNode fileNode1 = fread["multi_nodes"];
	int month = (int)fileNode1["month"];
	int day = (int)fileNode1["day"];
	int year = (int)fileNode1["year"];
	cout << "multi_nodes:" << endl 
		<< "  month=" << month << "  day=" << day << "  year=" << year;
	cout << "  time=[";
	for (int i = 0; i < 4; i++)
	{
		int a = (int)fileNode1["time"][i];
		cout << a << " ";
	}
	cout << "]" << endl;
	
	//关闭文件
	fread.release();
	return 0;
}

Figure 2-9 XML file and YAML file generated by the myXMLandYAML.cpp program

Figure 2-10 myXMLandYAML.cpp Program file reading result

5.Chapter Summary

In this chapter, we first introduce how to use the Mat class in OpenCV 4 for storing image data, then cover image reading and display, video loading and camera invocation, and finally explain how to save image and video files, as well as the saving and reading of XML and YAML files.

List of main functions in this chapter