Image registration is the process of aligning two or more images of the same scene. Typically, one image, called the base image or reference image, is considered the reference to which the other images, called input images, are compared. The object of image registration is to bring the input image into alignment with the base image by applying a spatial transformation to the input image. The differences between the input image and the output image might have occurred as a result of terrain relief and other changes in perspective when imaging the same scene from different viewpoints. Lens and other internal sensor distortions, or differences between sensors and sensor types, can also cause distortion.
A spatial transformation maps locations in one image to new locations in another image. (For more details, see Spatial Transformations) Determining the parameters of the spatial transformation needed to bring the images into alignment is key to the image registration process.
Image registration is often used as a preliminary step in other image processing applications. For example, you can use image registration to align satellite images of the earth's surface or images created by different medical diagnostic modalities (MRI and SPECT). After registration, you can compare features in the images to see how a river has migrated, how an area is flooded, or to see if a tumor is visible in an MRI or SPECT image.
The Image Processing Toolbox software provides tools to support point mapping to determine the parameters of the transformation required to bring an image into alignment with another image. In point mapping, you pick points in a pair of images that identify the same feature or landmark in the images. Then, a spatial mapping is inferred from the positions of these control points.
Note You might need to perform several iterations of this process, experimenting with different types of transformations, before you achieve a satisfactory result. In some cases, you might perform successive registrations, removing gross global distortions first, and then removing smaller local distortions in subsequent passes.
The following figure provides a graphic illustration of this process. This process is best understood by looking at an example. See Example: Registering to a Digital Orthophoto for an extended example.
If you need to perform the same kind of registration for many images, you automate the process by putting all the steps in a script. For example, you could create a script that launches the Control Point Selection Tool with an input and a base image. The script could then use the control points selected to create a TFORM structure and pass the TFORM and the input image to the imtransform function, outputting the registered image.
To do this, specify the 'Wait' option when you call cpselect to launch the Control Point Selection Tool. With the 'Wait' option, cpselect blocks the MATLAB command line until control points have been selected and returns the sets of control points selected in the input image and the base image as a return values. If you do not use the 'Wait' option, cpselect returns control immediately and your script will continue without allowing time for control point selection. In addition, without the 'Wait' option, cpselect does not return the control points as return values. For an example, see the cpselect reference page.
In this example, the base image is westconcordorthophoto.png, the MassGIS georegistered orthophoto. It is a panchromatic (grayscale) image, supplied by the Massachusetts Geographic Information System (MassGIS), that has been orthorectified to remove camera, perspective, and relief distortions (via a specialized image transformation process). The orthophoto is also georegistered (and geocoded) — the columns and rows of the digital orthophoto image are aligned to the axes of the Massachusetts State Plane coordinate system. In the orthophoto, each pixel center corresponds to a definite geographic location, and every pixel is 1 meter square in map units.
The image to be registered is westconcordaerial.png, a digital aerial photograph supplied by mPower3/Emerge, and is a visible-color RGB image. The aerial image is geometrically uncorrected: it includes camera perspective, terrain and building relief, internal (lens) distortions, and it does not have any particular alignment or registration with respect to the earth.
The following example reads both images into the MATLAB workspace and displays them usingorthophoto = imread('westconcordorthophoto.png'); figure, imshow(orthophoto) unregistered = imread('westconcordaerial.png'); figure, imshow(unregistered)
You do not have to read the images into the MATLAB workspace. The cpselect function accepts file specifications for grayscale images. However, if you want to use cross-correlation to tune your control point positioning, the images must be in the workspace.
The toolbox provides an interactive tool, called the Control Point Selection Tool, that you can use to pick pairs of corresponding control points in both images. Control points are landmarks that you can find in both images, like a road intersection, or a natural feature.
To start this tool, enter cpselect at the MATLAB prompt, specifying as arguments the input and base images.
The Control Point Selection Tool displays two views of both the input image and the base image in which you can pick control points by pointing and clicking. For more information, see Selecting Control Points. This figure shows the Control Point Selection Tool with four pairs of control points selected. The number of control point pairs you pick is at least partially determined by the type of transformation you want to perform (specified in Step 5). See Transformation Types for information about the minimum number of points required by each transformation.
In the Control Point Selection Tool, click the File menu and choose the Export Points to Workspace option. See Exporting Control Points to the Workspace for more information.
For example, the following set of control points in the input image represent spatial coordinates; the left column lists x-coordinates, the right column lists y-coordinates.input_points = 118.0000 96.0000 304.0000 87.0000 358.0000 281.0000 127.0000 292.0000
This is an optional step that uses cross-correlation to adjust the position of the control points you selected with cpselect. To use cross-correlation, features in the two images must be at the same scale and have the same orientation. They cannot be rotated relative to each other. Because the Concord image is rotated in relation to the base image, cpcorr cannot tune the control points. See Using Correlation to Improve Control Points for more information.
In this step, you pass the control points to the cp2tform function that determines the parameters of the transformation needed to bring the image into alignment. cp2tform is a data-fitting function that determines the transformation based on the geometric relationship of the control points. cp2tform returns the parameters in a geometric transformation structure, called a TFORM structure.
When you use cp2tform, you must specify the type of transformation you want to perform. The cp2tform function can infer the parameters for five types of transformations. You must choose which transformation will correct the type of distortion present in the input image. See Transformation Types for more information. Images can contain more than one type of distortion.
The predominant distortion in the aerial image of West Concord (the input image) results from the camera perspective. Ignoring terrain relief, which is minor in this area, image registration can correct for camera perspective distortion by using a projective transformation. The projective transformation also rotates the image into alignment with the map coordinate system underlying the base digital orthophoto image. (Given sufficient information about the terrain and camera, you could correct these other distortions at the same time by creating a composite transformation with maketform. See Performing General 2-D Spatial Transformations for more information.)
mytform = cp2tform(input_points, base_points, 'projective');
As the final step in image registration, transform the input image to bring it into alignment with the base image. You use imtransform to perform the transformation, passing it the input image and the TFORM structure, which defines the transformation. imtransform returns the transformed image. For more information about using imtransform, see Spatial Transformations
registered = imtransform(unregistered, mytform);
The following figure shows the transformed image transparently overlaid on the base image to show the results of the registration. (To see how this is done, see Example: Performing Image Registration.
The cp2tform function can infer the parameters for the following types of transformations, listed in order of complexity.
'polynomial' (Order 2, 3, or 4)
The first four transformations, 'nonreflective similarity', 'affine', 'projective', and 'polynomial' are global transformations. In these transformations, a single mathematical expression applies to an entire image. The last two transformations, 'piecewise linear' and 'lwm' (local weighted mean), are local transformations. In these transformations, different mathematical expressions apply to different regions within an image. When exploring how different transformations affect the images you are working with, try the global transformations first. If these transformations are not satisfactory, try the local transformations: the piecewise linear transformation first, and then the local weighted mean transformation.
Your choice of transformation type affects the number of control point pairs you need to select. For example, a nonreflective similarity transformation requires at least two control point pairs. A polynomial transformation of order 4 requires 15 control point pairs. For more information about these transformation types, and the special syntaxes they require, see cpselect.
To specify control points in a pair of images you want to register, use the Control Point Selection Tool, cpselect. The tool displays the image you want to register, called the input image, next to the image you want to compare it to, called the base image or reference image.
Specifying control points is a four-step process:
Start the tool, specifying the input image and the base image.
Use navigation aids to explore the image, looking for visual elements that you can identify in both images. cpselect provides many ways to navigate around the image, panning and zooming to view areas of the image in more detail.
Specify matching control point pairs in the input image and the base image.
Save the control points in the MATLAB workspace.
The following figure shows the default appearance of the tool when you first start it.
To use the Control Point Selection Tool, enter the cpselect command at the MATLAB prompt. As arguments, specify the image you want to register (the input image), and the image you want to compare it to (the base image).
For simplicity, this section uses the same image as the input and the base image. To walk through an example of an actual registration, see Registering an Image.moon_base = imread('moon.tif'); moon_input = moon_base; cpselect(moon_input, moon_base);
The cpselect command has other optional arguments. For example, you can restart a control point selection session by including a cpstruct structure as the third argument. For more information about restarting sessions, see Exporting Control Points to the Workspace. For complete details, see the cpselect reference page.
When the Control Point Selection Tool starts, it contains three primary components:
Details windows—The two windows displayed at the top of the tool are called the Detail windows. These windows show a close-up view of a portion of the images you are working with. The input image is on the left and the base image is on the right.
Overview windows—The two windows displayed at the bottom of the tool are called the Overview windows. These windows show the images in their entirety, at the largest scale that fits the window. The input image is on the left and the base image is on the right. You can control whether the Overview window appears by using the View menu.
Details rectangle—Superimposed on the images displayed in the two Overview windows is a rectangle, called the Detail rectangle. This rectangle controls the part of the image that is visible in the Detail window. By default, at startup, the detail rectangle covers one quarter of the entire image and is positioned over the center of the image. You can move the Detail rectangle to change the portion of the image displayed in the Detail windows.
The following figure shows these components of the Control Point Selection Tool.
To find visual elements that are common to both images, you might want to change the section of the image displayed in the detail view or zoom in on a part of the image to view it in more detail. The following sections describe the different ways to change your view of the images:
To view parts of an image that are not visible in the Detail or Overview windows, use the scroll bars provided for each window.
As you scroll the image in the Detail window, note how the Detail rectangle moves over the image in the Overview window. The position of the Detail rectangle always shows the portion of the image in the Detail window.
To get a closer view of any part of the image, move the Detail rectangle in the Overview window over that section of the image. cpselect displays that section of the image in the Detail window at a higher magnification than the Overview window.
To move the detail rectangle,
Move the pointer into the Detail rectangle. The cursor changes to the fleur shape, .
Press and hold the mouse button to drag the detail rectangle anywhere on the image.
Note As you move the Detail rectangle over the image in the Overview window, the view of the image displayed in the Detail window changes.
To change the section of the image displayed in the Detail window, use the pan tool to move the image in the window.
To use the pan tool,
Click the Pan button in the Control Point Selection Tool toolbar or select Pan from the Tools menu.
Move the pointer over the image in the Detail window. The cursor changes to the hand shape, .
Press and hold the mouse button. The cursor changes to a closed fist shape, . Use the mouse to move the image in the Detail window.
Note As you move the image in the Detail window, the Detail rectangle in the Overview window moves.
To enlarge an image to get a closer look or shrink an image to see the whole image in context, you can zoom in or zoom out on the images displayed. (You can also zoom in or out on an image by changing the magnification. See Specifying the Magnification of the Images for more information.)
To zoom in or zoom out on the base or input images,
Click the appropriate magnifying glass button on the Control Point Selection Tool toolbar or select Zoom In or Zoom Out from the Tools menu.
Move the pointer over the image in the Detail window that you want to zoom in or out on. The cursor changes to the appropriate magnifying glass shape, such as . Position the cursor over a location in the image and click the mouse. With each click, cpselect changes the magnification of the image by a preset amount. (SeeSpecifying the Magnification of the Images for a list of some of these magnifications.) cpselect centers the new view of the image on the spot where you clicked.
Another way to use the Zoom tool to zoom in on an image is to position the cursor over a location in the image and, while pressing and holding the mouse button, draw a rectangle defining the area you want to zoom in on. cpselect magnifies the image so that the chosen section fills the Detail window. cpselect resizes the detail rectangle in the Overview window as well.
The size of the Detail rectangle in the Overview window changes as you zoom in or out on the image in the Detail window.
To keep the relative magnifications of the base and input images synchronized as you zoom in or out, click the Lock ratio check box. See Locking the Relative Magnification of the Input and Base Images for more information.
To enlarge an image to get a closer look or to shrink an image to see the whole image in context, use the magnification edit box. (You can also use the Zoom buttons to enlarge or shrink an image. See Zooming In and Out on an Image for more information.)
To change the magnification of an image,
Move the cursor into the magnification edit box of the window you want to change. The cursor changes to the text entry cursor.
Type a new value in the magnification edit box and press Enter, or click the menu associated with the edit box and choose from a list of preset magnifications. cpselect changes the magnification of the image and displays the new view in the appropriate window. To keep the relative magnifications of the base and input images synchronized as you change the magnification, click the Lock ratio check box. See Locking the Relative Magnification of the Input and Base Images for more information.
To keep the relative magnification of the input and base images automatically synchronized in the Detail windows, click the Lock Ratio check box.
When the Lock Ratio check box is selected, cpselect changes the magnification of both the input and base images when you zoom in or out on either one of the images or specify a magnification value for either of the images.
The primary function of the Control Point Selection Tool is to enable you to pick control points in the image to be registered (the input image) and the image to which you are comparing it (the base image). When you start cpselect, point selection is enabled, by default.
You specify control points by pointing and clicking in the input and base images, in either the Detail or the Overview windows. Each point you specify in the input image must have a match in the base image. The following sections describe the ways you can use the Control Point Selection Tool to choose control point pairs:
To specify a pair of control points in your images,
Click the Control Point Selection button in the Control Point Selection Tool toolbar or select Add Points from the Tools menu. (Control point selection mode is active by default.) The cursor changes to a crosshairs shape
Position the cursor over a feature you have visually selected in any of the images displayed and click the mouse button. cpselect places a control point symbol, , at the position you specified, in both the Detail window and the corresponding Overview window. cpselect numbers the points as you select them. The appearance of the control point symbol indicates its current state. The circle around the point indicates that it is the currently selected point. The number identifies control point pairs.
Note Depending on where in the image you pick control points, the symbol for the point might be visible in the Overview window, but not in the Detail window.
The following figure illustrates control points in several states.
Instead of picking matching control points yourself, you can let the Control Point Selection Tool estimate the match for the control points you specify, automatically. The Control Point Selection Tool determines the position of the matching control point based on the geometric relationship of the previously selected control points, not on any feature of the underlying images.
To illustrate point prediction, this figure shows four control points selected in the input image, where the points form the four corners of a square. (The control point selections in the figure do not attempt to identify any landmarks in the image.) The figure shows the picking of a fourth point, in the left window, and the corresponding predicted point in the right window. Note how the Control Point Selection Tool places the predicted point at the same location relative to the other control points, forming the bottom right corner of the square.
Note By default, the Control Point Selection Tool does not include predicted points in the set of valid control points returned in input_points or base_points. To include predicted points, you must accept them by selecting the points and fine-tuning their position with the cursor. When you move a predicted point, the Control Point Selection Tool changes the symbol to indicate that it has changed to a standard control point. For more information, see Moving Control Points.
To use control point prediction,
Click the Control Point Prediction button .
Note Because the Control Point Selection Tool predicts control point locations based on the locations of the previous control points, you cannot use point prediction until you have a minimum of two pairs of matched points. Until this minimum is met, the Control Point Prediction button is disabled.
Position the cursor anywhere in any of the images displayed. The cursor changes to the crosshairs shape, .
You can pick control points in either of the Detail windows, input or base, or in either of the Overview windows, input or base. You also can work in either direction: input-to-base image or base-to-input image.
Click either mouse button. The Control Point Selection Tool places a control point symbol at the position you specified and places another control point symbol for a matching point in all the other windows. The symbol for the predicted point contains the letter P, indicating that it's a predicted control point,
To accept a predicted point, select it with the cursor and move it. The Control Point Selection Tool removes the P from the point.
To move a control point,
Click the Control Point Selection button .
Position the cursor over the control point you want to move. The cursor changes to the fleur shape,
Press and hold the mouse button and drag the control point. The state of the control point changes to selected when you move it.
If you move a predicted control point, the state of the control point changes to a regular (nonpredicted) control point.
To delete a control point, and its matching point, if one exists
Click the Control Point Selection button .
Click the control point you want to delete. Its state changes to selected. If the control point has a match, both points become active.
Delete the point (or points) using one of these methods:
Pressing the Backspace key
Pressing the Delete key
Choosing one of the delete options from the Edit menu
Using this menu you can delete individual points or pairs of matched points, in the input or base images.
After you specify control point pairs, you must save them in the MATLAB workspace to make them available for the next step in image registration, processing by cp2tform.
To save control points to the MATLAB workspace,
Select File on the Control Point Selection Tool menu bar.
Choose the Export Points to Workspace option. The Control Point Selection Tool displays this dialog box:
By default, the Control Point Selection Tool saves the x-coordinates and y-coordinates that specify the locations of the control points you selected in two arrays named input_points and base_points, although you can specify other names. These are n-by-2 arrays, where n is the number of valid control point pairs you selected. This example shows the input_points array containing four pairs of control points. The values in the left column represent the x-coordinates; the values in the right column represent the y-coordinates.input_points = 215.6667 262.3333 225.7778 311.3333 156.5556 340.1111 270.8889 368.8889
Whenever you exit the Control Point Selection Tool, it asks if you want to save your control points.
To save the current state of the Control Point Selection Tool, choose the Export Points to Workspace option from the File menu. In the Export Points to Workspace dialog box, select the Structure with all points check box.
This option saves the positions of all the control points you specified and their current states in a cpstruct structure.cpstruct = inputPoints: [4x2 double] basePoints: [4x2 double] inputBasePairs: [4x2 double] ids: [4x1 double] inputIdPairs: [4x2 double] baseIdPairs: [4x2 double] isInputPredicted: [4x1 double] isBasePredicted: [4x1 double]
You can use the cpstruct to restart a control point selection session at the point where you left off.
This option is useful if you are picking many points over a long time and want to preserve unmatched and predicted points when you resume work. The Control Point Selection Tool does not include unmatched and predicted points in the input_points and base_points arrays.
To extract the arrays of valid control point coordinates from a cpstruct, use the cpstruct2pairs function.
You might want to fine-tune the control points you selected using cpselect. Using cross-correlation, you can sometimes improve the points you selected by eye using the Control Point Selection Tool.
To use cross-correlation, pass sets of control points in the input and base images, along with the images themselves, to the cpcorr function.
input_pts_adj= cpcorr(input_points, base_points, input, base);
The cpcorr function defines 11-by-11 regions around each control point in the input image and around the matching control point in the base image, and then calculates the correlation between the values at each pixel in the region. Next, the cpcorr function looks for the position with the highest correlation value and uses this as the optimal position of the control point. The cpcorr function only moves control points up to 4 pixels based on the results of the cross-correlation.
Note Features in the two images must be at the same scale and have the same orientation. They cannot be rotated relative to each other.
If cpcorr cannot correlate some of the control points, it returns their values in input_points unmodified.