Calibration between TarantulaX and Your Mobile Phone

Camera Calibration for TarantulaX and Mobile Devices

Calibrating the TarantulaX system with a mobile phone camera involves determining both intrinsic and extrinsic camera parameters to ensure accurate data fusion and spatial alignment.

1. Intrinsic Parameters

Intrinsic parameters define the internal characteristics of the mobile phone camera, such as:

• Focal length

• Principal point (optical center)

• Lens distortion coefficients (e.g., radial and tangential distortion)

These parameters are used to correct for distortions and project 3D points onto the 2D image plane accurately.

2. Extrinsic Parameters

Extrinsic parameters define the spatial relationship between the mobile phone camera and the TarantulaX system. These include:

• Rotation matrix (R): Describes the orientation of the camera relative to the TarantulaX coordinate frame.

• Translation vector (T): Describes the position of the camera relative to the TarantulaX system.

Together, they enable proper alignment of sensor data and ensure accurate mapping and localization.

Intrinsics Parameters:

Camera intrinsics are structured as an array of floating-point values in the following order:

$$[f_x, f_y, c_x, c_y, k_1, k_2, k_3, p_1, p_2]$$

plus a lens distortion model type, which could be "Tan" or "ArcTan":

• [fₓ , fᵧ] = focal lengths in pixels. While the instructions above will produce correctly scaled values, the values in the EEPROM of the camera will be in millimeters. To convert these "actual focal lengths" into pixels, divide by the pixel-pitch of the camera (0.00375 mm/px). The focal lengths must also be scaled according to the image resolution that you are undistorting (ie. since vision uses half-sized frames, you must divide by 2).

• [cₓ , cᵧ] = center of distortion, in absolute pixel values. The EEPROM values will be relative to the images center (ie, -2,). All cameras will have a tolerance of +/-15 px, hence, having a nominal value of zero is common.

The meaning of the other coefficients depends on the lens distortion model. When it is "Tan" model. In short,

• [k₁, k₂, k₃] = radial distortion coefficients. These are ~arbitrary, resolution-independent values that distort the image as a function of radius from (cₓ , cᵧ). k₁ is the dominant term and generally defines the direction of the distortion correction.

  • x' = x(1 + k₁ r² + k₂ r⁴ + k₃ r⁶ )

  • k₁ > 0 = barrel distortion

  • k₁ < 0 = pincushion distortion

• [p₁, p₂] = tangential distortion coefficients. These account for lenses that are slightly tilted. For nominal values, these are often set to zero

  • x' = 2p₁ xy + p₂ (r² + 2x² )

  • y' = p₁ (r² + 2y² ) + 2p₂ xy

• please refer tohttps://docs.opencv.org/2.4/doc/tutorials/calib3d/camera_calibration/camera_calibration.html for detail

When it is "ArcTan" model. In short,

• [k₁, k₂, k₃] = radial distortion coefficients. These are resolution-independent values that distort the image as a function of angle from the principal point.

  • r =(x/z)^2 + (y/z)^2

  • θ= atan(r)

  • θ_d= theta * (1 + k₁ * θ^2 + k₂ * θ^4 + k₃ * θ^6)

  • [x', y’]= θ_d * r * [x/z, y/z]

  • y'= θ_d * (y/z) * r

• [p₁, p₂] are not uesd

• please refer tohttps://docs.opencv.org/trunk/db/d58/group__calib3d__fisheye.html for detail

Since we are using a mobile phone camera, which is generally a zoom camera, we need to first adjust the camera to a fixed-focus camera and set the focal length value. Below is the sample code for Samsung Camera SDK:

import com.samsung.android.sdk.camera.SCamera;
import com.samsung.android.sdk.camera.SCameraManager;
import com.samsung.android.sdk.camera.SCameraCharacteristics;
import com.samsung.android.sdk.camera.SCaptureRequest;
import com.samsung.android.sdk.camera.SCaptureRequest.Builder;
import com.samsung.android.sdk.camera.SCaptureSession;
import com.samsung.android.sdk.camera.SCameraCaptureSession;
import com.samsung.android.sdk.camera.SCaptureResult;

public class CameraActivity extends AppCompatActivity {
    
    private SCamera mCamera;
    private SCameraManager mCameraManager;
    private SCameraCharacteristics mCharacteristics;
    private SCaptureRequest.Builder mCaptureRequestBuilder;
    private SCaptureSession mCaptureSession;

    @Override
    protected void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.activity_camera);
        
        // Initialize the camera
        mCamera = SCamera.getInstance();
        mCameraManager = mCamera.getSCameraManager();
        
        // Get camera characteristics
        try {
            String cameraId = getCameraId();
            mCharacteristics = mCameraManager.getCameraCharacteristics(cameraId);
            
            // Set focus mode to manual focus
            mCaptureRequestBuilder.set(SCaptureRequest.CONTROL_AF_MODE, SCaptureRequest.CONTROL_AF_MODE_OFF);
            
            // Set focus distance (in millimeters)
            mCaptureRequestBuilder.set(SCaptureRequest.LENS_FOCUS_DISTANCE, 50f); // For example, set focus distance to 50 millimeters
            
            // Create capture request
            SCaptureRequest captureRequest = mCaptureRequestBuilder.build();
            
            // Create capture session
            mCamera.createCaptureSession(Arrays.asList(mSurface), new SCameraCaptureSession.StateCallback() {
                @Override
                public void onConfigured(SCaptureSession session) {
                    mCaptureSession = session;
                    try {
                        // Start capture
                        mCaptureSession.setRepeatingRequest(captureRequest, null, null);
                    } catch (SCameraAccessException e) {
                        e.printStackTrace();
                    }
                }
                
                @Override
                public void onConfigureFailed(SCaptureSession session) {
                    // Handle configuration failure
                }
            }, null);
        } catch (SCameraAccessException e) {
            e.printStackTrace();
        }
    }
    
    // Get camera ID
    private String getCameraId() throws SCameraAccessException {
        for (String cameraId : mCameraManager.getCameraIdList()) {
            SCameraCharacteristics characteristics = mCameraManager.getCameraCharacteristics(cameraId);
            // Check if it's a rear-facing camera
            Integer facing = characteristics.get(SCameraCharacteristics.LENS_FACING);
            if (facing != null && facing == SCameraCharacteristics.LENS_FACING_BACK) {
                return cameraId;
            }
        }
        return null;
    }
}

While we set the camera to fixed-focus, we can also retrieve the camera's intrinsic parameters. Here's an example code using the Samsung Camera SDK:

import android.content.Context;
import android.util.Log;

import com.samsung.android.sdk.camera.SCamera;
import com.samsung.android.sdk.camera.SCameraCharacteristics;
import com.samsung.android.sdk.camera.SCameraManager;
import com.samsung.android.sdk.SsdkUnsupportedException;

public class SamsungCameraParameters {
    private SCamera sCamera;
    private SCameraManager sCameraManager;
    private SCameraCharacteristics characteristics;

    public SamsungCameraParameters(Context context) {
        // Initialize SCamera instance
        sCamera = new SCamera();
        try {
            sCamera.initialize(context);
        } catch (SsdkUnsupportedException e) {
            e.printStackTrace();
        }
        // Get SCameraManager instance
        sCameraManager = sCamera.getSCameraManager();
    }

    public void getCameraParameters(String cameraId) {
        try {
            // Get camera characteristics for the specified camera ID
            characteristics = sCameraManager.getCameraCharacteristics(cameraId);

            // Get focal lengths
            float[] focalLengths = characteristics.get(SCameraCharacteristics.LENS_INFO_AVAILABLE_FOCAL_LENGTHS);
            if (focalLengths != null && focalLengths.length > 0) {
                Log.d("SamsungCamera", "Focal Length: " + focalLengths[0]);
            }

            // Get sensor size
            SizeF sensorSize = characteristics.get(SCameraCharacteristics.SENSOR_INFO_PHYSICAL_SIZE);
            if (sensorSize != null) {
                Log.d("SamsungCamera", "Sensor Width: " + sensorSize.getWidth());
                Log.d("SamsungCamera", "Sensor Height: " + sensorSize.getHeight());
            }

            // Get principal point (assumed to be at the center of the sensor)
            float principalPointX = sensorSize.getWidth() / 2;
            float principalPointY = sensorSize.getHeight() / 2;
            Log.d("SamsungCamera", "Principal Point X: " + principalPointX);
            Log.d("SamsungCamera", "Principal Point Y: " + principalPointY);

            // Get distortion coefficients
            float[] distortionCoefficients = characteristics.get(SCameraCharacteristics.LENS_DISTORTION);
            if (distortionCoefficients != null) {
                Log.d("SamsungCamera", "Distortion Coefficients: " + Arrays.toString(distortionCoefficients));
            } else {
                Log.d("SamsungCamera", "No distortion coefficients available.");
            }
        } catch (CameraAccessException e) {
            e.printStackTrace();
        }
    }
}

Extrinsics Parameters:

Camera extrinsics are also structured as an array of floating-point values:

$$[m_{11}, m_{12}, m_{13}, m_{14}, m_{21}, m_{22}, m_{23}, m_{24}, m_{31}, m_{32}, m_{33}, m_{34}, pitch, yaw, roll]$$

• [m₁₁ … m₃₄] is a 3x4 inverse RT matrix describing the camera's nominal position and rotation within respect to the car.

• [pitch, yaw, roll] = radian values describing the calibrated rotation offsets from the nominals above.

For the calibration process of the extrinsics parameters for the TarantulaX and the smartphone, we will employ a hand-eye calibration method. To ensure that the initial extrinsics parameters are as close to the true parameters as possible, we will impose certain restrictions on the User Manual, such as installing the device in front of the car sunroof, etc. For detailed instructions, please refer to our User Manual.

Once the installation is complete, users can drive freely on the road. When the calibration is completed, we will prompt you through the mobile app (of course, if you want to complete the calibration quickly, you can choose an open space to drive in a figure-eight pattern).

After the mobile app prompts that The calibration is complete, please start your ROVR journey.