Note

GPU is required in version 3.x.x

How to get?

Please visit https://www.insta360.com/sdk/apply to apply for the latest SDK.

Support

Developers' Page: https://www.insta360.com/developer/home

Insta360 Enterprise: https://www.insta360.com/enterprise

Issue Report: https://insta.jinshuju.com/f/hZ4aMW

中文文档

Overview

MediaSDK is mainly used for stitching panoramic materials. The currently supported device models include: ONE X, ONE R/RS (Standard fisheye and one-inch fisheye), ONE X2, X3, X4, X4 Air, X5 cameras for panoramic materials. This SDK supports video export and image export. The main supported platforms are Windows and Ubuntu 22.04. For specific API usage, please refer to MediaSDK/example/main.cc.

• Supported cameras

Model	Link
ONE X (Discontinued)	http://insta360.com/product/insta360-onex/
ONE R Twin Edition (Discontinued)	http://insta360.com/product/insta360-oner_twin-edition
ONE X2	https://www.insta360.com/product/insta360-onex2
ONE RS 1-Inch 360 Edition (Discontinued)	https://www.insta360.com/product/insta360-oners/1inch-360
X3	https://www.insta360.com/product/insta360-x3
X4	https://www.insta360.com/product/insta360-x4
X4 Air	https://www.insta360.com/product/insta360-x4air
X5	https://www.insta360.com/product/insta360-x5

• Supported platforms

Platform	Version
Windows	Windows 7 or later, only x64 supported
Linux	Ubuntu 22.04 (x86_64), other distributions need to be tested

• Supported file format

filetype	import format	export format
Video	insv	mp4
Image	insp/jpeg	jpg

Table of contents

• Initialize SDK Environment
• Input and Output Parameter Settings
• Stabilization Parameter Settings
• Stitching Parameter Settings
• Image Setting Parameters
• Stitching Process
• Logging Functionality
• Hardware Codec Acceleration
• Live stream preview splicing
• Error Codes

Notes

The SDK requires that all file path strings must be encoded in UTF-8.

The GPU version of the SDK requires a graphics card.

Ubuntu system testing for WSL on Windows is not supported.

API Description

Initialize SDK Environment

void InitEnv() (GPU Version)

This API must be called at the start of the main program to initialize the SDK environment.

Set the model root directory

SetModelFileRootDir(const std::string path)

Supported from version 3.1.x onwards

This interface primarily sets the root directory for model files, allowing you to use SDK_DIR/models as a parameter. Its main purpose is to eliminate the tedious process of setting up different AI models for different cameras when using AI functions.

Input and Output Parameter Settings

Input Path:

void SetInputPath(std::vector\<std::string>& input_paths)

This API is used to set the input paths of the materials. It is an array and is valid for both videos and photos.

For videos, this array typically contains at most two material files. Materials with a resolution of 5.7K or higher require two material files as input (except for materials captured with X4 cameras). For X4 cameras, dual video track storage is currently used. Regardless of resolution, there is only one original video file.

Example Usage

// For dual-track 5.7K materials
std::string _00_file = "/path/to/VID_XXX_..._00_XXX.insv";
std::string _10_file = "/path/to/VID_XXX_..._10_XXX.insv"
std::vector<std::string> input_path = {_00_file,_10_file};
videoStitcher->SetInputPath(input_path);

// For single-track material files
std::string insv_file = "/path/to/VID_XXX_..._00_XXX.insv";
std::vector<std::string> input_path = {insv_file};
videoStitcher->SetInputPath(input_path);

For photo files, this array can accept multiple inputs (but not exactly 2). If 3, 5, 7 or 9 materials are input, they are assumed to be HDR photos, and HDR fusion will be applied automatically. For X4 cameras, the default HDR materials captured by the camera have already undergone in-camera HDR fusion. Therefore, only one material file is output from the camera.

Output Path:

void SetOutputPath(const std::string& output_path)

This API is used to set the output path. It is valid for both video and photo outputs. The parameter should be a full path.

Supported Output Formats:

For videos, the path should end with .mp4 Example:/output/to/path/video.mp4

For images, the path should end with .jpg Example: /output/to/path/image.jpg

Output Resolution:

void SetOutputSize(int width, int height)

This API is used to set the output resolution. The parameter width:height must have a 2:1 ratio.

For the CPU version of the SDK, if a resolution that is too small is set and moiré patterns appear, you can use the EnableAreaSampleAPI to eliminate moiré effects.

Encoding Format:

void EnableH265Encoder()

This API is used to set the encoding format to H.265. The default encoding format is H.264.

When the output resolution is greater than 4K, it is recommended to use H.265 encoding, as H.265 encoding supports hardware acceleration, which can speed up the export process.

Output Bitrate:

void SetOutputBitRate(int64\_t bitRate)

This API is used to set the bitrate for video output. The unit is bps.

If not set, the original video bitrate will be used by default.

Example:

To output at 60 Mbps, set the value as follows: bitRate = 60 × 1000 × 1000

Export Video as Image Sequences:

void SetImageSequenceInfo(const std::string output_dir, IMAGE_TYPE image_type)

This function allows users to export video frames as an image sequence and configure the output path and image format.

Parameters: output_dirThis parameter specifies the directory-level output path, excluding file information.Before using this function, make sure that the target directory has already been created.

Example:/path/to/image_save_dir

The exported image files are named based on the video frame timestamp (ms).

Example: /path/to/image_save_dir/100.jpg (This means the image was saved at the 100 ms video frame timestamp.)

image_type specifies the image format to be used. This can be either .png or .jpg.

Note:If you have used SetOutputPath, this does not need to be set again.

Export Selected Frames from Video:

void SetExportFrameSequence(const std::vector <uint64_t>& vec)

This API is used to export images from specific video frame indices. This function must be used together with SetImageSequenceInfo. The output image file names will be based on the video frame index.The video frame index starts from 0.

Example: /path/to/image_save_dir/10.jpg (this means the image is saved for video frame index 10)

Demo Example:

// This sample code demonstrates extracting frames 0, 10, 20, and 30 from a video file
std::vector<uint64_t> seq_nos = {0，10，20，30};
const std::string image_seq_export_dir = /path/to/image_seq_dir;
const IMAGE_TYPE image_format = IMAGE_TYPE::JPEG;
...
videoStitcher->SetExportFrameSequence(seq_nos);
videoStitcher->SetImageSequenceInfo(image_seq_export_dir,image_format);
...
videoStitcher->StartStitch()

Stabilization Parameter Settings

Enable Stabilization:

void EnableFlowState(bool enable)

This API is used to configure the stabilization option, determining whether to enable stabilization.

Enable Direction Lock:

void EnableDirectionLock(bool enable)

This API is used to enable direction lock.

**Stitching Parameter Settings

Stitching Type:

void SetStitchType(STITCH_TYPE type)

This API is used to set the stitching type. The available stitching types are as follows:

enum class STITCH_TYPE {
    TEMPLATE,       // Template stitching
    OPTFLOW,        // Optical flow stitching
    DYNAMICSTITCH,  // Dynamic stitching
    AIFLOW          // AI stitching
};

Usage Scenarios

• Template Stitching: An older stitching algorithm that provides poor stitching results for near-field scenes, but is fast and has low computational cost.

• Dynamic Stitching: Suitable for scenes containing motion or situations with rapid changes in movement.

• Optical Flow Stitching: Similar in function to dynamic stitching but optimized for higher accuracy.

• AI Stitching: Based on Insta360’s optimized optical flow stitching technology, offering superior stitching results.

Performance consumption and cutting effect：

AI Stitching > Optical Flow Stitching > Dynamic Stitching > Template Stitching

Stitching Speed:

Template Stitching > Dynamic Stitching > Optical Flow Stitching > AI Stitching

**Note: **When using AI Stitching, you must call the SetAiStitchModelFile API to specify the model file. If this step is skipped, the stitching settings will be invalid.

AI Stitching Model:

Note: Deprecated after version 3.1.x

void SetAiStitchModelFile(const std::string& model_file)

This API is used to set the AI stitching model, which is required for AI stitching.

Model file v1: SDK_ROOT_DIR/data/ai_stitch_model_v1.ins

Model file v2: SDK_ROOT_DIR/data/ai_stitch_model_v2.ins

For materials before X4 camera, use the v1 version of the model file. For materials of X5 camera, use the v2 version of the model file

Chromatic Calibration:

void EnableStitchFusion(bool enable)

This API is used to enable Chromatic Calibration.

Causes of chromatic aberration: The two lenses are separate, and the resulting video exposure may not be consistent. When they are stitched together, there will be a more obvious brightness difference. In addition, because the lighting on both sides of the lens is different, the camera exposure is different, and sometimes the pictures taken by the front and back lenses will also have a significant brightness difference. This phenomenon is particularly obvious in places with large light difference ratios. Achromatic aberration is developed to solve this problem.

Lens Guard:

void SetCameraAccessoryType(CameraAccessoryType type)

This API is used to set the lens guard type. If a lens guard is used during shooting, it must also be specified when stitching. Otherwise, the stabilization effect may be incorrect.

The following are the available lens guard types:

enum class CameraAccessoryType {
    kNormal = 0,            
    kWaterproof,            // Waterproof case (one/onex/onex2/oner/oners/onex3)
    kOnerLensGuard,         // Adhesive lens guard (oner/oners)
    kOnerLensGuardPro,      // Clip-on lens guard (oner/oners)
    kOnex2LensGuard,        // Adhesive lens guard (oner/oners/onex2/onex3)
    kOnex2LensGuardPro,     // Clip-on lens guard (onex2)
    k283PanoLensGuardPro,   // Clip-on lens guard for 283 panoramic lens (oner/oners)
    kDiveCaseAir,           // Dive case (above water) (onex/onex2/oner/oners/onex3)
    kDiveCaseWater,         // Dive case (underwater) (onex/onex2/oner/oners/onex3)
    kInvisibleDiveCaseAir,  // Invisible Dive Case  (Above water) (X3/X4)
    kInvisibleDiveCaseWater,// Invisible Dive Case  (underwater) (X3/X4)
    kOnex4LensGuardA,       // X4 A-grade plastic lens guard
    kOnex4LensGuardS,       // X4 S-grade glass lens guard
    kOnex3LensGuardA,       // X3 A-grade plastic lens guard
    kOnex3LensGuardS        // X3 S-grade glass lens guard
};

Standard lens guards in the store are classified as A-grade, while Premium guards are classified as S-grade.

Heat sink inspection:

void EnableCoolingShellDetection(bool enable, const std::string& model_dir)

This interface is used to detect the heat sink case. If you use a heat sink case but have not selected whether to use a heat sink case in the camera interface, you need to turn on this function for detection. Otherwise, it will affect the stitching effect of the picture.

This is an AI function that requires the model file path to be passed in.

Model file v1: SDK_ROOT_DIR/modelfile/coolingshell/

Image Setting Parameters

Color Plus:

void EnableColorPlus(bool enable, const std::string& model_path)

Note: Model path settings have been removed in version 3.1.x and later.

This API is used to enable or disable the Color Plus function. This is an AI-based feature, requiring the path to an AI model to be set.

Model file: SDK_ROOT_DIR/data/colorplus_model.ins

Denoise:

void EnableDenoise(bool enable, const std::string& model_path)

Note: Model path settings have been removed in version 3.1.x and later.

This API is used to enable or disable the denoising feature.

Multi-frame denoising is used in the video. It is a process of reducing or removing noise in the video through image processing technology. Compared with single-frame denoising, video denoising often uses redundant information of multiple frames before and after. It also consumes performance and slows down the export speed.

For image materials, the model file path should be specified.

Model file: SDK_ROOT_DIR/data/jpg_denoise_9d006262.ins

Remove purple edge:

void EnableDefringe(bool enable, const std::string& defringe_model_path)

Note: Model path settings have been removed in version 3.1.x and later.

This interface is used to eliminate the purple edge phenomenon caused by lighting during recording, such as outdoor strong light and indoor lighting scenes.

Model file: SDK_ROOT/modelfile/defringe_hr_dynamic_7b56e80f.ins

Remove strobe:

void EnableDeflicker(bool enable, const std::string& deflicker_model_path)

Note: Model path settings have been removed in version 3.1.x and later.

This interface is used to eliminate screen flickering problems caused by lighting during recording.

Model file: SDK_ROOT/modelfile/deflicker_86ccba0d.ins

Color adjust

• Exposure： SetExposure Range [-100,100]

• Highlights： SetHighlights Range [-100,100]

• Shadows： SetShadows Range [-100,100]

• Contrast： SetContrast Range [-100,100]

• Brightness： SetBrightness Range [-100,100]

• Blackpoint： SetBlackpoint Range [-100,100]

• Saturation： SetSaturation Range [-100,100]

• Vibrance： SetVibrance Range [-100,100]

• Warmth： SetWarmth Range [-100,100]

• Tint： SetTint Range [-100,100]

• Sharpness： SetDefinition Range [0,100]

Stitching Process

Stitching Progress Callback:

void SetStitchStateCallback(stitch_error_callback callback)

This API is primarily used for stitching status and progress notifications.

It is recommended not to perform time-consuming operations within this callback, as this may affect stitching speed.

Example Code:

video_stitcher->SetStitchProgressCallback([&](int process, int error) {
    if (stitch_progress != process) {
        std::cout << "\r";
        std::cout << "process = " << process << "%";
        std::cout << std::flush;
        stitch_progress = process;
    }

    if (stitch_progress == 100) {
        std::cout << std::endl;
        std::unique_lock<std::mutex> lck(mutex_);
        cond_.notify_one();
        is_finisned = true;
    }
 });

Stitching Error Callback:

void SetStitchProgressCallback(stitch_process_callback callback)

This API is used to receive error messages during the stitching process.

Example Code:

video_stitcher->SetStitchStateCallback([&](int error, const char* errinfo) {
    std::cout << "error: " << errinfo << std::endl;
    has_error = true;
    cond_.notify_one();
});

Start Stitching:

void StartStitch()

This API is used to start the stitching process.

**Note:**Ensure that all parameter settings are completed before calling this API.If this API is executed before setting the necessary parameters, the parameters will not take effect.

Cancel Stitching:

bool CancelStitch()

This API is used to interrupt the stitching process.

Get Stitching Progress:

int GetStitchProgress() const

This API is used to retrieve the stitching progress.

Logging Functionality

Set Log Path:

void SetLogPath(const std::string log_path)

This API is primarily used to set the log file path in the SDK, allowing SDK log information to be saved.

Set Log Print Level:

void SetLogLevel(InsLogLevel level)

This API is used to set the logging level within the SDK.

Hardware Codec Acceleration

Set whether to enable software encoding and decoding：

SetSoftwareCodecUsage

This interface is mainly used by users to set whether to force the use of software codec. In an environment with only CPU, if an error occurs, it can be set to software codec.

Disablecuda:

EnableCuda

This interface mainly detects whether cuda acceleration is enabled. If cuda acceleration is not enabled, it is recommended to set it to false.

Set rendering acceleration type:

SetImageProcessingAccelType

This interface is mainly used to set the rendering acceleration: Auto is used for automatic detection. If you encounter Vulkan errors, it is recommended to set it to CPU.

Live stream preview stitching

Environmental preparation

Camera stitch preview is based on CameraSDK and MediaSDK together. Header File is located in MediaSDK_ROOT/include/ins_realtime_stitcher.

The main function of CameraSDK is to provide stitching parameters, video data, anti-shake data, and exposure data.

The main function of MediaSDK is to use the parameters and data provided by cameraSDK to stitch together images and generate a 2:1 panoramic image.

Please refer to MediaSDK_ROOT/example/realtime_stitcher_demo.cc

Preview parameter acquisition and setting

#include <ins_realtime_stitcher.h>
// This interface mainly obtains the parameters required by mediaSDK
//...
// Cam is the current camera instance object
auto preview_param = cam->GetPreviewParam();

// Create a stitching instance object
auto stitcher = std::make_shared<ins::RealTimeStitcher>();

// Set preview parameters for stitching instance objects
ins::CameraInfo camera_info;
camera_info.cameraName = preview_param.camera_name;
camera_info.decode_type = static_cast<ins::VideoDecodeType>(preview_param.encode_type);
camera_info.offset = preview_param.offset;
auto window_crop_info = preview_param.crop_info;
camera_info.window_crop_info_.crop_offset_x = window_crop_info.crop_offset_x;
camera_info.window_crop_info_.crop_offset_y = window_crop_info.crop_offset_y;
camera_info.window_crop_info_.dst_width = window_crop_info.dst_width;
camera_info.window_crop_info_.dst_height = window_crop_info.dst_height;
camera_info.window_crop_info_.src_width = window_crop_info.src_width;
camera_info.window_crop_info_.src_height = window_crop_info.src_height;
camera_info.gyro_timestamp = preview_param.gyro_timestamp;

stitcher->SetCameraInfo(camera_info);

Preview stream processing of original data source

// In the cameraSDK, you need to use the inheritance ins_camera :: StreamDelegate interface to achieve real-time data acquisition of the camera.
// Real-time data from cameraSDK can be transmitted to MediaSDK through demo examples

class StitchStreamDelegate : public ins_camera::StreamDelegate {
public:
    StitchDelegate(const std::shared_ptr<ins::RealTimeStitcher>& stitcher) :stitcher_(stitcher) {
    }

    virtual ~StitchDelegate() {
    }

    void OnAudioData(const uint8_t* data, size_t size, int64_t timestamp) override {}

    // Video data
    void OnVideoData(const uint8_t* data, size_t size, int64_t timestamp, uint8_t streamType, int stream_index) override {
        stitcher_->HandleVideoData(data, size, timestamp, streamType, stream_index);
    }
     
    // Anti-shake data
    void OnGyroData(const std::vector<ins_camera::GyroData>& data) override {
        std::vector<ins::GyroData> data_vec(data.size());
        memcpy(data_vec.data(), data.data(), data.size() * sizeof(ins_camera::GyroData));
        stitcher_->HandleGyroData(data_vec);
    }
 
    // Exposure data
    void OnExposureData(const ins_camera::ExposureData& data) override {
        ins::ExposureData exposure_data{};
        exposure_data.exposure_time = data.exposure_time;
        exposure_data.timestamp = data.timestamp;
        stitcher_->HandleExposureData(exposure_data);
    }

private:
    std::shared_ptr<ins::RealTimeStitcher> stitcher_;
};

Set preview parameters

stitching type

Refer to the Stitching Type above

Stabilization parameter

Refer to the Stabilization Parameter Settings above

Lens Guard

Refer to the Lens Guard Settings above

Output screen size

For output size, if not set, the output size is the resolution of the current preview.

If the performance output frame rate, the resolution size can be reduced.

Get stitched data

The currently supported format for stitched data is RGBA.

You can get the stitched video picture by setting SetStitchRealTimeDataCallback this callback interface. It is recommended not to perform time-consuming operations in this callback. The reference code is as follows:

stitcher->SetStitchRealTimeDataCallback([&](uint8_t* data[4], int linesize[4], int width, int height, int format, int64_t timestamp) {
        show_image_ = cv::Mat(height, width, CV_8UC4, data[0]).clone();
    });

Enable preview

// Set up the delegation interface for camera real-time data
std::shared_ptr<ins_camera::StreamDelegate> delegate = std::make_shared<StitchStreamDelegate>(stitcher);
cam->SetStreamDelegate(delegate);
ins_camera::LiveStreamParam param;
//...
// Open the preview of the camera
if (cam->StartLiveStreaming(param)) {
// Start the stitching process
    stitcher->StartStitch();
    std::cout << "successfully started live stream" << std::endl;
}

Close preview

// Close camera preview stream
if (cam->StopLiveStreaming()) {
// Stop the stitching process
    stitcher->CancelStitch();
    std::cout << "success!" << std::endl;
}

Error Codes

Error Code	Error Message
E_OPEN_FILE(1)	Failed to open file
E_PARSE_METADATA(2)	Failed to parse file metadata
E_CREATE_OFFSCREEN(3)	Common offscreen rendering failure
E_CREATE_RENDER_MODEL(4)	Failed to create render model
E_FRAME_PARSE(5)	Failed to retrieve data frame
E_CREATE_RENDER_SOURCE(6)	Failed to create rendering data source
E_UPDATE_RENDER_SOURCE(7)	Failed to update rendering data frame
E_RENDER_FRAME(7)	Failed to render frame
E_SAVE_FRAME(8)	Failed to save image
E_VIDEO_FRAME_EXPORTOR(9)	Failed to create video frame exporter
E_UNKNOWN(999)	Unknown error, please provide detailed information for analysis

FAQ

How to write a Dockerfile if using Docker

FROM nvidia/cuda:12.1.1-devel-ubuntu22.04
# Environment variables
# ENV __NV_PRIME_RENDER_OFFLOAD=1
# ENV __GLX_VENDOR_LIBRARY_NAME=nvidia
RUN apt-get update && apt-get install -y \
    gnupg \
    software-properties-common \
    libtiff5 \
    libegl1 \
    libxext6 \
    libvulkan1 \
    freeglut3-dev \
    vulkan-tools \
    libdc1394-25 
# Copy application code
COPY libMediaSDK-dev-3.0.3.3-20250616_094659-amd64.deb /home/
COPY VID_20221010_123325_00_064.insv /home/
COPY VID_20221010_123325_10_064.insv /home/
RUN cd /home
RUN apt-get update 
RUN VULKAN_API_VERSION=$(dpkg -s libvulkan1 | grep -oP 'Version: [0-9|\.]+' | grep -oP '[0-9]+(\.[0-9]+)(\.[0-9]+)') && \
    mkdir -pm755 /etc/vulkan/icd.d/ && echo "{\
    \"file_format_version\" : \"1.0.0\",\
    \"ICD\": {\
        \"library_path\": \"libGLX_nvidia.so.0\",\
        \"api_version\" : \"${VULKAN_API_VERSION}\"\
    }}" > /etc/vulkan/icd.d/nvidia_icd.json && \
    mkdir -pm755 /usr/share/glvnd/egl_vendor.d/ && echo "{\
    \"file_format_version\" : \"1.0.0\",\
    \"ICD\": {\
        \"library_path\": \"libEGL_nvidia.so.0\"\
    }}" > /usr/share/glvnd/egl_vendor.d/10_nvidia.json