# **Stereo Disparity Map and Point Cloud Generation** This project implements a dense stereo disparity map and point cloud generation pipeline using spatio-temporal Zero-Normalized Cross-Correlation (ZNCC) with subpixel interpolation methods and other advanced features. ## **Features** - **Dense Disparity Map Calculation**: Generates Disparity and vertical shift maps. - **Subpixel Interpolation**: Supports parabolic, Gaussian, and equiangular interpolation methods for subpixel accuracy. - **Correlation Map**: Computes ZNCC correlation for each pixel. - **Point Cloud Generation**: Reprojects disparity maps into 3D space using a Q matrix. - **Color Mapping**: Normalizes depth values and applies a colormap for visualization. - **Optional Median Filtering**: Smoothens the disparity map. - **Visualization**: Saves disparity maps, vertical shift maps, and correlation maps with colorbars. ## **Requirements** - Python 3.9 or later ```bash # This step only pip install -r ~/Speckle-Scanner/05_disparity/zncc/requirements.txt # Or install everything for the full pipeline pip install -r ~/Speckle-Scanner/requirements.txt # Optional GPU acceleration (pick one matching your CUDA version) pip install cupy-cuda12x # CUDA 12.x pip install cupy-cuda11x # CUDA 11.2–11.8 ``` Packages: `numpy`, `opencv-python`, `matplotlib`, `scipy`, `numba`, `open3d`. Without `cupy`, the pipeline falls back to CPU (`numba`). ## **Input Requirements** - **Stereo Rectified Images**: - Images must be stored in `.png` or `.bmp` format. - **Pipeline pairing** (in `02_rect_images/`): `lc_ts1634840093_ck….png` is matched with `rc_ts1634840093_ck….png` on the shared `ts` token; the `ck…` suffix is ignored. - Pairs are sorted by timestamp; `--num_images` defaults to **all** pairs; if limited, the **last N** (highest timestamps) are used. - Standalone runs may still use separate `left/` and `right/` folders with any naming; pipeline mode uses one folder with `lc_` / `rc_` prefixes. - PNG is preferred when both PNG and BMP pairs exist. - **Q Matrix File**: - A `.cvstore` file containing the Q matrix for 3D reprojection. ## **Outputs** - **Disparity Map**: Saved as `Disparity.npy`. - **Vertical Shift Map**: Saved as `vertical_shift_map.npy`. - **Correlation Map**: Saved as `correlation_map.npy`. - **Disparity and Shift Maps with Colorbars**: - `Disparity_map_colorbar.png` - `vertical_shift_map_colorbar.png` - `correlation_map_colorbar.png` - **Point Cloud**: - Saved as `Point_cloud.ply` in PLY format with depth-based color information. ## **Usage** 1. Clone the repository and navigate to the project directory. 2. Prepare your stereo images and paste them in data folder Example directory structure: ``` project_directory/ .data/ ├── left/ │ ├── 0.png │ ├── 1.png │ └── ... └── right/ ├── 0.png ├── 1.png └── ... Or .data/ ├── left/ │ ├── lc00000.bmp │ └── lc00001.bmp └── right/ ├── rc00000.bmp └── rc00001.bmp ``` 3. The Q matrix must be provided in a YAML file (e.g., `Q.cvstore`). Format example: ``` %YAML:1.0 --- Q: !!opencv-matrix rows: 4 cols: 4 dt: f data: [ 1.0, 0.0, 0.0, -5.5863211059570312e+02, 0.0, 1.0, 0.0, -7.3320780181884766e+02, 0.0, 0.0, 0.0, 3.3665836719173171e+03, 0.0, 0.0, 1.4262549294100269e+00, 1.0682440279589944e+03 ] ``` This matrix is essential for converting disparity maps into a 3D point cloud. 3. Run the script with desired parameters (see below). ### **Basic Command** ```bash # stereo_disparity_main.py auto-selects GPU (CUDA) or CPU. cd ~/Speckle-Scanner/05_disparity/zncc # Default parameters (saves disparity map only) python stereo_disparity_main.py \ --left_dir \ --right_dir \ --left_prefix lc_ \ --right_prefix rc_ \ --q_file \ --disp_output_dir \ --pcl_output_dir # Troubleshooting mode — saves all outputs (vertical shift, correlation, point cloud) python stereo_disparity_main.py \ --left_dir \ --right_dir \ --left_prefix lc_ \ --right_prefix rc_ \ --q_file \ --disp_output_dir \ --pcl_output_dir \ --troubleshooting ``` This runs the program with default parameters. Same commands for stereo_disparity_cpu.py and stereo_disparity_gpu.py ### **Custom Parameters** You can override any parameter through command-line arguments. Below is an example with some customized parameters: ```bash python stereo_disparity_main.py \ --window_size 5 \ --H_neg_range -165 \ --H_pos_range 65 \ --v_neg_range -4 \ --v_pos_range 0 \ --zncc_threshold 0.4 \ --num_images 10 \ # optional: last 10 pairs by timestamp; omit for all pairs --noise_filter "open3d" \ --interpolation True \ --noise_remove False \ --method "gaussian" \ --q_file "./data/Q_matrix.cvstore" \ --left_dir "./data/left" \ --right_dir "./data/right" \ --disp_output_dir "./output/disp" \ --pcl_output_dir "./output/pcl" \ --troubleshooting ``` ## **Available Parameters** | Parameter | Default Value | Description | | ------------------ | ------------------- | --------------------------------------------------------------------------------------| | `--window_size` | `5` | Size of the window for block matching. | | `--H_neg_range` | `0` | Horizontal negative disparity range. | | `--H_pos_range` | `65` | Horizontal positive disparity range. | | `--v_neg_range` | `-4` | Vertical negative disparity range. | | `--v_pos_range` | `2` | Vertical positive disparity range. | | `--zncc_threshold` | `0.4` | ZNCC threshold for valid matches. | | `--num_images` | all pairs | Stereo pairs to use per scan. Default: all `lc_ts*`/`rc_ts*` pairs matched on `ts`. If fewer than available, the **last N** pairs (highest timestamps) are used; `ck*` suffix is ignored. | | `--noise_filter` | `open3d` | Post-processing noise filter method (`median`[Disparity Map] or `open3d`[Pointcloud]).| | `--interpolation` | `True` | Enable subpixel interpolation (`True` or `False`). | | `--noise_remove` | `True` | noise removal using bidirectional disparity estimation (`True` or `False`). | | `--method` | `parabolic` | Subpixel interpolation method (`parabolic`, `gaussian`, or `equiangular`). | | `--q_file` | `./data1/Q.cvstore` | Path to the Q matrix file. | | `--left_dir` | `./data1/left` | Path to the directory containing left images. | | `--right_dir` | `./data1/right` | Path to the directory containing right images. | | `--left_prefix` | `""` | Filename prefix to filter left images (e.g. `lc_`) when both cameras share a folder. | | `--right_prefix` | `""` | Filename prefix to filter right images (e.g. `rc_`) when both cameras share a folder. | | `--output_dir` | `./results` | Fallback output directory (used when `--disp_output_dir`/`--pcl_output_dir` are not set). | | `--disp_output_dir` | same as `--output_dir` | Directory for disparity map outputs (npy + png). | | `--pcl_output_dir` | same as `--output_dir` | Directory for point cloud outputs (PLY + TXT). | | `--troubleshooting` | `False` | When set, saves all outputs (vertical shift map, correlation map, point cloud); otherwise only disparity map is saved. | ## **Details of Post-processing noise filter method** - **open3d**: used for denoising point clouds. - **median**: used to reduce noise in the disparity map. ## **Details of Subpixel Interpolation Methods** - **Parabolic**: Fits a parabola to the ZNCC values and finds the peak. - **Gaussian**: Uses the logarithm of ZNCC values to model a Gaussian distribution and find the peak. - **Equiangular**: Uses angular interpolation for smoother and more robust subpixel shifts. ## **Notes** - Ensure stereo images are rectified before processing. - Adjust the disparity ranges (`H_neg_range`, `H_pos_range`, etc.) based on your dataset. - If disparity maps appear noisy, enable median filtering or refine the ZNCC threshold. ## **Execution Time** - Execution time depends on the number of images, window size, and disparity ranges. - Subpixel interpolation adds computational overhead but improves accuracy. ## **Point Cloud File** - The generated `.ply` file can be visualized using point cloud tools like MeshLab or CloudCompare. - Depth values are normalized and color-coded using the `jet` colormap. ## **Troubleshooting** - Ensure the `Q.cvstore` file exists in the specified path. - Check file permissions. - Ensure the left and right images are rectified and have the same resolution. - Reduce the number of images or use a smaller window size. - Run on a machine with higher processing power. --- ## **Pipeline Usage (Automated Path Resolution)** Use `run_zncc_pipeline.py` instead of calling `stereo_disparity_main.py` directly. It resolves all paths automatically from the project folder structure and processes every scan in a session (or a single scan you name). ### **Folder structure assumed** ``` ~/Speckle-Scanner_Processing_data/ └── / └── / └── / ├── params_link/ │ └── lc-rc_Q.cvstore ← Q matrix (input) └── / ├── 02_rect_images/ ← lc_ts*.png + rc_ts*.png (input) ├── 04_zncc_disp_map/ ← disparity .npy + colorbar .png (created) └── 06_zncc_pcl/ ← Point_cloud.ply + .txt (created) ``` ### **Commands** ```bash cd ~/Speckle-Scanner/05_disparity/zncc # Process ALL scans in a session (all matched lc/rc pairs per scan) python run_zncc_pipeline.py \ --project Olsen_wings \ --date 2026-05-12 \ --session session1 # Process ALL sessions on a date (omit --session) python run_zncc_pipeline.py \ --project Olsen_wings \ --date 2026-05-12 # Process a SINGLE scan — only the last 3 pairs (highest timestamps) python run_zncc_pipeline.py \ --project Olsen_wings \ --date 2026-05-12 \ --session session1 \ --scan Scan000001 \ --num_images 3 # Full outputs including point cloud and debug maps (troubleshooting mode) python run_zncc_pipeline.py \ --project Olsen_wings \ --date 2026-05-12 \ --session session1 \ --scan Scan000001 \ --troubleshooting # Custom ZNCC parameters python run_zncc_pipeline.py \ --project Olsen_wings \ --date 2026-05-12 \ --session session1 \ --window_size 7 \ --H_neg_range 0 \ --H_pos_range 80 \ --v_neg_range -4 \ --v_pos_range 2 \ --zncc_threshold 0.5 \ --noise_filter open3d \ --method gaussian ``` ### **Pipeline parameters** | Parameter | Default | Description | |---------------------|---------|------------------------------------------------------------------------------------| | `--project` | — | Project name (e.g. `Olsen_wings`) | | `--date` | — | Date string (e.g. `2026-05-12`) | | `--session` | all | Session name (e.g. `session1`); omit to process **all sessions** on that date | | `--scan` | all | Single scan to process (e.g. `Scan000001`); omit to process all scans in session | | `--troubleshooting` | off | When set, saves vertical shift map, correlation map, and point cloud PLY/TXT; by default only `disparity.npy` and `Disparity_map_colorbar.png` are saved | | `--num_images` | all | Image pairs per scan (matched on `ts`; last N if limited) | | All other ZNCC params | see above table | Forwarded directly to `stereo_disparity_main.py` | ### **What gets saved** | File | Default | `--troubleshooting` | |------|:-------:|:-------------------:| | `04_zncc_disp_map/disparity.npy` | ✓ | ✓ | | `04_zncc_disp_map/Disparity_map_colorbar.png` | ✓ | ✓ | | `04_zncc_disp_map/vertical_shift_map.npy` | | ✓ | | `04_zncc_disp_map/vertical_shift_map_colorbar.png` | | ✓ | | `04_zncc_disp_map/correlation_map.npy` | | ✓ | | `04_zncc_disp_map/correlation_map_colorbar.png` | | ✓ | | `06_zncc_pcl/Point_cloud.ply` | | ✓ | | `06_zncc_pcl/Point_cloud.txt` | | ✓ | --- Contributions and improvements are welcome! Feel free to open an issue or submit a pull request on GitHub.