324 lines
13 KiB
Markdown
324 lines
13 KiB
Markdown
# **Stereo Disparity Map and Point Cloud Generation**
|
||
|
||
This project implements a dense stereo disparity map and point cloud generation pipeline using spatio-temporal Zero-Normalized Cross-Correlation (ZNCC) with subpixel interpolation methods and other advanced features.
|
||
|
||
## **Features**
|
||
|
||
- **Dense Disparity Map Calculation**: Generates Disparity and vertical shift maps.
|
||
- **Subpixel Interpolation**: Supports parabolic, Gaussian, and equiangular interpolation methods for subpixel accuracy.
|
||
- **Correlation Map**: Computes ZNCC correlation for each pixel.
|
||
- **Point Cloud Generation**: Reprojects disparity maps into 3D space using a Q matrix.
|
||
- **Color Mapping**: Normalizes depth values and applies a colormap for visualization.
|
||
- **Optional Median Filtering**: Smoothens the disparity map.
|
||
- **Visualization**: Saves disparity maps, vertical shift maps, and correlation maps with colorbars.
|
||
|
||
## **Requirements**
|
||
|
||
- Python 3.9 or later
|
||
|
||
```bash
|
||
# This step only
|
||
pip install -r ~/Speckle-Scanner/05_disparity/zncc/requirements.txt
|
||
|
||
# Or install everything for the full pipeline
|
||
pip install -r ~/Speckle-Scanner/requirements.txt
|
||
|
||
# Optional GPU acceleration (pick one matching your CUDA version)
|
||
pip install cupy-cuda12x # CUDA 12.x
|
||
pip install cupy-cuda11x # CUDA 11.2–11.8
|
||
```
|
||
|
||
Packages: `numpy`, `opencv-python`, `matplotlib`, `scipy`, `numba`, `open3d`. Without `cupy`, the pipeline falls back to CPU (`numba`).
|
||
|
||
## **Input Requirements**
|
||
|
||
- **Stereo Rectified Images**:
|
||
- Images must be stored in `.png` or `.bmp` format.
|
||
- **Pipeline pairing** (in `02_rect_images/`): `lc_ts1634840093_ck….png` is matched with `rc_ts1634840093_ck….png` on the shared `ts` token; the `ck…` suffix is ignored.
|
||
- Pairs are sorted by timestamp; `--num_images` defaults to **all** pairs; if limited, the **last N** (highest timestamps) are used.
|
||
- Standalone runs may still use separate `left/` and `right/` folders with any naming; pipeline mode uses one folder with `lc_` / `rc_` prefixes.
|
||
- PNG is preferred when both PNG and BMP pairs exist.
|
||
- **Q Matrix File**:
|
||
- A `.cvstore` file containing the Q matrix for 3D reprojection.
|
||
|
||
## **Outputs**
|
||
|
||
- **Disparity Map**: Saved as `Disparity.npy`.
|
||
- **Vertical Shift Map**: Saved as `vertical_shift_map.npy`.
|
||
- **Correlation Map**: Saved as `correlation_map.npy`.
|
||
- **Disparity and Shift Maps with Colorbars**:
|
||
- `Disparity_map_colorbar.png`
|
||
- `vertical_shift_map_colorbar.png`
|
||
- `correlation_map_colorbar.png`
|
||
- **Point Cloud**:
|
||
- Saved as `Point_cloud.ply` in PLY format with depth-based color information.
|
||
|
||
## **Usage**
|
||
|
||
1. Clone the repository and navigate to the project directory.
|
||
2. Prepare your stereo images and paste them in data folder
|
||
Example directory structure:
|
||
```
|
||
|
||
project_directory/
|
||
|
||
.data/
|
||
├── left/
|
||
│ ├── 0.png
|
||
│ ├── 1.png
|
||
│ └── ...
|
||
└── right/
|
||
├── 0.png
|
||
├── 1.png
|
||
└── ...
|
||
|
||
Or
|
||
|
||
.data/
|
||
├── left/
|
||
│ ├── lc00000.bmp
|
||
│ └── lc00001.bmp
|
||
└── right/
|
||
├── rc00000.bmp
|
||
└── rc00001.bmp
|
||
|
||
```
|
||
3. The Q matrix must be provided in a YAML file (e.g., `Q.cvstore`).
|
||
Format example:
|
||
```
|
||
|
||
%YAML:1.0
|
||
---
|
||
Q: !!opencv-matrix
|
||
rows: 4
|
||
cols: 4
|
||
dt: f
|
||
data: [ 1.0, 0.0, 0.0, -5.5863211059570312e+02,
|
||
0.0, 1.0, 0.0, -7.3320780181884766e+02,
|
||
0.0, 0.0, 0.0, 3.3665836719173171e+03,
|
||
0.0, 0.0, 1.4262549294100269e+00, 1.0682440279589944e+03 ]
|
||
|
||
```
|
||
This matrix is essential for converting disparity maps into a 3D point cloud.
|
||
3. Run the script with desired parameters (see below).
|
||
|
||
### **Basic Command**
|
||
|
||
```bash
|
||
# stereo_disparity_main.py auto-selects GPU (CUDA) or CPU.
|
||
|
||
cd ~/Speckle-Scanner/05_disparity/zncc
|
||
|
||
# Default parameters (saves disparity map only)
|
||
python stereo_disparity_main.py \
|
||
--left_dir <path_to_rect_images> \
|
||
--right_dir <path_to_rect_images> \
|
||
--left_prefix lc_ \
|
||
--right_prefix rc_ \
|
||
--q_file <path_to_lc-rc_Q.cvstore> \
|
||
--disp_output_dir <path_to_04_zncc_disp_map> \
|
||
--pcl_output_dir <path_to_06_zncc_pcl>
|
||
|
||
# Troubleshooting mode — saves all outputs (vertical shift, correlation, point cloud)
|
||
python stereo_disparity_main.py \
|
||
--left_dir <path_to_rect_images> \
|
||
--right_dir <path_to_rect_images> \
|
||
--left_prefix lc_ \
|
||
--right_prefix rc_ \
|
||
--q_file <path_to_lc-rc_Q.cvstore> \
|
||
--disp_output_dir <path_to_04_zncc_disp_map> \
|
||
--pcl_output_dir <path_to_06_zncc_pcl> \
|
||
--troubleshooting
|
||
```
|
||
This runs the program with default parameters.
|
||
Same commands for stereo_disparity_cpu.py and stereo_disparity_gpu.py
|
||
|
||
### **Custom Parameters**
|
||
|
||
|
||
You can override any parameter through command-line arguments. Below is an example with some customized parameters:
|
||
|
||
```bash
|
||
python stereo_disparity_main.py \
|
||
--window_size 5 \
|
||
--H_neg_range -165 \
|
||
--H_pos_range 65 \
|
||
--v_neg_range -4 \
|
||
--v_pos_range 0 \
|
||
--zncc_threshold 0.4 \
|
||
--num_images 10 \ # optional: last 10 pairs by timestamp; omit for all pairs
|
||
--noise_filter "open3d" \
|
||
--interpolation True \
|
||
--noise_remove False \
|
||
--method "gaussian" \
|
||
--q_file "./data/Q_matrix.cvstore" \
|
||
--left_dir "./data/left" \
|
||
--right_dir "./data/right" \
|
||
--disp_output_dir "./output/disp" \
|
||
--pcl_output_dir "./output/pcl" \
|
||
--troubleshooting
|
||
```
|
||
|
||
|
||
## **Available Parameters**
|
||
|
||
| Parameter | Default Value | Description |
|
||
| ------------------ | ------------------- | --------------------------------------------------------------------------------------|
|
||
| `--window_size` | `5` | Size of the window for block matching. |
|
||
| `--H_neg_range` | `0` | Horizontal negative disparity range. |
|
||
| `--H_pos_range` | `65` | Horizontal positive disparity range. |
|
||
| `--v_neg_range` | `-4` | Vertical negative disparity range. |
|
||
| `--v_pos_range` | `2` | Vertical positive disparity range. |
|
||
| `--zncc_threshold` | `0.4` | ZNCC threshold for valid matches. |
|
||
| `--num_images` | all pairs | Stereo pairs to use per scan. Default: all `lc_ts*`/`rc_ts*` pairs matched on `ts`. If fewer than available, the **last N** pairs (highest timestamps) are used; `ck*` suffix is ignored. |
|
||
| `--noise_filter` | `open3d` | Post-processing noise filter method (`median`[Disparity Map] or `open3d`[Pointcloud]).|
|
||
| `--interpolation` | `True` | Enable subpixel interpolation (`True` or `False`). |
|
||
| `--noise_remove` | `True` | noise removal using bidirectional disparity estimation (`True` or `False`). |
|
||
| `--method` | `parabolic` | Subpixel interpolation method (`parabolic`, `gaussian`, or `equiangular`). |
|
||
| `--q_file` | `./data1/Q.cvstore` | Path to the Q matrix file. |
|
||
| `--left_dir` | `./data1/left` | Path to the directory containing left images. |
|
||
| `--right_dir` | `./data1/right` | Path to the directory containing right images. |
|
||
| `--left_prefix` | `""` | Filename prefix to filter left images (e.g. `lc_`) when both cameras share a folder. |
|
||
| `--right_prefix` | `""` | Filename prefix to filter right images (e.g. `rc_`) when both cameras share a folder. |
|
||
| `--output_dir` | `./results` | Fallback output directory (used when `--disp_output_dir`/`--pcl_output_dir` are not set). |
|
||
| `--disp_output_dir` | same as `--output_dir` | Directory for disparity map outputs (npy + png). |
|
||
| `--pcl_output_dir` | same as `--output_dir` | Directory for point cloud outputs (PLY + TXT). |
|
||
| `--troubleshooting` | `False` | When set, saves all outputs (vertical shift map, correlation map, point cloud); otherwise only disparity map is saved. |
|
||
|
||
|
||
## **Details of Post-processing noise filter method**
|
||
|
||
- **open3d**: used for denoising point clouds.
|
||
- **median**: used to reduce noise in the disparity map.
|
||
|
||
## **Details of Subpixel Interpolation Methods**
|
||
|
||
- **Parabolic**: Fits a parabola to the ZNCC values and finds the peak.
|
||
- **Gaussian**: Uses the logarithm of ZNCC values to model a Gaussian distribution and find the peak.
|
||
- **Equiangular**: Uses angular interpolation for smoother and more robust subpixel shifts.
|
||
|
||
## **Notes**
|
||
|
||
- Ensure stereo images are rectified before processing.
|
||
- Adjust the disparity ranges (`H_neg_range`, `H_pos_range`, etc.) based on your dataset.
|
||
- If disparity maps appear noisy, enable median filtering or refine the ZNCC threshold.
|
||
|
||
## **Execution Time**
|
||
|
||
- Execution time depends on the number of images, window size, and disparity ranges.
|
||
- Subpixel interpolation adds computational overhead but improves accuracy.
|
||
|
||
## **Point Cloud File**
|
||
|
||
- The generated `.ply` file can be visualized using point cloud tools like MeshLab or CloudCompare.
|
||
- Depth values are normalized and color-coded using the `jet` colormap.
|
||
|
||
## **Troubleshooting**
|
||
|
||
- Ensure the `Q.cvstore` file exists in the specified path.
|
||
- Check file permissions.
|
||
- Ensure the left and right images are rectified and have the same resolution.
|
||
- Reduce the number of images or use a smaller window size.
|
||
- Run on a machine with higher processing power.
|
||
|
||
---
|
||
|
||
## **Pipeline Usage (Automated Path Resolution)**
|
||
|
||
Use `run_zncc_pipeline.py` instead of calling `stereo_disparity_main.py` directly.
|
||
It resolves all paths automatically from the project folder structure and processes
|
||
every scan in a session (or a single scan you name).
|
||
|
||
### **Folder structure assumed**
|
||
|
||
```
|
||
~/Speckle-Scanner_Processing_data/
|
||
└── <project>/
|
||
└── <date>/
|
||
└── <session>/
|
||
├── params_link/
|
||
│ └── lc-rc_Q.cvstore ← Q matrix (input)
|
||
└── <ScanXXXXXX>/
|
||
├── 02_rect_images/ ← lc_ts*.png + rc_ts*.png (input)
|
||
├── 04_zncc_disp_map/ ← disparity .npy + colorbar .png (created)
|
||
└── 06_zncc_pcl/ ← Point_cloud.ply + .txt (created)
|
||
```
|
||
|
||
### **Commands**
|
||
|
||
```bash
|
||
cd ~/Speckle-Scanner/05_disparity/zncc
|
||
|
||
# Process ALL scans in a session (all matched lc/rc pairs per scan)
|
||
python run_zncc_pipeline.py \
|
||
--project Olsen_wings \
|
||
--date 2026-05-12 \
|
||
--session session1
|
||
|
||
# Process ALL sessions on a date (omit --session)
|
||
python run_zncc_pipeline.py \
|
||
--project Olsen_wings \
|
||
--date 2026-05-12
|
||
|
||
# Process a SINGLE scan — only the last 3 pairs (highest timestamps)
|
||
python run_zncc_pipeline.py \
|
||
--project Olsen_wings \
|
||
--date 2026-05-12 \
|
||
--session session1 \
|
||
--scan Scan000001 \
|
||
--num_images 3
|
||
|
||
# Full outputs including point cloud and debug maps (troubleshooting mode)
|
||
python run_zncc_pipeline.py \
|
||
--project Olsen_wings \
|
||
--date 2026-05-12 \
|
||
--session session1 \
|
||
--scan Scan000001 \
|
||
--troubleshooting
|
||
|
||
# Custom ZNCC parameters
|
||
python run_zncc_pipeline.py \
|
||
--project Olsen_wings \
|
||
--date 2026-05-12 \
|
||
--session session1 \
|
||
--window_size 7 \
|
||
--H_neg_range 0 \
|
||
--H_pos_range 80 \
|
||
--v_neg_range -4 \
|
||
--v_pos_range 2 \
|
||
--zncc_threshold 0.5 \
|
||
--noise_filter open3d \
|
||
--method gaussian
|
||
```
|
||
|
||
### **Pipeline parameters**
|
||
|
||
| Parameter | Default | Description |
|
||
|---------------------|---------|------------------------------------------------------------------------------------|
|
||
| `--project` | — | Project name (e.g. `Olsen_wings`) |
|
||
| `--date` | — | Date string (e.g. `2026-05-12`) |
|
||
| `--session` | all | Session name (e.g. `session1`); omit to process **all sessions** on that date |
|
||
| `--scan` | all | Single scan to process (e.g. `Scan000001`); omit to process all scans in session |
|
||
| `--troubleshooting` | off | When set, saves vertical shift map, correlation map, and point cloud PLY/TXT; by default only `disparity.npy` and `Disparity_map_colorbar.png` are saved |
|
||
| `--num_images` | all | Image pairs per scan (matched on `ts`; last N if limited) |
|
||
| All other ZNCC params | see above table | Forwarded directly to `stereo_disparity_main.py` |
|
||
|
||
### **What gets saved**
|
||
|
||
| File | Default | `--troubleshooting` |
|
||
|------|:-------:|:-------------------:|
|
||
| `04_zncc_disp_map/disparity.npy` | ✓ | ✓ |
|
||
| `04_zncc_disp_map/Disparity_map_colorbar.png` | ✓ | ✓ |
|
||
| `04_zncc_disp_map/vertical_shift_map.npy` | | ✓ |
|
||
| `04_zncc_disp_map/vertical_shift_map_colorbar.png` | | ✓ |
|
||
| `04_zncc_disp_map/correlation_map.npy` | | ✓ |
|
||
| `04_zncc_disp_map/correlation_map_colorbar.png` | | ✓ |
|
||
| `06_zncc_pcl/Point_cloud.ply` | | ✓ |
|
||
| `06_zncc_pcl/Point_cloud.txt` | | ✓ |
|
||
|
||
---
|
||
|
||
Contributions and improvements are welcome! Feel free to open an issue or submit a pull request on GitHub.
|
||
|
||
|