ETPNav Reproduction Guide

🌟 Project Introduction (About)

Official open-source repository: MarSaKi/ETPNav Paper link: ETPNav: Evolving Topological Planning for Vision-Language Navigation in Continuous Environments (This paper was published in the top-tier journal IEEE TPAMI in 2024)

Statement: ETPNav is an excellent work proposed by Dong An et al., and currently serves as a strong baseline in the field of continuous environment visual language navigation (VLN-CE). This project and document mainly consist of detailed notes on challenges encountered and manipulation guides I compiled while studying and reproducing the official open-source project of ETPNav. I hope this clear, closed loop reference guide can assist researchers who wish to reproduce this work in the future.

Regarding the ETPNav (Evolving Topological Planning) algorithm itself, it primarily addresses the limitations of existing methods in long-distance planning and obstacle avoidance. It focuses on breaking through two core capabilities:

1. Online mapping and long-distance planning: The model does not need to explore the environment in advance, and a topological map can be dynamically built online using self-organized waypoints along the route. This enables the agent to decouple navigation tasks into high-level planning and low-level control. Based on this topological map, a cross-modal planner (based on Transformer) can better understand the instructions and generate long-distance navigation path plans.
2. Obstacle avoidance control in continuous environments: To address the complexity of continuous environments, ETPNav proposes a robust obstacle-avoidance controller (Tryout) based on trial-and-error heuristics, effectively preventing the agent from getting stuck in a deadlock due to collisions with obstacles in continuous environments.

1. Environment Configuration

This reproduction tutorial uses a Python 3.8 environment. Please follow these steps to configure the virtual environment and related dependencies:

1.1 Create a virtual environment and install PyTorch

bash# 创建并激活 conda 虚拟环境
conda create -n vlnce38 python=3.8
conda activate vlnce38

# 安装 PyTorch (1.9.1+cu111)
（方式一）
pip install torch==1.9.1+cu111 torchvision==0.10.1+cu111 \
-f https://download.pytorch.org/whl/torch_stable.html \
-i https://mirrors.cloud.tencent.com/pypi/simple

（方式二）
## 下载torch 1.9.1+cu111（Python3.8/Linux x86_64）
wget https://mirrors.aliyun.com/pytorch-wheels/cu111/torch-1.9.1%2Bcu111-cp38-cp38-linux_x86_64.whl

## 下载torchvision 0.10.1+cu111（匹配上面的torch版本）
wget https://mirrors.aliyun.com/pytorch-wheels/cu111/torchvision-0.10.1%2Bcu111-cp38-cp38-linux_x86_64.whl

pip install torch-1.9.1+cu111-cp38-cp38-linux_x86_64.whl torchvision-0.10.1+cu111-cp38-cp38-linux_x86_64.whl

1.2 Installing Project Dependencies

Download requirements.txt: Baidu Netdisk (extract code: 8je8)

bashpip install "pip<24.1" setuptools==65.5.0 wheel==0.38.4
python -m pip install -r requirements.txt

1.3 Installing the Habitat simulator

(1) Download the pre-compiled package of Habitat-sim v0.1.7 without a header version: Click to download (2) Download habitat-lab v0.1.7 Click to download

bash# 安装 habitat-sim
conda install habitat-sim-0.1.7-py3.8_headless_linux_856d4b08c1a2632626bf0d205bf46471a99502b7.tar.bz2

# 安装 habitat-lab
cd habitat-lab-0.1.7
pip install -e .

1.4 Fixing Environment Dependencies for Cloud Server (Optional)

When running the Habitat simulator in a cloud container or a lightweight Linux system, issues such as the lack of a graphics rendering library (OpenGL/EGL error) and version conflicts in the C++ underlying libraries (CXXABI error) often occur. Please follow these steps in order to resolve them simultaneously:

1. Install system-level graphics rendering dependencies (to resolve the “missing libEGL library” error)

The underlying 3D engine Magnum of Habitat requires the following libraries to initialize properly:

bashapt-get update
apt-get install -y libopengl0 libgl1-mesa-glx libglib2.0-0 libegl1

2. Upgrade the system-level C++ standard library (to resolve CXXABI errors)

bash# 安装管理 PPA 源的工具
apt-get install -y software-properties-common

# 添加 ubuntu-toolchain-r 测试源
add-apt-repository ppa:ubuntu-toolchain-r/test -y
apt-get update

# 仅升级系统的 C++ 标准库
apt-get install --only-upgrade libstdc++6 -y

1.5 Download the original ETPnav-main code

Offline download

bash# 克隆仓库
git clone https://gh-proxy.org/https://github.com/MarSaKi/ETPNav.git

2. Dataset Download

2.1 Scene Data (Scenes): Matterport3D (MP3D)

Download the Matterport3D scene reconstruction data. There are 90 scenes in total, and the dataset size is approximately 22GB. The final storage path should be: data/scene_datasets/mp3d/{scene}/{scene}.glb

• Method 1: Download the official script (requires Python 2.7) Please visit the official project homepage of Matterport3D, and follow the instructions on the page to obtain the official download script download_mp.py.

  bashpython download_mp.py --task habitat -o data/scene_datasets/mp3d/
  

• Method 2: Quick download from cloud storage

  • Link: Baidu Cloud Storage
  • Extraction code: 4kz6

2.2 Task Data (Episodes): R2R & RxR

Please place the downloaded Episode data in the data/datasets directory.

2.3 Connectivity Graphs

Visualization connectivity graph files:

• Download link: connectivity_graphs.pkl
• Storage path: data/connectivity_graphs.pkl

3. Model weights and pre-trained data

Please download the corresponding encoder weights, predictor weights, and pre-trained feature files as required, and place them in the specified directory.

3.1 Encoder and Component Weights

3.2 Pretraining Data

• R2R pre-trained data: download link $\rightarrow$ stored in pretrain_src/datasets/R2R
• Pre-computed visual features: download link $\rightarrow$ stored in pretrain_src/datasets/img_features
• LXMERT pre-trained weights: download link $\rightarrow$ stored in pretrain_src/datasets/pretrained/LXMERT

3.3 Final Pretrained Weights

If you want to skip the pre-training stage, you can directly download the provided pre-trained weights:

• Download link: Baidu Cloud Storage (Extract code: vfsh)
• Storage path: pretrained/ETP/model_step_82500.pt

The final folder structure is as follows:

textETPNav/
├── assets/
├── bert_config/
│   ├── bert-base-uncased/
│   │   ├── config.json
│   │   ├── pytorch_model.bin
│   │   ├── tokenizer_config.json
│   │   └── vocab.txt
│   └── xlm-roberta-base/
├── data/
│   ├── datasets/
│   │   ├── R2R_VLNCE_v1-2_preprocessed/
│   │   ├── R2R_VLNCE_v1-2_preprocessed_BERTidx/
│   │   └── RxR_VLNCE_v0_enc_xlmr/
│   ├── ddppo-models/
│   │   └── gibson-2plus-resnet50.pth
│   ├── scene_datasets/
│   │   └── mp3d/
│   ├── wp_pred/
│   │   ├── check_cwp_bestdist_hfov79
│   │   └── check_cwp_bestdist_hfov90
│   └── connectivity_graphs.pkl
├── habitat_extensions/
├── precompute_img_features/
├── pretrain_src/
│   ├── datasets/
│   │   ├── pretrained/
│   │   │   └── LXMERT/
│   │   │       └── model_LXRT.pth
│   │   └── R2R/
│   ├── img_features/
│   │   ├── CLIP-ViT-B-32-views-habitat.hdf5
│   │   └── ddppo_resnet50_depth_features.hdf5
│   ├── pretrain_src/
│   └── run_pt/
├── pretrained/
│   ├── ETP/
│   │   └── model_step_82500.pt
├── run_r2r/
├── run_rxr/
├── vlnce_baselines/
├── .gitignore
├── environment.yaml
├── LICENSE
├── README.md
└── run.py

4. Code Execution

4.1 Pretraining

(If you want to skip this step, you can use the pre-trained weights downloaded earlier and directly perform fine-tuning or evaluation.)

1. Modify the reference path: If the external network is unreachable, replace the BERT weight address with a local one.

   

2. Adjust the number of GPUs: Adjust the number of GPUs for distributed training according to actual hardware conditions.

   

3. Start pre-training: This step mainly performs the two pre-training tasks: MLM (Masked Language Modeling) and SAP.

   bashCUDA_VISIBLE_DEVICES=0 bash pretrain_src/run_pt/run_r2r.bash 233
   

   The training logs will be saved in /ETPNav-main/pretrained/r2r_ce/mlm.sap_habitat_depth/logs/log.txt. You can select the best pre-trained weights based on the actual evaluation metrics:

   

4.2 Fine-tuning (Finetuning)

1. Configure pre-trained weight path: Point the loading path in the script to pretrained/ETP/model_step_82500.pt.

   
   

2. Adjust the number of GPUs: Adjust the number of GPUs for distributed training according to the actual hardware conditions.

   
   

3. Initial tuning: Note: It takes approximately 1.5 days for a single RTX 4090 card to complete the tuning process.

   bashCUDA_VISIBLE_DEVICES=0 bash run_r2r/main.bash train 2333  # training
   

4. Disable TensorFlow warnings (optional): export TF_ENABLE_ONEDNN_OPTS=0 export TF_CPP_MIN_LOG_LEVEL=2

4.3 Testing and Evaluation (Eval & Testing)

Evaluation

Run the following command for evaluation:

bashCUDA_VISIBLE_DEVICES=0 bash run_r2r/main.bash eval  2333  # evaluation

The evaluation metrics obtained in the eval mode are as follows:

Add visualization (optional)

Here, you can choose to save the test video locally or use TensorBoard for visualization.

If you select the disk mode and run the evaluation code again, the navigation video will be saved to: ETPNav-main/data/logs/video/release_r2r/*

Here are some visual video demonstrations for specific scenarios:

https://github.com/user-attachments/assets/e5797cce-265b-4e95-9ba4-f732c1a2813c

Reasoning

Run the following command for reasoning:

bashCUDA_VISIBLE_DEVICES=0 bash run_r2r/main.bash inference  2333  # inference

5. References and Acknowledgments (Acknowledgements)

This reproduction tutorial and code primarily refer to and rely on the following excellent open-source projects. We would like to express our sincere gratitude to the original authors for their open-source spirit:

• ETPNav Official Repository: MarSaKi/ETPNav —— The core reproduction goal of this project.
• VLN-CE Official Framework: jacobkrantz/VLN-CE —— Provides the infrastructure and evaluation standards for continuous environment navigation.
• Discrete-Continuous-VLN: YicongHong/Discrete-Continuous-VLN —— Offers important references for pre-training, waypoint prediction, and other modules.