ActiveFly-Bench: Aligning Embodied Question Answering with Vision-Language-Action for Aerial Embodied Perception

TODO list

• Dataset Released
  • Simulated Datasets
  • Real-world Datasets
• Pretrained Weights Released
• Simulation Deployment Released
  • OpenVLA-based Agent
  • OpenPi-based Agent
• Real-world Deployment Released
  • OpenVLA-based Agent
  • OpenPi-based Agent

1. Environment Setup

OpenVLA-based Agent

Use this environment for OpenVLA training, rollout, and FLUC evaluation.

conda create -n activefly-openvla python=3.11 -y
conda activate activefly-openvla

pip install -r requirements-openvla.txt

Install flash-attn after the main requirements are installed.

pip install packaging ninja
pip install flash-attn==2.5.5 --no-build-isolation

flash-attn can also be installed from a prebuilt wheel.

OpenPi-based Agent

Use this environment for OpenPi training, rollout, and FLUC evaluation.

conda create -n activefly-pi python=3.11 -y
conda activate activefly-pi

pip install -r requirements-pi.txt

Dataset

Download the packaged ActiveFly train/test archives from Hugging Face Datasets.

Simulated Environments

Download the simulation assets from Hugging Face or EmbodiedCity.

Then see Configuration to set API keys and external dataset paths.

2. Configuration

Configure environment variables before running evaluation, the pipeline, or the Web UI. Use export in your shell, or place variables in a local .env file (loaded by run_pipeline.sh and run_eval.sh).

Dataset Paths Config (set on your machine)

GT and rollout frames are not in the repo. Point these to where you downloaded ActiveFly on your machine:

Variable	Purpose
ACTIVEFLY_QA_DIR	Air-EQA / OBP metadata directory (default: ./Air-EQA)
ACTIVEFLY_FLUC_SIM_DIR	GT trajectory frames (sim)
ACTIVEFLY_FLUC_REAL_DIR	GT trajectory frames (real-world)
ACTIVEFLY_ROLLOUT_DIR	Rollout frame root (pi/, openvla/ subdirs)
ACTIVEFLY_PI0_DIR	Optional Pi0 rollout output override (default: $ACTIVEFLY_ROLLOUT_DIR/pi)
ACTIVEFLY_OPENVLA_DIR	Optional OpenVLA rollout output override (default: $ACTIVEFLY_ROLLOUT_DIR/openvla)

Example:

export ACTIVEFLY_FLUC_SIM_DIR=<path_to_ActiveFly_on_your_machine>
export ACTIVEFLY_FLUC_REAL_DIR=<path_to_ActiveFly_on_your_machine>/realworld_indoor
export ACTIVEFLY_PI0_DIR=<pi0_rollout_or_checkpoint_dir>
export ACTIVEFLY_OPENVLA_DIR=<openvla_rollout_dir>
export ACTIVEFLY_ROLLOUT_DIR=<rollout_root>   # optional; defaults to pi/ and openvla/ under it

Create a local .env with the variables above (and API keys), then run run_pipeline.sh / run_eval.sh (they load .env automatically). pipeline_web.py also loads .env on start.

Output Directory (Defaults)

These paths are default output directory in this repo:

Path	Purpose
eval/api_results/	VLM answer JSONL files
eval/	Collected summary tables (eval_items.csv, eval.json, …)
eval/pipeline_web/	Static debug pages built by run_eval.sh

API Keys

Air-EQA/test_qa.py reads API credentials from environment variables:

• OpenAI: OPENAI_API_KEY or ACTIVEFLY_OPENAI_API_KEY
• Gemini: GEMINI_API_KEY, GOOGLE_API_KEY, or ACTIVEFLY_GEMINI_API_KEY
• Qwen: DASHSCOPE_API_KEY, QWEN_API_KEY, or ACTIVEFLY_QWEN_API_KEY

export DASHSCOPE_API_KEY=<your_qwen_api_key>

3. ActiveFly Workflow

3.1 Air-EQA and OBP Evaluation

Run the following command to evaluate a VLM to answer EQA or OBP from GT or rollout frames.

python Air-EQA/test_qa.py --model <gpt|gemini|qwen> --traj <gt|openvla|pi> --dataset <EQA|OBP>

Notes:

• --traj gt reads the ground-truth trajectory frames.
• --traj openvla reads rollout frames under openvla/ in your configured rollout directory.
• --traj pi reads rollout frames under pi/ in your configured rollout directory.
• VLM answers are written to eval/api_results/ by default. See Configuration for external dataset paths.

Example:

python Air-EQA/test_qa.py --model qwen --traj openvla --dataset EQA
python Air-EQA/test_qa.py --model qwen --traj pi --dataset OBP

3.2 FLUC Evaluation

Download OpenVLA or OpenPi weights. Run the server and client in separate terminals.

OpenVLA

Server:

Start the OpenVLA policy server before launching the FLUC rollout client.

conda activate activefly-openvla
cd FLUC-OpenVLA/run

python vla-scripts/run_act_server.py \
  --model-path <model_checkpoint> \
  --http-port 5007 \
  --gpu-id 0 \
  --unnorm-key sim

Client:

Run the rollout client after the OpenVLA server is already serving requests.

conda activate activefly-openvla
cd FLUC-OpenVLA/eval

python run_act_client.py \
  -r <data_root>/ActiveFly_Test/<split> \
  -o ../outputs/openvla_activefly_eval \
  --server_port 5007 \
  --instruction_type instruction

Metrics:

Run the shared metric script after the OpenVLA rollout.

python ../../scripts/metric.py \
  <data_root>/ActiveFly_Test/<split> \
  ../outputs/openvla_activefly_eval \
  --metric all \
  --output-dir ./eval_outputs

OpenPi

Server:

Start the OpenPi policy server before launching the FLUC rollout client.

conda activate activefly-pi
cd FLUC-OpenPi

python run/scripts/run_pi_server.py \
  --policy-dir <checkpoint_step_dir> \
  --port 8000

Use the concrete checkpoint step directory, for example <checkpoint_root>/<exp_name>/<step>.

Client:

Run the rollout client after the OpenPi server is already serving requests.

conda activate activefly-pi
cd FLUC-OpenPi

python eval/run_rollout.py \
  -r <data_root>/ActiveFly_Test/<split> \
  -o <openpi_output_dir> \
  --server_port 8000 \
  --instruction_type instruction

Metrics:

Run the shared metric script after the OpenPi rollout.

python ../scripts/metric.py \
  <data_root>/ActiveFly_Test/<split> \
  <openpi_output_dir> \
  --metric all \
  --traj-len-thresh 5 \
  --output-dir <metric_output_dir>

The summary reports nDTW, NE, SR, and OSR.

Run metric.py only after a rollout client has finished writing trajectory outputs. It is the post-rollout evaluation step for OpenVLA and OpenPi.

3.3 Workflow Demo

Run the whole embodied perception test that comprise EQA -> OBP -> rollout -> EQA pipeline. Make sure the simulated environment and the VLA server are launched. Set API keys first.

OpenVLA:

conda activate activefly-openvla

python scripts/test_activefly_workflow.py \
  --eqa-idx 0 \
  --model qwen \
  --server-ip 127.0.0.1 \
  --server-port 5007

OpenPi:

conda activate activefly-pi

python scripts/test_activefly_openpi_workflow.py \
  --eqa-idx 0 \
  --model qwen \
  --server-ip 127.0.0.1 \
  --server-port 8000

Supported VLMs: qwen, gemini, gpt.

3.4 Collect Evaluation Results

After EQA, OBP, and FLUC runs finish, use this script to summarize all saved outputs into eval/ and build static debug pages under eval/pipeline_web/.

bash run_eval.sh

Then open the Web UI in debug mode to browse the results.

4. Training VLA Models

Use this section when you want to finetune the VLA models on the current ActiveFly training set instead of only running evaluation.

4.1 Train OpenVLA

Use the OpenVLA finetuning launcher from FLUC-OpenVLA.

The launcher reads trajectory folders directly from ActiveFly_Train, so no extra conversion step is required if your current data is already organized in that format.

conda activate activefly-openvla
cd FLUC-OpenVLA

sh run/vla-scripts/finetune_act.sh

To train from a specific pretrained checkpoint or data root, adjust the arguments inside FLUC-OpenVLA/run/vla-scripts/finetune_act.sh or call FLUC-OpenVLA/run/vla-scripts/finetune_act.py directly with your custom arguments.

4.2 Train OpenPi

OpenPi finetuning first converts ActiveFly_Train to LeRobot format, then starts training. By default both OpenVLA and OpenPi read datasets/ActiveFly_Train; set ACTIVEFLY_FLUC_SIM_DIR to override.

conda activate activefly-pi
cd FLUC-OpenPi

sh run/scripts/finetune_pi.sh

The launcher writes checkpoints under the data root configured in FLUC-OpenPi/run/scripts/finetune_pi.sh and computes missing norm stats before training starts.

To train from a specific pretrained checkpoint or data root, adjust the arguments inside FLUC-OpenPi/run/scripts/finetune_pi.sh or call FLUC-OpenPi/run/scripts/finetune_pi.py directly with your custom arguments.

5. Web UI

Interactive runner / debug UI:

Launch the interactive web UI for manual debugging and pipeline inspection. Make sure dataset paths are set before launching.

python pipeline_web.py

Static debug view:

Use the static mode when you only need to inspect saved results without the interactive controls. Pages are served from eval/pipeline_web/ (built by run_eval.sh).

python pipeline_web.py --static

Debug mode reads trajectory frames from your configured dataset directories and model answers from eval/api_results/, then displays them as GIFs in the page. Run mode lets you execute the pipeline on one item from the browser; after the job finishes, open that item's detail page to view the updated videos and model answers.

5. Real-world Deployment

We provide the sample deployment code for real UAVs in ActiveFly-Client-ROS. Note that this repository only includes the VLA client code. The UAV flight control and path planning modules are not included. Please refer to here for more details.

6. Acknowledgement

We sincerely thank the following projects for their excellent work: UAV-Flow, OpenVLA, OpenPi.

License

This project is released under the CC BY 4.0 license.