fix(evo1): update Evo1 model to higher effience

这个提交包含在:
t0mo0n
2026-03-29 22:34:05 +08:00
父节点 c3b5cf4ee2
当前提交 7782473872
共有 72 个文件被更改,包括 286 次插入 和 307 次删除

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fd4a3d6d0266205fe3f725f048645be66592c23564db22e08b0807d5569b2c64
size 607384

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9ee6e1d67c0241f42a0dbaf5c676d19230375ed078ee1943bb8c4c37ab303d14
size 447784

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0d37675b699b643f2e15348de99aed535fac10dc5f09a79b8e4b0acfaf4e0378
size 3658384

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e8ae51f17c0c7f6d67b753922e2cc269f5e8372a65734f859231639a64fe8fad
size 1845784

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:efa33022976218f246f736a6133494b5d0e5486e8b50b52900c42f285ae85558
size 877984

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a1c69d1c7f6f8c60e2c82331c0f444725200ed4053d3e5ec23aa82cb5db5b29f
size 833684

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:88374973bc3bd9e7389b5853cd6e555af9731c30b658b1ee6f85be34a126cb92
size 58884

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a93c1e3fc245d9d336a427fcc506cb1fc15615268053392b249816f90c011e50
size 8273

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:0861e122e406f891a57a550551ad1ce0ef4ca5babf58f476cf8914395e144319
size 162084

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e78dee99bc84857c09183463f3fb3fdea292ea7570f1bfb0001a79674bb69d45
size 2323489

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:97e9079580af6e423317c33b2a0f4fe0f6dbc67c3e1d9e6f4836c87c6c8f34a2
size 14480060

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:db5b39eef745655a67e57870d8ee56f26a03651d2161635ca2ec3c58c9269907
size 5098355

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ccc08d3ef1cb8bdfb8ab5395ff65893d13386ad526d7bc954d468b7dc885d557
size 2037373

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:78718679ffdf9d81130047c4267ed66a081315eff36f216833bc3781c6241417
size 2155609

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:486dddcd07f072b2e3978087dc2f9b641a91722bccb1bcb3421c4c7ce64a39ed
size 1001764

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d6dc6d31b0fa56341b2323fee5e8a375c2500ae6fb02e31fb52e8e917b608439
size 8169

查看文件

@ -40,19 +40,36 @@
- 删除了原始 Evo1 里永假的 `action_dim != horizon * per_action_dim` 检查。
- `InternVL3Embedder` 不再把 `device` 固定保存成字符串语义,而是运行时从模型参数读取实际 device。
## 相较原始 Evo1 的性能优化(已落地)
### 1. VLM 图文融合改为 batch 处理(主要收益)
- 由逐样本循环调用改为整批处理,减少 Python 循环和重复前向开销,提升吞吐。
- 优化位置:
- `src/lerobot/policies/evo1/modeling_evo1.py`:`_compute_fused_tokens`
- `src/lerobot/policies/evo1/evo1_model.py`:`get_vl_embeddings`
- `src/lerobot/policies/evo1/internvl3_embedder.py`:`_build_multimodal_prompt`、`_prepare_and_fuse_embeddings`、`get_fused_image_text_embedding_from_tensor_images`
### 2. 图像预处理避免不必要的 PIL 转换(主要收益)
- 对已是 `[3, image_size, image_size]` 的 Tensor 走纯 Tensor 快路径,直接归一化,不再强制 Tensor->PIL->Tensor。
- 优化位置:
- `src/lerobot/policies/evo1/internvl3_embedder.py`:`_preprocess_images`
### 3. 次要优化(收益次于以上两项)
- `repeat` 改 `expand`/广播,减少中间张量分配。
- 位置:`src/lerobot/policies/evo1/flow_matching.py`:`MultiEmbodimentActionEncoder.forward`
- 推理阶段时间嵌入预计算,减少循环内重复计算。
- 位置:`src/lerobot/policies/evo1/flow_matching.py`:`FlowmatchingActionHead.get_action`
- 强化 dtype/device 对齐,减少隐式 cast。
- 位置:`src/lerobot/policies/evo1/flow_matching.py`、`src/lerobot/policies/evo1/internvl3_embedder.py`
## 当前仍保留的原始 Evo1 遗留问题
下面这些问题在当前 Evo-RL 端口里仍然基本沿用原始 Evo1 的实现,后续如果要继续打磨,建议由 Evo1 本体维护侧统一评估:
### 1. VLM forward 仍为逐 sample 串行
- 现象:batch 内逐样本调用 VLM。
- 影响:
- 训练吞吐较低,尤其 stage2 更明显。
- 归因:原始 Evo1 即如此实现。
- 建议:后续如有性能优化需求,可考虑 batched VLM forward。
### 2. 配置链较绕
### 1. 配置链较绕
- 现象:`dict -> SimpleNamespace` 形式在模型层继续传递配置。
- 影响:

查看文件

@ -57,15 +57,34 @@ class EVO1(nn.Module):
def get_vl_embeddings(
self,
images: list[Image.Image | torch.Tensor],
images: list[list[Image.Image | torch.Tensor]],
image_mask: torch.Tensor,
prompt: str = "",
prompt: str | list[str] = "",
return_cls_only: bool | None = None,
) -> torch.Tensor:
if return_cls_only is None:
return_cls_only = self.return_cls_only
if not images:
raise ValueError("EVO1 expects at least one image per sample.")
raise ValueError("EVO1 expects at least one batch item.")
if not isinstance(images[0], list):
raise TypeError("EVO1 expects batched images as list[list[Image|Tensor]].")
batch_size = len(images)
if image_mask.ndim == 1:
image_mask = image_mask.unsqueeze(0)
if image_mask.ndim != 2:
raise ValueError(f"EVO1 expects image_mask with shape [B, V], got {tuple(image_mask.shape)}")
if image_mask.shape[0] != batch_size:
raise ValueError(
f"EVO1 batch mismatch: num_images_batch={batch_size} image_mask_batch={image_mask.shape[0]}"
)
if isinstance(prompt, list) and len(prompt) != batch_size:
raise ValueError(
f"EVO1 batch mismatch: num_prompts={len(prompt)} num_images_batch={batch_size}"
)
return self.embedder.get_fused_image_text_embedding_from_tensor_images(
image_tensors=images,
image_mask=image_mask,
@ -83,17 +102,40 @@ class EVO1(nn.Module):
if state_tensor.ndim == 1:
state_tensor = state_tensor.unsqueeze(0)
elif state_tensor.ndim == 3:
state_tensor = state_tensor[:, -1]
elif state_tensor.ndim != 2:
raise ValueError(f"EVO1 expects state tensor with shape [B, D], got {tuple(state_tensor.shape)}")
return state_tensor.to(self._device)
def predict_action(
self,
fused_tokens: torch.Tensor,
state: torch.Tensor,
state: torch.Tensor | None,
actions_gt: torch.Tensor | None = None,
action_mask: torch.Tensor | None = None,
embodiment_ids: torch.Tensor | None = None,
):
batch_size = fused_tokens.shape[0]
if state is not None and state.shape[0] != batch_size:
raise ValueError(f"EVO1 batch mismatch: state_batch={state.shape[0]} fused_batch={batch_size}")
if actions_gt is not None and actions_gt.shape[0] != batch_size:
raise ValueError(f"EVO1 batch mismatch: actions_gt_batch={actions_gt.shape[0]} fused_batch={batch_size}")
if action_mask is not None and action_mask.shape[0] != batch_size:
raise ValueError(f"EVO1 batch mismatch: action_mask_batch={action_mask.shape[0]} fused_batch={batch_size}")
if embodiment_ids is not None:
if embodiment_ids.ndim == 0:
embodiment_ids = embodiment_ids.unsqueeze(0)
elif embodiment_ids.ndim > 1:
embodiment_ids = embodiment_ids[:, -1]
if embodiment_ids.shape[0] != batch_size:
raise ValueError(
f"EVO1 batch mismatch: embodiment_ids_batch={embodiment_ids.shape[0]} fused_batch={batch_size}"
)
if actions_gt is None:
return self.action_head.get_action(
fused_tokens,
@ -112,9 +154,9 @@ class EVO1(nn.Module):
@torch.no_grad()
def run_inference(
self,
images: list[Image.Image | torch.Tensor],
images: list[list[Image.Image | torch.Tensor]],
image_mask: torch.Tensor,
prompt: str,
prompt: str | list[str],
state_input: list | torch.Tensor,
return_cls_only: bool | None = None,
action_mask: torch.Tensor | None = None,
@ -127,6 +169,10 @@ class EVO1(nn.Module):
return_cls_only=return_cls_only,
)
state_tensor = self.prepare_state(state_input)
if state_tensor.shape[0] != fused_tokens.shape[0]:
raise ValueError(
f"EVO1 batch mismatch: state_batch={state_tensor.shape[0]} fused_batch={fused_tokens.shape[0]}"
)
return self.predict_action(
fused_tokens,
state_tensor,

查看文件

@ -46,8 +46,13 @@ class CategorySpecificLinear(nn.Module):
def forward(self, x: torch.Tensor, category_id: torch.LongTensor):
if self.num_categories <= 1:
if x.dtype != self.linear.weight.dtype:
x = x.to(dtype=self.linear.weight.dtype)
return self.linear(x)
if x.dtype != self.weight.dtype:
x = x.to(dtype=self.weight.dtype)
orig_shape = x.shape
x_flat = x.reshape(-1, orig_shape[-1])
if category_id.dim() == 0:
@ -98,14 +103,14 @@ class MultiEmbodimentActionEncoder(nn.Module):
x = action_seq.reshape(batch_size * horizon, action_dim)
if category_id.dim() == 0:
cat_ids = category_id.repeat(horizon * batch_size)
cat_ids = category_id.expand(horizon * batch_size)
else:
cat_ids = category_id.unsqueeze(1).repeat(1, horizon).reshape(batch_size * horizon)
cat_ids = category_id.unsqueeze(1).expand(batch_size, horizon).reshape(batch_size * horizon)
out = self.activation(self.W1(x, cat_ids))
pos_enc = self.pos_encoding(horizon).to(out.device)
pos_enc = pos_enc.repeat(batch_size, 1, 1).reshape(batch_size * horizon, -1)
out = out + pos_enc
pos_enc = self.pos_encoding(horizon).to(device=out.device, dtype=out.dtype)
out = out.view(batch_size, horizon, -1) + pos_enc
out = out.view(batch_size * horizon, -1)
out = self.activation(self.W2(out, cat_ids))
out = self.W3(out, cat_ids)
out = out.view(batch_size, horizon, self.embed_dim)
@ -179,8 +184,8 @@ class FlowmatchingActionHead(nn.Module):
)
self.embed_dim = embed_dim
self.horizon = horizon
self.per_action_dim = config.per_action_dim
self.action_dim = config.action_dim
self.per_action_dim = getattr(self.config, "per_action_dim", per_action_dim)
self.action_dim = getattr(self.config, "action_dim", action_dim)
self.time_pos_enc = SinusoidalPositionalEncoding(embed_dim, max_len=1000)
@ -237,7 +242,12 @@ class FlowmatchingActionHead(nn.Module):
action_mask: torch.Tensor = None,
):
if actions_gt is None:
return self.get_action(fused_tokens, state=state, embodiment_id=embodiment_id)
return self.get_action(
fused_tokens,
state=state,
embodiment_id=embodiment_id,
action_mask=action_mask,
)
batch_size = fused_tokens.size(0)
device = fused_tokens.device
@ -256,7 +266,7 @@ class FlowmatchingActionHead(nn.Module):
)
time_index = (t * 1000).long()
time_emb = self.time_pos_enc(1000)[:, time_index, :].squeeze(0)
time_emb = self.time_pos_enc(1000)[:, time_index, :].squeeze(0).to(dtype=context_tokens.dtype)
actions_gt_seq = actions_gt
@ -271,11 +281,12 @@ class FlowmatchingActionHead(nn.Module):
if self.horizon > 1:
noise_seq = noise.view(batch_size, self.horizon, self.per_action_dim)
actions_gt_seq = actions_gt_seq.view(batch_size, self.horizon, self.per_action_dim)
else:
noise_seq = noise.unsqueeze(1)
noise_seq = noise.view(batch_size, 1, self.per_action_dim)
actions_gt_seq = actions_gt_seq.view(batch_size, 1, self.per_action_dim)
t_broadcast = t.view(batch_size, 1, 1) if self.horizon > 1 else t.view(batch_size, 1)
t_broadcast = t.view(batch_size, 1, 1)
action_intermediate_seq = (1 - t_broadcast) * noise_seq + t_broadcast * actions_gt_seq
if self.horizon > 1 and self.action_encoder is not None:
@ -330,7 +341,7 @@ class FlowmatchingActionHead(nn.Module):
else:
per_action_dim = action_dim_total
action = torch.rand(batch_size, action_dim_total, device=device) * 2 - 1
action = torch.rand(batch_size, action_dim_total, device=device, dtype=context_tokens.dtype) * 2 - 1
if self.horizon > 1:
action_seq = action.view(batch_size, self.horizon, per_action_dim)
@ -341,19 +352,20 @@ class FlowmatchingActionHead(nn.Module):
action_mask = action_mask.to(dtype=action_seq.dtype, device=action_seq.device)
if action_mask.shape != action_seq.shape:
raise ValueError(f"action_mask shape {action_mask.shape} != noise shape {action_seq.shape}")
flat_action_mask = action_mask.view(batch_size, -1)
action_seq = action_seq * action_mask
num_steps = int(getattr(self.config, "num_inference_timesteps", 32))
dt = 1.0 / num_steps
for i in range(num_steps):
t = i / num_steps
t_vals = torch.arange(num_steps, device=device, dtype=torch.float32) / num_steps
time_indices = (t_vals * 1000).long()
all_time_embs = self.time_pos_enc(1000)[0, time_indices, :].to(device=device, dtype=context_tokens.dtype)
time_index = int(t * 1000)
time_emb = self.time_pos_enc(1000)[:, time_index, :].to(device).squeeze(0)
time_emb = time_emb.unsqueeze(0).repeat(batch_size, 1)
for i in range(num_steps):
time_emb = all_time_embs[i].unsqueeze(0).expand(batch_size, -1)
if self.horizon > 1 and self.action_encoder is not None:
action_seq = action_seq * action_mask
action_tokens = self.action_encoder(action_seq, embodiment_id)
else:
if self.single_action_proj is None:
@ -378,7 +390,7 @@ class FlowmatchingActionHead(nn.Module):
pred = self.mlp_head(x_pooled, embodiment_id)
action = action + dt * pred
action = action * action_mask.view(batch_size, -1)
action = action * flat_action_mask
if self.horizon > 1:
action_seq = action.view(batch_size, self.horizon, per_action_dim)

查看文件

@ -128,50 +128,94 @@ class InternVL3Embedder(nn.Module):
self.img_context_token_id = self.tokenizer.convert_tokens_to_ids(IMG_CONTEXT_TOKEN)
def _preprocess_images(
self, image_tensors: list[Image.Image | torch.Tensor]
) -> tuple[torch.Tensor, list[int]]:
self, image_tensors_batch: list[list[Image.Image | torch.Tensor]]
) -> tuple[torch.Tensor, list[list[int]]]:
pixel_values_list = []
for image in image_tensors:
if isinstance(image, torch.Tensor):
image = to_pil_image(image)
tiles = dynamic_preprocess(image, image_size=self.image_size)
tile_tensors = torch.stack([self.transform(t) for t in tiles]) # (T_i, 3, 448, 448)
pixel_values_list.append(tile_tensors)
batch_num_tiles_list = []
model_dtype = next(self.model.parameters()).dtype
mean = torch.tensor(IMAGENET_MEAN, device=self.device, dtype=torch.float32).view(1, 3, 1, 1)
std = torch.tensor(IMAGENET_STD, device=self.device, dtype=torch.float32).view(1, 3, 1, 1)
pixel_values = torch.cat(pixel_values_list, dim=0).to(dtype=torch.bfloat16, device=self.device)
num_tiles_list = [pv.shape[0] for pv in pixel_values_list]
for image_tensors in image_tensors_batch:
num_tiles_list = []
for image in image_tensors:
if isinstance(image, torch.Tensor):
if image.ndim == 3 and image.shape[0] == 3 and image.shape[1] == self.image_size and image.shape[2] == self.image_size:
image_t = image.to(device=self.device)
if not torch.is_floating_point(image_t):
image_t = image_t.to(torch.float32) / 255.0
else:
image_t = image_t.to(torch.float32)
if float(image_t.max().item()) > 1.0:
image_t = image_t / 255.0
tile_tensors = (image_t.unsqueeze(0) - mean) / std
else:
image = to_pil_image(image.detach().cpu())
tiles = dynamic_preprocess(image, image_size=self.image_size)
tile_tensors = torch.stack([self.transform(t) for t in tiles]).to(device=self.device)
else:
tiles = dynamic_preprocess(image, image_size=self.image_size)
tile_tensors = torch.stack([self.transform(t) for t in tiles]).to(device=self.device)
return pixel_values, num_tiles_list
tile_tensors = tile_tensors.to(dtype=model_dtype)
pixel_values_list.append(tile_tensors)
num_tiles_list.append(tile_tensors.shape[0])
batch_num_tiles_list.append(num_tiles_list)
def _build_multimodal_prompt(self, num_tiles_list: list[int], text_prompt: str) -> str:
prompt = ""
for i in range(len(num_tiles_list)):
prompt += f"Image-{i + 1}: <image>\n"
prompt += text_prompt.strip()
if pixel_values_list:
pixel_values = torch.cat(pixel_values_list, dim=0)
else:
pixel_values = torch.empty(
0,
3,
self.image_size,
self.image_size,
dtype=model_dtype,
device=self.device,
)
for tile_count in num_tiles_list:
token_count = self.model.num_image_token * tile_count
image_tokens = IMG_START_TOKEN + IMG_CONTEXT_TOKEN * token_count + IMG_END_TOKEN
prompt = prompt.replace("<image>", image_tokens, 1)
return pixel_values, batch_num_tiles_list
return prompt
def _build_multimodal_prompt(
self,
batch_num_tiles_list: list[list[int]],
text_prompts: list[str],
) -> list[str]:
if len(batch_num_tiles_list) != len(text_prompts):
raise ValueError(
f"InternVL3 batch mismatch: num_image_batches={len(batch_num_tiles_list)} num_text_prompts={len(text_prompts)}"
)
prompts = []
for num_tiles_list, text_prompt in zip(batch_num_tiles_list, text_prompts, strict=True):
prompt_segments = []
for i, tile_count in enumerate(num_tiles_list):
token_count = self.model.num_image_token * tile_count
image_tokens = IMG_START_TOKEN + IMG_CONTEXT_TOKEN * token_count + IMG_END_TOKEN
prompt_segments.append(f"Image-{i + 1}: {image_tokens}\n")
prompts.append("".join(prompt_segments) + text_prompt.strip())
return prompts
def _prepare_and_fuse_embeddings(
self, prompt: str, vit_embeds: torch.Tensor, image_mask: torch.Tensor, num_tiles_list: list[int]
self,
prompts: list[str],
vit_embeds: torch.Tensor,
image_masks: torch.Tensor,
batch_num_tiles_list: list[list[int]],
) -> tuple[torch.Tensor, torch.Tensor]:
untruncated_ids = self.tokenizer(prompt, return_tensors="pt").input_ids
true_sequence_length = untruncated_ids.shape[1]
untruncated_ids = self.tokenizer(prompts, padding=False, truncation=False)["input_ids"]
true_sequence_length = max((len(ids) for ids in untruncated_ids), default=0)
if true_sequence_length > self.max_text_length:
logging.warning(
"InternVL3 prompt truncated: max_length=%s actual_length=%s prompt_prefix=%r",
"InternVL3 prompt truncated in batch: max_length=%s actual_max_length=%s",
self.max_text_length,
true_sequence_length,
prompt[:100],
)
model_inputs = self.tokenizer(
prompt,
prompts,
return_tensors="pt",
padding="max_length",
truncation=True,
@ -181,61 +225,113 @@ class InternVL3Embedder(nn.Module):
attention_mask = model_inputs["attention_mask"]
img_token_mask = input_ids == self.img_context_token_id
img_token_locations = torch.where(img_token_mask)[1]
input_embeds = self.model.language_model.get_input_embeddings()(input_ids).clone()
batch_size, seq_len, channels = input_embeds.shape
input_embeds = input_embeds.reshape(batch_size * seq_len, channels)
input_ids = input_ids.reshape(batch_size * seq_len)
selected = input_ids == self.img_context_token_id
batch_size, _, channels = input_embeds.shape
vit_embeds = vit_embeds.reshape(-1, channels).to(device=input_embeds.device, dtype=input_embeds.dtype)
n_token = int(selected.sum().item())
if vit_embeds.shape[0] < n_token:
raise ValueError(
f"InternVL3 produced fewer image tokens than expected: got {vit_embeds.shape[0]}, need {n_token}"
)
input_embeds[selected] = vit_embeds[:n_token]
tokens_per_tile = self.model.num_image_token
vit_idx = 0
actual_vis_tokens_list = img_token_mask.sum(dim=1).tolist()
current_token_idx = 0
for i in range(len(image_mask)):
num_tiles_for_this_image = num_tiles_list[i]
num_tokens_for_this_image = num_tiles_for_this_image * tokens_per_tile
for batch_idx in range(batch_size):
expected_vis_tokens = sum(batch_num_tiles_list[batch_idx]) * tokens_per_tile
actual_vis_tokens = int(actual_vis_tokens_list[batch_idx])
if actual_vis_tokens > expected_vis_tokens:
raise ValueError(
"InternVL3 detected more image placeholder tokens than expected in prompt construction: "
f"batch_idx={batch_idx}, actual={actual_vis_tokens}, expected={expected_vis_tokens}"
)
if not image_mask[i]:
start_idx = img_token_locations[current_token_idx]
end_idx = start_idx + num_tokens_for_this_image
if vit_idx + expected_vis_tokens > vit_embeds.shape[0]:
raise ValueError(
"InternVL3 produced fewer image tokens than expected for batch fusion: "
f"need_up_to={vit_idx + expected_vis_tokens}, got={vit_embeds.shape[0]}"
)
attention_mask[0, start_idx:end_idx] = 0
item_vit_embeds = vit_embeds[vit_idx : vit_idx + expected_vis_tokens]
vit_idx += expected_vis_tokens
current_token_idx += num_tokens_for_this_image
if actual_vis_tokens > 0:
input_embeds[batch_idx, img_token_mask[batch_idx]] = item_vit_embeds[:actual_vis_tokens]
input_embeds = input_embeds.reshape(batch_size, seq_len, channels)
current_token_idx = 0
img_token_locations = torch.where(img_token_mask[batch_idx])[0]
for image_idx, num_tiles_for_this_image in enumerate(batch_num_tiles_list[batch_idx]):
num_tokens_for_this_image = num_tiles_for_this_image * tokens_per_tile
if not bool(image_masks[batch_idx, image_idx].item()):
start_offset = current_token_idx
end_offset = min(
current_token_idx + num_tokens_for_this_image,
int(img_token_locations.shape[0]),
)
if start_offset < end_offset:
masked_token_indices = img_token_locations[start_offset:end_offset]
attention_mask[batch_idx, masked_token_indices] = 0
current_token_idx += num_tokens_for_this_image
return input_embeds, attention_mask
def get_fused_image_text_embedding_from_tensor_images(
self,
image_tensors: list[Image.Image | torch.Tensor],
image_tensors: list[Image.Image | torch.Tensor] | list[list[Image.Image | torch.Tensor]],
image_mask: torch.Tensor,
text_prompt: str,
text_prompt: str | list[str],
return_cls_only: bool = True,
):
pixel_values, num_tiles_list = self._preprocess_images(image_tensors)
is_batched_input = bool(image_tensors) and isinstance(image_tensors[0], list)
if is_batched_input:
image_tensors_batch = image_tensors
else:
image_tensors_batch = [image_tensors]
batch_size = len(image_tensors_batch)
if batch_size == 0:
raise ValueError("InternVL3 expects at least one batch item.")
if isinstance(text_prompt, str):
text_prompts = [text_prompt] * batch_size
else:
text_prompts = text_prompt
if len(text_prompts) != batch_size:
raise ValueError(
f"InternVL3 batch mismatch: num_text_prompts={len(text_prompts)} num_batches={batch_size}"
)
if image_mask.ndim == 1:
image_masks = image_mask.unsqueeze(0)
else:
image_masks = image_mask
if image_masks.shape[0] != batch_size:
raise ValueError(
f"InternVL3 batch mismatch: image_mask_batch={image_masks.shape[0]} num_batches={batch_size}"
)
for batch_idx, image_list in enumerate(image_tensors_batch):
if len(image_list) == 0:
raise ValueError(f"InternVL3 expects at least one image per batch item, got empty at batch_idx={batch_idx}")
if int(image_masks.shape[1]) != len(image_list):
raise ValueError(
"InternVL3 mask/image count mismatch: "
f"batch_idx={batch_idx}, num_masks={image_masks.shape[1]}, num_images={len(image_list)}"
)
image_masks = image_masks.to(device=self.device, dtype=torch.bool)
pixel_values, batch_num_tiles_list = self._preprocess_images(image_tensors_batch)
if pixel_values.shape[0] == 0:
logging.warning("InternVL3 received an empty image batch after preprocessing.")
hidden_size = self.model.language_model.get_input_embeddings().embedding_dim
vit_embeds = torch.empty(0, hidden_size, dtype=next(self.model.parameters()).dtype, device=self.device)
else:
vit_embeds = self.model.extract_feature(pixel_values)
vit_embeds = self.model.extract_feature(pixel_values)
fused_embeds = vit_embeds
prompt = self._build_multimodal_prompt(num_tiles_list, text_prompt)
prompts = self._build_multimodal_prompt(batch_num_tiles_list, text_prompts)
inputs_embeds, attention_mask = self._prepare_and_fuse_embeddings(
prompt, fused_embeds, image_mask, num_tiles_list
prompts,
vit_embeds,
image_masks,
batch_num_tiles_list,
)
outputs = self.model.language_model(

查看文件

@ -272,7 +272,15 @@ class EVO1Policy(PreTrainedPolicy):
camera_keys = self._camera_keys or sorted(key for key in batch if key.startswith(f"{OBS_IMAGES}."))
if not camera_keys:
raise ValueError("EVO1 requires at least one visual observation feature.")
batch_size = batch[camera_keys[0]].shape[0]
first_image = batch[camera_keys[0]]
if first_image.dim() == 3:
batch_size = 1
elif first_image.dim() in (4, 5):
batch_size = first_image.shape[0]
else:
raise ValueError(
f"Unsupported image tensor shape for EVO1: key={camera_keys[0]} shape={tuple(first_image.shape)}"
)
image_batches: list[list[Tensor]] = []
image_masks = torch.zeros(batch_size, self.config.max_views, dtype=torch.bool)
@ -288,6 +296,10 @@ class EVO1Policy(PreTrainedPolicy):
raise ValueError(
f"Unsupported image tensor shape for EVO1: key={camera_key} shape={tuple(image.shape)}"
)
if image.shape[0] != batch_size:
raise ValueError(
f"EVO1 batch mismatch across cameras: key={camera_key} batch={image.shape[0]} expected={batch_size}"
)
sample_images.append(image[batch_index].detach().cpu())
if not sample_images:
raise ValueError("EVO1 received a batch without any image tensor.")
@ -304,16 +316,13 @@ class EVO1Policy(PreTrainedPolicy):
image_batches: list[list[Tensor]],
image_masks: Tensor,
) -> Tensor:
fused_tokens = []
for prompt, images, image_mask in zip(prompts, image_batches, image_masks, strict=True):
fused = self.model.get_vl_embeddings(
images=images,
image_mask=image_mask,
prompt=prompt,
return_cls_only=self.config.return_cls_only,
)
fused_tokens.append(fused.to(device=self.config.device, dtype=self._compute_dtype))
return torch.cat(fused_tokens, dim=0)
fused_tokens = self.model.get_vl_embeddings(
images=image_batches,
image_mask=image_masks,
prompt=prompts,
return_cls_only=self.config.return_cls_only,
)
return fused_tokens.to(device=self.config.device, dtype=self._compute_dtype)
def _compute_masked_loss(
self,

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9dc9df05797dc0e7b92edc845caab2e4c37c3cfcabb4ee6339c67212b5baba3b
size 38023

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7e11af87616b83c1cdb30330e951b91e86b51c64a1326e1ba5b4a3fbcdec1a11
size 55698

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b8840fb643afe903191248703b1f95a57faf5812ecd9978ac502ee939646fdb2
size 121115

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f79d14daafb1c0cf2fec5d46ee8029a73fe357402fdd31a7cd4a4794d7319a7c
size 260367

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a8d6e64d6cb0e02c94ae125630ee758055bd2e695772c0463a30d63ddc6c5e17
size 3520862

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6bdf22208d49cd36d24bc844d4d8bda5e321eafe39d2b470e4fc95c7812fdb24
size 3687117

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:8920d5ebab36ffcba9aa74dcd91677c121f504b4d945b472352d379f9272fabf
size 3687117

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:35723f2db499da3d9d121aa79d2ff4c748effd7c2ea92f277ec543a82fb843ca
size 3687117

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:53172b773d4a78bb3140f10280105c2c4ebcb467f3097579988d42cb87790ab9
size 3687117

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:58a5d91573e7dd2352a1454a5c9118c9ad3798428a0104e5e0b57fc01f780ae7
size 3687117

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:bb65a25e989a32a8b6258d368bd077e4548379c74ab5ada01cc532d658670df0
size 3687117

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c3dcff0a705ebfdaf11b7f49ad85b464eff03477ace3d63ce45d6a3a10b429d5
size 111338

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d8ab0274761cdd758bafdf274ce3e6398cd6f0df23393971f3e1b6b465d66ef3
size 111338

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:aee60956925da9687546aafa770d5e6a04f99576f903b08d0bd5f8003a7f4f3e
size 111338

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c8d9f9cc9e232820760fe4a46b47000c921fa5d868420e55d8dbc05dae56e8bd
size 111338

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:01cfe50c537e3aef0cd5947ec0b15b321b54ecb461baf7b4f2506897158eebc8
size 111338

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:96431ca3479eef2379406ef901cad7ba5eac4f7edcc48ecc9e8d1fa0e99d8017
size 111338

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:3763d7bff7873cb40ea9d6f2f98d45fcf163addcd2809b6c59f273b6c3627ad5
size 85353

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:24150994c6959631dc081b43e4001a8664e13b194ac194a32100f7d3fd2c0d0f
size 85353

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c9c3fdf34debe47d4b80570a19e676185449df749f37daa2111184c1f439ae5f
size 85353

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:f8cfbe444c14d643da2faea9f6a402ddb37114ab15395c381f1a7982e541f868
size 85353

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:07c5c1a63998884ee747a6d0aa8f49217da3c32af2760dad2a9da794d3517003
size 85353

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9927ec508e3335f8b10cf3682e41dedb7e647f92a2063a4196f1e48749c47bc5
size 85353

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:6b1e600768a8771c5fe650e038a1193597e3810f032041b2a0d021e4496381c1
size 3686488

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:9d4ebab73eabddc58879a4e770289d19e00a1a4cf2fa5fa33cd3a3246992bc90
size 40551392

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ee0c29d3782aa1cadcf4dc6ed767d9460ff00fff9fc70b460502340b832eefcc
size 5104

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:1a7a8b1a457149109f843c32bcbb047d09de2201847b9b79f7501b447f77ecf4
size 31672

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:5e6ce85296b2009e7c2060d336c0429b1c7197d9adb159e7df0ba18003067b36
size 68

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:ea76e6711959fd3f905ec2bdc306f488920f00ec99421e4870d05f6205eb323e
size 31672

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:c2b8f8532c7a0b776de5e536b8b54e30b1a0c2e3d5cc25a2d86fe43e40ae5e8c
size 515400

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:224b5fa4828aa88171b68c036e8919c1eae563e2113f03b6461eadf5bf8525a6
size 31672

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:016d2fa8fe5f58017dfd46f4632fdc19dfd751e32a2c7cde2077c6f95546d6bd
size 68

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:eca0d87a699620e4fec7e68539b0be91e4cc933f6bf12032da52c182ab6f38cf
size 31672

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:19eaaa85f66ba4aa6388dbb83819ffad6ea4363247208f871a8dc385689f6fc8
size 992

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:227296eaeeb54acdc3dae2eb8af3d4d08fb87e245337624447140b1e91cfd002
size 47424

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:271b00cb2f0cd5fd26b1d53463638e3d1a6e92692ec625fcffb420ca190869e5
size 68

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:778fddbbaa64248cee35cb377c02cc2b6076f7ce5855146de677128900617ddf
size 47424

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:d640988f2269cf6aa03c8ee17f9d096edace83d837f90025011fafec5bf53c61
size 200

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:32ddf36af25791935b395c7641531cda14d5c4a2cf654a2e76ac45271665d07a
size 16904

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:22a1031a2acfc36a455bff73ffbe097cfeb7742b6485e7422507e78d7a682703
size 164

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b5dca7940998421ae58e9e26b2b2641b058d23b0270b7a147ebf85fbbdce7184
size 35496

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:2212ae7b910d14d723214f5af50985e419f7bd0f4261565ef48b1ef495443d6d
size 200

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:32ddf36af25791935b395c7641531cda14d5c4a2cf654a2e76ac45271665d07a
size 16904

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:22a1031a2acfc36a455bff73ffbe097cfeb7742b6485e7422507e78d7a682703
size 164

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:b5dca7940998421ae58e9e26b2b2641b058d23b0270b7a147ebf85fbbdce7184
size 35496

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:7681c2076cba6a36046647861b03efb870aea38bba9b92b4d4db8cd916818e5e
size 26504161

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:a38c59bc93ab9cb2f2d438debd98ba06bcb6d7a6aa8f7e092caa77591db2ffe7
size 21564949

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:e22f7cca684d15d66126bef75dbce9f3e8224d7eaa057133a8bbe151b8955002
size 15873071

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:798082de42937101e9320bce75859921ae7e26347c9a66f06a13be51888ce445
size 15879505

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:91ca17d0484ce7b91030c02f515af8da60291f7aa2fbd2649205c3b9e522a6d4
size 48327985

查看文件

@ -1,3 +0,0 @@
version https://git-lfs.github.com/spec/v1
oid sha256:fe94178b947be9a0d05e167cd82ebdb44ecf38b9376ca0588e99e49b3496dfc0
size 54471462