Skip to content

fix(core): route image requests to vision fallback#8089

Open
he-yufeng wants to merge 3 commits intoAstrBotDevs:masterfrom
he-yufeng:fix/image-fallback-provider
Open

fix(core): route image requests to vision fallback#8089
he-yufeng wants to merge 3 commits intoAstrBotDevs:masterfrom
he-yufeng:fix/image-fallback-provider

Conversation

@he-yufeng
Copy link
Copy Markdown
Contributor

@he-yufeng he-yufeng commented May 8, 2026

Summary

  • choose an image-capable fallback chat provider before assembling the request when the primary provider cannot accept image input
  • keep the existing placeholder behavior when no fallback provider supports images
  • clear a primary-provider-specific request model when switching this request to a fallback provider

Fixes #8087.

To verify

  • .venv\Scripts\python.exe -m pytest tests\unit\test_astr_main_agent.py::TestBuildMainAgent::test_build_main_agent_with_images tests\unit\test_astr_main_agent.py::TestBuildMainAgent::test_build_main_agent_uses_image_fallback_provider tests\unit\test_astr_main_agent.py::TestBuildMainAgent::test_build_main_agent_keeps_text_provider_without_image_fallback tests\test_tool_loop_agent_runner.py::test_runner_builds_placeholder_for_unsupported_request_image -q
  • .venv\Scripts\python.exe -m ruff check astrbot\core\astr_main_agent.py tests\unit\test_astr_main_agent.py
  • .venv\Scripts\python.exe -m ruff format astrbot\core\astr_main_agent.py tests\unit\test_astr_main_agent.py --check
  • .venv\Scripts\python.exe -m py_compile astrbot\core\astr_main_agent.py tests\unit\test_astr_main_agent.py
  • git diff --check

Summary by Sourcery

Ensure image-based chat requests are handled by an image-capable provider when available, while preserving existing behavior when no such fallback exists.

Bug Fixes:

  • Route image requests to an image-capable fallback chat provider when the primary provider does not support image input.
  • Avoid sending provider-specific model identifiers to fallback providers when switching an image request away from the primary provider.

Tests:

  • Add unit tests covering selection of an image-capable fallback provider for image requests and retaining the primary provider when no suitable image fallback exists.

@dosubot dosubot Bot added size:M This PR changes 30-99 lines, ignoring generated files. area:core The bug / feature is about astrbot's core, backend area:provider The bug / feature is about AI Provider, Models, LLM Agent, LLM Agent Runner. labels May 8, 2026
Copy link
Copy Markdown
Contributor

@sourcery-ai sourcery-ai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hey - I've left some high level feedback:

  • In _select_image_chat_provider, mutating req.model as a side effect of choosing a fallback provider can be surprising for callers; consider either returning a new/updated ProviderRequest or making the mutation explicit at the call site to keep the helper side-effect-free.
  • The _provider_supports_modality helper treats a missing or non-list modalities entry as supporting all modalities; if a misconfigured provider shouldn’t be assumed to support images, consider tightening this to isinstance(modalities, list) and modality in modalities (and treating other cases as unsupported).
Prompt for AI Agents
Please address the comments from this code review:

## Overall Comments
- In `_select_image_chat_provider`, mutating `req.model` as a side effect of choosing a fallback provider can be surprising for callers; consider either returning a new/updated `ProviderRequest` or making the mutation explicit at the call site to keep the helper side-effect-free.
- The `_provider_supports_modality` helper treats a missing or non-list `modalities` entry as supporting all modalities; if a misconfigured provider shouldn’t be assumed to support images, consider tightening this to `isinstance(modalities, list) and modality in modalities` (and treating other cases as unsupported).

Sourcery is free for open source - if you like our reviews please consider sharing them ✨
Help me be more useful! Please click 👍 or 👎 on each comment and I'll use the feedback to improve your reviews.

Copy link
Copy Markdown
Contributor

@gemini-code-assist gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a mechanism to automatically switch to an image-capable fallback provider when the primary provider does not support image inputs. It adds helper functions to check provider modalities and select appropriate fallbacks, along with corresponding unit tests. The reviewer suggested filtering the newly selected primary provider from the fallback list to avoid redundant retry attempts in the AgentRunner.

Comment thread astrbot/core/astr_main_agent.py Outdated
@he-yufeng
Copy link
Copy Markdown
Contributor Author

The Windows Python 3.12 smoke job timed out waiting for the health endpoint, while the rest of the smoke matrix and the unit/format/build checks completed successfully. I cannot rerun that upstream job from this fork (gh run rerun reports the job cannot be rerun), so I reproduced the same smoke script locally on Windows with Python 3.12.13:

  • .venv\Scripts\python.exe scripts\smoke_startup_check.py

It passed locally (Smoke test passed). I am leaving the PR unchanged unless the CI timeout repeats or points at a concrete startup failure.

@he-yufeng
Copy link
Copy Markdown
Contributor Author

Addressed the remaining Sourcery point in 282f3b0: provider modality support now requires an explicit list containing the target modality, so missing or malformed modalities are not treated as image-capable.

The other Sourcery point was already handled in the current branch: _select_image_chat_provider only returns the provider, and the request model reset is explicit at the call site.

Validation:

  • ..venv\Scripts\python.exe -m ruff check astrbot\core\astr_main_agent.py tests\unit\test_astr_main_agent.py
  • ..venv\Scripts\python.exe -m pytest tests\unit\test_astr_main_agent.py::test_provider_supports_modality_requires_explicit_list tests\unit\test_astr_main_agent.py::TestBuildMainAgent::test_build_main_agent_uses_image_fallback_provider tests\unit\test_astr_main_agent.py::TestBuildMainAgent::test_build_main_agent_keeps_text_provider_without_image_fallback -q

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:core The bug / feature is about astrbot's core, backend area:provider The bug / feature is about AI Provider, Models, LLM Agent, LLM Agent Runner. size:M This PR changes 30-99 lines, ignoring generated files.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Bug] 传递图片时,若未设定图片转述模型,未能fallback

1 participant