Is there an existing issue for this?
Is your issue described in the documentation?
Is your issue present in the latest beta/pre-release?
This issue is present in the latest pre-release
Describe the Bug
When selecting VideoTool box and restricting the toolbox from using software nearly every frame becomes an IDR keyframe and P-frames are never produced resulting in severe performance degradation.
I am running a VM with MacOS Sequoia 15.7.4 with a HD 630 iGPU and AMD Radeon 6600 passed through. SMBIOS has been cycled between MaPro1,1 and iMac19,3. The system is able to run games and play video normally. The system will run Parsec without any issues. When Sunshine encoder is set to software, the encoder performs well with relatively low latency.
Expected Behavior
Videotool box should perform on par with Software encoding.
Additional Context
The Sunshine fork Lumen addressed this issue by removing max_ref_frames=1 for H.264 VT (fixes all-IDR bug) from src/video.cpp. In addition the enabled PARALLEL_ENCODING flag was also added.
I have attempted a similar fix by compiling sunshine myself with similar modifications but I have not had success yet. I think this is because I have been unable to get the compiler to build an app that will use the GPU successfully, so that is my own skill issue, but I hope some one might be able to try this as well.
The following commands have also been applied on and off in terminal.
defaults write com.apple.coremedia hardwareVideoDecoder -string force
defaults write com.apple.coremedia hardwareVideoEncoder -string force
defaults write com.apple.AppleGVA gvaForceAMDHEVCDecode -boolean yes
defaults write com.apple.AppleGVA gvaForceAMDAVCDecode -boolean yes
defaults write com.apple.AppleGVA gvaForceAMDAVCEncode -boolean yes
defaults write com.apple.AppleGVA gvaForceAMDKE -boolean yes
defaults write com.Apple.ActivityMonitor ShowGPUTab -bool true
Host Operating System
macOS
Operating System Version
15.7.4
Architecture
amd64/x86_64
Package
macOS - dmg
GPU Type
AMD
GPU Model
AMD 6600
GPU Driver/Mesa Version
115-D534P00-100
Capture Method
AVCaptureScreen (macOS)
Apps
Log output
[2026-04-17 13:43:46.602]: Info: Sunshine version: 2026.415.34134 commit: a16c070a8feb509a55a434393690cf76447e9423
[2026-04-17 13:43:46.602]: Info: Package Publisher: LizardByte
[2026-04-17 13:43:46.602]: Info: Publisher Website: https://app.lizardbyte.dev
[2026-04-17 13:43:46.602]: Info: Get support: https://app.lizardbyte.dev/support
[2026-04-17 13:43:46.602]: Info: config: 'hevc_mode' = 0
[2026-04-17 13:43:46.602]: Info: config: 'output_name' = 2
[2026-04-17 13:43:46.602]: Info: config: 'high_resolution_scrolling' = disabled
[2026-04-17 13:43:46.602]: Info: config: 'fec_percentage' = 2
[2026-04-17 13:43:46.602]: Info: config: 'av1_mode' = 0
[2026-04-17 13:43:46.602]: Info: config: 'min_fps_factor' = 0
[2026-04-17 13:43:46.602]: Info: config: 'min_log_level' = 2
[2026-04-17 13:43:46.602]: Info: config: 'vt_coder' = cavlc
[2026-04-17 13:43:46.602]: Info: config: 'wan_encryption_mode' = 2
[2026-04-17 13:43:46.602]: Info: config: 'min_threads' = 8
[2026-04-17 13:43:46.602]: Info: config: 'minimum_fps_target' = 240
[2026-04-17 13:43:46.602]: Info: config: 'encoder' = videotoolbox
[2026-04-17 13:43:46.602]: Info: config: 'stream_audio' = true
[2026-04-17 13:43:46.602]: Info: config: 'qp' = 20
[2026-04-17 13:43:46.602]: Info: config: 'sunshine_name' = Mako-iMP
[2026-04-17 13:43:46.602]: Info: config: 'sw_preset' = ultrafast
[2026-04-17 13:43:46.602]: Info: config: 'notify_pre_releases' = enabled
[2026-04-17 13:43:46.602]: Info: config: 'vt_software' = allowed
[2026-04-17 13:43:46.683]: Warning: No gamepad input is available
[2026-04-17 13:43:46.684]: Info: Trying encoder [videotoolbox]
[2026-04-17 13:43:46.718]: Info: Detecting displays
[2026-04-17 13:43:46.718]: Info: Detected display: WOIEYEK 4K120 (id: 1703442200) connected: true
[2026-04-17 13:43:46.719]: Info: Configuring selected display (1703442200) to stream
[2026-04-17 13:43:46.724]: Info: Creating encoder [h264_videotoolbox]
[2026-04-17 13:43:46.724]: Info: Color coding: SDR (Rec. 601)
[2026-04-17 13:43:46.724]: Info: Color depth: 8-bit
[2026-04-17 13:43:46.724]: Info: Color range: JPEG
[2026-04-17 13:43:46.725]: Info: Streaming bitrate is 1000000
[2026-04-17 13:43:46.942]: Error: [h264_videotoolbox @ 0x7f9ef2a093c0] Error: cannot create compression session: -12903
[2026-04-17 13:43:46.942]: Info: Retrying with fallback configuration options for [h264_videotoolbox] after error: Generic error in an external library
[2026-04-17 13:43:46.942]: Info: Streaming bitrate is 1000000
[2026-04-17 13:43:47.023]: Warning: [h264_videotoolbox @ 0x7f9ef2d0b240] PrioritizeEncodingSpeedOverQuality property is not supported on this device. Ignoring.
[2026-04-17 13:43:47.026]: Info: [h264_videotoolbox @ 0x7f9ef2d0b240] This device does not support the AllowOpenGop option. Value ignored.
[2026-04-17 13:43:47.331]: Info: Creating encoder [hevc_videotoolbox]
[2026-04-17 13:43:47.332]: Info: Color coding: SDR (Rec. 601)
[2026-04-17 13:43:47.332]: Info: Color depth: 8-bit
[2026-04-17 13:43:47.332]: Info: Color range: JPEG
[2026-04-17 13:43:47.332]: Info: Streaming bitrate is 1000000
[2026-04-17 13:43:47.373]: Warning: [hevc_videotoolbox @ 0x7f9ef1f6f380] PrioritizeEncodingSpeedOverQuality property is not supported on this device. Ignoring.
[2026-04-17 13:43:47.373]: Error: [hevc_videotoolbox @ 0x7f9ef1f6f380] Error setting max bitrate property: -12900
[2026-04-17 13:43:47.376]: Info: [hevc_videotoolbox @ 0x7f9ef1f6f380] This device does not support the AllowOpenGop option. Value ignored.
[2026-04-17 13:43:47.376]: Info: [hevc_videotoolbox @ 0x7f9ef1f6f380] This device does not support the max_ref_frames option. Value ignored.
[2026-04-17 13:43:48.510]: Info: Creating encoder [av1_videotoolbox]
[2026-04-17 13:43:48.510]: Info: Color coding: SDR (Rec. 601)
[2026-04-17 13:43:48.510]: Info: Color depth: 8-bit
[2026-04-17 13:43:48.510]: Info: Color range: JPEG
[2026-04-17 13:43:48.510]: Error: Couldn't open [av1_videotoolbox]
[2026-04-17 13:43:48.510]: Info: Detecting displays
[2026-04-17 13:43:48.510]: Info: Detected display: WOIEYEK 4K120 (id: 1703442200) connected: true
[2026-04-17 13:43:48.510]: Info: Configuring selected display (1703442200) to stream
[2026-04-17 13:43:48.510]: Info: Creating encoder [hevc_videotoolbox]
[2026-04-17 13:43:48.510]: Info: Color coding: SDR (Rec. 709)
[2026-04-17 13:43:48.510]: Info: Color depth: 10-bit
[2026-04-17 13:43:48.510]: Info: Color range: JPEG
[2026-04-17 13:43:48.510]: Info: Streaming bitrate is 1000000
[2026-04-17 13:43:48.522]: Warning: [hevc_videotoolbox @ 0x7f9ef2a0dcc0] PrioritizeEncodingSpeedOverQuality property is not supported on this device. Ignoring.
[2026-04-17 13:43:48.522]: Error: [hevc_videotoolbox @ 0x7f9ef2a0dcc0] Error setting max bitrate property: -12900
[2026-04-17 13:43:48.525]: Info: [hevc_videotoolbox @ 0x7f9ef2a0dcc0] This device does not support the AllowOpenGop option. Value ignored.
[2026-04-17 13:43:48.526]: Info: [hevc_videotoolbox @ 0x7f9ef2a0dcc0] This device does not support the max_ref_frames option. Value ignored.
[2026-04-17 13:43:49.820]: Info: // Testing for available encoders, this may generate errors. You can safely ignore those errors. //
[2026-04-17 13:43:49.821]: Info:
[2026-04-17 13:43:49.821]: Info: // Ignore any errors mentioned above, they are not relevant. //
[2026-04-17 13:43:49.821]: Info:
[2026-04-17 13:43:49.821]: Info: Found H.264 encoder: h264_videotoolbox [videotoolbox]
[2026-04-17 13:43:49.821]: Info: Found HEVC encoder: hevc_videotoolbox [videotoolbox]
[2026-04-17 13:43:49.823]: Info: Starting system tray
[2026-04-17 13:43:49.827]: Info: Configuration UI available at [https://localhost:47990]
[2026-04-17 13:43:49.950]: Info: System tray created
[2026-04-17 13:43:49.950]: Info: Starting main loop
[2026-04-17 13:43:50.531]: Info: Successfully registered DNS service.
[2026-04-17 13:43:53.483]: Info: Trying encoder [videotoolbox]
[2026-04-17 13:43:53.483]: Info: Detecting displays
[2026-04-17 13:43:53.483]: Info: Detected display: WOIEYEK 4K120 (id: 1703442200) connected: true
[2026-04-17 13:43:53.483]: Info: Configuring selected display (1703442200) to stream
[2026-04-17 13:43:53.483]: Info: Creating encoder [h264_videotoolbox]
[2026-04-17 13:43:53.483]: Info: Color coding: SDR (Rec. 601)
[2026-04-17 13:43:53.483]: Info: Color depth: 8-bit
[2026-04-17 13:43:53.483]: Info: Color range: JPEG
[2026-04-17 13:43:53.483]: Info: Streaming bitrate is 1000000
[2026-04-17 13:43:53.488]: Error: [h264_videotoolbox @ 0x7f9ef2c12d80] Error: cannot create compression session: -12903
[2026-04-17 13:43:53.488]: Info: Retrying with fallback configuration options for [h264_videotoolbox] after error: Generic error in an external library
[2026-04-17 13:43:53.488]: Info: Streaming bitrate is 1000000
[2026-04-17 13:43:53.563]: Warning: [h264_videotoolbox @ 0x7f9ef2a113c0] PrioritizeEncodingSpeedOverQuality property is not supported on this device. Ignoring.
[2026-04-17 13:43:53.566]: Info: [h264_videotoolbox @ 0x7f9ef2a113c0] This device does not support the AllowOpenGop option. Value ignored.
[2026-04-17 13:43:53.733]: Info: Creating encoder [hevc_videotoolbox]
[2026-04-17 13:43:53.733]: Info: Color coding: SDR (Rec. 601)
[2026-04-17 13:43:53.733]: Info: Color depth: 8-bit
[2026-04-17 13:43:53.733]: Info: Color range: JPEG
[2026-04-17 13:43:53.733]: Info: Streaming bitrate is 1000000
[2026-04-17 13:43:53.744]: Warning: [hevc_videotoolbox @ 0x7f9ef2d24d40] PrioritizeEncodingSpeedOverQuality property is not supported on this device. Ignoring.
[2026-04-17 13:43:53.744]: Error: [hevc_videotoolbox @ 0x7f9ef2d24d40] Error setting max bitrate property: -12900
[2026-04-17 13:43:53.746]: Info: [hevc_videotoolbox @ 0x7f9ef2d24d40] This device does not support the AllowOpenGop option. Value ignored.
[2026-04-17 13:43:53.746]: Info: [hevc_videotoolbox @ 0x7f9ef2d24d40] This device does not support the max_ref_frames option. Value ignored.
[2026-04-17 13:43:54.900]: Info: Creating encoder [av1_videotoolbox]
[2026-04-17 13:43:54.900]: Info: Color coding: SDR (Rec. 601)
[2026-04-17 13:43:54.900]: Info: Color depth: 8-bit
[2026-04-17 13:43:54.900]: Info: Color range: JPEG
[2026-04-17 13:43:54.900]: Error: Couldn't open [av1_videotoolbox]
[2026-04-17 13:43:54.900]: Info: Detecting displays
[2026-04-17 13:43:54.900]: Info: Detected display: WOIEYEK 4K120 (id: 1703442200) connected: true
[2026-04-17 13:43:54.900]: Info: Configuring selected display (1703442200) to stream
[2026-04-17 13:43:54.900]: Info: Creating encoder [hevc_videotoolbox]
[2026-04-17 13:43:54.900]: Info: Color coding: SDR (Rec. 709)
[2026-04-17 13:43:54.900]: Info: Color depth: 10-bit
[2026-04-17 13:43:54.900]: Info: Color range: JPEG
[2026-04-17 13:43:54.900]: Info: Streaming bitrate is 1000000
[2026-04-17 13:43:54.914]: Warning: [hevc_videotoolbox @ 0x7f9ef290d900] PrioritizeEncodingSpeedOverQuality property is not supported on this device. Ignoring.
[2026-04-17 13:43:54.914]: Error: [hevc_videotoolbox @ 0x7f9ef290d900] Error setting max bitrate property: -12900
[2026-04-17 13:43:54.917]: Info: [hevc_videotoolbox @ 0x7f9ef290d900] This device does not support the AllowOpenGop option. Value ignored.
[2026-04-17 13:43:54.917]: Info: [hevc_videotoolbox @ 0x7f9ef290d900] This device does not support the max_ref_frames option. Value ignored.
[2026-04-17 13:43:56.134]: Info: // Testing for available encoders, this may generate errors. You can safely ignore those errors. //
[2026-04-17 13:43:56.134]: Info:
[2026-04-17 13:43:56.134]: Info: // Ignore any errors mentioned above, they are not relevant. //
[2026-04-17 13:43:56.134]: Info:
[2026-04-17 13:43:56.134]: Info: Found H.264 encoder: h264_videotoolbox [videotoolbox]
[2026-04-17 13:43:56.134]: Info: Found HEVC encoder: hevc_videotoolbox [videotoolbox]
[2026-04-17 13:43:56.134]: Info: Executing [Desktop]
[2026-04-17 13:43:56.144]: Info: New streaming session started [active sessions: 1]
[2026-04-17 13:43:56.150]: Info: CLIENT CONNECTED
[2026-04-17 13:43:56.153]: Info: Detecting displays
[2026-04-17 13:43:56.153]: Info: Detected display: WOIEYEK 4K120 (id: 1703442200) connected: true
[2026-04-17 13:43:56.153]: Info: Configuring selected display (1703442200) to stream
[2026-04-17 13:43:56.250]: Info: Creating encoder [h264_videotoolbox]
[2026-04-17 13:43:56.250]: Info: Color coding: SDR (Rec. 709)
[2026-04-17 13:43:56.250]: Info: Color depth: 8-bit
[2026-04-17 13:43:56.250]: Info: Color range: MPEG
[2026-04-17 13:43:56.250]: Info: Streaming bitrate is 9108000
[2026-04-17 13:43:56.258]: Error: [h264_videotoolbox @ 0x7f9ef290d4c0] Error: cannot create compression session: -12903
[2026-04-17 13:43:56.258]: Info: Retrying with fallback configuration options for [h264_videotoolbox] after error: Generic error in an external library
[2026-04-17 13:43:56.258]: Info: Streaming bitrate is 9108000
[2026-04-17 13:43:56.325]: Warning: [h264_videotoolbox @ 0x7f9ef2c11080] PrioritizeEncodingSpeedOverQuality property is not supported on this device. Ignoring.
[2026-04-17 13:43:56.328]: Info: [h264_videotoolbox @ 0x7f9ef2c11080] This device does not support the AllowOpenGop option. Value ignored.
[2026-04-17 13:43:56.445]: Error: Encoder did not produce IDR frame when requested!
[2026-04-17 13:43:56.640]: Info: Using macOS system audio tap for capture.
[2026-04-17 13:43:56.640]: Info: Sample rate: 48000, Frame size: 240, Channels: 2
[2026-04-17 13:43:56.931]: Info: Opus initialized: 48 kHz, 2 channels, 96 kbps (total), LOWDELAY
[2026-04-17 13:43:58.118]: Error: Encoder did not produce IDR frame when requested!
[2026-04-17 13:43:58.418]: Error: Encoder did not produce IDR frame when requested!
[2026-04-17 13:43:58.469]: Error: Encoder did not produce IDR frame when requested!
[2026-04-17 13:43:58.519]: Error: Encoder did not produce IDR frame when requested!
Online logs
No response
Is there an existing issue for this?
Is your issue described in the documentation?
Is your issue present in the latest beta/pre-release?
This issue is present in the latest pre-release
Describe the Bug
When selecting VideoTool box and restricting the toolbox from using software nearly every frame becomes an IDR keyframe and P-frames are never produced resulting in severe performance degradation.
I am running a VM with MacOS Sequoia 15.7.4 with a HD 630 iGPU and AMD Radeon 6600 passed through. SMBIOS has been cycled between MaPro1,1 and iMac19,3. The system is able to run games and play video normally. The system will run Parsec without any issues. When Sunshine encoder is set to software, the encoder performs well with relatively low latency.
Expected Behavior
Videotool box should perform on par with Software encoding.
Additional Context
The Sunshine fork Lumen addressed this issue by removing max_ref_frames=1 for H.264 VT (fixes all-IDR bug) from src/video.cpp. In addition the enabled PARALLEL_ENCODING flag was also added.
I have attempted a similar fix by compiling sunshine myself with similar modifications but I have not had success yet. I think this is because I have been unable to get the compiler to build an app that will use the GPU successfully, so that is my own skill issue, but I hope some one might be able to try this as well.
The following commands have also been applied on and off in terminal.
Host Operating System
macOS
Operating System Version
15.7.4
Architecture
amd64/x86_64
Package
macOS - dmg
GPU Type
AMD
GPU Model
AMD 6600
GPU Driver/Mesa Version
115-D534P00-100
Capture Method
AVCaptureScreen (macOS)
Apps
Log output
Online logs
No response