"Encoding overloaded! Consider turning down video settings or using a faster encoding preset."
TL;DR
I kept Base (Canvas) Resolution at 1080p, changed Output (Scaled) Resolution to 900p, then set Rescale Output to 1080p. Both with Bicubic 16 samples.
For context, I got this immediately every single time I started a new recording in OBS. I do not stream, I do long form videos and shorts for YouTube and TikTok.
My current (working) settings are as follows:
VIDEO
-- General --
Base (Canvas) Resolution: 1920x1080 (16:9)
Output (Scaled) Resolution: 1600x900 (16:9)
Downscale Filter: Bicubic (Sharpened scaling, 16 samples)
Common FPS Values: 60
OUTPUT
-- Recording Settings --
Recording Format: Matroska Video (.mkv)
Video Encoder: NVIDIA NVENC H.264
Audio Encoder: FFmpeg AAC
Rescale Output: Bicubic (Sharpened scaling, 16 samples) | 1920x1080
-- Encoder Settings --
Rate Control: Constant QP
Constant QP: 20
Keyframe interval (seconds, 0=auto): 0 s
Preset: P5: Slow (Good Quality)
Tuning: High Quality
Multipass Mode: Two Passes (Quarter Resolution)
Profile: high
Look-ahead: Checked
Adaptive Quantization: Checked
B-Frames: 2
I have tried nearly every combination and I kept getting the warning immediately. It didn't matter if I changed rate control to Constant Bitrate, unchecked Look-ahead or Adaptive Quantization, lessened the Constant QP/Bitrate, changed the video encoder to HEVC or AVI, etc. What removed the warning was changing the output scaled resolution to 900p, and then setting rescale output to 1080p.
It was a rare situation when it froze on me, usually there's a chance during a more demanding PC game. Interestingly, I noticed it even did that occasionally when capturing from Switch or Xbox via elgato. I recorded Ocarina of Time and I had no issues, but then it seized up for maybe :30 while recording GTA VI. Thankfully it wasn't anywhere crucial and I didn't lose the entire recording being that I output in Matroska, I was able to edit that portion out and carry on.
OBS never freezes the image per se, but I did notice the program itself flashing/blinking very quickly on occasion. I made a mental note to check that portion of the video when finished and my assumption was right. The quick OBS blinks were when it froze the gameplay for about :30, but then it recovered and was fine after.
From the settings I had then to now, what changed were the Output (Scaled) Resolution and Rescale Output resolution. With that being said, I don't see the warning anymore, which would display instantly, but then go away after a few seconds. According to the stats, I still have rendering lag, but encoding lag is nonexistent. I will chime in again if I notice any anomalies, because I haven't had the time to do a full gameplay video since "fixing" this. I'm just happy after years of seeing that warning, it's gone (for now).
My PC Specs:
Intel Core Ultra 7
32GB RAM
RTX 5060Ti (16GB VRAM)
My old PC would produce the same error which was a 1660Ti and a decade old processor. I would like to think my newer PC would be capable of capturing when my older PC was able to do it. I also assume capturing from a console would be overall less taxing than recording in OBS while playing a Steam game at the same time?
I also personally don't try to play games on Ultra settings, I usually shoot for something medium-high settings. A lot of times my eye balls cannot tell the difference from running games at 80% to 98% GPU utilization, so my brain says the trade off of slightly better visuals is more placebo than some major leap. To each their own, but I don't need to max out my hardware at Ultra settings or 2K/4K when the monitor is 3ft from my face. I also assume the less demanding a game is the better OBS will perform? I am perfectly fine at 1080p or lesser visuals at a higher resolution as long as I can maintain steady FPS at 60/90/120Hz. My eyes prefer more atmospheric elements like weather, lighting, shadows than higher pixel count when it's so close. I don't care much for Lumen, Nanite or Ray Tracing.
Also, outputting at 1080p means smaller file sizes, quicker edits, renders and upload times.