path: root/guides/sysadmin/machines/amdahl30/killing-time.org

On November 19 2024, LDLC's off-brand SSD died on me.  RIP.
Re-installed Tumbleweed on the replacement (Kingston SA400S3) on
November 28.

Since then, I have been getting uncannily reproducible stuttering and
frame drops (60↘40±10) in Hades Ⅱ when moving toward effect- or
particle-heavy areas of the hub rooms (Crossroads, Training Grounds).
No idea WTF, those areas ran fine before.

- "High" graphics setting at native 1920×1080 resolution.
  - Tried "Low" graphics, lowered resolution, disabled vsync, switched
    to Windowed mode: symptoms persist.
- Proton Experimental.
  - Tried a couple of old Proton versions: symptoms persist.
- Reinstalled game & nuked everything under
  - =~/.cache/mesa_shader_cache*=
  - =~/.cache/radv_builtin_shaders*=
  - =~/.config/unity3d=
  - =~/.local/share/Steam=
  - =~/.local/share/vulkan/=
  - =~/.steam*=
  in case "stale shaders" were to blame or something.
- Tumbleweed/Plasma/Wayland session.
  - Tried X11: symptoms persist.
- Reducing noise with
  - ~balooctl6 suspend~
  - ~swapoff -a~ (RAM nowhere near exhausted)

Well then.
* CPU frequency scaling?
(Hey 👋 A warning: this was the first rabbit hole I burrowed into.
Spoiler alert: nothing I learned here solved the problem.  Chronicled
the journey anyway since I wandered through interesting spots)

Started by noticing that the Plasma "Power Management" tray widget
says "Power Profile" is "Not available".  Not sure whether that was
the case with the old installation; maybe I had something configured
or installed to enable this?

Internet says "install and enable power-profiles-daemon", except
that's on:

#+begin_example
$ systemctl status power-profiles-daemon.service
● power-profiles-daemon.service - Power Profiles daemon
     Loaded: loaded (/usr/lib/systemd/system/power-profiles-daemon.service; disabled; preset: disabled)
     Active: active (running) since Sun 2024-12-01 11:46:32 CET; 45min ago
 Invocation: b2545a02bc9642b7aeb5f370e8b50e7c
   Main PID: 2289 (power-profiles-)
      Tasks: 4 (limit: 18320)
        CPU: 52ms
     CGroup: /system.slice/power-profiles-daemon.service
             └─2289 /usr/libexec/power-profiles-daemon
#+end_example

But:

#+begin_example
$ powerprofilesctl
,* balanced:
    PlatformDriver:     placeholder

  power-saver:
    PlatformDriver:     placeholder
#+end_example

Internet says I am missing the right scaling driver, and sounds very
keen on enabling =amd_pstate=, which I do not seem to have available.
=/proc/config.gz= suggests the kernel configuration supports it, but
=cpupower= does not appear to know about it:

#+begin_example
$ zcat /proc/config.gz | grep -i pstate
CONFIG_X86_INTEL_PSTATE=y
CONFIG_X86_AMD_PSTATE=y
CONFIG_X86_AMD_PSTATE_DEFAULT_MODE=3
# CONFIG_X86_AMD_PSTATE_UT is not set

$ cpupower frequency-info
analyzing CPU 5:
  driver: acpi-cpufreq
  CPUs which run at the same hardware frequency: 5
  CPUs which need to have their frequency coordinated by software: 5
  maximum transition latency:  Cannot determine or is not supported.
  hardware limits: 1.40 GHz - 3.70 GHz
  available frequency steps:  3.70 GHz, 1.70 GHz, 1.40 GHz
  available cpufreq governors: ondemand performance schedutil
  current policy: frequency should be within 1.40 GHz and 3.70 GHz.
                  The governor "schedutil" may decide which speed to use
                  within this range.
  current CPU frequency: Unable to call hardware
  current CPU frequency: 3.30 GHz (asserted by call to kernel)
  boost state support:
    Supported: yes
    Active: no
#+end_example

=dmesg= offers:

#+begin_example
$ sudo dmesg -H
[…] amd_pstate: the _CPC object is not present in SBIOS or ACPI disabled
#+end_example

Though:

#+begin_example
$ lscpu | grep -i cppc
Flags:                                […] cppc […]
#+end_example

So ACPI problem?  Lots of posts mentioning =amd_= parameters on the
kernel command-line, but AFAIU those posts are stale with newer
kernels (6.11 here) which automatically (attempt to) load the
=amd_pstate= driver.

Went through the UEFI menu and found nothing related to ACPI or
[[https://forum.level1techs.com/t/amd-p-state-driver/197885/24][X2APIC]].  Skeptical of UEFI settings anyway, since I did not change
them between the old and new installations.

{{{narrator(Some time later)}}}

Probably not ACPI, =dmesg= is choke full of ACPI noise.  OTOH, using
some diagnosis methods from [[https://bugzilla.kernel.org/show_bug.cgi?id=218171][this kernel bug report]]:

#+begin_example
$ find /sys/devices -name '*cppc*'
🦗
#+end_example

(~acpidump ; acpixtract ; iasl ; grep -i cpc *.dsl~ also yields 🦗,
but =iasl= complains about "unresolved" "control methods", so 🤷)

{{{narrator(Some time later)}}}

[[https://wiki.archlinux.org/title/CPU_frequency_scaling#amd_pstate][ArchWiki]] does say "Change /Enable CPPC/ […] from /Auto/ to /Enabled/".
My UEFI menu tucks that under /Overclocking → Advanced CPU
Configuration → AMD CBS → CPPC CTRL/.  That change *does* convince
Linux to enable =amd_pstate=; going over the previous tests in reverse
order:

#+begin_example
$ [… acpidump && acpixtract && iasl … ] && grep -i cpc *.dsl
ssdt1.dsl:        Name (_CPC, Package (0x17)  // _CPC: Continuous Performance Control
[… repeats 12 times …]

$ find /sys/devices -name '*cppc*' -o -name '*pstate*' | tr -s '[:digit:]' N | sort -u
/sys/devices/system/cpu/amd_pstate
/sys/devices/system/cpu/cpufreq/policyN/amd_pstate_highest_perf
/sys/devices/system/cpu/cpufreq/policyN/amd_pstate_hw_prefcore
/sys/devices/system/cpu/cpufreq/policyN/amd_pstate_lowest_nonlinear_freq
/sys/devices/system/cpu/cpufreq/policyN/amd_pstate_max_freq
/sys/devices/system/cpu/cpufreq/policyN/amd_pstate_prefcore_ranking
/sys/devices/system/cpu/cpuN/acpi_cppc

$ sudo dmesg -H
[… ominous silence about amd_pstate …]

$ cpupower frequency-info
analyzing CPU 1:
  driver: amd-pstate-epp
  CPUs which run at the same hardware frequency: 1
  CPUs which need to have their frequency coordinated by software: 1
  maximum transition latency:  Cannot determine or is not supported.
  hardware limits: 400 MHz - 4.31 GHz
  available cpufreq governors: performance powersave
  current policy: frequency should be within 2.38 GHz and 4.31 GHz.
                  The governor "powersave" may decide which speed to use
                  within this range.
  current CPU frequency: Unable to call hardware
  current CPU frequency: 3.57 GHz (asserted by call to kernel)
  boost state support:
    Supported: yes
    Active: yes
    AMD PSTATE Highest Performance: 255. Maximum Frequency: 4.31 GHz.
    AMD PSTATE Nominal Performance: 219. Nominal Frequency: 3.70 GHz.
    AMD PSTATE Lowest Non-linear Performance: 141. Lowest Non-linear Frequency: 2.38 GHz.
    AMD PSTATE Lowest Performance: 24. Lowest Frequency: 400 MHz.

$ powerprofilesctl
  performance:
    CpuDriver:	amd_pstate
    Degraded:   no

,* balanced:
    CpuDriver:	amd_pstate
    PlatformDriver:	placeholder

  power-saver:
    CpuDriver:	amd_pstate
    PlatformDriver:	placeholder
#+end_example

And lo, the 🍃↔🚀 slider appears in the Power Management tray widget.

Nervous about entering the "Overclocking" UEFI zone tho, and concerned
about these "Maximum frequencies".

/And does it even help with the game?/

🥁

No.  No it does not; no discernible difference in FPS nor vibes.

Will assume this new baseline cannot hurt - OT1H "overclocking" is
scary, OTOH Linux now has a finer handle on the CPU and hopefully will
not overwork it to death?

-----

Chronologically, the events of the following addenda were weaved
together with those of the next section; figured I would append them
here to give the reader some closure on this shaggy dog subplot.
** Addendum Ⅰ — CPPC Considered Harmful, apparently
[[https://www.techpowerup.com/forums/threads/what-fixed-stuttering-and-random-framerate-spikes-in-games-for-me.327264/][aska33j proclaims]] that /disabling CPPC/ "fixed stuttering and random
framerate spikes in games for [them]" so… roundtrip to UEFI, disabling
that.  The =amd_pstate= warning is back; the "Power Profile" slider is
no longer accessible in the systray widget; no discernible effect
in-game anyway.
** Addendum Ⅱ — BIOS update
For the modest price of [[file:maintenance.org::*Our protagonist sets forth][breaking Secure Boot]], the =amd_pstate= driver
now manages to initialize successfully without me having to mess with
{{{glitch(Overclocking Settings)}}}.  Welcome back =amd_pstate= 🤝
* Sᴇᴠᴇʀᴀʟ Wᴇᴇᴋꜱ Lᴀᴛᴇʀ
- [[https://www.gamingonlinux.com/forum/topic/5475/page=1/][ridge reports]] "bad frame pacing on ADMGPU",
  - when vsync is turned off: a non-factor in my testing,
  - lots of useful information in that thread tho and
    interesting-sounding pointers,
  - [[https://www.gamingonlinux.com/forum/topic/5475/page=2/#r42519][Shmerl]] says:
    - games can cause stutter by underloading the GPU, causing it to
      drop out of "high performance mode",
      - (=amdgpu_top= and =radeontop= do confirm that lag spikes
        correlate with GPU usage drop)
    - see [[https://gitlab.freedesktop.org/drm/amd/-/issues/1500][drm/amd#1500]]:
      - /lots/ of sysfs noodling there; unfortunately, none of the
        suggested settings for =power_dpm_force_performance_level= &
        =pp_power_profile_mode= change the symptoms.
  - Since this forum seems full of knowledgeable folks, posted [[https://www.gamingonlinux.com/forum/topic/6437/][a new
    topic]] there… but then [[https://www.gamingonlinux.com/forum/topic/6463/][the UK OSA dropped]].

- In [[https://gitlab.freedesktop.org/drm/amd/-/issues/3618#note_2689087][this drm/amd#3618 thread]], @agd5f suggests "6.11 stable kernels"
  include a fix for the issue at hand there and a further rework "was
  submitted to 6.13"; @mattipulkkinen reports happy results with
  6.13-rc2 (FTR, symptoms persist here with 6.12.8).

- Piggybacked onto [[https://gitlab.freedesktop.org/mesa/mesa/-/issues/11300][mesa/mesa#11300]]:
  - common: Hades Ⅱ, iGPU, recent kernel & Mesa, Proton Experimental,
  - differences: Fedora, GNOME, X11,
  - noteworthy: good performance on Windows,
  - suggestion by @Venemo: downgrade & bisect Mesa.
    - Tempting, though scared of bricking graphical sessions and/or
      ending up with a frankensystem (intalling binaries under a
      prefix is probably easy, but then keeping track of config tweaks
      and cache artifacts sounds fraught).
    - {{{narrator(10 false leads later)}}}
    - The factory@ announcements say Mesa was upgraded from 24.2.7 to
      24.3.0 in snapshot 20241124.  The Mesa docs explain how to test
      "local builds" (i.e. uninstalled; kudos![fn:mesa-builddeps]) so
      gave it a shot: a freshly built 24.2.7 behaves exactly like the
      current distro version (24.3.4), so no smoking gun there.

[fn:mesa-builddeps] Took some trial-and-error to get a build going:
~zypper source-install --build-deps-only Mesa~ did not cut it.
Probably should have looked at Tumbleweed's spec file to get a
matching build configuration; instead ~zypper install~'ed my way to
victory - needed ~libclc llvm llvm-devel libLLVMSPIRVLib-devel clang
clang-devel python3-ply rust rust-bindgen rust-cbindgen~.


- In [[https://gitlab.freedesktop.org/upower/power-profiles-daemon/-/issues/164][upower/power-profiles-daemon#164]], @Nyan reports problematic iGPU
  capping; not convinced this is applicable though, given the reported
  symptoms (video playback is fine here).

- Seen reports of Variable Refresh Rate causing problems:
  - searched high and low to understand why VRR appears nowhere in
    Plasma settings, despite the start menu turning up "Display
    Configuration" when searching for "VRR",
  - mystery solved by ~kscreen-doctor -o~: =Vrr: incapable= 🤷

- (FTR: chronologically, this is the point where I found that forum
  post about leaving CPPC disabled to fix stuttering—see addendum Ⅰ in
  the previous section.

  That did not work for me, but I left it disabled at this point since
  enabling it never had an effect in the first place, and I was keen
  on restoring every "variable" to their original state)

- Looking at Steam forums, [[https://steamcommunity.com/app/1145350/discussions/1/596260472619121965/][some folks]] do report FPS drops /shortly
  after the update/:
  #+begin_quote
  it started fine after the major update, now suddenly im stuck with
  40~50 fps with micro sutters
  — December 6 2024
  #+end_quote

- After AMD drivers & Mesa, figured I could look at vkd3d's issue
  tracker.  [[https://github.com/doitsujin/dxvk/issues/4436][doitsujin/dxvk#4436]] and
  [[https://github.com/ValveSoftware/steam-for-linux/issues/11446][ValveSoftware/steam-for-linux#11446]] looked somewhat promising:
  reports of lag on "KDE Tumbleweed Wayland", reported not long before
  my symptoms began (November 2024)); alas, ~LD_PRELOAD=~ does not
  help.
  - {{{narrator(clicks through duplicates\, out of GitHub & into
    [[https://reddit.com/r/linux_gaming/comments/1htcxfj/system_green_screens_regularly_during_more/m5da9ey/][Reddit]])}}}

    #+begin_quote
    Alternatively, remove the offending line in
    =/usr/share/drirc.d/00-radv-defaults.conf=
    #+end_quote

    {{{narrator(discovers [[https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/src/util/00-radv-defaults.conf][=/usr/share/drirc.d/=]])}}}

    Computers were a mistake.

- Peeked at [[https://github.com/HansKristian-Work/vkd3d-proton/blob/master/.github/ISSUE_TEMPLATE/bug_report.md][vkd3d-proton's issue template]] and idly ran with
  ~PROTON_LOG=1~.  Over the course of 30 seconds or so, the log file
  gets flooded with 3MB's worth of =trace:unwind:dump_unwind_info= 🤨

- VRAM usage is always close to full, even when not playing games.
  "At rest", the Plasma shell consumes ≈410MB over 512MB available.
  - [[https://forum.kde.org/viewtopic.php%3Ff=111&t=165779.html][Lissanro reported]] in 2020 that changing Plasma's rendering backend
    to /Software/ freed up some VRAM.
  - Indeed, bringing up the Plasma Renderer menu, switching to
    /Software/, logging out & back in frees up some VRAM.  It also
    yields compositing glitches 🤷
  - More to the point, /it has no effect on the symptoms in-game/.

- Figured I would ask ValveSoftware/Proton about the logs; filed
  [[https://github.com/ValveSoftware/Proton/issues/8424][#8424]]; got dup'd into [[https://github.com/ValveSoftware/Proton/issues/7805][#7805]], per the "one report per game" policy.
  That issue is about a /crash on Alt-Tab/, with an /Nvidia dGPU/;
  unsure how lumping our two reports together will help.  Had to try 🤷

- Found [[https://gitlab.freedesktop.org/drm/amd/-/issues/2516][drm/amd#2516]]; noticed that I have
  - =/sys/module/gpu_sched/parameters/sched_policy=: 1
  - =/sys/module/amdgpu/parameters/sched_policy=: 0
  Changed the kernel command-line to set the former to 0, as suggested
  in that issue; symptoms persist.  No idea what the latter is about,
  nor how it differs from the former.  I can find [[https://docs.kernel.org/gpu/amdgpu/module-parameters.html#sched-policy-int][the docs for amdgpu]]
  but nor for gpu_sched.

- [[https://old.reddit.com/r/archlinux/comments/1gzy0xd/amdgpu_regression_on_kernel_612_choppy/m1dn05z/][Some folks]] report =amdgpu.dcdebugmask=0x10= (≡ =DC_DISABLE_PSR=)
  fixing "choppy performance".  No effect here.  Could try setting
  [[https://www.kernel.org/doc/html/v6.13/gpu/amdgpu/module-parameters.html#dcdebugmask-uint][other values]]…

- (FTR: chronologically, this is the point where I caved—not sure why,
  I think seeing the amount of unhappy kernel noises in journalctl got
  to me—and grabbed the latest BIOS from msi.com, flashed it, breaking
  Secure Boot & restoring =amd_pstate= 🤷 see addendum Ⅱ in the
  previous section)
* This is insane
Selected subset of moving parts; "testability" considering ease of
clean reverts:

| Part         | Testability                                                                          |
|--------------+--------------------------------------------------------------------------------------|
| Linux kernel | 🫣 [[https://en.opensuse.org/SDB:InstallNewerKernel][some distro documentation]]; afraid of side-effects                                |
| AMD drivers  | 🤷 no clue; maybe inextricable from kernel?                                         |
| Mesa         | 🤔 easy to rebuild; circumspect about transient state in cache & config folders     |
| Steam        | 🫥 under Steam's control                                                            |
| Wine         | 🫥 under Steam's control                                                            |
| Proton       | 👌 as long as I stick to versions under Steam's control; have not considered GE yet |
| vkd3d-proton | 🫥 under Steam's control                                                            |
| Hades Ⅱ     | 🫥 under Steam's control                                                            |

That's looking at software packages as individual blackboxes;
config-wise, worth noting:

| Part            | Testability                                       |
|-----------------+---------------------------------------------------|
| +AMD pstate+    | +😬 UEFI roundtrip+[fn:amd-pstate]               |
| sysfs           | OK; worst case: reboot & edit kernel command-line |
| Plasma Renderer | OK                                                |

[fn:amd-pstate] Always on since I updated the BIOS, so I no longer
consider it a factor.  I guess I /could/ keep doing so and manually
disable it.


Let's throw in:

| Part | Testability              |
|------+--------------------------|
| BIOS | 🔥 [[file:maintenance.org::*Firmware updates][breaks boot settings]] |