ARM64: Fix fpsimd save/restore for cpu suspend

Currently FPSIMD context is saved and restored on CPU_PM_ENTER and CPU_PM_EXIT
if the current thread is not a kernel thread and TIF_FOREIGN_FPSTATE has not
been set. However, if CPU_PM_ENTER/EXIT are called from an interrupt context,
then the check on TIF_FOREIGN_FPSTATE could potentially race with an inflight
fpsimd_restore_current_state() call as seen below:

ret_to_user
    do_notify_resume
        fpsimd_restore_current_state
            test_and_clear_thread_flag => flag cleared

=> IPI comes in to suspend current CPU

CPU_PM_ENTER
    TIF_FOREIGN_FPSTATE not set as it was just cleared
        fpsimd_save_state => Overrides current thread's context with stale data

To avoid the above race, save and restore the CPU's fpsimd state locally if the
current thread is a userspace thread. If it isn't, then clear the
fpsimd_last_state variable to ensure that the userspace thread's context is
properly restored.

Bug 1567470

Change-Id: Id0e28964d16b3dedc4cec66a0e58ab3dac5b9cbc
Signed-off-by: Sai Gurrappadi <sgurrappadi@nvidia.com>
Reviewed-on: http://git-master/r/603148
Reviewed-by: Robert Goldman <rgoldman@nvidia.com>
Reviewed-by: Bo Yan <byan@nvidia.com>

diff --git a/arch/arm64/kernel/fpsimd.c b/arch/arm64/kernel/fpsimd.c
index 53b7c029a3b3a3f695bfdcb5b4f74b8450e6f6c0..7ac90cf1d7e3211e230d88f109799fe3cc7e1527 100644 (file)
--- a/arch/arm64/kernel/fpsimd.c
+++ b/arch/arm64/kernel/fpsimd.c
@@ -296,18 +296,21 @@ static inline void fpsimd_cpu_hotplug_init(void) { }
 #endif /* CONFIG_CPU_HOTPLUG */
 
 #ifdef CONFIG_CPU_PM
+static DEFINE_PER_CPU(struct fpsimd_state, fpsimd_cached_state);
+
 static int fpsimd_cpu_pm_notifier(struct notifier_block *self,
                                  unsigned long cmd, void *v)
 {
        switch (cmd) {
        case CPU_PM_ENTER:
-               if (current->mm && !test_thread_flag(TIF_FOREIGN_FPSTATE))
-                       fpsimd_save_state(&current->thread.fpsimd_state);
-               this_cpu_write(fpsimd_last_state, NULL);
+               if (current->mm)
+                       fpsimd_save_state(this_cpu_ptr(&fpsimd_cached_state));
+               else
+                       this_cpu_write(fpsimd_last_state, NULL);
                break;
        case CPU_PM_EXIT:
                if (current->mm)
-                       set_thread_flag(TIF_FOREIGN_FPSTATE);
+                       fpsimd_load_state(this_cpu_ptr(&fpsimd_cached_state));
                break;
        case CPU_PM_ENTER_FAILED:
        default: