License cleanup: add SPDX GPL-2.0 license identifier to files with no license
Many source files in the tree are missing licensing information, which
makes it harder for compliance tools to determine the correct license.
By default all files without license information are under the default
license of the kernel, which is GPL version 2.
Update the files which contain no license information with the 'GPL-2.0'
SPDX license identifier. The SPDX identifier is a legally binding
shorthand, which can be used instead of the full boiler plate text.
This patch is based on work done by Thomas Gleixner and Kate Stewart and
Philippe Ombredanne.
How this work was done:
Patches were generated and checked against linux-4.14-rc6 for a subset of
the use cases:
- file had no licensing information it it.
- file was a */uapi/* one with no licensing information in it,
- file was a */uapi/* one with existing licensing information,
Further patches will be generated in subsequent months to fix up cases
where non-standard license headers were used, and references to license
had to be inferred by heuristics based on keywords.
The analysis to determine which SPDX License Identifier to be applied to
a file was done in a spreadsheet of side by side results from of the
output of two independent scanners (ScanCode & Windriver) producing SPDX
tag:value files created by Philippe Ombredanne. Philippe prepared the
base worksheet, and did an initial spot review of a few 1000 files.
The 4.13 kernel was the starting point of the analysis with 60,537 files
assessed. Kate Stewart did a file by file comparison of the scanner
results in the spreadsheet to determine which SPDX license identifier(s)
to be applied to the file. She confirmed any determination that was not
immediately clear with lawyers working with the Linux Foundation.
Criteria used to select files for SPDX license identifier tagging was:
- Files considered eligible had to be source code files.
- Make and config files were included as candidates if they contained >5
lines of source
- File already had some variant of a license header in it (even if <5
lines).
All documentation files were explicitly excluded.
The following heuristics were used to determine which SPDX license
identifiers to apply.
- when both scanners couldn't find any license traces, file was
considered to have no license information in it, and the top level
COPYING file license applied.
For non */uapi/* files that summary was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 11139
and resulted in the first patch in this series.
If that file was a */uapi/* path one, it was "GPL-2.0 WITH
Linux-syscall-note" otherwise it was "GPL-2.0". Results of that was:
SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 WITH Linux-syscall-note 930
and resulted in the second patch in this series.
- if a file had some form of licensing information in it, and was one
of the */uapi/* ones, it was denoted with the Linux-syscall-note if
any GPL family license was found in the file or had no licensing in
it (per prior point). Results summary:
SPDX license identifier # files
---------------------------------------------------|------
GPL-2.0 WITH Linux-syscall-note 270
GPL-2.0+ WITH Linux-syscall-note 169
((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) 21
((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) 17
LGPL-2.1+ WITH Linux-syscall-note 15
GPL-1.0+ WITH Linux-syscall-note 14
((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) 5
LGPL-2.0+ WITH Linux-syscall-note 4
LGPL-2.1 WITH Linux-syscall-note 3
((GPL-2.0 WITH Linux-syscall-note) OR MIT) 3
((GPL-2.0 WITH Linux-syscall-note) AND MIT) 1
and that resulted in the third patch in this series.
- when the two scanners agreed on the detected license(s), that became
the concluded license(s).
- when there was disagreement between the two scanners (one detected a
license but the other didn't, or they both detected different
licenses) a manual inspection of the file occurred.
- In most cases a manual inspection of the information in the file
resulted in a clear resolution of the license that should apply (and
which scanner probably needed to revisit its heuristics).
- When it was not immediately clear, the license identifier was
confirmed with lawyers working with the Linux Foundation.
- If there was any question as to the appropriate license identifier,
the file was flagged for further research and to be revisited later
in time.
In total, over 70 hours of logged manual review was done on the
spreadsheet to determine the SPDX license identifiers to apply to the
source files by Kate, Philippe, Thomas and, in some cases, confirmation
by lawyers working with the Linux Foundation.
Kate also obtained a third independent scan of the 4.13 code base from
FOSSology, and compared selected files where the other two scanners
disagreed against that SPDX file, to see if there was new insights. The
Windriver scanner is based on an older version of FOSSology in part, so
they are related.
Thomas did random spot checks in about 500 files from the spreadsheets
for the uapi headers and agreed with SPDX license identifier in the
files he inspected. For the non-uapi files Thomas did random spot checks
in about 15000 files.
In initial set of patches against 4.14-rc6, 3 files were found to have
copy/paste license identifier errors, and have been fixed to reflect the
correct identifier.
Additionally Philippe spent 10 hours this week doing a detailed manual
inspection and review of the 12,461 patched files from the initial patch
version early this week with:
- a full scancode scan run, collecting the matched texts, detected
license ids and scores
- reviewing anything where there was a license detected (about 500+
files) to ensure that the applied SPDX license was correct
- reviewing anything where there was no detection but the patch license
was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
SPDX license was correct
This produced a worksheet with 20 files needing minor correction. This
worksheet was then exported into 3 different .csv files for the
different types of files to be modified.
These .csv files were then reviewed by Greg. Thomas wrote a script to
parse the csv files and add the proper SPDX tag to the file, in the
format that the file expected. This script was further refined by Greg
based on the output to detect more types of files automatically and to
distinguish between header and source .c files (which need different
comment types.) Finally Greg ran the script using the .csv files to
generate the patches.
Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Philippe Ombredanne <pombredanne@nexb.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
2017-11-01 14:07:57 +00:00
|
|
|
// SPDX-License-Identifier: GPL-2.0
|
2017-02-01 12:10:18 +00:00
|
|
|
/*
|
|
|
|
|
* Scheduler topology setup/handling methods
|
|
|
|
|
*/
|
|
|
|
|
#include "sched.h"
|
|
|
|
|
|
|
|
|
|
DEFINE_MUTEX(sched_domains_mutex);
|
|
|
|
|
|
|
|
|
|
/* Protected by sched_domains_mutex: */
|
2018-08-03 12:37:32 +00:00
|
|
|
static cpumask_var_t sched_domains_tmpmask;
|
|
|
|
|
static cpumask_var_t sched_domains_tmpmask2;
|
2017-02-01 12:10:18 +00:00
|
|
|
|
|
|
|
|
#ifdef CONFIG_SCHED_DEBUG
|
|
|
|
|
|
|
|
|
|
static int __init sched_debug_setup(char *str)
|
|
|
|
|
{
|
2017-09-07 15:03:53 +00:00
|
|
|
sched_debug_enabled = true;
|
2017-02-01 12:10:18 +00:00
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
|
}
|
|
|
|
|
early_param("sched_debug", sched_debug_setup);
|
|
|
|
|
|
|
|
|
|
static inline bool sched_debug(void)
|
|
|
|
|
{
|
|
|
|
|
return sched_debug_enabled;
|
|
|
|
|
}
|
|
|
|
|
|
2020-09-08 18:49:56 +00:00
|
|
|
#define SD_FLAG(_name, mflags) [__##_name] = { .meta_flags = mflags, .name = #_name },
|
|
|
|
|
const struct sd_flag_debug sd_flag_debug[] = {
|
|
|
|
|
#include <linux/sched/sd_flags.h>
|
|
|
|
|
};
|
|
|
|
|
#undef SD_FLAG
|
|
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
static int sched_domain_debug_one(struct sched_domain *sd, int cpu, int level,
|
|
|
|
|
struct cpumask *groupmask)
|
|
|
|
|
{
|
|
|
|
|
struct sched_group *group = sd->groups;
|
2020-08-17 11:29:51 +00:00
|
|
|
unsigned long flags = sd->flags;
|
|
|
|
|
unsigned int idx;
|
2017-02-01 12:10:18 +00:00
|
|
|
|
|
|
|
|
cpumask_clear(groupmask);
|
|
|
|
|
|
sched/topology: Add sched_group_capacity debugging
Add sgc::id to easier spot domain construction issues.
Take the opportunity to slightly rework the group printing, because
adding more "(id: %d)" strings makes the entire thing very hard to
read. Also the individual groups are very hard to separate, so add
explicit visual grouping, which allows replacing all the "(%s: %d)"
format things with shorter "%s=%d" variants.
Then fix up some inconsistencies in surrounding prints for domains.
The end result looks like:
[] CPU0 attaching sched-domain(s):
[] domain-0: span=0,4 level=DIE
[] groups: 0:{ span=0 }, 4:{ span=4 }
[] domain-1: span=0-1,3-5,7 level=NUMA
[] groups: 0:{ span=0,4 mask=0,4 cap=2048 }, 1:{ span=1,5 mask=1,5 cap=2048 }, 3:{ span=3,7 mask=3,7 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 0:{ span=0-1,3-5,7 mask=0,4 cap=6144 }, 2:{ span=1-3,5-7 mask=2,6 cap=6144 }
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-04-26 15:35:35 +00:00
|
|
|
printk(KERN_DEBUG "%*s domain-%d: ", level, "", level);
|
|
|
|
|
printk(KERN_CONT "span=%*pbl level=%s\n",
|
2017-02-01 12:10:18 +00:00
|
|
|
cpumask_pr_args(sched_domain_span(sd)), sd->name);
|
|
|
|
|
|
|
|
|
|
if (!cpumask_test_cpu(cpu, sched_domain_span(sd))) {
|
sched: Clean up and harmonize the coding style of the scheduler code base
A good number of small style inconsistencies have accumulated
in the scheduler core, so do a pass over them to harmonize
all these details:
- fix speling in comments,
- use curly braces for multi-line statements,
- remove unnecessary parentheses from integer literals,
- capitalize consistently,
- remove stray newlines,
- add comments where necessary,
- remove invalid/unnecessary comments,
- align structure definitions and other data types vertically,
- add missing newlines for increased readability,
- fix vertical tabulation where it's misaligned,
- harmonize preprocessor conditional block labeling
and vertical alignment,
- remove line-breaks where they uglify the code,
- add newline after local variable definitions,
No change in functionality:
md5:
1191fa0a890cfa8132156d2959d7e9e2 built-in.o.before.asm
1191fa0a890cfa8132156d2959d7e9e2 built-in.o.after.asm
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-03-03 13:01:12 +00:00
|
|
|
printk(KERN_ERR "ERROR: domain->span does not contain CPU%d\n", cpu);
|
2017-02-01 12:10:18 +00:00
|
|
|
}
|
2018-07-23 04:19:07 +00:00
|
|
|
if (group && !cpumask_test_cpu(cpu, sched_group_span(group))) {
|
sched: Clean up and harmonize the coding style of the scheduler code base
A good number of small style inconsistencies have accumulated
in the scheduler core, so do a pass over them to harmonize
all these details:
- fix speling in comments,
- use curly braces for multi-line statements,
- remove unnecessary parentheses from integer literals,
- capitalize consistently,
- remove stray newlines,
- add comments where necessary,
- remove invalid/unnecessary comments,
- align structure definitions and other data types vertically,
- add missing newlines for increased readability,
- fix vertical tabulation where it's misaligned,
- harmonize preprocessor conditional block labeling
and vertical alignment,
- remove line-breaks where they uglify the code,
- add newline after local variable definitions,
No change in functionality:
md5:
1191fa0a890cfa8132156d2959d7e9e2 built-in.o.before.asm
1191fa0a890cfa8132156d2959d7e9e2 built-in.o.after.asm
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-03-03 13:01:12 +00:00
|
|
|
printk(KERN_ERR "ERROR: domain->groups does not contain CPU%d\n", cpu);
|
2017-02-01 12:10:18 +00:00
|
|
|
}
|
|
|
|
|
|
2020-08-17 11:29:51 +00:00
|
|
|
for_each_set_bit(idx, &flags, __SD_FLAG_CNT) {
|
|
|
|
|
unsigned int flag = BIT(idx);
|
|
|
|
|
unsigned int meta_flags = sd_flag_debug[idx].meta_flags;
|
|
|
|
|
|
|
|
|
|
if ((meta_flags & SDF_SHARED_CHILD) && sd->child &&
|
|
|
|
|
!(sd->child->flags & flag))
|
|
|
|
|
printk(KERN_ERR "ERROR: flag %s set here but not in child\n",
|
|
|
|
|
sd_flag_debug[idx].name);
|
|
|
|
|
|
|
|
|
|
if ((meta_flags & SDF_SHARED_PARENT) && sd->parent &&
|
|
|
|
|
!(sd->parent->flags & flag))
|
|
|
|
|
printk(KERN_ERR "ERROR: flag %s set here but not in parent\n",
|
|
|
|
|
sd_flag_debug[idx].name);
|
|
|
|
|
}
|
|
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
printk(KERN_DEBUG "%*s groups:", level + 1, "");
|
|
|
|
|
do {
|
|
|
|
|
if (!group) {
|
|
|
|
|
printk("\n");
|
|
|
|
|
printk(KERN_ERR "ERROR: group is NULL\n");
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
|
2017-05-01 09:03:12 +00:00
|
|
|
if (!cpumask_weight(sched_group_span(group))) {
|
2017-02-01 12:10:18 +00:00
|
|
|
printk(KERN_CONT "\n");
|
|
|
|
|
printk(KERN_ERR "ERROR: empty group\n");
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (!(sd->flags & SD_OVERLAP) &&
|
2017-05-01 09:03:12 +00:00
|
|
|
cpumask_intersects(groupmask, sched_group_span(group))) {
|
2017-02-01 12:10:18 +00:00
|
|
|
printk(KERN_CONT "\n");
|
|
|
|
|
printk(KERN_ERR "ERROR: repeated CPUs\n");
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
|
2017-05-01 09:03:12 +00:00
|
|
|
cpumask_or(groupmask, groupmask, sched_group_span(group));
|
2017-02-01 12:10:18 +00:00
|
|
|
|
sched/topology: Add sched_group_capacity debugging
Add sgc::id to easier spot domain construction issues.
Take the opportunity to slightly rework the group printing, because
adding more "(id: %d)" strings makes the entire thing very hard to
read. Also the individual groups are very hard to separate, so add
explicit visual grouping, which allows replacing all the "(%s: %d)"
format things with shorter "%s=%d" variants.
Then fix up some inconsistencies in surrounding prints for domains.
The end result looks like:
[] CPU0 attaching sched-domain(s):
[] domain-0: span=0,4 level=DIE
[] groups: 0:{ span=0 }, 4:{ span=4 }
[] domain-1: span=0-1,3-5,7 level=NUMA
[] groups: 0:{ span=0,4 mask=0,4 cap=2048 }, 1:{ span=1,5 mask=1,5 cap=2048 }, 3:{ span=3,7 mask=3,7 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 0:{ span=0-1,3-5,7 mask=0,4 cap=6144 }, 2:{ span=1-3,5-7 mask=2,6 cap=6144 }
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-04-26 15:35:35 +00:00
|
|
|
printk(KERN_CONT " %d:{ span=%*pbl",
|
|
|
|
|
group->sgc->id,
|
2017-05-01 09:03:12 +00:00
|
|
|
cpumask_pr_args(sched_group_span(group)));
|
2017-04-14 15:29:16 +00:00
|
|
|
|
2017-05-01 06:51:05 +00:00
|
|
|
if ((sd->flags & SD_OVERLAP) &&
|
2017-05-01 09:03:12 +00:00
|
|
|
!cpumask_equal(group_balance_mask(group), sched_group_span(group))) {
|
sched/topology: Add sched_group_capacity debugging
Add sgc::id to easier spot domain construction issues.
Take the opportunity to slightly rework the group printing, because
adding more "(id: %d)" strings makes the entire thing very hard to
read. Also the individual groups are very hard to separate, so add
explicit visual grouping, which allows replacing all the "(%s: %d)"
format things with shorter "%s=%d" variants.
Then fix up some inconsistencies in surrounding prints for domains.
The end result looks like:
[] CPU0 attaching sched-domain(s):
[] domain-0: span=0,4 level=DIE
[] groups: 0:{ span=0 }, 4:{ span=4 }
[] domain-1: span=0-1,3-5,7 level=NUMA
[] groups: 0:{ span=0,4 mask=0,4 cap=2048 }, 1:{ span=1,5 mask=1,5 cap=2048 }, 3:{ span=3,7 mask=3,7 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 0:{ span=0-1,3-5,7 mask=0,4 cap=6144 }, 2:{ span=1-3,5-7 mask=2,6 cap=6144 }
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-04-26 15:35:35 +00:00
|
|
|
printk(KERN_CONT " mask=%*pbl",
|
2017-05-01 08:47:02 +00:00
|
|
|
cpumask_pr_args(group_balance_mask(group)));
|
2017-04-14 15:29:16 +00:00
|
|
|
}
|
|
|
|
|
|
sched/topology: Add sched_group_capacity debugging
Add sgc::id to easier spot domain construction issues.
Take the opportunity to slightly rework the group printing, because
adding more "(id: %d)" strings makes the entire thing very hard to
read. Also the individual groups are very hard to separate, so add
explicit visual grouping, which allows replacing all the "(%s: %d)"
format things with shorter "%s=%d" variants.
Then fix up some inconsistencies in surrounding prints for domains.
The end result looks like:
[] CPU0 attaching sched-domain(s):
[] domain-0: span=0,4 level=DIE
[] groups: 0:{ span=0 }, 4:{ span=4 }
[] domain-1: span=0-1,3-5,7 level=NUMA
[] groups: 0:{ span=0,4 mask=0,4 cap=2048 }, 1:{ span=1,5 mask=1,5 cap=2048 }, 3:{ span=3,7 mask=3,7 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 0:{ span=0-1,3-5,7 mask=0,4 cap=6144 }, 2:{ span=1-3,5-7 mask=2,6 cap=6144 }
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-04-26 15:35:35 +00:00
|
|
|
if (group->sgc->capacity != SCHED_CAPACITY_SCALE)
|
|
|
|
|
printk(KERN_CONT " cap=%lu", group->sgc->capacity);
|
2017-02-01 12:10:18 +00:00
|
|
|
|
2017-04-14 16:20:48 +00:00
|
|
|
if (group == sd->groups && sd->child &&
|
|
|
|
|
!cpumask_equal(sched_domain_span(sd->child),
|
2017-05-01 09:03:12 +00:00
|
|
|
sched_group_span(group))) {
|
2017-04-14 16:20:48 +00:00
|
|
|
printk(KERN_ERR "ERROR: domain->groups does not match domain->child\n");
|
|
|
|
|
}
|
|
|
|
|
|
sched/topology: Add sched_group_capacity debugging
Add sgc::id to easier spot domain construction issues.
Take the opportunity to slightly rework the group printing, because
adding more "(id: %d)" strings makes the entire thing very hard to
read. Also the individual groups are very hard to separate, so add
explicit visual grouping, which allows replacing all the "(%s: %d)"
format things with shorter "%s=%d" variants.
Then fix up some inconsistencies in surrounding prints for domains.
The end result looks like:
[] CPU0 attaching sched-domain(s):
[] domain-0: span=0,4 level=DIE
[] groups: 0:{ span=0 }, 4:{ span=4 }
[] domain-1: span=0-1,3-5,7 level=NUMA
[] groups: 0:{ span=0,4 mask=0,4 cap=2048 }, 1:{ span=1,5 mask=1,5 cap=2048 }, 3:{ span=3,7 mask=3,7 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 0:{ span=0-1,3-5,7 mask=0,4 cap=6144 }, 2:{ span=1-3,5-7 mask=2,6 cap=6144 }
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-04-26 15:35:35 +00:00
|
|
|
printk(KERN_CONT " }");
|
|
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
group = group->next;
|
2017-04-14 15:29:16 +00:00
|
|
|
|
|
|
|
|
if (group != sd->groups)
|
|
|
|
|
printk(KERN_CONT ",");
|
|
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
} while (group != sd->groups);
|
|
|
|
|
printk(KERN_CONT "\n");
|
|
|
|
|
|
|
|
|
|
if (!cpumask_equal(sched_domain_span(sd), groupmask))
|
|
|
|
|
printk(KERN_ERR "ERROR: groups don't span domain->span\n");
|
|
|
|
|
|
|
|
|
|
if (sd->parent &&
|
|
|
|
|
!cpumask_subset(groupmask, sched_domain_span(sd->parent)))
|
sched: Clean up and harmonize the coding style of the scheduler code base
A good number of small style inconsistencies have accumulated
in the scheduler core, so do a pass over them to harmonize
all these details:
- fix speling in comments,
- use curly braces for multi-line statements,
- remove unnecessary parentheses from integer literals,
- capitalize consistently,
- remove stray newlines,
- add comments where necessary,
- remove invalid/unnecessary comments,
- align structure definitions and other data types vertically,
- add missing newlines for increased readability,
- fix vertical tabulation where it's misaligned,
- harmonize preprocessor conditional block labeling
and vertical alignment,
- remove line-breaks where they uglify the code,
- add newline after local variable definitions,
No change in functionality:
md5:
1191fa0a890cfa8132156d2959d7e9e2 built-in.o.before.asm
1191fa0a890cfa8132156d2959d7e9e2 built-in.o.after.asm
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-03-03 13:01:12 +00:00
|
|
|
printk(KERN_ERR "ERROR: parent span is not a superset of domain->span\n");
|
2017-02-01 12:10:18 +00:00
|
|
|
return 0;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void sched_domain_debug(struct sched_domain *sd, int cpu)
|
|
|
|
|
{
|
|
|
|
|
int level = 0;
|
|
|
|
|
|
|
|
|
|
if (!sched_debug_enabled)
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
if (!sd) {
|
|
|
|
|
printk(KERN_DEBUG "CPU%d attaching NULL sched-domain.\n", cpu);
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
|
sched/topology: Add sched_group_capacity debugging
Add sgc::id to easier spot domain construction issues.
Take the opportunity to slightly rework the group printing, because
adding more "(id: %d)" strings makes the entire thing very hard to
read. Also the individual groups are very hard to separate, so add
explicit visual grouping, which allows replacing all the "(%s: %d)"
format things with shorter "%s=%d" variants.
Then fix up some inconsistencies in surrounding prints for domains.
The end result looks like:
[] CPU0 attaching sched-domain(s):
[] domain-0: span=0,4 level=DIE
[] groups: 0:{ span=0 }, 4:{ span=4 }
[] domain-1: span=0-1,3-5,7 level=NUMA
[] groups: 0:{ span=0,4 mask=0,4 cap=2048 }, 1:{ span=1,5 mask=1,5 cap=2048 }, 3:{ span=3,7 mask=3,7 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 0:{ span=0-1,3-5,7 mask=0,4 cap=6144 }, 2:{ span=1-3,5-7 mask=2,6 cap=6144 }
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-04-26 15:35:35 +00:00
|
|
|
printk(KERN_DEBUG "CPU%d attaching sched-domain(s):\n", cpu);
|
2017-02-01 12:10:18 +00:00
|
|
|
|
|
|
|
|
for (;;) {
|
|
|
|
|
if (sched_domain_debug_one(sd, cpu, level, sched_domains_tmpmask))
|
|
|
|
|
break;
|
|
|
|
|
level++;
|
|
|
|
|
sd = sd->parent;
|
|
|
|
|
if (!sd)
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
#else /* !CONFIG_SCHED_DEBUG */
|
|
|
|
|
|
|
|
|
|
# define sched_debug_enabled 0
|
|
|
|
|
# define sched_domain_debug(sd, cpu) do { } while (0)
|
|
|
|
|
static inline bool sched_debug(void)
|
|
|
|
|
{
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
#endif /* CONFIG_SCHED_DEBUG */
|
|
|
|
|
|
2020-08-25 13:32:16 +00:00
|
|
|
/* Generate a mask of SD flags with the SDF_NEEDS_GROUPS metaflag */
|
|
|
|
|
#define SD_FLAG(name, mflags) (name * !!((mflags) & SDF_NEEDS_GROUPS)) |
|
|
|
|
|
static const unsigned int SD_DEGENERATE_GROUPS_MASK =
|
|
|
|
|
#include <linux/sched/sd_flags.h>
|
|
|
|
|
0;
|
|
|
|
|
#undef SD_FLAG
|
|
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
static int sd_degenerate(struct sched_domain *sd)
|
|
|
|
|
{
|
|
|
|
|
if (cpumask_weight(sched_domain_span(sd)) == 1)
|
|
|
|
|
return 1;
|
|
|
|
|
|
|
|
|
|
/* Following flags need at least 2 groups */
|
2020-08-17 11:29:54 +00:00
|
|
|
if ((sd->flags & SD_DEGENERATE_GROUPS_MASK) &&
|
|
|
|
|
(sd->groups != sd->groups->next))
|
|
|
|
|
return 0;
|
2017-02-01 12:10:18 +00:00
|
|
|
|
|
|
|
|
/* Following flags don't use groups */
|
|
|
|
|
if (sd->flags & (SD_WAKE_AFFINE))
|
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
|
|
return 1;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static int
|
|
|
|
|
sd_parent_degenerate(struct sched_domain *sd, struct sched_domain *parent)
|
|
|
|
|
{
|
|
|
|
|
unsigned long cflags = sd->flags, pflags = parent->flags;
|
|
|
|
|
|
|
|
|
|
if (sd_degenerate(parent))
|
|
|
|
|
return 1;
|
|
|
|
|
|
|
|
|
|
if (!cpumask_equal(sched_domain_span(sd), sched_domain_span(parent)))
|
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
|
|
/* Flags needing groups don't count if only 1 group in parent */
|
2020-08-17 11:29:55 +00:00
|
|
|
if (parent->groups == parent->groups->next)
|
2020-08-17 11:29:57 +00:00
|
|
|
pflags &= ~SD_DEGENERATE_GROUPS_MASK;
|
2020-08-17 11:29:55 +00:00
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
if (~cflags & pflags)
|
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
|
|
return 1;
|
|
|
|
|
}
|
|
|
|
|
|
2018-12-03 09:56:21 +00:00
|
|
|
#if defined(CONFIG_ENERGY_MODEL) && defined(CONFIG_CPU_FREQ_GOV_SCHEDUTIL)
|
2018-12-05 10:23:56 +00:00
|
|
|
DEFINE_STATIC_KEY_FALSE(sched_energy_present);
|
2018-12-03 09:56:23 +00:00
|
|
|
unsigned int sysctl_sched_energy_aware = 1;
|
2018-12-03 09:56:21 +00:00
|
|
|
DEFINE_MUTEX(sched_energy_mutex);
|
|
|
|
|
bool sched_energy_update;
|
|
|
|
|
|
2020-10-27 18:07:11 +00:00
|
|
|
void rebuild_sched_domains_energy(void)
|
|
|
|
|
{
|
|
|
|
|
mutex_lock(&sched_energy_mutex);
|
|
|
|
|
sched_energy_update = true;
|
|
|
|
|
rebuild_sched_domains();
|
|
|
|
|
sched_energy_update = false;
|
|
|
|
|
mutex_unlock(&sched_energy_mutex);
|
|
|
|
|
}
|
|
|
|
|
|
2018-12-03 09:56:23 +00:00
|
|
|
#ifdef CONFIG_PROC_SYSCTL
|
|
|
|
|
int sched_energy_aware_handler(struct ctl_table *table, int write,
|
2020-04-24 06:43:38 +00:00
|
|
|
void *buffer, size_t *lenp, loff_t *ppos)
|
2018-12-03 09:56:23 +00:00
|
|
|
{
|
|
|
|
|
int ret, state;
|
|
|
|
|
|
|
|
|
|
if (write && !capable(CAP_SYS_ADMIN))
|
|
|
|
|
return -EPERM;
|
|
|
|
|
|
|
|
|
|
ret = proc_dointvec_minmax(table, write, buffer, lenp, ppos);
|
|
|
|
|
if (!ret && write) {
|
|
|
|
|
state = static_branch_unlikely(&sched_energy_present);
|
2020-10-27 18:07:11 +00:00
|
|
|
if (state != sysctl_sched_energy_aware)
|
|
|
|
|
rebuild_sched_domains_energy();
|
2018-12-03 09:56:23 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return ret;
|
|
|
|
|
}
|
|
|
|
|
#endif
|
|
|
|
|
|
2018-12-03 09:56:18 +00:00
|
|
|
static void free_pd(struct perf_domain *pd)
|
|
|
|
|
{
|
|
|
|
|
struct perf_domain *tmp;
|
|
|
|
|
|
|
|
|
|
while (pd) {
|
|
|
|
|
tmp = pd->next;
|
|
|
|
|
kfree(pd);
|
|
|
|
|
pd = tmp;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static struct perf_domain *find_pd(struct perf_domain *pd, int cpu)
|
|
|
|
|
{
|
|
|
|
|
while (pd) {
|
|
|
|
|
if (cpumask_test_cpu(cpu, perf_domain_span(pd)))
|
|
|
|
|
return pd;
|
|
|
|
|
pd = pd->next;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return NULL;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static struct perf_domain *pd_init(int cpu)
|
|
|
|
|
{
|
|
|
|
|
struct em_perf_domain *obj = em_cpu_get(cpu);
|
|
|
|
|
struct perf_domain *pd;
|
|
|
|
|
|
|
|
|
|
if (!obj) {
|
|
|
|
|
if (sched_debug())
|
|
|
|
|
pr_info("%s: no EM found for CPU%d\n", __func__, cpu);
|
|
|
|
|
return NULL;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
pd = kzalloc(sizeof(*pd), GFP_KERNEL);
|
|
|
|
|
if (!pd)
|
|
|
|
|
return NULL;
|
|
|
|
|
pd->em_pd = obj;
|
|
|
|
|
|
|
|
|
|
return pd;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void perf_domain_debug(const struct cpumask *cpu_map,
|
|
|
|
|
struct perf_domain *pd)
|
|
|
|
|
{
|
|
|
|
|
if (!sched_debug() || !pd)
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
printk(KERN_DEBUG "root_domain %*pbl:", cpumask_pr_args(cpu_map));
|
|
|
|
|
|
|
|
|
|
while (pd) {
|
2020-05-27 09:58:47 +00:00
|
|
|
printk(KERN_CONT " pd%d:{ cpus=%*pbl nr_pstate=%d }",
|
2018-12-03 09:56:18 +00:00
|
|
|
cpumask_first(perf_domain_span(pd)),
|
|
|
|
|
cpumask_pr_args(perf_domain_span(pd)),
|
2020-05-27 09:58:47 +00:00
|
|
|
em_pd_nr_perf_states(pd->em_pd));
|
2018-12-03 09:56:18 +00:00
|
|
|
pd = pd->next;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
printk(KERN_CONT "\n");
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void destroy_perf_domain_rcu(struct rcu_head *rp)
|
|
|
|
|
{
|
|
|
|
|
struct perf_domain *pd;
|
|
|
|
|
|
|
|
|
|
pd = container_of(rp, struct perf_domain, rcu);
|
|
|
|
|
free_pd(pd);
|
|
|
|
|
}
|
|
|
|
|
|
2018-12-03 09:56:22 +00:00
|
|
|
static void sched_energy_set(bool has_eas)
|
|
|
|
|
{
|
|
|
|
|
if (!has_eas && static_branch_unlikely(&sched_energy_present)) {
|
|
|
|
|
if (sched_debug())
|
|
|
|
|
pr_info("%s: stopping EAS\n", __func__);
|
|
|
|
|
static_branch_disable_cpuslocked(&sched_energy_present);
|
|
|
|
|
} else if (has_eas && !static_branch_unlikely(&sched_energy_present)) {
|
|
|
|
|
if (sched_debug())
|
|
|
|
|
pr_info("%s: starting EAS\n", __func__);
|
|
|
|
|
static_branch_enable_cpuslocked(&sched_energy_present);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2018-12-03 09:56:20 +00:00
|
|
|
/*
|
|
|
|
|
* EAS can be used on a root domain if it meets all the following conditions:
|
|
|
|
|
* 1. an Energy Model (EM) is available;
|
|
|
|
|
* 2. the SD_ASYM_CPUCAPACITY flag is set in the sched_domain hierarchy.
|
2020-02-27 19:14:32 +00:00
|
|
|
* 3. no SMT is detected.
|
|
|
|
|
* 4. the EM complexity is low enough to keep scheduling overheads low;
|
|
|
|
|
* 5. schedutil is driving the frequency of all CPUs of the rd;
|
2020-10-27 18:07:13 +00:00
|
|
|
* 6. frequency invariance support is present;
|
2018-12-03 09:56:20 +00:00
|
|
|
*
|
|
|
|
|
* The complexity of the Energy Model is defined as:
|
|
|
|
|
*
|
2020-05-27 09:58:47 +00:00
|
|
|
* C = nr_pd * (nr_cpus + nr_ps)
|
2018-12-03 09:56:20 +00:00
|
|
|
*
|
|
|
|
|
* with parameters defined as:
|
|
|
|
|
* - nr_pd: the number of performance domains
|
|
|
|
|
* - nr_cpus: the number of CPUs
|
2020-05-27 09:58:47 +00:00
|
|
|
* - nr_ps: the sum of the number of performance states of all performance
|
2018-12-03 09:56:20 +00:00
|
|
|
* domains (for example, on a system with 2 performance domains,
|
2020-05-27 09:58:47 +00:00
|
|
|
* with 10 performance states each, nr_ps = 2 * 10 = 20).
|
2018-12-03 09:56:20 +00:00
|
|
|
*
|
|
|
|
|
* It is generally not a good idea to use such a model in the wake-up path on
|
|
|
|
|
* very complex platforms because of the associated scheduling overheads. The
|
|
|
|
|
* arbitrary constraint below prevents that. It makes EAS usable up to 16 CPUs
|
2020-05-27 09:58:47 +00:00
|
|
|
* with per-CPU DVFS and less than 8 performance states each, for example.
|
2018-12-03 09:56:20 +00:00
|
|
|
*/
|
|
|
|
|
#define EM_MAX_COMPLEXITY 2048
|
|
|
|
|
|
2018-12-03 09:56:21 +00:00
|
|
|
extern struct cpufreq_governor schedutil_gov;
|
2018-12-03 09:56:22 +00:00
|
|
|
static bool build_perf_domains(const struct cpumask *cpu_map)
|
2018-12-03 09:56:18 +00:00
|
|
|
{
|
2020-05-27 09:58:47 +00:00
|
|
|
int i, nr_pd = 0, nr_ps = 0, nr_cpus = cpumask_weight(cpu_map);
|
2018-12-03 09:56:18 +00:00
|
|
|
struct perf_domain *pd = NULL, *tmp;
|
|
|
|
|
int cpu = cpumask_first(cpu_map);
|
|
|
|
|
struct root_domain *rd = cpu_rq(cpu)->rd;
|
2018-12-03 09:56:21 +00:00
|
|
|
struct cpufreq_policy *policy;
|
|
|
|
|
struct cpufreq_governor *gov;
|
2018-12-03 09:56:20 +00:00
|
|
|
|
2018-12-03 09:56:23 +00:00
|
|
|
if (!sysctl_sched_energy_aware)
|
|
|
|
|
goto free;
|
|
|
|
|
|
2018-12-03 09:56:20 +00:00
|
|
|
/* EAS is enabled for asymmetric CPU capacity topologies. */
|
|
|
|
|
if (!per_cpu(sd_asym_cpucapacity, cpu)) {
|
|
|
|
|
if (sched_debug()) {
|
|
|
|
|
pr_info("rd %*pbl: CPUs do not have asymmetric capacities\n",
|
|
|
|
|
cpumask_pr_args(cpu_map));
|
|
|
|
|
}
|
|
|
|
|
goto free;
|
|
|
|
|
}
|
2018-12-03 09:56:18 +00:00
|
|
|
|
2020-02-27 19:14:32 +00:00
|
|
|
/* EAS definitely does *not* handle SMT */
|
|
|
|
|
if (sched_smt_active()) {
|
|
|
|
|
pr_warn("rd %*pbl: Disabling EAS, SMT is not supported\n",
|
|
|
|
|
cpumask_pr_args(cpu_map));
|
|
|
|
|
goto free;
|
|
|
|
|
}
|
|
|
|
|
|
2020-10-27 18:07:13 +00:00
|
|
|
if (!arch_scale_freq_invariant()) {
|
|
|
|
|
if (sched_debug()) {
|
|
|
|
|
pr_warn("rd %*pbl: Disabling EAS: frequency-invariant load tracking not yet supported",
|
|
|
|
|
cpumask_pr_args(cpu_map));
|
|
|
|
|
}
|
|
|
|
|
goto free;
|
|
|
|
|
}
|
|
|
|
|
|
2018-12-03 09:56:18 +00:00
|
|
|
for_each_cpu(i, cpu_map) {
|
|
|
|
|
/* Skip already covered CPUs. */
|
|
|
|
|
if (find_pd(pd, i))
|
|
|
|
|
continue;
|
|
|
|
|
|
2018-12-03 09:56:21 +00:00
|
|
|
/* Do not attempt EAS if schedutil is not being used. */
|
|
|
|
|
policy = cpufreq_cpu_get(i);
|
|
|
|
|
if (!policy)
|
|
|
|
|
goto free;
|
|
|
|
|
gov = policy->governor;
|
|
|
|
|
cpufreq_cpu_put(policy);
|
|
|
|
|
if (gov != &schedutil_gov) {
|
|
|
|
|
if (rd->pd)
|
|
|
|
|
pr_warn("rd %*pbl: Disabling EAS, schedutil is mandatory\n",
|
|
|
|
|
cpumask_pr_args(cpu_map));
|
|
|
|
|
goto free;
|
|
|
|
|
}
|
|
|
|
|
|
2018-12-03 09:56:18 +00:00
|
|
|
/* Create the new pd and add it to the local list. */
|
|
|
|
|
tmp = pd_init(i);
|
|
|
|
|
if (!tmp)
|
|
|
|
|
goto free;
|
|
|
|
|
tmp->next = pd;
|
|
|
|
|
pd = tmp;
|
2018-12-03 09:56:20 +00:00
|
|
|
|
|
|
|
|
/*
|
2020-05-27 09:58:47 +00:00
|
|
|
* Count performance domains and performance states for the
|
2018-12-03 09:56:20 +00:00
|
|
|
* complexity check.
|
|
|
|
|
*/
|
|
|
|
|
nr_pd++;
|
2020-05-27 09:58:47 +00:00
|
|
|
nr_ps += em_pd_nr_perf_states(pd->em_pd);
|
2018-12-03 09:56:20 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* Bail out if the Energy Model complexity is too high. */
|
2020-05-27 09:58:47 +00:00
|
|
|
if (nr_pd * (nr_ps + nr_cpus) > EM_MAX_COMPLEXITY) {
|
2018-12-03 09:56:20 +00:00
|
|
|
WARN(1, "rd %*pbl: Failed to start EAS, EM complexity is too high\n",
|
|
|
|
|
cpumask_pr_args(cpu_map));
|
|
|
|
|
goto free;
|
2018-12-03 09:56:18 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
perf_domain_debug(cpu_map, pd);
|
|
|
|
|
|
|
|
|
|
/* Attach the new list of performance domains to the root domain. */
|
|
|
|
|
tmp = rd->pd;
|
|
|
|
|
rcu_assign_pointer(rd->pd, pd);
|
|
|
|
|
if (tmp)
|
|
|
|
|
call_rcu(&tmp->rcu, destroy_perf_domain_rcu);
|
|
|
|
|
|
2018-12-03 09:56:22 +00:00
|
|
|
return !!pd;
|
2018-12-03 09:56:18 +00:00
|
|
|
|
|
|
|
|
free:
|
|
|
|
|
free_pd(pd);
|
|
|
|
|
tmp = rd->pd;
|
|
|
|
|
rcu_assign_pointer(rd->pd, NULL);
|
|
|
|
|
if (tmp)
|
|
|
|
|
call_rcu(&tmp->rcu, destroy_perf_domain_rcu);
|
2018-12-03 09:56:22 +00:00
|
|
|
|
|
|
|
|
return false;
|
2018-12-03 09:56:18 +00:00
|
|
|
}
|
|
|
|
|
#else
|
|
|
|
|
static void free_pd(struct perf_domain *pd) { }
|
2018-12-03 09:56:21 +00:00
|
|
|
#endif /* CONFIG_ENERGY_MODEL && CONFIG_CPU_FREQ_GOV_SCHEDUTIL*/
|
2018-12-03 09:56:18 +00:00
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
static void free_rootdomain(struct rcu_head *rcu)
|
|
|
|
|
{
|
|
|
|
|
struct root_domain *rd = container_of(rcu, struct root_domain, rcu);
|
|
|
|
|
|
|
|
|
|
cpupri_cleanup(&rd->cpupri);
|
|
|
|
|
cpudl_cleanup(&rd->cpudl);
|
|
|
|
|
free_cpumask_var(rd->dlo_mask);
|
|
|
|
|
free_cpumask_var(rd->rto_mask);
|
|
|
|
|
free_cpumask_var(rd->online);
|
|
|
|
|
free_cpumask_var(rd->span);
|
2018-12-03 09:56:18 +00:00
|
|
|
free_pd(rd->pd);
|
2017-02-01 12:10:18 +00:00
|
|
|
kfree(rd);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void rq_attach_root(struct rq *rq, struct root_domain *rd)
|
|
|
|
|
{
|
|
|
|
|
struct root_domain *old_rd = NULL;
|
|
|
|
|
unsigned long flags;
|
|
|
|
|
|
|
|
|
|
raw_spin_lock_irqsave(&rq->lock, flags);
|
|
|
|
|
|
|
|
|
|
if (rq->rd) {
|
|
|
|
|
old_rd = rq->rd;
|
|
|
|
|
|
|
|
|
|
if (cpumask_test_cpu(rq->cpu, old_rd->online))
|
|
|
|
|
set_rq_offline(rq);
|
|
|
|
|
|
|
|
|
|
cpumask_clear_cpu(rq->cpu, old_rd->span);
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* If we dont want to free the old_rd yet then
|
|
|
|
|
* set old_rd to NULL to skip the freeing later
|
|
|
|
|
* in this function:
|
|
|
|
|
*/
|
|
|
|
|
if (!atomic_dec_and_test(&old_rd->refcount))
|
|
|
|
|
old_rd = NULL;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
atomic_inc(&rd->refcount);
|
|
|
|
|
rq->rd = rd;
|
|
|
|
|
|
|
|
|
|
cpumask_set_cpu(rq->cpu, rd->span);
|
|
|
|
|
if (cpumask_test_cpu(rq->cpu, cpu_active_mask))
|
|
|
|
|
set_rq_online(rq);
|
|
|
|
|
|
|
|
|
|
raw_spin_unlock_irqrestore(&rq->lock, flags);
|
|
|
|
|
|
|
|
|
|
if (old_rd)
|
2018-11-07 03:10:53 +00:00
|
|
|
call_rcu(&old_rd->rcu, free_rootdomain);
|
2017-02-01 12:10:18 +00:00
|
|
|
}
|
|
|
|
|
|
2018-01-24 01:45:38 +00:00
|
|
|
void sched_get_rd(struct root_domain *rd)
|
|
|
|
|
{
|
|
|
|
|
atomic_inc(&rd->refcount);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void sched_put_rd(struct root_domain *rd)
|
|
|
|
|
{
|
|
|
|
|
if (!atomic_dec_and_test(&rd->refcount))
|
|
|
|
|
return;
|
|
|
|
|
|
2018-11-07 03:10:53 +00:00
|
|
|
call_rcu(&rd->rcu, free_rootdomain);
|
2018-01-24 01:45:38 +00:00
|
|
|
}
|
|
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
static int init_rootdomain(struct root_domain *rd)
|
|
|
|
|
{
|
|
|
|
|
if (!zalloc_cpumask_var(&rd->span, GFP_KERNEL))
|
|
|
|
|
goto out;
|
|
|
|
|
if (!zalloc_cpumask_var(&rd->online, GFP_KERNEL))
|
|
|
|
|
goto free_span;
|
|
|
|
|
if (!zalloc_cpumask_var(&rd->dlo_mask, GFP_KERNEL))
|
|
|
|
|
goto free_online;
|
|
|
|
|
if (!zalloc_cpumask_var(&rd->rto_mask, GFP_KERNEL))
|
|
|
|
|
goto free_dlo_mask;
|
|
|
|
|
|
sched/rt: Simplify the IPI based RT balancing logic
When a CPU lowers its priority (schedules out a high priority task for a
lower priority one), a check is made to see if any other CPU has overloaded
RT tasks (more than one). It checks the rto_mask to determine this and if so
it will request to pull one of those tasks to itself if the non running RT
task is of higher priority than the new priority of the next task to run on
the current CPU.
When we deal with large number of CPUs, the original pull logic suffered
from large lock contention on a single CPU run queue, which caused a huge
latency across all CPUs. This was caused by only having one CPU having
overloaded RT tasks and a bunch of other CPUs lowering their priority. To
solve this issue, commit:
b6366f048e0c ("sched/rt: Use IPI to trigger RT task push migration instead of pulling")
changed the way to request a pull. Instead of grabbing the lock of the
overloaded CPU's runqueue, it simply sent an IPI to that CPU to do the work.
Although the IPI logic worked very well in removing the large latency build
up, it still could suffer from a large number of IPIs being sent to a single
CPU. On a 80 CPU box, I measured over 200us of processing IPIs. Worse yet,
when I tested this on a 120 CPU box, with a stress test that had lots of
RT tasks scheduling on all CPUs, it actually triggered the hard lockup
detector! One CPU had so many IPIs sent to it, and due to the restart
mechanism that is triggered when the source run queue has a priority status
change, the CPU spent minutes! processing the IPIs.
Thinking about this further, I realized there's no reason for each run queue
to send its own IPI. As all CPUs with overloaded tasks must be scanned
regardless if there's one or many CPUs lowering their priority, because
there's no current way to find the CPU with the highest priority task that
can schedule to one of these CPUs, there really only needs to be one IPI
being sent around at a time.
This greatly simplifies the code!
The new approach is to have each root domain have its own irq work, as the
rto_mask is per root domain. The root domain has the following fields
attached to it:
rto_push_work - the irq work to process each CPU set in rto_mask
rto_lock - the lock to protect some of the other rto fields
rto_loop_start - an atomic that keeps contention down on rto_lock
the first CPU scheduling in a lower priority task
is the one to kick off the process.
rto_loop_next - an atomic that gets incremented for each CPU that
schedules in a lower priority task.
rto_loop - a variable protected by rto_lock that is used to
compare against rto_loop_next
rto_cpu - The cpu to send the next IPI to, also protected by
the rto_lock.
When a CPU schedules in a lower priority task and wants to make sure
overloaded CPUs know about it. It increments the rto_loop_next. Then it
atomically sets rto_loop_start with a cmpxchg. If the old value is not "0",
then it is done, as another CPU is kicking off the IPI loop. If the old
value is "0", then it will take the rto_lock to synchronize with a possible
IPI being sent around to the overloaded CPUs.
If rto_cpu is greater than or equal to nr_cpu_ids, then there's either no
IPI being sent around, or one is about to finish. Then rto_cpu is set to the
first CPU in rto_mask and an IPI is sent to that CPU. If there's no CPUs set
in rto_mask, then there's nothing to be done.
When the CPU receives the IPI, it will first try to push any RT tasks that is
queued on the CPU but can't run because a higher priority RT task is
currently running on that CPU.
Then it takes the rto_lock and looks for the next CPU in the rto_mask. If it
finds one, it simply sends an IPI to that CPU and the process continues.
If there's no more CPUs in the rto_mask, then rto_loop is compared with
rto_loop_next. If they match, everything is done and the process is over. If
they do not match, then a CPU scheduled in a lower priority task as the IPI
was being passed around, and the process needs to start again. The first CPU
in rto_mask is sent the IPI.
This change removes this duplication of work in the IPI logic, and greatly
lowers the latency caused by the IPIs. This removed the lockup happening on
the 120 CPU machine. It also simplifies the code tremendously. What else
could anyone ask for?
Thanks to Peter Zijlstra for simplifying the rto_loop_start atomic logic and
supplying me with the rto_start_trylock() and rto_start_unlock() helper
functions.
Signed-off-by: Steven Rostedt (VMware) <rostedt@goodmis.org>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Clark Williams <williams@redhat.com>
Cc: Daniel Bristot de Oliveira <bristot@redhat.com>
Cc: John Kacur <jkacur@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Scott Wood <swood@redhat.com>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/20170424114732.1aac6dc4@gandalf.local.home
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-10-06 18:05:04 +00:00
|
|
|
#ifdef HAVE_RT_PUSH_IPI
|
|
|
|
|
rd->rto_cpu = -1;
|
|
|
|
|
raw_spin_lock_init(&rd->rto_lock);
|
|
|
|
|
init_irq_work(&rd->rto_push_work, rto_push_irq_work_func);
|
|
|
|
|
#endif
|
|
|
|
|
|
2020-10-08 15:48:46 +00:00
|
|
|
rd->visit_gen = 0;
|
2017-02-01 12:10:18 +00:00
|
|
|
init_dl_bw(&rd->dl_bw);
|
|
|
|
|
if (cpudl_init(&rd->cpudl) != 0)
|
|
|
|
|
goto free_rto_mask;
|
|
|
|
|
|
|
|
|
|
if (cpupri_init(&rd->cpupri) != 0)
|
|
|
|
|
goto free_cpudl;
|
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
|
|
free_cpudl:
|
|
|
|
|
cpudl_cleanup(&rd->cpudl);
|
|
|
|
|
free_rto_mask:
|
|
|
|
|
free_cpumask_var(rd->rto_mask);
|
|
|
|
|
free_dlo_mask:
|
|
|
|
|
free_cpumask_var(rd->dlo_mask);
|
|
|
|
|
free_online:
|
|
|
|
|
free_cpumask_var(rd->online);
|
|
|
|
|
free_span:
|
|
|
|
|
free_cpumask_var(rd->span);
|
|
|
|
|
out:
|
|
|
|
|
return -ENOMEM;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* By default the system creates a single root-domain with all CPUs as
|
|
|
|
|
* members (mimicking the global state we have today).
|
|
|
|
|
*/
|
|
|
|
|
struct root_domain def_root_domain;
|
|
|
|
|
|
|
|
|
|
void init_defrootdomain(void)
|
|
|
|
|
{
|
|
|
|
|
init_rootdomain(&def_root_domain);
|
|
|
|
|
|
|
|
|
|
atomic_set(&def_root_domain.refcount, 1);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static struct root_domain *alloc_rootdomain(void)
|
|
|
|
|
{
|
|
|
|
|
struct root_domain *rd;
|
|
|
|
|
|
2017-04-13 09:15:48 +00:00
|
|
|
rd = kzalloc(sizeof(*rd), GFP_KERNEL);
|
2017-02-01 12:10:18 +00:00
|
|
|
if (!rd)
|
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
|
|
if (init_rootdomain(rd) != 0) {
|
|
|
|
|
kfree(rd);
|
|
|
|
|
return NULL;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return rd;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void free_sched_groups(struct sched_group *sg, int free_sgc)
|
|
|
|
|
{
|
|
|
|
|
struct sched_group *tmp, *first;
|
|
|
|
|
|
|
|
|
|
if (!sg)
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
first = sg;
|
|
|
|
|
do {
|
|
|
|
|
tmp = sg->next;
|
|
|
|
|
|
|
|
|
|
if (free_sgc && atomic_dec_and_test(&sg->sgc->ref))
|
|
|
|
|
kfree(sg->sgc);
|
|
|
|
|
|
2017-08-10 07:52:16 +00:00
|
|
|
if (atomic_dec_and_test(&sg->ref))
|
|
|
|
|
kfree(sg);
|
2017-02-01 12:10:18 +00:00
|
|
|
sg = tmp;
|
|
|
|
|
} while (sg != first);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void destroy_sched_domain(struct sched_domain *sd)
|
|
|
|
|
{
|
|
|
|
|
/*
|
2017-08-21 13:42:52 +00:00
|
|
|
* A normal sched domain may have multiple group references, an
|
|
|
|
|
* overlapping domain, having private groups, only one. Iterate,
|
|
|
|
|
* dropping group/capacity references, freeing where none remain.
|
2017-02-01 12:10:18 +00:00
|
|
|
*/
|
2017-08-10 07:52:16 +00:00
|
|
|
free_sched_groups(sd->groups, 1);
|
|
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
if (sd->shared && atomic_dec_and_test(&sd->shared->ref))
|
|
|
|
|
kfree(sd->shared);
|
|
|
|
|
kfree(sd);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void destroy_sched_domains_rcu(struct rcu_head *rcu)
|
|
|
|
|
{
|
|
|
|
|
struct sched_domain *sd = container_of(rcu, struct sched_domain, rcu);
|
|
|
|
|
|
|
|
|
|
while (sd) {
|
|
|
|
|
struct sched_domain *parent = sd->parent;
|
|
|
|
|
destroy_sched_domain(sd);
|
|
|
|
|
sd = parent;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void destroy_sched_domains(struct sched_domain *sd)
|
|
|
|
|
{
|
|
|
|
|
if (sd)
|
|
|
|
|
call_rcu(&sd->rcu, destroy_sched_domains_rcu);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Keep a special pointer to the highest sched_domain that has
|
|
|
|
|
* SD_SHARE_PKG_RESOURCE set (Last Level Cache Domain) for this
|
|
|
|
|
* allows us to avoid some pointer chasing select_idle_sibling().
|
|
|
|
|
*
|
|
|
|
|
* Also keep a unique ID per domain (we use the first CPU number in
|
|
|
|
|
* the cpumask of the domain), this allows us to quickly tell if
|
|
|
|
|
* two CPUs are in the same cache domain, see cpus_share_cache().
|
|
|
|
|
*/
|
2019-03-21 00:34:24 +00:00
|
|
|
DEFINE_PER_CPU(struct sched_domain __rcu *, sd_llc);
|
2017-02-01 12:10:18 +00:00
|
|
|
DEFINE_PER_CPU(int, sd_llc_size);
|
|
|
|
|
DEFINE_PER_CPU(int, sd_llc_id);
|
2019-03-21 00:34:24 +00:00
|
|
|
DEFINE_PER_CPU(struct sched_domain_shared __rcu *, sd_llc_shared);
|
|
|
|
|
DEFINE_PER_CPU(struct sched_domain __rcu *, sd_numa);
|
|
|
|
|
DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_packing);
|
|
|
|
|
DEFINE_PER_CPU(struct sched_domain __rcu *, sd_asym_cpucapacity);
|
2018-07-04 10:17:39 +00:00
|
|
|
DEFINE_STATIC_KEY_FALSE(sched_asym_cpucapacity);
|
2017-02-01 12:10:18 +00:00
|
|
|
|
|
|
|
|
static void update_top_cache_domain(int cpu)
|
|
|
|
|
{
|
|
|
|
|
struct sched_domain_shared *sds = NULL;
|
|
|
|
|
struct sched_domain *sd;
|
|
|
|
|
int id = cpu;
|
|
|
|
|
int size = 1;
|
|
|
|
|
|
|
|
|
|
sd = highest_flag_domain(cpu, SD_SHARE_PKG_RESOURCES);
|
|
|
|
|
if (sd) {
|
|
|
|
|
id = cpumask_first(sched_domain_span(sd));
|
|
|
|
|
size = cpumask_weight(sched_domain_span(sd));
|
|
|
|
|
sds = sd->shared;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
rcu_assign_pointer(per_cpu(sd_llc, cpu), sd);
|
|
|
|
|
per_cpu(sd_llc_size, cpu) = size;
|
|
|
|
|
per_cpu(sd_llc_id, cpu) = id;
|
|
|
|
|
rcu_assign_pointer(per_cpu(sd_llc_shared, cpu), sds);
|
|
|
|
|
|
|
|
|
|
sd = lowest_flag_domain(cpu, SD_NUMA);
|
|
|
|
|
rcu_assign_pointer(per_cpu(sd_numa, cpu), sd);
|
|
|
|
|
|
|
|
|
|
sd = highest_flag_domain(cpu, SD_ASYM_PACKING);
|
2018-12-03 09:56:19 +00:00
|
|
|
rcu_assign_pointer(per_cpu(sd_asym_packing, cpu), sd);
|
|
|
|
|
|
|
|
|
|
sd = lowest_flag_domain(cpu, SD_ASYM_CPUCAPACITY);
|
|
|
|
|
rcu_assign_pointer(per_cpu(sd_asym_cpucapacity, cpu), sd);
|
2017-02-01 12:10:18 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Attach the domain 'sd' to 'cpu' as its base domain. Callers must
|
|
|
|
|
* hold the hotplug lock.
|
|
|
|
|
*/
|
|
|
|
|
static void
|
|
|
|
|
cpu_attach_domain(struct sched_domain *sd, struct root_domain *rd, int cpu)
|
|
|
|
|
{
|
|
|
|
|
struct rq *rq = cpu_rq(cpu);
|
|
|
|
|
struct sched_domain *tmp;
|
sched/topology: Warn when NUMA diameter > 2
NUMA topologies where the shortest path between some two nodes requires
three or more hops (i.e. diameter > 2) end up being misrepresented in the
scheduler topology structures.
This is currently detected when booting a kernel with CONFIG_SCHED_DEBUG=y
+ sched_debug on the cmdline, although this will only yield a warning about
sched_group spans not matching sched_domain spans:
ERROR: groups don't span domain->span
Add an explicit warning for that case, triggered regardless of
CONFIG_SCHED_DEBUG, and decorate it with an appropriate comment.
The topology described in the comment can be booted up on QEMU by appending
the following to your usual QEMU incantation:
-smp cores=4 \
-numa node,cpus=0,nodeid=0 -numa node,cpus=1,nodeid=1, \
-numa node,cpus=2,nodeid=2, -numa node,cpus=3,nodeid=3, \
-numa dist,src=0,dst=1,val=20, -numa dist,src=0,dst=2,val=30, \
-numa dist,src=0,dst=3,val=40, -numa dist,src=1,dst=2,val=20, \
-numa dist,src=1,dst=3,val=30, -numa dist,src=2,dst=3,val=20
A somewhat more realistic topology (6-node mesh) with the same affliction
can be conjured with:
-smp cores=6 \
-numa node,cpus=0,nodeid=0 -numa node,cpus=1,nodeid=1, \
-numa node,cpus=2,nodeid=2, -numa node,cpus=3,nodeid=3, \
-numa node,cpus=4,nodeid=4, -numa node,cpus=5,nodeid=5, \
-numa dist,src=0,dst=1,val=20, -numa dist,src=0,dst=2,val=30, \
-numa dist,src=0,dst=3,val=40, -numa dist,src=0,dst=4,val=30, \
-numa dist,src=0,dst=5,val=20, \
-numa dist,src=1,dst=2,val=20, -numa dist,src=1,dst=3,val=30, \
-numa dist,src=1,dst=4,val=20, -numa dist,src=1,dst=5,val=30, \
-numa dist,src=2,dst=3,val=20, -numa dist,src=2,dst=4,val=30, \
-numa dist,src=2,dst=5,val=40, \
-numa dist,src=3,dst=4,val=20, -numa dist,src=3,dst=5,val=30, \
-numa dist,src=4,dst=5,val=20
Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Mel Gorman <mgorman@techsingularity.net>
Link: https://lore.kernel.org/lkml/jhjtux5edo2.mognet@arm.com
2020-11-10 18:43:00 +00:00
|
|
|
int numa_distance = 0;
|
2017-02-01 12:10:18 +00:00
|
|
|
|
|
|
|
|
/* Remove the sched domains which do not contribute to scheduling. */
|
|
|
|
|
for (tmp = sd; tmp; ) {
|
|
|
|
|
struct sched_domain *parent = tmp->parent;
|
|
|
|
|
if (!parent)
|
|
|
|
|
break;
|
|
|
|
|
|
|
|
|
|
if (sd_parent_degenerate(tmp, parent)) {
|
|
|
|
|
tmp->parent = parent->parent;
|
|
|
|
|
if (parent->parent)
|
|
|
|
|
parent->parent->child = tmp;
|
|
|
|
|
/*
|
|
|
|
|
* Transfer SD_PREFER_SIBLING down in case of a
|
|
|
|
|
* degenerate parent; the spans match for this
|
|
|
|
|
* so the property transfers.
|
|
|
|
|
*/
|
|
|
|
|
if (parent->flags & SD_PREFER_SIBLING)
|
|
|
|
|
tmp->flags |= SD_PREFER_SIBLING;
|
|
|
|
|
destroy_sched_domain(parent);
|
|
|
|
|
} else
|
|
|
|
|
tmp = tmp->parent;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (sd && sd_degenerate(sd)) {
|
|
|
|
|
tmp = sd;
|
|
|
|
|
sd = sd->parent;
|
|
|
|
|
destroy_sched_domain(tmp);
|
|
|
|
|
if (sd)
|
|
|
|
|
sd->child = NULL;
|
|
|
|
|
}
|
|
|
|
|
|
sched/topology: Warn when NUMA diameter > 2
NUMA topologies where the shortest path between some two nodes requires
three or more hops (i.e. diameter > 2) end up being misrepresented in the
scheduler topology structures.
This is currently detected when booting a kernel with CONFIG_SCHED_DEBUG=y
+ sched_debug on the cmdline, although this will only yield a warning about
sched_group spans not matching sched_domain spans:
ERROR: groups don't span domain->span
Add an explicit warning for that case, triggered regardless of
CONFIG_SCHED_DEBUG, and decorate it with an appropriate comment.
The topology described in the comment can be booted up on QEMU by appending
the following to your usual QEMU incantation:
-smp cores=4 \
-numa node,cpus=0,nodeid=0 -numa node,cpus=1,nodeid=1, \
-numa node,cpus=2,nodeid=2, -numa node,cpus=3,nodeid=3, \
-numa dist,src=0,dst=1,val=20, -numa dist,src=0,dst=2,val=30, \
-numa dist,src=0,dst=3,val=40, -numa dist,src=1,dst=2,val=20, \
-numa dist,src=1,dst=3,val=30, -numa dist,src=2,dst=3,val=20
A somewhat more realistic topology (6-node mesh) with the same affliction
can be conjured with:
-smp cores=6 \
-numa node,cpus=0,nodeid=0 -numa node,cpus=1,nodeid=1, \
-numa node,cpus=2,nodeid=2, -numa node,cpus=3,nodeid=3, \
-numa node,cpus=4,nodeid=4, -numa node,cpus=5,nodeid=5, \
-numa dist,src=0,dst=1,val=20, -numa dist,src=0,dst=2,val=30, \
-numa dist,src=0,dst=3,val=40, -numa dist,src=0,dst=4,val=30, \
-numa dist,src=0,dst=5,val=20, \
-numa dist,src=1,dst=2,val=20, -numa dist,src=1,dst=3,val=30, \
-numa dist,src=1,dst=4,val=20, -numa dist,src=1,dst=5,val=30, \
-numa dist,src=2,dst=3,val=20, -numa dist,src=2,dst=4,val=30, \
-numa dist,src=2,dst=5,val=40, \
-numa dist,src=3,dst=4,val=20, -numa dist,src=3,dst=5,val=30, \
-numa dist,src=4,dst=5,val=20
Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Acked-by: Mel Gorman <mgorman@techsingularity.net>
Link: https://lore.kernel.org/lkml/jhjtux5edo2.mognet@arm.com
2020-11-10 18:43:00 +00:00
|
|
|
for (tmp = sd; tmp; tmp = tmp->parent)
|
|
|
|
|
numa_distance += !!(tmp->flags & SD_NUMA);
|
|
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
sched_domain_debug(sd, cpu);
|
|
|
|
|
|
|
|
|
|
rq_attach_root(rq, rd);
|
|
|
|
|
tmp = rq->sd;
|
|
|
|
|
rcu_assign_pointer(rq->sd, sd);
|
2017-08-10 15:10:26 +00:00
|
|
|
dirty_sched_domain_sysctl(cpu);
|
2017-02-01 12:10:18 +00:00
|
|
|
destroy_sched_domains(tmp);
|
|
|
|
|
|
|
|
|
|
update_top_cache_domain(cpu);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
struct s_data {
|
2019-01-18 14:49:36 +00:00
|
|
|
struct sched_domain * __percpu *sd;
|
2017-02-01 12:10:18 +00:00
|
|
|
struct root_domain *rd;
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
enum s_alloc {
|
|
|
|
|
sa_rootdomain,
|
|
|
|
|
sa_sd,
|
|
|
|
|
sa_sd_storage,
|
|
|
|
|
sa_none,
|
|
|
|
|
};
|
|
|
|
|
|
2017-04-28 08:54:26 +00:00
|
|
|
/*
|
|
|
|
|
* Return the canonical balance CPU for this group, this is the first CPU
|
2017-05-01 08:47:02 +00:00
|
|
|
* of this group that's also in the balance mask.
|
2017-04-28 08:54:26 +00:00
|
|
|
*
|
2017-05-01 08:47:02 +00:00
|
|
|
* The balance mask are all those CPUs that could actually end up at this
|
|
|
|
|
* group. See build_balance_mask().
|
2017-04-28 08:54:26 +00:00
|
|
|
*
|
|
|
|
|
* Also see should_we_balance().
|
|
|
|
|
*/
|
|
|
|
|
int group_balance_cpu(struct sched_group *sg)
|
|
|
|
|
{
|
2017-05-01 08:47:02 +00:00
|
|
|
return cpumask_first(group_balance_mask(sg));
|
2017-04-28 08:54:26 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* NUMA topology (first read the regular topology blurb below)
|
|
|
|
|
*
|
|
|
|
|
* Given a node-distance table, for example:
|
|
|
|
|
*
|
|
|
|
|
* node 0 1 2 3
|
|
|
|
|
* 0: 10 20 30 20
|
|
|
|
|
* 1: 20 10 20 30
|
|
|
|
|
* 2: 30 20 10 20
|
|
|
|
|
* 3: 20 30 20 10
|
|
|
|
|
*
|
|
|
|
|
* which represents a 4 node ring topology like:
|
|
|
|
|
*
|
|
|
|
|
* 0 ----- 1
|
|
|
|
|
* | |
|
|
|
|
|
* | |
|
|
|
|
|
* | |
|
|
|
|
|
* 3 ----- 2
|
|
|
|
|
*
|
|
|
|
|
* We want to construct domains and groups to represent this. The way we go
|
|
|
|
|
* about doing this is to build the domains on 'hops'. For each NUMA level we
|
|
|
|
|
* construct the mask of all nodes reachable in @level hops.
|
|
|
|
|
*
|
|
|
|
|
* For the above NUMA topology that gives 3 levels:
|
|
|
|
|
*
|
|
|
|
|
* NUMA-2 0-3 0-3 0-3 0-3
|
|
|
|
|
* groups: {0-1,3},{1-3} {0-2},{0,2-3} {1-3},{0-1,3} {0,2-3},{0-2}
|
|
|
|
|
*
|
|
|
|
|
* NUMA-1 0-1,3 0-2 1-3 0,2-3
|
|
|
|
|
* groups: {0},{1},{3} {0},{1},{2} {1},{2},{3} {0},{2},{3}
|
|
|
|
|
*
|
|
|
|
|
* NUMA-0 0 1 2 3
|
|
|
|
|
*
|
|
|
|
|
*
|
|
|
|
|
* As can be seen; things don't nicely line up as with the regular topology.
|
|
|
|
|
* When we iterate a domain in child domain chunks some nodes can be
|
|
|
|
|
* represented multiple times -- hence the "overlap" naming for this part of
|
|
|
|
|
* the topology.
|
|
|
|
|
*
|
|
|
|
|
* In order to minimize this overlap, we only build enough groups to cover the
|
|
|
|
|
* domain. For instance Node-0 NUMA-2 would only get groups: 0-1,3 and 1-3.
|
|
|
|
|
*
|
|
|
|
|
* Because:
|
|
|
|
|
*
|
|
|
|
|
* - the first group of each domain is its child domain; this
|
|
|
|
|
* gets us the first 0-1,3
|
|
|
|
|
* - the only uncovered node is 2, who's child domain is 1-3.
|
|
|
|
|
*
|
|
|
|
|
* However, because of the overlap, computing a unique CPU for each group is
|
|
|
|
|
* more complicated. Consider for instance the groups of NODE-1 NUMA-2, both
|
|
|
|
|
* groups include the CPUs of Node-0, while those CPUs would not in fact ever
|
|
|
|
|
* end up at those groups (they would end up in group: 0-1,3).
|
|
|
|
|
*
|
2017-05-01 08:47:02 +00:00
|
|
|
* To correct this we have to introduce the group balance mask. This mask
|
2017-04-28 08:54:26 +00:00
|
|
|
* will contain those CPUs in the group that can reach this group given the
|
|
|
|
|
* (child) domain tree.
|
|
|
|
|
*
|
|
|
|
|
* With this we can once again compute balance_cpu and sched_group_capacity
|
|
|
|
|
* relations.
|
|
|
|
|
*
|
|
|
|
|
* XXX include words on how balance_cpu is unique and therefore can be
|
|
|
|
|
* used for sched_group_capacity links.
|
|
|
|
|
*
|
|
|
|
|
*
|
|
|
|
|
* Another 'interesting' topology is:
|
|
|
|
|
*
|
|
|
|
|
* node 0 1 2 3
|
|
|
|
|
* 0: 10 20 20 30
|
|
|
|
|
* 1: 20 10 20 20
|
|
|
|
|
* 2: 20 20 10 20
|
|
|
|
|
* 3: 30 20 20 10
|
|
|
|
|
*
|
|
|
|
|
* Which looks a little like:
|
|
|
|
|
*
|
|
|
|
|
* 0 ----- 1
|
|
|
|
|
* | / |
|
|
|
|
|
* | / |
|
|
|
|
|
* | / |
|
|
|
|
|
* 2 ----- 3
|
|
|
|
|
*
|
|
|
|
|
* This topology is asymmetric, nodes 1,2 are fully connected, but nodes 0,3
|
|
|
|
|
* are not.
|
|
|
|
|
*
|
|
|
|
|
* This leads to a few particularly weird cases where the sched_domain's are
|
sched: Clean up and harmonize the coding style of the scheduler code base
A good number of small style inconsistencies have accumulated
in the scheduler core, so do a pass over them to harmonize
all these details:
- fix speling in comments,
- use curly braces for multi-line statements,
- remove unnecessary parentheses from integer literals,
- capitalize consistently,
- remove stray newlines,
- add comments where necessary,
- remove invalid/unnecessary comments,
- align structure definitions and other data types vertically,
- add missing newlines for increased readability,
- fix vertical tabulation where it's misaligned,
- harmonize preprocessor conditional block labeling
and vertical alignment,
- remove line-breaks where they uglify the code,
- add newline after local variable definitions,
No change in functionality:
md5:
1191fa0a890cfa8132156d2959d7e9e2 built-in.o.before.asm
1191fa0a890cfa8132156d2959d7e9e2 built-in.o.after.asm
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-03-03 13:01:12 +00:00
|
|
|
* not of the same number for each CPU. Consider:
|
2017-04-28 08:54:26 +00:00
|
|
|
*
|
|
|
|
|
* NUMA-2 0-3 0-3
|
|
|
|
|
* groups: {0-2},{1-3} {1-3},{0-2}
|
|
|
|
|
*
|
|
|
|
|
* NUMA-1 0-2 0-3 0-3 1-3
|
|
|
|
|
*
|
|
|
|
|
* NUMA-0 0 1 2 3
|
|
|
|
|
*
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
/*
|
2017-05-01 08:47:02 +00:00
|
|
|
* Build the balance mask; it contains only those CPUs that can arrive at this
|
|
|
|
|
* group and should be considered to continue balancing.
|
2017-04-28 08:54:26 +00:00
|
|
|
*
|
|
|
|
|
* We do this during the group creation pass, therefore the group information
|
|
|
|
|
* isn't complete yet, however since each group represents a (child) domain we
|
|
|
|
|
* can fully construct this using the sched_domain bits (which are already
|
|
|
|
|
* complete).
|
2017-02-01 12:10:18 +00:00
|
|
|
*/
|
sched/topology: Fix overlapping sched_group_capacity
When building the overlapping groups we need to attach a consistent
sched_group_capacity structure. That is, all 'identical' sched_group's
should have the _same_ sched_group_capacity.
This can (once again) be demonstrated with a topology like:
node 0 1 2 3
0: 10 20 30 20
1: 20 10 20 30
2: 30 20 10 20
3: 20 30 20 10
But we need at least 2 CPUs per node for this to show up, after all,
if there is only one CPU per node, our CPU @i is per definition a
unique CPU that reaches this domain (aka balance-cpu).
Given the above NUMA topo and 2 CPUs per node:
[] CPU0 attaching sched-domain(s):
[] domain-0: span=0,4 level=DIE
[] groups: 0:{ span=0 }, 4:{ span=4 }
[] domain-1: span=0-1,3-5,7 level=NUMA
[] groups: 0:{ span=0,4 mask=0,4 cap=2048 }, 1:{ span=1,5 mask=1,5 cap=2048 }, 3:{ span=3,7 mask=3,7 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 0:{ span=0-1,3-5,7 mask=0,4 cap=6144 }, 2:{ span=1-3,5-7 mask=2,6 cap=6144 }
[] CPU1 attaching sched-domain(s):
[] domain-0: span=1,5 level=DIE
[] groups: 1:{ span=1 }, 5:{ span=5 }
[] domain-1: span=0-2,4-6 level=NUMA
[] groups: 1:{ span=1,5 mask=1,5 cap=2048 }, 2:{ span=2,6 mask=2,6 cap=2048 }, 4:{ span=0,4 mask=0,4 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 1:{ span=0-2,4-6 mask=1,5 cap=6144 }, 3:{ span=0,2-4,6-7 mask=3,7 cap=6144 }
Observe how CPU0-domain1-group0 and CPU1-domain1-group4 are the
'same' but have a different id (0 vs 4).
To fix this, use the group balance CPU to select the SGC. This means
we have to compute the full mask for each CPU and require a second
temporary mask to store the group mask in (it otherwise lives in the
SGC).
The fixed topology looks like:
[] CPU0 attaching sched-domain(s):
[] domain-0: span=0,4 level=DIE
[] groups: 0:{ span=0 }, 4:{ span=4 }
[] domain-1: span=0-1,3-5,7 level=NUMA
[] groups: 0:{ span=0,4 mask=0,4 cap=2048 }, 1:{ span=1,5 mask=1,5 cap=2048 }, 3:{ span=3,7 mask=3,7 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 0:{ span=0-1,3-5,7 mask=0,4 cap=6144 }, 2:{ span=1-3,5-7 mask=2,6 cap=6144 }
[] CPU1 attaching sched-domain(s):
[] domain-0: span=1,5 level=DIE
[] groups: 1:{ span=1 }, 5:{ span=5 }
[] domain-1: span=0-2,4-6 level=NUMA
[] groups: 1:{ span=1,5 mask=1,5 cap=2048 }, 2:{ span=2,6 mask=2,6 cap=2048 }, 0:{ span=0,4 mask=0,4 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 1:{ span=0-2,4-6 mask=1,5 cap=6144 }, 3:{ span=0,2-4,6-7 mask=3,7 cap=6144 }
Debugged-by: Lauro Ramos Venancio <lvenanci@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Fixes: e3589f6c81e4 ("sched: Allow for overlapping sched_domain spans")
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-04-25 12:31:11 +00:00
|
|
|
static void
|
2017-05-01 08:47:02 +00:00
|
|
|
build_balance_mask(struct sched_domain *sd, struct sched_group *sg, struct cpumask *mask)
|
2017-02-01 12:10:18 +00:00
|
|
|
{
|
2017-05-01 09:03:12 +00:00
|
|
|
const struct cpumask *sg_span = sched_group_span(sg);
|
2017-02-01 12:10:18 +00:00
|
|
|
struct sd_data *sdd = sd->private;
|
|
|
|
|
struct sched_domain *sibling;
|
|
|
|
|
int i;
|
|
|
|
|
|
sched/topology: Fix overlapping sched_group_capacity
When building the overlapping groups we need to attach a consistent
sched_group_capacity structure. That is, all 'identical' sched_group's
should have the _same_ sched_group_capacity.
This can (once again) be demonstrated with a topology like:
node 0 1 2 3
0: 10 20 30 20
1: 20 10 20 30
2: 30 20 10 20
3: 20 30 20 10
But we need at least 2 CPUs per node for this to show up, after all,
if there is only one CPU per node, our CPU @i is per definition a
unique CPU that reaches this domain (aka balance-cpu).
Given the above NUMA topo and 2 CPUs per node:
[] CPU0 attaching sched-domain(s):
[] domain-0: span=0,4 level=DIE
[] groups: 0:{ span=0 }, 4:{ span=4 }
[] domain-1: span=0-1,3-5,7 level=NUMA
[] groups: 0:{ span=0,4 mask=0,4 cap=2048 }, 1:{ span=1,5 mask=1,5 cap=2048 }, 3:{ span=3,7 mask=3,7 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 0:{ span=0-1,3-5,7 mask=0,4 cap=6144 }, 2:{ span=1-3,5-7 mask=2,6 cap=6144 }
[] CPU1 attaching sched-domain(s):
[] domain-0: span=1,5 level=DIE
[] groups: 1:{ span=1 }, 5:{ span=5 }
[] domain-1: span=0-2,4-6 level=NUMA
[] groups: 1:{ span=1,5 mask=1,5 cap=2048 }, 2:{ span=2,6 mask=2,6 cap=2048 }, 4:{ span=0,4 mask=0,4 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 1:{ span=0-2,4-6 mask=1,5 cap=6144 }, 3:{ span=0,2-4,6-7 mask=3,7 cap=6144 }
Observe how CPU0-domain1-group0 and CPU1-domain1-group4 are the
'same' but have a different id (0 vs 4).
To fix this, use the group balance CPU to select the SGC. This means
we have to compute the full mask for each CPU and require a second
temporary mask to store the group mask in (it otherwise lives in the
SGC).
The fixed topology looks like:
[] CPU0 attaching sched-domain(s):
[] domain-0: span=0,4 level=DIE
[] groups: 0:{ span=0 }, 4:{ span=4 }
[] domain-1: span=0-1,3-5,7 level=NUMA
[] groups: 0:{ span=0,4 mask=0,4 cap=2048 }, 1:{ span=1,5 mask=1,5 cap=2048 }, 3:{ span=3,7 mask=3,7 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 0:{ span=0-1,3-5,7 mask=0,4 cap=6144 }, 2:{ span=1-3,5-7 mask=2,6 cap=6144 }
[] CPU1 attaching sched-domain(s):
[] domain-0: span=1,5 level=DIE
[] groups: 1:{ span=1 }, 5:{ span=5 }
[] domain-1: span=0-2,4-6 level=NUMA
[] groups: 1:{ span=1,5 mask=1,5 cap=2048 }, 2:{ span=2,6 mask=2,6 cap=2048 }, 0:{ span=0,4 mask=0,4 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 1:{ span=0-2,4-6 mask=1,5 cap=6144 }, 3:{ span=0,2-4,6-7 mask=3,7 cap=6144 }
Debugged-by: Lauro Ramos Venancio <lvenanci@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Fixes: e3589f6c81e4 ("sched: Allow for overlapping sched_domain spans")
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-04-25 12:31:11 +00:00
|
|
|
cpumask_clear(mask);
|
|
|
|
|
|
2017-04-20 19:51:40 +00:00
|
|
|
for_each_cpu(i, sg_span) {
|
2017-02-01 12:10:18 +00:00
|
|
|
sibling = *per_cpu_ptr(sdd->sd, i);
|
sched/topology: Fix overlapping sched_group_mask
The point of sched_group_mask is to select those CPUs from
sched_group_cpus that can actually arrive at this balance domain.
The current code gets it wrong, as can be readily demonstrated with a
topology like:
node 0 1 2 3
0: 10 20 30 20
1: 20 10 20 30
2: 30 20 10 20
3: 20 30 20 10
Where (for example) domain 1 on CPU1 ends up with a mask that includes
CPU0:
[] CPU1 attaching sched-domain:
[] domain 0: span 0-2 level NUMA
[] groups: 1 (mask: 1), 2, 0
[] domain 1: span 0-3 level NUMA
[] groups: 0-2 (mask: 0-2) (cpu_capacity: 3072), 0,2-3 (cpu_capacity: 3072)
This causes sched_balance_cpu() to compute the wrong CPU and
consequently should_we_balance() will terminate early resulting in
missed load-balance opportunities.
The fixed topology looks like:
[] CPU1 attaching sched-domain:
[] domain 0: span 0-2 level NUMA
[] groups: 1 (mask: 1), 2, 0
[] domain 1: span 0-3 level NUMA
[] groups: 0-2 (mask: 1) (cpu_capacity: 3072), 0,2-3 (cpu_capacity: 3072)
(note: this relies on OVERLAP domains to always have children, this is
true because the regular topology domains are still here -- this is
before degenerate trimming)
Debugged-by: Lauro Ramos Venancio <lvenanci@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Cc: stable@vger.kernel.org
Fixes: e3589f6c81e4 ("sched: Allow for overlapping sched_domain spans")
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-04-25 12:00:49 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Can happen in the asymmetric case, where these siblings are
|
|
|
|
|
* unused. The mask will not be empty because those CPUs that
|
|
|
|
|
* do have the top domain _should_ span the domain.
|
|
|
|
|
*/
|
|
|
|
|
if (!sibling->child)
|
|
|
|
|
continue;
|
|
|
|
|
|
|
|
|
|
/* If we would not end up here, we can't continue from here */
|
|
|
|
|
if (!cpumask_equal(sg_span, sched_domain_span(sibling->child)))
|
2017-02-01 12:10:18 +00:00
|
|
|
continue;
|
|
|
|
|
|
sched/topology: Fix overlapping sched_group_capacity
When building the overlapping groups we need to attach a consistent
sched_group_capacity structure. That is, all 'identical' sched_group's
should have the _same_ sched_group_capacity.
This can (once again) be demonstrated with a topology like:
node 0 1 2 3
0: 10 20 30 20
1: 20 10 20 30
2: 30 20 10 20
3: 20 30 20 10
But we need at least 2 CPUs per node for this to show up, after all,
if there is only one CPU per node, our CPU @i is per definition a
unique CPU that reaches this domain (aka balance-cpu).
Given the above NUMA topo and 2 CPUs per node:
[] CPU0 attaching sched-domain(s):
[] domain-0: span=0,4 level=DIE
[] groups: 0:{ span=0 }, 4:{ span=4 }
[] domain-1: span=0-1,3-5,7 level=NUMA
[] groups: 0:{ span=0,4 mask=0,4 cap=2048 }, 1:{ span=1,5 mask=1,5 cap=2048 }, 3:{ span=3,7 mask=3,7 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 0:{ span=0-1,3-5,7 mask=0,4 cap=6144 }, 2:{ span=1-3,5-7 mask=2,6 cap=6144 }
[] CPU1 attaching sched-domain(s):
[] domain-0: span=1,5 level=DIE
[] groups: 1:{ span=1 }, 5:{ span=5 }
[] domain-1: span=0-2,4-6 level=NUMA
[] groups: 1:{ span=1,5 mask=1,5 cap=2048 }, 2:{ span=2,6 mask=2,6 cap=2048 }, 4:{ span=0,4 mask=0,4 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 1:{ span=0-2,4-6 mask=1,5 cap=6144 }, 3:{ span=0,2-4,6-7 mask=3,7 cap=6144 }
Observe how CPU0-domain1-group0 and CPU1-domain1-group4 are the
'same' but have a different id (0 vs 4).
To fix this, use the group balance CPU to select the SGC. This means
we have to compute the full mask for each CPU and require a second
temporary mask to store the group mask in (it otherwise lives in the
SGC).
The fixed topology looks like:
[] CPU0 attaching sched-domain(s):
[] domain-0: span=0,4 level=DIE
[] groups: 0:{ span=0 }, 4:{ span=4 }
[] domain-1: span=0-1,3-5,7 level=NUMA
[] groups: 0:{ span=0,4 mask=0,4 cap=2048 }, 1:{ span=1,5 mask=1,5 cap=2048 }, 3:{ span=3,7 mask=3,7 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 0:{ span=0-1,3-5,7 mask=0,4 cap=6144 }, 2:{ span=1-3,5-7 mask=2,6 cap=6144 }
[] CPU1 attaching sched-domain(s):
[] domain-0: span=1,5 level=DIE
[] groups: 1:{ span=1 }, 5:{ span=5 }
[] domain-1: span=0-2,4-6 level=NUMA
[] groups: 1:{ span=1,5 mask=1,5 cap=2048 }, 2:{ span=2,6 mask=2,6 cap=2048 }, 0:{ span=0,4 mask=0,4 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 1:{ span=0-2,4-6 mask=1,5 cap=6144 }, 3:{ span=0,2-4,6-7 mask=3,7 cap=6144 }
Debugged-by: Lauro Ramos Venancio <lvenanci@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Fixes: e3589f6c81e4 ("sched: Allow for overlapping sched_domain spans")
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-04-25 12:31:11 +00:00
|
|
|
cpumask_set_cpu(i, mask);
|
2017-02-01 12:10:18 +00:00
|
|
|
}
|
sched/topology: Fix overlapping sched_group_mask
The point of sched_group_mask is to select those CPUs from
sched_group_cpus that can actually arrive at this balance domain.
The current code gets it wrong, as can be readily demonstrated with a
topology like:
node 0 1 2 3
0: 10 20 30 20
1: 20 10 20 30
2: 30 20 10 20
3: 20 30 20 10
Where (for example) domain 1 on CPU1 ends up with a mask that includes
CPU0:
[] CPU1 attaching sched-domain:
[] domain 0: span 0-2 level NUMA
[] groups: 1 (mask: 1), 2, 0
[] domain 1: span 0-3 level NUMA
[] groups: 0-2 (mask: 0-2) (cpu_capacity: 3072), 0,2-3 (cpu_capacity: 3072)
This causes sched_balance_cpu() to compute the wrong CPU and
consequently should_we_balance() will terminate early resulting in
missed load-balance opportunities.
The fixed topology looks like:
[] CPU1 attaching sched-domain:
[] domain 0: span 0-2 level NUMA
[] groups: 1 (mask: 1), 2, 0
[] domain 1: span 0-3 level NUMA
[] groups: 0-2 (mask: 1) (cpu_capacity: 3072), 0,2-3 (cpu_capacity: 3072)
(note: this relies on OVERLAP domains to always have children, this is
true because the regular topology domains are still here -- this is
before degenerate trimming)
Debugged-by: Lauro Ramos Venancio <lvenanci@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Cc: stable@vger.kernel.org
Fixes: e3589f6c81e4 ("sched: Allow for overlapping sched_domain spans")
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-04-25 12:00:49 +00:00
|
|
|
|
|
|
|
|
/* We must not have empty masks here */
|
sched/topology: Fix overlapping sched_group_capacity
When building the overlapping groups we need to attach a consistent
sched_group_capacity structure. That is, all 'identical' sched_group's
should have the _same_ sched_group_capacity.
This can (once again) be demonstrated with a topology like:
node 0 1 2 3
0: 10 20 30 20
1: 20 10 20 30
2: 30 20 10 20
3: 20 30 20 10
But we need at least 2 CPUs per node for this to show up, after all,
if there is only one CPU per node, our CPU @i is per definition a
unique CPU that reaches this domain (aka balance-cpu).
Given the above NUMA topo and 2 CPUs per node:
[] CPU0 attaching sched-domain(s):
[] domain-0: span=0,4 level=DIE
[] groups: 0:{ span=0 }, 4:{ span=4 }
[] domain-1: span=0-1,3-5,7 level=NUMA
[] groups: 0:{ span=0,4 mask=0,4 cap=2048 }, 1:{ span=1,5 mask=1,5 cap=2048 }, 3:{ span=3,7 mask=3,7 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 0:{ span=0-1,3-5,7 mask=0,4 cap=6144 }, 2:{ span=1-3,5-7 mask=2,6 cap=6144 }
[] CPU1 attaching sched-domain(s):
[] domain-0: span=1,5 level=DIE
[] groups: 1:{ span=1 }, 5:{ span=5 }
[] domain-1: span=0-2,4-6 level=NUMA
[] groups: 1:{ span=1,5 mask=1,5 cap=2048 }, 2:{ span=2,6 mask=2,6 cap=2048 }, 4:{ span=0,4 mask=0,4 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 1:{ span=0-2,4-6 mask=1,5 cap=6144 }, 3:{ span=0,2-4,6-7 mask=3,7 cap=6144 }
Observe how CPU0-domain1-group0 and CPU1-domain1-group4 are the
'same' but have a different id (0 vs 4).
To fix this, use the group balance CPU to select the SGC. This means
we have to compute the full mask for each CPU and require a second
temporary mask to store the group mask in (it otherwise lives in the
SGC).
The fixed topology looks like:
[] CPU0 attaching sched-domain(s):
[] domain-0: span=0,4 level=DIE
[] groups: 0:{ span=0 }, 4:{ span=4 }
[] domain-1: span=0-1,3-5,7 level=NUMA
[] groups: 0:{ span=0,4 mask=0,4 cap=2048 }, 1:{ span=1,5 mask=1,5 cap=2048 }, 3:{ span=3,7 mask=3,7 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 0:{ span=0-1,3-5,7 mask=0,4 cap=6144 }, 2:{ span=1-3,5-7 mask=2,6 cap=6144 }
[] CPU1 attaching sched-domain(s):
[] domain-0: span=1,5 level=DIE
[] groups: 1:{ span=1 }, 5:{ span=5 }
[] domain-1: span=0-2,4-6 level=NUMA
[] groups: 1:{ span=1,5 mask=1,5 cap=2048 }, 2:{ span=2,6 mask=2,6 cap=2048 }, 0:{ span=0,4 mask=0,4 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 1:{ span=0-2,4-6 mask=1,5 cap=6144 }, 3:{ span=0,2-4,6-7 mask=3,7 cap=6144 }
Debugged-by: Lauro Ramos Venancio <lvenanci@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Fixes: e3589f6c81e4 ("sched: Allow for overlapping sched_domain spans")
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-04-25 12:31:11 +00:00
|
|
|
WARN_ON_ONCE(cpumask_empty(mask));
|
2017-02-01 12:10:18 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
2017-04-28 08:54:26 +00:00
|
|
|
* XXX: This creates per-node group entries; since the load-balancer will
|
|
|
|
|
* immediately access remote memory to construct this group's load-balance
|
|
|
|
|
* statistics having the groups node local is of dubious benefit.
|
2017-02-01 12:10:18 +00:00
|
|
|
*/
|
2017-04-13 13:56:07 +00:00
|
|
|
static struct sched_group *
|
|
|
|
|
build_group_from_child_sched_domain(struct sched_domain *sd, int cpu)
|
|
|
|
|
{
|
|
|
|
|
struct sched_group *sg;
|
|
|
|
|
struct cpumask *sg_span;
|
|
|
|
|
|
|
|
|
|
sg = kzalloc_node(sizeof(struct sched_group) + cpumask_size(),
|
|
|
|
|
GFP_KERNEL, cpu_to_node(cpu));
|
|
|
|
|
|
|
|
|
|
if (!sg)
|
|
|
|
|
return NULL;
|
|
|
|
|
|
2017-05-01 09:03:12 +00:00
|
|
|
sg_span = sched_group_span(sg);
|
2017-04-13 13:56:07 +00:00
|
|
|
if (sd->child)
|
|
|
|
|
cpumask_copy(sg_span, sched_domain_span(sd->child));
|
|
|
|
|
else
|
|
|
|
|
cpumask_copy(sg_span, sched_domain_span(sd));
|
|
|
|
|
|
2017-08-10 07:52:16 +00:00
|
|
|
atomic_inc(&sg->ref);
|
2017-04-13 13:56:07 +00:00
|
|
|
return sg;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void init_overlap_sched_group(struct sched_domain *sd,
|
sched/topology: Fix overlapping sched_group_capacity
When building the overlapping groups we need to attach a consistent
sched_group_capacity structure. That is, all 'identical' sched_group's
should have the _same_ sched_group_capacity.
This can (once again) be demonstrated with a topology like:
node 0 1 2 3
0: 10 20 30 20
1: 20 10 20 30
2: 30 20 10 20
3: 20 30 20 10
But we need at least 2 CPUs per node for this to show up, after all,
if there is only one CPU per node, our CPU @i is per definition a
unique CPU that reaches this domain (aka balance-cpu).
Given the above NUMA topo and 2 CPUs per node:
[] CPU0 attaching sched-domain(s):
[] domain-0: span=0,4 level=DIE
[] groups: 0:{ span=0 }, 4:{ span=4 }
[] domain-1: span=0-1,3-5,7 level=NUMA
[] groups: 0:{ span=0,4 mask=0,4 cap=2048 }, 1:{ span=1,5 mask=1,5 cap=2048 }, 3:{ span=3,7 mask=3,7 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 0:{ span=0-1,3-5,7 mask=0,4 cap=6144 }, 2:{ span=1-3,5-7 mask=2,6 cap=6144 }
[] CPU1 attaching sched-domain(s):
[] domain-0: span=1,5 level=DIE
[] groups: 1:{ span=1 }, 5:{ span=5 }
[] domain-1: span=0-2,4-6 level=NUMA
[] groups: 1:{ span=1,5 mask=1,5 cap=2048 }, 2:{ span=2,6 mask=2,6 cap=2048 }, 4:{ span=0,4 mask=0,4 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 1:{ span=0-2,4-6 mask=1,5 cap=6144 }, 3:{ span=0,2-4,6-7 mask=3,7 cap=6144 }
Observe how CPU0-domain1-group0 and CPU1-domain1-group4 are the
'same' but have a different id (0 vs 4).
To fix this, use the group balance CPU to select the SGC. This means
we have to compute the full mask for each CPU and require a second
temporary mask to store the group mask in (it otherwise lives in the
SGC).
The fixed topology looks like:
[] CPU0 attaching sched-domain(s):
[] domain-0: span=0,4 level=DIE
[] groups: 0:{ span=0 }, 4:{ span=4 }
[] domain-1: span=0-1,3-5,7 level=NUMA
[] groups: 0:{ span=0,4 mask=0,4 cap=2048 }, 1:{ span=1,5 mask=1,5 cap=2048 }, 3:{ span=3,7 mask=3,7 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 0:{ span=0-1,3-5,7 mask=0,4 cap=6144 }, 2:{ span=1-3,5-7 mask=2,6 cap=6144 }
[] CPU1 attaching sched-domain(s):
[] domain-0: span=1,5 level=DIE
[] groups: 1:{ span=1 }, 5:{ span=5 }
[] domain-1: span=0-2,4-6 level=NUMA
[] groups: 1:{ span=1,5 mask=1,5 cap=2048 }, 2:{ span=2,6 mask=2,6 cap=2048 }, 0:{ span=0,4 mask=0,4 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 1:{ span=0-2,4-6 mask=1,5 cap=6144 }, 3:{ span=0,2-4,6-7 mask=3,7 cap=6144 }
Debugged-by: Lauro Ramos Venancio <lvenanci@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Fixes: e3589f6c81e4 ("sched: Allow for overlapping sched_domain spans")
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-04-25 12:31:11 +00:00
|
|
|
struct sched_group *sg)
|
2017-04-13 13:56:07 +00:00
|
|
|
{
|
sched/topology: Fix overlapping sched_group_capacity
When building the overlapping groups we need to attach a consistent
sched_group_capacity structure. That is, all 'identical' sched_group's
should have the _same_ sched_group_capacity.
This can (once again) be demonstrated with a topology like:
node 0 1 2 3
0: 10 20 30 20
1: 20 10 20 30
2: 30 20 10 20
3: 20 30 20 10
But we need at least 2 CPUs per node for this to show up, after all,
if there is only one CPU per node, our CPU @i is per definition a
unique CPU that reaches this domain (aka balance-cpu).
Given the above NUMA topo and 2 CPUs per node:
[] CPU0 attaching sched-domain(s):
[] domain-0: span=0,4 level=DIE
[] groups: 0:{ span=0 }, 4:{ span=4 }
[] domain-1: span=0-1,3-5,7 level=NUMA
[] groups: 0:{ span=0,4 mask=0,4 cap=2048 }, 1:{ span=1,5 mask=1,5 cap=2048 }, 3:{ span=3,7 mask=3,7 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 0:{ span=0-1,3-5,7 mask=0,4 cap=6144 }, 2:{ span=1-3,5-7 mask=2,6 cap=6144 }
[] CPU1 attaching sched-domain(s):
[] domain-0: span=1,5 level=DIE
[] groups: 1:{ span=1 }, 5:{ span=5 }
[] domain-1: span=0-2,4-6 level=NUMA
[] groups: 1:{ span=1,5 mask=1,5 cap=2048 }, 2:{ span=2,6 mask=2,6 cap=2048 }, 4:{ span=0,4 mask=0,4 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 1:{ span=0-2,4-6 mask=1,5 cap=6144 }, 3:{ span=0,2-4,6-7 mask=3,7 cap=6144 }
Observe how CPU0-domain1-group0 and CPU1-domain1-group4 are the
'same' but have a different id (0 vs 4).
To fix this, use the group balance CPU to select the SGC. This means
we have to compute the full mask for each CPU and require a second
temporary mask to store the group mask in (it otherwise lives in the
SGC).
The fixed topology looks like:
[] CPU0 attaching sched-domain(s):
[] domain-0: span=0,4 level=DIE
[] groups: 0:{ span=0 }, 4:{ span=4 }
[] domain-1: span=0-1,3-5,7 level=NUMA
[] groups: 0:{ span=0,4 mask=0,4 cap=2048 }, 1:{ span=1,5 mask=1,5 cap=2048 }, 3:{ span=3,7 mask=3,7 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 0:{ span=0-1,3-5,7 mask=0,4 cap=6144 }, 2:{ span=1-3,5-7 mask=2,6 cap=6144 }
[] CPU1 attaching sched-domain(s):
[] domain-0: span=1,5 level=DIE
[] groups: 1:{ span=1 }, 5:{ span=5 }
[] domain-1: span=0-2,4-6 level=NUMA
[] groups: 1:{ span=1,5 mask=1,5 cap=2048 }, 2:{ span=2,6 mask=2,6 cap=2048 }, 0:{ span=0,4 mask=0,4 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 1:{ span=0-2,4-6 mask=1,5 cap=6144 }, 3:{ span=0,2-4,6-7 mask=3,7 cap=6144 }
Debugged-by: Lauro Ramos Venancio <lvenanci@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Fixes: e3589f6c81e4 ("sched: Allow for overlapping sched_domain spans")
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-04-25 12:31:11 +00:00
|
|
|
struct cpumask *mask = sched_domains_tmpmask2;
|
2017-04-13 13:56:07 +00:00
|
|
|
struct sd_data *sdd = sd->private;
|
|
|
|
|
struct cpumask *sg_span;
|
sched/topology: Fix overlapping sched_group_capacity
When building the overlapping groups we need to attach a consistent
sched_group_capacity structure. That is, all 'identical' sched_group's
should have the _same_ sched_group_capacity.
This can (once again) be demonstrated with a topology like:
node 0 1 2 3
0: 10 20 30 20
1: 20 10 20 30
2: 30 20 10 20
3: 20 30 20 10
But we need at least 2 CPUs per node for this to show up, after all,
if there is only one CPU per node, our CPU @i is per definition a
unique CPU that reaches this domain (aka balance-cpu).
Given the above NUMA topo and 2 CPUs per node:
[] CPU0 attaching sched-domain(s):
[] domain-0: span=0,4 level=DIE
[] groups: 0:{ span=0 }, 4:{ span=4 }
[] domain-1: span=0-1,3-5,7 level=NUMA
[] groups: 0:{ span=0,4 mask=0,4 cap=2048 }, 1:{ span=1,5 mask=1,5 cap=2048 }, 3:{ span=3,7 mask=3,7 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 0:{ span=0-1,3-5,7 mask=0,4 cap=6144 }, 2:{ span=1-3,5-7 mask=2,6 cap=6144 }
[] CPU1 attaching sched-domain(s):
[] domain-0: span=1,5 level=DIE
[] groups: 1:{ span=1 }, 5:{ span=5 }
[] domain-1: span=0-2,4-6 level=NUMA
[] groups: 1:{ span=1,5 mask=1,5 cap=2048 }, 2:{ span=2,6 mask=2,6 cap=2048 }, 4:{ span=0,4 mask=0,4 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 1:{ span=0-2,4-6 mask=1,5 cap=6144 }, 3:{ span=0,2-4,6-7 mask=3,7 cap=6144 }
Observe how CPU0-domain1-group0 and CPU1-domain1-group4 are the
'same' but have a different id (0 vs 4).
To fix this, use the group balance CPU to select the SGC. This means
we have to compute the full mask for each CPU and require a second
temporary mask to store the group mask in (it otherwise lives in the
SGC).
The fixed topology looks like:
[] CPU0 attaching sched-domain(s):
[] domain-0: span=0,4 level=DIE
[] groups: 0:{ span=0 }, 4:{ span=4 }
[] domain-1: span=0-1,3-5,7 level=NUMA
[] groups: 0:{ span=0,4 mask=0,4 cap=2048 }, 1:{ span=1,5 mask=1,5 cap=2048 }, 3:{ span=3,7 mask=3,7 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 0:{ span=0-1,3-5,7 mask=0,4 cap=6144 }, 2:{ span=1-3,5-7 mask=2,6 cap=6144 }
[] CPU1 attaching sched-domain(s):
[] domain-0: span=1,5 level=DIE
[] groups: 1:{ span=1 }, 5:{ span=5 }
[] domain-1: span=0-2,4-6 level=NUMA
[] groups: 1:{ span=1,5 mask=1,5 cap=2048 }, 2:{ span=2,6 mask=2,6 cap=2048 }, 0:{ span=0,4 mask=0,4 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 1:{ span=0-2,4-6 mask=1,5 cap=6144 }, 3:{ span=0,2-4,6-7 mask=3,7 cap=6144 }
Debugged-by: Lauro Ramos Venancio <lvenanci@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Fixes: e3589f6c81e4 ("sched: Allow for overlapping sched_domain spans")
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-04-25 12:31:11 +00:00
|
|
|
int cpu;
|
|
|
|
|
|
2017-05-01 08:47:02 +00:00
|
|
|
build_balance_mask(sd, sg, mask);
|
2021-03-25 02:31:40 +00:00
|
|
|
cpu = cpumask_first(mask);
|
2017-04-13 13:56:07 +00:00
|
|
|
|
|
|
|
|
sg->sgc = *per_cpu_ptr(sdd->sgc, cpu);
|
|
|
|
|
if (atomic_inc_return(&sg->sgc->ref) == 1)
|
2017-05-01 08:47:02 +00:00
|
|
|
cpumask_copy(group_balance_mask(sg), mask);
|
2017-04-28 08:54:26 +00:00
|
|
|
else
|
2017-05-01 08:47:02 +00:00
|
|
|
WARN_ON_ONCE(!cpumask_equal(group_balance_mask(sg), mask));
|
2017-04-13 13:56:07 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Initialize sgc->capacity such that even if we mess up the
|
|
|
|
|
* domains and no possible iteration will get us here, we won't
|
|
|
|
|
* die on a /0 trap.
|
|
|
|
|
*/
|
2017-05-01 09:03:12 +00:00
|
|
|
sg_span = sched_group_span(sg);
|
2017-04-13 13:56:07 +00:00
|
|
|
sg->sgc->capacity = SCHED_CAPACITY_SCALE * cpumask_weight(sg_span);
|
|
|
|
|
sg->sgc->min_capacity = SCHED_CAPACITY_SCALE;
|
2018-07-04 10:17:41 +00:00
|
|
|
sg->sgc->max_capacity = SCHED_CAPACITY_SCALE;
|
2017-04-13 13:56:07 +00:00
|
|
|
}
|
|
|
|
|
|
sched/topology: fix the issue groups don't span domain->span for NUMA diameter > 2
As long as NUMA diameter > 2, building sched_domain by sibling's child
domain will definitely create a sched_domain with sched_group which will
span out of the sched_domain:
+------+ +------+ +-------+ +------+
| node | 12 |node | 20 | node | 12 |node |
| 0 +---------+1 +--------+ 2 +-------+3 |
+------+ +------+ +-------+ +------+
domain0 node0 node1 node2 node3
domain1 node0+1 node0+1 node2+3 node2+3
+
domain2 node0+1+2 |
group: node0+1 |
group:node2+3 <-------------------+
when node2 is added into the domain2 of node0, kernel is using the child
domain of node2's domain2, which is domain1(node2+3). Node 3 is outside
the span of the domain including node0+1+2.
This will make load_balance() run based on screwed avg_load and group_type
in the sched_group spanning out of the sched_domain, and it also makes
select_task_rq_fair() pick an idle CPU outside the sched_domain.
Real servers which suffer from this problem include Kunpeng920 and 8-node
Sun Fire X4600-M2, at least.
Here we move to use the *child* domain of the *child* domain of node2's
domain2 as the new added sched_group. At the same, we re-use the lower
level sgc directly.
+------+ +------+ +-------+ +------+
| node | 12 |node | 20 | node | 12 |node |
| 0 +---------+1 +--------+ 2 +-------+3 |
+------+ +------+ +-------+ +------+
domain0 node0 node1 +- node2 node3
|
domain1 node0+1 node0+1 | node2+3 node2+3
|
domain2 node0+1+2 |
group: node0+1 |
group:node2 <-------------------+
While the lower level sgc is re-used, this patch only changes the remote
sched_groups for those sched_domains playing grandchild trick, therefore,
sgc->next_update is still safe since it's only touched by CPUs that have
the group span as local group. And sgc->imbalance is also safe because
sd_parent remains the same in load_balance and LB only tries other CPUs
from the local group.
Moreover, since local groups are not touched, they are still getting
roughly equal size in a TL. And should_we_balance() only matters with
local groups, so the pull probability of those groups are still roughly
equal.
Tested by the below topology:
qemu-system-aarch64 -M virt -nographic \
-smp cpus=8 \
-numa node,cpus=0-1,nodeid=0 \
-numa node,cpus=2-3,nodeid=1 \
-numa node,cpus=4-5,nodeid=2 \
-numa node,cpus=6-7,nodeid=3 \
-numa dist,src=0,dst=1,val=12 \
-numa dist,src=0,dst=2,val=20 \
-numa dist,src=0,dst=3,val=22 \
-numa dist,src=1,dst=2,val=22 \
-numa dist,src=2,dst=3,val=12 \
-numa dist,src=1,dst=3,val=24 \
-m 4G -cpu cortex-a57 -kernel arch/arm64/boot/Image
w/o patch, we get lots of "groups don't span domain->span":
[ 0.802139] CPU0 attaching sched-domain(s):
[ 0.802193] domain-0: span=0-1 level=MC
[ 0.802443] groups: 0:{ span=0 cap=1013 }, 1:{ span=1 cap=979 }
[ 0.802693] domain-1: span=0-3 level=NUMA
[ 0.802731] groups: 0:{ span=0-1 cap=1992 }, 2:{ span=2-3 cap=1943 }
[ 0.802811] domain-2: span=0-5 level=NUMA
[ 0.802829] groups: 0:{ span=0-3 cap=3935 }, 4:{ span=4-7 cap=3937 }
[ 0.802881] ERROR: groups don't span domain->span
[ 0.803058] domain-3: span=0-7 level=NUMA
[ 0.803080] groups: 0:{ span=0-5 mask=0-1 cap=5843 }, 6:{ span=4-7 mask=6-7 cap=4077 }
[ 0.804055] CPU1 attaching sched-domain(s):
[ 0.804072] domain-0: span=0-1 level=MC
[ 0.804096] groups: 1:{ span=1 cap=979 }, 0:{ span=0 cap=1013 }
[ 0.804152] domain-1: span=0-3 level=NUMA
[ 0.804170] groups: 0:{ span=0-1 cap=1992 }, 2:{ span=2-3 cap=1943 }
[ 0.804219] domain-2: span=0-5 level=NUMA
[ 0.804236] groups: 0:{ span=0-3 cap=3935 }, 4:{ span=4-7 cap=3937 }
[ 0.804302] ERROR: groups don't span domain->span
[ 0.804520] domain-3: span=0-7 level=NUMA
[ 0.804546] groups: 0:{ span=0-5 mask=0-1 cap=5843 }, 6:{ span=4-7 mask=6-7 cap=4077 }
[ 0.804677] CPU2 attaching sched-domain(s):
[ 0.804687] domain-0: span=2-3 level=MC
[ 0.804705] groups: 2:{ span=2 cap=934 }, 3:{ span=3 cap=1009 }
[ 0.804754] domain-1: span=0-3 level=NUMA
[ 0.804772] groups: 2:{ span=2-3 cap=1943 }, 0:{ span=0-1 cap=1992 }
[ 0.804820] domain-2: span=0-5 level=NUMA
[ 0.804836] groups: 2:{ span=0-3 mask=2-3 cap=3991 }, 4:{ span=0-1,4-7 mask=4-5 cap=5985 }
[ 0.804944] ERROR: groups don't span domain->span
[ 0.805108] domain-3: span=0-7 level=NUMA
[ 0.805134] groups: 2:{ span=0-5 mask=2-3 cap=5899 }, 6:{ span=0-1,4-7 mask=6-7 cap=6125 }
[ 0.805223] CPU3 attaching sched-domain(s):
[ 0.805232] domain-0: span=2-3 level=MC
[ 0.805249] groups: 3:{ span=3 cap=1009 }, 2:{ span=2 cap=934 }
[ 0.805319] domain-1: span=0-3 level=NUMA
[ 0.805336] groups: 2:{ span=2-3 cap=1943 }, 0:{ span=0-1 cap=1992 }
[ 0.805383] domain-2: span=0-5 level=NUMA
[ 0.805399] groups: 2:{ span=0-3 mask=2-3 cap=3991 }, 4:{ span=0-1,4-7 mask=4-5 cap=5985 }
[ 0.805458] ERROR: groups don't span domain->span
[ 0.805605] domain-3: span=0-7 level=NUMA
[ 0.805626] groups: 2:{ span=0-5 mask=2-3 cap=5899 }, 6:{ span=0-1,4-7 mask=6-7 cap=6125 }
[ 0.805712] CPU4 attaching sched-domain(s):
[ 0.805721] domain-0: span=4-5 level=MC
[ 0.805738] groups: 4:{ span=4 cap=984 }, 5:{ span=5 cap=924 }
[ 0.805787] domain-1: span=4-7 level=NUMA
[ 0.805803] groups: 4:{ span=4-5 cap=1908 }, 6:{ span=6-7 cap=2029 }
[ 0.805851] domain-2: span=0-1,4-7 level=NUMA
[ 0.805867] groups: 4:{ span=4-7 cap=3937 }, 0:{ span=0-3 cap=3935 }
[ 0.805915] ERROR: groups don't span domain->span
[ 0.806108] domain-3: span=0-7 level=NUMA
[ 0.806130] groups: 4:{ span=0-1,4-7 mask=4-5 cap=5985 }, 2:{ span=0-3 mask=2-3 cap=3991 }
[ 0.806214] CPU5 attaching sched-domain(s):
[ 0.806222] domain-0: span=4-5 level=MC
[ 0.806240] groups: 5:{ span=5 cap=924 }, 4:{ span=4 cap=984 }
[ 0.806841] domain-1: span=4-7 level=NUMA
[ 0.806866] groups: 4:{ span=4-5 cap=1908 }, 6:{ span=6-7 cap=2029 }
[ 0.806934] domain-2: span=0-1,4-7 level=NUMA
[ 0.806953] groups: 4:{ span=4-7 cap=3937 }, 0:{ span=0-3 cap=3935 }
[ 0.807004] ERROR: groups don't span domain->span
[ 0.807312] domain-3: span=0-7 level=NUMA
[ 0.807386] groups: 4:{ span=0-1,4-7 mask=4-5 cap=5985 }, 2:{ span=0-3 mask=2-3 cap=3991 }
[ 0.807686] CPU6 attaching sched-domain(s):
[ 0.807710] domain-0: span=6-7 level=MC
[ 0.807750] groups: 6:{ span=6 cap=1017 }, 7:{ span=7 cap=1012 }
[ 0.807840] domain-1: span=4-7 level=NUMA
[ 0.807870] groups: 6:{ span=6-7 cap=2029 }, 4:{ span=4-5 cap=1908 }
[ 0.807952] domain-2: span=0-1,4-7 level=NUMA
[ 0.807985] groups: 6:{ span=4-7 mask=6-7 cap=4077 }, 0:{ span=0-5 mask=0-1 cap=5843 }
[ 0.808045] ERROR: groups don't span domain->span
[ 0.808257] domain-3: span=0-7 level=NUMA
[ 0.808571] groups: 6:{ span=0-1,4-7 mask=6-7 cap=6125 }, 2:{ span=0-5 mask=2-3 cap=5899 }
[ 0.808848] CPU7 attaching sched-domain(s):
[ 0.808860] domain-0: span=6-7 level=MC
[ 0.808880] groups: 7:{ span=7 cap=1012 }, 6:{ span=6 cap=1017 }
[ 0.808953] domain-1: span=4-7 level=NUMA
[ 0.808974] groups: 6:{ span=6-7 cap=2029 }, 4:{ span=4-5 cap=1908 }
[ 0.809034] domain-2: span=0-1,4-7 level=NUMA
[ 0.809055] groups: 6:{ span=4-7 mask=6-7 cap=4077 }, 0:{ span=0-5 mask=0-1 cap=5843 }
[ 0.809128] ERROR: groups don't span domain->span
[ 0.810361] domain-3: span=0-7 level=NUMA
[ 0.810400] groups: 6:{ span=0-1,4-7 mask=6-7 cap=5961 }, 2:{ span=0-5 mask=2-3 cap=5903 }
w/ patch, we don't get "groups don't span domain->span" any more:
[ 1.486271] CPU0 attaching sched-domain(s):
[ 1.486820] domain-0: span=0-1 level=MC
[ 1.500924] groups: 0:{ span=0 cap=980 }, 1:{ span=1 cap=994 }
[ 1.515717] domain-1: span=0-3 level=NUMA
[ 1.515903] groups: 0:{ span=0-1 cap=1974 }, 2:{ span=2-3 cap=1989 }
[ 1.516989] domain-2: span=0-5 level=NUMA
[ 1.517124] groups: 0:{ span=0-3 cap=3963 }, 4:{ span=4-5 cap=1949 }
[ 1.517369] domain-3: span=0-7 level=NUMA
[ 1.517423] groups: 0:{ span=0-5 mask=0-1 cap=5912 }, 6:{ span=4-7 mask=6-7 cap=4054 }
[ 1.520027] CPU1 attaching sched-domain(s):
[ 1.520097] domain-0: span=0-1 level=MC
[ 1.520184] groups: 1:{ span=1 cap=994 }, 0:{ span=0 cap=980 }
[ 1.520429] domain-1: span=0-3 level=NUMA
[ 1.520487] groups: 0:{ span=0-1 cap=1974 }, 2:{ span=2-3 cap=1989 }
[ 1.520687] domain-2: span=0-5 level=NUMA
[ 1.520744] groups: 0:{ span=0-3 cap=3963 }, 4:{ span=4-5 cap=1949 }
[ 1.520948] domain-3: span=0-7 level=NUMA
[ 1.521038] groups: 0:{ span=0-5 mask=0-1 cap=5912 }, 6:{ span=4-7 mask=6-7 cap=4054 }
[ 1.522068] CPU2 attaching sched-domain(s):
[ 1.522348] domain-0: span=2-3 level=MC
[ 1.522606] groups: 2:{ span=2 cap=1003 }, 3:{ span=3 cap=986 }
[ 1.522832] domain-1: span=0-3 level=NUMA
[ 1.522885] groups: 2:{ span=2-3 cap=1989 }, 0:{ span=0-1 cap=1974 }
[ 1.523043] domain-2: span=0-5 level=NUMA
[ 1.523092] groups: 2:{ span=0-3 mask=2-3 cap=4037 }, 4:{ span=4-5 cap=1949 }
[ 1.523302] domain-3: span=0-7 level=NUMA
[ 1.523352] groups: 2:{ span=0-5 mask=2-3 cap=5986 }, 6:{ span=0-1,4-7 mask=6-7 cap=6102 }
[ 1.523748] CPU3 attaching sched-domain(s):
[ 1.523774] domain-0: span=2-3 level=MC
[ 1.523825] groups: 3:{ span=3 cap=986 }, 2:{ span=2 cap=1003 }
[ 1.524009] domain-1: span=0-3 level=NUMA
[ 1.524086] groups: 2:{ span=2-3 cap=1989 }, 0:{ span=0-1 cap=1974 }
[ 1.524281] domain-2: span=0-5 level=NUMA
[ 1.524331] groups: 2:{ span=0-3 mask=2-3 cap=4037 }, 4:{ span=4-5 cap=1949 }
[ 1.524534] domain-3: span=0-7 level=NUMA
[ 1.524586] groups: 2:{ span=0-5 mask=2-3 cap=5986 }, 6:{ span=0-1,4-7 mask=6-7 cap=6102 }
[ 1.524847] CPU4 attaching sched-domain(s):
[ 1.524873] domain-0: span=4-5 level=MC
[ 1.524954] groups: 4:{ span=4 cap=958 }, 5:{ span=5 cap=991 }
[ 1.525105] domain-1: span=4-7 level=NUMA
[ 1.525153] groups: 4:{ span=4-5 cap=1949 }, 6:{ span=6-7 cap=2006 }
[ 1.525368] domain-2: span=0-1,4-7 level=NUMA
[ 1.525428] groups: 4:{ span=4-7 cap=3955 }, 0:{ span=0-1 cap=1974 }
[ 1.532726] domain-3: span=0-7 level=NUMA
[ 1.532811] groups: 4:{ span=0-1,4-7 mask=4-5 cap=6003 }, 2:{ span=0-3 mask=2-3 cap=4037 }
[ 1.534125] CPU5 attaching sched-domain(s):
[ 1.534159] domain-0: span=4-5 level=MC
[ 1.534303] groups: 5:{ span=5 cap=991 }, 4:{ span=4 cap=958 }
[ 1.534490] domain-1: span=4-7 level=NUMA
[ 1.534572] groups: 4:{ span=4-5 cap=1949 }, 6:{ span=6-7 cap=2006 }
[ 1.534734] domain-2: span=0-1,4-7 level=NUMA
[ 1.534783] groups: 4:{ span=4-7 cap=3955 }, 0:{ span=0-1 cap=1974 }
[ 1.536057] domain-3: span=0-7 level=NUMA
[ 1.536430] groups: 4:{ span=0-1,4-7 mask=4-5 cap=6003 }, 2:{ span=0-3 mask=2-3 cap=3896 }
[ 1.536815] CPU6 attaching sched-domain(s):
[ 1.536846] domain-0: span=6-7 level=MC
[ 1.536934] groups: 6:{ span=6 cap=1005 }, 7:{ span=7 cap=1001 }
[ 1.537144] domain-1: span=4-7 level=NUMA
[ 1.537262] groups: 6:{ span=6-7 cap=2006 }, 4:{ span=4-5 cap=1949 }
[ 1.537553] domain-2: span=0-1,4-7 level=NUMA
[ 1.537613] groups: 6:{ span=4-7 mask=6-7 cap=4054 }, 0:{ span=0-1 cap=1805 }
[ 1.537872] domain-3: span=0-7 level=NUMA
[ 1.537998] groups: 6:{ span=0-1,4-7 mask=6-7 cap=6102 }, 2:{ span=0-5 mask=2-3 cap=5845 }
[ 1.538448] CPU7 attaching sched-domain(s):
[ 1.538505] domain-0: span=6-7 level=MC
[ 1.538586] groups: 7:{ span=7 cap=1001 }, 6:{ span=6 cap=1005 }
[ 1.538746] domain-1: span=4-7 level=NUMA
[ 1.538798] groups: 6:{ span=6-7 cap=2006 }, 4:{ span=4-5 cap=1949 }
[ 1.539048] domain-2: span=0-1,4-7 level=NUMA
[ 1.539111] groups: 6:{ span=4-7 mask=6-7 cap=4054 }, 0:{ span=0-1 cap=1805 }
[ 1.539571] domain-3: span=0-7 level=NUMA
[ 1.539610] groups: 6:{ span=0-1,4-7 mask=6-7 cap=6102 }, 2:{ span=0-5 mask=2-3 cap=5845 }
Signed-off-by: Barry Song <song.bao.hua@hisilicon.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Reviewed-by: Valentin Schneider <valentin.schneider@arm.com>
Tested-by: Meelis Roos <mroos@linux.ee>
Link: https://lkml.kernel.org/r/20210224030944.15232-1-song.bao.hua@hisilicon.com
2021-02-24 03:09:44 +00:00
|
|
|
static struct sched_domain *
|
|
|
|
|
find_descended_sibling(struct sched_domain *sd, struct sched_domain *sibling)
|
|
|
|
|
{
|
|
|
|
|
/*
|
|
|
|
|
* The proper descendant would be the one whose child won't span out
|
|
|
|
|
* of sd
|
|
|
|
|
*/
|
|
|
|
|
while (sibling->child &&
|
|
|
|
|
!cpumask_subset(sched_domain_span(sibling->child),
|
|
|
|
|
sched_domain_span(sd)))
|
|
|
|
|
sibling = sibling->child;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* As we are referencing sgc across different topology level, we need
|
|
|
|
|
* to go down to skip those sched_domains which don't contribute to
|
|
|
|
|
* scheduling because they will be degenerated in cpu_attach_domain
|
|
|
|
|
*/
|
|
|
|
|
while (sibling->child &&
|
|
|
|
|
cpumask_equal(sched_domain_span(sibling->child),
|
|
|
|
|
sched_domain_span(sibling)))
|
|
|
|
|
sibling = sibling->child;
|
|
|
|
|
|
|
|
|
|
return sibling;
|
|
|
|
|
}
|
|
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
static int
|
|
|
|
|
build_overlap_sched_groups(struct sched_domain *sd, int cpu)
|
|
|
|
|
{
|
2017-04-14 15:32:07 +00:00
|
|
|
struct sched_group *first = NULL, *last = NULL, *sg;
|
2017-02-01 12:10:18 +00:00
|
|
|
const struct cpumask *span = sched_domain_span(sd);
|
|
|
|
|
struct cpumask *covered = sched_domains_tmpmask;
|
|
|
|
|
struct sd_data *sdd = sd->private;
|
|
|
|
|
struct sched_domain *sibling;
|
|
|
|
|
int i;
|
|
|
|
|
|
|
|
|
|
cpumask_clear(covered);
|
|
|
|
|
|
2017-04-14 15:24:02 +00:00
|
|
|
for_each_cpu_wrap(i, span, cpu) {
|
2017-02-01 12:10:18 +00:00
|
|
|
struct cpumask *sg_span;
|
|
|
|
|
|
|
|
|
|
if (cpumask_test_cpu(i, covered))
|
|
|
|
|
continue;
|
|
|
|
|
|
|
|
|
|
sibling = *per_cpu_ptr(sdd->sd, i);
|
|
|
|
|
|
2017-04-20 19:51:42 +00:00
|
|
|
/*
|
|
|
|
|
* Asymmetric node setups can result in situations where the
|
|
|
|
|
* domain tree is of unequal depth, make sure to skip domains
|
|
|
|
|
* that already cover the entire range.
|
|
|
|
|
*
|
|
|
|
|
* In that case build_sched_domains() will have terminated the
|
|
|
|
|
* iteration early and our sibling sd spans will be empty.
|
|
|
|
|
* Domains should always include the CPU they're built on, so
|
|
|
|
|
* check that.
|
|
|
|
|
*/
|
2017-02-01 12:10:18 +00:00
|
|
|
if (!cpumask_test_cpu(i, sched_domain_span(sibling)))
|
|
|
|
|
continue;
|
|
|
|
|
|
sched/topology: fix the issue groups don't span domain->span for NUMA diameter > 2
As long as NUMA diameter > 2, building sched_domain by sibling's child
domain will definitely create a sched_domain with sched_group which will
span out of the sched_domain:
+------+ +------+ +-------+ +------+
| node | 12 |node | 20 | node | 12 |node |
| 0 +---------+1 +--------+ 2 +-------+3 |
+------+ +------+ +-------+ +------+
domain0 node0 node1 node2 node3
domain1 node0+1 node0+1 node2+3 node2+3
+
domain2 node0+1+2 |
group: node0+1 |
group:node2+3 <-------------------+
when node2 is added into the domain2 of node0, kernel is using the child
domain of node2's domain2, which is domain1(node2+3). Node 3 is outside
the span of the domain including node0+1+2.
This will make load_balance() run based on screwed avg_load and group_type
in the sched_group spanning out of the sched_domain, and it also makes
select_task_rq_fair() pick an idle CPU outside the sched_domain.
Real servers which suffer from this problem include Kunpeng920 and 8-node
Sun Fire X4600-M2, at least.
Here we move to use the *child* domain of the *child* domain of node2's
domain2 as the new added sched_group. At the same, we re-use the lower
level sgc directly.
+------+ +------+ +-------+ +------+
| node | 12 |node | 20 | node | 12 |node |
| 0 +---------+1 +--------+ 2 +-------+3 |
+------+ +------+ +-------+ +------+
domain0 node0 node1 +- node2 node3
|
domain1 node0+1 node0+1 | node2+3 node2+3
|
domain2 node0+1+2 |
group: node0+1 |
group:node2 <-------------------+
While the lower level sgc is re-used, this patch only changes the remote
sched_groups for those sched_domains playing grandchild trick, therefore,
sgc->next_update is still safe since it's only touched by CPUs that have
the group span as local group. And sgc->imbalance is also safe because
sd_parent remains the same in load_balance and LB only tries other CPUs
from the local group.
Moreover, since local groups are not touched, they are still getting
roughly equal size in a TL. And should_we_balance() only matters with
local groups, so the pull probability of those groups are still roughly
equal.
Tested by the below topology:
qemu-system-aarch64 -M virt -nographic \
-smp cpus=8 \
-numa node,cpus=0-1,nodeid=0 \
-numa node,cpus=2-3,nodeid=1 \
-numa node,cpus=4-5,nodeid=2 \
-numa node,cpus=6-7,nodeid=3 \
-numa dist,src=0,dst=1,val=12 \
-numa dist,src=0,dst=2,val=20 \
-numa dist,src=0,dst=3,val=22 \
-numa dist,src=1,dst=2,val=22 \
-numa dist,src=2,dst=3,val=12 \
-numa dist,src=1,dst=3,val=24 \
-m 4G -cpu cortex-a57 -kernel arch/arm64/boot/Image
w/o patch, we get lots of "groups don't span domain->span":
[ 0.802139] CPU0 attaching sched-domain(s):
[ 0.802193] domain-0: span=0-1 level=MC
[ 0.802443] groups: 0:{ span=0 cap=1013 }, 1:{ span=1 cap=979 }
[ 0.802693] domain-1: span=0-3 level=NUMA
[ 0.802731] groups: 0:{ span=0-1 cap=1992 }, 2:{ span=2-3 cap=1943 }
[ 0.802811] domain-2: span=0-5 level=NUMA
[ 0.802829] groups: 0:{ span=0-3 cap=3935 }, 4:{ span=4-7 cap=3937 }
[ 0.802881] ERROR: groups don't span domain->span
[ 0.803058] domain-3: span=0-7 level=NUMA
[ 0.803080] groups: 0:{ span=0-5 mask=0-1 cap=5843 }, 6:{ span=4-7 mask=6-7 cap=4077 }
[ 0.804055] CPU1 attaching sched-domain(s):
[ 0.804072] domain-0: span=0-1 level=MC
[ 0.804096] groups: 1:{ span=1 cap=979 }, 0:{ span=0 cap=1013 }
[ 0.804152] domain-1: span=0-3 level=NUMA
[ 0.804170] groups: 0:{ span=0-1 cap=1992 }, 2:{ span=2-3 cap=1943 }
[ 0.804219] domain-2: span=0-5 level=NUMA
[ 0.804236] groups: 0:{ span=0-3 cap=3935 }, 4:{ span=4-7 cap=3937 }
[ 0.804302] ERROR: groups don't span domain->span
[ 0.804520] domain-3: span=0-7 level=NUMA
[ 0.804546] groups: 0:{ span=0-5 mask=0-1 cap=5843 }, 6:{ span=4-7 mask=6-7 cap=4077 }
[ 0.804677] CPU2 attaching sched-domain(s):
[ 0.804687] domain-0: span=2-3 level=MC
[ 0.804705] groups: 2:{ span=2 cap=934 }, 3:{ span=3 cap=1009 }
[ 0.804754] domain-1: span=0-3 level=NUMA
[ 0.804772] groups: 2:{ span=2-3 cap=1943 }, 0:{ span=0-1 cap=1992 }
[ 0.804820] domain-2: span=0-5 level=NUMA
[ 0.804836] groups: 2:{ span=0-3 mask=2-3 cap=3991 }, 4:{ span=0-1,4-7 mask=4-5 cap=5985 }
[ 0.804944] ERROR: groups don't span domain->span
[ 0.805108] domain-3: span=0-7 level=NUMA
[ 0.805134] groups: 2:{ span=0-5 mask=2-3 cap=5899 }, 6:{ span=0-1,4-7 mask=6-7 cap=6125 }
[ 0.805223] CPU3 attaching sched-domain(s):
[ 0.805232] domain-0: span=2-3 level=MC
[ 0.805249] groups: 3:{ span=3 cap=1009 }, 2:{ span=2 cap=934 }
[ 0.805319] domain-1: span=0-3 level=NUMA
[ 0.805336] groups: 2:{ span=2-3 cap=1943 }, 0:{ span=0-1 cap=1992 }
[ 0.805383] domain-2: span=0-5 level=NUMA
[ 0.805399] groups: 2:{ span=0-3 mask=2-3 cap=3991 }, 4:{ span=0-1,4-7 mask=4-5 cap=5985 }
[ 0.805458] ERROR: groups don't span domain->span
[ 0.805605] domain-3: span=0-7 level=NUMA
[ 0.805626] groups: 2:{ span=0-5 mask=2-3 cap=5899 }, 6:{ span=0-1,4-7 mask=6-7 cap=6125 }
[ 0.805712] CPU4 attaching sched-domain(s):
[ 0.805721] domain-0: span=4-5 level=MC
[ 0.805738] groups: 4:{ span=4 cap=984 }, 5:{ span=5 cap=924 }
[ 0.805787] domain-1: span=4-7 level=NUMA
[ 0.805803] groups: 4:{ span=4-5 cap=1908 }, 6:{ span=6-7 cap=2029 }
[ 0.805851] domain-2: span=0-1,4-7 level=NUMA
[ 0.805867] groups: 4:{ span=4-7 cap=3937 }, 0:{ span=0-3 cap=3935 }
[ 0.805915] ERROR: groups don't span domain->span
[ 0.806108] domain-3: span=0-7 level=NUMA
[ 0.806130] groups: 4:{ span=0-1,4-7 mask=4-5 cap=5985 }, 2:{ span=0-3 mask=2-3 cap=3991 }
[ 0.806214] CPU5 attaching sched-domain(s):
[ 0.806222] domain-0: span=4-5 level=MC
[ 0.806240] groups: 5:{ span=5 cap=924 }, 4:{ span=4 cap=984 }
[ 0.806841] domain-1: span=4-7 level=NUMA
[ 0.806866] groups: 4:{ span=4-5 cap=1908 }, 6:{ span=6-7 cap=2029 }
[ 0.806934] domain-2: span=0-1,4-7 level=NUMA
[ 0.806953] groups: 4:{ span=4-7 cap=3937 }, 0:{ span=0-3 cap=3935 }
[ 0.807004] ERROR: groups don't span domain->span
[ 0.807312] domain-3: span=0-7 level=NUMA
[ 0.807386] groups: 4:{ span=0-1,4-7 mask=4-5 cap=5985 }, 2:{ span=0-3 mask=2-3 cap=3991 }
[ 0.807686] CPU6 attaching sched-domain(s):
[ 0.807710] domain-0: span=6-7 level=MC
[ 0.807750] groups: 6:{ span=6 cap=1017 }, 7:{ span=7 cap=1012 }
[ 0.807840] domain-1: span=4-7 level=NUMA
[ 0.807870] groups: 6:{ span=6-7 cap=2029 }, 4:{ span=4-5 cap=1908 }
[ 0.807952] domain-2: span=0-1,4-7 level=NUMA
[ 0.807985] groups: 6:{ span=4-7 mask=6-7 cap=4077 }, 0:{ span=0-5 mask=0-1 cap=5843 }
[ 0.808045] ERROR: groups don't span domain->span
[ 0.808257] domain-3: span=0-7 level=NUMA
[ 0.808571] groups: 6:{ span=0-1,4-7 mask=6-7 cap=6125 }, 2:{ span=0-5 mask=2-3 cap=5899 }
[ 0.808848] CPU7 attaching sched-domain(s):
[ 0.808860] domain-0: span=6-7 level=MC
[ 0.808880] groups: 7:{ span=7 cap=1012 }, 6:{ span=6 cap=1017 }
[ 0.808953] domain-1: span=4-7 level=NUMA
[ 0.808974] groups: 6:{ span=6-7 cap=2029 }, 4:{ span=4-5 cap=1908 }
[ 0.809034] domain-2: span=0-1,4-7 level=NUMA
[ 0.809055] groups: 6:{ span=4-7 mask=6-7 cap=4077 }, 0:{ span=0-5 mask=0-1 cap=5843 }
[ 0.809128] ERROR: groups don't span domain->span
[ 0.810361] domain-3: span=0-7 level=NUMA
[ 0.810400] groups: 6:{ span=0-1,4-7 mask=6-7 cap=5961 }, 2:{ span=0-5 mask=2-3 cap=5903 }
w/ patch, we don't get "groups don't span domain->span" any more:
[ 1.486271] CPU0 attaching sched-domain(s):
[ 1.486820] domain-0: span=0-1 level=MC
[ 1.500924] groups: 0:{ span=0 cap=980 }, 1:{ span=1 cap=994 }
[ 1.515717] domain-1: span=0-3 level=NUMA
[ 1.515903] groups: 0:{ span=0-1 cap=1974 }, 2:{ span=2-3 cap=1989 }
[ 1.516989] domain-2: span=0-5 level=NUMA
[ 1.517124] groups: 0:{ span=0-3 cap=3963 }, 4:{ span=4-5 cap=1949 }
[ 1.517369] domain-3: span=0-7 level=NUMA
[ 1.517423] groups: 0:{ span=0-5 mask=0-1 cap=5912 }, 6:{ span=4-7 mask=6-7 cap=4054 }
[ 1.520027] CPU1 attaching sched-domain(s):
[ 1.520097] domain-0: span=0-1 level=MC
[ 1.520184] groups: 1:{ span=1 cap=994 }, 0:{ span=0 cap=980 }
[ 1.520429] domain-1: span=0-3 level=NUMA
[ 1.520487] groups: 0:{ span=0-1 cap=1974 }, 2:{ span=2-3 cap=1989 }
[ 1.520687] domain-2: span=0-5 level=NUMA
[ 1.520744] groups: 0:{ span=0-3 cap=3963 }, 4:{ span=4-5 cap=1949 }
[ 1.520948] domain-3: span=0-7 level=NUMA
[ 1.521038] groups: 0:{ span=0-5 mask=0-1 cap=5912 }, 6:{ span=4-7 mask=6-7 cap=4054 }
[ 1.522068] CPU2 attaching sched-domain(s):
[ 1.522348] domain-0: span=2-3 level=MC
[ 1.522606] groups: 2:{ span=2 cap=1003 }, 3:{ span=3 cap=986 }
[ 1.522832] domain-1: span=0-3 level=NUMA
[ 1.522885] groups: 2:{ span=2-3 cap=1989 }, 0:{ span=0-1 cap=1974 }
[ 1.523043] domain-2: span=0-5 level=NUMA
[ 1.523092] groups: 2:{ span=0-3 mask=2-3 cap=4037 }, 4:{ span=4-5 cap=1949 }
[ 1.523302] domain-3: span=0-7 level=NUMA
[ 1.523352] groups: 2:{ span=0-5 mask=2-3 cap=5986 }, 6:{ span=0-1,4-7 mask=6-7 cap=6102 }
[ 1.523748] CPU3 attaching sched-domain(s):
[ 1.523774] domain-0: span=2-3 level=MC
[ 1.523825] groups: 3:{ span=3 cap=986 }, 2:{ span=2 cap=1003 }
[ 1.524009] domain-1: span=0-3 level=NUMA
[ 1.524086] groups: 2:{ span=2-3 cap=1989 }, 0:{ span=0-1 cap=1974 }
[ 1.524281] domain-2: span=0-5 level=NUMA
[ 1.524331] groups: 2:{ span=0-3 mask=2-3 cap=4037 }, 4:{ span=4-5 cap=1949 }
[ 1.524534] domain-3: span=0-7 level=NUMA
[ 1.524586] groups: 2:{ span=0-5 mask=2-3 cap=5986 }, 6:{ span=0-1,4-7 mask=6-7 cap=6102 }
[ 1.524847] CPU4 attaching sched-domain(s):
[ 1.524873] domain-0: span=4-5 level=MC
[ 1.524954] groups: 4:{ span=4 cap=958 }, 5:{ span=5 cap=991 }
[ 1.525105] domain-1: span=4-7 level=NUMA
[ 1.525153] groups: 4:{ span=4-5 cap=1949 }, 6:{ span=6-7 cap=2006 }
[ 1.525368] domain-2: span=0-1,4-7 level=NUMA
[ 1.525428] groups: 4:{ span=4-7 cap=3955 }, 0:{ span=0-1 cap=1974 }
[ 1.532726] domain-3: span=0-7 level=NUMA
[ 1.532811] groups: 4:{ span=0-1,4-7 mask=4-5 cap=6003 }, 2:{ span=0-3 mask=2-3 cap=4037 }
[ 1.534125] CPU5 attaching sched-domain(s):
[ 1.534159] domain-0: span=4-5 level=MC
[ 1.534303] groups: 5:{ span=5 cap=991 }, 4:{ span=4 cap=958 }
[ 1.534490] domain-1: span=4-7 level=NUMA
[ 1.534572] groups: 4:{ span=4-5 cap=1949 }, 6:{ span=6-7 cap=2006 }
[ 1.534734] domain-2: span=0-1,4-7 level=NUMA
[ 1.534783] groups: 4:{ span=4-7 cap=3955 }, 0:{ span=0-1 cap=1974 }
[ 1.536057] domain-3: span=0-7 level=NUMA
[ 1.536430] groups: 4:{ span=0-1,4-7 mask=4-5 cap=6003 }, 2:{ span=0-3 mask=2-3 cap=3896 }
[ 1.536815] CPU6 attaching sched-domain(s):
[ 1.536846] domain-0: span=6-7 level=MC
[ 1.536934] groups: 6:{ span=6 cap=1005 }, 7:{ span=7 cap=1001 }
[ 1.537144] domain-1: span=4-7 level=NUMA
[ 1.537262] groups: 6:{ span=6-7 cap=2006 }, 4:{ span=4-5 cap=1949 }
[ 1.537553] domain-2: span=0-1,4-7 level=NUMA
[ 1.537613] groups: 6:{ span=4-7 mask=6-7 cap=4054 }, 0:{ span=0-1 cap=1805 }
[ 1.537872] domain-3: span=0-7 level=NUMA
[ 1.537998] groups: 6:{ span=0-1,4-7 mask=6-7 cap=6102 }, 2:{ span=0-5 mask=2-3 cap=5845 }
[ 1.538448] CPU7 attaching sched-domain(s):
[ 1.538505] domain-0: span=6-7 level=MC
[ 1.538586] groups: 7:{ span=7 cap=1001 }, 6:{ span=6 cap=1005 }
[ 1.538746] domain-1: span=4-7 level=NUMA
[ 1.538798] groups: 6:{ span=6-7 cap=2006 }, 4:{ span=4-5 cap=1949 }
[ 1.539048] domain-2: span=0-1,4-7 level=NUMA
[ 1.539111] groups: 6:{ span=4-7 mask=6-7 cap=4054 }, 0:{ span=0-1 cap=1805 }
[ 1.539571] domain-3: span=0-7 level=NUMA
[ 1.539610] groups: 6:{ span=0-1,4-7 mask=6-7 cap=6102 }, 2:{ span=0-5 mask=2-3 cap=5845 }
Signed-off-by: Barry Song <song.bao.hua@hisilicon.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Reviewed-by: Valentin Schneider <valentin.schneider@arm.com>
Tested-by: Meelis Roos <mroos@linux.ee>
Link: https://lkml.kernel.org/r/20210224030944.15232-1-song.bao.hua@hisilicon.com
2021-02-24 03:09:44 +00:00
|
|
|
/*
|
|
|
|
|
* Usually we build sched_group by sibling's child sched_domain
|
|
|
|
|
* But for machines whose NUMA diameter are 3 or above, we move
|
|
|
|
|
* to build sched_group by sibling's proper descendant's child
|
|
|
|
|
* domain because sibling's child sched_domain will span out of
|
|
|
|
|
* the sched_domain being built as below.
|
|
|
|
|
*
|
|
|
|
|
* Smallest diameter=3 topology is:
|
|
|
|
|
*
|
|
|
|
|
* node 0 1 2 3
|
|
|
|
|
* 0: 10 20 30 40
|
|
|
|
|
* 1: 20 10 20 30
|
|
|
|
|
* 2: 30 20 10 20
|
|
|
|
|
* 3: 40 30 20 10
|
|
|
|
|
*
|
|
|
|
|
* 0 --- 1 --- 2 --- 3
|
|
|
|
|
*
|
|
|
|
|
* NUMA-3 0-3 N/A N/A 0-3
|
|
|
|
|
* groups: {0-2},{1-3} {1-3},{0-2}
|
|
|
|
|
*
|
|
|
|
|
* NUMA-2 0-2 0-3 0-3 1-3
|
|
|
|
|
* groups: {0-1},{1-3} {0-2},{2-3} {1-3},{0-1} {2-3},{0-2}
|
|
|
|
|
*
|
|
|
|
|
* NUMA-1 0-1 0-2 1-3 2-3
|
|
|
|
|
* groups: {0},{1} {1},{2},{0} {2},{3},{1} {3},{2}
|
|
|
|
|
*
|
|
|
|
|
* NUMA-0 0 1 2 3
|
|
|
|
|
*
|
|
|
|
|
* The NUMA-2 groups for nodes 0 and 3 are obviously buggered, as the
|
|
|
|
|
* group span isn't a subset of the domain span.
|
|
|
|
|
*/
|
|
|
|
|
if (sibling->child &&
|
|
|
|
|
!cpumask_subset(sched_domain_span(sibling->child), span))
|
|
|
|
|
sibling = find_descended_sibling(sd, sibling);
|
|
|
|
|
|
2017-04-13 13:56:07 +00:00
|
|
|
sg = build_group_from_child_sched_domain(sibling, cpu);
|
2017-02-01 12:10:18 +00:00
|
|
|
if (!sg)
|
|
|
|
|
goto fail;
|
|
|
|
|
|
2017-05-01 09:03:12 +00:00
|
|
|
sg_span = sched_group_span(sg);
|
2017-02-01 12:10:18 +00:00
|
|
|
cpumask_or(covered, covered, sg_span);
|
|
|
|
|
|
sched/topology: fix the issue groups don't span domain->span for NUMA diameter > 2
As long as NUMA diameter > 2, building sched_domain by sibling's child
domain will definitely create a sched_domain with sched_group which will
span out of the sched_domain:
+------+ +------+ +-------+ +------+
| node | 12 |node | 20 | node | 12 |node |
| 0 +---------+1 +--------+ 2 +-------+3 |
+------+ +------+ +-------+ +------+
domain0 node0 node1 node2 node3
domain1 node0+1 node0+1 node2+3 node2+3
+
domain2 node0+1+2 |
group: node0+1 |
group:node2+3 <-------------------+
when node2 is added into the domain2 of node0, kernel is using the child
domain of node2's domain2, which is domain1(node2+3). Node 3 is outside
the span of the domain including node0+1+2.
This will make load_balance() run based on screwed avg_load and group_type
in the sched_group spanning out of the sched_domain, and it also makes
select_task_rq_fair() pick an idle CPU outside the sched_domain.
Real servers which suffer from this problem include Kunpeng920 and 8-node
Sun Fire X4600-M2, at least.
Here we move to use the *child* domain of the *child* domain of node2's
domain2 as the new added sched_group. At the same, we re-use the lower
level sgc directly.
+------+ +------+ +-------+ +------+
| node | 12 |node | 20 | node | 12 |node |
| 0 +---------+1 +--------+ 2 +-------+3 |
+------+ +------+ +-------+ +------+
domain0 node0 node1 +- node2 node3
|
domain1 node0+1 node0+1 | node2+3 node2+3
|
domain2 node0+1+2 |
group: node0+1 |
group:node2 <-------------------+
While the lower level sgc is re-used, this patch only changes the remote
sched_groups for those sched_domains playing grandchild trick, therefore,
sgc->next_update is still safe since it's only touched by CPUs that have
the group span as local group. And sgc->imbalance is also safe because
sd_parent remains the same in load_balance and LB only tries other CPUs
from the local group.
Moreover, since local groups are not touched, they are still getting
roughly equal size in a TL. And should_we_balance() only matters with
local groups, so the pull probability of those groups are still roughly
equal.
Tested by the below topology:
qemu-system-aarch64 -M virt -nographic \
-smp cpus=8 \
-numa node,cpus=0-1,nodeid=0 \
-numa node,cpus=2-3,nodeid=1 \
-numa node,cpus=4-5,nodeid=2 \
-numa node,cpus=6-7,nodeid=3 \
-numa dist,src=0,dst=1,val=12 \
-numa dist,src=0,dst=2,val=20 \
-numa dist,src=0,dst=3,val=22 \
-numa dist,src=1,dst=2,val=22 \
-numa dist,src=2,dst=3,val=12 \
-numa dist,src=1,dst=3,val=24 \
-m 4G -cpu cortex-a57 -kernel arch/arm64/boot/Image
w/o patch, we get lots of "groups don't span domain->span":
[ 0.802139] CPU0 attaching sched-domain(s):
[ 0.802193] domain-0: span=0-1 level=MC
[ 0.802443] groups: 0:{ span=0 cap=1013 }, 1:{ span=1 cap=979 }
[ 0.802693] domain-1: span=0-3 level=NUMA
[ 0.802731] groups: 0:{ span=0-1 cap=1992 }, 2:{ span=2-3 cap=1943 }
[ 0.802811] domain-2: span=0-5 level=NUMA
[ 0.802829] groups: 0:{ span=0-3 cap=3935 }, 4:{ span=4-7 cap=3937 }
[ 0.802881] ERROR: groups don't span domain->span
[ 0.803058] domain-3: span=0-7 level=NUMA
[ 0.803080] groups: 0:{ span=0-5 mask=0-1 cap=5843 }, 6:{ span=4-7 mask=6-7 cap=4077 }
[ 0.804055] CPU1 attaching sched-domain(s):
[ 0.804072] domain-0: span=0-1 level=MC
[ 0.804096] groups: 1:{ span=1 cap=979 }, 0:{ span=0 cap=1013 }
[ 0.804152] domain-1: span=0-3 level=NUMA
[ 0.804170] groups: 0:{ span=0-1 cap=1992 }, 2:{ span=2-3 cap=1943 }
[ 0.804219] domain-2: span=0-5 level=NUMA
[ 0.804236] groups: 0:{ span=0-3 cap=3935 }, 4:{ span=4-7 cap=3937 }
[ 0.804302] ERROR: groups don't span domain->span
[ 0.804520] domain-3: span=0-7 level=NUMA
[ 0.804546] groups: 0:{ span=0-5 mask=0-1 cap=5843 }, 6:{ span=4-7 mask=6-7 cap=4077 }
[ 0.804677] CPU2 attaching sched-domain(s):
[ 0.804687] domain-0: span=2-3 level=MC
[ 0.804705] groups: 2:{ span=2 cap=934 }, 3:{ span=3 cap=1009 }
[ 0.804754] domain-1: span=0-3 level=NUMA
[ 0.804772] groups: 2:{ span=2-3 cap=1943 }, 0:{ span=0-1 cap=1992 }
[ 0.804820] domain-2: span=0-5 level=NUMA
[ 0.804836] groups: 2:{ span=0-3 mask=2-3 cap=3991 }, 4:{ span=0-1,4-7 mask=4-5 cap=5985 }
[ 0.804944] ERROR: groups don't span domain->span
[ 0.805108] domain-3: span=0-7 level=NUMA
[ 0.805134] groups: 2:{ span=0-5 mask=2-3 cap=5899 }, 6:{ span=0-1,4-7 mask=6-7 cap=6125 }
[ 0.805223] CPU3 attaching sched-domain(s):
[ 0.805232] domain-0: span=2-3 level=MC
[ 0.805249] groups: 3:{ span=3 cap=1009 }, 2:{ span=2 cap=934 }
[ 0.805319] domain-1: span=0-3 level=NUMA
[ 0.805336] groups: 2:{ span=2-3 cap=1943 }, 0:{ span=0-1 cap=1992 }
[ 0.805383] domain-2: span=0-5 level=NUMA
[ 0.805399] groups: 2:{ span=0-3 mask=2-3 cap=3991 }, 4:{ span=0-1,4-7 mask=4-5 cap=5985 }
[ 0.805458] ERROR: groups don't span domain->span
[ 0.805605] domain-3: span=0-7 level=NUMA
[ 0.805626] groups: 2:{ span=0-5 mask=2-3 cap=5899 }, 6:{ span=0-1,4-7 mask=6-7 cap=6125 }
[ 0.805712] CPU4 attaching sched-domain(s):
[ 0.805721] domain-0: span=4-5 level=MC
[ 0.805738] groups: 4:{ span=4 cap=984 }, 5:{ span=5 cap=924 }
[ 0.805787] domain-1: span=4-7 level=NUMA
[ 0.805803] groups: 4:{ span=4-5 cap=1908 }, 6:{ span=6-7 cap=2029 }
[ 0.805851] domain-2: span=0-1,4-7 level=NUMA
[ 0.805867] groups: 4:{ span=4-7 cap=3937 }, 0:{ span=0-3 cap=3935 }
[ 0.805915] ERROR: groups don't span domain->span
[ 0.806108] domain-3: span=0-7 level=NUMA
[ 0.806130] groups: 4:{ span=0-1,4-7 mask=4-5 cap=5985 }, 2:{ span=0-3 mask=2-3 cap=3991 }
[ 0.806214] CPU5 attaching sched-domain(s):
[ 0.806222] domain-0: span=4-5 level=MC
[ 0.806240] groups: 5:{ span=5 cap=924 }, 4:{ span=4 cap=984 }
[ 0.806841] domain-1: span=4-7 level=NUMA
[ 0.806866] groups: 4:{ span=4-5 cap=1908 }, 6:{ span=6-7 cap=2029 }
[ 0.806934] domain-2: span=0-1,4-7 level=NUMA
[ 0.806953] groups: 4:{ span=4-7 cap=3937 }, 0:{ span=0-3 cap=3935 }
[ 0.807004] ERROR: groups don't span domain->span
[ 0.807312] domain-3: span=0-7 level=NUMA
[ 0.807386] groups: 4:{ span=0-1,4-7 mask=4-5 cap=5985 }, 2:{ span=0-3 mask=2-3 cap=3991 }
[ 0.807686] CPU6 attaching sched-domain(s):
[ 0.807710] domain-0: span=6-7 level=MC
[ 0.807750] groups: 6:{ span=6 cap=1017 }, 7:{ span=7 cap=1012 }
[ 0.807840] domain-1: span=4-7 level=NUMA
[ 0.807870] groups: 6:{ span=6-7 cap=2029 }, 4:{ span=4-5 cap=1908 }
[ 0.807952] domain-2: span=0-1,4-7 level=NUMA
[ 0.807985] groups: 6:{ span=4-7 mask=6-7 cap=4077 }, 0:{ span=0-5 mask=0-1 cap=5843 }
[ 0.808045] ERROR: groups don't span domain->span
[ 0.808257] domain-3: span=0-7 level=NUMA
[ 0.808571] groups: 6:{ span=0-1,4-7 mask=6-7 cap=6125 }, 2:{ span=0-5 mask=2-3 cap=5899 }
[ 0.808848] CPU7 attaching sched-domain(s):
[ 0.808860] domain-0: span=6-7 level=MC
[ 0.808880] groups: 7:{ span=7 cap=1012 }, 6:{ span=6 cap=1017 }
[ 0.808953] domain-1: span=4-7 level=NUMA
[ 0.808974] groups: 6:{ span=6-7 cap=2029 }, 4:{ span=4-5 cap=1908 }
[ 0.809034] domain-2: span=0-1,4-7 level=NUMA
[ 0.809055] groups: 6:{ span=4-7 mask=6-7 cap=4077 }, 0:{ span=0-5 mask=0-1 cap=5843 }
[ 0.809128] ERROR: groups don't span domain->span
[ 0.810361] domain-3: span=0-7 level=NUMA
[ 0.810400] groups: 6:{ span=0-1,4-7 mask=6-7 cap=5961 }, 2:{ span=0-5 mask=2-3 cap=5903 }
w/ patch, we don't get "groups don't span domain->span" any more:
[ 1.486271] CPU0 attaching sched-domain(s):
[ 1.486820] domain-0: span=0-1 level=MC
[ 1.500924] groups: 0:{ span=0 cap=980 }, 1:{ span=1 cap=994 }
[ 1.515717] domain-1: span=0-3 level=NUMA
[ 1.515903] groups: 0:{ span=0-1 cap=1974 }, 2:{ span=2-3 cap=1989 }
[ 1.516989] domain-2: span=0-5 level=NUMA
[ 1.517124] groups: 0:{ span=0-3 cap=3963 }, 4:{ span=4-5 cap=1949 }
[ 1.517369] domain-3: span=0-7 level=NUMA
[ 1.517423] groups: 0:{ span=0-5 mask=0-1 cap=5912 }, 6:{ span=4-7 mask=6-7 cap=4054 }
[ 1.520027] CPU1 attaching sched-domain(s):
[ 1.520097] domain-0: span=0-1 level=MC
[ 1.520184] groups: 1:{ span=1 cap=994 }, 0:{ span=0 cap=980 }
[ 1.520429] domain-1: span=0-3 level=NUMA
[ 1.520487] groups: 0:{ span=0-1 cap=1974 }, 2:{ span=2-3 cap=1989 }
[ 1.520687] domain-2: span=0-5 level=NUMA
[ 1.520744] groups: 0:{ span=0-3 cap=3963 }, 4:{ span=4-5 cap=1949 }
[ 1.520948] domain-3: span=0-7 level=NUMA
[ 1.521038] groups: 0:{ span=0-5 mask=0-1 cap=5912 }, 6:{ span=4-7 mask=6-7 cap=4054 }
[ 1.522068] CPU2 attaching sched-domain(s):
[ 1.522348] domain-0: span=2-3 level=MC
[ 1.522606] groups: 2:{ span=2 cap=1003 }, 3:{ span=3 cap=986 }
[ 1.522832] domain-1: span=0-3 level=NUMA
[ 1.522885] groups: 2:{ span=2-3 cap=1989 }, 0:{ span=0-1 cap=1974 }
[ 1.523043] domain-2: span=0-5 level=NUMA
[ 1.523092] groups: 2:{ span=0-3 mask=2-3 cap=4037 }, 4:{ span=4-5 cap=1949 }
[ 1.523302] domain-3: span=0-7 level=NUMA
[ 1.523352] groups: 2:{ span=0-5 mask=2-3 cap=5986 }, 6:{ span=0-1,4-7 mask=6-7 cap=6102 }
[ 1.523748] CPU3 attaching sched-domain(s):
[ 1.523774] domain-0: span=2-3 level=MC
[ 1.523825] groups: 3:{ span=3 cap=986 }, 2:{ span=2 cap=1003 }
[ 1.524009] domain-1: span=0-3 level=NUMA
[ 1.524086] groups: 2:{ span=2-3 cap=1989 }, 0:{ span=0-1 cap=1974 }
[ 1.524281] domain-2: span=0-5 level=NUMA
[ 1.524331] groups: 2:{ span=0-3 mask=2-3 cap=4037 }, 4:{ span=4-5 cap=1949 }
[ 1.524534] domain-3: span=0-7 level=NUMA
[ 1.524586] groups: 2:{ span=0-5 mask=2-3 cap=5986 }, 6:{ span=0-1,4-7 mask=6-7 cap=6102 }
[ 1.524847] CPU4 attaching sched-domain(s):
[ 1.524873] domain-0: span=4-5 level=MC
[ 1.524954] groups: 4:{ span=4 cap=958 }, 5:{ span=5 cap=991 }
[ 1.525105] domain-1: span=4-7 level=NUMA
[ 1.525153] groups: 4:{ span=4-5 cap=1949 }, 6:{ span=6-7 cap=2006 }
[ 1.525368] domain-2: span=0-1,4-7 level=NUMA
[ 1.525428] groups: 4:{ span=4-7 cap=3955 }, 0:{ span=0-1 cap=1974 }
[ 1.532726] domain-3: span=0-7 level=NUMA
[ 1.532811] groups: 4:{ span=0-1,4-7 mask=4-5 cap=6003 }, 2:{ span=0-3 mask=2-3 cap=4037 }
[ 1.534125] CPU5 attaching sched-domain(s):
[ 1.534159] domain-0: span=4-5 level=MC
[ 1.534303] groups: 5:{ span=5 cap=991 }, 4:{ span=4 cap=958 }
[ 1.534490] domain-1: span=4-7 level=NUMA
[ 1.534572] groups: 4:{ span=4-5 cap=1949 }, 6:{ span=6-7 cap=2006 }
[ 1.534734] domain-2: span=0-1,4-7 level=NUMA
[ 1.534783] groups: 4:{ span=4-7 cap=3955 }, 0:{ span=0-1 cap=1974 }
[ 1.536057] domain-3: span=0-7 level=NUMA
[ 1.536430] groups: 4:{ span=0-1,4-7 mask=4-5 cap=6003 }, 2:{ span=0-3 mask=2-3 cap=3896 }
[ 1.536815] CPU6 attaching sched-domain(s):
[ 1.536846] domain-0: span=6-7 level=MC
[ 1.536934] groups: 6:{ span=6 cap=1005 }, 7:{ span=7 cap=1001 }
[ 1.537144] domain-1: span=4-7 level=NUMA
[ 1.537262] groups: 6:{ span=6-7 cap=2006 }, 4:{ span=4-5 cap=1949 }
[ 1.537553] domain-2: span=0-1,4-7 level=NUMA
[ 1.537613] groups: 6:{ span=4-7 mask=6-7 cap=4054 }, 0:{ span=0-1 cap=1805 }
[ 1.537872] domain-3: span=0-7 level=NUMA
[ 1.537998] groups: 6:{ span=0-1,4-7 mask=6-7 cap=6102 }, 2:{ span=0-5 mask=2-3 cap=5845 }
[ 1.538448] CPU7 attaching sched-domain(s):
[ 1.538505] domain-0: span=6-7 level=MC
[ 1.538586] groups: 7:{ span=7 cap=1001 }, 6:{ span=6 cap=1005 }
[ 1.538746] domain-1: span=4-7 level=NUMA
[ 1.538798] groups: 6:{ span=6-7 cap=2006 }, 4:{ span=4-5 cap=1949 }
[ 1.539048] domain-2: span=0-1,4-7 level=NUMA
[ 1.539111] groups: 6:{ span=4-7 mask=6-7 cap=4054 }, 0:{ span=0-1 cap=1805 }
[ 1.539571] domain-3: span=0-7 level=NUMA
[ 1.539610] groups: 6:{ span=0-1,4-7 mask=6-7 cap=6102 }, 2:{ span=0-5 mask=2-3 cap=5845 }
Signed-off-by: Barry Song <song.bao.hua@hisilicon.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Reviewed-by: Valentin Schneider <valentin.schneider@arm.com>
Tested-by: Meelis Roos <mroos@linux.ee>
Link: https://lkml.kernel.org/r/20210224030944.15232-1-song.bao.hua@hisilicon.com
2021-02-24 03:09:44 +00:00
|
|
|
init_overlap_sched_group(sibling, sg);
|
2017-02-01 12:10:18 +00:00
|
|
|
|
|
|
|
|
if (!first)
|
|
|
|
|
first = sg;
|
|
|
|
|
if (last)
|
|
|
|
|
last->next = sg;
|
|
|
|
|
last = sg;
|
|
|
|
|
last->next = first;
|
|
|
|
|
}
|
2017-04-14 15:32:07 +00:00
|
|
|
sd->groups = first;
|
2017-02-01 12:10:18 +00:00
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
|
|
|
|
|
|
fail:
|
|
|
|
|
free_sched_groups(first, 0);
|
|
|
|
|
|
|
|
|
|
return -ENOMEM;
|
|
|
|
|
}
|
|
|
|
|
|
2017-04-28 08:54:26 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Package topology (also see the load-balance blurb in fair.c)
|
|
|
|
|
*
|
|
|
|
|
* The scheduler builds a tree structure to represent a number of important
|
|
|
|
|
* topology features. By default (default_topology[]) these include:
|
|
|
|
|
*
|
|
|
|
|
* - Simultaneous multithreading (SMT)
|
|
|
|
|
* - Multi-Core Cache (MC)
|
|
|
|
|
* - Package (DIE)
|
|
|
|
|
*
|
|
|
|
|
* Where the last one more or less denotes everything up to a NUMA node.
|
|
|
|
|
*
|
|
|
|
|
* The tree consists of 3 primary data structures:
|
|
|
|
|
*
|
|
|
|
|
* sched_domain -> sched_group -> sched_group_capacity
|
|
|
|
|
* ^ ^ ^ ^
|
|
|
|
|
* `-' `-'
|
|
|
|
|
*
|
sched: Clean up and harmonize the coding style of the scheduler code base
A good number of small style inconsistencies have accumulated
in the scheduler core, so do a pass over them to harmonize
all these details:
- fix speling in comments,
- use curly braces for multi-line statements,
- remove unnecessary parentheses from integer literals,
- capitalize consistently,
- remove stray newlines,
- add comments where necessary,
- remove invalid/unnecessary comments,
- align structure definitions and other data types vertically,
- add missing newlines for increased readability,
- fix vertical tabulation where it's misaligned,
- harmonize preprocessor conditional block labeling
and vertical alignment,
- remove line-breaks where they uglify the code,
- add newline after local variable definitions,
No change in functionality:
md5:
1191fa0a890cfa8132156d2959d7e9e2 built-in.o.before.asm
1191fa0a890cfa8132156d2959d7e9e2 built-in.o.after.asm
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-03-03 13:01:12 +00:00
|
|
|
* The sched_domains are per-CPU and have a two way link (parent & child) and
|
2017-04-28 08:54:26 +00:00
|
|
|
* denote the ever growing mask of CPUs belonging to that level of topology.
|
|
|
|
|
*
|
|
|
|
|
* Each sched_domain has a circular (double) linked list of sched_group's, each
|
|
|
|
|
* denoting the domains of the level below (or individual CPUs in case of the
|
|
|
|
|
* first domain level). The sched_group linked by a sched_domain includes the
|
|
|
|
|
* CPU of that sched_domain [*].
|
|
|
|
|
*
|
|
|
|
|
* Take for instance a 2 threaded, 2 core, 2 cache cluster part:
|
|
|
|
|
*
|
|
|
|
|
* CPU 0 1 2 3 4 5 6 7
|
|
|
|
|
*
|
|
|
|
|
* DIE [ ]
|
|
|
|
|
* MC [ ] [ ]
|
|
|
|
|
* SMT [ ] [ ] [ ] [ ]
|
|
|
|
|
*
|
|
|
|
|
* - or -
|
|
|
|
|
*
|
|
|
|
|
* DIE 0-7 0-7 0-7 0-7 0-7 0-7 0-7 0-7
|
|
|
|
|
* MC 0-3 0-3 0-3 0-3 4-7 4-7 4-7 4-7
|
|
|
|
|
* SMT 0-1 0-1 2-3 2-3 4-5 4-5 6-7 6-7
|
|
|
|
|
*
|
|
|
|
|
* CPU 0 1 2 3 4 5 6 7
|
|
|
|
|
*
|
|
|
|
|
* One way to think about it is: sched_domain moves you up and down among these
|
|
|
|
|
* topology levels, while sched_group moves you sideways through it, at child
|
|
|
|
|
* domain granularity.
|
|
|
|
|
*
|
|
|
|
|
* sched_group_capacity ensures each unique sched_group has shared storage.
|
|
|
|
|
*
|
|
|
|
|
* There are two related construction problems, both require a CPU that
|
|
|
|
|
* uniquely identify each group (for a given domain):
|
|
|
|
|
*
|
|
|
|
|
* - The first is the balance_cpu (see should_we_balance() and the
|
|
|
|
|
* load-balance blub in fair.c); for each group we only want 1 CPU to
|
|
|
|
|
* continue balancing at a higher domain.
|
|
|
|
|
*
|
|
|
|
|
* - The second is the sched_group_capacity; we want all identical groups
|
|
|
|
|
* to share a single sched_group_capacity.
|
|
|
|
|
*
|
|
|
|
|
* Since these topologies are exclusive by construction. That is, its
|
|
|
|
|
* impossible for an SMT thread to belong to multiple cores, and cores to
|
|
|
|
|
* be part of multiple caches. There is a very clear and unique location
|
|
|
|
|
* for each CPU in the hierarchy.
|
|
|
|
|
*
|
|
|
|
|
* Therefore computing a unique CPU for each group is trivial (the iteration
|
|
|
|
|
* mask is redundant and set all 1s; all CPUs in a group will end up at _that_
|
|
|
|
|
* group), we can simply pick the first CPU in each group.
|
|
|
|
|
*
|
|
|
|
|
*
|
|
|
|
|
* [*] in other words, the first group of each domain is its child domain.
|
|
|
|
|
*/
|
|
|
|
|
|
2017-05-03 12:18:06 +00:00
|
|
|
static struct sched_group *get_group(int cpu, struct sd_data *sdd)
|
2017-02-01 12:10:18 +00:00
|
|
|
{
|
|
|
|
|
struct sched_domain *sd = *per_cpu_ptr(sdd->sd, cpu);
|
|
|
|
|
struct sched_domain *child = sd->child;
|
2017-05-03 12:18:06 +00:00
|
|
|
struct sched_group *sg;
|
sched/topology: Skip duplicate group rewrites in build_sched_groups()
While staring at build_sched_domains(), I realized that get_group()
does several duplicate (thus useless) writes.
If you take the Arm Juno r0 (LITTLEs = [0, 3, 4, 5], bigs = [1, 2]), the
sched_group build flow would look like this:
('MC[cpu]->sg' means 'per_cpu_ptr(&tl->data->sg, cpu)' with 'tl == MC')
build_sched_groups(MC[CPU0]->sd, CPU0)
get_group(0) -> MC[CPU0]->sg
get_group(3) -> MC[CPU3]->sg
get_group(4) -> MC[CPU4]->sg
get_group(5) -> MC[CPU5]->sg
build_sched_groups(DIE[CPU0]->sd, CPU0)
get_group(0) -> DIE[CPU0]->sg
get_group(1) -> DIE[CPU1]->sg <=================+
|
build_sched_groups(MC[CPU1]->sd, CPU1) |
get_group(1) -> MC[CPU1]->sg |
get_group(2) -> MC[CPU2]->sg |
|
build_sched_groups(DIE[CPU1]->sd, CPU1) ^
get_group(1) -> DIE[CPU1]->sg } We've set up these two up here!
get_group(3) -> DIE[CPU0]->sg }
From this point on, we will only use sched_groups that have been
previously visited & initialized. The only new operation will
be which group pointer we affect to sd->groups.
On the Juno r0 we get 32 get_group() calls, every single one of them
writing to a sched_group->cpumask. However, all of the data structures
we need are set up after 8 visits (see above).
Return early from get_group() if we've already visited (and thus
initialized) the sched_group we're looking at. Overlapping domains
are not affected as they do not use build_sched_groups().
Tested on a Juno and a 2 * (Xeon E5-2690) system.
( FWIW I initially checked the refs for both sg && sg->sgc, but figured if
they weren't both 0 or > 1 then something must have gone wrong, so I
threw in a WARN_ON(). )
No change in functionality intended.
Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2019-04-09 17:35:45 +00:00
|
|
|
bool already_visited;
|
2017-02-01 12:10:18 +00:00
|
|
|
|
|
|
|
|
if (child)
|
|
|
|
|
cpu = cpumask_first(sched_domain_span(child));
|
|
|
|
|
|
2017-05-03 12:18:06 +00:00
|
|
|
sg = *per_cpu_ptr(sdd->sg, cpu);
|
|
|
|
|
sg->sgc = *per_cpu_ptr(sdd->sgc, cpu);
|
|
|
|
|
|
sched/topology: Skip duplicate group rewrites in build_sched_groups()
While staring at build_sched_domains(), I realized that get_group()
does several duplicate (thus useless) writes.
If you take the Arm Juno r0 (LITTLEs = [0, 3, 4, 5], bigs = [1, 2]), the
sched_group build flow would look like this:
('MC[cpu]->sg' means 'per_cpu_ptr(&tl->data->sg, cpu)' with 'tl == MC')
build_sched_groups(MC[CPU0]->sd, CPU0)
get_group(0) -> MC[CPU0]->sg
get_group(3) -> MC[CPU3]->sg
get_group(4) -> MC[CPU4]->sg
get_group(5) -> MC[CPU5]->sg
build_sched_groups(DIE[CPU0]->sd, CPU0)
get_group(0) -> DIE[CPU0]->sg
get_group(1) -> DIE[CPU1]->sg <=================+
|
build_sched_groups(MC[CPU1]->sd, CPU1) |
get_group(1) -> MC[CPU1]->sg |
get_group(2) -> MC[CPU2]->sg |
|
build_sched_groups(DIE[CPU1]->sd, CPU1) ^
get_group(1) -> DIE[CPU1]->sg } We've set up these two up here!
get_group(3) -> DIE[CPU0]->sg }
From this point on, we will only use sched_groups that have been
previously visited & initialized. The only new operation will
be which group pointer we affect to sd->groups.
On the Juno r0 we get 32 get_group() calls, every single one of them
writing to a sched_group->cpumask. However, all of the data structures
we need are set up after 8 visits (see above).
Return early from get_group() if we've already visited (and thus
initialized) the sched_group we're looking at. Overlapping domains
are not affected as they do not use build_sched_groups().
Tested on a Juno and a 2 * (Xeon E5-2690) system.
( FWIW I initially checked the refs for both sg && sg->sgc, but figured if
they weren't both 0 or > 1 then something must have gone wrong, so I
threw in a WARN_ON(). )
No change in functionality intended.
Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2019-04-09 17:35:45 +00:00
|
|
|
/* Increase refcounts for claim_allocations: */
|
|
|
|
|
already_visited = atomic_inc_return(&sg->ref) > 1;
|
|
|
|
|
/* sgc visits should follow a similar trend as sg */
|
|
|
|
|
WARN_ON(already_visited != (atomic_inc_return(&sg->sgc->ref) > 1));
|
|
|
|
|
|
|
|
|
|
/* If we have already visited that group, it's already initialized. */
|
|
|
|
|
if (already_visited)
|
|
|
|
|
return sg;
|
2017-02-01 12:10:18 +00:00
|
|
|
|
2017-05-03 12:18:06 +00:00
|
|
|
if (child) {
|
2017-05-01 09:03:12 +00:00
|
|
|
cpumask_copy(sched_group_span(sg), sched_domain_span(child));
|
|
|
|
|
cpumask_copy(group_balance_mask(sg), sched_group_span(sg));
|
2017-05-03 12:18:06 +00:00
|
|
|
} else {
|
2017-05-01 09:03:12 +00:00
|
|
|
cpumask_set_cpu(cpu, sched_group_span(sg));
|
2017-05-01 08:47:02 +00:00
|
|
|
cpumask_set_cpu(cpu, group_balance_mask(sg));
|
2017-02-01 12:10:18 +00:00
|
|
|
}
|
|
|
|
|
|
2017-05-01 09:03:12 +00:00
|
|
|
sg->sgc->capacity = SCHED_CAPACITY_SCALE * cpumask_weight(sched_group_span(sg));
|
2017-05-03 12:18:06 +00:00
|
|
|
sg->sgc->min_capacity = SCHED_CAPACITY_SCALE;
|
2018-07-04 10:17:41 +00:00
|
|
|
sg->sgc->max_capacity = SCHED_CAPACITY_SCALE;
|
2017-05-03 12:18:06 +00:00
|
|
|
|
|
|
|
|
return sg;
|
2017-02-01 12:10:18 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* build_sched_groups will build a circular linked list of the groups
|
2019-04-09 17:35:46 +00:00
|
|
|
* covered by the given span, will set each group's ->cpumask correctly,
|
|
|
|
|
* and will initialize their ->sgc.
|
2017-02-01 12:10:18 +00:00
|
|
|
*
|
|
|
|
|
* Assumes the sched_domain tree is fully constructed
|
|
|
|
|
*/
|
|
|
|
|
static int
|
|
|
|
|
build_sched_groups(struct sched_domain *sd, int cpu)
|
|
|
|
|
{
|
|
|
|
|
struct sched_group *first = NULL, *last = NULL;
|
|
|
|
|
struct sd_data *sdd = sd->private;
|
|
|
|
|
const struct cpumask *span = sched_domain_span(sd);
|
|
|
|
|
struct cpumask *covered;
|
|
|
|
|
int i;
|
|
|
|
|
|
|
|
|
|
lockdep_assert_held(&sched_domains_mutex);
|
|
|
|
|
covered = sched_domains_tmpmask;
|
|
|
|
|
|
|
|
|
|
cpumask_clear(covered);
|
|
|
|
|
|
2017-05-03 12:18:06 +00:00
|
|
|
for_each_cpu_wrap(i, span, cpu) {
|
2017-02-01 12:10:18 +00:00
|
|
|
struct sched_group *sg;
|
|
|
|
|
|
|
|
|
|
if (cpumask_test_cpu(i, covered))
|
|
|
|
|
continue;
|
|
|
|
|
|
2017-05-03 12:18:06 +00:00
|
|
|
sg = get_group(i, sdd);
|
2017-02-01 12:10:18 +00:00
|
|
|
|
2017-05-01 09:03:12 +00:00
|
|
|
cpumask_or(covered, covered, sched_group_span(sg));
|
2017-02-01 12:10:18 +00:00
|
|
|
|
|
|
|
|
if (!first)
|
|
|
|
|
first = sg;
|
|
|
|
|
if (last)
|
|
|
|
|
last->next = sg;
|
|
|
|
|
last = sg;
|
|
|
|
|
}
|
|
|
|
|
last->next = first;
|
2017-05-03 12:18:06 +00:00
|
|
|
sd->groups = first;
|
2017-02-01 12:10:18 +00:00
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Initialize sched groups cpu_capacity.
|
|
|
|
|
*
|
|
|
|
|
* cpu_capacity indicates the capacity of sched group, which is used while
|
|
|
|
|
* distributing the load between different sched groups in a sched domain.
|
|
|
|
|
* Typically cpu_capacity for all the groups in a sched domain will be same
|
|
|
|
|
* unless there are asymmetries in the topology. If there are asymmetries,
|
|
|
|
|
* group having more cpu_capacity will pickup more load compared to the
|
|
|
|
|
* group having less cpu_capacity.
|
|
|
|
|
*/
|
|
|
|
|
static void init_sched_groups_capacity(int cpu, struct sched_domain *sd)
|
|
|
|
|
{
|
|
|
|
|
struct sched_group *sg = sd->groups;
|
|
|
|
|
|
|
|
|
|
WARN_ON(!sg);
|
|
|
|
|
|
|
|
|
|
do {
|
|
|
|
|
int cpu, max_cpu = -1;
|
|
|
|
|
|
2017-05-01 09:03:12 +00:00
|
|
|
sg->group_weight = cpumask_weight(sched_group_span(sg));
|
2017-02-01 12:10:18 +00:00
|
|
|
|
|
|
|
|
if (!(sd->flags & SD_ASYM_PACKING))
|
|
|
|
|
goto next;
|
|
|
|
|
|
2017-05-01 09:03:12 +00:00
|
|
|
for_each_cpu(cpu, sched_group_span(sg)) {
|
2017-02-01 12:10:18 +00:00
|
|
|
if (max_cpu < 0)
|
|
|
|
|
max_cpu = cpu;
|
|
|
|
|
else if (sched_asym_prefer(cpu, max_cpu))
|
|
|
|
|
max_cpu = cpu;
|
|
|
|
|
}
|
|
|
|
|
sg->asym_prefer_cpu = max_cpu;
|
|
|
|
|
|
|
|
|
|
next:
|
|
|
|
|
sg = sg->next;
|
|
|
|
|
} while (sg != sd->groups);
|
|
|
|
|
|
|
|
|
|
if (cpu != group_balance_cpu(sg))
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
update_group_capacity(sd, cpu);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Initializers for schedule domains
|
|
|
|
|
* Non-inlined to reduce accumulated stack pressure in build_sched_domains()
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
static int default_relax_domain_level = -1;
|
|
|
|
|
int sched_domain_level_max;
|
|
|
|
|
|
|
|
|
|
static int __init setup_relax_domain_level(char *str)
|
|
|
|
|
{
|
|
|
|
|
if (kstrtoint(str, 0, &default_relax_domain_level))
|
|
|
|
|
pr_warn("Unable to set relax_domain_level\n");
|
|
|
|
|
|
|
|
|
|
return 1;
|
|
|
|
|
}
|
|
|
|
|
__setup("relax_domain_level=", setup_relax_domain_level);
|
|
|
|
|
|
|
|
|
|
static void set_domain_attribute(struct sched_domain *sd,
|
|
|
|
|
struct sched_domain_attr *attr)
|
|
|
|
|
{
|
|
|
|
|
int request;
|
|
|
|
|
|
|
|
|
|
if (!attr || attr->relax_domain_level < 0) {
|
|
|
|
|
if (default_relax_domain_level < 0)
|
|
|
|
|
return;
|
2019-10-14 16:44:08 +00:00
|
|
|
request = default_relax_domain_level;
|
2017-02-01 12:10:18 +00:00
|
|
|
} else
|
|
|
|
|
request = attr->relax_domain_level;
|
2019-10-14 16:44:08 +00:00
|
|
|
|
|
|
|
|
if (sd->level > request) {
|
2017-02-01 12:10:18 +00:00
|
|
|
/* Turn off idle balance on this domain: */
|
|
|
|
|
sd->flags &= ~(SD_BALANCE_WAKE|SD_BALANCE_NEWIDLE);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void __sdt_free(const struct cpumask *cpu_map);
|
|
|
|
|
static int __sdt_alloc(const struct cpumask *cpu_map);
|
|
|
|
|
|
|
|
|
|
static void __free_domain_allocs(struct s_data *d, enum s_alloc what,
|
|
|
|
|
const struct cpumask *cpu_map)
|
|
|
|
|
{
|
|
|
|
|
switch (what) {
|
|
|
|
|
case sa_rootdomain:
|
|
|
|
|
if (!atomic_read(&d->rd->refcount))
|
|
|
|
|
free_rootdomain(&d->rd->rcu);
|
2020-08-23 22:36:59 +00:00
|
|
|
fallthrough;
|
2017-02-01 12:10:18 +00:00
|
|
|
case sa_sd:
|
|
|
|
|
free_percpu(d->sd);
|
2020-08-23 22:36:59 +00:00
|
|
|
fallthrough;
|
2017-02-01 12:10:18 +00:00
|
|
|
case sa_sd_storage:
|
|
|
|
|
__sdt_free(cpu_map);
|
2020-08-23 22:36:59 +00:00
|
|
|
fallthrough;
|
2017-02-01 12:10:18 +00:00
|
|
|
case sa_none:
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static enum s_alloc
|
|
|
|
|
__visit_domain_allocation_hell(struct s_data *d, const struct cpumask *cpu_map)
|
|
|
|
|
{
|
|
|
|
|
memset(d, 0, sizeof(*d));
|
|
|
|
|
|
|
|
|
|
if (__sdt_alloc(cpu_map))
|
|
|
|
|
return sa_sd_storage;
|
|
|
|
|
d->sd = alloc_percpu(struct sched_domain *);
|
|
|
|
|
if (!d->sd)
|
|
|
|
|
return sa_sd_storage;
|
|
|
|
|
d->rd = alloc_rootdomain();
|
|
|
|
|
if (!d->rd)
|
|
|
|
|
return sa_sd;
|
sched: Clean up and harmonize the coding style of the scheduler code base
A good number of small style inconsistencies have accumulated
in the scheduler core, so do a pass over them to harmonize
all these details:
- fix speling in comments,
- use curly braces for multi-line statements,
- remove unnecessary parentheses from integer literals,
- capitalize consistently,
- remove stray newlines,
- add comments where necessary,
- remove invalid/unnecessary comments,
- align structure definitions and other data types vertically,
- add missing newlines for increased readability,
- fix vertical tabulation where it's misaligned,
- harmonize preprocessor conditional block labeling
and vertical alignment,
- remove line-breaks where they uglify the code,
- add newline after local variable definitions,
No change in functionality:
md5:
1191fa0a890cfa8132156d2959d7e9e2 built-in.o.before.asm
1191fa0a890cfa8132156d2959d7e9e2 built-in.o.after.asm
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-03-03 13:01:12 +00:00
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
return sa_rootdomain;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* NULL the sd_data elements we've used to build the sched_domain and
|
|
|
|
|
* sched_group structure so that the subsequent __free_domain_allocs()
|
|
|
|
|
* will not free the data we're using.
|
|
|
|
|
*/
|
|
|
|
|
static void claim_allocations(int cpu, struct sched_domain *sd)
|
|
|
|
|
{
|
|
|
|
|
struct sd_data *sdd = sd->private;
|
|
|
|
|
|
|
|
|
|
WARN_ON_ONCE(*per_cpu_ptr(sdd->sd, cpu) != sd);
|
|
|
|
|
*per_cpu_ptr(sdd->sd, cpu) = NULL;
|
|
|
|
|
|
|
|
|
|
if (atomic_read(&(*per_cpu_ptr(sdd->sds, cpu))->ref))
|
|
|
|
|
*per_cpu_ptr(sdd->sds, cpu) = NULL;
|
|
|
|
|
|
|
|
|
|
if (atomic_read(&(*per_cpu_ptr(sdd->sg, cpu))->ref))
|
|
|
|
|
*per_cpu_ptr(sdd->sg, cpu) = NULL;
|
|
|
|
|
|
|
|
|
|
if (atomic_read(&(*per_cpu_ptr(sdd->sgc, cpu))->ref))
|
|
|
|
|
*per_cpu_ptr(sdd->sgc, cpu) = NULL;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#ifdef CONFIG_NUMA
|
|
|
|
|
enum numa_topology_type sched_numa_topology_type;
|
sched: Clean up and harmonize the coding style of the scheduler code base
A good number of small style inconsistencies have accumulated
in the scheduler core, so do a pass over them to harmonize
all these details:
- fix speling in comments,
- use curly braces for multi-line statements,
- remove unnecessary parentheses from integer literals,
- capitalize consistently,
- remove stray newlines,
- add comments where necessary,
- remove invalid/unnecessary comments,
- align structure definitions and other data types vertically,
- add missing newlines for increased readability,
- fix vertical tabulation where it's misaligned,
- harmonize preprocessor conditional block labeling
and vertical alignment,
- remove line-breaks where they uglify the code,
- add newline after local variable definitions,
No change in functionality:
md5:
1191fa0a890cfa8132156d2959d7e9e2 built-in.o.before.asm
1191fa0a890cfa8132156d2959d7e9e2 built-in.o.after.asm
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-03-03 13:01:12 +00:00
|
|
|
|
|
|
|
|
static int sched_domains_numa_levels;
|
|
|
|
|
static int sched_domains_curr_level;
|
|
|
|
|
|
|
|
|
|
int sched_max_numa_distance;
|
|
|
|
|
static int *sched_domains_numa_distance;
|
|
|
|
|
static struct cpumask ***sched_domains_numa_masks;
|
2019-08-08 19:53:01 +00:00
|
|
|
int __read_mostly node_reclaim_distance = RECLAIM_DISTANCE;
|
2017-02-01 12:10:18 +00:00
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* SD_flags allowed in topology descriptions.
|
|
|
|
|
*
|
|
|
|
|
* These flags are purely descriptive of the topology and do not prescribe
|
|
|
|
|
* behaviour. Behaviour is artificial and mapped in the below sd_init()
|
|
|
|
|
* function:
|
|
|
|
|
*
|
|
|
|
|
* SD_SHARE_CPUCAPACITY - describes SMT topologies
|
|
|
|
|
* SD_SHARE_PKG_RESOURCES - describes shared caches
|
|
|
|
|
* SD_NUMA - describes NUMA topologies
|
|
|
|
|
*
|
|
|
|
|
* Odd one out, which beside describing the topology has a quirk also
|
|
|
|
|
* prescribes the desired behaviour that goes along with it:
|
|
|
|
|
*
|
|
|
|
|
* SD_ASYM_PACKING - describes SMT quirks
|
|
|
|
|
*/
|
|
|
|
|
#define TOPOLOGY_SD_FLAGS \
|
sched: Clean up and harmonize the coding style of the scheduler code base
A good number of small style inconsistencies have accumulated
in the scheduler core, so do a pass over them to harmonize
all these details:
- fix speling in comments,
- use curly braces for multi-line statements,
- remove unnecessary parentheses from integer literals,
- capitalize consistently,
- remove stray newlines,
- add comments where necessary,
- remove invalid/unnecessary comments,
- align structure definitions and other data types vertically,
- add missing newlines for increased readability,
- fix vertical tabulation where it's misaligned,
- harmonize preprocessor conditional block labeling
and vertical alignment,
- remove line-breaks where they uglify the code,
- add newline after local variable definitions,
No change in functionality:
md5:
1191fa0a890cfa8132156d2959d7e9e2 built-in.o.before.asm
1191fa0a890cfa8132156d2959d7e9e2 built-in.o.after.asm
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-03-03 13:01:12 +00:00
|
|
|
(SD_SHARE_CPUCAPACITY | \
|
2017-02-01 12:10:18 +00:00
|
|
|
SD_SHARE_PKG_RESOURCES | \
|
sched: Clean up and harmonize the coding style of the scheduler code base
A good number of small style inconsistencies have accumulated
in the scheduler core, so do a pass over them to harmonize
all these details:
- fix speling in comments,
- use curly braces for multi-line statements,
- remove unnecessary parentheses from integer literals,
- capitalize consistently,
- remove stray newlines,
- add comments where necessary,
- remove invalid/unnecessary comments,
- align structure definitions and other data types vertically,
- add missing newlines for increased readability,
- fix vertical tabulation where it's misaligned,
- harmonize preprocessor conditional block labeling
and vertical alignment,
- remove line-breaks where they uglify the code,
- add newline after local variable definitions,
No change in functionality:
md5:
1191fa0a890cfa8132156d2959d7e9e2 built-in.o.before.asm
1191fa0a890cfa8132156d2959d7e9e2 built-in.o.after.asm
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-03-03 13:01:12 +00:00
|
|
|
SD_NUMA | \
|
2020-08-17 11:29:47 +00:00
|
|
|
SD_ASYM_PACKING)
|
2017-02-01 12:10:18 +00:00
|
|
|
|
|
|
|
|
static struct sched_domain *
|
|
|
|
|
sd_init(struct sched_domain_topology_level *tl,
|
|
|
|
|
const struct cpumask *cpu_map,
|
2018-07-20 13:32:31 +00:00
|
|
|
struct sched_domain *child, int dflags, int cpu)
|
2017-02-01 12:10:18 +00:00
|
|
|
{
|
|
|
|
|
struct sd_data *sdd = &tl->data;
|
|
|
|
|
struct sched_domain *sd = *per_cpu_ptr(sdd->sd, cpu);
|
|
|
|
|
int sd_id, sd_weight, sd_flags = 0;
|
|
|
|
|
|
|
|
|
|
#ifdef CONFIG_NUMA
|
|
|
|
|
/*
|
|
|
|
|
* Ugly hack to pass state to sd_numa_mask()...
|
|
|
|
|
*/
|
|
|
|
|
sched_domains_curr_level = tl->numa_level;
|
|
|
|
|
#endif
|
|
|
|
|
|
|
|
|
|
sd_weight = cpumask_weight(tl->mask(cpu));
|
|
|
|
|
|
|
|
|
|
if (tl->sd_flags)
|
|
|
|
|
sd_flags = (*tl->sd_flags)();
|
|
|
|
|
if (WARN_ONCE(sd_flags & ~TOPOLOGY_SD_FLAGS,
|
|
|
|
|
"wrong sd_flags in topology description\n"))
|
2020-06-09 15:09:36 +00:00
|
|
|
sd_flags &= TOPOLOGY_SD_FLAGS;
|
2017-02-01 12:10:18 +00:00
|
|
|
|
2018-07-20 13:32:31 +00:00
|
|
|
/* Apply detected topology flags */
|
|
|
|
|
sd_flags |= dflags;
|
|
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
*sd = (struct sched_domain){
|
|
|
|
|
.min_interval = sd_weight,
|
|
|
|
|
.max_interval = 2*sd_weight,
|
2020-09-21 07:24:24 +00:00
|
|
|
.busy_factor = 16,
|
2020-09-21 07:24:22 +00:00
|
|
|
.imbalance_pct = 117,
|
2017-02-01 12:10:18 +00:00
|
|
|
|
|
|
|
|
.cache_nice_tries = 0,
|
|
|
|
|
|
2020-04-15 21:05:07 +00:00
|
|
|
.flags = 1*SD_BALANCE_NEWIDLE
|
2017-02-01 12:10:18 +00:00
|
|
|
| 1*SD_BALANCE_EXEC
|
|
|
|
|
| 1*SD_BALANCE_FORK
|
|
|
|
|
| 0*SD_BALANCE_WAKE
|
|
|
|
|
| 1*SD_WAKE_AFFINE
|
|
|
|
|
| 0*SD_SHARE_CPUCAPACITY
|
|
|
|
|
| 0*SD_SHARE_PKG_RESOURCES
|
|
|
|
|
| 0*SD_SERIALIZE
|
2018-07-04 10:17:50 +00:00
|
|
|
| 1*SD_PREFER_SIBLING
|
2017-02-01 12:10:18 +00:00
|
|
|
| 0*SD_NUMA
|
|
|
|
|
| sd_flags
|
|
|
|
|
,
|
|
|
|
|
|
|
|
|
|
.last_balance = jiffies,
|
|
|
|
|
.balance_interval = sd_weight,
|
|
|
|
|
.max_newidle_lb_cost = 0,
|
|
|
|
|
.next_decay_max_lb_cost = jiffies,
|
|
|
|
|
.child = child,
|
|
|
|
|
#ifdef CONFIG_SCHED_DEBUG
|
|
|
|
|
.name = tl->name,
|
|
|
|
|
#endif
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
cpumask_and(sched_domain_span(sd), cpu_map, tl->mask(cpu));
|
|
|
|
|
sd_id = cpumask_first(sched_domain_span(sd));
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Convert topological properties into behaviour.
|
|
|
|
|
*/
|
|
|
|
|
|
2020-02-06 19:19:55 +00:00
|
|
|
/* Don't attempt to spread across CPUs of different capacities. */
|
|
|
|
|
if ((sd->flags & SD_ASYM_CPUCAPACITY) && sd->child)
|
|
|
|
|
sd->child->flags &= ~SD_PREFER_SIBLING;
|
2017-02-01 12:10:18 +00:00
|
|
|
|
|
|
|
|
if (sd->flags & SD_SHARE_CPUCAPACITY) {
|
|
|
|
|
sd->imbalance_pct = 110;
|
|
|
|
|
|
|
|
|
|
} else if (sd->flags & SD_SHARE_PKG_RESOURCES) {
|
|
|
|
|
sd->imbalance_pct = 117;
|
|
|
|
|
sd->cache_nice_tries = 1;
|
|
|
|
|
|
|
|
|
|
#ifdef CONFIG_NUMA
|
|
|
|
|
} else if (sd->flags & SD_NUMA) {
|
|
|
|
|
sd->cache_nice_tries = 2;
|
|
|
|
|
|
2018-07-04 10:17:50 +00:00
|
|
|
sd->flags &= ~SD_PREFER_SIBLING;
|
2017-02-01 12:10:18 +00:00
|
|
|
sd->flags |= SD_SERIALIZE;
|
2019-08-08 19:53:01 +00:00
|
|
|
if (sched_domains_numa_distance[tl->numa_level] > node_reclaim_distance) {
|
2017-02-01 12:10:18 +00:00
|
|
|
sd->flags &= ~(SD_BALANCE_EXEC |
|
|
|
|
|
SD_BALANCE_FORK |
|
|
|
|
|
SD_WAKE_AFFINE);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#endif
|
|
|
|
|
} else {
|
|
|
|
|
sd->cache_nice_tries = 1;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* For all levels sharing cache; connect a sched_domain_shared
|
|
|
|
|
* instance.
|
|
|
|
|
*/
|
|
|
|
|
if (sd->flags & SD_SHARE_PKG_RESOURCES) {
|
|
|
|
|
sd->shared = *per_cpu_ptr(sdd->sds, sd_id);
|
|
|
|
|
atomic_inc(&sd->shared->ref);
|
|
|
|
|
atomic_set(&sd->shared->nr_busy_cpus, sd_weight);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
sd->private = sdd;
|
|
|
|
|
|
|
|
|
|
return sd;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Topology list, bottom-up.
|
|
|
|
|
*/
|
|
|
|
|
static struct sched_domain_topology_level default_topology[] = {
|
|
|
|
|
#ifdef CONFIG_SCHED_SMT
|
|
|
|
|
{ cpu_smt_mask, cpu_smt_flags, SD_INIT_NAME(SMT) },
|
|
|
|
|
#endif
|
|
|
|
|
#ifdef CONFIG_SCHED_MC
|
|
|
|
|
{ cpu_coregroup_mask, cpu_core_flags, SD_INIT_NAME(MC) },
|
|
|
|
|
#endif
|
|
|
|
|
{ cpu_cpu_mask, SD_INIT_NAME(DIE) },
|
|
|
|
|
{ NULL, },
|
|
|
|
|
};
|
|
|
|
|
|
|
|
|
|
static struct sched_domain_topology_level *sched_domain_topology =
|
|
|
|
|
default_topology;
|
|
|
|
|
|
|
|
|
|
#define for_each_sd_topology(tl) \
|
|
|
|
|
for (tl = sched_domain_topology; tl->mask; tl++)
|
|
|
|
|
|
|
|
|
|
void set_sched_topology(struct sched_domain_topology_level *tl)
|
|
|
|
|
{
|
|
|
|
|
if (WARN_ON_ONCE(sched_smp_initialized))
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
sched_domain_topology = tl;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
#ifdef CONFIG_NUMA
|
|
|
|
|
|
|
|
|
|
static const struct cpumask *sd_numa_mask(int cpu)
|
|
|
|
|
{
|
|
|
|
|
return sched_domains_numa_masks[sched_domains_curr_level][cpu_to_node(cpu)];
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void sched_numa_warn(const char *str)
|
|
|
|
|
{
|
|
|
|
|
static int done = false;
|
|
|
|
|
int i,j;
|
|
|
|
|
|
|
|
|
|
if (done)
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
done = true;
|
|
|
|
|
|
|
|
|
|
printk(KERN_WARNING "ERROR: %s\n\n", str);
|
|
|
|
|
|
|
|
|
|
for (i = 0; i < nr_node_ids; i++) {
|
|
|
|
|
printk(KERN_WARNING " ");
|
|
|
|
|
for (j = 0; j < nr_node_ids; j++)
|
|
|
|
|
printk(KERN_CONT "%02d ", node_distance(i,j));
|
|
|
|
|
printk(KERN_CONT "\n");
|
|
|
|
|
}
|
|
|
|
|
printk(KERN_WARNING "\n");
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
bool find_numa_distance(int distance)
|
|
|
|
|
{
|
|
|
|
|
int i;
|
|
|
|
|
|
|
|
|
|
if (distance == node_distance(0, 0))
|
|
|
|
|
return true;
|
|
|
|
|
|
|
|
|
|
for (i = 0; i < sched_domains_numa_levels; i++) {
|
|
|
|
|
if (sched_domains_numa_distance[i] == distance)
|
|
|
|
|
return true;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* A system can have three types of NUMA topology:
|
|
|
|
|
* NUMA_DIRECT: all nodes are directly connected, or not a NUMA system
|
|
|
|
|
* NUMA_GLUELESS_MESH: some nodes reachable through intermediary nodes
|
|
|
|
|
* NUMA_BACKPLANE: nodes can reach other nodes through a backplane
|
|
|
|
|
*
|
|
|
|
|
* The difference between a glueless mesh topology and a backplane
|
|
|
|
|
* topology lies in whether communication between not directly
|
|
|
|
|
* connected nodes goes through intermediary nodes (where programs
|
|
|
|
|
* could run), or through backplane controllers. This affects
|
|
|
|
|
* placement of programs.
|
|
|
|
|
*
|
|
|
|
|
* The type of topology can be discerned with the following tests:
|
|
|
|
|
* - If the maximum distance between any nodes is 1 hop, the system
|
|
|
|
|
* is directly connected.
|
|
|
|
|
* - If for two nodes A and B, located N > 1 hops away from each other,
|
|
|
|
|
* there is an intermediary node C, which is < N hops away from both
|
|
|
|
|
* nodes A and B, the system is a glueless mesh.
|
|
|
|
|
*/
|
|
|
|
|
static void init_numa_topology_type(void)
|
|
|
|
|
{
|
|
|
|
|
int a, b, c, n;
|
|
|
|
|
|
|
|
|
|
n = sched_max_numa_distance;
|
|
|
|
|
|
2018-08-10 17:00:18 +00:00
|
|
|
if (sched_domains_numa_levels <= 2) {
|
2017-02-01 12:10:18 +00:00
|
|
|
sched_numa_topology_type = NUMA_DIRECT;
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
for_each_online_node(a) {
|
|
|
|
|
for_each_online_node(b) {
|
|
|
|
|
/* Find two nodes furthest removed from each other. */
|
|
|
|
|
if (node_distance(a, b) < n)
|
|
|
|
|
continue;
|
|
|
|
|
|
|
|
|
|
/* Is there an intermediary node between a and b? */
|
|
|
|
|
for_each_online_node(c) {
|
|
|
|
|
if (node_distance(a, c) < n &&
|
|
|
|
|
node_distance(b, c) < n) {
|
|
|
|
|
sched_numa_topology_type =
|
|
|
|
|
NUMA_GLUELESS_MESH;
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
sched_numa_topology_type = NUMA_BACKPLANE;
|
|
|
|
|
return;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2021-01-22 12:39:43 +00:00
|
|
|
|
|
|
|
|
#define NR_DISTANCE_VALUES (1 << DISTANCE_BITS)
|
|
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
void sched_init_numa(void)
|
|
|
|
|
{
|
|
|
|
|
struct sched_domain_topology_level *tl;
|
2021-01-22 12:39:43 +00:00
|
|
|
unsigned long *distance_map;
|
|
|
|
|
int nr_levels = 0;
|
|
|
|
|
int i, j;
|
sched/topology: Introduce NUMA identity node sched domain
On AMD Family17h-based (EPYC) system, a logical NUMA node can contain
upto 8 cores (16 threads) with the following topology.
----------------------------
C0 | T0 T1 | || | T0 T1 | C4
--------| || |--------
C1 | T0 T1 | L3 || L3 | T0 T1 | C5
--------| || |--------
C2 | T0 T1 | #0 || #1 | T0 T1 | C6
--------| || |--------
C3 | T0 T1 | || | T0 T1 | C7
----------------------------
Here, there are 2 last-level (L3) caches per logical NUMA node.
A socket can contain upto 4 NUMA nodes, and a system can support
upto 2 sockets. With full system configuration, current scheduler
creates 4 sched domains:
domain0 SMT (span a core)
domain1 MC (span a last-level-cache)
domain2 NUMA (span a socket: 4 nodes)
domain3 NUMA (span a system: 8 nodes)
Note that there is no domain to represent cpus spaning a logical
NUMA node. With this hierarchy of sched domains, the scheduler does
not balance properly in the following cases:
Case1:
When running 8 tasks, a properly balanced system should
schedule a task per logical NUMA node. This is not the case for
the current scheduler.
Case2:
In some cases, threads are scheduled on the same cpu, while other
cpus are idle. This results in run-to-run inconsistency. For example:
taskset -c 0-7 sysbench --num-threads=8 --test=cpu \
--cpu-max-prime=100000 run
Total execution time ranges from 25.1s to 33.5s depending on threads
placement, where 25.1s is when all 8 threads are balanced properly
on 8 cpus.
Introducing NUMA identity node sched domain, which is based on how
SRAT/SLIT table define a logical NUMA node. This results in the following
hierarchy of sched domains on the same system described above.
domain0 SMT (span a core)
domain1 MC (span a last-level-cache)
domain2 NODE (span a logical NUMA node)
domain3 NUMA (span a socket: 4 nodes)
domain4 NUMA (span a system: 8 nodes)
This fixes the improper load balancing cases mentioned above.
Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: bp@suse.de
Link: http://lkml.kernel.org/r/1504768805-46716-1-git-send-email-suravee.suthikulpanit@amd.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-09-07 07:20:05 +00:00
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
/*
|
|
|
|
|
* O(nr_nodes^2) deduplicating selection sort -- in order to find the
|
|
|
|
|
* unique distances in the node_distance() table.
|
|
|
|
|
*/
|
2021-01-22 12:39:43 +00:00
|
|
|
distance_map = bitmap_alloc(NR_DISTANCE_VALUES, GFP_KERNEL);
|
|
|
|
|
if (!distance_map)
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
bitmap_zero(distance_map, NR_DISTANCE_VALUES);
|
2017-02-01 12:10:18 +00:00
|
|
|
for (i = 0; i < nr_node_ids; i++) {
|
|
|
|
|
for (j = 0; j < nr_node_ids; j++) {
|
2021-01-22 12:39:43 +00:00
|
|
|
int distance = node_distance(i, j);
|
2017-02-01 12:10:18 +00:00
|
|
|
|
2021-01-22 12:39:43 +00:00
|
|
|
if (distance < LOCAL_DISTANCE || distance >= NR_DISTANCE_VALUES) {
|
|
|
|
|
sched_numa_warn("Invalid distance value range");
|
|
|
|
|
return;
|
2017-02-01 12:10:18 +00:00
|
|
|
}
|
2021-01-22 12:39:43 +00:00
|
|
|
|
|
|
|
|
bitmap_set(distance_map, distance, 1);
|
2017-02-01 12:10:18 +00:00
|
|
|
}
|
2021-01-22 12:39:43 +00:00
|
|
|
}
|
|
|
|
|
/*
|
|
|
|
|
* We can now figure out how many unique distance values there are and
|
|
|
|
|
* allocate memory accordingly.
|
|
|
|
|
*/
|
|
|
|
|
nr_levels = bitmap_weight(distance_map, NR_DISTANCE_VALUES);
|
2017-02-01 12:10:18 +00:00
|
|
|
|
2021-01-22 12:39:43 +00:00
|
|
|
sched_domains_numa_distance = kcalloc(nr_levels, sizeof(int), GFP_KERNEL);
|
|
|
|
|
if (!sched_domains_numa_distance) {
|
|
|
|
|
bitmap_free(distance_map);
|
|
|
|
|
return;
|
2017-02-01 12:10:18 +00:00
|
|
|
}
|
|
|
|
|
|
2021-01-22 12:39:43 +00:00
|
|
|
for (i = 0, j = 0; i < nr_levels; i++, j++) {
|
|
|
|
|
j = find_next_bit(distance_map, NR_DISTANCE_VALUES, j);
|
|
|
|
|
sched_domains_numa_distance[i] = j;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
bitmap_free(distance_map);
|
|
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
/*
|
2021-01-22 12:39:43 +00:00
|
|
|
* 'nr_levels' contains the number of unique distances
|
2017-02-01 12:10:18 +00:00
|
|
|
*
|
|
|
|
|
* The sched_domains_numa_distance[] array includes the actual distance
|
|
|
|
|
* numbers.
|
|
|
|
|
*/
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Here, we should temporarily reset sched_domains_numa_levels to 0.
|
|
|
|
|
* If it fails to allocate memory for array sched_domains_numa_masks[][],
|
2021-01-22 12:39:43 +00:00
|
|
|
* the array will contain less then 'nr_levels' members. This could be
|
2017-02-01 12:10:18 +00:00
|
|
|
* dangerous when we use it to iterate array sched_domains_numa_masks[][]
|
|
|
|
|
* in other functions.
|
|
|
|
|
*
|
2021-01-22 12:39:43 +00:00
|
|
|
* We reset it to 'nr_levels' at the end of this function.
|
2017-02-01 12:10:18 +00:00
|
|
|
*/
|
|
|
|
|
sched_domains_numa_levels = 0;
|
|
|
|
|
|
2021-01-22 12:39:43 +00:00
|
|
|
sched_domains_numa_masks = kzalloc(sizeof(void *) * nr_levels, GFP_KERNEL);
|
2017-02-01 12:10:18 +00:00
|
|
|
if (!sched_domains_numa_masks)
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Now for each level, construct a mask per node which contains all
|
|
|
|
|
* CPUs of nodes that are that many hops away from us.
|
|
|
|
|
*/
|
2021-01-22 12:39:43 +00:00
|
|
|
for (i = 0; i < nr_levels; i++) {
|
2017-02-01 12:10:18 +00:00
|
|
|
sched_domains_numa_masks[i] =
|
|
|
|
|
kzalloc(nr_node_ids * sizeof(void *), GFP_KERNEL);
|
|
|
|
|
if (!sched_domains_numa_masks[i])
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
for (j = 0; j < nr_node_ids; j++) {
|
|
|
|
|
struct cpumask *mask = kzalloc(cpumask_size(), GFP_KERNEL);
|
2021-01-22 12:39:43 +00:00
|
|
|
int k;
|
|
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
if (!mask)
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
sched_domains_numa_masks[i][j] = mask;
|
|
|
|
|
|
|
|
|
|
for_each_node(k) {
|
2021-01-22 12:39:43 +00:00
|
|
|
if (sched_debug() && (node_distance(j, k) != node_distance(k, j)))
|
|
|
|
|
sched_numa_warn("Node-distance not symmetric");
|
|
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
if (node_distance(j, k) > sched_domains_numa_distance[i])
|
|
|
|
|
continue;
|
|
|
|
|
|
|
|
|
|
cpumask_or(mask, mask, cpumask_of_node(k));
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* Compute default topology size */
|
|
|
|
|
for (i = 0; sched_domain_topology[i].mask; i++);
|
|
|
|
|
|
2021-02-01 09:53:53 +00:00
|
|
|
tl = kzalloc((i + nr_levels + 1) *
|
2017-02-01 12:10:18 +00:00
|
|
|
sizeof(struct sched_domain_topology_level), GFP_KERNEL);
|
|
|
|
|
if (!tl)
|
|
|
|
|
return;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Copy the default topology bits..
|
|
|
|
|
*/
|
|
|
|
|
for (i = 0; sched_domain_topology[i].mask; i++)
|
|
|
|
|
tl[i] = sched_domain_topology[i];
|
|
|
|
|
|
sched/topology: Introduce NUMA identity node sched domain
On AMD Family17h-based (EPYC) system, a logical NUMA node can contain
upto 8 cores (16 threads) with the following topology.
----------------------------
C0 | T0 T1 | || | T0 T1 | C4
--------| || |--------
C1 | T0 T1 | L3 || L3 | T0 T1 | C5
--------| || |--------
C2 | T0 T1 | #0 || #1 | T0 T1 | C6
--------| || |--------
C3 | T0 T1 | || | T0 T1 | C7
----------------------------
Here, there are 2 last-level (L3) caches per logical NUMA node.
A socket can contain upto 4 NUMA nodes, and a system can support
upto 2 sockets. With full system configuration, current scheduler
creates 4 sched domains:
domain0 SMT (span a core)
domain1 MC (span a last-level-cache)
domain2 NUMA (span a socket: 4 nodes)
domain3 NUMA (span a system: 8 nodes)
Note that there is no domain to represent cpus spaning a logical
NUMA node. With this hierarchy of sched domains, the scheduler does
not balance properly in the following cases:
Case1:
When running 8 tasks, a properly balanced system should
schedule a task per logical NUMA node. This is not the case for
the current scheduler.
Case2:
In some cases, threads are scheduled on the same cpu, while other
cpus are idle. This results in run-to-run inconsistency. For example:
taskset -c 0-7 sysbench --num-threads=8 --test=cpu \
--cpu-max-prime=100000 run
Total execution time ranges from 25.1s to 33.5s depending on threads
placement, where 25.1s is when all 8 threads are balanced properly
on 8 cpus.
Introducing NUMA identity node sched domain, which is based on how
SRAT/SLIT table define a logical NUMA node. This results in the following
hierarchy of sched domains on the same system described above.
domain0 SMT (span a core)
domain1 MC (span a last-level-cache)
domain2 NODE (span a logical NUMA node)
domain3 NUMA (span a socket: 4 nodes)
domain4 NUMA (span a system: 8 nodes)
This fixes the improper load balancing cases mentioned above.
Signed-off-by: Suravee Suthikulpanit <suravee.suthikulpanit@amd.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: bp@suse.de
Link: http://lkml.kernel.org/r/1504768805-46716-1-git-send-email-suravee.suthikulpanit@amd.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-09-07 07:20:05 +00:00
|
|
|
/*
|
|
|
|
|
* Add the NUMA identity distance, aka single NODE.
|
|
|
|
|
*/
|
|
|
|
|
tl[i++] = (struct sched_domain_topology_level){
|
|
|
|
|
.mask = sd_numa_mask,
|
|
|
|
|
.numa_level = 0,
|
|
|
|
|
SD_INIT_NAME(NODE)
|
|
|
|
|
};
|
|
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
/*
|
|
|
|
|
* .. and append 'j' levels of NUMA goodness.
|
|
|
|
|
*/
|
2021-01-22 12:39:43 +00:00
|
|
|
for (j = 1; j < nr_levels; i++, j++) {
|
2017-02-01 12:10:18 +00:00
|
|
|
tl[i] = (struct sched_domain_topology_level){
|
|
|
|
|
.mask = sd_numa_mask,
|
|
|
|
|
.sd_flags = cpu_numa_flags,
|
|
|
|
|
.flags = SDTL_OVERLAP,
|
|
|
|
|
.numa_level = j,
|
|
|
|
|
SD_INIT_NAME(NUMA)
|
|
|
|
|
};
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
sched_domain_topology = tl;
|
|
|
|
|
|
2021-01-22 12:39:43 +00:00
|
|
|
sched_domains_numa_levels = nr_levels;
|
|
|
|
|
sched_max_numa_distance = sched_domains_numa_distance[nr_levels - 1];
|
2017-02-01 12:10:18 +00:00
|
|
|
|
|
|
|
|
init_numa_topology_type();
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void sched_domains_numa_masks_set(unsigned int cpu)
|
|
|
|
|
{
|
|
|
|
|
int node = cpu_to_node(cpu);
|
|
|
|
|
int i, j;
|
|
|
|
|
|
|
|
|
|
for (i = 0; i < sched_domains_numa_levels; i++) {
|
|
|
|
|
for (j = 0; j < nr_node_ids; j++) {
|
|
|
|
|
if (node_distance(j, node) <= sched_domains_numa_distance[i])
|
|
|
|
|
cpumask_set_cpu(cpu, sched_domains_numa_masks[i][j]);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void sched_domains_numa_masks_clear(unsigned int cpu)
|
|
|
|
|
{
|
|
|
|
|
int i, j;
|
|
|
|
|
|
|
|
|
|
for (i = 0; i < sched_domains_numa_levels; i++) {
|
|
|
|
|
for (j = 0; j < nr_node_ids; j++)
|
|
|
|
|
cpumask_clear_cpu(cpu, sched_domains_numa_masks[i][j]);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2019-06-28 08:51:41 +00:00
|
|
|
/*
|
|
|
|
|
* sched_numa_find_closest() - given the NUMA topology, find the cpu
|
|
|
|
|
* closest to @cpu from @cpumask.
|
|
|
|
|
* cpumask: cpumask to find a cpu from
|
|
|
|
|
* cpu: cpu to be close to
|
|
|
|
|
*
|
|
|
|
|
* returns: cpu, or nr_cpu_ids when nothing found.
|
|
|
|
|
*/
|
|
|
|
|
int sched_numa_find_closest(const struct cpumask *cpus, int cpu)
|
|
|
|
|
{
|
|
|
|
|
int i, j = cpu_to_node(cpu);
|
|
|
|
|
|
|
|
|
|
for (i = 0; i < sched_domains_numa_levels; i++) {
|
|
|
|
|
cpu = cpumask_any_and(cpus, sched_domains_numa_masks[i][j]);
|
|
|
|
|
if (cpu < nr_cpu_ids)
|
|
|
|
|
return cpu;
|
|
|
|
|
}
|
|
|
|
|
return nr_cpu_ids;
|
|
|
|
|
}
|
|
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
#endif /* CONFIG_NUMA */
|
|
|
|
|
|
|
|
|
|
static int __sdt_alloc(const struct cpumask *cpu_map)
|
|
|
|
|
{
|
|
|
|
|
struct sched_domain_topology_level *tl;
|
|
|
|
|
int j;
|
|
|
|
|
|
|
|
|
|
for_each_sd_topology(tl) {
|
|
|
|
|
struct sd_data *sdd = &tl->data;
|
|
|
|
|
|
|
|
|
|
sdd->sd = alloc_percpu(struct sched_domain *);
|
|
|
|
|
if (!sdd->sd)
|
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
|
|
sdd->sds = alloc_percpu(struct sched_domain_shared *);
|
|
|
|
|
if (!sdd->sds)
|
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
|
|
sdd->sg = alloc_percpu(struct sched_group *);
|
|
|
|
|
if (!sdd->sg)
|
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
|
|
sdd->sgc = alloc_percpu(struct sched_group_capacity *);
|
|
|
|
|
if (!sdd->sgc)
|
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
|
|
for_each_cpu(j, cpu_map) {
|
|
|
|
|
struct sched_domain *sd;
|
|
|
|
|
struct sched_domain_shared *sds;
|
|
|
|
|
struct sched_group *sg;
|
|
|
|
|
struct sched_group_capacity *sgc;
|
|
|
|
|
|
|
|
|
|
sd = kzalloc_node(sizeof(struct sched_domain) + cpumask_size(),
|
|
|
|
|
GFP_KERNEL, cpu_to_node(j));
|
|
|
|
|
if (!sd)
|
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
|
|
*per_cpu_ptr(sdd->sd, j) = sd;
|
|
|
|
|
|
|
|
|
|
sds = kzalloc_node(sizeof(struct sched_domain_shared),
|
|
|
|
|
GFP_KERNEL, cpu_to_node(j));
|
|
|
|
|
if (!sds)
|
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
|
|
*per_cpu_ptr(sdd->sds, j) = sds;
|
|
|
|
|
|
|
|
|
|
sg = kzalloc_node(sizeof(struct sched_group) + cpumask_size(),
|
|
|
|
|
GFP_KERNEL, cpu_to_node(j));
|
|
|
|
|
if (!sg)
|
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
|
|
|
|
sg->next = sg;
|
|
|
|
|
|
|
|
|
|
*per_cpu_ptr(sdd->sg, j) = sg;
|
|
|
|
|
|
|
|
|
|
sgc = kzalloc_node(sizeof(struct sched_group_capacity) + cpumask_size(),
|
|
|
|
|
GFP_KERNEL, cpu_to_node(j));
|
|
|
|
|
if (!sgc)
|
|
|
|
|
return -ENOMEM;
|
|
|
|
|
|
sched/topology: Add sched_group_capacity debugging
Add sgc::id to easier spot domain construction issues.
Take the opportunity to slightly rework the group printing, because
adding more "(id: %d)" strings makes the entire thing very hard to
read. Also the individual groups are very hard to separate, so add
explicit visual grouping, which allows replacing all the "(%s: %d)"
format things with shorter "%s=%d" variants.
Then fix up some inconsistencies in surrounding prints for domains.
The end result looks like:
[] CPU0 attaching sched-domain(s):
[] domain-0: span=0,4 level=DIE
[] groups: 0:{ span=0 }, 4:{ span=4 }
[] domain-1: span=0-1,3-5,7 level=NUMA
[] groups: 0:{ span=0,4 mask=0,4 cap=2048 }, 1:{ span=1,5 mask=1,5 cap=2048 }, 3:{ span=3,7 mask=3,7 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 0:{ span=0-1,3-5,7 mask=0,4 cap=6144 }, 2:{ span=1-3,5-7 mask=2,6 cap=6144 }
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-04-26 15:35:35 +00:00
|
|
|
#ifdef CONFIG_SCHED_DEBUG
|
|
|
|
|
sgc->id = j;
|
|
|
|
|
#endif
|
|
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
*per_cpu_ptr(sdd->sgc, j) = sgc;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return 0;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
static void __sdt_free(const struct cpumask *cpu_map)
|
|
|
|
|
{
|
|
|
|
|
struct sched_domain_topology_level *tl;
|
|
|
|
|
int j;
|
|
|
|
|
|
|
|
|
|
for_each_sd_topology(tl) {
|
|
|
|
|
struct sd_data *sdd = &tl->data;
|
|
|
|
|
|
|
|
|
|
for_each_cpu(j, cpu_map) {
|
|
|
|
|
struct sched_domain *sd;
|
|
|
|
|
|
|
|
|
|
if (sdd->sd) {
|
|
|
|
|
sd = *per_cpu_ptr(sdd->sd, j);
|
|
|
|
|
if (sd && (sd->flags & SD_OVERLAP))
|
|
|
|
|
free_sched_groups(sd->groups, 0);
|
|
|
|
|
kfree(*per_cpu_ptr(sdd->sd, j));
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (sdd->sds)
|
|
|
|
|
kfree(*per_cpu_ptr(sdd->sds, j));
|
|
|
|
|
if (sdd->sg)
|
|
|
|
|
kfree(*per_cpu_ptr(sdd->sg, j));
|
|
|
|
|
if (sdd->sgc)
|
|
|
|
|
kfree(*per_cpu_ptr(sdd->sgc, j));
|
|
|
|
|
}
|
|
|
|
|
free_percpu(sdd->sd);
|
|
|
|
|
sdd->sd = NULL;
|
|
|
|
|
free_percpu(sdd->sds);
|
|
|
|
|
sdd->sds = NULL;
|
|
|
|
|
free_percpu(sdd->sg);
|
|
|
|
|
sdd->sg = NULL;
|
|
|
|
|
free_percpu(sdd->sgc);
|
|
|
|
|
sdd->sgc = NULL;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
2017-04-27 08:28:59 +00:00
|
|
|
static struct sched_domain *build_sched_domain(struct sched_domain_topology_level *tl,
|
2017-02-01 12:10:18 +00:00
|
|
|
const struct cpumask *cpu_map, struct sched_domain_attr *attr,
|
2018-07-20 13:32:31 +00:00
|
|
|
struct sched_domain *child, int dflags, int cpu)
|
2017-02-01 12:10:18 +00:00
|
|
|
{
|
2018-07-20 13:32:31 +00:00
|
|
|
struct sched_domain *sd = sd_init(tl, cpu_map, child, dflags, cpu);
|
2017-02-01 12:10:18 +00:00
|
|
|
|
|
|
|
|
if (child) {
|
|
|
|
|
sd->level = child->level + 1;
|
|
|
|
|
sched_domain_level_max = max(sched_domain_level_max, sd->level);
|
|
|
|
|
child->parent = sd;
|
|
|
|
|
|
|
|
|
|
if (!cpumask_subset(sched_domain_span(child),
|
|
|
|
|
sched_domain_span(sd))) {
|
|
|
|
|
pr_err("BUG: arch topology borken\n");
|
|
|
|
|
#ifdef CONFIG_SCHED_DEBUG
|
|
|
|
|
pr_err(" the %s domain not a subset of the %s domain\n",
|
|
|
|
|
child->name, sd->name);
|
|
|
|
|
#endif
|
sched: Clean up and harmonize the coding style of the scheduler code base
A good number of small style inconsistencies have accumulated
in the scheduler core, so do a pass over them to harmonize
all these details:
- fix speling in comments,
- use curly braces for multi-line statements,
- remove unnecessary parentheses from integer literals,
- capitalize consistently,
- remove stray newlines,
- add comments where necessary,
- remove invalid/unnecessary comments,
- align structure definitions and other data types vertically,
- add missing newlines for increased readability,
- fix vertical tabulation where it's misaligned,
- harmonize preprocessor conditional block labeling
and vertical alignment,
- remove line-breaks where they uglify the code,
- add newline after local variable definitions,
No change in functionality:
md5:
1191fa0a890cfa8132156d2959d7e9e2 built-in.o.before.asm
1191fa0a890cfa8132156d2959d7e9e2 built-in.o.after.asm
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-03-03 13:01:12 +00:00
|
|
|
/* Fixup, ensure @sd has at least @child CPUs. */
|
2017-02-01 12:10:18 +00:00
|
|
|
cpumask_or(sched_domain_span(sd),
|
|
|
|
|
sched_domain_span(sd),
|
|
|
|
|
sched_domain_span(child));
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
}
|
|
|
|
|
set_domain_attribute(sd, attr);
|
|
|
|
|
|
|
|
|
|
return sd;
|
|
|
|
|
}
|
|
|
|
|
|
sched/topology: Assert non-NUMA topology masks don't (partially) overlap
topology.c::get_group() relies on the assumption that non-NUMA domains do
not partially overlap. Zeng Tao pointed out in [1] that such topology
descriptions, while completely bogus, can end up being exposed to the
scheduler.
In his example (8 CPUs, 2-node system), we end up with:
MC span for CPU3 == 3-7
MC span for CPU4 == 4-7
The first pass through get_group(3, sdd@MC) will result in the following
sched_group list:
3 -> 4 -> 5 -> 6 -> 7
^ /
`----------------'
And a later pass through get_group(4, sdd@MC) will "corrupt" that to:
3 -> 4 -> 5 -> 6 -> 7
^ /
`-----------'
which will completely break things like 'while (sg != sd->groups)' when
using CPU3's base sched_domain.
There already are some architecture-specific checks in place such as
x86/kernel/smpboot.c::topology.sane(), but this is something we can detect
in the core scheduler, so it seems worthwhile to do so.
Warn and abort the construction of the sched domains if such a broken
topology description is detected. Note that this is somewhat
expensive (O(t.c²), 't' non-NUMA topology levels and 'c' CPUs) and could be
gated under SCHED_DEBUG if deemed necessary.
Testing
=======
Dietmar managed to reproduce this using the following qemu incantation:
$ qemu-system-aarch64 -kernel ./Image -hda ./qemu-image-aarch64.img \
-append 'root=/dev/vda console=ttyAMA0 loglevel=8 sched_debug' -smp \
cores=8 --nographic -m 512 -cpu cortex-a53 -machine virt -numa \
node,cpus=0-2,nodeid=0 -numa node,cpus=3-7,nodeid=1
alongside the following drivers/base/arch_topology.c hack (AIUI wouldn't be
needed if '-smp cores=X, sockets=Y' would work with qemu):
8<---
@@ -465,6 +465,9 @@ void update_siblings_masks(unsigned int cpuid)
if (cpuid_topo->package_id != cpu_topo->package_id)
continue;
+ if ((cpu < 4 && cpuid > 3) || (cpu > 3 && cpuid < 4))
+ continue;
+
cpumask_set_cpu(cpuid, &cpu_topo->core_sibling);
cpumask_set_cpu(cpu, &cpuid_topo->core_sibling);
8<---
[1]: https://lkml.kernel.org/r/1577088979-8545-1-git-send-email-prime.zeng@hisilicon.com
Reported-by: Zeng Tao <prime.zeng@hisilicon.com>
Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lkml.kernel.org/r/20200115160915.22575-1-valentin.schneider@arm.com
2020-01-15 16:09:15 +00:00
|
|
|
/*
|
|
|
|
|
* Ensure topology masks are sane, i.e. there are no conflicts (overlaps) for
|
|
|
|
|
* any two given CPUs at this (non-NUMA) topology level.
|
|
|
|
|
*/
|
|
|
|
|
static bool topology_span_sane(struct sched_domain_topology_level *tl,
|
|
|
|
|
const struct cpumask *cpu_map, int cpu)
|
|
|
|
|
{
|
|
|
|
|
int i;
|
|
|
|
|
|
|
|
|
|
/* NUMA levels are allowed to overlap */
|
|
|
|
|
if (tl->flags & SDTL_OVERLAP)
|
|
|
|
|
return true;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Non-NUMA levels cannot partially overlap - they must be either
|
|
|
|
|
* completely equal or completely disjoint. Otherwise we can end up
|
|
|
|
|
* breaking the sched_group lists - i.e. a later get_group() pass
|
|
|
|
|
* breaks the linking done for an earlier span.
|
|
|
|
|
*/
|
|
|
|
|
for_each_cpu(i, cpu_map) {
|
|
|
|
|
if (i == cpu)
|
|
|
|
|
continue;
|
|
|
|
|
/*
|
|
|
|
|
* We should 'and' all those masks with 'cpu_map' to exactly
|
|
|
|
|
* match the topology we're about to build, but that can only
|
|
|
|
|
* remove CPUs, which only lessens our ability to detect
|
|
|
|
|
* overlaps
|
|
|
|
|
*/
|
|
|
|
|
if (!cpumask_equal(tl->mask(cpu), tl->mask(i)) &&
|
|
|
|
|
cpumask_intersects(tl->mask(cpu), tl->mask(i)))
|
|
|
|
|
return false;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return true;
|
|
|
|
|
}
|
|
|
|
|
|
2018-07-20 13:32:31 +00:00
|
|
|
/*
|
|
|
|
|
* Find the sched_domain_topology_level where all CPU capacities are visible
|
|
|
|
|
* for all CPUs.
|
|
|
|
|
*/
|
|
|
|
|
static struct sched_domain_topology_level
|
|
|
|
|
*asym_cpu_capacity_level(const struct cpumask *cpu_map)
|
|
|
|
|
{
|
|
|
|
|
int i, j, asym_level = 0;
|
|
|
|
|
bool asym = false;
|
|
|
|
|
struct sched_domain_topology_level *tl, *asym_tl = NULL;
|
|
|
|
|
unsigned long cap;
|
|
|
|
|
|
|
|
|
|
/* Is there any asymmetry? */
|
2019-06-17 15:00:17 +00:00
|
|
|
cap = arch_scale_cpu_capacity(cpumask_first(cpu_map));
|
2018-07-20 13:32:31 +00:00
|
|
|
|
|
|
|
|
for_each_cpu(i, cpu_map) {
|
2019-06-17 15:00:17 +00:00
|
|
|
if (arch_scale_cpu_capacity(i) != cap) {
|
2018-07-20 13:32:31 +00:00
|
|
|
asym = true;
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
if (!asym)
|
|
|
|
|
return NULL;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Examine topology from all CPU's point of views to detect the lowest
|
|
|
|
|
* sched_domain_topology_level where a highest capacity CPU is visible
|
|
|
|
|
* to everyone.
|
|
|
|
|
*/
|
|
|
|
|
for_each_cpu(i, cpu_map) {
|
2019-06-17 15:00:17 +00:00
|
|
|
unsigned long max_capacity = arch_scale_cpu_capacity(i);
|
2018-07-20 13:32:31 +00:00
|
|
|
int tl_id = 0;
|
|
|
|
|
|
|
|
|
|
for_each_sd_topology(tl) {
|
|
|
|
|
if (tl_id < asym_level)
|
|
|
|
|
goto next_level;
|
|
|
|
|
|
|
|
|
|
for_each_cpu_and(j, tl->mask(i), cpu_map) {
|
|
|
|
|
unsigned long capacity;
|
|
|
|
|
|
2019-06-17 15:00:17 +00:00
|
|
|
capacity = arch_scale_cpu_capacity(j);
|
2018-07-20 13:32:31 +00:00
|
|
|
|
|
|
|
|
if (capacity <= max_capacity)
|
|
|
|
|
continue;
|
|
|
|
|
|
|
|
|
|
max_capacity = capacity;
|
|
|
|
|
asym_level = tl_id;
|
|
|
|
|
asym_tl = tl;
|
|
|
|
|
}
|
|
|
|
|
next_level:
|
|
|
|
|
tl_id++;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
return asym_tl;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
/*
|
|
|
|
|
* Build sched domains for a given set of CPUs and attach the sched domains
|
|
|
|
|
* to the individual CPUs
|
|
|
|
|
*/
|
|
|
|
|
static int
|
|
|
|
|
build_sched_domains(const struct cpumask *cpu_map, struct sched_domain_attr *attr)
|
|
|
|
|
{
|
2019-10-23 15:37:44 +00:00
|
|
|
enum s_alloc alloc_state = sa_none;
|
2017-02-01 12:10:18 +00:00
|
|
|
struct sched_domain *sd;
|
|
|
|
|
struct s_data d;
|
|
|
|
|
struct rq *rq = NULL;
|
|
|
|
|
int i, ret = -ENOMEM;
|
2018-07-20 13:32:31 +00:00
|
|
|
struct sched_domain_topology_level *tl_asym;
|
2018-07-04 10:17:39 +00:00
|
|
|
bool has_asym = false;
|
2017-02-01 12:10:18 +00:00
|
|
|
|
2019-10-23 15:37:44 +00:00
|
|
|
if (WARN_ON(cpumask_empty(cpu_map)))
|
|
|
|
|
goto error;
|
|
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
alloc_state = __visit_domain_allocation_hell(&d, cpu_map);
|
|
|
|
|
if (alloc_state != sa_rootdomain)
|
|
|
|
|
goto error;
|
|
|
|
|
|
2018-07-20 13:32:31 +00:00
|
|
|
tl_asym = asym_cpu_capacity_level(cpu_map);
|
|
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
/* Set up domains for CPUs specified by the cpu_map: */
|
|
|
|
|
for_each_cpu(i, cpu_map) {
|
|
|
|
|
struct sched_domain_topology_level *tl;
|
2020-08-17 11:29:56 +00:00
|
|
|
int dflags = 0;
|
2017-02-01 12:10:18 +00:00
|
|
|
|
|
|
|
|
sd = NULL;
|
|
|
|
|
for_each_sd_topology(tl) {
|
2018-07-04 10:17:39 +00:00
|
|
|
if (tl == tl_asym) {
|
2018-07-20 13:32:31 +00:00
|
|
|
dflags |= SD_ASYM_CPUCAPACITY;
|
2018-07-04 10:17:39 +00:00
|
|
|
has_asym = true;
|
|
|
|
|
}
|
2018-07-20 13:32:31 +00:00
|
|
|
|
sched/topology: Assert non-NUMA topology masks don't (partially) overlap
topology.c::get_group() relies on the assumption that non-NUMA domains do
not partially overlap. Zeng Tao pointed out in [1] that such topology
descriptions, while completely bogus, can end up being exposed to the
scheduler.
In his example (8 CPUs, 2-node system), we end up with:
MC span for CPU3 == 3-7
MC span for CPU4 == 4-7
The first pass through get_group(3, sdd@MC) will result in the following
sched_group list:
3 -> 4 -> 5 -> 6 -> 7
^ /
`----------------'
And a later pass through get_group(4, sdd@MC) will "corrupt" that to:
3 -> 4 -> 5 -> 6 -> 7
^ /
`-----------'
which will completely break things like 'while (sg != sd->groups)' when
using CPU3's base sched_domain.
There already are some architecture-specific checks in place such as
x86/kernel/smpboot.c::topology.sane(), but this is something we can detect
in the core scheduler, so it seems worthwhile to do so.
Warn and abort the construction of the sched domains if such a broken
topology description is detected. Note that this is somewhat
expensive (O(t.c²), 't' non-NUMA topology levels and 'c' CPUs) and could be
gated under SCHED_DEBUG if deemed necessary.
Testing
=======
Dietmar managed to reproduce this using the following qemu incantation:
$ qemu-system-aarch64 -kernel ./Image -hda ./qemu-image-aarch64.img \
-append 'root=/dev/vda console=ttyAMA0 loglevel=8 sched_debug' -smp \
cores=8 --nographic -m 512 -cpu cortex-a53 -machine virt -numa \
node,cpus=0-2,nodeid=0 -numa node,cpus=3-7,nodeid=1
alongside the following drivers/base/arch_topology.c hack (AIUI wouldn't be
needed if '-smp cores=X, sockets=Y' would work with qemu):
8<---
@@ -465,6 +465,9 @@ void update_siblings_masks(unsigned int cpuid)
if (cpuid_topo->package_id != cpu_topo->package_id)
continue;
+ if ((cpu < 4 && cpuid > 3) || (cpu > 3 && cpuid < 4))
+ continue;
+
cpumask_set_cpu(cpuid, &cpu_topo->core_sibling);
cpumask_set_cpu(cpu, &cpuid_topo->core_sibling);
8<---
[1]: https://lkml.kernel.org/r/1577088979-8545-1-git-send-email-prime.zeng@hisilicon.com
Reported-by: Zeng Tao <prime.zeng@hisilicon.com>
Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Link: https://lkml.kernel.org/r/20200115160915.22575-1-valentin.schneider@arm.com
2020-01-15 16:09:15 +00:00
|
|
|
if (WARN_ON(!topology_span_sane(tl, cpu_map, i)))
|
|
|
|
|
goto error;
|
|
|
|
|
|
2018-07-20 13:32:31 +00:00
|
|
|
sd = build_sched_domain(tl, cpu_map, attr, sd, dflags, i);
|
|
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
if (tl == sched_domain_topology)
|
|
|
|
|
*per_cpu_ptr(d.sd, i) = sd;
|
2017-04-26 15:36:41 +00:00
|
|
|
if (tl->flags & SDTL_OVERLAP)
|
2017-02-01 12:10:18 +00:00
|
|
|
sd->flags |= SD_OVERLAP;
|
|
|
|
|
if (cpumask_equal(cpu_map, sched_domain_span(sd)))
|
|
|
|
|
break;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* Build the groups for the domains */
|
|
|
|
|
for_each_cpu(i, cpu_map) {
|
|
|
|
|
for (sd = *per_cpu_ptr(d.sd, i); sd; sd = sd->parent) {
|
|
|
|
|
sd->span_weight = cpumask_weight(sched_domain_span(sd));
|
|
|
|
|
if (sd->flags & SD_OVERLAP) {
|
|
|
|
|
if (build_overlap_sched_groups(sd, i))
|
|
|
|
|
goto error;
|
|
|
|
|
} else {
|
|
|
|
|
if (build_sched_groups(sd, i))
|
|
|
|
|
goto error;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* Calculate CPU capacity for physical packages and nodes */
|
|
|
|
|
for (i = nr_cpumask_bits-1; i >= 0; i--) {
|
|
|
|
|
if (!cpumask_test_cpu(i, cpu_map))
|
|
|
|
|
continue;
|
|
|
|
|
|
|
|
|
|
for (sd = *per_cpu_ptr(d.sd, i); sd; sd = sd->parent) {
|
|
|
|
|
claim_allocations(i, sd);
|
|
|
|
|
init_sched_groups_capacity(i, sd);
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* Attach the domains */
|
|
|
|
|
rcu_read_lock();
|
|
|
|
|
for_each_cpu(i, cpu_map) {
|
|
|
|
|
rq = cpu_rq(i);
|
|
|
|
|
sd = *per_cpu_ptr(d.sd, i);
|
|
|
|
|
|
|
|
|
|
/* Use READ_ONCE()/WRITE_ONCE() to avoid load/store tearing: */
|
|
|
|
|
if (rq->cpu_capacity_orig > READ_ONCE(d.rd->max_cpu_capacity))
|
|
|
|
|
WRITE_ONCE(d.rd->max_cpu_capacity, rq->cpu_capacity_orig);
|
|
|
|
|
|
|
|
|
|
cpu_attach_domain(sd, d.rd, i);
|
|
|
|
|
}
|
|
|
|
|
rcu_read_unlock();
|
|
|
|
|
|
2018-07-04 10:17:39 +00:00
|
|
|
if (has_asym)
|
sched/topology: Allow sched_asym_cpucapacity to be disabled
While the static key is correctly initialized as being disabled, it will
remain forever enabled once turned on. This means that if we start with an
asymmetric system and hotplug out enough CPUs to end up with an SMP system,
the static key will remain set - which is obviously wrong. We should detect
this and turn off things like misfit migration and capacity aware wakeups.
As Quentin pointed out, having separate root domains makes this slightly
trickier. We could have exclusive cpusets that create an SMP island - IOW,
the domains within this root domain will not see any asymmetry. This means
we can't just disable the key on domain destruction, we need to count how
many asymmetric root domains we have.
Consider the following example using Juno r0 which is 2+4 big.LITTLE, where
two identical cpusets are created: they both span both big and LITTLE CPUs:
asym0 asym1
[ ][ ]
L L B L L B
$ cgcreate -g cpuset:asym0
$ cgset -r cpuset.cpus=0,1,3 asym0
$ cgset -r cpuset.mems=0 asym0
$ cgset -r cpuset.cpu_exclusive=1 asym0
$ cgcreate -g cpuset:asym1
$ cgset -r cpuset.cpus=2,4,5 asym1
$ cgset -r cpuset.mems=0 asym1
$ cgset -r cpuset.cpu_exclusive=1 asym1
$ cgset -r cpuset.sched_load_balance=0 .
(the CPU numbering may look odd because on the Juno LITTLEs are CPUs 0,3-5
and bigs are CPUs 1-2)
If we make one of those SMP (IOW remove asymmetry) by e.g. hotplugging its
big core, we would end up with an SMP cpuset and an asymmetric cpuset - the
static key must remain set, because we still have one asymmetric root domain.
With the above example, this could be done with:
$ echo 0 > /sys/devices/system/cpu/cpu2/online
Which would result in:
asym0 asym1
[ ][ ]
L L B L L
When both SMP and asymmetric cpusets are present, all CPUs will observe
sched_asym_cpucapacity being set (it is system-wide), but not all CPUs
observe asymmetry in their sched domain hierarchy:
per_cpu(sd_asym_cpucapacity, <any CPU in asym0>) == <some SD at DIE level>
per_cpu(sd_asym_cpucapacity, <any CPU in asym1>) == NULL
Change the simple key enablement to an increment, and decrement the key
counter when destroying domains that cover asymmetric CPUs.
Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Dietmar.Eggemann@arm.com
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: hannes@cmpxchg.org
Cc: lizefan@huawei.com
Cc: morten.rasmussen@arm.com
Cc: qperret@google.com
Cc: tj@kernel.org
Cc: vincent.guittot@linaro.org
Fixes: df054e8445a4 ("sched/topology: Add static_key for asymmetric CPU capacity optimizations")
Link: https://lkml.kernel.org/r/20191023153745.19515-3-valentin.schneider@arm.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2019-10-23 15:37:45 +00:00
|
|
|
static_branch_inc_cpuslocked(&sched_asym_cpucapacity);
|
2018-07-04 10:17:39 +00:00
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
if (rq && sched_debug_enabled) {
|
sched/topology: Clarify root domain(s) debug string
When scheduler debug is enabled, building scheduling domains outputs
information about how the domains are laid out and to which root domain
each CPU (or sets of CPUs) belongs, e.g.:
CPU0 attaching sched-domain(s):
domain-0: span=0-5 level=MC
groups: 0:{ span=0 }, 1:{ span=1 }, 2:{ span=2 }, 3:{ span=3 }, 4:{ span=4 }, 5:{ span=5 }
CPU1 attaching sched-domain(s):
domain-0: span=0-5 level=MC
groups: 1:{ span=1 }, 2:{ span=2 }, 3:{ span=3 }, 4:{ span=4 }, 5:{ span=5 }, 0:{ span=0 }
[...]
span: 0-5 (max cpu_capacity = 1024)
The fact that latest line refers to CPUs 0-5 root domain doesn't however look
immediately obvious to me: one might wonder why span 0-5 is reported "again".
Make it more clear by adding "root domain" to it, as to end with the
following:
CPU0 attaching sched-domain(s):
domain-0: span=0-5 level=MC
groups: 0:{ span=0 }, 1:{ span=1 }, 2:{ span=2 }, 3:{ span=3 }, 4:{ span=4 }, 5:{ span=5 }
CPU1 attaching sched-domain(s):
domain-0: span=0-5 level=MC
groups: 1:{ span=1 }, 2:{ span=2 }, 3:{ span=3 }, 4:{ span=4 }, 5:{ span=5 }, 0:{ span=0 }
[...]
root domain span: 0-5 (max cpu_capacity = 1024)
Signed-off-by: Juri Lelli <juri.lelli@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Patrick Bellasi <patrick.bellasi@arm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Link: http://lkml.kernel.org/r/20180524152936.17611-1-juri.lelli@redhat.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-05-24 15:29:36 +00:00
|
|
|
pr_info("root domain span: %*pbl (max cpu_capacity = %lu)\n",
|
2017-02-01 12:10:18 +00:00
|
|
|
cpumask_pr_args(cpu_map), rq->rd->max_cpu_capacity);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
ret = 0;
|
|
|
|
|
error:
|
|
|
|
|
__free_domain_allocs(&d, alloc_state, cpu_map);
|
sched: Clean up and harmonize the coding style of the scheduler code base
A good number of small style inconsistencies have accumulated
in the scheduler core, so do a pass over them to harmonize
all these details:
- fix speling in comments,
- use curly braces for multi-line statements,
- remove unnecessary parentheses from integer literals,
- capitalize consistently,
- remove stray newlines,
- add comments where necessary,
- remove invalid/unnecessary comments,
- align structure definitions and other data types vertically,
- add missing newlines for increased readability,
- fix vertical tabulation where it's misaligned,
- harmonize preprocessor conditional block labeling
and vertical alignment,
- remove line-breaks where they uglify the code,
- add newline after local variable definitions,
No change in functionality:
md5:
1191fa0a890cfa8132156d2959d7e9e2 built-in.o.before.asm
1191fa0a890cfa8132156d2959d7e9e2 built-in.o.after.asm
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-03-03 13:01:12 +00:00
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
return ret;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* Current sched domains: */
|
|
|
|
|
static cpumask_var_t *doms_cur;
|
|
|
|
|
|
|
|
|
|
/* Number of sched domains in 'doms_cur': */
|
|
|
|
|
static int ndoms_cur;
|
|
|
|
|
|
2021-03-18 12:38:50 +00:00
|
|
|
/* Attributes of custom domains in 'doms_cur' */
|
2017-02-01 12:10:18 +00:00
|
|
|
static struct sched_domain_attr *dattr_cur;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Special case: If a kmalloc() of a doms_cur partition (array of
|
|
|
|
|
* cpumask) fails, then fallback to a single sched domain,
|
|
|
|
|
* as determined by the single cpumask fallback_doms.
|
|
|
|
|
*/
|
2017-04-25 13:29:40 +00:00
|
|
|
static cpumask_var_t fallback_doms;
|
2017-02-01 12:10:18 +00:00
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* arch_update_cpu_topology lets virtualized architectures update the
|
|
|
|
|
* CPU core maps. It is supposed to return 1 if the topology changed
|
|
|
|
|
* or 0 if it stayed the same.
|
|
|
|
|
*/
|
|
|
|
|
int __weak arch_update_cpu_topology(void)
|
|
|
|
|
{
|
|
|
|
|
return 0;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
cpumask_var_t *alloc_sched_domains(unsigned int ndoms)
|
|
|
|
|
{
|
|
|
|
|
int i;
|
|
|
|
|
cpumask_var_t *doms;
|
|
|
|
|
|
treewide: kmalloc() -> kmalloc_array()
The kmalloc() function has a 2-factor argument form, kmalloc_array(). This
patch replaces cases of:
kmalloc(a * b, gfp)
with:
kmalloc_array(a * b, gfp)
as well as handling cases of:
kmalloc(a * b * c, gfp)
with:
kmalloc(array3_size(a, b, c), gfp)
as it's slightly less ugly than:
kmalloc_array(array_size(a, b), c, gfp)
This does, however, attempt to ignore constant size factors like:
kmalloc(4 * 1024, gfp)
though any constants defined via macros get caught up in the conversion.
Any factors with a sizeof() of "unsigned char", "char", and "u8" were
dropped, since they're redundant.
The tools/ directory was manually excluded, since it has its own
implementation of kmalloc().
The Coccinelle script used for this was:
// Fix redundant parens around sizeof().
@@
type TYPE;
expression THING, E;
@@
(
kmalloc(
- (sizeof(TYPE)) * E
+ sizeof(TYPE) * E
, ...)
|
kmalloc(
- (sizeof(THING)) * E
+ sizeof(THING) * E
, ...)
)
// Drop single-byte sizes and redundant parens.
@@
expression COUNT;
typedef u8;
typedef __u8;
@@
(
kmalloc(
- sizeof(u8) * (COUNT)
+ COUNT
, ...)
|
kmalloc(
- sizeof(__u8) * (COUNT)
+ COUNT
, ...)
|
kmalloc(
- sizeof(char) * (COUNT)
+ COUNT
, ...)
|
kmalloc(
- sizeof(unsigned char) * (COUNT)
+ COUNT
, ...)
|
kmalloc(
- sizeof(u8) * COUNT
+ COUNT
, ...)
|
kmalloc(
- sizeof(__u8) * COUNT
+ COUNT
, ...)
|
kmalloc(
- sizeof(char) * COUNT
+ COUNT
, ...)
|
kmalloc(
- sizeof(unsigned char) * COUNT
+ COUNT
, ...)
)
// 2-factor product with sizeof(type/expression) and identifier or constant.
@@
type TYPE;
expression THING;
identifier COUNT_ID;
constant COUNT_CONST;
@@
(
- kmalloc
+ kmalloc_array
(
- sizeof(TYPE) * (COUNT_ID)
+ COUNT_ID, sizeof(TYPE)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(TYPE) * COUNT_ID
+ COUNT_ID, sizeof(TYPE)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(TYPE) * (COUNT_CONST)
+ COUNT_CONST, sizeof(TYPE)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(TYPE) * COUNT_CONST
+ COUNT_CONST, sizeof(TYPE)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(THING) * (COUNT_ID)
+ COUNT_ID, sizeof(THING)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(THING) * COUNT_ID
+ COUNT_ID, sizeof(THING)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(THING) * (COUNT_CONST)
+ COUNT_CONST, sizeof(THING)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(THING) * COUNT_CONST
+ COUNT_CONST, sizeof(THING)
, ...)
)
// 2-factor product, only identifiers.
@@
identifier SIZE, COUNT;
@@
- kmalloc
+ kmalloc_array
(
- SIZE * COUNT
+ COUNT, SIZE
, ...)
// 3-factor product with 1 sizeof(type) or sizeof(expression), with
// redundant parens removed.
@@
expression THING;
identifier STRIDE, COUNT;
type TYPE;
@@
(
kmalloc(
- sizeof(TYPE) * (COUNT) * (STRIDE)
+ array3_size(COUNT, STRIDE, sizeof(TYPE))
, ...)
|
kmalloc(
- sizeof(TYPE) * (COUNT) * STRIDE
+ array3_size(COUNT, STRIDE, sizeof(TYPE))
, ...)
|
kmalloc(
- sizeof(TYPE) * COUNT * (STRIDE)
+ array3_size(COUNT, STRIDE, sizeof(TYPE))
, ...)
|
kmalloc(
- sizeof(TYPE) * COUNT * STRIDE
+ array3_size(COUNT, STRIDE, sizeof(TYPE))
, ...)
|
kmalloc(
- sizeof(THING) * (COUNT) * (STRIDE)
+ array3_size(COUNT, STRIDE, sizeof(THING))
, ...)
|
kmalloc(
- sizeof(THING) * (COUNT) * STRIDE
+ array3_size(COUNT, STRIDE, sizeof(THING))
, ...)
|
kmalloc(
- sizeof(THING) * COUNT * (STRIDE)
+ array3_size(COUNT, STRIDE, sizeof(THING))
, ...)
|
kmalloc(
- sizeof(THING) * COUNT * STRIDE
+ array3_size(COUNT, STRIDE, sizeof(THING))
, ...)
)
// 3-factor product with 2 sizeof(variable), with redundant parens removed.
@@
expression THING1, THING2;
identifier COUNT;
type TYPE1, TYPE2;
@@
(
kmalloc(
- sizeof(TYPE1) * sizeof(TYPE2) * COUNT
+ array3_size(COUNT, sizeof(TYPE1), sizeof(TYPE2))
, ...)
|
kmalloc(
- sizeof(TYPE1) * sizeof(THING2) * (COUNT)
+ array3_size(COUNT, sizeof(TYPE1), sizeof(TYPE2))
, ...)
|
kmalloc(
- sizeof(THING1) * sizeof(THING2) * COUNT
+ array3_size(COUNT, sizeof(THING1), sizeof(THING2))
, ...)
|
kmalloc(
- sizeof(THING1) * sizeof(THING2) * (COUNT)
+ array3_size(COUNT, sizeof(THING1), sizeof(THING2))
, ...)
|
kmalloc(
- sizeof(TYPE1) * sizeof(THING2) * COUNT
+ array3_size(COUNT, sizeof(TYPE1), sizeof(THING2))
, ...)
|
kmalloc(
- sizeof(TYPE1) * sizeof(THING2) * (COUNT)
+ array3_size(COUNT, sizeof(TYPE1), sizeof(THING2))
, ...)
)
// 3-factor product, only identifiers, with redundant parens removed.
@@
identifier STRIDE, SIZE, COUNT;
@@
(
kmalloc(
- (COUNT) * STRIDE * SIZE
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kmalloc(
- COUNT * (STRIDE) * SIZE
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kmalloc(
- COUNT * STRIDE * (SIZE)
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kmalloc(
- (COUNT) * (STRIDE) * SIZE
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kmalloc(
- COUNT * (STRIDE) * (SIZE)
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kmalloc(
- (COUNT) * STRIDE * (SIZE)
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kmalloc(
- (COUNT) * (STRIDE) * (SIZE)
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
|
kmalloc(
- COUNT * STRIDE * SIZE
+ array3_size(COUNT, STRIDE, SIZE)
, ...)
)
// Any remaining multi-factor products, first at least 3-factor products,
// when they're not all constants...
@@
expression E1, E2, E3;
constant C1, C2, C3;
@@
(
kmalloc(C1 * C2 * C3, ...)
|
kmalloc(
- (E1) * E2 * E3
+ array3_size(E1, E2, E3)
, ...)
|
kmalloc(
- (E1) * (E2) * E3
+ array3_size(E1, E2, E3)
, ...)
|
kmalloc(
- (E1) * (E2) * (E3)
+ array3_size(E1, E2, E3)
, ...)
|
kmalloc(
- E1 * E2 * E3
+ array3_size(E1, E2, E3)
, ...)
)
// And then all remaining 2 factors products when they're not all constants,
// keeping sizeof() as the second factor argument.
@@
expression THING, E1, E2;
type TYPE;
constant C1, C2, C3;
@@
(
kmalloc(sizeof(THING) * C2, ...)
|
kmalloc(sizeof(TYPE) * C2, ...)
|
kmalloc(C1 * C2 * C3, ...)
|
kmalloc(C1 * C2, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(TYPE) * (E2)
+ E2, sizeof(TYPE)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(TYPE) * E2
+ E2, sizeof(TYPE)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(THING) * (E2)
+ E2, sizeof(THING)
, ...)
|
- kmalloc
+ kmalloc_array
(
- sizeof(THING) * E2
+ E2, sizeof(THING)
, ...)
|
- kmalloc
+ kmalloc_array
(
- (E1) * E2
+ E1, E2
, ...)
|
- kmalloc
+ kmalloc_array
(
- (E1) * (E2)
+ E1, E2
, ...)
|
- kmalloc
+ kmalloc_array
(
- E1 * E2
+ E1, E2
, ...)
)
Signed-off-by: Kees Cook <keescook@chromium.org>
2018-06-12 20:55:00 +00:00
|
|
|
doms = kmalloc_array(ndoms, sizeof(*doms), GFP_KERNEL);
|
2017-02-01 12:10:18 +00:00
|
|
|
if (!doms)
|
|
|
|
|
return NULL;
|
|
|
|
|
for (i = 0; i < ndoms; i++) {
|
|
|
|
|
if (!alloc_cpumask_var(&doms[i], GFP_KERNEL)) {
|
|
|
|
|
free_sched_domains(doms, i);
|
|
|
|
|
return NULL;
|
|
|
|
|
}
|
|
|
|
|
}
|
|
|
|
|
return doms;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
void free_sched_domains(cpumask_var_t doms[], unsigned int ndoms)
|
|
|
|
|
{
|
|
|
|
|
unsigned int i;
|
|
|
|
|
for (i = 0; i < ndoms; i++)
|
|
|
|
|
free_cpumask_var(doms[i]);
|
|
|
|
|
kfree(doms);
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
2018-12-19 13:34:45 +00:00
|
|
|
* Set up scheduler domains and groups. For now this just excludes isolated
|
|
|
|
|
* CPUs, but could be used to exclude other special cases in the future.
|
2017-02-01 12:10:18 +00:00
|
|
|
*/
|
2017-04-25 13:29:40 +00:00
|
|
|
int sched_init_domains(const struct cpumask *cpu_map)
|
2017-02-01 12:10:18 +00:00
|
|
|
{
|
|
|
|
|
int err;
|
|
|
|
|
|
2017-04-25 13:29:40 +00:00
|
|
|
zalloc_cpumask_var(&sched_domains_tmpmask, GFP_KERNEL);
|
sched/topology: Fix overlapping sched_group_capacity
When building the overlapping groups we need to attach a consistent
sched_group_capacity structure. That is, all 'identical' sched_group's
should have the _same_ sched_group_capacity.
This can (once again) be demonstrated with a topology like:
node 0 1 2 3
0: 10 20 30 20
1: 20 10 20 30
2: 30 20 10 20
3: 20 30 20 10
But we need at least 2 CPUs per node for this to show up, after all,
if there is only one CPU per node, our CPU @i is per definition a
unique CPU that reaches this domain (aka balance-cpu).
Given the above NUMA topo and 2 CPUs per node:
[] CPU0 attaching sched-domain(s):
[] domain-0: span=0,4 level=DIE
[] groups: 0:{ span=0 }, 4:{ span=4 }
[] domain-1: span=0-1,3-5,7 level=NUMA
[] groups: 0:{ span=0,4 mask=0,4 cap=2048 }, 1:{ span=1,5 mask=1,5 cap=2048 }, 3:{ span=3,7 mask=3,7 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 0:{ span=0-1,3-5,7 mask=0,4 cap=6144 }, 2:{ span=1-3,5-7 mask=2,6 cap=6144 }
[] CPU1 attaching sched-domain(s):
[] domain-0: span=1,5 level=DIE
[] groups: 1:{ span=1 }, 5:{ span=5 }
[] domain-1: span=0-2,4-6 level=NUMA
[] groups: 1:{ span=1,5 mask=1,5 cap=2048 }, 2:{ span=2,6 mask=2,6 cap=2048 }, 4:{ span=0,4 mask=0,4 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 1:{ span=0-2,4-6 mask=1,5 cap=6144 }, 3:{ span=0,2-4,6-7 mask=3,7 cap=6144 }
Observe how CPU0-domain1-group0 and CPU1-domain1-group4 are the
'same' but have a different id (0 vs 4).
To fix this, use the group balance CPU to select the SGC. This means
we have to compute the full mask for each CPU and require a second
temporary mask to store the group mask in (it otherwise lives in the
SGC).
The fixed topology looks like:
[] CPU0 attaching sched-domain(s):
[] domain-0: span=0,4 level=DIE
[] groups: 0:{ span=0 }, 4:{ span=4 }
[] domain-1: span=0-1,3-5,7 level=NUMA
[] groups: 0:{ span=0,4 mask=0,4 cap=2048 }, 1:{ span=1,5 mask=1,5 cap=2048 }, 3:{ span=3,7 mask=3,7 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 0:{ span=0-1,3-5,7 mask=0,4 cap=6144 }, 2:{ span=1-3,5-7 mask=2,6 cap=6144 }
[] CPU1 attaching sched-domain(s):
[] domain-0: span=1,5 level=DIE
[] groups: 1:{ span=1 }, 5:{ span=5 }
[] domain-1: span=0-2,4-6 level=NUMA
[] groups: 1:{ span=1,5 mask=1,5 cap=2048 }, 2:{ span=2,6 mask=2,6 cap=2048 }, 0:{ span=0,4 mask=0,4 cap=2048 }
[] domain-2: span=0-7 level=NUMA
[] groups: 1:{ span=0-2,4-6 mask=1,5 cap=6144 }, 3:{ span=0,2-4,6-7 mask=3,7 cap=6144 }
Debugged-by: Lauro Ramos Venancio <lvenanci@redhat.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Fixes: e3589f6c81e4 ("sched: Allow for overlapping sched_domain spans")
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2017-04-25 12:31:11 +00:00
|
|
|
zalloc_cpumask_var(&sched_domains_tmpmask2, GFP_KERNEL);
|
2017-04-25 13:29:40 +00:00
|
|
|
zalloc_cpumask_var(&fallback_doms, GFP_KERNEL);
|
|
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
arch_update_cpu_topology();
|
|
|
|
|
ndoms_cur = 1;
|
|
|
|
|
doms_cur = alloc_sched_domains(ndoms_cur);
|
|
|
|
|
if (!doms_cur)
|
|
|
|
|
doms_cur = &fallback_doms;
|
2017-10-27 02:42:37 +00:00
|
|
|
cpumask_and(doms_cur[0], cpu_map, housekeeping_cpumask(HK_FLAG_DOMAIN));
|
2017-02-01 12:10:18 +00:00
|
|
|
err = build_sched_domains(doms_cur[0], NULL);
|
|
|
|
|
|
|
|
|
|
return err;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Detach sched domains from a group of CPUs specified in cpu_map
|
|
|
|
|
* These CPUs will now be attached to the NULL domain
|
|
|
|
|
*/
|
|
|
|
|
static void detach_destroy_domains(const struct cpumask *cpu_map)
|
|
|
|
|
{
|
sched/topology: Allow sched_asym_cpucapacity to be disabled
While the static key is correctly initialized as being disabled, it will
remain forever enabled once turned on. This means that if we start with an
asymmetric system and hotplug out enough CPUs to end up with an SMP system,
the static key will remain set - which is obviously wrong. We should detect
this and turn off things like misfit migration and capacity aware wakeups.
As Quentin pointed out, having separate root domains makes this slightly
trickier. We could have exclusive cpusets that create an SMP island - IOW,
the domains within this root domain will not see any asymmetry. This means
we can't just disable the key on domain destruction, we need to count how
many asymmetric root domains we have.
Consider the following example using Juno r0 which is 2+4 big.LITTLE, where
two identical cpusets are created: they both span both big and LITTLE CPUs:
asym0 asym1
[ ][ ]
L L B L L B
$ cgcreate -g cpuset:asym0
$ cgset -r cpuset.cpus=0,1,3 asym0
$ cgset -r cpuset.mems=0 asym0
$ cgset -r cpuset.cpu_exclusive=1 asym0
$ cgcreate -g cpuset:asym1
$ cgset -r cpuset.cpus=2,4,5 asym1
$ cgset -r cpuset.mems=0 asym1
$ cgset -r cpuset.cpu_exclusive=1 asym1
$ cgset -r cpuset.sched_load_balance=0 .
(the CPU numbering may look odd because on the Juno LITTLEs are CPUs 0,3-5
and bigs are CPUs 1-2)
If we make one of those SMP (IOW remove asymmetry) by e.g. hotplugging its
big core, we would end up with an SMP cpuset and an asymmetric cpuset - the
static key must remain set, because we still have one asymmetric root domain.
With the above example, this could be done with:
$ echo 0 > /sys/devices/system/cpu/cpu2/online
Which would result in:
asym0 asym1
[ ][ ]
L L B L L
When both SMP and asymmetric cpusets are present, all CPUs will observe
sched_asym_cpucapacity being set (it is system-wide), but not all CPUs
observe asymmetry in their sched domain hierarchy:
per_cpu(sd_asym_cpucapacity, <any CPU in asym0>) == <some SD at DIE level>
per_cpu(sd_asym_cpucapacity, <any CPU in asym1>) == NULL
Change the simple key enablement to an increment, and decrement the key
counter when destroying domains that cover asymmetric CPUs.
Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Dietmar.Eggemann@arm.com
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: hannes@cmpxchg.org
Cc: lizefan@huawei.com
Cc: morten.rasmussen@arm.com
Cc: qperret@google.com
Cc: tj@kernel.org
Cc: vincent.guittot@linaro.org
Fixes: df054e8445a4 ("sched/topology: Add static_key for asymmetric CPU capacity optimizations")
Link: https://lkml.kernel.org/r/20191023153745.19515-3-valentin.schneider@arm.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2019-10-23 15:37:45 +00:00
|
|
|
unsigned int cpu = cpumask_any(cpu_map);
|
2017-02-01 12:10:18 +00:00
|
|
|
int i;
|
|
|
|
|
|
sched/topology: Allow sched_asym_cpucapacity to be disabled
While the static key is correctly initialized as being disabled, it will
remain forever enabled once turned on. This means that if we start with an
asymmetric system and hotplug out enough CPUs to end up with an SMP system,
the static key will remain set - which is obviously wrong. We should detect
this and turn off things like misfit migration and capacity aware wakeups.
As Quentin pointed out, having separate root domains makes this slightly
trickier. We could have exclusive cpusets that create an SMP island - IOW,
the domains within this root domain will not see any asymmetry. This means
we can't just disable the key on domain destruction, we need to count how
many asymmetric root domains we have.
Consider the following example using Juno r0 which is 2+4 big.LITTLE, where
two identical cpusets are created: they both span both big and LITTLE CPUs:
asym0 asym1
[ ][ ]
L L B L L B
$ cgcreate -g cpuset:asym0
$ cgset -r cpuset.cpus=0,1,3 asym0
$ cgset -r cpuset.mems=0 asym0
$ cgset -r cpuset.cpu_exclusive=1 asym0
$ cgcreate -g cpuset:asym1
$ cgset -r cpuset.cpus=2,4,5 asym1
$ cgset -r cpuset.mems=0 asym1
$ cgset -r cpuset.cpu_exclusive=1 asym1
$ cgset -r cpuset.sched_load_balance=0 .
(the CPU numbering may look odd because on the Juno LITTLEs are CPUs 0,3-5
and bigs are CPUs 1-2)
If we make one of those SMP (IOW remove asymmetry) by e.g. hotplugging its
big core, we would end up with an SMP cpuset and an asymmetric cpuset - the
static key must remain set, because we still have one asymmetric root domain.
With the above example, this could be done with:
$ echo 0 > /sys/devices/system/cpu/cpu2/online
Which would result in:
asym0 asym1
[ ][ ]
L L B L L
When both SMP and asymmetric cpusets are present, all CPUs will observe
sched_asym_cpucapacity being set (it is system-wide), but not all CPUs
observe asymmetry in their sched domain hierarchy:
per_cpu(sd_asym_cpucapacity, <any CPU in asym0>) == <some SD at DIE level>
per_cpu(sd_asym_cpucapacity, <any CPU in asym1>) == NULL
Change the simple key enablement to an increment, and decrement the key
counter when destroying domains that cover asymmetric CPUs.
Signed-off-by: Valentin Schneider <valentin.schneider@arm.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Dietmar Eggemann <dietmar.eggemann@arm.com>
Cc: Dietmar.Eggemann@arm.com
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: hannes@cmpxchg.org
Cc: lizefan@huawei.com
Cc: morten.rasmussen@arm.com
Cc: qperret@google.com
Cc: tj@kernel.org
Cc: vincent.guittot@linaro.org
Fixes: df054e8445a4 ("sched/topology: Add static_key for asymmetric CPU capacity optimizations")
Link: https://lkml.kernel.org/r/20191023153745.19515-3-valentin.schneider@arm.com
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2019-10-23 15:37:45 +00:00
|
|
|
if (rcu_access_pointer(per_cpu(sd_asym_cpucapacity, cpu)))
|
|
|
|
|
static_branch_dec_cpuslocked(&sched_asym_cpucapacity);
|
|
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
rcu_read_lock();
|
|
|
|
|
for_each_cpu(i, cpu_map)
|
|
|
|
|
cpu_attach_domain(NULL, &def_root_domain, i);
|
|
|
|
|
rcu_read_unlock();
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* handle null as "default" */
|
|
|
|
|
static int dattrs_equal(struct sched_domain_attr *cur, int idx_cur,
|
|
|
|
|
struct sched_domain_attr *new, int idx_new)
|
|
|
|
|
{
|
|
|
|
|
struct sched_domain_attr tmp;
|
|
|
|
|
|
|
|
|
|
/* Fast path: */
|
|
|
|
|
if (!new && !cur)
|
|
|
|
|
return 1;
|
|
|
|
|
|
|
|
|
|
tmp = SD_ATTR_INIT;
|
sched: Clean up and harmonize the coding style of the scheduler code base
A good number of small style inconsistencies have accumulated
in the scheduler core, so do a pass over them to harmonize
all these details:
- fix speling in comments,
- use curly braces for multi-line statements,
- remove unnecessary parentheses from integer literals,
- capitalize consistently,
- remove stray newlines,
- add comments where necessary,
- remove invalid/unnecessary comments,
- align structure definitions and other data types vertically,
- add missing newlines for increased readability,
- fix vertical tabulation where it's misaligned,
- harmonize preprocessor conditional block labeling
and vertical alignment,
- remove line-breaks where they uglify the code,
- add newline after local variable definitions,
No change in functionality:
md5:
1191fa0a890cfa8132156d2959d7e9e2 built-in.o.before.asm
1191fa0a890cfa8132156d2959d7e9e2 built-in.o.after.asm
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
2018-03-03 13:01:12 +00:00
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
return !memcmp(cur ? (cur + idx_cur) : &tmp,
|
|
|
|
|
new ? (new + idx_new) : &tmp,
|
|
|
|
|
sizeof(struct sched_domain_attr));
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* Partition sched domains as specified by the 'ndoms_new'
|
|
|
|
|
* cpumasks in the array doms_new[] of cpumasks. This compares
|
|
|
|
|
* doms_new[] to the current sched domain partitioning, doms_cur[].
|
|
|
|
|
* It destroys each deleted domain and builds each new domain.
|
|
|
|
|
*
|
|
|
|
|
* 'doms_new' is an array of cpumask_var_t's of length 'ndoms_new'.
|
|
|
|
|
* The masks don't intersect (don't overlap.) We should setup one
|
|
|
|
|
* sched domain for each mask. CPUs not in any of the cpumasks will
|
|
|
|
|
* not be load balanced. If the same cpumask appears both in the
|
|
|
|
|
* current 'doms_cur' domains and in the new 'doms_new', we can leave
|
|
|
|
|
* it as it is.
|
|
|
|
|
*
|
|
|
|
|
* The passed in 'doms_new' should be allocated using
|
|
|
|
|
* alloc_sched_domains. This routine takes ownership of it and will
|
|
|
|
|
* free_sched_domains it when done with it. If the caller failed the
|
|
|
|
|
* alloc call, then it can pass in doms_new == NULL && ndoms_new == 1,
|
|
|
|
|
* and partition_sched_domains() will fallback to the single partition
|
|
|
|
|
* 'fallback_doms', it also forces the domains to be rebuilt.
|
|
|
|
|
*
|
|
|
|
|
* If doms_new == NULL it will be replaced with cpu_online_mask.
|
|
|
|
|
* ndoms_new == 0 is a special case for destroying existing domains,
|
|
|
|
|
* and it will not create the default domain.
|
|
|
|
|
*
|
2019-07-19 13:59:53 +00:00
|
|
|
* Call with hotplug lock and sched_domains_mutex held
|
2017-02-01 12:10:18 +00:00
|
|
|
*/
|
2019-07-19 13:59:53 +00:00
|
|
|
void partition_sched_domains_locked(int ndoms_new, cpumask_var_t doms_new[],
|
|
|
|
|
struct sched_domain_attr *dattr_new)
|
2017-02-01 12:10:18 +00:00
|
|
|
{
|
2018-12-03 09:56:22 +00:00
|
|
|
bool __maybe_unused has_eas = false;
|
2017-02-01 12:10:18 +00:00
|
|
|
int i, j, n;
|
|
|
|
|
int new_topology;
|
|
|
|
|
|
2019-07-19 13:59:53 +00:00
|
|
|
lockdep_assert_held(&sched_domains_mutex);
|
2017-02-01 12:10:18 +00:00
|
|
|
|
|
|
|
|
/* Let the architecture update CPU core mappings: */
|
|
|
|
|
new_topology = arch_update_cpu_topology();
|
|
|
|
|
|
2017-08-08 10:16:24 +00:00
|
|
|
if (!doms_new) {
|
|
|
|
|
WARN_ON_ONCE(dattr_new);
|
|
|
|
|
n = 0;
|
|
|
|
|
doms_new = alloc_sched_domains(1);
|
|
|
|
|
if (doms_new) {
|
|
|
|
|
n = 1;
|
2017-10-27 02:42:37 +00:00
|
|
|
cpumask_and(doms_new[0], cpu_active_mask,
|
|
|
|
|
housekeeping_cpumask(HK_FLAG_DOMAIN));
|
2017-08-08 10:16:24 +00:00
|
|
|
}
|
|
|
|
|
} else {
|
|
|
|
|
n = ndoms_new;
|
|
|
|
|
}
|
2017-02-01 12:10:18 +00:00
|
|
|
|
|
|
|
|
/* Destroy deleted domains: */
|
|
|
|
|
for (i = 0; i < ndoms_cur; i++) {
|
|
|
|
|
for (j = 0; j < n && !new_topology; j++) {
|
2018-12-03 09:56:18 +00:00
|
|
|
if (cpumask_equal(doms_cur[i], doms_new[j]) &&
|
2019-07-19 13:59:55 +00:00
|
|
|
dattrs_equal(dattr_cur, i, dattr_new, j)) {
|
|
|
|
|
struct root_domain *rd;
|
|
|
|
|
|
|
|
|
|
/*
|
|
|
|
|
* This domain won't be destroyed and as such
|
|
|
|
|
* its dl_bw->total_bw needs to be cleared. It
|
|
|
|
|
* will be recomputed in function
|
|
|
|
|
* update_tasks_root_domain().
|
|
|
|
|
*/
|
|
|
|
|
rd = cpu_rq(cpumask_any(doms_cur[i]))->rd;
|
|
|
|
|
dl_clear_root_domain(rd);
|
2017-02-01 12:10:18 +00:00
|
|
|
goto match1;
|
2019-07-19 13:59:55 +00:00
|
|
|
}
|
2017-02-01 12:10:18 +00:00
|
|
|
}
|
|
|
|
|
/* No match - a current sched domain not in new doms_new[] */
|
|
|
|
|
detach_destroy_domains(doms_cur[i]);
|
|
|
|
|
match1:
|
|
|
|
|
;
|
|
|
|
|
}
|
|
|
|
|
|
|
|
|
|
n = ndoms_cur;
|
2017-08-08 10:16:24 +00:00
|
|
|
if (!doms_new) {
|
2017-02-01 12:10:18 +00:00
|
|
|
n = 0;
|
|
|
|
|
doms_new = &fallback_doms;
|
2017-10-27 02:42:37 +00:00
|
|
|
cpumask_and(doms_new[0], cpu_active_mask,
|
|
|
|
|
housekeeping_cpumask(HK_FLAG_DOMAIN));
|
2017-02-01 12:10:18 +00:00
|
|
|
}
|
|
|
|
|
|
|
|
|
|
/* Build new domains: */
|
|
|
|
|
for (i = 0; i < ndoms_new; i++) {
|
|
|
|
|
for (j = 0; j < n && !new_topology; j++) {
|
2018-12-03 09:56:18 +00:00
|
|
|
if (cpumask_equal(doms_new[i], doms_cur[j]) &&
|
|
|
|
|
dattrs_equal(dattr_new, i, dattr_cur, j))
|
2017-02-01 12:10:18 +00:00
|
|
|
goto match2;
|
|
|
|
|
}
|
|
|
|
|
/* No match - add a new doms_new */
|
|
|
|
|
build_sched_domains(doms_new[i], dattr_new ? dattr_new + i : NULL);
|
|
|
|
|
match2:
|
|
|
|
|
;
|
|
|
|
|
}
|
|
|
|
|
|
2018-12-03 09:56:21 +00:00
|
|
|
#if defined(CONFIG_ENERGY_MODEL) && defined(CONFIG_CPU_FREQ_GOV_SCHEDUTIL)
|
2018-12-03 09:56:18 +00:00
|
|
|
/* Build perf. domains: */
|
|
|
|
|
for (i = 0; i < ndoms_new; i++) {
|
2018-12-03 09:56:21 +00:00
|
|
|
for (j = 0; j < n && !sched_energy_update; j++) {
|
2018-12-03 09:56:18 +00:00
|
|
|
if (cpumask_equal(doms_new[i], doms_cur[j]) &&
|
2018-12-03 09:56:22 +00:00
|
|
|
cpu_rq(cpumask_first(doms_cur[j]))->rd->pd) {
|
|
|
|
|
has_eas = true;
|
2018-12-03 09:56:18 +00:00
|
|
|
goto match3;
|
2018-12-03 09:56:22 +00:00
|
|
|
}
|
2018-12-03 09:56:18 +00:00
|
|
|
}
|
|
|
|
|
/* No match - add perf. domains for a new rd */
|
2018-12-03 09:56:22 +00:00
|
|
|
has_eas |= build_perf_domains(doms_new[i]);
|
2018-12-03 09:56:18 +00:00
|
|
|
match3:
|
|
|
|
|
;
|
|
|
|
|
}
|
2018-12-03 09:56:22 +00:00
|
|
|
sched_energy_set(has_eas);
|
2018-12-03 09:56:18 +00:00
|
|
|
#endif
|
|
|
|
|
|
2017-02-01 12:10:18 +00:00
|
|
|
/* Remember the new sched domains: */
|
|
|
|
|
if (doms_cur != &fallback_doms)
|
|
|
|
|
free_sched_domains(doms_cur, ndoms_cur);
|
|
|
|
|
|
|
|
|
|
kfree(dattr_cur);
|
|
|
|
|
doms_cur = doms_new;
|
|
|
|
|
dattr_cur = dattr_new;
|
|
|
|
|
ndoms_cur = ndoms_new;
|
|
|
|
|
|
2021-03-25 10:31:20 +00:00
|
|
|
update_sched_domain_debugfs();
|
2019-07-19 13:59:53 +00:00
|
|
|
}
|
2017-02-01 12:10:18 +00:00
|
|
|
|
2019-07-19 13:59:53 +00:00
|
|
|
/*
|
|
|
|
|
* Call with hotplug lock held
|
|
|
|
|
*/
|
|
|
|
|
void partition_sched_domains(int ndoms_new, cpumask_var_t doms_new[],
|
|
|
|
|
struct sched_domain_attr *dattr_new)
|
|
|
|
|
{
|
|
|
|
|
mutex_lock(&sched_domains_mutex);
|
|
|
|
|
partition_sched_domains_locked(ndoms_new, doms_new, dattr_new);
|
2017-02-01 12:10:18 +00:00
|
|
|
mutex_unlock(&sched_domains_mutex);
|
|
|
|
|
}
|
