During remove & rescan cycle, PCI subsystem will recalculate and adjust
the bridge window sizing that was initially done by "BIOS". The size
calculation is based on the required alignment of the largest resource
among the downstream resources as per pbus_size_mem() (unimportant or
zero parameters marked with "..."):
min_align = calculate_mem_align(aligns, max_order);
size0 = calculate_memsize(size, ..., min_align);
inside calculate_memsize(), for the largest alignment:
min_align = align1 >> 1;
...
return min_align;
and then in calculate_memsize():
return ALIGN(max(size, ...), align);
If the original bridge window sizing tried to conserve space, this will
lead to massive increase of the required bridge window size when the
downstream has a large disparity in BAR sizes. E.g., with 16MiB and
16GiB BARs this results in 24GiB bridge window size even if 16MiB BAR
does not require gigabytes of space to fit.
When doing remove & rescan for a bus that contains such a PCI device, a
larger bridge window is suddenly required on rescan but when there is a
bridge window upstream that is already assigned based on the original
size, it cannot be enlarged to the new requirement. This causes the
allocation of the bridge window to fail (0x600000000 > 0x400ffffff):
pci 0000:02:01.0: PCI bridge to [bus 03]
pci 0000:02:01.0: bridge window [mem 0x40400000-0x405fffff]
pci 0000:02:01.0: bridge window [mem 0x6000000000-0x6400ffffff 64bit pref]
pci 0000:01:00.0: PCI bridge to [bus 02-04]
pci 0000:01:00.0: bridge window [mem 0x40400000-0x406fffff]
pci 0000:01:00.0: bridge window [mem 0x6000000000-0x6400ffffff 64bit pref]
pci 0000:03:00.0: device released
pci 0000:02:01.0: device released
pcieport 0000:01:00.0: scanning [bus 02-04] behind bridge, pass 0
pci 0000:02:01.0: PCI bridge to [bus 03]
pci 0000:02:01.0: bridge window [mem 0x40400000-0x405fffff]
pci 0000:02:01.0: bridge window [mem 0x6000000000-0x6400ffffff 64bit pref]
pci 0000:02:01.0: scanning [bus 03-03] behind bridge, pass 0
pci 0000:03:00.0: BAR 0 [mem 0x6400000000-0x6400ffffff 64bit pref]
pci 0000:03:00.0: BAR 2 [mem 0x6000000000-0x63ffffffff 64bit pref]
pci 0000:03:00.0: ROM [mem 0x40400000-0x405fffff pref]
pci 0000:02:01.0: PCI bridge to [bus 03]
pci 0000:02:01.0: scanning [bus 03-03] behind bridge, pass 1
pcieport 0000:01:00.0: scanning [bus 02-04] behind bridge, pass 1
pci 0000:02:01.0: bridge window [mem size 0x600000000 64bit pref]: can't assign; no space
pci 0000:02:01.0: bridge window [mem size 0x600000000 64bit pref]: failed to assign
pci 0000:02:01.0: bridge window [mem 0x40400000-0x405fffff]: assigned
pci 0000:03:00.0: BAR 2 [mem size 0x400000000 64bit pref]: can't assign; no space
pci 0000:03:00.0: BAR 2 [mem size 0x400000000 64bit pref]: failed to assign
pci 0000:03:00.0: BAR 0 [mem size 0x01000000 64bit pref]: can't assign; no space
pci 0000:03:00.0: BAR 0 [mem size 0x01000000 64bit pref]: failed to assign
pci 0000:03:00.0: ROM [mem 0x40400000-0x405fffff pref]: assigned
pci 0000:02:01.0: PCI bridge to [bus 03]
pci 0000:02:01.0: bridge window [mem 0x40400000-0x405fffff]
This is a major surprise for users who are suddenly left with a device that
was working fine with the original bridge window sizing.
Even if the already assigned bridge window could be enlarged by
reallocation in some cases (something the current code does not attempt
to do), it is not possible in general case and the large amount of
wasted space at the tail of the bridge window may lead to other
resource exhaustion problems on Root Complex level (think of multiple
PCIe cards with VFs and BAR size disparity in a single system).
PCI BARs only need natural alignment (PCIe r6.1, sec 7.5.1.2.1) and bridge
memory windows need 1MiB (sec 7.5.1.3). The current bridge window tail
alignment rule was introduced in the commit 5d0a8965aea9 ("[PATCH] 2.5.14:
New PCI allocation code (alpha, arm, parisc) [2/2]") that only states:
"pbus_size_mem: core stuff; tested with randomly generated sets of
resources". It does not explain the motivation for the extra tail space
allocated that is not truly needed by the downstream resources. As such, it
is far from clear if it ever has been required by any HW.
To prevent devices with BAR size disparity from becoming unusable after
remove & rescan cycle, attempt to do a truly minimal allocation for memory
resources if needed. First check if the normally calculated bridge window
will not fit into an already assigned upstream resource. In such case, try
with relaxed bridge window tail sizing rules instead where no extra tail
space is requested beyond what the downstream resources require. Only
enforce the alignment requirement of the bridge window itself (normally
1MiB).
With this patch, the resources are successfully allocated:
pci 0000:02:01.0: PCI bridge to [bus 03]
pci 0000:02:01.0: scanning [bus 03-03] behind bridge, pass 1
pcieport 0000:01:00.0: scanning [bus 02-04] behind bridge, pass 1
pcieport 0000:01:00.0: Assigned bridge window [mem 0x6000000000-0x6400ffffff 64bit pref] to [bus 02-04] cannot fit 0x600000000 required for 0000:02:01.0 bridging to [bus 03]
pci 0000:02:01.0: bridge window [mem 0x6000000000-0x6400ffffff 64bit pref] to [bus 03] requires relaxed alignment rules
pcieport 0000:01:00.0: Assigned bridge window [mem 0x40400000-0x406fffff] to [bus 02-04] free space at [mem 0x40400000-0x405fffff]
pci 0000:02:01.0: bridge window [mem 0x6000000000-0x6400ffffff 64bit pref]: assigned
pci 0000:02:01.0: bridge window [mem 0x40400000-0x405fffff]: assigned
pci 0000:03:00.0: BAR 2 [mem 0x6000000000-0x63ffffffff 64bit pref]: assigned
pci 0000:03:00.0: BAR 0 [mem 0x6400000000-0x6400ffffff 64bit pref]: assigned
pci 0000:03:00.0: ROM [mem 0x40400000-0x405fffff pref]: assigned
pci 0000:02:01.0: PCI bridge to [bus 03]
pci 0000:02:01.0: bridge window [mem 0x40400000-0x405fffff]
pci 0000:02:01.0: bridge window [mem 0x6000000000-0x6400ffffff 64bit pref]
This patch draws inspiration from the initial investigations and work by
Mika Westerberg.
Closes: https://bugzilla.kernel.org/show_bug.cgi?id=216795
Link: https://lore.kernel.org/linux-pci/20190812144144.2646-1-mika.westerberg@linux.intel.com/
Fixes: 5d0a8965aea9 ("[PATCH] 2.5.14: New PCI allocation code (alpha, arm, parisc) [2/2]")
Link: https://lore.kernel.org/r/20240507102523.57320-9-ilpo.jarvinen@linux.intel.com
Tested-by: Lidong Wang <lidong.wang@intel.com>
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Mika Westerberg <mika.westerberg@linux.intel.com>
pbus_size_mem() keeps the size of the optional resources in
children_add_size. When calculating the PCI bridge window size,
calculate_memsize() lower bounds size by old_size before adding
children_add_size and performing the window size alignment. This
results in double counting for the resources in children_add_size
because old_size may be based on the previous size of the bridge
window after it has already included children_add_size (that is,
size1 in pbus_size_mem() from an earlier invocation of that
function).
As a result, on repeated remove of the bus & rescan cycles the resource
size keeps increasing when children_add_size is non-zero as can be seen
from this extract:
iomem0: 23fffd00000-23fffdfffff : PCI Bus 0000:03 # 1MiB
iomem1: 20000000000-200001fffff : PCI Bus 0000:03 # 2MiB
iomem2: 20000000000-200002fffff : PCI Bus 0000:03 # 3MiB
iomem3: 20000000000-200003fffff : PCI Bus 0000:03 # 4MiB
iomem4: 20000000000-200004fffff : PCI Bus 0000:03 # 5MiB
Solve the double counting by moving old_size check later in
calculate_memsize() so that children_add_size is already accounted for.
After the patch, the bridge window retains its size as expected:
iomem0: 23fffd00000-23fffdfffff : PCI Bus 0000:03 # 1MiB
iomem1: 20000000000-200000fffff : PCI Bus 0000:03 # 1MiB
iomem2: 20000000000-200000fffff : PCI Bus 0000:03 # 1MiB
Fixes: a4ac9fea01 ("PCI : Calculate right add_size")
Link: https://lore.kernel.org/r/20240507102523.57320-2-ilpo.jarvinen@linux.intel.com
Tested-by: Lidong Wang <lidong.wang@intel.com>
Signed-off-by: Ilpo Järvinen <ilpo.jarvinen@linux.intel.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Use the pci_resource_name() to get the name of the resource and use it
while printing log messages.
[bhelgaas: rename to match struct resource * names, also use names in other
BAR messages]
Link: https://lore.kernel.org/r/20211106112606.192563-3-puranjay12@gmail.com
Signed-off-by: Puranjay Mohan <puranjay12@gmail.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Fix a section mismatch warning on Sparc 32-bit:
WARNING: modpost: vmlinux: section mismatch in reference: leon_pci_init+0xf8 (section: .text) -> pci_assign_unassigned_resources (section: .init.text)
This is due to this comment from arch/sparc/kernel/leon_pci.c:
The LEON architecture does not rely on a BIOS or bootloader to setup PCI
for us. The Linux generic routines are used to setup resources, reset
values of configuration-space register settings are preserved.
Link: https://lore.kernel.org/r/20230925042316.15415-1-rdunlap@infradead.org
Signed-off-by: Randy Dunlap <rdunlap@infradead.org>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: sparclinux@vger.kernel.org
Refactor pci_bus_for_each_resource() in the same way as
pci_dev_for_each_resource(). This allows the index to be hidden inside the
implementation so the caller can omit it when it's not used otherwise.
No functional changes intended.
Link: https://lore.kernel.org/r/20230330162434.35055-6-andriy.shevchenko@linux.intel.com
Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Krzysztof Wilczyński <kw@linux.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Instead of open-coding it everywhere introduce a tiny helper that can be
used to iterate over each resource of a PCI device, and convert the most
obvious users into it.
While at it drop doubled empty line before pdev_sort_resources().
No functional changes intended.
Suggested-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Link: https://lore.kernel.org/r/20230330162434.35055-4-andriy.shevchenko@linux.intel.com
Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Signed-off-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Krzysztof Wilczyński <kw@linux.com>
Previously we distributed spare resources only upon hot-add, so if the
initial root bus scan found devices that had not been fully configured by
the BIOS, we allocated only enough resources to cover what was then
present. If some of those devices were hotplug bridges, we did not leave
any additional resource space for future expansion.
Distribute the available resources for root buses, too, to make this work
the same way as the normal hotplug case.
A previous commit to do this was reverted due to a regression reported by
Jonathan Cameron:
e96e27fc6f ("PCI: Distribute available resources for root buses, too")
5632e2beaf ("Revert "PCI: Distribute available resources for root buses, too"")
This commit changes pci_bridge_resources_not_assigned() to work with
bridges that do not have all the resource windows programmed by the boot
firmware (previously we expected all I/O, memory and prefetchable memory
were programmed).
Link: https://bugzilla.kernel.org/show_bug.cgi?id=216000
Link: https://lore.kernel.org/r/20220905080232.36087-5-mika.westerberg@linux.intel.com
Link: https://lore.kernel.org/r/20230131092405.29121-4-mika.westerberg@linux.intel.com
Reported-by: Chris Chiu <chris.chiu@canonical.com>
Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
A PCI bridge may reside on a bus with other devices as well. The resource
distribution code does not take this into account and therefore it expands
the bridge resource windows too much, not leaving space for the other
devices (or functions of a multifunction device). This leads to an issue
that Jonathan reported when running QEMU with the following topology (QEMU
parameters):
-device pcie-root-port,port=0,id=root_port13,chassis=0,slot=2 \
-device x3130-upstream,id=sw1,bus=root_port13,multifunction=on \
-device e1000,bus=root_port13,addr=0.1 \
-device xio3130-downstream,id=fun1,bus=sw1,chassis=0,slot=3 \
-device e1000,bus=fun1
The first e1000 NIC here is another function in the switch upstream port.
This leads to following errors:
pci 0000:00:04.0: bridge window [mem 0x10200000-0x103fffff] to [bus 02-04]
pci 0000:02:00.0: bridge window [mem 0x10200000-0x103fffff] to [bus 03-04]
pci 0000:02:00.1: BAR 0: failed to assign [mem size 0x00020000]
e1000 0000:02:00.1: can't ioremap BAR 0: [??? 0x00000000 flags 0x0]
Fix this by taking into account bridge windows, device BARs and SR-IOV PF
BARs on the bus (PF BARs include space for VF BARS so only account PF
BARs), including the ones belonging to bridges themselves if it has any.
Link: https://lore.kernel.org/linux-pci/20221014124553.0000696f@huawei.com/
Link: https://lore.kernel.org/linux-pci/6053736d-1923-41e7-def9-7585ce1772d9@ixsystems.com/
Link: https://lore.kernel.org/r/20230131092405.29121-3-mika.westerberg@linux.intel.com
Reported-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Reported-by: Alexander Motin <mav@ixsystems.com>
Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
After division the extra resource space per hotplug bridge may not be
aligned according to the window alignment, so align it before passing it
down for further distribution.
Link: https://lore.kernel.org/r/20230131092405.29121-2-mika.westerberg@linux.intel.com
Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
This reverts commit e96e27fc6f.
Jonathan reported that this commit broke this topology, where all the space
available on bus 02 was assigned to the 02:00.0 bridge window, leaving none
for the e1000 device at 02:00.1:
pci 0000:00:04.0: bridge window [mem 0x10200000-0x103fffff] to [bus 02-04]
pci 0000:02:00.0: bridge window [mem 0x10200000-0x103fffff] to [bus 03-04]
pci 0000:02:00.1: BAR 0: failed to assign [mem size 0x00020000]
e1000 0000:02:00.1: can't ioremap BAR 0: [??? 0x00000000 flags 0x0]
Link: https://lore.kernel.org/r/20221014124553.0000696f@huawei.com
Reported-by: Jonathan Cameron <Jonathan.Cameron@huawei.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Drop two empty lines from pci_scan_child_bus_extend() and correct
indentation in pci_bridge_distribute_available_resources() to better
follow the kernel coding style.
No functional impact.
Link: https://lore.kernel.org/r/20220905080232.36087-6-mika.westerberg@linux.intel.com
Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Previously we distributed spare resources only upon hot-add, so if the
initial root bus scan found devices that had not been fully configured by
the BIOS, we allocated only enough resources to cover what was then
present. If some of those devices were hotplug bridges, we did not leave
any additional resource space for future expansion.
Distribute the available resources for root buses, too, to make this work
the same way as the normal hotplug case.
Link: https://bugzilla.kernel.org/show_bug.cgi?id=216000
Link: https://lore.kernel.org/r/20220905080232.36087-5-mika.westerberg@linux.intel.com
Reported-by: Chris Chiu <chris.chiu@canonical.com>
Tested-by: Chris Chiu <chris.chiu@canonical.com>
Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
We need to be able to call pci_bridge_distribute_available_resources()
from this function so move it accordingly to avoid need for forward
declaration.
No functional impact.
Link: https://lore.kernel.org/r/20220905080232.36087-4-mika.westerberg@linux.intel.com
Signed-off-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Andy Shevchenko <andriy.shevchenko@linux.intel.com>
Remove variables and assignments that are never used.
Found by Krzysztof using cppcheck, e.g.,
$ cppcheck --enable=all --force
uselessAssignmentPtrArg drivers/pci/proc.c:102 Assignment of function parameter has no effect outside the function. Did you forget dereferencing it?
unreadVariable drivers/pci/setup-bus.c:1528 Variable 'old_flags' is assigned a value that is never used.
Reported-by: Krzysztof Wilczyński <kw@linux.com>
Link: https://lore.kernel.org/r/20220313192933.434746-2-helgaas@kernel.org
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Current kernel reports that BARs larger than 128GB, e.g., this 4TB BAR, are
disabled:
pci 0000:01:00.0: disabling BAR 4: [mem 0x00000000-0x3ffffffffff 64bit pref] (bad alignment 0x40000000000)
Increase the maximum BAR size from 128GB to 8TB for future expansion.
[bhelgaas: commit log]
Link: https://lore.kernel.org/r/20220118092117.10089-1-liudongdong3@huawei.com
Signed-off-by: Dongdong Liu <liudongdong3@huawei.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
The bare "unsigned" type implicitly means "unsigned int", but the preferred
coding style is to use the complete type name.
Update the bare use of "unsigned" to the preferred "unsigned int".
No change to functionality intended.
See a1ce18e4f9 ("checkpatch: warn on bare unsigned or signed declarations
without int").
Link: https://lore.kernel.org/r/20211013014136.1117543-1-kw@linux.com
Signed-off-by: Krzysztof Wilczyński <kw@linux.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Use kzalloc() instead of kmalloc() in the panic message.
[bhelgaas: drop similar ibmphp_pci.c change since it's not obviously
correct]
Link: https://lore.kernel.org/r/1594279708-34369-1-git-send-email-wang.yi59@zte.com.cn
Signed-off-by: Liao Pingfang <liao.pingfang@zte.com.cn>
Signed-off-by: Yi Wang <wang.yi59@zte.com.cn>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
- Constify struct pci_ecam_ops (Rob Herring)
- Support building as modules (Rob Herring)
- Eliminate wrappers for pci_host_common_probe() by using DT match table
data (Rob Herring)
* remotes/lorenzo/pci/host-generic:
PCI: host-generic: Eliminate pci_host_common_probe wrappers
PCI: host-generic: Support building as modules
PCI: Constify struct pci_ecam_ops
# Conflicts:
# drivers/pci/controller/dwc/pcie-hisi.c
Use bridge resource definitions instead of using the PCI_BRIDGE_RESOURCES
constant with an integer offeset.
Link: https://lore.kernel.org/r/20200520183411.1534621-2-kw@linux.com
Signed-off-by: Krzysztof Wilczynski <kw@linux.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Enable building host-generic and its host-common dependency as a
module.
Link: https://lore.kernel.org/r/20200409234923.21598-3-robh@kernel.org
Signed-off-by: Rob Herring <robh@kernel.org>
Signed-off-by: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Acked-by: Will Deacon <will@kernel.org>
Acked-by: Bjorn Helgaas <bhelgaas@google.com>
Cc: Lorenzo Pieralisi <lorenzo.pieralisi@arm.com>
Cc: Andrew Murray <amurray@thegoodpenguin.co.uk>
Cc: Bjorn Helgaas <bhelgaas@google.com>
Cc: Will Deacon <will@kernel.org>
Cc: linux-pci@vger.kernel.org
Cc: linux-arm-kernel@lists.infradead.org
In certain cases we should be able to enumerate IO and MEM ranges of all
PCI devices installed in the system, and then set respective host bridge
apertures basing on calculated size and alignment. Particularly when
firmware is broken and fails to assign bridge windows properly, like on
Alpha UP1500 platform.
Actually, almost everything is already in place, and required changes are
minimal:
- add "size_windows" flag to struct pci_host_bridge: when set, it
instructs __pci_bus_size_bridges() to continue with the root bus;
- in the __pci_bus_size_bridges() path: add checks for bus->self,
as it can legitimately be null for the root bus.
Link: https://lore.kernel.org/r/20200314194355.GA12510@mail.rc.ru
Tested-by: Matt Turner <mattst88@gmail.com>
Signed-off-by: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Remove checks for resource size in adjust_bridge_window(). This is
necessary to allow pci_bus_distribute_available_resources() to function
when the kernel parameter "pci=hpmemsize=nn[KMG]" is used to allocate
resources. Because the kernel parameter sets the size of all hotplug
bridges to be the same, there are problems when nested hotplug bridges are
encountered. Fitting a downstream hotplug bridge with size X and normal
bridges with non-zero size Y into parent hotplug bridge with size X is
impossible, and hence the downstream hotplug bridge needs to shrink to fit
into its parent.
Add check for if bridge is extended or shrunken and reflect that in the
call to pci_dbg().
Reset the resource if its new size is zero (if we have run out of a bridge
window resource) to prevent the PCI resource assignment code from
attempting to assign a zero-sized resource.
Link: https://lore.kernel.org/r/PSXP216MB0438D3E2CFE64EBAA32AF691803C0@PSXP216MB0438.KORP216.PROD.OUTLOOK.COM
Signed-off-by: Nicholas Johnson <nicholas.johnson-opensource@outlook.com.au>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Change adjust_bridge_window() to set resource size directly instead of
using additional resource lists.
Because additional resource lists are optional resources, any algorithm
that requires guaranteed allocation that uses them cannot be guaranteed to
work.
Remove the resource from add_list, as a zero-sized additional resource is
redundant.
Update comment in pci_bus_distribute_available_resources() to reflect the
above changes.
Link: https://lore.kernel.org/r/PSXP216MB04386BA48874B56BC5CB0292803C0@PSXP216MB0438.KORP216.PROD.OUTLOOK.COM
Signed-off-by: Nicholas Johnson <nicholas.johnson-opensource@outlook.com.au>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Rename extend_bridge_window() to adjust_bridge_window() to prepare for the
fact that the window will be able to shrink. No functional change
intended.
Link: https://lore.kernel.org/r/PSXP216MB0438C47B3473D0C9DE531F18803C0@PSXP216MB0438.KORP216.PROD.OUTLOOK.COM
Signed-off-by: Nicholas Johnson <nicholas.johnson-opensource@outlook.com.au>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Mika Westerberg <mika.westerberg@linux.intel.com>
In extend_bridge_window(), change "available" parameter name to "new_size".
This makes more sense as this parameter represents the new size for the
window. No functional change intended.
Link: https://lore.kernel.org/r/PSXP216MB043853617ECA4118C472A417803C0@PSXP216MB0438.KORP216.PROD.OUTLOOK.COM
Signed-off-by: Nicholas Johnson <nicholas.johnson-opensource@outlook.com.au>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Change pci_bus_distribute_available_resources() to better handle bridges
with different resource alignment requirements.
The arguments io, mmio and mmio_pref represent the start and end
addresses of resource, into which we must fit the current bridge window.
The steps taken by pci_bus_distribute_available_resources():
- For io, mmio and mmio_pref, increase .start to align with the alignment
of the current bridge window (otherwise the current bridge window may
not fit within the available range).
- For io, mmio and mmio_pref, adjust the current bridge window to the
size after the above.
- Count the number of hotplug bridges and normal bridges on this bus.
- If the total number of bridges is one, give that bridge all of the
resources and return.
- If there are no hotplug bridges, return.
- For io, mmio and mmio_pref, increase .start by the amount required for
each bridge resource on the bus for non hotplug bridges, giving extra
room to make up for alignment of those resources.
- For io, mmio and mmio_pref, calculate the resource size per hotplug
bridge which is available after the previous steps.
- For io, mmio and mmio_pref, distribute the resources to each hotplug
bridge, with the sizes calculated above.
The motivation for fixing this is enabling devices that require greater
than 1MB alignment. This fixes the case where the user hot-adds devices
with BAR alignment >1MB and Linux fails to assign resources to it.
Link: https://bugzilla.kernel.org/show_bug.cgi?id=199581
Link: https://lore.kernel.org/r/PSXP216MB0438C2BFD0FD3691ED9C83F4803C0@PSXP216MB0438.KORP216.PROD.OUTLOOK.COM
Reported-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Signed-off-by: Nicholas Johnson <nicholas.johnson-opensource@outlook.com.au>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Mika Westerberg <mika.westerberg@linux.intel.com>
In pci_bus_distribute_available_resources(), use resource_size() rather
than the local available_io, etc. No functional change intended; this just
makes the preceding patch smaller.
[bhelgaas: extracted from https://lore.kernel.org/r/PSXP216MB0438587C47CBEDF365B1EA27803C0@PSXP216MB0438.KORP216.PROD.OUTLOOK.COM]
Signed-off-by: Nicholas Johnson <nicholas.johnson-opensource@outlook.com.au>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Change pci_bus_distribute_available_resources() arguments from
resource_size_t to struct resource to add more information required to get
the alignment correct for bridge windows with alignment >1M.
We require (size, alignment), instead of just (size) which is what is
currently available. The change from resource_size_t to struct resource
does just that.
Note that the struct resource arguments are passed by value and not by
reference. We do not want to pass by reference and change the resource size
of the parent bridge window. We only want the size information.
No functional change intended.
Link: https://lore.kernel.org/r/PSXP216MB0438587C47CBEDF365B1EA27803C0@PSXP216MB0438.KORP216.PROD.OUTLOOK.COM
[bhelgaas: split parts to other patches to reduce the size of this one]
Signed-off-by: Nicholas Johnson <nicholas.johnson-opensource@outlook.com.au>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Mika Westerberg <mika.westerberg@linux.intel.com>
In pci_bus_distribute_available_resources(), rename:
io => io_per_hp
mmio => mmio_per_hp
mmio_pref => mmio_pref_per_hp
No functional change; this is just to make a subsequent patch smaller.
[bhelgaas: extracted from https://lore.kernel.org/r/PSXP216MB0438587C47CBEDF365B1EA27803C0@PSXP216MB0438.KORP216.PROD.OUTLOOK.COM]
Signed-off-by: Nicholas Johnson <nicholas.johnson-opensource@outlook.com.au>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Some PCI bridges implement BARs in addition to bridge windows. For
example, here's a PLX switch:
04:00.0 PCI bridge: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI
Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA (rev ca)
(prog-if 00 [Normal decode])
Flags: bus master, fast devsel, latency 0, IRQ 30, NUMA node 0
Memory at 90a00000 (32-bit, non-prefetchable) [size=256K]
Bus: primary=04, secondary=05, subordinate=0a, sec-latency=0
I/O behind bridge: 00002000-00003fff
Memory behind bridge: 90000000-909fffff
Prefetchable memory behind bridge: 0000380000800000-0000380000bfffff
Previously, when the kernel assigned resource addresses (with the
pci=realloc command line parameter, for example) it could clear the struct
resource corresponding to the BAR. When this happened, lspci would report
this BAR as "ignored":
Region 0: Memory at <ignored> (32-bit, non-prefetchable) [size=256K]
This is because the kernel reports a zero start address and zero flags
in the corresponding sysfs resource file and in /proc/bus/pci/devices.
Investigation with 'lspci -x', however, shows the BIOS-assigned address
will still be programmed in the device's BAR registers.
It's clearly a bug that the kernel lost track of the BAR value, but in most
cases, this still won't result in a visible issue because nothing uses the
memory, so nothing is affected. However, when an IOMMU is in use, it will
not reserve this space in the IOVA because the kernel no longer thinks the
range is valid. (See dmar_init_reserved_ranges() for the Intel
implementation of this.)
Without the proper reserved range, a DMA mapping may allocate an IOVA that
matches a bridge BAR, which results in DMA accesses going to the BAR
instead of the intended RAM.
The problem was in pci_assign_unassigned_root_bus_resources(). When any
resource from a bridge device fails to get assigned, the code set the
resource's flags to zero. This makes sense for bridge windows, as they
will be re-enabled later, but for regular BARs, it makes the kernel
permanently lose track of the fact that they decode address space.
Change pci_assign_unassigned_root_bus_resources() and
pci_assign_unassigned_bridge_resources() so they only clear "res->flags"
for bridge *windows*, not bridge BARs.
Fixes: da7822e5ad ("PCI: update bridge resources to get more big ranges when allocating space (again)")
Link: https://lore.kernel.org/r/20200108213208.4612-1-logang@deltatee.com
[bhelgaas: commit log, check for pci_is_bridge()]
Reported-by: Kit Chow <kchow@gigaio.com>
Signed-off-by: Logan Gunthorpe <logang@deltatee.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Previously, the kernel sometimes assigned more MMIO or MMIO_PREF space than
desired. For example, if the user requested 128M of space with
"pci=realloc,hpmemsize=128M", we sometimes assigned 256M:
pci 0000:06:01.0: BAR 14: assigned [mem 0x90100000-0xa00fffff] = 256M
pci 0000:06:04.0: BAR 14: assigned [mem 0xa0200000-0xb01fffff] = 256M
With this patch applied:
pci 0000:06:01.0: BAR 14: assigned [mem 0x90100000-0x980fffff] = 128M
pci 0000:06:04.0: BAR 14: assigned [mem 0x98200000-0xa01fffff] = 128M
This happened when in the first pass, the MMIO_PREF succeeded but the MMIO
failed. In the next pass, because MMIO_PREF was already assigned, the
attempt to assign MMIO_PREF returned an error code instead of success
(nothing more to do, already allocated). Hence, the size which was actually
allocated, but thought to have failed, was placed in the MMIO window.
The bug resulted in the MMIO_PREF being added to the MMIO window, which
meant doubling if MMIO_PREF size = MMIO size. With a large MMIO_PREF, the
MMIO window would likely fail to be assigned altogether due to lack of
32-bit address space.
Change find_free_bus_resource() to do the following:
- Return first unassigned resource of the correct type.
- If there is none, return first assigned resource of the correct type.
- If none of the above, return NULL.
Returning an assigned resource of the correct type allows the caller to
distinguish between already assigned and no resource of the correct type.
Add checks in pbus_size_io() and pbus_size_mem() to return success if
resource returned from find_free_bus_resource() is already allocated.
This avoids pbus_size_io() and pbus_size_mem() returning error code to
__pci_bus_size_bridges() when a resource has been successfully assigned in
a previous pass. This fixes the existing behaviour where space for a
resource could be reserved multiple times in different parent bridge
windows.
Link: https://lore.kernel.org/lkml/20190531171216.20532-2-logang@deltatee.com/T/#u
Link: https://bugzilla.kernel.org/show_bug.cgi?id=203243
Link: https://lore.kernel.org/r/PS2P216MB075563AA6AD242AA666EDC6A80760@PS2P216MB0755.KORP216.PROD.OUTLOOK.COM
Reported-by: Kit Chow <kchow@gigaio.com>
Reported-by: Nicholas Johnson <nicholas.johnson-opensource@outlook.com.au>
Signed-off-by: Nicholas Johnson <nicholas.johnson-opensource@outlook.com.au>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
The existing "pci=hpmemsize=nn[KMG]" kernel parameter overrides the default
size of both the non-prefetchable and the prefetchable MMIO windows for
hotplug bridges.
Add "pci=hpmmiosize=nn[KMG]" to override the default size of only the
non-prefetchable MMIO window.
Add "pci=hpmmioprefsize=nn[KMG]" to override the default size of only the
prefetchable MMIO window.
Link: https://lore.kernel.org/r/SL2P216MB0187E4D0055791957B7E2660806B0@SL2P216MB0187.KORP216.PROD.OUTLOOK.COM
Signed-off-by: Nicholas Johnson <nicholas.johnson-opensource@outlook.com.au>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Mika Westerberg <mika.westerberg@linux.intel.com>
pci_reassign_bridge_resources() can be called by pci_resize_resource() at
runtime, it walks the PCI tree up and down, and it isn't currently
protected against any changes or hotplug operation.
Hold the pci_bus_sem to protect it.
Link: https://lore.kernel.org/r/7339fd73ccaf58552737ab10008333fd9f7723f2.camel@kernel.crashing.org
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Writing loop conditions as "i < NUM" is a common C idiom; using "i <= END"
is unusual and thus prone to errors. Change loops to use the former.
Link: https://lore.kernel.org/r/20190806140715.19847-1-efremov@linux.com
Signed-off-by: Denis Efremov <efremov@linux.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Kuppuswamy Sathyanarayanan <sathyanarayanan.kuppuswamy@linux.intel.com>
If "hotplug_bridges == 0", "!dev->is_hotplug_bridge" is always true, so the
loop that divides the remaining resources among hotplug-capable bridges
does nothing.
Check for "hotplug_bridges == 0" earlier, so we don't even have to compute
the amount of remaining resources. No functional change intended.
Link: https://lore.kernel.org/r/PS2P216MB0642C7A485649D2D787A1C6F80000@PS2P216MB0642.KORP216.PROD.OUTLOOK.COM
Link: https://lore.kernel.org/r/20190622210310.180905-3-helgaas@kernel.org
Signed-off-by: Nicholas Johnson <nicholas.johnson-opensource@outlook.com.au>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Reviewed-by: Logan Gunthorpe <logang@deltatee.com>
Reviewed-by: Mika Westerberg <mika.westerberg@linux.intel.com>
Prevent auto-enabling of bridges reallocation when the FW tells us that the
initial configuration must be preserved for a given host bridge.
Link: https://lore.kernel.org/r/20190615002359.29577-3-benh@kernel.crashing.org
Signed-off-by: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Replace dev_printk(KERN_DEBUG) with dev_info(), etc to be more consistent
with other logging and avoid checkpatch warnings.
The KERN_DEBUG messages could be converted to dev_dbg(), but that depends
on CONFIG_DYNAMIC_DEBUG and DEBUG, and we want most of these messages to
*always* be in the dmesg log.
Link: https://lore.kernel.org/lkml/1555733240-19875-1-git-send-email-mohankumar718@gmail.com
Signed-off-by: Mohan Kumar <mohankumar718@gmail.com>
[bhelgaas: commit log]
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
The PCI configuration space header type tells us whether the device is a
bridge, a CardBus bridge, or a normal device, and defines the layout of the
rest of the header (PCI r3.0 sec 6.1, PCIe r4.0 sec 7.5.1.1.9).
When we rely on the header format, e.g., when we're dealing with bridge
windows, we should check the header type, not the class code. The class
code is loosely related to the header type, but is often incorrect and the
spec doesn't actually require it to be related to the header format.
Suggested-by: Bjorn Helgaas <bhelgaas@google.com>
Signed-off-by: Honghui Zhang <honghui.zhang@mediatek.com>
[bhelgaas: changelog, keep the PCI_CLASS_BRIDGE_HOST check]
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
pci_bridge_check_ranges() determines whether a bridge supports the optional
I/O and prefetchable memory windows and sets the flag bits in the bridge
resources. This *could* be done once during enumeration except that the
resource allocation code completely clears the flag bits, e.g., in the
pci_assign_unassigned_bridge_resources() path.
The problem with pci_bridge_check_ranges() in the resource allocation path
is that we may allocate resources after devices have been claimed by
drivers, and pci_bridge_check_ranges() *changes* the window registers to
determine whether they're writable. This may break concurrent accesses to
devices behind the bridge.
Add a new pci_read_bridge_windows() to determine whether a bridge supports
the optional windows, call it once during enumeration, remember the
results, and change pci_bridge_check_ranges() so it doesn't touch the
bridge windows but sets the flag bits based on those remembered results.
Link: https://lore.kernel.org/linux-pci/1506151482-113560-1-git-send-email-wangzhou1@hisilicon.com
Link: https://lists.gnu.org/archive/html/qemu-devel/2018-12/msg02082.html
Reported-by: Yandong Xu <xuyandong2@huawei.com>
Tested-by: Yandong Xu <xuyandong2@huawei.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
Cc: Michael S. Tsirkin <mst@redhat.com>
Cc: Sagi Grimberg <sagi@grimberg.me>
Cc: Ofer Hayut <ofer@lightbitslabs.com>
Cc: Roy Shterman <roys@lightbitslabs.com>
Cc: Keith Busch <keith.busch@intel.com>
Cc: Zhou Wang <wangzhou1@hisilicon.com>
Currently, a hotplug bridge will be given hpmemsize additional memory
and hpiosize additional io if available, in order to satisfy any future
hotplug allocation requirements.
These calculations don't consider the current memory/io size of the
hotplug bridge/slot, so hotplug bridges/slots which have downstream
devices will be allocated their current allocation in addition to the
hpmemsize value.
This makes for possibly undesirable results with a mix of unoccupied and
occupied slots (ex, with hpmemsize=2M):
02:03.0 PCI bridge: <-- Occupied
Memory behind bridge: d6200000-d64fffff [size=3M]
02:04.0 PCI bridge: <-- Unoccupied
Memory behind bridge: d6500000-d66fffff [size=2M]
This change considers the current allocation size when using the
hpmemsize/hpiosize parameters to make the reservations predictable for
the mix of unoccupied and occupied slots:
02:03.0 PCI bridge: <-- Occupied
Memory behind bridge: d6200000-d63fffff [size=2M]
02:04.0 PCI bridge: <-- Unoccupied
Memory behind bridge: d6400000-d65fffff [size=2M]
Signed-off-by: Jon Derrick <jonathan.derrick@intel.com>
Signed-off-by: Bjorn Helgaas <bhelgaas@google.com>
