If the VMA is being deleted, we don't need to explicity unmap first
anymore. The MMU code will automatically merge the operations into
a single page tree walk.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
These are the new priviledged interfaces to the VMM backends, and expose
some functionality that wasn't previously available.
It's now possible to allocate a chunk of address-space (even all of it),
without causing page tables to be allocated up-front, and then map into
it at arbitrary locations. This is the basic primitive used to support
features such as sparse mapping, or to allow userspace control over its
own address-space, or HMM (where the GPU driver isn't in control of the
address-space layout).
Rather than being tied to a subtle combination of memory object and VMA
properties, arguments that control map flags (ro, kind, etc) are passed
explicitly at map time.
The compatibility hacks to implement the old frontend on top of the new
driver backends have been replaced with something similar to implement
the old frontend's interfaces on top of the new frontend.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Adds support for:
- 64KiB/2MiB big page sizes (128KiB not supported by HW with new PT layout).
- System-memory PTs.
- LPTE "invalid" state.
- (Tegra) Use of video memory aperture.
- Sparse PDEs/PTEs.
- Additional blocklinear kinds.
- 49-bit address-space.
GP100 supports an entirely new 5-level page table layout that provides
an expanded 49-bit address-space. It also supports the layout present
on previous generations, which we've been making do with until now.
This commit implements support for the new layout, and enables it by
default.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Adds support for:
- 64KiB big page size.
- System-memory PTs.
- LPTE "invalid" state.
- (Tegra) Use of video memory aperture.
Adds support for marking LPTEs invalid, resulting in the corresponding
SPTEs being ignored, which is supposed to speed up TLB invalidates.
On The Tegra side, this will switch to using the video memory aperture
for all mappings. The HW will still target non-coherent system memory,
but this aperture needs to be selected in order to support compression.
Tegra's instmem backend somewhat cheated to get this effect previously.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
This is the common code to support a rework of the VMM backends.
It adds support for more than 2 levels of page table nesting, which
is required to be able to support GP100's MMU layout.
Sparse mappings (that don't cause MMU faults when accessed) are now
supported, where the backend provides it.
Dual-PT handling had to become more sophisticated to support sparse,
but this also allows us to support an optimisation the MMU provides
on GK104 and newer.
Certain operations can now be combined into a single page tree walk
to avoid some overhead, but also enables optimsations like skipping
PTE unmap writes when the PT will be destroyed anyway.
The old backend has been hacked up to forward requests onto the new
backend, if present, so that it's possible to bisect between issues
in the backend changes vs the upcoming frontend changes.
Until the new frontend has been merged, new backends will leak BAR2
page tables on module unload. This is expected, and it's not worth
the effort of hacking around this as it doesn't effect runtime.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
To avoid wasting compression tags when using 64KiB pages, we need to
enable this so we can select between upper/lower comptagline in PTEs.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
If NV_PFB_MMU_CTRL_USE_FULL_COMP_TAG_LINE is TRUE, then the last bit of
NV_MMU_PTE_COMPTAGLINE is re-purposed to select the upper/lower half of
a compression tag when using 64KiB big pages.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
We previously required each VMM user to allocate their own page directory
and fill in the instance block themselves.
It makes more sense to handle this in a common location.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Adds support for:
- Selection of old/new-style page table layout (GP100MmuLayout=0/1).
- System-memory PDs.
New layout disabled by default for the moment, as we don't have a
backend that can handle it yet.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
This is the first chunk of the new VMM code that provides the structures
needed to describe a GPU virtual address-space layout, as well as common
interfaces to handle VMM creation, and connecting instances to a VMM.
The constructor now allocates the PD itself, rather than having the user
handle that manually. This won't/can't be used until after all backends
have been ported to these interfaces, so a little bit of memory will be
wasted on Fermi and newer for a couple of commits in the series.
Compatibility has been hacked into the old code to allow each GPU backend
to be ported individually.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
GP100 "big" (which is a funny name, when it supports "even bigger") page
tables are small enough that we want to be able to suballocate them from
a larger block of memory.
This builds on the previous page table cache interfaces so that the VMM
code doesn't need to know the difference.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Builds up and maintains a small cache of each page table size in order
to reduce the frequency of expensive allocations, particularly in the
pathological case where an address range ping-pongs between allocated
and free.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Removes the need to expose internals outside of MMU, and GP100 is both
different, and a lot harder to deal with.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
This will cause a subtle behaviour change on GPUs that are in mixed-memory
configurations in that VRAM in the degraded section of VRAM will no longer
be used for TTM buffer objects.
That section of VRAM is not meant to be used for displayable/compressed
surfaces, and we have no reliable way with the current interfaces to be
able to make that decision properly.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Another transition step to allow finer-grained patches transitioning to
new MMU backends.
Old backends will continue operate as before (accessing nvkm_mem::tag),
and new backends will get a reference to the tags allocated here.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Upcoming MMU changes use nvkm_memory as its basic representation of memory,
so we need to be able to allocate VRAM like this.
The code is basically identical to the current chipset-specific allocators,
minus support for compression tags (which will be handled elsewhere anyway).
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Adds support for 64-bit writes, and optimised filling of buffers with
fixed 32/64-bit values.
These will all be used by the upcoming MMU changes.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
We need to be able to prevent memory from being freed while it's still
mapped in a GPU's address-space.
Will be used by upcoming MMU changes.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Needed by VMM code to determine whether an allocation is compatible with
a given page size (ie. you can't map 4KiB system memory pages into 64KiB
GPU pages).
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Map flags (access, kind, etc) are currently defined in either the VMA,
or the memory object, which turns out to not be ideal for things like
suballocated buffers, etc.
These will become per-map flags instead, so we need to support passing
these arguments in nvkm_memory_map().
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
nvkm_memory is going to be used by the upcoming mmu rework for the basic
representation of a memory allocation, as such, this commit adds support
for comptag allocation to nvkm_memory.
This is very simple for now, in that it requires comptags for the entire
memory allocation even if only certain ranges are compressed.
Support for tracking ranges will be added at a later date.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
We're moving towards having a central place to handle comptag allocation,
and as some GPUs don't have a ram submodule (ie. Tegra), we need to move
the mm somewhere else.
It probably never belonged in ram anyways.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Different sections of VRAM may have different properties (ie. can't be used
for compression/display, can't be mapped, etc).
We currently already support this, but it's a bit magic. This change makes
it more obvious where we're allocating from.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
TTM memory allocations will be hanging off the DRM's client, but the
locking needed to do so gets really tricky with all the other use of
the DRM's object tree.
To solve this, we make the normal DRM client a child of a new master,
where the memory allocations will be done from instead.
This also solves a potential race with client creation.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
We don't really care about where the memory is, just that it's compatible
with a VMA allocated for a given page size.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Before: "imem: init completed in 299277us"
After: "imem: init completed in 11574us"
Suspend from Fedora 26 gnome desktop on GP102.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Before: "imem: suspend completed in 5540487us"
After: "imem: suspend completed in 1871526us"
Suspend from Fedora 26 gnome desktop on GP102.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
A good deal of the structures we map into here aren't accessed very often
at all, and Fedora 26 has exposed an issue where after creating a heap of
channels, BAR2 space would run out, and we'd need to make use of the slow
path while accessing important structures like page tables.
This implements an LRU on BAR2 space, which allows eviction of mappings
that aren't currently needed, to make space for other objects.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
Another piece of solving the "GP100 BAR2 VMM bootstrap" puzzle.
Without doing this, we'd attempt to write PDEs for the lower page table
levels through BAR2 before BAR2 access has been fully initialised.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
This is not as simple as it was for earlier GPUs, due to the need to swap
accessor functions depending on whether BAR2 is usable or not.
We were previously protected by nvkm_instobj's accessor functions keeping
an object mapped permanently, with some unclear magic that managed to hit
the slow-path where needed even if an object was marked as mapped.
That's been replaced here by reference counting maps (some objects, like
page tables can be accessed concurrently), and swapping the functions as
necessary.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
This is to simplify upcoming changes. The slow-path is something that
currently occurs during bootstrap of the BAR2 VMM, while backing up an
object during suspend/resume, or when BAR2 address space runs out.
The latter is a real problem that can happen at runtime, and occurs in
Fedora 26 already (due to some change that causes a lot of channels to
be created at login), so ideally we'd prefer not to make it any slower.
We'd also like suspend/resume speed to not suffer.
Upcoming commits will solve those problems in a better way, making the
extra overhead of moving the locking here a non-issue.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
The accessor functions can change as a result of acquire()/release() calls,
and are protected by any refcounting done there.
Other functions must remain constant, as they can be called any time.
Signed-off-by: Ben Skeggs <bskeggs@redhat.com>
