The logic was reversed. This feature is not enabled
at the moment, but fix it now for the future.
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Using the wrong watermwark value for the secondary
watermark. Copy paste typo. Noticed by Mykola.
Reviewed-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org
Using the wrong watermwark value for the secondary
watermark. Copy paste typo. Noticed by Mykola.
Reviewed-by: Harry Wentland <harry.wentland@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org
Should help avoid fragmentation of vram due to CPU access
requirements.
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
For kernel driver BOs, be explicit about whether we need
vram access up front. This avoids unecessary migrations and
avoids using visible vram for buffers were it's not needed.
v2: line wrap fixes
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Customize the insert_nop func for SDMA rings, and use burst NOP for
ring/IB submissions in other places as well
Signed-off-by: Jammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
The insert_nop function is added to amdgpu_ring_funcs structure as
well as the default implementation
Signed-off-by: Jammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
This function is added to map the ring to sdma instance
Signed-off-by: Jammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
The burst NOP is supported for SDMA when feature_version is >= 20.
Signed-off-by: Jammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
This is added to support the burst NOP
v2: squash the typo fix
Signed-off-by: Jammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Instead of the array which is used for ID management.
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Jammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
No copy_(to|from)_user while BO is reserved.
v2: handle default path as well
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
That isn't used any more.
v2: rebase
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
sync_resv is to handle both amdgpu_fence and sched_fence.
Signed-off-by: Chunming Zhou <david1.zhou@amd.com>
Reviewed-by: Jammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: Christian K?nig <christian.koenig@amd.com>
AUX addresses are 20 bits long. Send out the entire address instead of
just the low 16 bits.
Port of:
drm/radeon/atom: Send out the full AUX address
to amdgpu
Reviewed-by: Jammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org
The latter is a default version of <asm/mman.h> and not for driver use.
Signed-off-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Oded Gabbay <oded.gabbay@gmail.com>
Saving the current UVD state on suspend and restoring it on resume
just doesn't work reliable. Just close cleanup all sessions on suspend.
Ported from radeon commit "12e49feadff6d7b7ebbe852b36943a71524d8d34".
v2: rebased
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com> (v1)
This way the scheduler doesn't wait in it's work thread any more.
v2: fix race conditions
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Reviewed-by: Jammy Zhou <Jammy.Zhou@amd.com>
Freeing up a queue after signalling it isn't race free.
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Jammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Removing the entity from scheduling can deadlock the whole system.
Wait forever till the remaining IBs are scheduled.
v2: fix comment as well
Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Jammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com> (v1)
The context needs to finish before everything else.
Signed-off-by: Christian König <christian.koenig@amd.com>
Acked-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Jammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
This aids handling buffers moves with the scheduler.
Signed-off-by: Chunming Zhou <david1.zhou@amd.com>
Reviewed-by: Christian K?nig <christian.koenig@amd.com>
The fence in the array may be skipped if wait_all is false,
thus the related callback is not initialized with list head.
So removing this kind callback will cause NULL pointer reference.
Signed-off-by: Junwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: Jammy Zhou <Jammy.Zhou@amd.com>
when eviction is happening, if don't handle
dependency, then the fence could be dead off.
Signed-off-by: Chunming Zhou <david1.zhou@amd.com>
Reviewed-by: Jammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: Christian K?nig <christian.koenig@amd.com>
Entity don't live as long as scheduler fences.
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Calling schedule() is probably the worse things we can do.
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Keep run queue, entity and scheduler handling together.
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Cleanup function name, stop checking scheduler ready twice, but
check if kernel thread should stop instead.
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
None of them are used any more.
v2: fix type in error message
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Signed-off-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Chunming Zhou <david1.zhou@amd.com>
Reviewed-by: Christian K?nig <christian.koenig@amd.com>
Remove active_hw_rq and it's protecting queue_lock, they are unused.
User 32bit atomic for hw_rq_count, 64bits for counting to three is a bit
overkill.
Cleanup the function name and remove incorrect comments.
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Jammy Zhou <Jammy.Zhou@amd.com>
Simply not used any more. Only keep 32bit atomic for fence sequence numbering.
v2: trivial rebase
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com> (v1)
Reviewed-by: Jammy Zhou <Jammy.Zhou@amd.com> (v1)
Reviewed-by: Chunming Zhou <david1.zhou@amd.com> (v1)
Rename the function and update the related code with this modified function.
Add the new parameter of bool wait_all.
If wait_all is true, it will return when all fences are signaled or timeout.
If wait_all is false, it will return when any fence is signaled or timeout.
Signed-off-by: Junwei Zhang <Jerry.Zhang@amd.com>
Reviewed-by: Monk Liu <monk.liu@amd.com>
Reviewed-by: Jammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Use pci_alloc_consistent rather than kzalloc since we
need 256 byte aligned memory for the ring buffer.
v2: fix copy paste typo in free function noticed
by Jammy.
bug:
https://bugs.freedesktop.org/show_bug.cgi?id=91749
Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Jammy Zhou <Jammy.Zhou@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org
This is a port of:
DRM - radeon: Don't link train DisplayPort on HPD until we get the dpcd
to amdgpu.
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Cc: stable@vger.kernel.org
Stop double freeing the the BO list by pulling the content
of amdgpu_cs_parser_prepare_job() into the IOCTL function again.
v2: better commit message
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com> (v1)
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
The problem now is that we don't necessarily call amdgpu_ib_get()
in some error paths and so work with uninitialized data.
Better require that the memory is already zeroed.
v2: better commit message
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com> (v1)
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
More appropriate and fixes some nasty lockdep warnings.
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Used by mesa, etc. for profiling.
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Signed-off-by: Marek Olšák <marek.olsak@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
Pagetables can be moved and therefore the page directory update can be necessary
for the current cs even if none of the the bo's are moved. In that scenario
there is no fence between the sdma0 and gfx ring, so we add one.
v2 (chk): rebased
Signed-off-by: Bas Nieuwenhuizen <bas@basnieuwenhuizen.nl>
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Rework run queue implementation, especially remove the odd list handling.
v2: cleanup the code only, no algorithem change.
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
fix the bug that there is duplicated bo_update_mapping issued
Signed-off-by: monk.liu <monk.liu@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
bo_list_clone() will take a lot of time when bo_list hold too much
elements, like above 7000
Signed-off-by: Monk.Liu <monk.liu@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Reviewed-by: Jammy Zhou <jammy.zhou@amd.com>
It's not validated yet and causes more harm than good.
Avoids spurious resets.
Reviewed-by: Leo Liu <leo.liu@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
remaining timeout returned by amdgpu_fence_wait_any can be larger than
max int value, thus the truncated 32 bit value in r ends up being
negative while its original long value is positive.
Signed-off-by: monk.liu <monk.liu@amd.com>
Reviewed-by: Michel Dänzer <michel.daenzer@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Jammy Zhou <jammy.zhou@amd.com>
fix fence is released when pass to **fence sometimes.
add reference for it.
Signed-off-by: Chunming Zhou <david1.zhou@amd.com>
Reviewed-by: Christian K?nig <christian.koenig@amd.com>
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Tested-and-Reviewed-by: Leo Liu <leo.liu@amd.com>
Not used any more.
v2: remove amd_sched_emit as well.
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Similar to radeon, except that amdgpu doesn't even use struct_mutex to
protect anything like the shared z buffer (sane gpu architecture,
yay!). And the code already grabs the globa adev->ring_lock, so this
code can't race with itself. Which makes struct_mutex completely
redundnant. Remove it.
Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: "Christian König" <christian.koenig@amd.com>
Reviewed-by: Thierry Reding <treding@nvidia.com>
Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
It really doesn't protect anything which doesn't have other locks
already. Also this is run from driver unload code so not much need for
locks anyway.
Same changes as for radeon really.
Cc: Alex Deucher <alexander.deucher@amd.com>
Cc: "Christian König" <christian.koenig@amd.com>
Reviewed-by: Thierry Reding <treding@nvidia.com>
Signed-off-by: Daniel Vetter <daniel.vetter@intel.com>
Signed-off-by: Alex Deucher <alexander.deucher@amd.com>
The workaround simply doesn't work because VM mappings
are controlled by userspace not the kernel.
Additional to that this is just a performance problem
which happens if you have holes in your VM mapping.
v2: adjust virtual addr alignment as well.
v3: fix trivial warning
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Monk Liu <monk.liu@amd.com> (v1)
Reviewed-by: Jammy Zhou <Jammy.Zhou@amd.com> (v2)
Looks like that somehow got missed while during porting the radeon changes.
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
It was just a wrapper for fence_wait anyway.
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
The common kernel function does the same thing.
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
scheduler fence is based on kernel fence framework.
v2: squash in Christian's build fix
Signed-off-by: Chunming Zhou <david1.zhou@amd.com>
Reviewed-by: Christian K?nig <christian.koenig@amd.com>
v2: rebased
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com> (v1)
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Avoiding a couple of casts.
v2: rename c_entity to entity as well
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
Cleanup the kernel context handling.
v2: rebased
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com> (v1)
Id's are for the IOCTL ABI only.
v2: remove tgid as well
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
We didn't initialized the mutex in the cloned bo list resulting in nice
warnings from lockdep. Also fixes error handling in this function.
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
This way can avoid interrupt lost, and can process sched job exactly.
Signed-off-by: Chunming Zhou <david1.zhou@amd.com>
Reviewed-by: Jammy Zhou <Jammy.Zhou@amd.com>
This function is used to get the next queued sequence number
Signed-off-by: Jammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Signed-off-by: Jammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
This function is to update last_emitted_v_seq and wake up the waiters.
It should be called by driver in the run_job backend function
Signed-off-by: Jammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
It is clean to update last_queued_v_seq in the scheduler module
Signed-off-by: Jammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Fix the code alignment, etc.
v2: rebase the code
Signed-off-by: Jammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
every sbumission should be able to get a fence.
Signed-off-by: Chunming Zhou <david1.zhou@amd.com>
Reviewed-by: Christian K?nig <christian.koenig@amd.com>
Reviewed-by: Jammy Zhou <jammy.zhou@amd.com>
It is theoretically possible that a swapped out BO gets the
same GTT address, but different backing pages while being swapped in.
Instead just use another VA state to note updated areas.
Ported from not upstream yet radeon commit with the same name.
v2: fix some bugs in the original implementation found in the radeon code.
v3: squash in VCE/UVD fix
Signed-off-by: Christian König <christian.koenig@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
thus unnecessary wake up could be avoid between rings
v2:
move wait_queue_head to fence_drv from ring
Signed-off-by: monk.liu <monk.liu@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
origninal method will sleep/schedule at the granurarity of HZ/2 and
based on seq signal method, the new implement is based on kernel fance
interface, no unnecessary schedule at all
v2: replace logic of original amdgpu_fence_wait_any
Signed-off-by: monk.liu <monk.liu@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
if enabling scheduler, then the queued seq is assigned
when pushing job before emitting job.
Signed-off-by: Chunming Zhou <david1.zhou@amd.com>
Reviewed-by: Christian K?nig <christian.koenig@amd.com>
the job must be emitted by scheduler, otherwise scheduler is abnormal.
Signed-off-by: Chunming Zhou <david1.zhou@amd.com>
Reviewed-by: Christian K?nig <christian.koenig@amd.com>
This option can be used to specify the max number of submissions in the
active HW queue. The default value is 2 now.
Signed-off-by: Jammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
This option can be used to specify the max job number in the job queue,
and it is 16 by default.
Signed-off-by: Jammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: Chunming Zhou <david1.zhou@amd.com>
it is possible that the callback isn't defined sometimes.
Signed-off-by: Chunming Zhou <david1.zhou@amd.com>
Reviewed-by: Christian K?nig <christian.koenig@amd.com>
fence_process may be called from kthread, user thread and interrupt context.
it is possible to called concurrently, then will wake up fence queue multiple times.
Signed-off-by: Chunming Zhou <david1.zhou@amd.com>
Reviewed-by: Jammy Zhou <Jammy.Zhou@amd.com>
use kernel context to submit command for vm
Signed-off-by: Chunming Zhou <david1.zhou@amd.com>
Acked-by: Christian K?nig <christian.koenig@amd.com>
Reviewed-by: Jammy Zhou <Jammy.Zhou@amd.com>
user mode will still use pte ring as a normal ring.
if the prepare job generates another command(update pte) on its ring in scheduler,
then will kill scheduler which is going to waiting later job but pending running job.
Signed-off-by: Chunming Zhou <david1.zhou@amd.com>
Acked-by: Christian K?nig <christian.koenig@amd.com>
Reviewed-by: Jammy Zhou <Jammy.Zhou@amd.com>
BO validation is moved to scheduler except usrptr which must be validated
in user process
Signed-off-by: Chunming Zhou <david1.zhou@amd.com>
Acked-by: Christian K?nig <christian.koenig@amd.com>
Reviewed-by: Jammy Zhou <Jammy.Zhou@amd.com>
run queue:
A set of entities scheduling commands for the same ring. It
implements the scheduling policy that selects the next entity to
emit commands from.
entity:
A scheduler entity is a wrapper around a job queue or a group of
other entities. This can be used to build hierarchies of entities.
For example all job queue entities belonging to the same process
may be placed in a higher level entity and scheduled against other
process entities.
Entities take turns emitting jobs from their job queue to the
corresponding hardware ring, in accordance with the scheduler policy.
Signed-off-by: Shaoyun Liu <Shaoyun.Liu@amd.com>
Signed-off-by: Chunming Zhou <David1.Zhou@amd.com>
Signed-off-by: Jammy Zhou <Jammy.Zhou@amd.com>
Acked-by: Christian K?nig <christian.koenig@amd.com>
Reviewed-by: Jammy Zhou <Jammy.Zhou@amd.com>
Signed-off-by: David Zhang <david1.zhang@amd.com>
Reviewed-by: Alex Deucher <alexander.deucher@amd.com>
Reviewed-by: Jammy Zhou <Jammy.Zhou@amd.com>
Reviewed-by: Christian König <christian.koenig@amd.com>
