Nixyri/godot
1
0
Fork 0
mirror of https://github.com/godotengine/godot.git synced 2024-11-22 04:06:14 +00:00
Code Issues Actions 1 Packages Projects Releases Wiki Activity

Improvements from TheForge (see description)

Browse Source 
The work was performed by collaboration of TheForge and Google. I am
merely splitting it up into smaller PRs and cleaning it up.

This is the most "risky" PR so far because the previous ones have been
miscellaneous stuff aimed at either [improve
debugging](https://github.com/godotengine/godot/pull/90993) (e.g. device
lost), [improve Android
experience](https://github.com/godotengine/godot/pull/96439) (add Swappy
for better Frame Pacing + Pre-Transformed Swapchains for slightly better
performance), or harmless [ASTC
improvements](https://github.com/godotengine/godot/pull/96045) (better
performance by simply toggling a feature when available).

However this PR contains larger modifications aimed at improving
performance or reducing memory fragmentation. With greater
modifications, come greater risks of bugs or breakage.

Changes introduced by this PR:

## Transient memory

TBDR GPUs (e.g. most of Android + iOS + M1 Apple) support rendering to
Render Targets that are not backed by actual GPU memory (everything
stays in cache). This works as long as load action isn't `LOAD`, and
store action must be `DONT_CARE`. This saves VRAM (it also makes
painfully obvious when a mistake introduces a performance regression).
Of particular usefulness is when doing MSAA and keeping the raw MSAA
content is not necessary.

## Immutable samplers

Some GPUs get faster when the sampler settings are hard-coded into the
GLSL shaders (instead of being dynamically bound at runtime). This
required changes to the GLSL shaders, PSO creation routines, Descriptor
creation routines, and Descriptor binding routines.

### Toggle

 - `bool immutable_samplers_enabled = true`

Setting it to false enforces the old behavior. Useful for debugging bugs
and regressions.

Immutable samplers requires that the samplers stay... immutable, hence
this boolean is useful if the promise gets broken. We might want to turn
this into a `GLOBAL_DEF` setting.

## Linear Descriptor Pools

Instead of creating dozen/hundreds/thousands of `VkDescriptorSet` every
frame that need to be freed individually when they are no longer needed,
they all get freed at once by resetting the whole pool. Once the whole
pool is no longer in use by the GPU, it gets reset and its memory
recycled. Descriptor sets that are created to be kept around for longer
or forever (i.e. not created and freed within the same frame) **must
not** use linear pools. There may be more than one pool per frame. How
many pools per frame Godot ends up with depends on its capacity, and
that is controlled by
`rendering/rendering_device/vulkan/max_descriptors_per_pool`.

- **Possible improvement for later:** It should be possible for Godot
to adapt to how many descriptors per pool are needed on a per-key basis
(i.e. grow their capacity like `std::vector` does) after rendering a few
frames; which would be better than the current solution of having a
single global value for all pools (`max_descriptors_per_pool`) that the
user needs to tweak.

### Toggle
 - `bool linear_descriptor_pools_enabled = true`

Setting it to false enforces the old behavior. Useful for debugging bugs
and regressions.
Setting it to false is required when workarounding driver bugs (e.g.
Adreno 730).

## Reset Command Pools

A ridiculous optimization. Ridiculous because the original code
should've done this in the first place. Previously Godot was doing the
following:

  1. Create a command buffer **pool**. One per frame.
  2. Create multiple command buffers from the pool in point 1.
3. Call `vkBeginCommandBuffer` on the cmd buffer in point 2. This
resets the cmd buffer because Godot requests the
`VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT` flag.
  4. Add commands to the cmd buffers from point 2.
  5. Submit those commands.
6. On frame N + 2, recycle the buffer pool and cmd buffers from pt 1 &
2, and repeat from step 3.

The problem here is that step 3 resets each command buffer individually.
Initially Godot used to have 1 cmd buffer per pool, thus the impact is
very low.

But not anymore (specially with Adreno workarounds to force splitting
compute dispatches into a new cmd buffer, more on this later). However
Godot keeps around a very low amount of command buffers per frame.

The recommended method is to reset the whole pool, to reset all cmd
buffers at once. Hence the new steps would be:

  1. Create a command buffer **pool**. One per frame.
  2. Create multiple command buffers from the pool in point 1.
3. Call `vkBeginCommandBuffer` on the cmd buffer in point 2, which is
already reset/empty (see step 6).
  4. Add commands to the cmd buffers from point 2.
  5. Submit those commands.
6. On frame N + 2, recycle the buffer pool and cmd buffers from pt 1 &
2, call `vkResetCommandPool` and repeat from step 3.

**Possible issues:** @dariosamo added `transfer_worker` which creates a
command buffer pool:

```cpp
transfer_worker->command_pool =
driver->command_pool_create(transfer_queue_family,
RDD::COMMAND_BUFFER_TYPE_PRIMARY);
```

As expected, validation was complaining that command buffers were being
reused without being reset (that's good, we now know Validation Layers
will warn us of wrong use).
I fixed it by adding:

```cpp
void RenderingDevice::_wait_for_transfer_worker(TransferWorker
*p_transfer_worker) {
	driver->fence_wait(p_transfer_worker->command_fence);
	driver->command_pool_reset(p_transfer_worker->command_pool); //
! New line !
```

**Secondary cmd buffers are subject to the same issue but I didn't alter
them. I talked this with Dario and he is aware of this.**
Secondary cmd buffers are currently disabled due to other issues (it's
disabled on master).

### Toggle

 - `bool RenderingDeviceCommons::command_pool_reset_enabled`

Setting it to false enforces the old behavior. Useful for debugging bugs
and regressions.

There's no other reason for this boolean. Possibly once it becomes well
tested, the boolean could be removed entirely.

## Descriptor set batched binding

Adds `command_bind_render_uniform_sets` and
`add_draw_list_bind_uniform_sets` (+ compute variants).

It performs the same as `add_draw_list_bind_uniform_set` (notice
singular vs plural), but on multiple consecutive uniform sets, thus
reducing graph and draw call overhead.

### Toggle

 - `bool descriptor_set_batching = true;`

Setting it to false enforces the old behavior. Useful for debugging bugs
and regressions.

There's no other reason for this boolean. Possibly once it becomes well
tested, the boolean could be removed entirely.

## Do not wait so long for swapchain

Godot currently does the following:

 1. Fill the entire cmd buffer with commands.
 2. `submit()`
    - Wait with a semaphore for the swapchain.
- Trigger a semaphore to indicate when we're done (so the swapchain
can submit).
 3. `present()`

The optimization opportunity here is that 95% of Godot's rendering is
done offscreen.
Then a fullscreen pass copies everything to the swapchain. Godot doesn't
practically render directly to the swapchain.

The problem with this is that the GPU has to wait for the swapchain to
be released **to start anything**, when we could start *much earlier*.
Only the final blit pass must wait for the swapchain.

TheForge changed it to the following (more complicated, I'm simplifying
the idea):

 1. Fill the entire cmd buffer with commands.
 2. In `screen_prepare_for_drawing` do `submit()`
    - There are no semaphore waits for the swapchain.
    - Trigger a semaphore to indicate when we're done.
3. Fill a new cmd buffer that only does the final blit to the
swapchain.
 4. `submit()`
    - Wait with a semaphore for the submit() from step 2.
- Wait with a semaphore for the swapchain (so the swapchain can
submit).
- Trigger a semaphore to indicate when we're done (so the swapchain
can submit).
 5. `present()`

Dario discovered this problem independently while working on a different
platform.

**However TheForge's solution had to be rewritten from scratch:** The
complexity to achieve the solution was high and quite difficult to
maintain with the way Godot works now (after Übershaders PR).
But on the other hand, re-implementing the solution became much simpler
because Dario already had to do something similar: To fix an Adreno 730
driver bug, he had to implement splitting command buffers. **This is
exactly what we need!**. Thus it was re-written using this existing
functionality for a new purpose.

To achieve this, I added a new argument, `bool p_split_cmd_buffer`, to
`RenderingDeviceGraph::add_draw_list_begin`, which is only set to true
by `RenderingDevice::draw_list_begin_for_screen`.

The graph will split the draw list into its own command buffer.

### Toggle

 - `bool split_swapchain_into_its_own_cmd_buffer = true;`

Setting it to false enforces the old behavior. This might be necessary
for consoles which follow an alternate solution to the same problem.
If not, then we should consider removing it.

## Free Shader memory

PR #90993 added `shader_destroy_modules()` but it was not actually in
use.

This PR adds several places where `shader_destroy_modules()` is called
after initialization to free up memory of SPIR-V structures that are no
longer needed.
This commit is contained in:
Matias N. Goldberg 2024-11-14 13:03:14 -03:00
parent 76fa7b2914
commit 8cf50cc9f9
23 changed files with 869 additions and 197 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

3
doc/classes/ProjectSettings.xml
View File

@ -2861,6 +2861,9 @@
[b]Note:[/b] Some platforms may restrict the actual value.
</member>
<member name="rendering/rendering_device/vulkan/max_descriptors_per_pool" type="int" setter="" getter="" default="64">
The number of descriptors per pool. Godot's Vulkan backend uses linear pools for descriptors that will be created and destroyed within a single frame. Instead of destroying every single descriptor every frame, they all can be destroyed at once by resetting the pool they belong to.
A larger number is more efficient up to a limit, after that it will only waste RAM (maximum efficiency is achieved when there is no more than 1 pool per frame). A small number could end up with one pool per descriptor, which negatively impacts performance.
[b]Note:[/b] Changing this property requires a restart to take effect.
</member>
<member name="rendering/scaling_3d/fsr_sharpness" type="float" setter="" getter="" default="0.2">
Determines how sharp the upscaled image will be when using the FSR upscaling mode. Sharpness halves with every whole number. Values go from 0.0 (sharpest) to 2.0. Values above 2.0 won't make a visible difference.

28
drivers/d3d12/rendering_device_driver_d3d12.cpp
View File

@ -2286,6 +2286,10 @@ RDD::CommandPoolID RenderingDeviceDriverD3D12::command_pool_create(CommandQueueF
return CommandPoolID(command_pool);
}
bool RenderingDeviceDriverD3D12::command_pool_reset(CommandPoolID p_cmd_pool) {
return true;
}
void RenderingDeviceDriverD3D12::command_pool_free(CommandPoolID p_cmd_pool) {
CommandPoolInfo *command_pool = (CommandPoolInfo *)(p_cmd_pool.id);
memdelete(command_pool);
@ -3589,7 +3593,7 @@ Vector<uint8_t> RenderingDeviceDriverD3D12::shader_compile_binary_from_spirv(Vec
return ret;
}
RDD::ShaderID RenderingDeviceDriverD3D12::shader_create_from_bytecode(const Vector<uint8_t> &p_shader_binary, ShaderDescription &r_shader_desc, String &r_name) {
RDD::ShaderID RenderingDeviceDriverD3D12::shader_create_from_bytecode(const Vector<uint8_t> &p_shader_binary, ShaderDescription &r_shader_desc, String &r_name, const Vector<ImmutableSampler> &p_immutable_samplers) {
r_shader_desc = {}; // Driver-agnostic.
ShaderInfo shader_info_in; // Driver-specific.
@ -3798,7 +3802,9 @@ static void _add_descriptor_count_for_uniform(RenderingDevice::UniformType p_typ
}
}
RDD::UniformSetID RenderingDeviceDriverD3D12::uniform_set_create(VectorView<BoundUniform> p_uniforms, ShaderID p_shader, uint32_t p_set_index) {
RDD::UniformSetID RenderingDeviceDriverD3D12::uniform_set_create(VectorView<BoundUniform> p_uniforms, ShaderID p_shader, uint32_t p_set_index, int p_linear_pool_index) {
// p_linear_pool_index = -1; // TODO:? Linear pools not implemented or not supported by API backend.
// Pre-bookkeep.
UniformSetInfo *uniform_set_info = VersatileResource::allocate<UniformSetInfo>(resources_allocator);
@ -5325,6 +5331,13 @@ void RenderingDeviceDriverD3D12::command_bind_render_uniform_set(CommandBufferID
_command_bind_uniform_set(p_cmd_buffer, p_uniform_set, p_shader, p_set_index, false);
}
void RenderingDeviceDriverD3D12::command_bind_render_uniform_sets(CommandBufferID p_cmd_buffer, VectorView<UniformSetID> p_uniform_sets, ShaderID p_shader, uint32_t p_first_set_index, uint32_t p_set_count) {
for (uint32_t i = 0u; i < p_set_count; ++i) {
// TODO: _command_bind_uniform_set() does WAAAAY too much stuff. A lot of it should be already cached in UniformSetID when uniform_set_create() was called. Binding is supposed to be a cheap operation, ideally a memcpy.
_command_bind_uniform_set(p_cmd_buffer, p_uniform_sets[i], p_shader, p_first_set_index + i, false);
}
}
void RenderingDeviceDriverD3D12::command_render_draw(CommandBufferID p_cmd_buffer, uint32_t p_vertex_count, uint32_t p_instance_count, uint32_t p_base_vertex, uint32_t p_first_instance) {
CommandBufferInfo *cmd_buf_info = (CommandBufferInfo *)p_cmd_buffer.id;
_bind_vertex_buffers(cmd_buf_info);
@ -5829,6 +5842,13 @@ void RenderingDeviceDriverD3D12::command_bind_compute_uniform_set(CommandBufferI
_command_bind_uniform_set(p_cmd_buffer, p_uniform_set, p_shader, p_set_index, true);
}
void RenderingDeviceDriverD3D12::command_bind_compute_uniform_sets(CommandBufferID p_cmd_buffer, VectorView<UniformSetID> p_uniform_sets, ShaderID p_shader, uint32_t p_first_set_index, uint32_t p_set_count) {
for (uint32_t i = 0u; i < p_set_count; ++i) {
// TODO: _command_bind_uniform_set() does WAAAAY too much stuff. A lot of it should be already cached in UniformSetID when uniform_set_create() was called. Binding is supposed to be a cheap operation, ideally a memcpy.
_command_bind_uniform_set(p_cmd_buffer, p_uniform_sets[i], p_shader, p_first_set_index + i, true);
}
}
void RenderingDeviceDriverD3D12::command_compute_dispatch(CommandBufferID p_cmd_buffer, uint32_t p_x_groups, uint32_t p_y_groups, uint32_t p_z_groups) {
CommandBufferInfo *cmd_buf_info = (CommandBufferInfo *)p_cmd_buffer.id;
if (!barrier_capabilities.enhanced_barriers_supported) {
@ -6112,6 +6132,10 @@ uint64_t RenderingDeviceDriverD3D12::get_total_memory_used() {
return stats.Total.Stats.BlockBytes;
}
uint64_t RenderingDeviceDriverD3D12::get_lazily_memory_used() {
return 0;
}
uint64_t RenderingDeviceDriverD3D12::limit_get(Limit p_limit) {
uint64_t safe_unbounded = ((uint64_t)1 << 30);
switch (p_limit) {

9
drivers/d3d12/rendering_device_driver_d3d12.h
View File

@ -434,6 +434,7 @@ private:
public:
virtual CommandPoolID command_pool_create(CommandQueueFamilyID p_cmd_queue_family, CommandBufferType p_cmd_buffer_type) override final;
virtual bool command_pool_reset(CommandPoolID p_cmd_pool) override final;
virtual void command_pool_free(CommandPoolID p_cmd_pool) override final;
// ----- BUFFER -----
@ -697,7 +698,7 @@ private:
public:
virtual String shader_get_binary_cache_key() override final;
virtual Vector<uint8_t> shader_compile_binary_from_spirv(VectorView<ShaderStageSPIRVData> p_spirv, const String &p_shader_name) override final;
virtual ShaderID shader_create_from_bytecode(const Vector<uint8_t> &p_shader_binary, ShaderDescription &r_shader_desc, String &r_name) override final;
virtual ShaderID shader_create_from_bytecode(const Vector<uint8_t> &p_shader_binary, ShaderDescription &r_shader_desc, String &r_name, const Vector<ImmutableSampler> &p_immutable_samplers) override final;
virtual uint32_t shader_get_layout_hash(ShaderID p_shader) override final;
virtual void shader_free(ShaderID p_shader) override final;
virtual void shader_destroy_modules(ShaderID p_shader) override final;
@ -747,7 +748,7 @@ private:
};
public:
virtual UniformSetID uniform_set_create(VectorView<BoundUniform> p_uniforms, ShaderID p_shader, uint32_t p_set_index) override final;
virtual UniformSetID uniform_set_create(VectorView<BoundUniform> p_uniforms, ShaderID p_shader, uint32_t p_set_index, int p_linear_pool_index) override final;
virtual void uniform_set_free(UniformSetID p_uniform_set) override final;
// ----- COMMANDS -----
@ -757,6 +758,7 @@ public:
private:
void _command_check_descriptor_sets(CommandBufferID p_cmd_buffer);
void _command_bind_uniform_set(CommandBufferID p_cmd_buffer, UniformSetID p_uniform_set, ShaderID p_shader, uint32_t p_set_index, bool p_for_compute);
void _command_bind_uniform_sets(CommandBufferID p_cmd_buffer, VectorView<UniformSetID> p_uniform_sets, ShaderID p_shader, uint32_t p_first_set_index, uint32_t p_set_count, bool p_for_compute);
public:
/******************/
@ -846,6 +848,7 @@ public:
// Binding.
virtual void command_bind_render_pipeline(CommandBufferID p_cmd_buffer, PipelineID p_pipeline) override final;
virtual void command_bind_render_uniform_set(CommandBufferID p_cmd_buffer, UniformSetID p_uniform_set, ShaderID p_shader, uint32_t p_set_index) override final;
virtual void command_bind_render_uniform_sets(CommandBufferID p_cmd_buffer, VectorView<UniformSetID> p_uniform_sets, ShaderID p_shader, uint32_t p_first_set_index, uint32_t p_set_count) override final;
// Drawing.
virtual void command_render_draw(CommandBufferID p_cmd_buffer, uint32_t p_vertex_count, uint32_t p_instance_count, uint32_t p_base_vertex, uint32_t p_first_instance) override final;
@ -893,6 +896,7 @@ public:
// Binding.
virtual void command_bind_compute_pipeline(CommandBufferID p_cmd_buffer, PipelineID p_pipeline) override final;
virtual void command_bind_compute_uniform_set(CommandBufferID p_cmd_buffer, UniformSetID p_uniform_set, ShaderID p_shader, uint32_t p_set_index) override final;
virtual void command_bind_compute_uniform_sets(CommandBufferID p_cmd_buffer, VectorView<UniformSetID> p_uniform_sets, ShaderID p_shader, uint32_t p_first_set_index, uint32_t p_set_count) override final;
// Dispatching.
virtual void command_compute_dispatch(CommandBufferID p_cmd_buffer, uint32_t p_x_groups, uint32_t p_y_groups, uint32_t p_z_groups) override final;
@ -986,6 +990,7 @@ public:
virtual void set_object_name(ObjectType p_type, ID p_driver_id, const String &p_name) override final;
virtual uint64_t get_resource_native_handle(DriverResource p_type, ID p_driver_id) override final;
virtual uint64_t get_total_memory_used() override final;
virtual uint64_t get_lazily_memory_used() override final;
virtual uint64_t limit_get(Limit p_limit) override final;
virtual uint64_t api_trait_get(ApiTrait p_trait) override final;
virtual bool has_feature(Features p_feature) override final;

2
drivers/metal/metal_objects.h
View File

@ -502,6 +502,7 @@ public:
#pragma mark - Render Commands
void render_bind_uniform_set(RDD::UniformSetID p_uniform_set, RDD::ShaderID p_shader, uint32_t p_set_index);
void render_bind_uniform_sets(VectorView<RDD::UniformSetID> p_uniform_sets, RDD::ShaderID p_shader, uint32_t p_first_set_index, uint32_t p_set_count);
void render_clear_attachments(VectorView<RDD::AttachmentClear> p_attachment_clears, VectorView<Rect2i> p_rects);
void render_set_viewport(VectorView<Rect2i> p_viewports);
void render_set_scissor(VectorView<Rect2i> p_scissors);
@ -535,6 +536,7 @@ public:
#pragma mark - Compute Commands
void compute_bind_uniform_set(RDD::UniformSetID p_uniform_set, RDD::ShaderID p_shader, uint32_t p_set_index);
void compute_bind_uniform_sets(VectorView<RDD::UniformSetID> p_uniform_sets, RDD::ShaderID p_shader, uint32_t p_first_set_index, uint32_t p_set_count);
void compute_dispatch(uint32_t p_x_groups, uint32_t p_y_groups, uint32_t p_z_groups);
void compute_dispatch_indirect(RDD::BufferID p_indirect_buffer, uint64_t p_offset);

54
drivers/metal/metal_objects.mm
View File

@ -223,6 +223,26 @@ void MDCommandBuffer::render_bind_uniform_set(RDD::UniformSetID p_uniform_set, R
}
}
void MDCommandBuffer::render_bind_uniform_sets(VectorView<RDD::UniformSetID> p_uniform_sets, RDD::ShaderID p_shader, uint32_t p_first_set_index, uint32_t p_set_count) {
DEV_ASSERT(type == MDCommandBufferStateType::Render);
for (size_t i = 0u; i < p_set_count; ++i) {
MDUniformSet *set = (MDUniformSet *)(p_uniform_sets[i].id);
if (render.uniform_sets.size() <= set->index) {
uint32_t s = render.uniform_sets.size();
render.uniform_sets.resize(set->index + 1);
// Set intermediate values to null.
std::fill(&render.uniform_sets[s], &render.uniform_sets[set->index] + 1, nullptr);
}
if (render.uniform_sets[set->index] != set) {
render.dirty.set_flag(RenderState::DIRTY_UNIFORMS);
render.uniform_set_mask |= 1ULL << set->index;
render.uniform_sets[set->index] = set;
}
}
}
void MDCommandBuffer::render_clear_attachments(VectorView<RDD::AttachmentClear> p_attachment_clears, VectorView<Rect2i> p_rects) {
DEV_ASSERT(type == MDCommandBufferStateType::Render);
@ -964,6 +984,40 @@ void MDCommandBuffer::compute_bind_uniform_set(RDD::UniformSetID p_uniform_set,
}
}
void MDCommandBuffer::compute_bind_uniform_sets(VectorView<RDD::UniformSetID> p_uniform_sets, RDD::ShaderID p_shader, uint32_t p_first_set_index, uint32_t p_set_count) {
DEV_ASSERT(type == MDCommandBufferStateType::Compute);
id<MTLComputeCommandEncoder> enc = compute.encoder;
id<MTLDevice> device = enc.device;
MDShader *shader = (MDShader *)(p_shader.id);
thread_local LocalVector<__unsafe_unretained id<MTLBuffer>> buffers;
thread_local LocalVector<NSUInteger> offsets;
buffers.resize(p_set_count);
offsets.resize(p_set_count);
for (size_t i = 0u; i < p_set_count; ++i) {
UniformSet const &set_info = shader->sets[p_first_set_index + i];
MDUniformSet *set = (MDUniformSet *)(p_uniform_sets[i].id);
BoundUniformSet &bus = set->boundUniformSetForShader(shader, device);
bus.merge_into(compute.resource_usage);
uint32_t const *offset = set_info.offsets.getptr(RDD::SHADER_STAGE_COMPUTE);
if (offset) {
buffers[i] = bus.buffer;
offsets[i] = *offset;
} else {
buffers[i] = nullptr;
offsets[i] = 0u;
}
}
[enc setBuffers:buffers.ptr() offsets:offsets.ptr() withRange:NSMakeRange(p_first_set_index, p_set_count)];
}
void MDCommandBuffer::compute_dispatch(uint32_t p_x_groups, uint32_t p_y_groups, uint32_t p_z_groups) {
DEV_ASSERT(type == MDCommandBufferStateType::Compute);

8
drivers/metal/rendering_device_driver_metal.h
View File

@ -185,6 +185,7 @@ public:
// ----- POOL -----
virtual CommandPoolID command_pool_create(CommandQueueFamilyID p_cmd_queue_family, CommandBufferType p_cmd_buffer_type) override final;
virtual bool command_pool_reset(CommandPoolID p_cmd_pool) override final;
virtual void command_pool_free(CommandPoolID p_cmd_pool) override final;
// ----- BUFFER -----
@ -250,14 +251,14 @@ private:
public:
virtual String shader_get_binary_cache_key() override final;
virtual Vector<uint8_t> shader_compile_binary_from_spirv(VectorView<ShaderStageSPIRVData> p_spirv, const String &p_shader_name) override final;
virtual ShaderID shader_create_from_bytecode(const Vector<uint8_t> &p_shader_binary, ShaderDescription &r_shader_desc, String &r_name) override final;
virtual ShaderID shader_create_from_bytecode(const Vector<uint8_t> &p_shader_binary, ShaderDescription &r_shader_desc, String &r_name, const Vector<ImmutableSampler> &p_immutable_samplers) override final;
virtual void shader_free(ShaderID p_shader) override final;
virtual void shader_destroy_modules(ShaderID p_shader) override final;
#pragma mark - Uniform Set
public:
virtual UniformSetID uniform_set_create(VectorView<BoundUniform> p_uniforms, ShaderID p_shader, uint32_t p_set_index) override final;
virtual UniformSetID uniform_set_create(VectorView<BoundUniform> p_uniforms, ShaderID p_shader, uint32_t p_set_index, int p_linear_pool_index) override final;
virtual void uniform_set_free(UniformSetID p_uniform_set) override final;
#pragma mark - Commands
@ -330,6 +331,7 @@ public:
// Binding.
virtual void command_bind_render_pipeline(CommandBufferID p_cmd_buffer, PipelineID p_pipeline) override final;
virtual void command_bind_render_uniform_set(CommandBufferID p_cmd_buffer, UniformSetID p_uniform_set, ShaderID p_shader, uint32_t p_set_index) override final;
virtual void command_bind_render_uniform_sets(CommandBufferID p_cmd_buffer, VectorView<UniformSetID> p_uniform_sets, ShaderID p_shader, uint32_t p_first_set_index, uint32_t p_set_count) override final;
// Drawing.
virtual void command_render_draw(CommandBufferID p_cmd_buffer, uint32_t p_vertex_count, uint32_t p_instance_count, uint32_t p_base_vertex, uint32_t p_first_instance) override final;
@ -370,6 +372,7 @@ public:
// Binding.
virtual void command_bind_compute_pipeline(CommandBufferID p_cmd_buffer, PipelineID p_pipeline) override final;
virtual void command_bind_compute_uniform_set(CommandBufferID p_cmd_buffer, UniformSetID p_uniform_set, ShaderID p_shader, uint32_t p_set_index) override final;
virtual void command_bind_compute_uniform_sets(CommandBufferID p_cmd_buffer, VectorView<UniformSetID> p_uniform_sets, ShaderID p_shader, uint32_t p_first_set_index, uint32_t p_set_count) override final;
// Dispatching.
virtual void command_compute_dispatch(CommandBufferID p_cmd_buffer, uint32_t p_x_groups, uint32_t p_y_groups, uint32_t p_z_groups) override final;
@ -412,6 +415,7 @@ public:
virtual void set_object_name(ObjectType p_type, ID p_driver_id, const String &p_name) override final;
virtual uint64_t get_resource_native_handle(DriverResource p_type, ID p_driver_id) override final;
virtual uint64_t get_total_memory_used() override final;
virtual uint64_t get_lazily_memory_used() override final;
virtual uint64_t limit_get(Limit p_limit) override final;
virtual uint64_t api_trait_get(ApiTrait p_trait) override final;
virtual bool has_feature(Features p_feature) override final;

41
drivers/metal/rendering_device_driver_metal.mm
View File

@ -275,13 +275,20 @@ RDD::TextureID RenderingDeviceDriverMetal::texture_create(const TextureFormat &p
}
// Usage.
MTLResourceOptions options = MTLResourceCPUCacheModeDefaultCache | MTLResourceHazardTrackingModeTracked;
if (p_format.usage_bits & TEXTURE_USAGE_CPU_READ_BIT) {
options |= MTLResourceStorageModeShared;
const bool supports_memoryless = (*metal_device_properties).features.highestFamily >= MTLGPUFamilyApple2 && (*metal_device_properties).features.highestFamily < MTLGPUFamilyMac1;
if (supports_memoryless && p_format.usage_bits & TEXTURE_USAGE_TRANSIENT_BIT) {
desc.resourceOptions = MTLResourceStorageModeMemoryless | MTLResourceHazardTrackingModeTracked;
desc.storageMode = MTLStorageModeMemoryless;
} else {
options |= MTLResourceStorageModePrivate;
MTLResourceOptions options = MTLResourceCPUCacheModeDefaultCache | MTLResourceHazardTrackingModeTracked;
if (p_format.usage_bits & TEXTURE_USAGE_CPU_READ_BIT) {
options |= MTLResourceStorageModeShared;
} else {
options |= MTLResourceStorageModePrivate;
}
desc.resourceOptions = options;
}
desc.resourceOptions = options;
if (p_format.usage_bits & TEXTURE_USAGE_SAMPLING_BIT) {
desc.usage |= MTLTextureUsageShaderRead;
@ -876,6 +883,10 @@ RDD::CommandPoolID RenderingDeviceDriverMetal::command_pool_create(CommandQueueF
return rid::make(device_queue);
}
bool RenderingDeviceDriverMetal::command_pool_reset(CommandPoolID p_cmd_pool) {
return true;
}
void RenderingDeviceDriverMetal::command_pool_free(CommandPoolID p_cmd_pool) {
rid::release(p_cmd_pool);
}
@ -2315,7 +2326,7 @@ void RenderingDeviceDriverMetal::shader_cache_free_entry(const SHA256Digest &key
}
}
RDD::ShaderID RenderingDeviceDriverMetal::shader_create_from_bytecode(const Vector<uint8_t> &p_shader_binary, ShaderDescription &r_shader_desc, String &r_name) {
RDD::ShaderID RenderingDeviceDriverMetal::shader_create_from_bytecode(const Vector<uint8_t> &p_shader_binary, ShaderDescription &r_shader_desc, String &r_name, const Vector<ImmutableSampler> &p_immutable_samplers) {
r_shader_desc = {}; // Driver-agnostic.
const uint8_t *binptr = p_shader_binary.ptr();
@ -2524,7 +2535,9 @@ void RenderingDeviceDriverMetal::shader_destroy_modules(ShaderID p_shader) {
/**** UNIFORM SET ****/
/*********************/
RDD::UniformSetID RenderingDeviceDriverMetal::uniform_set_create(VectorView<BoundUniform> p_uniforms, ShaderID p_shader, uint32_t p_set_index) {
RDD::UniformSetID RenderingDeviceDriverMetal::uniform_set_create(VectorView<BoundUniform> p_uniforms, ShaderID p_shader, uint32_t p_set_index, int p_linear_pool_index) {
// p_linear_pool_index = -1; // TODO:? Linear pools not implemented or not supported by API backend.
MDUniformSet *set = new MDUniformSet();
Vector<BoundUniform> bound_uniforms;
bound_uniforms.resize(p_uniforms.size());
@ -3079,6 +3092,11 @@ void RenderingDeviceDriverMetal::command_bind_render_uniform_set(CommandBufferID
cb->render_bind_uniform_set(p_uniform_set, p_shader, p_set_index);
}
void RenderingDeviceDriverMetal::command_bind_render_uniform_sets(CommandBufferID p_cmd_buffer, VectorView<UniformSetID> p_uniform_sets, ShaderID p_shader, uint32_t p_first_set_index, uint32_t p_set_count) {
MDCommandBuffer *cb = (MDCommandBuffer *)(p_cmd_buffer.id);
cb->render_bind_uniform_sets(p_uniform_sets, p_shader, p_first_set_index, p_set_count);
}
void RenderingDeviceDriverMetal::command_render_draw(CommandBufferID p_cmd_buffer, uint32_t p_vertex_count, uint32_t p_instance_count, uint32_t p_base_vertex, uint32_t p_first_instance) {
MDCommandBuffer *cb = (MDCommandBuffer *)(p_cmd_buffer.id);
cb->render_draw(p_vertex_count, p_instance_count, p_base_vertex, p_first_instance);
@ -3550,6 +3568,11 @@ void RenderingDeviceDriverMetal::command_bind_compute_uniform_set(CommandBufferI
cb->compute_bind_uniform_set(p_uniform_set, p_shader, p_set_index);
}
void RenderingDeviceDriverMetal::command_bind_compute_uniform_sets(CommandBufferID p_cmd_buffer, VectorView<UniformSetID> p_uniform_sets, ShaderID p_shader, uint32_t p_first_set_index, uint32_t p_set_count) {
MDCommandBuffer *cb = (MDCommandBuffer *)(p_cmd_buffer.id);
cb->compute_bind_uniform_sets(p_uniform_sets, p_shader, p_first_set_index, p_set_count);
}
void RenderingDeviceDriverMetal::command_compute_dispatch(CommandBufferID p_cmd_buffer, uint32_t p_x_groups, uint32_t p_y_groups, uint32_t p_z_groups) {
MDCommandBuffer *cb = (MDCommandBuffer *)(p_cmd_buffer.id);
cb->compute_dispatch(p_x_groups, p_y_groups, p_z_groups);
@ -3753,6 +3776,10 @@ uint64_t RenderingDeviceDriverMetal::get_total_memory_used() {
return device.currentAllocatedSize;
}
uint64_t RenderingDeviceDriverMetal::get_lazily_memory_used() {
return 0; // TODO: Track this (grep for memoryless in Godot's Metal backend).
}
uint64_t RenderingDeviceDriverMetal::limit_get(Limit p_limit) {
MetalDeviceProperties const &props = (*metal_device_properties);
MetalLimits const &limits = props.limits;

250
drivers/vulkan/rendering_device_driver_vulkan.cpp
View File

@ -1383,6 +1383,15 @@ Error RenderingDeviceDriverVulkan::initialize(uint32_t p_device_index, uint32_t
physical_device = context_driver->physical_device_get(p_device_index);
vkGetPhysicalDeviceProperties(physical_device, &physical_device_properties);
// Workaround a driver bug on Adreno 730 GPUs that keeps leaking memory on each call to vkResetDescriptorPool.
// Which eventually run out of memory. in such case we should not be using linear allocated pools
// Bug introduced in driver 512.597.0 and fixed in 512.671.0
// Confirmed by Qualcomm
if (linear_descriptor_pools_enabled) {
const uint32_t reset_descriptor_pool_broken_driver_begin = VK_MAKE_VERSION(512u, 597u, 0u);
const uint32_t reset_descriptor_pool_fixed_driver_begin = VK_MAKE_VERSION(512u, 671u, 0u);
linear_descriptor_pools_enabled = physical_device_properties.driverVersion < reset_descriptor_pool_broken_driver_begin || physical_device_properties.driverVersion > reset_descriptor_pool_fixed_driver_begin;
}
frame_count = p_frame_count;
// Copy the queue family properties the context already retrieved.
@ -1728,7 +1737,27 @@ RDD::TextureID RenderingDeviceDriverVulkan::texture_create(const TextureFormat &
VmaAllocationCreateInfo alloc_create_info = {};
alloc_create_info.flags = (p_format.usage_bits & TEXTURE_USAGE_CPU_READ_BIT) ? VMA_ALLOCATION_CREATE_HOST_ACCESS_RANDOM_BIT : 0;
alloc_create_info.preferredFlags = VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT;
if (p_format.usage_bits & TEXTURE_USAGE_TRANSIENT_BIT) {
uint32_t memory_type_index = 0;
VmaAllocationCreateInfo lazy_memory_requirements = alloc_create_info;
lazy_memory_requirements.usage = VMA_MEMORY_USAGE_GPU_LAZILY_ALLOCATED;
VkResult result = vmaFindMemoryTypeIndex(allocator, UINT32_MAX, &lazy_memory_requirements, &memory_type_index);
if (VK_SUCCESS == result) {
alloc_create_info = lazy_memory_requirements;
create_info.usage |= VK_IMAGE_USAGE_TRANSIENT_ATTACHMENT_BIT;
// VUID-VkImageCreateInfo-usage-00963 :
// If usage includes VK_IMAGE_USAGE_TRANSIENT_ATTACHMENT_BIT,
// then bits other than VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT, VK_IMAGE_USAGE_DEPTH_STENCIL_ATTACHMENT_BIT,
// and VK_IMAGE_USAGE_INPUT_ATTACHMENT_BIT must not be set
create_info.usage &= (VK_IMAGE_USAGE_TRANSIENT_ATTACHMENT_BIT | VK_IMAGE_USAGE_COLOR_ATTACHMENT_BIT | VK_IMAGE_USAGE_DEPTH_STENCIL_ATTACHMENT_BIT | VK_IMAGE_USAGE_INPUT_ATTACHMENT_BIT);
} else {
alloc_create_info.preferredFlags = VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT;
}
} else {
alloc_create_info.preferredFlags = VK_MEMORY_PROPERTY_DEVICE_LOCAL_BIT;
}
if (image_size <= SMALL_ALLOCATION_MAX_SIZE) {
uint32_t mem_type_index = 0;
vmaFindMemoryTypeIndexForImageInfo(allocator, &create_info, &alloc_create_info, &mem_type_index);
@ -2657,7 +2686,10 @@ RDD::CommandPoolID RenderingDeviceDriverVulkan::command_pool_create(CommandQueue
VkCommandPoolCreateInfo cmd_pool_info = {};
cmd_pool_info.sType = VK_STRUCTURE_TYPE_COMMAND_POOL_CREATE_INFO;
cmd_pool_info.queueFamilyIndex = family_index;
cmd_pool_info.flags = VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT;
if (!command_pool_reset_enabled) {
cmd_pool_info.flags = VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT;
}
VkCommandPool vk_command_pool = VK_NULL_HANDLE;
VkResult res = vkCreateCommandPool(vk_device, &cmd_pool_info, VKC::get_allocation_callbacks(VK_OBJECT_TYPE_COMMAND_POOL), &vk_command_pool);
@ -2669,6 +2701,16 @@ RDD::CommandPoolID RenderingDeviceDriverVulkan::command_pool_create(CommandQueue
return CommandPoolID(command_pool);
}
bool RenderingDeviceDriverVulkan::command_pool_reset(CommandPoolID p_cmd_pool) {
DEV_ASSERT(p_cmd_pool);
CommandPool *command_pool = (CommandPool *)(p_cmd_pool.id);
VkResult err = vkResetCommandPool(vk_device, command_pool->vk_command_pool, 0);
ERR_FAIL_COND_V_MSG(err, false, "vkResetCommandPool failed with error " + itos(err) + ".");
return true;
}
void RenderingDeviceDriverVulkan::command_pool_free(CommandPoolID p_cmd_pool) {
DEV_ASSERT(p_cmd_pool);
@ -2702,8 +2744,6 @@ RDD::CommandBufferID RenderingDeviceDriverVulkan::command_buffer_create(CommandP
}
bool RenderingDeviceDriverVulkan::command_buffer_begin(CommandBufferID p_cmd_buffer) {
// Reset is implicit (VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT).
VkCommandBufferBeginInfo cmd_buf_begin_info = {};
cmd_buf_begin_info.sType = VK_STRUCTURE_TYPE_COMMAND_BUFFER_BEGIN_INFO;
cmd_buf_begin_info.flags = VK_COMMAND_BUFFER_USAGE_ONE_TIME_SUBMIT_BIT;
@ -2715,8 +2755,6 @@ bool RenderingDeviceDriverVulkan::command_buffer_begin(CommandBufferID p_cmd_buf
}
bool RenderingDeviceDriverVulkan::command_buffer_begin_secondary(CommandBufferID p_cmd_buffer, RenderPassID p_render_pass, uint32_t p_subpass, FramebufferID p_framebuffer) {
// Reset is implicit (VK_COMMAND_POOL_CREATE_RESET_COMMAND_BUFFER_BIT).
Framebuffer *framebuffer = (Framebuffer *)(p_framebuffer.id);
VkCommandBufferInheritanceInfo inheritance_info = {};
@ -3475,7 +3513,7 @@ Vector<uint8_t> RenderingDeviceDriverVulkan::shader_compile_binary_from_spirv(Ve
return ret;
}
RDD::ShaderID RenderingDeviceDriverVulkan::shader_create_from_bytecode(const Vector<uint8_t> &p_shader_binary, ShaderDescription &r_shader_desc, String &r_name) {
RDD::ShaderID RenderingDeviceDriverVulkan::shader_create_from_bytecode(const Vector<uint8_t> &p_shader_binary, ShaderDescription &r_shader_desc, String &r_name, const Vector<ImmutableSampler> &p_immutable_samplers) {
r_shader_desc = {}; // Driver-agnostic.
ShaderInfo shader_info; // Driver-specific.
@ -3547,6 +3585,19 @@ RDD::ShaderID RenderingDeviceDriverVulkan::shader_create_from_bytecode(const Vec
case UNIFORM_TYPE_SAMPLER: {
layout_binding.descriptorType = VK_DESCRIPTOR_TYPE_SAMPLER;
layout_binding.descriptorCount = set_ptr[j].length;
// Immutable samplers: here they get set in the layoutbinding, given that they will not be changed later.
int immutable_bind_index = -1;
if (immutable_samplers_enabled && p_immutable_samplers.size() > 0) {
for (int k = 0; k < p_immutable_samplers.size(); k++) {
if (p_immutable_samplers[k].binding == layout_binding.binding) {
immutable_bind_index = k;
break;
}
}
if (immutable_bind_index >= 0) {
layout_binding.pImmutableSamplers = (VkSampler *)&p_immutable_samplers[immutable_bind_index].ids[0].id;
}
}
} break;
case UNIFORM_TYPE_SAMPLER_WITH_TEXTURE: {
layout_binding.descriptorType = VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER;
@ -3768,9 +3819,9 @@ void RenderingDeviceDriverVulkan::shader_destroy_modules(ShaderID p_shader) {
/*********************/
/**** UNIFORM SET ****/
/*********************/
VkDescriptorPool RenderingDeviceDriverVulkan::_descriptor_set_pool_find_or_create(const DescriptorSetPoolKey &p_key, DescriptorSetPools::Iterator *r_pool_sets_it) {
DescriptorSetPools::Iterator pool_sets_it = descriptor_set_pools.find(p_key);
VkDescriptorPool RenderingDeviceDriverVulkan::_descriptor_set_pool_find_or_create(const DescriptorSetPoolKey &p_key, DescriptorSetPools::Iterator *r_pool_sets_it, int p_linear_pool_index) {
bool linear_pool = p_linear_pool_index >= 0;
DescriptorSetPools::Iterator pool_sets_it = linear_pool ? linear_descriptor_set_pools[p_linear_pool_index].find(p_key) : descriptor_set_pools.find(p_key);
if (pool_sets_it) {
for (KeyValue<VkDescriptorPool, uint32_t> &E : pool_sets_it->value) {
@ -3856,7 +3907,11 @@ VkDescriptorPool RenderingDeviceDriverVulkan::_descriptor_set_pool_find_or_creat
VkDescriptorPoolCreateInfo descriptor_set_pool_create_info = {};
descriptor_set_pool_create_info.sType = VK_STRUCTURE_TYPE_DESCRIPTOR_POOL_CREATE_INFO;
descriptor_set_pool_create_info.flags = VK_DESCRIPTOR_POOL_CREATE_FREE_DESCRIPTOR_SET_BIT; // Can't think how somebody may NOT need this flag.
if (linear_descriptor_pools_enabled && linear_pool) {
descriptor_set_pool_create_info.flags = 0;
} else {
descriptor_set_pool_create_info.flags = VK_DESCRIPTOR_POOL_CREATE_FREE_DESCRIPTOR_SET_BIT; // Can't think how somebody may NOT need this flag.
}
descriptor_set_pool_create_info.maxSets = max_descriptor_sets_per_pool;
descriptor_set_pool_create_info.poolSizeCount = vk_sizes_count;
descriptor_set_pool_create_info.pPoolSizes = vk_sizes;
@ -3870,7 +3925,11 @@ VkDescriptorPool RenderingDeviceDriverVulkan::_descriptor_set_pool_find_or_creat
// Bookkeep.
if (!pool_sets_it) {
pool_sets_it = descriptor_set_pools.insert(p_key, HashMap<VkDescriptorPool, uint32_t>());
if (linear_pool) {
pool_sets_it = linear_descriptor_set_pools[p_linear_pool_index].insert(p_key, HashMap<VkDescriptorPool, uint32_t>());
} else {
pool_sets_it = descriptor_set_pools.insert(p_key, HashMap<VkDescriptorPool, uint32_t>());
}
}
HashMap<VkDescriptorPool, uint32_t> &pool_rcs = pool_sets_it->value;
pool_rcs.insert(vk_pool, 0);
@ -3878,34 +3937,43 @@ VkDescriptorPool RenderingDeviceDriverVulkan::_descriptor_set_pool_find_or_creat
return vk_pool;
}
void RenderingDeviceDriverVulkan::_descriptor_set_pool_unreference(DescriptorSetPools::Iterator p_pool_sets_it, VkDescriptorPool p_vk_descriptor_pool) {
void RenderingDeviceDriverVulkan::_descriptor_set_pool_unreference(DescriptorSetPools::Iterator p_pool_sets_it, VkDescriptorPool p_vk_descriptor_pool, int p_linear_pool_index) {
HashMap<VkDescriptorPool, uint32_t>::Iterator pool_rcs_it = p_pool_sets_it->value.find(p_vk_descriptor_pool);
pool_rcs_it->value--;
if (pool_rcs_it->value == 0) {
vkDestroyDescriptorPool(vk_device, p_vk_descriptor_pool, VKC::get_allocation_callbacks(VK_OBJECT_TYPE_DESCRIPTOR_POOL));
p_pool_sets_it->value.erase(p_vk_descriptor_pool);
if (p_pool_sets_it->value.is_empty()) {
descriptor_set_pools.remove(p_pool_sets_it);
if (linear_descriptor_pools_enabled && p_linear_pool_index >= 0) {
linear_descriptor_set_pools[p_linear_pool_index].remove(p_pool_sets_it);
} else {
descriptor_set_pools.remove(p_pool_sets_it);
}
}
}
}
RDD::UniformSetID RenderingDeviceDriverVulkan::uniform_set_create(VectorView<BoundUniform> p_uniforms, ShaderID p_shader, uint32_t p_set_index) {
RDD::UniformSetID RenderingDeviceDriverVulkan::uniform_set_create(VectorView<BoundUniform> p_uniforms, ShaderID p_shader, uint32_t p_set_index, int p_linear_pool_index) {
if (!linear_descriptor_pools_enabled) {
p_linear_pool_index = -1;
}
DescriptorSetPoolKey pool_key;
// Immutable samplers will be skipped so we need to track the number of vk_writes used.
VkWriteDescriptorSet *vk_writes = ALLOCA_ARRAY(VkWriteDescriptorSet, p_uniforms.size());
uint32_t writes_amount = 0;
for (uint32_t i = 0; i < p_uniforms.size(); i++) {
const BoundUniform &uniform = p_uniforms[i];
vk_writes[i] = {};
vk_writes[i].sType = VK_STRUCTURE_TYPE_WRITE_DESCRIPTOR_SET;
vk_writes[i].dstBinding = uniform.binding;
vk_writes[i].descriptorType = VK_DESCRIPTOR_TYPE_MAX_ENUM; // Invalid value.
vk_writes[writes_amount] = {};
vk_writes[writes_amount].sType = VK_STRUCTURE_TYPE_WRITE_DESCRIPTOR_SET;
uint32_t num_descriptors = 1;
switch (uniform.type) {
case UNIFORM_TYPE_SAMPLER: {
if (uniform.immutable_sampler && immutable_samplers_enabled) {
continue; // Skipping immutable samplers.
}
num_descriptors = uniform.ids.size();
VkDescriptorImageInfo *vk_img_infos = ALLOCA_ARRAY(VkDescriptorImageInfo, num_descriptors);
@ -3916,8 +3984,8 @@ RDD::UniformSetID RenderingDeviceDriverVulkan::uniform_set_create(VectorView<Bou
vk_img_infos[j].imageLayout = VK_IMAGE_LAYOUT_UNDEFINED;
}
vk_writes[i].descriptorType = VK_DESCRIPTOR_TYPE_SAMPLER;
vk_writes[i].pImageInfo = vk_img_infos;
vk_writes[writes_amount].descriptorType = VK_DESCRIPTOR_TYPE_SAMPLER;
vk_writes[writes_amount].pImageInfo = vk_img_infos;
} break;
case UNIFORM_TYPE_SAMPLER_WITH_TEXTURE: {
num_descriptors = uniform.ids.size() / 2;
@ -3930,8 +3998,8 @@ RDD::UniformSetID RenderingDeviceDriverVulkan::uniform_set_create(VectorView<Bou
vk_img_infos[j].imageLayout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL;
}
vk_writes[i].descriptorType = VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER;
vk_writes[i].pImageInfo = vk_img_infos;
vk_writes[writes_amount].descriptorType = VK_DESCRIPTOR_TYPE_COMBINED_IMAGE_SAMPLER;
vk_writes[writes_amount].pImageInfo = vk_img_infos;
} break;
case UNIFORM_TYPE_TEXTURE: {
num_descriptors = uniform.ids.size();
@ -3943,8 +4011,8 @@ RDD::UniformSetID RenderingDeviceDriverVulkan::uniform_set_create(VectorView<Bou
vk_img_infos[j].imageLayout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL;
}
vk_writes[i].descriptorType = VK_DESCRIPTOR_TYPE_SAMPLED_IMAGE;
vk_writes[i].pImageInfo = vk_img_infos;
vk_writes[writes_amount].descriptorType = VK_DESCRIPTOR_TYPE_SAMPLED_IMAGE;
vk_writes[writes_amount].pImageInfo = vk_img_infos;
} break;
case UNIFORM_TYPE_IMAGE: {
num_descriptors = uniform.ids.size();
@ -3956,8 +4024,8 @@ RDD::UniformSetID RenderingDeviceDriverVulkan::uniform_set_create(VectorView<Bou
vk_img_infos[j].imageLayout = VK_IMAGE_LAYOUT_GENERAL;
}
vk_writes[i].descriptorType = VK_DESCRIPTOR_TYPE_STORAGE_IMAGE;
vk_writes[i].pImageInfo = vk_img_infos;
vk_writes[writes_amount].descriptorType = VK_DESCRIPTOR_TYPE_STORAGE_IMAGE;
vk_writes[writes_amount].pImageInfo = vk_img_infos;
} break;
case UNIFORM_TYPE_TEXTURE_BUFFER: {
num_descriptors = uniform.ids.size();
@ -3973,9 +4041,9 @@ RDD::UniformSetID RenderingDeviceDriverVulkan::uniform_set_create(VectorView<Bou
vk_buf_views[j] = buf_info->vk_view;
}
vk_writes[i].descriptorType = VK_DESCRIPTOR_TYPE_UNIFORM_TEXEL_BUFFER;
vk_writes[i].pBufferInfo = vk_buf_infos;
vk_writes[i].pTexelBufferView = vk_buf_views;
vk_writes[writes_amount].descriptorType = VK_DESCRIPTOR_TYPE_UNIFORM_TEXEL_BUFFER;
vk_writes[writes_amount].pBufferInfo = vk_buf_infos;
vk_writes[writes_amount].pTexelBufferView = vk_buf_views;
} break;
case UNIFORM_TYPE_SAMPLER_WITH_TEXTURE_BUFFER: {
num_descriptors = uniform.ids.size() / 2;
@ -3995,10 +4063,10 @@ RDD::UniformSetID RenderingDeviceDriverVulkan::uniform_set_create(VectorView<Bou
vk_buf_views[j] = buf_info->vk_view;
}
vk_writes[i].descriptorType = VK_DESCRIPTOR_TYPE_UNIFORM_TEXEL_BUFFER;
vk_writes[i].pImageInfo = vk_img_infos;
vk_writes[i].pBufferInfo = vk_buf_infos;
vk_writes[i].pTexelBufferView = vk_buf_views;
vk_writes[writes_amount].descriptorType = VK_DESCRIPTOR_TYPE_UNIFORM_TEXEL_BUFFER;
vk_writes[writes_amount].pImageInfo = vk_img_infos;
vk_writes[writes_amount].pBufferInfo = vk_buf_infos;
vk_writes[writes_amount].pTexelBufferView = vk_buf_views;
} break;
case UNIFORM_TYPE_IMAGE_BUFFER: {
CRASH_NOW_MSG("Unimplemented!"); // TODO.
@ -4010,8 +4078,8 @@ RDD::UniformSetID RenderingDeviceDriverVulkan::uniform_set_create(VectorView<Bou
vk_buf_info->buffer = buf_info->vk_buffer;
vk_buf_info->range = buf_info->size;
vk_writes[i].descriptorType = VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER;
vk_writes[i].pBufferInfo = vk_buf_info;
vk_writes[writes_amount].descriptorType = VK_DESCRIPTOR_TYPE_UNIFORM_BUFFER;
vk_writes[writes_amount].pBufferInfo = vk_buf_info;
} break;
case UNIFORM_TYPE_STORAGE_BUFFER: {
const BufferInfo *buf_info = (const BufferInfo *)uniform.ids[0].id;
@ -4020,8 +4088,8 @@ RDD::UniformSetID RenderingDeviceDriverVulkan::uniform_set_create(VectorView<Bou
vk_buf_info->buffer = buf_info->vk_buffer;
vk_buf_info->range = buf_info->size;
vk_writes[i].descriptorType = VK_DESCRIPTOR_TYPE_STORAGE_BUFFER;
vk_writes[i].pBufferInfo = vk_buf_info;
vk_writes[writes_amount].descriptorType = VK_DESCRIPTOR_TYPE_STORAGE_BUFFER;
vk_writes[writes_amount].pBufferInfo = vk_buf_info;
} break;
case UNIFORM_TYPE_INPUT_ATTACHMENT: {
num_descriptors = uniform.ids.size();
@ -4033,24 +4101,26 @@ RDD::UniformSetID RenderingDeviceDriverVulkan::uniform_set_create(VectorView<Bou
vk_img_infos[j].imageLayout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL;
}
vk_writes[i].descriptorType = VK_DESCRIPTOR_TYPE_INPUT_ATTACHMENT;
vk_writes[i].pImageInfo = vk_img_infos;
vk_writes[writes_amount].descriptorType = VK_DESCRIPTOR_TYPE_INPUT_ATTACHMENT;
vk_writes[writes_amount].pImageInfo = vk_img_infos;
} break;
default: {
DEV_ASSERT(false);
}
}
vk_writes[i].descriptorCount = num_descriptors;
vk_writes[writes_amount].dstBinding = uniform.binding;
vk_writes[writes_amount].descriptorCount = num_descriptors;
ERR_FAIL_COND_V_MSG(pool_key.uniform_type[uniform.type] == MAX_UNIFORM_POOL_ELEMENT, UniformSetID(),
"Uniform set reached the limit of bindings for the same type (" + itos(MAX_UNIFORM_POOL_ELEMENT) + ").");
pool_key.uniform_type[uniform.type] += num_descriptors;
writes_amount++;
}
// Need a descriptor pool.
DescriptorSetPools::Iterator pool_sets_it;
VkDescriptorPool vk_pool = _descriptor_set_pool_find_or_create(pool_key, &pool_sets_it);
VkDescriptorPool vk_pool = _descriptor_set_pool_find_or_create(pool_key, &pool_sets_it, p_linear_pool_index);
DEV_ASSERT(vk_pool);
pool_sets_it->value[vk_pool]++;
@ -4062,22 +4132,27 @@ RDD::UniformSetID RenderingDeviceDriverVulkan::uniform_set_create(VectorView<Bou
descriptor_set_allocate_info.pSetLayouts = &shader_info->vk_descriptor_set_layouts[p_set_index];
VkDescriptorSet vk_descriptor_set = VK_NULL_HANDLE;
VkResult res = vkAllocateDescriptorSets(vk_device, &descriptor_set_allocate_info, &vk_descriptor_set);
if (res) {
_descriptor_set_pool_unreference(pool_sets_it, vk_pool);
_descriptor_set_pool_unreference(pool_sets_it, vk_pool, p_linear_pool_index);
ERR_FAIL_V_MSG(UniformSetID(), "Cannot allocate descriptor sets, error " + itos(res) + ".");
}
for (uint32_t i = 0; i < p_uniforms.size(); i++) {
for (uint32_t i = 0; i < writes_amount; i++) {
vk_writes[i].dstSet = vk_descriptor_set;
}
vkUpdateDescriptorSets(vk_device, p_uniforms.size(), vk_writes, 0, nullptr);
vkUpdateDescriptorSets(vk_device, writes_amount, vk_writes, 0, nullptr);
// Bookkeep.
UniformSetInfo *usi = VersatileResource::allocate<UniformSetInfo>(resources_allocator);
usi->vk_descriptor_set = vk_descriptor_set;
usi->vk_descriptor_pool = vk_pool;
if (p_linear_pool_index >= 0) {
usi->vk_linear_descriptor_pool = vk_pool;
} else {
usi->vk_descriptor_pool = vk_pool;
}
usi->pool_sets_it = pool_sets_it;
return UniformSetID(usi);
@ -4085,13 +4160,43 @@ RDD::UniformSetID RenderingDeviceDriverVulkan::uniform_set_create(VectorView<Bou
void RenderingDeviceDriverVulkan::uniform_set_free(UniformSetID p_uniform_set) {
UniformSetInfo *usi = (UniformSetInfo *)p_uniform_set.id;
vkFreeDescriptorSets(vk_device, usi->vk_descriptor_pool, 1, &usi->vk_descriptor_set);
_descriptor_set_pool_unreference(usi->pool_sets_it, usi->vk_descriptor_pool);
if (usi->vk_linear_descriptor_pool) {
// Nothing to do. All sets are freed at once using vkResetDescriptorPool.
//
// We can NOT decrease the reference count (i.e. call _descriptor_set_pool_unreference())
// because the pool is linear (i.e. the freed set can't be recycled) and further calls to
// _descriptor_set_pool_find_or_create() need usi->pool_sets_it->value to stay so that we can
// tell if the pool has ran out of space and we need to create a new pool.
} else {
vkFreeDescriptorSets(vk_device, usi->vk_descriptor_pool, 1, &usi->vk_descriptor_set);
_descriptor_set_pool_unreference(usi->pool_sets_it, usi->vk_descriptor_pool, -1);
}
VersatileResource::free(resources_allocator, usi);
}
bool RenderingDeviceDriverVulkan::uniform_sets_have_linear_pools() const {
return true;
}
void RenderingDeviceDriverVulkan::linear_uniform_set_pools_reset(int p_linear_pool_index) {
if (linear_descriptor_pools_enabled) {
DescriptorSetPools &pools_to_reset = linear_descriptor_set_pools[p_linear_pool_index];
DescriptorSetPools::Iterator curr_pool = pools_to_reset.begin();
while (curr_pool != pools_to_reset.end()) {
HashMap<VkDescriptorPool, uint32_t>::Iterator curr_pair = curr_pool->value.begin();
while (curr_pair != curr_pool->value.end()) {
vkResetDescriptorPool(vk_device, curr_pair->key, 0);
curr_pair->value = 0;
++curr_pair;
}
++curr_pool;
}
}
}
// ----- COMMANDS -----
void RenderingDeviceDriverVulkan::command_uniform_set_prepare_for_use(CommandBufferID p_cmd_buffer, UniformSetID p_uniform_set, ShaderID p_shader, uint32_t p_set_index) {
@ -4600,6 +4705,23 @@ void RenderingDeviceDriverVulkan::command_bind_render_uniform_set(CommandBufferI
vkCmdBindDescriptorSets((VkCommandBuffer)p_cmd_buffer.id, VK_PIPELINE_BIND_POINT_GRAPHICS, shader_info->vk_pipeline_layout, p_set_index, 1, &usi->vk_descriptor_set, 0, nullptr);
}
void RenderingDeviceDriverVulkan::command_bind_render_uniform_sets(CommandBufferID p_cmd_buffer, VectorView<UniformSetID> p_uniform_sets, ShaderID p_shader, uint32_t p_first_set_index, uint32_t p_set_count) {
if (p_set_count == 0) {
return;
}
thread_local LocalVector<VkDescriptorSet> sets;
sets.clear();
sets.resize(p_set_count);
for (uint32_t i = 0; i < p_set_count; i++) {
sets[i] = ((const UniformSetInfo *)p_uniform_sets[i].id)->vk_descriptor_set;
}
const ShaderInfo *shader_info = (const ShaderInfo *)p_shader.id;
vkCmdBindDescriptorSets((VkCommandBuffer)p_cmd_buffer.id, VK_PIPELINE_BIND_POINT_GRAPHICS, shader_info->vk_pipeline_layout, p_first_set_index, p_set_count, &sets[0], 0, nullptr);
}
void RenderingDeviceDriverVulkan::command_render_draw(CommandBufferID p_cmd_buffer, uint32_t p_vertex_count, uint32_t p_instance_count, uint32_t p_base_vertex, uint32_t p_first_instance) {
vkCmdDraw((VkCommandBuffer)p_cmd_buffer.id, p_vertex_count, p_instance_count, p_base_vertex, p_first_instance);
}
@ -5015,6 +5137,23 @@ void RenderingDeviceDriverVulkan::command_bind_compute_uniform_set(CommandBuffer
vkCmdBindDescriptorSets((VkCommandBuffer)p_cmd_buffer.id, VK_PIPELINE_BIND_POINT_COMPUTE, shader_info->vk_pipeline_layout, p_set_index, 1, &usi->vk_descriptor_set, 0, nullptr);
}
void RenderingDeviceDriverVulkan::command_bind_compute_uniform_sets(CommandBufferID p_cmd_buffer, VectorView<UniformSetID> p_uniform_sets, ShaderID p_shader, uint32_t p_first_set_index, uint32_t p_set_count) {
if (p_set_count == 0) {
return;
}
thread_local LocalVector<VkDescriptorSet> sets;
sets.clear();
sets.resize(p_set_count);
for (uint32_t i = 0; i < p_set_count; i++) {
sets[i] = ((const UniformSetInfo *)p_uniform_sets[i].id)->vk_descriptor_set;
}
const ShaderInfo *shader_info = (const ShaderInfo *)p_shader.id;
vkCmdBindDescriptorSets((VkCommandBuffer)p_cmd_buffer.id, VK_PIPELINE_BIND_POINT_COMPUTE, shader_info->vk_pipeline_layout, p_first_set_index, p_set_count, &sets[0], 0, nullptr);
}
void RenderingDeviceDriverVulkan::command_compute_dispatch(CommandBufferID p_cmd_buffer, uint32_t p_x_groups, uint32_t p_y_groups, uint32_t p_z_groups) {
vkCmdDispatch((VkCommandBuffer)p_cmd_buffer.id, p_x_groups, p_y_groups, p_z_groups);
}
@ -5555,6 +5694,10 @@ uint64_t RenderingDeviceDriverVulkan::get_total_memory_used() {
return stats.total.statistics.allocationBytes;
}
uint64_t RenderingDeviceDriverVulkan::get_lazily_memory_used() {
return vmaCalculateLazilyAllocatedBytes(allocator);
}
uint64_t RenderingDeviceDriverVulkan::limit_get(Limit p_limit) {
const VkPhysicalDeviceLimits &limits = physical_device_properties.limits;
switch (p_limit) {
@ -5728,6 +5871,15 @@ RenderingDeviceDriverVulkan::~RenderingDeviceDriverVulkan() {
}
vmaDestroyAllocator(allocator);
// Destroy linearly allocated descriptor pools
for (KeyValue<int, DescriptorSetPools> &pool_map : linear_descriptor_set_pools) {
for (KeyValue<DescriptorSetPoolKey, HashMap<VkDescriptorPool, uint32_t>> pools : pool_map.value) {
for (KeyValue<VkDescriptorPool, uint32_t> descriptor_pool : pools.value) {
vkDestroyDescriptorPool(vk_device, descriptor_pool.key, VKC::get_allocation_callbacks(VK_OBJECT_TYPE_DESCRIPTOR_POOL));
}
}
}
if (vk_device != VK_NULL_HANDLE) {
vkDestroyDevice(vk_device, VKC::get_allocation_callbacks(VK_OBJECT_TYPE_DEVICE));
}

22
drivers/vulkan/rendering_device_driver_vulkan.h
View File

@ -333,6 +333,7 @@ private:
public:
virtual CommandPoolID command_pool_create(CommandQueueFamilyID p_cmd_queue_family, CommandBufferType p_cmd_buffer_type) override final;
virtual bool command_pool_reset(CommandPoolID p_cmd_pool) override final;
virtual void command_pool_free(CommandPoolID p_cmd_pool) override final;
// ----- BUFFER -----
@ -444,7 +445,7 @@ private:
public:
virtual String shader_get_binary_cache_key() override final;
virtual Vector<uint8_t> shader_compile_binary_from_spirv(VectorView<ShaderStageSPIRVData> p_spirv, const String &p_shader_name) override final;
virtual ShaderID shader_create_from_bytecode(const Vector<uint8_t> &p_shader_binary, ShaderDescription &r_shader_desc, String &r_name) override final;
virtual ShaderID shader_create_from_bytecode(const Vector<uint8_t> &p_shader_binary, ShaderDescription &r_shader_desc, String &r_name, const Vector<ImmutableSampler> &p_immutable_samplers) override final;
virtual void shader_free(ShaderID p_shader) override final;
virtual void shader_destroy_modules(ShaderID p_shader) override final;
@ -482,18 +483,27 @@ private:
DescriptorSetPools descriptor_set_pools;
uint32_t max_descriptor_sets_per_pool = 0;
VkDescriptorPool _descriptor_set_pool_find_or_create(const DescriptorSetPoolKey &p_key, DescriptorSetPools::Iterator *r_pool_sets_it);
void _descriptor_set_pool_unreference(DescriptorSetPools::Iterator p_pool_sets_it, VkDescriptorPool p_vk_descriptor_pool);
HashMap<int, DescriptorSetPools> linear_descriptor_set_pools;
bool linear_descriptor_pools_enabled = true;
VkDescriptorPool _descriptor_set_pool_find_or_create(const DescriptorSetPoolKey &p_key, DescriptorSetPools::Iterator *r_pool_sets_it, int p_linear_pool_index);
void _descriptor_set_pool_unreference(DescriptorSetPools::Iterator p_pool_sets_it, VkDescriptorPool p_vk_descriptor_pool, int p_linear_pool_index);
// Global flag to toggle usage of immutable sampler when creating pipeline layouts.
// It cannot change after creating the PSOs, since we need to skipping samplers when creating uniform sets.
bool immutable_samplers_enabled = true;
struct UniformSetInfo {
VkDescriptorSet vk_descriptor_set = VK_NULL_HANDLE;
VkDescriptorPool vk_descriptor_pool = VK_NULL_HANDLE;
VkDescriptorPool vk_linear_descriptor_pool = VK_NULL_HANDLE;
DescriptorSetPools::Iterator pool_sets_it;
};
public:
virtual UniformSetID uniform_set_create(VectorView<BoundUniform> p_uniforms, ShaderID p_shader, uint32_t p_set_index) override final;
virtual UniformSetID uniform_set_create(VectorView<BoundUniform> p_uniforms, ShaderID p_shader, uint32_t p_set_index, int p_linear_pool_index) override final;
virtual void linear_uniform_set_pools_reset(int p_linear_pool_index) override final;
virtual void uniform_set_free(UniformSetID p_uniform_set) override final;
virtual bool uniform_sets_have_linear_pools() const override final;
// ----- COMMANDS -----
@ -575,6 +585,7 @@ public:
// Binding.
virtual void command_bind_render_pipeline(CommandBufferID p_cmd_buffer, PipelineID p_pipeline) override final;
virtual void command_bind_render_uniform_set(CommandBufferID p_cmd_buffer, UniformSetID p_uniform_set, ShaderID p_shader, uint32_t p_set_index) override final;
virtual void command_bind_render_uniform_sets(CommandBufferID p_cmd_buffer, VectorView<UniformSetID> p_uniform_sets, ShaderID p_shader, uint32_t p_first_set_index, uint32_t p_set_count) override final;
// Drawing.
virtual void command_render_draw(CommandBufferID p_cmd_buffer, uint32_t p_vertex_count, uint32_t p_instance_count, uint32_t p_base_vertex, uint32_t p_first_instance) override final;
@ -617,6 +628,7 @@ public:
// Binding.
virtual void command_bind_compute_pipeline(CommandBufferID p_cmd_buffer, PipelineID p_pipeline) override final;
virtual void command_bind_compute_uniform_set(CommandBufferID p_cmd_buffer, UniformSetID p_uniform_set, ShaderID p_shader, uint32_t p_set_index) override final;
virtual void command_bind_compute_uniform_sets(CommandBufferID p_cmd_buffer, VectorView<UniformSetID> p_uniform_sets, ShaderID p_shader, uint32_t p_first_set_index, uint32_t p_set_count) override final;
// Dispatching.
virtual void command_compute_dispatch(CommandBufferID p_cmd_buffer, uint32_t p_x_groups, uint32_t p_y_groups, uint32_t p_z_groups) override final;
@ -671,7 +683,7 @@ public:
virtual void set_object_name(ObjectType p_type, ID p_driver_id, const String &p_name) override final;
virtual uint64_t get_resource_native_handle(DriverResource p_type, ID p_driver_id) override final;
virtual uint64_t get_total_memory_used() override final;
virtual uint64_t get_lazily_memory_used() override final;
virtual uint64_t limit_get(Limit p_limit) override final;
virtual uint64_t api_trait_get(ApiTrait p_trait) override final;
virtual bool has_feature(Features p_feature) override final;

10
servers/rendering/renderer_rd/forward_mobile/render_forward_mobile.cpp
View File

@ -614,8 +614,7 @@ RID RenderForwardMobile::_setup_render_pass_uniform_set(RenderListType p_render_
if (render_pass_uniform_sets[p_index].is_valid() && RD::get_singleton()->uniform_set_is_valid(render_pass_uniform_sets[p_index])) {
RD::get_singleton()->free(render_pass_uniform_sets[p_index]);
}
render_pass_uniform_sets[p_index] = RD::get_singleton()->uniform_set_create(uniforms, scene_shader.default_shader_rd, RENDER_PASS_UNIFORM_SET);
render_pass_uniform_sets[p_index] = RD::get_singleton()->uniform_set_create(uniforms, scene_shader.default_shader_rd, RENDER_PASS_UNIFORM_SET, true);
return render_pass_uniform_sets[p_index];
}
@ -1673,7 +1672,9 @@ void RenderForwardMobile::base_uniforms_changed() {
void RenderForwardMobile::_update_render_base_uniform_set() {
RendererRD::LightStorage *light_storage = RendererRD::LightStorage::get_singleton();
if (render_base_uniform_set.is_null() || !RD::get_singleton()->uniform_set_is_valid(render_base_uniform_set) || (lightmap_texture_array_version != light_storage->lightmap_array_get_version())) {
// We must always recreate the uniform set every frame if we're using linear pools (since we requested it on creation).
// This pays off as long as we often get inside the if() block (i.e. the settings end up changing often).
if (RD::get_singleton()->uniform_sets_have_linear_pools() || render_base_uniform_set.is_null() || !RD::get_singleton()->uniform_set_is_valid(render_base_uniform_set) || (lightmap_texture_array_version != light_storage->lightmap_array_get_version())) {
if (render_base_uniform_set.is_valid() && RD::get_singleton()->uniform_set_is_valid(render_base_uniform_set)) {
RD::get_singleton()->free(render_base_uniform_set);
}
@ -1687,6 +1688,7 @@ void RenderForwardMobile::_update_render_base_uniform_set() {
u.binding = 2;
u.uniform_type = RD::UNIFORM_TYPE_SAMPLER;
u.append_id(scene_shader.shadow_sampler);
u.immutable_sampler = true;
uniforms.push_back(u);
}
@ -1773,7 +1775,7 @@ void RenderForwardMobile::_update_render_base_uniform_set() {
uniforms.push_back(u);
}
render_base_uniform_set = RD::get_singleton()->uniform_set_create(uniforms, scene_shader.default_shader_rd, SCENE_UNIFORM_SET);
render_base_uniform_set = RD::get_singleton()->uniform_set_create(uniforms, scene_shader.default_shader_rd, SCENE_UNIFORM_SET, true);
}
}

27
servers/rendering/renderer_rd/forward_mobile/scene_shader_forward_mobile.cpp
View File

@ -456,6 +456,16 @@ SceneShaderForwardMobile::SceneShaderForwardMobile() {
void SceneShaderForwardMobile::init(const String p_defines) {
RendererRD::MaterialStorage *material_storage = RendererRD::MaterialStorage::get_singleton();
// Immutable samplers : create the shadow sampler to be passed when creating the pipeline.
{
RD::SamplerState sampler;
sampler.mag_filter = RD::SAMPLER_FILTER_LINEAR;
sampler.min_filter = RD::SAMPLER_FILTER_LINEAR;
sampler.enable_compare = true;
sampler.compare_op = RD::COMPARE_OP_LESS;
shadow_sampler = RD::get_singleton()->sampler_create(sampler);
}
/* SCENE SHADER */
{
@ -474,8 +484,13 @@ void SceneShaderForwardMobile::init(const String p_defines) {
shader_versions.push_back(base_define + "\n#define USE_MULTIVIEW\n#define MODE_RENDER_DEPTH\n"); // SHADER_VERSION_SHADOW_PASS_MULTIVIEW
}
shader.initialize(shader_versions, p_defines);
Vector<RD::PipelineImmutableSampler> immutable_samplers;
RD::PipelineImmutableSampler immutable_shadow_sampler;
immutable_shadow_sampler.binding = 2;
immutable_shadow_sampler.append_id(shadow_sampler);
immutable_shadow_sampler.uniform_type = RenderingDeviceCommons::UNIFORM_TYPE_SAMPLER;
immutable_samplers.push_back(immutable_shadow_sampler);
shader.initialize(shader_versions, p_defines, immutable_samplers);
if (!RendererCompositorRD::get_singleton()->is_xr_enabled()) {
for (uint32_t ubershader = 0; ubershader < 2; ubershader++) {
uint32_t base_variant = ubershader ? SHADER_VERSION_MAX : 0;
@ -776,14 +791,6 @@ void fragment() {
default_vec4_xform_uniform_set = RD::get_singleton()->uniform_set_create(uniforms, default_shader_rd, RenderForwardMobile::TRANSFORMS_UNIFORM_SET);
}
{
RD::SamplerState sampler;
sampler.mag_filter = RD::SAMPLER_FILTER_LINEAR;
sampler.min_filter = RD::SAMPLER_FILTER_LINEAR;
sampler.enable_compare = true;
sampler.compare_op = RD::COMPARE_OP_GREATER;
shadow_sampler = RD::get_singleton()->sampler_create(sampler);
}
}
void SceneShaderForwardMobile::set_default_specialization(const ShaderSpecialization &p_specialization) {

4
servers/rendering/renderer_rd/renderer_canvas_render_rd.cpp
View File

@ -1843,6 +1843,10 @@ RendererCanvasRenderRD::RendererCanvasRenderRD() {
for (int i = 0; i < 2; i++) {
shadow_render.sdf_render_pipelines[i] = RD::get_singleton()->render_pipeline_create(shadow_render.shader.version_get_shader(shadow_render.shader_version, SHADOW_RENDER_MODE_SDF), shadow_render.sdf_framebuffer_format, shadow_render.sdf_vertex_format, i == 0 ? RD::RENDER_PRIMITIVE_TRIANGLES : RD::RENDER_PRIMITIVE_LINES, RD::PipelineRasterizationState(), RD::PipelineMultisampleState(), RD::PipelineDepthStencilState(), RD::PipelineColorBlendState::create_disabled(), 0);
}
// Unload shader modules to save memory.
RD::get_singleton()->shader_destroy_modules(shadow_render.shader.version_get_shader(shadow_render.shader_version, SHADOW_RENDER_MODE_SHADOW));
RD::get_singleton()->shader_destroy_modules(shadow_render.shader.version_get_shader(shadow_render.shader_version, SHADOW_RENDER_MODE_SDF));
}
{ //bindings

3
servers/rendering/renderer_rd/renderer_compositor_rd.cpp
View File

@ -133,6 +133,9 @@ void RendererCompositorRD::initialize() {
for (int i = 0; i < BLIT_MODE_MAX; i++) {
blit.pipelines[i] = RD::get_singleton()->render_pipeline_create(blit.shader.version_get_shader(blit.shader_version, i), RD::get_singleton()->screen_get_framebuffer_format(DisplayServer::MAIN_WINDOW_ID), RD::INVALID_ID, RD::RENDER_PRIMITIVE_TRIANGLES, RD::PipelineRasterizationState(), RD::PipelineMultisampleState(), RD::PipelineDepthStencilState(), i == BLIT_MODE_NORMAL_ALPHA ? RenderingDevice::PipelineColorBlendState::create_blend() : RenderingDevice::PipelineColorBlendState::create_disabled(), 0);
// Unload shader modules to save memory.
RD::get_singleton()->shader_destroy_modules(blit.shader.version_get_shader(blit.shader_version, i));
}
//create index array for copy shader

7
servers/rendering/renderer_rd/shader_rd.cpp
View File

@ -318,7 +318,7 @@ void ShaderRD::_compile_variant(uint32_t p_variant, CompileData p_data) {
{
MutexLock lock(variant_set_mutex);
p_data.version->variants.write[variant] = RD::get_singleton()->shader_create_from_bytecode(shader_data, p_data.version->variants[variant]);
p_data.version->variants.write[variant] = RD::get_singleton()->shader_create_from_bytecode_with_samplers(shader_data, p_data.version->variants[variant], immutable_samplers);
p_data.version->variant_data.write[variant] = shader_data;
}
}
@ -460,7 +460,7 @@ bool ShaderRD::_load_from_cache(Version *p_version, int p_group) {
}
{
MutexLock lock(variant_set_mutex);
RID shader = RD::get_singleton()->shader_create_from_bytecode(p_version->variant_data[variant_id], p_version->variants[variant_id]);
RID shader = RD::get_singleton()->shader_create_from_bytecode_with_samplers(p_version->variant_data[variant_id], p_version->variants[variant_id], immutable_samplers);
if (shader.is_null()) {
for (uint32_t j = 0; j < i; j++) {
int variant_free_id = group_to_variant_map[p_group][j];
@ -738,7 +738,8 @@ ShaderRD::ShaderRD() {
base_compute_defines = base_compute_define_text.ascii();
}
void ShaderRD::initialize(const Vector<String> &p_variant_defines, const String &p_general_defines) {
void ShaderRD::initialize(const Vector<String> &p_variant_defines, const String &p_general_defines, const Vector<RD::PipelineImmutableSampler> &r_immutable_samplers) {
immutable_samplers = r_immutable_samplers;
ERR_FAIL_COND(variant_defines.size());
ERR_FAIL_COND(p_variant_defines.is_empty());

4
servers/rendering/renderer_rd/shader_rd.h
View File

@ -63,6 +63,8 @@ private:
HashMap<int, LocalVector<int>> group_to_variant_map;
Vector<bool> group_enabled;
Vector<RD::PipelineImmutableSampler> immutable_samplers;
struct Version {
CharString uniforms;
CharString vertex_globals;
@ -211,7 +213,7 @@ public:
RS::ShaderNativeSourceCode version_get_native_source_code(RID p_version);
void initialize(const Vector<String> &p_variant_defines, const String &p_general_defines = "");
void initialize(const Vector<String> &p_variant_defines, const String &p_general_defines = "", const Vector<RD::PipelineImmutableSampler> &r_immutable_samplers = Vector<RD::PipelineImmutableSampler>());
void initialize(const Vector<VariantDefine> &p_variant_defines, const String &p_general_defines = "");
virtual ~ShaderRD();

8
servers/rendering/renderer_rd/storage_rd/render_scene_buffers_rd.cpp
View File

@ -171,16 +171,18 @@ void RenderSceneBuffersRD::configure(const RenderSceneBuffersConfiguration *p_co
const bool resolve_target = msaa_3d != RS::VIEWPORT_MSAA_DISABLED;
create_texture(RB_SCOPE_BUFFERS, RB_TEX_COLOR, base_data_format, get_color_usage_bits(resolve_target, false, can_be_storage));
const uint32_t extra_bits = RD::TEXTURE_USAGE_TRANSIENT_BIT;
// Create our depth buffer.
create_texture(RB_SCOPE_BUFFERS, RB_TEX_DEPTH, get_depth_format(resolve_target, false, can_be_storage), get_depth_usage_bits(resolve_target, false, can_be_storage));
create_texture(RB_SCOPE_BUFFERS, RB_TEX_DEPTH, get_depth_format(resolve_target, false, can_be_storage), get_depth_usage_bits(resolve_target, false, can_be_storage) | extra_bits);
// Create our MSAA buffers.
if (msaa_3d == RS::VIEWPORT_MSAA_DISABLED) {
texture_samples = RD::TEXTURE_SAMPLES_1;
} else {
texture_samples = msaa_to_samples(msaa_3d);
create_texture(RB_SCOPE_BUFFERS, RB_TEX_COLOR_MSAA, base_data_format, get_color_usage_bits(false, true, can_be_storage), texture_samples);
create_texture(RB_SCOPE_BUFFERS, RB_TEX_DEPTH_MSAA, get_depth_format(false, true, can_be_storage), get_depth_usage_bits(false, true, can_be_storage), texture_samples);
create_texture(RB_SCOPE_BUFFERS, RB_TEX_COLOR_MSAA, base_data_format, get_color_usage_bits(false, true, can_be_storage) | extra_bits, texture_samples);
create_texture(RB_SCOPE_BUFFERS, RB_TEX_DEPTH_MSAA, get_depth_format(false, true, can_be_storage), get_depth_usage_bits(false, true, can_be_storage) | extra_bits, texture_samples);
}
// VRS (note, our vrs object will only be set if VRS is supported)

322
servers/rendering/rendering_device.cpp
View File

@ -561,9 +561,12 @@ String RenderingDevice::get_perf_report() const {
}
void RenderingDevice::update_perf_report() {
perf_report_text = " gpu:" + String::num_int64(gpu_copy_count);
perf_report_text = "";
perf_report_text += " gpu:" + String::num_int64(gpu_copy_count);
perf_report_text += " bytes:" + String::num_int64(copy_bytes_count);
perf_report_text += " lazily alloc:" + String::num_int64(driver->get_lazily_memory_used());
gpu_copy_count = 0;
copy_bytes_count = 0;
}
@ -2557,6 +2560,15 @@ RenderingDevice::FramebufferFormatID RenderingDevice::framebuffer_get_format(RID
return framebuffer->format_id;
}
Size2 RenderingDevice::framebuffer_get_size(RID p_framebuffer) {
_THREAD_SAFE_METHOD_
Framebuffer *framebuffer = framebuffer_owner.get_or_null(p_framebuffer);
ERR_FAIL_NULL_V(framebuffer, Size2(0, 0));
return framebuffer->size;
}
bool RenderingDevice::framebuffer_is_valid(RID p_framebuffer) const {
_THREAD_SAFE_METHOD_
@ -2872,11 +2884,33 @@ Vector<uint8_t> RenderingDevice::shader_compile_binary_from_spirv(const Vector<S
}
RID RenderingDevice::shader_create_from_bytecode(const Vector<uint8_t> &p_shader_binary, RID p_placeholder) {
// Immutable samplers :
// Expanding api when creating shader to allow passing optionally a set of immutable samplers
// keeping existing api but extending it by sending an empty set.
Vector<PipelineImmutableSampler> immutable_samplers;
return shader_create_from_bytecode_with_samplers(p_shader_binary, p_placeholder, immutable_samplers);
}
RID RenderingDevice::shader_create_from_bytecode_with_samplers(const Vector<uint8_t> &p_shader_binary, RID p_placeholder, const Vector<PipelineImmutableSampler> &p_immutable_samplers) {
_THREAD_SAFE_METHOD_
ShaderDescription shader_desc;
String name;
RDD::ShaderID shader_id = driver->shader_create_from_bytecode(p_shader_binary, shader_desc, name);
Vector<RDD::ImmutableSampler> driver_immutable_samplers;
for (const PipelineImmutableSampler &source_sampler : p_immutable_samplers) {
RDD::ImmutableSampler driver_sampler;
driver_sampler.type = source_sampler.uniform_type;
driver_sampler.binding = source_sampler.binding;
for (uint32_t j = 0; j < source_sampler.get_id_count(); j++) {
RDD::SamplerID *sampler_driver_id = sampler_owner.get_or_null(source_sampler.get_id(j));
driver_sampler.ids.push_back(*sampler_driver_id);
}
driver_immutable_samplers.append(driver_sampler);
}
RDD::ShaderID shader_id = driver->shader_create_from_bytecode(p_shader_binary, shader_desc, name, driver_immutable_samplers);
ERR_FAIL_COND_V(!shader_id, RID());
// All good, let's create modules.
@ -2947,6 +2981,12 @@ RID RenderingDevice::shader_create_from_bytecode(const Vector<uint8_t> &p_shader
return id;
}
void RenderingDevice::shader_destroy_modules(RID p_shader) {
Shader *shader = shader_owner.get_or_null(p_shader);
ERR_FAIL_NULL(shader);
driver->shader_destroy_modules(shader->driver_id);
}
RID RenderingDevice::shader_create_placeholder() {
_THREAD_SAFE_METHOD_
@ -3004,7 +3044,7 @@ void RenderingDevice::_uniform_set_update_shared(UniformSet *p_uniform_set) {
}
}
RID RenderingDevice::uniform_set_create(const Vector<Uniform> &p_uniforms, RID p_shader, uint32_t p_shader_set) {
RID RenderingDevice::uniform_set_create(const Vector<Uniform> &p_uniforms, RID p_shader, uint32_t p_shader_set, bool p_linear_pool) {
_THREAD_SAFE_METHOD_
ERR_FAIL_COND_V(p_uniforms.is_empty(), RID());
@ -3055,6 +3095,9 @@ RID RenderingDevice::uniform_set_create(const Vector<Uniform> &p_uniforms, RID p
driver_uniform.type = uniform.uniform_type;
driver_uniform.binding = uniform.binding;
// Mark immutable samplers to be skipped when creating uniform set.
driver_uniform.immutable_sampler = uniform.immutable_sampler;
switch (uniform.uniform_type) {
case UNIFORM_TYPE_SAMPLER: {
if (uniform.get_id_count() != (uint32_t)set_uniform.length) {
@ -3370,7 +3413,7 @@ RID RenderingDevice::uniform_set_create(const Vector<Uniform> &p_uniforms, RID p
}
}
RDD::UniformSetID driver_uniform_set = driver->uniform_set_create(driver_uniforms, shader->driver_id, p_shader_set);
RDD::UniformSetID driver_uniform_set = driver->uniform_set_create(driver_uniforms, shader->driver_id, p_shader_set, p_linear_pool ? frame : -1);
ERR_FAIL_COND_V(!driver_uniform_set, RID());
UniformSet uniform_set;
@ -3416,6 +3459,10 @@ void RenderingDevice::uniform_set_set_invalidation_callback(RID p_uniform_set, I
us->invalidated_callback_userdata = p_userdata;
}
bool RenderingDevice::uniform_sets_have_linear_pools() const {
return driver->uniform_sets_have_linear_pools();
}
/*******************/
/**** PIPELINES ****/
/*******************/
@ -3695,6 +3742,7 @@ Error RenderingDevice::screen_create(DisplayServer::WindowID p_screen) {
Error RenderingDevice::screen_prepare_for_drawing(DisplayServer::WindowID p_screen) {
_THREAD_SAFE_METHOD_
// After submitting work, acquire the swapchain image(s)
HashMap<DisplayServer::WindowID, RDD::SwapChainID>::ConstIterator it = screen_swap_chains.find(p_screen);
ERR_FAIL_COND_V_MSG(it == screen_swap_chains.end(), ERR_CANT_CREATE, "A swap chain was not created for the screen.");
@ -3831,7 +3879,7 @@ RenderingDevice::DrawListID RenderingDevice::draw_list_begin_for_screen(DisplayS
clear_value.color = p_clear_color;
RDD::RenderPassID render_pass = driver->swap_chain_get_render_pass(sc_it->value);
draw_graph.add_draw_list_begin(render_pass, fb_it->value, viewport, clear_value, true, false, RDD::BreadcrumbMarker::BLIT_PASS);
draw_graph.add_draw_list_begin(render_pass, fb_it->value, viewport, clear_value, true, false, RDD::BreadcrumbMarker::BLIT_PASS, split_swapchain_into_its_own_cmd_buffer);
draw_graph.add_draw_list_set_viewport(viewport);
draw_graph.add_draw_list_set_scissor(viewport);
@ -3890,8 +3938,8 @@ Error RenderingDevice::_draw_list_render_pass_begin(Framebuffer *p_framebuffer,
clear_values.resize(p_framebuffer->texture_ids.size());
resource_trackers.clear();
resource_usages.clear();
int clear_values_count = 0;
{
int clear_values_count = 0;
int color_index = 0;
for (int i = 0; i < p_framebuffer->texture_ids.size(); i++) {
RDD::RenderPassClearValue clear_value;
@ -4365,37 +4413,69 @@ void RenderingDevice::draw_list_draw(DrawListID p_list, bool p_use_indices, uint
}
}
#endif
thread_local LocalVector<RDD::UniformSetID> valid_descriptor_ids;
valid_descriptor_ids.clear();
valid_descriptor_ids.resize(dl->state.set_count);
uint32_t valid_set_count = 0;
uint32_t first_set_index = 0;
uint32_t last_set_index = 0;
bool found_first_set = false;
// Prepare descriptor sets if the API doesn't use pipeline barriers.
if (!driver->api_trait_get(RDD::API_TRAIT_HONORS_PIPELINE_BARRIERS)) {
for (uint32_t i = 0; i < dl->state.set_count; i++) {
if (dl->state.sets[i].pipeline_expected_format == 0) {
// Nothing expected by this pipeline.
continue;
}
for (uint32_t i = 0; i < dl->state.set_count; i++) {
if (dl->state.sets[i].pipeline_expected_format == 0) {
continue; // Nothing expected by this pipeline.
}
if (!dl->state.sets[i].bound && !found_first_set) {
first_set_index = i;
found_first_set = true;
}
// Prepare descriptor sets if the API doesn't use pipeline barriers.
if (!driver->api_trait_get(RDD::API_TRAIT_HONORS_PIPELINE_BARRIERS)) {
draw_graph.add_draw_list_uniform_set_prepare_for_use(dl->state.pipeline_shader_driver_id, dl->state.sets[i].uniform_set_driver_id, i);
}
}
// Bind descriptor sets.
for (uint32_t i = 0; i < dl->state.set_count; i++) {
for (uint32_t i = first_set_index; i < dl->state.set_count; i++) {
if (dl->state.sets[i].pipeline_expected_format == 0) {
continue; // Nothing expected by this pipeline.
}
if (!dl->state.sets[i].bound) {
// All good, see if this requires re-binding.
draw_graph.add_draw_list_bind_uniform_set(dl->state.pipeline_shader_driver_id, dl->state.sets[i].uniform_set_driver_id, i);
// Batch contiguous descriptor sets in a single call
if (descriptor_set_batching) {
// All good, see if this requires re-binding.
if (i - last_set_index > 1) {
// If the descriptor sets are not contiguous, bind the previous ones and start a new batch
draw_graph.add_draw_list_bind_uniform_sets(dl->state.pipeline_shader_driver_id, valid_descriptor_ids, first_set_index, valid_set_count);
UniformSet *uniform_set = uniform_set_owner.get_or_null(dl->state.sets[i].uniform_set);
_uniform_set_update_shared(uniform_set);
first_set_index = i;
valid_set_count = 1;
valid_descriptor_ids[0] = dl->state.sets[i].uniform_set_driver_id;
} else {
// Otherwise, keep storing in the current batch
valid_descriptor_ids[valid_set_count] = dl->state.sets[i].uniform_set_driver_id;
valid_set_count++;
}
draw_graph.add_draw_list_usages(uniform_set->draw_trackers, uniform_set->draw_trackers_usage);
UniformSet *uniform_set = uniform_set_owner.get_or_null(dl->state.sets[i].uniform_set);
_uniform_set_update_shared(uniform_set);
draw_graph.add_draw_list_usages(uniform_set->draw_trackers, uniform_set->draw_trackers_usage);
dl->state.sets[i].bound = true;
dl->state.sets[i].bound = true;
last_set_index = i;
} else {
draw_graph.add_draw_list_bind_uniform_set(dl->state.pipeline_shader_driver_id, dl->state.sets[i].uniform_set_driver_id, i);
}
}
}
// Bind the remaining batch
if (descriptor_set_batching && valid_set_count > 0) {
draw_graph.add_draw_list_bind_uniform_sets(dl->state.pipeline_shader_driver_id, valid_descriptor_ids, first_set_index, valid_set_count);
}
if (p_use_indices) {
#ifdef DEBUG_ENABLED
ERR_FAIL_COND_MSG(p_procedural_vertices > 0,
@ -4560,6 +4640,22 @@ void RenderingDevice::draw_list_draw_indirect(DrawListID p_list, bool p_use_indi
_check_transfer_worker_buffer(buffer);
}
void RenderingDevice::draw_list_set_viewport(DrawListID p_list, const Rect2 &p_rect) {
DrawList *dl = _get_draw_list_ptr(p_list);
ERR_FAIL_NULL(dl);
#ifdef DEBUG_ENABLED
ERR_FAIL_COND_MSG(!dl->validation.active, "Submitted Draw Lists can no longer be modified.");
#endif
if (p_rect.get_area() == 0) {
return;
}
dl->viewport = p_rect;
draw_graph.add_draw_list_set_viewport(p_rect);
}
void RenderingDevice::draw_list_enable_scissor(DrawListID p_list, const Rect2 &p_rect) {
ERR_RENDER_THREAD_GUARD();
@ -4884,37 +4980,70 @@ void RenderingDevice::compute_list_dispatch(ComputeListID p_list, uint32_t p_x_g
}
}
#endif
thread_local LocalVector<RDD::UniformSetID> valid_descriptor_ids;
valid_descriptor_ids.clear();
valid_descriptor_ids.resize(cl->state.set_count);
// Prepare descriptor sets if the API doesn't use pipeline barriers.
if (!driver->api_trait_get(RDD::API_TRAIT_HONORS_PIPELINE_BARRIERS)) {
for (uint32_t i = 0; i < cl->state.set_count; i++) {
if (cl->state.sets[i].pipeline_expected_format == 0) {
// Nothing expected by this pipeline.
continue;
}
uint32_t valid_set_count = 0;
uint32_t first_set_index = 0;
uint32_t last_set_index = 0;
bool found_first_set = false;
for (uint32_t i = 0; i < cl->state.set_count; i++) {
if (cl->state.sets[i].pipeline_expected_format == 0) {
// Nothing expected by this pipeline.
continue;
}
if (!cl->state.sets[i].bound && !found_first_set) {
first_set_index = i;
found_first_set = true;
}
// Prepare descriptor sets if the API doesn't use pipeline barriers.
if (!driver->api_trait_get(RDD::API_TRAIT_HONORS_PIPELINE_BARRIERS)) {
draw_graph.add_compute_list_uniform_set_prepare_for_use(cl->state.pipeline_shader_driver_id, cl->state.sets[i].uniform_set_driver_id, i);
}
}
// Bind descriptor sets.
for (uint32_t i = 0; i < cl->state.set_count; i++) {
for (uint32_t i = first_set_index; i < cl->state.set_count; i++) {
if (cl->state.sets[i].pipeline_expected_format == 0) {
continue; // Nothing expected by this pipeline.
}
if (!cl->state.sets[i].bound) {
// All good, see if this requires re-binding.
draw_graph.add_compute_list_bind_uniform_set(cl->state.pipeline_shader_driver_id, cl->state.sets[i].uniform_set_driver_id, i);
if (!cl->state.sets[i].bound) {
// Descriptor set batching
if (descriptor_set_batching) {
// All good, see if this requires re-binding.
if (i - last_set_index > 1) {
// If the descriptor sets are not contiguous, bind the previous ones and start a new batch
draw_graph.add_compute_list_bind_uniform_sets(cl->state.pipeline_shader_driver_id, valid_descriptor_ids, first_set_index, valid_set_count);
first_set_index = i;
valid_set_count = 1;
valid_descriptor_ids[0] = cl->state.sets[i].uniform_set_driver_id;
} else {
// Otherwise, keep storing in the current batch
valid_descriptor_ids[valid_set_count] = cl->state.sets[i].uniform_set_driver_id;
valid_set_count++;
}
last_set_index = i;
} else {
draw_graph.add_compute_list_bind_uniform_set(cl->state.pipeline_shader_driver_id, cl->state.sets[i].uniform_set_driver_id, i);
}
UniformSet *uniform_set = uniform_set_owner.get_or_null(cl->state.sets[i].uniform_set);
_uniform_set_update_shared(uniform_set);
draw_graph.add_compute_list_usages(uniform_set->draw_trackers, uniform_set->draw_trackers_usage);
cl->state.sets[i].bound = true;
}
}
// Bind the remaining batch
if (valid_set_count > 0) {
draw_graph.add_compute_list_bind_uniform_sets(cl->state.pipeline_shader_driver_id, valid_descriptor_ids, first_set_index, valid_set_count);
}
draw_graph.add_compute_list_dispatch(p_x_groups, p_y_groups, p_z_groups);
cl->state.dispatch_count++;
}
@ -4997,37 +5126,68 @@ void RenderingDevice::compute_list_dispatch_indirect(ComputeListID p_list, RID p
}
}
#endif
thread_local LocalVector<RDD::UniformSetID> valid_descriptor_ids;
valid_descriptor_ids.clear();
valid_descriptor_ids.resize(cl->state.set_count);
// Prepare descriptor sets if the API doesn't use pipeline barriers.
if (!driver->api_trait_get(RDD::API_TRAIT_HONORS_PIPELINE_BARRIERS)) {
for (uint32_t i = 0; i < cl->state.set_count; i++) {
if (cl->state.sets[i].pipeline_expected_format == 0) {
// Nothing expected by this pipeline.
continue;
}
uint32_t valid_set_count = 0;
uint32_t first_set_index = 0;
uint32_t last_set_index = 0;
bool found_first_set = false;
for (uint32_t i = 0; i < cl->state.set_count; i++) {
if (cl->state.sets[i].pipeline_expected_format == 0) {
// Nothing expected by this pipeline.
continue;
}
if (!cl->state.sets[i].bound && !found_first_set) {
first_set_index = i;
found_first_set = true;
}
// Prepare descriptor sets if the API doesn't use pipeline barriers.
if (!driver->api_trait_get(RDD::API_TRAIT_HONORS_PIPELINE_BARRIERS)) {
draw_graph.add_compute_list_uniform_set_prepare_for_use(cl->state.pipeline_shader_driver_id, cl->state.sets[i].uniform_set_driver_id, i);
}
}
// Bind descriptor sets.
for (uint32_t i = 0; i < cl->state.set_count; i++) {
for (uint32_t i = first_set_index; i < cl->state.set_count; i++) {
if (cl->state.sets[i].pipeline_expected_format == 0) {
continue; // Nothing expected by this pipeline.
}
if (!cl->state.sets[i].bound) {
// All good, see if this requires re-binding.
draw_graph.add_compute_list_bind_uniform_set(cl->state.pipeline_shader_driver_id, cl->state.sets[i].uniform_set_driver_id, i);
if (i - last_set_index > 1) {
// If the descriptor sets are not contiguous, bind the previous ones and start a new batch
draw_graph.add_compute_list_bind_uniform_sets(cl->state.pipeline_shader_driver_id, valid_descriptor_ids, first_set_index, valid_set_count);
first_set_index = i;
valid_set_count = 1;
valid_descriptor_ids[0] = cl->state.sets[i].uniform_set_driver_id;
} else {
// Otherwise, keep storing in the current batch
valid_descriptor_ids[valid_set_count] = cl->state.sets[i].uniform_set_driver_id;
valid_set_count++;
}
last_set_index = i;
UniformSet *uniform_set = uniform_set_owner.get_or_null(cl->state.sets[i].uniform_set);
_uniform_set_update_shared(uniform_set);
draw_graph.add_compute_list_usages(uniform_set->draw_trackers, uniform_set->draw_trackers_usage);
cl->state.sets[i].bound = true;
}
}
// Bind the remaining batch
if (valid_set_count > 0) {
draw_graph.add_compute_list_bind_uniform_sets(cl->state.pipeline_shader_driver_id, valid_descriptor_ids, first_set_index, valid_set_count);
}
draw_graph.add_compute_list_dispatch_indirect(buffer->driver_id, p_offset);
cl->state.dispatch_count++;
@ -5264,6 +5424,7 @@ void RenderingDevice::_submit_transfer_worker(TransferWorker *p_transfer_worker,
void RenderingDevice::_wait_for_transfer_worker(TransferWorker *p_transfer_worker) {
driver->fence_wait(p_transfer_worker->command_fence);
driver->command_pool_reset(p_transfer_worker->command_pool);
p_transfer_worker->staging_buffer_size_in_use = 0;
p_transfer_worker->submitted = false;
@ -5778,7 +5939,8 @@ void RenderingDevice::swap_buffers() {
// Advance to the next frame and begin recording again.
frame = (frame + 1) % frames.size();
_begin_frame();
_begin_frame(true);
}
void RenderingDevice::submit() {
@ -5796,7 +5958,7 @@ void RenderingDevice::sync() {
ERR_FAIL_COND_MSG(is_main_instance, "Only local devices can submit and sync.");
ERR_FAIL_COND_MSG(!local_device_processing, "sync can only be called after a submit");
_begin_frame();
_begin_frame(true);
local_device_processing = false;
}
@ -5906,14 +6068,22 @@ uint64_t RenderingDevice::get_memory_usage(MemoryType p_type) const {
}
}
void RenderingDevice::_begin_frame() {
void RenderingDevice::_begin_frame(bool p_presented) {
// Before beginning this frame, wait on the fence if it was signaled to make sure its work is finished.
if (frames[frame].fence_signaled) {
driver->fence_wait(frames[frame].fence);
frames[frame].fence_signaled = false;
}
update_perf_report();
if (command_pool_reset_enabled) {
bool reset = driver->command_pool_reset(frames[frame].command_pool);
ERR_FAIL_COND(!reset);
}
if (p_presented) {
update_perf_report();
driver->linear_uniform_set_pools_reset(frame);
}
// Begin recording on the frame's command buffers.
driver->begin_segment(frame, frames_drawn++);
@ -5962,15 +6132,11 @@ void RenderingDevice::_end_frame() {
driver->end_segment();
}
void RenderingDevice::_execute_frame(bool p_present) {
// Check whether this frame should present the swap chains and in which queue.
const bool frame_can_present = p_present && !frames[frame].swap_chains_to_present.is_empty();
const bool separate_present_queue = main_queue != present_queue;
thread_local LocalVector<RDD::SwapChainID> swap_chains;
swap_chains.clear();
// Execute command buffers and use semaphores to wait on the execution of the previous one. Normally there's only one command buffer,
// but driver workarounds can force situations where there'll be more.
void RenderingDevice::execute_chained_cmds(bool p_present_swap_chain, RenderingDeviceDriver::FenceID p_draw_fence,
RenderingDeviceDriver::SemaphoreID p_dst_draw_semaphore_to_signal) {
// Execute command buffers and use semaphores to wait on the execution of the previous one.
// Normally there's only one command buffer, but driver workarounds can force situations where
// there'll be more.
uint32_t command_buffer_count = 1;
RDG::CommandBufferPool &buffer_pool = frames[frame].command_buffer_pool;
if (buffer_pool.buffers_used > 0) {
@ -5978,6 +6144,12 @@ void RenderingDevice::_execute_frame(bool p_present) {
buffer_pool.buffers_used = 0;
}
thread_local LocalVector<RDD::SwapChainID> swap_chains;
swap_chains.clear();
// Instead of having just one command; we have potentially many (which had to be split due to an
// Adreno workaround on mobile, only if the workaround is active). Thus we must execute all of them
// and chain them together via semaphores as dependent executions.
thread_local LocalVector<RDD::SemaphoreID> wait_semaphores;
wait_semaphores = frames[frame].semaphores_to_wait_on;
@ -5987,45 +6159,57 @@ void RenderingDevice::_execute_frame(bool p_present) {
RDD::FenceID signal_fence;
if (i > 0) {
command_buffer = buffer_pool.buffers[i - 1];
signal_semaphore = buffer_pool.semaphores[i - 1];
} else {
command_buffer = frames[frame].command_buffer;
signal_semaphore = frames[frame].semaphore;
}
bool signal_semaphore_valid;
if (i == (command_buffer_count - 1)) {
// This is the last command buffer, it should signal the fence.
signal_fence = frames[frame].fence;
signal_semaphore_valid = false;
// This is the last command buffer, it should signal the semaphore & fence.
signal_semaphore = p_dst_draw_semaphore_to_signal;
signal_fence = p_draw_fence;
if (frame_can_present && separate_present_queue) {
// The semaphore is required if the frame can be presented and a separate present queue is used.
signal_semaphore_valid = true;
} else if (frame_can_present) {
if (p_present_swap_chain) {
// Just present the swap chains as part of the last command execution.
swap_chains = frames[frame].swap_chains_to_present;
}
} else {
signal_semaphore = buffer_pool.semaphores[i];
// Semaphores always need to be signaled if it's not the last command buffer.
signal_semaphore_valid = true;
}
driver->command_queue_execute_and_present(main_queue, wait_semaphores, command_buffer, signal_semaphore_valid ? signal_semaphore : VectorView<RDD::SemaphoreID>(), signal_fence, swap_chains);
driver->command_queue_execute_and_present(main_queue, wait_semaphores, command_buffer,
signal_semaphore ? signal_semaphore : VectorView<RDD::SemaphoreID>(), signal_fence,
swap_chains);
// Make the next command buffer wait on the semaphore signaled by this one.
wait_semaphores.resize(1);
wait_semaphores[0] = signal_semaphore;
}
// Indicate the fence has been signaled so the next time the frame's contents need to be used, the CPU needs to wait on the work to be completed.
frames[frame].semaphores_to_wait_on.clear();
}
void RenderingDevice::_execute_frame(bool p_present) {
// Check whether this frame should present the swap chains and in which queue.
const bool frame_can_present = p_present && !frames[frame].swap_chains_to_present.is_empty();
const bool separate_present_queue = main_queue != present_queue;
// The semaphore is required if the frame can be presented and a separate present queue is used;
// since the separate queue will wait for that semaphore before presenting.
const RDD::SemaphoreID semaphore = (frame_can_present && separate_present_queue)
? frames[frame].semaphore
: RDD::SemaphoreID(nullptr);
const bool present_swap_chain = frame_can_present && !separate_present_queue;
execute_chained_cmds(present_swap_chain, frames[frame].fence, semaphore);
// Indicate the fence has been signaled so the next time the frame's contents need to be
// used, the CPU needs to wait on the work to be completed.
frames[frame].fence_signaled = true;
if (frame_can_present) {
if (separate_present_queue) {
// Issue the presentation separately if the presentation queue is different from the main queue.
driver->command_queue_execute_and_present(present_queue, wait_semaphores, {}, {}, {}, frames[frame].swap_chains_to_present);
driver->command_queue_execute_and_present(present_queue, frames[frame].semaphore, {}, {}, {}, frames[frame].swap_chains_to_present);
}
frames[frame].swap_chains_to_present.clear();

45
servers/rendering/rendering_device.h
View File

@ -191,7 +191,12 @@ private:
Error _buffer_initialize(Buffer *p_buffer, const uint8_t *p_data, size_t p_data_size, uint32_t p_required_align = 32);
void update_perf_report();
// flag for batching descriptor sets
bool descriptor_set_batching = true;
// When true, the final draw call that copies our offscreen result into the Swapchain is put into its
// own cmd buffer, so that the whole rendering can start early instead of having to wait for the
// swapchain semaphore to be signaled (which causes bubbles).
bool split_swapchain_into_its_own_cmd_buffer = true;
uint32_t gpu_copy_count = 0;
uint32_t copy_bytes_count = 0;
String perf_report_text;
@ -599,6 +604,7 @@ public:
void framebuffer_set_invalidation_callback(RID p_framebuffer, InvalidationCallback p_callback, void *p_userdata);
FramebufferFormatID framebuffer_get_format(RID p_framebuffer);
Size2 framebuffer_get_size(RID p_framebuffer);
/*****************/
/**** SAMPLER ****/
@ -877,6 +883,7 @@ public:
RID shader_create_from_spirv(const Vector<ShaderStageSPIRVData> &p_spirv, const String &p_shader_name = "");
RID shader_create_from_bytecode(const Vector<uint8_t> &p_shader_binary, RID p_placeholder = RID());
RID shader_create_placeholder();
void shader_destroy_modules(RID p_shader);
uint64_t shader_get_vertex_input_attribute_mask(RID p_shader);
@ -889,13 +896,20 @@ public:
STORAGE_BUFFER_USAGE_DISPATCH_INDIRECT = 1,
};
/*****************/
/**** BUFFERS ****/
/*****************/
RID uniform_buffer_create(uint32_t p_size_bytes, const Vector<uint8_t> &p_data = Vector<uint8_t>());
RID storage_buffer_create(uint32_t p_size, const Vector<uint8_t> &p_data = Vector<uint8_t>(), BitField<StorageBufferUsage> p_usage = 0);
RID texture_buffer_create(uint32_t p_size_elements, DataFormat p_format, const Vector<uint8_t> &p_data = Vector<uint8_t>());
struct Uniform {
UniformType uniform_type = UNIFORM_TYPE_IMAGE;
uint32_t binding = 0; // Binding index as specified in shader.
// This flag specifies that this is an immutable sampler to be set when creating pipeline layout.
bool immutable_sampler = false;
private:
// In most cases only one ID is provided per binding, so avoid allocating memory unnecessarily for performance.
@ -956,6 +970,9 @@ public:
_FORCE_INLINE_ Uniform() = default;
};
typedef Uniform PipelineImmutableSampler;
RID shader_create_from_bytecode_with_samplers(const Vector<uint8_t> &p_shader_binary, RID p_placeholder = RID(), const Vector<PipelineImmutableSampler> &p_immutable_samplers = Vector<PipelineImmutableSampler>());
private:
static const uint32_t MAX_UNIFORM_SETS = 16;
static const uint32_t MAX_PUSH_CONSTANT_SIZE = 128;
@ -997,10 +1014,22 @@ private:
void _uniform_set_update_shared(UniformSet *p_uniform_set);
public:
RID uniform_set_create(const Vector<Uniform> &p_uniforms, RID p_shader, uint32_t p_shader_set);
/** Bake a set of uniforms that can be bound at runtime with the given shader.
* @remark Setting p_linear_pool = true while keeping the RID around for longer than the current frame will result in undefined behavior.
* @param p_uniforms The uniforms to bake into a set.
* @param p_shader The shader you intend to bind these uniforms with.
* @param p_set_index The set. Should be in range [0; 4)
* The value 4 comes from physical_device_properties.limits.maxBoundDescriptorSets. Vulkan only guarantees maxBoundDescriptorSets >= 4 (== 4 is very common on Mobile).
* @param p_linear_pool If you call this function every frame (and free the returned RID within the same frame!), set it to true for better performance.
* If you plan on keeping the return value around for more than one frame (e.g. Sets that are created once and reused forever) you MUST set it to false.
* @return Baked descriptor set.
*/
RID uniform_set_create(const Vector<Uniform> &p_uniforms, RID p_shader, uint32_t p_shader_set, bool p_linear_pool = false);
bool uniform_set_is_valid(RID p_uniform_set);
void uniform_set_set_invalidation_callback(RID p_uniform_set, InvalidationCallback p_callback, void *p_userdata);
bool uniform_sets_have_linear_pools() const;
/*******************/
/**** PIPELINES ****/
/*******************/
@ -1189,6 +1218,7 @@ public:
void draw_list_draw(DrawListID p_list, bool p_use_indices, uint32_t p_instances = 1, uint32_t p_procedural_vertices = 0);
void draw_list_draw_indirect(DrawListID p_list, bool p_use_indices, RID p_buffer, uint32_t p_offset = 0, uint32_t p_draw_count = 1, uint32_t p_stride = 0);
void draw_list_set_viewport(DrawListID p_list, const Rect2 &p_rect);
void draw_list_enable_scissor(DrawListID p_list, const Rect2 &p_rect);
void draw_list_disable_scissor(DrawListID p_list);
@ -1382,7 +1412,8 @@ private:
// This must have the same size of the transfer worker pool.
TightLocalVector<RDD::SemaphoreID> transfer_worker_semaphores;
// Extra command buffer pool used for driver workarounds.
// Extra command buffer pool used for driver workarounds or to reduce GPU bubbles by
// splitting the final render pass to the swapchain into its own cmd buffer.
RDG::CommandBufferPool command_buffer_pool;
struct Timestamp {
@ -1413,8 +1444,14 @@ private:
uint64_t texture_memory = 0;
uint64_t buffer_memory = 0;
protected:
void execute_chained_cmds(bool p_present_swap_chain,
RenderingDeviceDriver::FenceID p_draw_fence,
RenderingDeviceDriver::SemaphoreID p_dst_draw_semaphore_to_signal);
public:
void _free_internal(RID p_id);
void _begin_frame();
void _begin_frame(bool p_presented = false);
void _end_frame();
void _execute_frame(bool p_present);
void _stall_for_previous_frames();

18
servers/rendering/rendering_device_commons.h
View File

@ -43,6 +43,8 @@ class RenderingDeviceCommons : public Object {
// with RenderingDeviceDriver.
////////////////////////////////////////////
public:
static const bool command_pool_reset_enabled = true;
/*****************/
/**** GENERIC ****/
/*****************/
@ -359,6 +361,22 @@ public:
TEXTURE_USAGE_CAN_COPY_TO_BIT = (1 << 8),
TEXTURE_USAGE_INPUT_ATTACHMENT_BIT = (1 << 9),
TEXTURE_USAGE_VRS_ATTACHMENT_BIT = (1 << 10),
// When set, the texture is not backed by actual memory. It only ever lives in the cache.
// This is particularly useful for:
// 1. Depth/stencil buffers that are not needed after producing the colour output.
// 2. MSAA surfaces that are immediately resolved (i.e. its raw content isn't needed).
//
// This flag heavily improves performance & saves memory on TBDR GPUs (e.g. mobile).
// On Desktop this flag won't save memory but it still instructs the render graph that data will
// be discarded aggressively which may still improve some performance.
//
// It is not valid to perform copies from/to this texture, since it doesn't occupy actual RAM.
// It is also not valid to sample from this texture except using subpasses or via read/write
// pixel shader extensions (e.g. VK_EXT_rasterization_order_attachment_access).
//
// Try to set this bit as much as possible. If you set it, validation doesn't complain
// and it works fine on mobile, then go ahead.
TEXTURE_USAGE_TRANSIENT_BIT = (1 << 11),
};
struct TextureFormat {

27
servers/rendering/rendering_device_driver.h
View File

@ -426,6 +426,7 @@ public:
};
virtual CommandPoolID command_pool_create(CommandQueueFamilyID p_cmd_queue_family, CommandBufferType p_cmd_buffer_type) = 0;
virtual bool command_pool_reset(CommandPoolID p_cmd_pool) = 0;
virtual void command_pool_free(CommandPoolID p_cmd_pool) = 0;
// ----- BUFFER -----
@ -478,7 +479,21 @@ public:
virtual String shader_get_binary_cache_key() = 0;
virtual Vector<uint8_t> shader_compile_binary_from_spirv(VectorView<ShaderStageSPIRVData> p_spirv, const String &p_shader_name) = 0;
virtual ShaderID shader_create_from_bytecode(const Vector<uint8_t> &p_shader_binary, ShaderDescription &r_shader_desc, String &r_name) = 0;
struct ImmutableSampler {
UniformType type = UNIFORM_TYPE_MAX;
uint32_t binding = 0xffffffff; // Binding index as specified in shader.
LocalVector<ID> ids;
};
/** Creates a Pipeline State Object (PSO) out of the shader and all the input data it needs.
@param p_shader_binary Shader binary bytecode (e.g. SPIR-V).
@param r_shader_desc TBD.
@param r_name TBD.
@param p_immutable_samplers Immutable samplers can be embedded when creating the pipeline layout on the condition they
remain valid and unchanged, so they don't need to be specified when creating uniform sets.
@return PSO resource for binding.
*/
virtual ShaderID shader_create_from_bytecode(const Vector<uint8_t> &p_shader_binary, ShaderDescription &r_shader_desc, String &r_name, const Vector<ImmutableSampler> &p_immutable_samplers) = 0;
// Only meaningful if API_TRAIT_SHADER_CHANGE_INVALIDATION is SHADER_CHANGE_INVALIDATION_ALL_OR_NONE_ACCORDING_TO_LAYOUT_HASH.
virtual uint32_t shader_get_layout_hash(ShaderID p_shader) { return 0; }
virtual void shader_free(ShaderID p_shader) = 0;
@ -497,10 +512,15 @@ public:
UniformType type = UNIFORM_TYPE_MAX;
uint32_t binding = 0xffffffff; // Binding index as specified in shader.
LocalVector<ID> ids;
// Flag to indicate that this is an immutable sampler so it is skipped when creating uniform
// sets, as it would be set previously when creating the pipeline layout.
bool immutable_sampler = false;
};
virtual UniformSetID uniform_set_create(VectorView<BoundUniform> p_uniforms, ShaderID p_shader, uint32_t p_set_index) = 0;
virtual UniformSetID uniform_set_create(VectorView<BoundUniform> p_uniforms, ShaderID p_shader, uint32_t p_set_index, int p_linear_pool_index) = 0;
virtual void linear_uniform_set_pools_reset(int p_linear_pool_index) {}
virtual void uniform_set_free(UniformSetID p_uniform_set) = 0;
virtual bool uniform_sets_have_linear_pools() const { return false; }
// ----- COMMANDS -----
@ -642,6 +662,7 @@ public:
// Binding.
virtual void command_bind_render_pipeline(CommandBufferID p_cmd_buffer, PipelineID p_pipeline) = 0;
virtual void command_bind_render_uniform_set(CommandBufferID p_cmd_buffer, UniformSetID p_uniform_set, ShaderID p_shader, uint32_t p_set_index) = 0;
virtual void command_bind_render_uniform_sets(CommandBufferID p_cmd_buffer, VectorView<UniformSetID> p_uniform_sets, ShaderID p_shader, uint32_t p_first_set_index, uint32_t p_set_count) = 0;
// Drawing.
virtual void command_render_draw(CommandBufferID p_cmd_buffer, uint32_t p_vertex_count, uint32_t p_instance_count, uint32_t p_base_vertex, uint32_t p_first_instance) = 0;
@ -684,6 +705,7 @@ public:
// Binding.
virtual void command_bind_compute_pipeline(CommandBufferID p_cmd_buffer, PipelineID p_pipeline) = 0;
virtual void command_bind_compute_uniform_set(CommandBufferID p_cmd_buffer, UniformSetID p_uniform_set, ShaderID p_shader, uint32_t p_set_index) = 0;
virtual void command_bind_compute_uniform_sets(CommandBufferID p_cmd_buffer, VectorView<UniformSetID> p_uniform_sets, ShaderID p_shader, uint32_t p_first_set_index, uint32_t p_set_count) = 0;
// Dispatching.
virtual void command_compute_dispatch(CommandBufferID p_cmd_buffer, uint32_t p_x_groups, uint32_t p_y_groups, uint32_t p_z_groups) = 0;
@ -785,6 +807,7 @@ public:
virtual void set_object_name(ObjectType p_type, ID p_driver_id, const String &p_name) = 0;
virtual uint64_t get_resource_native_handle(DriverResource p_type, ID p_driver_id) = 0;
virtual uint64_t get_total_memory_used() = 0;
virtual uint64_t get_lazily_memory_used() = 0;
virtual uint64_t limit_get(Limit p_limit) = 0;
virtual uint64_t api_trait_get(ApiTrait p_trait);
virtual bool has_feature(Features p_feature) = 0;

99
servers/rendering/rendering_device_graph.cpp
View File

@ -650,10 +650,10 @@ void RenderingDeviceGraph::_run_compute_list_command(RDD::CommandBufferID p_comm
driver->command_bind_compute_pipeline(p_command_buffer, bind_pipeline_instruction->pipeline);
instruction_data_cursor += sizeof(ComputeListBindPipelineInstruction);
} break;
case ComputeListInstruction::TYPE_BIND_UNIFORM_SET: {
const ComputeListBindUniformSetInstruction *bind_uniform_set_instruction = reinterpret_cast<const ComputeListBindUniformSetInstruction *>(instruction);
driver->command_bind_compute_uniform_set(p_command_buffer, bind_uniform_set_instruction->uniform_set, bind_uniform_set_instruction->shader, bind_uniform_set_instruction->set_index);
instruction_data_cursor += sizeof(ComputeListBindUniformSetInstruction);
case ComputeListInstruction::TYPE_BIND_UNIFORM_SETS: {
const ComputeListBindUniformSetsInstruction *bind_uniform_sets_instruction = reinterpret_cast<const ComputeListBindUniformSetsInstruction *>(instruction);
driver->command_bind_compute_uniform_sets(p_command_buffer, VectorView<RDD::UniformSetID>(bind_uniform_sets_instruction->uniform_set_ids(), bind_uniform_sets_instruction->set_count), bind_uniform_sets_instruction->shader, bind_uniform_sets_instruction->first_set_index, bind_uniform_sets_instruction->set_count);
instruction_data_cursor += sizeof(ComputeListBindUniformSetsInstruction) + sizeof(RDD::UniformSetID) * bind_uniform_sets_instruction->set_count;
} break;
case ComputeListInstruction::TYPE_DISPATCH: {
const ComputeListDispatchInstruction *dispatch_instruction = reinterpret_cast<const ComputeListDispatchInstruction *>(instruction);
@ -701,10 +701,10 @@ void RenderingDeviceGraph::_run_draw_list_command(RDD::CommandBufferID p_command
driver->command_bind_render_pipeline(p_command_buffer, bind_pipeline_instruction->pipeline);
instruction_data_cursor += sizeof(DrawListBindPipelineInstruction);
} break;
case DrawListInstruction::TYPE_BIND_UNIFORM_SET: {
const DrawListBindUniformSetInstruction *bind_uniform_set_instruction = reinterpret_cast<const DrawListBindUniformSetInstruction *>(instruction);
driver->command_bind_render_uniform_set(p_command_buffer, bind_uniform_set_instruction->uniform_set, bind_uniform_set_instruction->shader, bind_uniform_set_instruction->set_index);
instruction_data_cursor += sizeof(DrawListBindUniformSetInstruction);
case DrawListInstruction::TYPE_BIND_UNIFORM_SETS: {
const DrawListBindUniformSetsInstruction *bind_uniform_sets_instruction = reinterpret_cast<const DrawListBindUniformSetsInstruction *>(instruction);
driver->command_bind_render_uniform_sets(p_command_buffer, VectorView<RDD::UniformSetID>(bind_uniform_sets_instruction->uniform_set_ids(), bind_uniform_sets_instruction->set_count), bind_uniform_sets_instruction->shader, bind_uniform_sets_instruction->first_set_index, bind_uniform_sets_instruction->set_count);
instruction_data_cursor += sizeof(DrawListBindUniformSetsInstruction) + sizeof(RDD::UniformSetID) * bind_uniform_sets_instruction->set_count;
} break;
case DrawListInstruction::TYPE_BIND_VERTEX_BUFFERS: {
const DrawListBindVertexBuffersInstruction *bind_vertex_buffers_instruction = reinterpret_cast<const DrawListBindVertexBuffersInstruction *>(instruction);
@ -865,6 +865,24 @@ void RenderingDeviceGraph::_run_render_commands(int32_t p_level, const RecordedC
}
const RecordedDrawListCommand *draw_list_command = reinterpret_cast<const RecordedDrawListCommand *>(command);
if (draw_list_command->split_cmd_buffer) {
// Create or reuse a command buffer and finish recording the current one.
driver->command_buffer_end(r_command_buffer);
while (r_command_buffer_pool.buffers_used >= r_command_buffer_pool.buffers.size()) {
RDD::CommandBufferID command_buffer = driver->command_buffer_create(r_command_buffer_pool.pool);
RDD::SemaphoreID command_semaphore = driver->semaphore_create();
r_command_buffer_pool.buffers.push_back(command_buffer);
r_command_buffer_pool.semaphores.push_back(command_semaphore);
}
// Start recording on the next usable command buffer from the pool.
uint32_t command_buffer_index = r_command_buffer_pool.buffers_used++;
r_command_buffer = r_command_buffer_pool.buffers[command_buffer_index];
driver->command_buffer_begin(r_command_buffer);
}
const VectorView clear_values(draw_list_command->clear_values(), draw_list_command->clear_values_count);
#if defined(DEBUG_ENABLED) || defined(DEV_ENABLED)
driver->command_insert_breadcrumb(r_command_buffer, draw_list_command->breadcrumb);
@ -1194,10 +1212,13 @@ void RenderingDeviceGraph::_print_draw_list(const uint8_t *p_instruction_data, u
print_line("\tBIND PIPELINE ID", itos(bind_pipeline_instruction->pipeline.id));
instruction_data_cursor += sizeof(DrawListBindPipelineInstruction);
} break;
case DrawListInstruction::TYPE_BIND_UNIFORM_SET: {
const DrawListBindUniformSetInstruction *bind_uniform_set_instruction = reinterpret_cast<const DrawListBindUniformSetInstruction *>(instruction);
print_line("\tBIND UNIFORM SET ID", itos(bind_uniform_set_instruction->uniform_set.id), "SET INDEX", bind_uniform_set_instruction->set_index);
instruction_data_cursor += sizeof(DrawListBindUniformSetInstruction);
case DrawListInstruction::TYPE_BIND_UNIFORM_SETS: {
const DrawListBindUniformSetsInstruction *bind_uniform_sets_instruction = reinterpret_cast<const DrawListBindUniformSetsInstruction *>(instruction);
print_line("\tBIND UNIFORM SETS COUNT", bind_uniform_sets_instruction->set_count);
for (uint32_t i = 0; i < bind_uniform_sets_instruction->set_count; i++) {
print_line("\tBIND UNIFORM SET ID", itos(bind_uniform_sets_instruction->uniform_set_ids()[i].id), "START INDEX", bind_uniform_sets_instruction->first_set_index);
}
instruction_data_cursor += sizeof(DrawListBindUniformSetsInstruction) + sizeof(RDD::UniformSetID) * bind_uniform_sets_instruction->set_count;
} break;
case DrawListInstruction::TYPE_BIND_VERTEX_BUFFERS: {
const DrawListBindVertexBuffersInstruction *bind_vertex_buffers_instruction = reinterpret_cast<const DrawListBindVertexBuffersInstruction *>(instruction);
@ -1291,10 +1312,13 @@ void RenderingDeviceGraph::_print_compute_list(const uint8_t *p_instruction_data
print_line("\tBIND PIPELINE ID", itos(bind_pipeline_instruction->pipeline.id));
instruction_data_cursor += sizeof(ComputeListBindPipelineInstruction);
} break;
case ComputeListInstruction::TYPE_BIND_UNIFORM_SET: {
const ComputeListBindUniformSetInstruction *bind_uniform_set_instruction = reinterpret_cast<const ComputeListBindUniformSetInstruction *>(instruction);
print_line("\tBIND UNIFORM SET ID", itos(bind_uniform_set_instruction->uniform_set.id), "SHADER ID", itos(bind_uniform_set_instruction->shader.id));
instruction_data_cursor += sizeof(ComputeListBindUniformSetInstruction);
case ComputeListInstruction::TYPE_BIND_UNIFORM_SETS: {
const ComputeListBindUniformSetsInstruction *bind_uniform_sets_instruction = reinterpret_cast<const ComputeListBindUniformSetsInstruction *>(instruction);
print_line("\tBIND UNIFORM SETS COUNT", bind_uniform_sets_instruction->set_count);
for (uint32_t i = 0; i < bind_uniform_sets_instruction->set_count; i++) {
print_line("\tBIND UNIFORM SET ID", itos(bind_uniform_sets_instruction->uniform_set_ids()[i].id), "START INDEX", bind_uniform_sets_instruction->first_set_index);
}
instruction_data_cursor += sizeof(ComputeListBindUniformSetsInstruction) + sizeof(RDD::UniformSetID) * bind_uniform_sets_instruction->set_count;
} break;
case ComputeListInstruction::TYPE_DISPATCH: {
const ComputeListDispatchInstruction *dispatch_instruction = reinterpret_cast<const ComputeListDispatchInstruction *>(instruction);
@ -1474,11 +1498,23 @@ void RenderingDeviceGraph::add_compute_list_bind_pipeline(RDD::PipelineID p_pipe
}
void RenderingDeviceGraph::add_compute_list_bind_uniform_set(RDD::ShaderID p_shader, RDD::UniformSetID p_uniform_set, uint32_t set_index) {
ComputeListBindUniformSetInstruction *instruction = reinterpret_cast<ComputeListBindUniformSetInstruction *>(_allocate_compute_list_instruction(sizeof(ComputeListBindUniformSetInstruction)));
instruction->type = ComputeListInstruction::TYPE_BIND_UNIFORM_SET;
add_compute_list_bind_uniform_sets(p_shader, VectorView(&p_uniform_set, 1), set_index, 1);
}
void RenderingDeviceGraph::add_compute_list_bind_uniform_sets(RDD::ShaderID p_shader, VectorView<RDD::UniformSetID> p_uniform_sets, uint32_t p_first_set_index, uint32_t p_set_count) {
DEV_ASSERT(p_uniform_sets.size() >= p_set_count);
uint32_t instruction_size = sizeof(ComputeListBindUniformSetsInstruction) + sizeof(RDD::UniformSetID) * p_set_count;
ComputeListBindUniformSetsInstruction *instruction = reinterpret_cast<ComputeListBindUniformSetsInstruction *>(_allocate_compute_list_instruction(instruction_size));
instruction->type = ComputeListInstruction::TYPE_BIND_UNIFORM_SETS;
instruction->shader = p_shader;
instruction->uniform_set = p_uniform_set;
instruction->set_index = set_index;
instruction->first_set_index = p_first_set_index;
instruction->set_count = p_set_count;
RDD::UniformSetID *ids = instruction->uniform_set_ids();
for (uint32_t i = 0; i < p_set_count; i++) {
ids[i] = p_uniform_sets[i];
}
}
void RenderingDeviceGraph::add_compute_list_dispatch(uint32_t p_x_groups, uint32_t p_y_groups, uint32_t p_z_groups) {
@ -1552,7 +1588,7 @@ void RenderingDeviceGraph::add_compute_list_end() {
_add_command_to_graph(compute_instruction_list.command_trackers.ptr(), compute_instruction_list.command_tracker_usages.ptr(), compute_instruction_list.command_trackers.size(), command_index, command);
}
void RenderingDeviceGraph::add_draw_list_begin(RDD::RenderPassID p_render_pass, RDD::FramebufferID p_framebuffer, Rect2i p_region, VectorView<RDD::RenderPassClearValue> p_clear_values, bool p_uses_color, bool p_uses_depth, uint32_t p_breadcrumb) {
void RenderingDeviceGraph::add_draw_list_begin(RDD::RenderPassID p_render_pass, RDD::FramebufferID p_framebuffer, Rect2i p_region, VectorView<RDD::RenderPassClearValue> p_clear_values, bool p_uses_color, bool p_uses_depth, uint32_t p_breadcrumb, bool p_split_cmd_buffer) {
draw_instruction_list.clear();
draw_instruction_list.index++;
draw_instruction_list.render_pass = p_render_pass;
@ -1561,6 +1597,7 @@ void RenderingDeviceGraph::add_draw_list_begin(RDD::RenderPassID p_render_pass,
#if defined(DEBUG_ENABLED) || defined(DEV_ENABLED)
draw_instruction_list.breadcrumb = p_breadcrumb;
#endif
draw_instruction_list.split_cmd_buffer = p_split_cmd_buffer;
draw_instruction_list.clear_values.resize(p_clear_values.size());
for (uint32_t i = 0; i < p_clear_values.size(); i++) {
draw_instruction_list.clear_values[i] = p_clear_values[i];
@ -1596,11 +1633,22 @@ void RenderingDeviceGraph::add_draw_list_bind_pipeline(RDD::PipelineID p_pipelin
}
void RenderingDeviceGraph::add_draw_list_bind_uniform_set(RDD::ShaderID p_shader, RDD::UniformSetID p_uniform_set, uint32_t set_index) {
DrawListBindUniformSetInstruction *instruction = reinterpret_cast<DrawListBindUniformSetInstruction *>(_allocate_draw_list_instruction(sizeof(DrawListBindUniformSetInstruction)));
instruction->type = DrawListInstruction::TYPE_BIND_UNIFORM_SET;
add_draw_list_bind_uniform_sets(p_shader, VectorView(&p_uniform_set, 1), set_index, 1);
}
void RenderingDeviceGraph::add_draw_list_bind_uniform_sets(RDD::ShaderID p_shader, VectorView<RDD::UniformSetID> p_uniform_sets, uint32_t p_first_index, uint32_t p_set_count) {
DEV_ASSERT(p_uniform_sets.size() >= p_set_count);
uint32_t instruction_size = sizeof(DrawListBindUniformSetsInstruction) + sizeof(RDD::UniformSetID) * p_set_count;
DrawListBindUniformSetsInstruction *instruction = reinterpret_cast<DrawListBindUniformSetsInstruction *>(_allocate_draw_list_instruction(instruction_size));
instruction->type = DrawListInstruction::TYPE_BIND_UNIFORM_SETS;
instruction->shader = p_shader;
instruction->uniform_set = p_uniform_set;
instruction->set_index = set_index;
instruction->first_set_index = p_first_index;
instruction->set_count = p_set_count;
for (uint32_t i = 0; i < p_set_count; i++) {
instruction->uniform_set_ids()[i] = p_uniform_sets[i];
}
}
void RenderingDeviceGraph::add_draw_list_bind_vertex_buffers(VectorView<RDD::BufferID> p_vertex_buffers, VectorView<uint64_t> p_vertex_buffer_offsets) {
@ -1794,6 +1842,7 @@ void RenderingDeviceGraph::add_draw_list_end() {
#if defined(DEBUG_ENABLED) || defined(DEV_ENABLED)
command->breadcrumb = draw_instruction_list.breadcrumb;
#endif
command->split_cmd_buffer = draw_instruction_list.split_cmd_buffer;
command->clear_values_count = draw_instruction_list.clear_values.size();
RDD::RenderPassClearValue *clear_values = command->clear_values();

38
servers/rendering/rendering_device_graph.h
View File

@ -49,7 +49,7 @@ public:
enum Type {
TYPE_NONE,
TYPE_BIND_PIPELINE,
TYPE_BIND_UNIFORM_SET,
TYPE_BIND_UNIFORM_SETS,
TYPE_DISPATCH,
TYPE_DISPATCH_INDIRECT,
TYPE_SET_PUSH_CONSTANT,
@ -64,7 +64,7 @@ public:
TYPE_NONE,
TYPE_BIND_INDEX_BUFFER,
TYPE_BIND_PIPELINE,
TYPE_BIND_UNIFORM_SET,
TYPE_BIND_UNIFORM_SETS,
TYPE_BIND_VERTEX_BUFFERS,
TYPE_CLEAR_ATTACHMENTS,
TYPE_DRAW,
@ -236,6 +236,7 @@ private:
uint32_t breadcrumb;
#endif
LocalVector<RDD::RenderPassClearValue> clear_values;
bool split_cmd_buffer = false;
};
struct RecordedCommandSort {
@ -327,6 +328,7 @@ private:
uint32_t breadcrumb = 0;
#endif
uint32_t clear_values_count = 0;
bool split_cmd_buffer = false;
_FORCE_INLINE_ RDD::RenderPassClearValue *clear_values() {
return reinterpret_cast<RDD::RenderPassClearValue *>(&this[1]);
@ -416,10 +418,18 @@ private:
RDD::PipelineID pipeline;
};
struct DrawListBindUniformSetInstruction : DrawListInstruction {
RDD::UniformSetID uniform_set;
struct DrawListBindUniformSetsInstruction : DrawListInstruction {
RDD::ShaderID shader;
uint32_t set_index = 0;
uint32_t first_set_index = 0;
uint32_t set_count = 0;
_FORCE_INLINE_ RDD::UniformSetID *uniform_set_ids() {
return reinterpret_cast<RDD::UniformSetID *>(&this[1]);
}
_FORCE_INLINE_ const RDD::UniformSetID *uniform_set_ids() const {
return reinterpret_cast<const RDD::UniformSetID *>(&this[1]);
}
};
struct DrawListBindVertexBuffersInstruction : DrawListInstruction {
@ -539,10 +549,18 @@ private:
RDD::PipelineID pipeline;
};
struct ComputeListBindUniformSetInstruction : ComputeListInstruction {
RDD::UniformSetID uniform_set;
struct ComputeListBindUniformSetsInstruction : ComputeListInstruction {
RDD::ShaderID shader;
uint32_t set_index = 0;
uint32_t first_set_index = 0;
uint32_t set_count = 0;
_FORCE_INLINE_ RDD::UniformSetID *uniform_set_ids() {
return reinterpret_cast<RDD::UniformSetID *>(&this[1]);
}
_FORCE_INLINE_ const RDD::UniformSetID *uniform_set_ids() const {
return reinterpret_cast<const RDD::UniformSetID *>(&this[1]);
}
};
struct ComputeListDispatchInstruction : ComputeListInstruction {
@ -687,6 +705,7 @@ public:
void add_compute_list_begin(RDD::BreadcrumbMarker p_phase = RDD::BreadcrumbMarker::NONE, uint32_t p_breadcrumb_data = 0);
void add_compute_list_bind_pipeline(RDD::PipelineID p_pipeline);
void add_compute_list_bind_uniform_set(RDD::ShaderID p_shader, RDD::UniformSetID p_uniform_set, uint32_t set_index);
void add_compute_list_bind_uniform_sets(RDD::ShaderID p_shader, VectorView<RDD::UniformSetID> p_uniform_set, uint32_t p_first_set_index, uint32_t p_set_count);
void add_compute_list_dispatch(uint32_t p_x_groups, uint32_t p_y_groups, uint32_t p_z_groups);
void add_compute_list_dispatch_indirect(RDD::BufferID p_buffer, uint32_t p_offset);
void add_compute_list_set_push_constant(RDD::ShaderID p_shader, const void *p_data, uint32_t p_data_size);
@ -694,10 +713,11 @@ public:
void add_compute_list_usage(ResourceTracker *p_tracker, ResourceUsage p_usage);
void add_compute_list_usages(VectorView<ResourceTracker *> p_trackers, VectorView<ResourceUsage> p_usages);
void add_compute_list_end();
void add_draw_list_begin(RDD::RenderPassID p_render_pass, RDD::FramebufferID p_framebuffer, Rect2i p_region, VectorView<RDD::RenderPassClearValue> p_clear_values, bool p_uses_color, bool p_uses_depth, uint32_t p_breadcrumb = 0);
void add_draw_list_begin(RDD::RenderPassID p_render_pass, RDD::FramebufferID p_framebuffer, Rect2i p_region, VectorView<RDD::RenderPassClearValue> p_clear_values, bool p_uses_color, bool p_uses_depth, uint32_t p_breadcrumb = 0, bool p_split_cmd_buffer = false);
void add_draw_list_bind_index_buffer(RDD::BufferID p_buffer, RDD::IndexBufferFormat p_format, uint32_t p_offset);
void add_draw_list_bind_pipeline(RDD::PipelineID p_pipeline, BitField<RDD::PipelineStageBits> p_pipeline_stage_bits);
void add_draw_list_bind_uniform_set(RDD::ShaderID p_shader, RDD::UniformSetID p_uniform_set, uint32_t set_index);
void add_draw_list_bind_uniform_sets(RDD::ShaderID p_shader, VectorView<RDD::UniformSetID> p_uniform_set, uint32_t p_first_index, uint32_t p_set_count);
void add_draw_list_bind_vertex_buffers(VectorView<RDD::BufferID> p_vertex_buffers, VectorView<uint64_t> p_vertex_buffer_offsets);
void add_draw_list_clear_attachments(VectorView<RDD::AttachmentClear> p_attachments_clear, VectorView<Rect2i> p_attachments_clear_rect);
void add_draw_list_draw(uint32_t p_vertex_count, uint32_t p_instance_count);

37
thirdparty/vulkan/vk_mem_alloc.h vendored
View File

@ -1713,6 +1713,21 @@ VMA_CALL_PRE void VMA_CALL_POST vmaCalculateStatistics(
VmaAllocator VMA_NOT_NULL allocator,
VmaTotalStatistics* VMA_NOT_NULL pStats);
// -- GODOT begin --
/** \brief Retrieves lazily allocated bytes
This function is called "calculate" not "get" because it has to traverse all
internal data structures, so it may be quite slow. Use it for debugging purposes.
For faster but more brief statistics suitable to be called every frame or every allocation,
use vmaGetHeapBudgets().
Note that when using allocator from multiple threads, returned information may immediately
become outdated.
*/
VMA_CALL_PRE uint64_t VMA_CALL_POST vmaCalculateLazilyAllocatedBytes(
VmaAllocator VMA_NOT_NULL allocator);
// -- GODOT end --
/** \brief Retrieves information about current memory usage and budget for all memory heaps.
\param allocator
@ -14912,6 +14927,28 @@ VMA_CALL_PRE void VMA_CALL_POST vmaCalculateStatistics(
allocator->CalculateStatistics(pStats);
}
// -- GODOT begin --
VMA_CALL_PRE uint64_t VMA_CALL_POST vmaCalculateLazilyAllocatedBytes(
VmaAllocator allocator)
{
VMA_ASSERT(allocator);
VMA_DEBUG_GLOBAL_MUTEX_LOCK
VmaTotalStatistics stats;
allocator->CalculateStatistics(&stats);
uint64_t total_lazilily_allocated_bytes = 0;
for (uint32_t heapIndex = 0; heapIndex < allocator->GetMemoryHeapCount(); ++heapIndex) {
for (uint32_t typeIndex = 0; typeIndex < allocator->GetMemoryTypeCount(); ++typeIndex) {
if (allocator->MemoryTypeIndexToHeapIndex(typeIndex) == heapIndex) {
VkMemoryPropertyFlags flags = allocator->m_MemProps.memoryTypes[typeIndex].propertyFlags;
if (flags & VK_MEMORY_PROPERTY_LAZILY_ALLOCATED_BIT)
total_lazilily_allocated_bytes += stats.memoryType[typeIndex].statistics.allocationBytes;
}
}
}
return total_lazilily_allocated_bytes;
}
// -- GODOT end --
VMA_CALL_PRE void VMA_CALL_POST vmaGetHeapBudgets(
VmaAllocator allocator,
VmaBudget* pBudgets)