Minki/linux
1
0
Fork 1
mirror of https://github.com/torvalds/linux.git synced 2024-11-27 14:41:39 +00:00
Code Issues Packages Projects Releases Wiki Activity

Support Multi-PF netdev (Socket Direct)

Browse Source 
This series adds support for combining multiple devices (PFs) of the
 same port under one netdev instance. Passing traffic through different
 devices belonging to different NUMA sockets saves cross-numa traffic and
 allows apps running on the same netdev from different numas to still
 feel a sense of proximity to the device and achieve improved
 performance.
 
 We achieve this by grouping PFs together, and creating the netdev only
 once all group members are probed. Symmetrically, we destroy the netdev
 once any of the PFs is removed.
 
 The channels are distributed between all devices, a proper configuration
 would utilize the correct close numa when working on a certain app/cpu.
 
 We pick one device to be a primary (leader), and it fills a special
 role.  The other devices (secondaries) are disconnected from the network
 in the chip level (set to silent mode). All RX/TX traffic is steered
 through the primary to/from the secondaries.
 
 Currently, we limit the support to PFs only, and up to two devices
 (sockets).
 
 V6:
 - Address documentation comments from Jakub.
 
 V5:
  - Address documentation comments from Przemek Kitszel.
 
 V4:
  - Improve documentation for better user observability and understanding
    of the feature, in terms of queues and their expected NUMA/CPU/IRQ
    affinity.
 
 V3:
  - Fix documentation per Jakubs feedback.
  - Fix typos
  - Link new documentation in the networking index.rst
 
 V2:
  - Add documentation in a new patch.
  - Add debugfs in a new patch.
  - Add mlx5_ifc bit for MPIR cap check and use it before query.
 -----BEGIN PGP SIGNATURE-----
 
 iQEzBAABCAAdFiEEGhZs6bAKwk/OTgTpSD+KveBX+j4FAmXpfYgACgkQSD+KveBX
 +j5jIAf/VGIX/UQttq74MzK9pWgJNKtf7l8aSYtZuKXx68pmpr+25DfsxbKEeVfy
 KzjvGFx5peoKisWILyaljQXSn7snmSqOsQf/IwDzmsmF/2ZTDyf6NPC6gND0bIjJ
 Uu6cJ2T6Sa9ktg+ANz/gLDvGBBfPqSYTYIXrJnNQKsnW6nV8mDvy4WVf6etvCxOi
 rMjfcqwNijf3GPTJd/qkaWhwneDG2AFWd5HzdORpNh6iuv8Cbc9aNhWgAPh18o7v
 VWuAiFraTgaz6jj2H/NfziAk4ZrtVsCqhaFjJe3eLO+MCk/bZ/SizsAcR61JLkjL
 pFqh5wqxA6v+5YJm4zVatZqPLIt4gQ==
 =GZBa
 -----END PGP SIGNATURE-----

Merge tag 'mlx5-socket-direct-v3' of git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux

Saeed Mahameed says:

====================
Support Multi-PF netdev (Socket Direct)

This series adds support for combining multiple devices (PFs) of the
same port under one netdev instance. Passing traffic through different
devices belonging to different NUMA sockets saves cross-numa traffic and
allows apps running on the same netdev from different numas to still
feel a sense of proximity to the device and achieve improved
performance.

We achieve this by grouping PFs together, and creating the netdev only
once all group members are probed. Symmetrically, we destroy the netdev
once any of the PFs is removed.

The channels are distributed between all devices, a proper configuration
would utilize the correct close numa when working on a certain app/cpu.

We pick one device to be a primary (leader), and it fills a special
role.  The other devices (secondaries) are disconnected from the network
in the chip level (set to silent mode). All RX/TX traffic is steered
through the primary to/from the secondaries.

Currently, we limit the support to PFs only, and up to two devices
(sockets).

* tag 'mlx5-socket-direct-v3' of git://git.kernel.org/pub/scm/linux/kernel/git/saeed/linux:
  Documentation: networking: Add description for multi-pf netdev
  net/mlx5: Enable SD feature
  net/mlx5e: Block TLS device offload on combined SD netdev
  net/mlx5e: Support per-mdev queue counter
  net/mlx5e: Support cross-vhca RSS
  net/mlx5e: Let channels be SD-aware
  net/mlx5e: Create EN core HW resources for all secondary devices
  net/mlx5e: Create single netdev per SD group
  net/mlx5: SD, Add debugfs
  net/mlx5: SD, Add informative prints in kernel log
  net/mlx5: SD, Implement steering for primary and secondaries
  net/mlx5: SD, Implement devcom communication and primary election
  net/mlx5: SD, Implement basic query and instantiation
  net/mlx5: SD, Introduce SD lib
  net/mlx5: Add MPIR bit in mcam_access_reg
====================

Link: https://lore.kernel.org/r/20240307084229.500776-1-saeed@kernel.org
Signed-off-by: Jakub Kicinski <kuba@kernel.org>
This commit is contained in:
Jakub Kicinski 2024-03-08 20:45:17 -08:00
commit d7e14e5344
34 changed files with 1171 additions and 176 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

1
Documentation/networking/index.rst
View File

@ -74,6 +74,7 @@ Contents:
mpls-sysctl
mptcp-sysctl
multiqueue
multi-pf-netdev
napi
net_cachelines/index
netconsole

174
Documentation/networking/multi-pf-netdev.rst Normal file
View File

@ -0,0 +1,174 @@
.. SPDX-License-Identifier: GPL-2.0
.. include:: <isonum.txt>
===============
Multi-PF Netdev
===============
Contents
========
- `Background`_
- `Overview`_
- `mlx5 implementation`_
- `Channels distribution`_
- `Observability`_
- `Steering`_
- `Mutually exclusive features`_
Background
==========
The Multi-PF NIC technology enables several CPUs within a multi-socket server to connect directly to
the network, each through its own dedicated PCIe interface. Through either a connection harness that
splits the PCIe lanes between two cards or by bifurcating a PCIe slot for a single card. This
results in eliminating the network traffic traversing over the internal bus between the sockets,
significantly reducing overhead and latency, in addition to reducing CPU utilization and increasing
network throughput.
Overview
========
The feature adds support for combining multiple PFs of the same port in a Multi-PF environment under
one netdev instance. It is implemented in the netdev layer. Lower-layer instances like pci func,
sysfs entry, and devlink are kept separate.
Passing traffic through different devices belonging to different NUMA sockets saves cross-NUMA
traffic and allows apps running on the same netdev from different NUMAs to still feel a sense of
proximity to the device and achieve improved performance.
mlx5 implementation
===================
Multi-PF or Socket-direct in mlx5 is achieved by grouping PFs together which belong to the same
NIC and has the socket-direct property enabled, once all PFs are probed, we create a single netdev
to represent all of them, symmetrically, we destroy the netdev whenever any of the PFs is removed.
The netdev network channels are distributed between all devices, a proper configuration would utilize
the correct close NUMA node when working on a certain app/CPU.
We pick one PF to be a primary (leader), and it fills a special role. The other devices
(secondaries) are disconnected from the network at the chip level (set to silent mode). In silent
mode, no south <-> north traffic flowing directly through a secondary PF. It needs the assistance of
the leader PF (east <-> west traffic) to function. All Rx/Tx traffic is steered through the primary
to/from the secondaries.
Currently, we limit the support to PFs only, and up to two PFs (sockets).
Channels distribution
=====================
We distribute the channels between the different PFs to achieve local NUMA node performance
on multiple NUMA nodes.
Each combined channel works against one specific PF, creating all its datapath queues against it. We
distribute channels to PFs in a round-robin policy.
::
Example for 2 PFs and 5 channels:
+--------+--------+
| ch idx | PF idx |
+--------+--------+
| 0 | 0 |
| 1 | 1 |
| 2 | 0 |
| 3 | 1 |
| 4 | 0 |
+--------+--------+
The reason we prefer round-robin is, it is less influenced by changes in the number of channels. The
mapping between a channel index and a PF is fixed, no matter how many channels the user configures.
As the channel stats are persistent across channel's closure, changing the mapping every single time
would turn the accumulative stats less representing of the channel's history.
This is achieved by using the correct core device instance (mdev) in each channel, instead of them
all using the same instance under "priv->mdev".
Observability
=============
The relation between PF, irq, napi, and queue can be observed via netlink spec:
$ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml --dump queue-get --json='{"ifindex": 13}'
[{'id': 0, 'ifindex': 13, 'napi-id': 539, 'type': 'rx'},
{'id': 1, 'ifindex': 13, 'napi-id': 540, 'type': 'rx'},
{'id': 2, 'ifindex': 13, 'napi-id': 541, 'type': 'rx'},
{'id': 3, 'ifindex': 13, 'napi-id': 542, 'type': 'rx'},
{'id': 4, 'ifindex': 13, 'napi-id': 543, 'type': 'rx'},
{'id': 0, 'ifindex': 13, 'napi-id': 539, 'type': 'tx'},
{'id': 1, 'ifindex': 13, 'napi-id': 540, 'type': 'tx'},
{'id': 2, 'ifindex': 13, 'napi-id': 541, 'type': 'tx'},
{'id': 3, 'ifindex': 13, 'napi-id': 542, 'type': 'tx'},
{'id': 4, 'ifindex': 13, 'napi-id': 543, 'type': 'tx'}]
$ ./tools/net/ynl/cli.py --spec Documentation/netlink/specs/netdev.yaml --dump napi-get --json='{"ifindex": 13}'
[{'id': 543, 'ifindex': 13, 'irq': 42},
{'id': 542, 'ifindex': 13, 'irq': 41},
{'id': 541, 'ifindex': 13, 'irq': 40},
{'id': 540, 'ifindex': 13, 'irq': 39},
{'id': 539, 'ifindex': 13, 'irq': 36}]
Here you can clearly observe our channels distribution policy:
$ ls /proc/irq/{36,39,40,41,42}/mlx5* -d -1
/proc/irq/36/mlx5_comp1@pci:0000:08:00.0
/proc/irq/39/mlx5_comp1@pci:0000:09:00.0
/proc/irq/40/mlx5_comp2@pci:0000:08:00.0
/proc/irq/41/mlx5_comp2@pci:0000:09:00.0
/proc/irq/42/mlx5_comp3@pci:0000:08:00.0
Steering
========
Secondary PFs are set to "silent" mode, meaning they are disconnected from the network.
In Rx, the steering tables belong to the primary PF only, and it is its role to distribute incoming
traffic to other PFs, via cross-vhca steering capabilities. Still maintain a single default RSS table,
that is capable of pointing to the receive queues of a different PF.
In Tx, the primary PF creates a new Tx flow table, which is aliased by the secondaries, so they can
go out to the network through it.
In addition, we set default XPS configuration that, based on the CPU, selects an SQ belonging to the
PF on the same node as the CPU.
XPS default config example:
NUMA node(s): 2
NUMA node0 CPU(s): 0-11
NUMA node1 CPU(s): 12-23
PF0 on node0, PF1 on node1.
- /sys/class/net/eth2/queues/tx-0/xps_cpus:000001
- /sys/class/net/eth2/queues/tx-1/xps_cpus:001000
- /sys/class/net/eth2/queues/tx-2/xps_cpus:000002
- /sys/class/net/eth2/queues/tx-3/xps_cpus:002000
- /sys/class/net/eth2/queues/tx-4/xps_cpus:000004
- /sys/class/net/eth2/queues/tx-5/xps_cpus:004000
- /sys/class/net/eth2/queues/tx-6/xps_cpus:000008
- /sys/class/net/eth2/queues/tx-7/xps_cpus:008000
- /sys/class/net/eth2/queues/tx-8/xps_cpus:000010
- /sys/class/net/eth2/queues/tx-9/xps_cpus:010000
- /sys/class/net/eth2/queues/tx-10/xps_cpus:000020
- /sys/class/net/eth2/queues/tx-11/xps_cpus:020000
- /sys/class/net/eth2/queues/tx-12/xps_cpus:000040
- /sys/class/net/eth2/queues/tx-13/xps_cpus:040000
- /sys/class/net/eth2/queues/tx-14/xps_cpus:000080
- /sys/class/net/eth2/queues/tx-15/xps_cpus:080000
- /sys/class/net/eth2/queues/tx-16/xps_cpus:000100
- /sys/class/net/eth2/queues/tx-17/xps_cpus:100000
- /sys/class/net/eth2/queues/tx-18/xps_cpus:000200
- /sys/class/net/eth2/queues/tx-19/xps_cpus:200000
- /sys/class/net/eth2/queues/tx-20/xps_cpus:000400
- /sys/class/net/eth2/queues/tx-21/xps_cpus:400000
- /sys/class/net/eth2/queues/tx-22/xps_cpus:000800
- /sys/class/net/eth2/queues/tx-23/xps_cpus:800000
Mutually exclusive features
===========================
The nature of Multi-PF, where different channels work with different PFs, conflicts with
stateful features where the state is maintained in one of the PFs.
For example, in the TLS device-offload feature, special context objects are created per connection
and maintained in the PF. Transitioning between different RQs/SQs would break the feature. Hence,
we disable this combination for now.

2
drivers/net/ethernet/mellanox/mlx5/core/Makefile
View File

@ -29,7 +29,7 @@ mlx5_core-$(CONFIG_MLX5_CORE_EN) += en/rqt.o en/tir.o en/rss.o en/rx_res.o \
en/reporter_tx.o en/reporter_rx.o en/params.o en/xsk/pool.o \
en/xsk/setup.o en/xsk/rx.o en/xsk/tx.o en/devlink.o en/ptp.o \
en/qos.o en/htb.o en/trap.o en/fs_tt_redirect.o en/selq.o \
lib/crypto.o
lib/crypto.o lib/sd.o
#
# Netdev extra

9
drivers/net/ethernet/mellanox/mlx5/core/en.h
View File

@ -60,6 +60,7 @@
#include "lib/clock.h"
#include "en/rx_res.h"
#include "en/selq.h"
#include "lib/sd.h"
extern const struct net_device_ops mlx5e_netdev_ops;
struct page_pool;
@ -791,6 +792,8 @@ struct mlx5e_channel {
struct hwtstamp_config *tstamp;
DECLARE_BITMAP(state, MLX5E_CHANNEL_NUM_STATES);
int ix;
int vec_ix;
int sd_ix;
int cpu;
/* Sync between icosq recovery and XSK enable/disable. */
struct mutex icosq_recovery_lock;
@ -914,7 +917,7 @@ struct mlx5e_priv {
bool tx_ptp_opened;
bool rx_ptp_opened;
struct hwtstamp_config tstamp;
u16 q_counter;
u16 q_counter[MLX5_SD_MAX_GROUP_SZ];
u16 drop_rq_q_counter;
struct notifier_block events_nb;
struct notifier_block blocking_events_nb;
@ -1029,12 +1032,12 @@ struct mlx5e_xsk_param;
struct mlx5e_rq_param;
int mlx5e_open_rq(struct mlx5e_params *params, struct mlx5e_rq_param *param,
struct mlx5e_xsk_param *xsk, int node,
struct mlx5e_xsk_param *xsk, int node, u16 q_counter,
struct mlx5e_rq *rq);
#define MLX5E_RQ_WQES_TIMEOUT 20000 /* msecs */
int mlx5e_wait_for_min_rx_wqes(struct mlx5e_rq *rq, int wait_time);
void mlx5e_close_rq(struct mlx5e_rq *rq);
int mlx5e_create_rq(struct mlx5e_rq *rq, struct mlx5e_rq_param *param);
int mlx5e_create_rq(struct mlx5e_rq *rq, struct mlx5e_rq_param *param, u16 q_counter);
void mlx5e_destroy_rq(struct mlx5e_rq *rq);
struct mlx5e_sq_param;

10
drivers/net/ethernet/mellanox/mlx5/core/en/channels.c
View File

@ -23,20 +23,26 @@ bool mlx5e_channels_is_xsk(struct mlx5e_channels *chs, unsigned int ix)
return test_bit(MLX5E_CHANNEL_STATE_XSK, c->state);
}
void mlx5e_channels_get_regular_rqn(struct mlx5e_channels *chs, unsigned int ix, u32 *rqn)
void mlx5e_channels_get_regular_rqn(struct mlx5e_channels *chs, unsigned int ix, u32 *rqn,
u32 *vhca_id)
{
struct mlx5e_channel *c = mlx5e_channels_get(chs, ix);
*rqn = c->rq.rqn;
if (vhca_id)
*vhca_id = MLX5_CAP_GEN(c->mdev, vhca_id);
}
void mlx5e_channels_get_xsk_rqn(struct mlx5e_channels *chs, unsigned int ix, u32 *rqn)
void mlx5e_channels_get_xsk_rqn(struct mlx5e_channels *chs, unsigned int ix, u32 *rqn,
u32 *vhca_id)
{
struct mlx5e_channel *c = mlx5e_channels_get(chs, ix);
WARN_ON_ONCE(!test_bit(MLX5E_CHANNEL_STATE_XSK, c->state));
*rqn = c->xskrq.rqn;
if (vhca_id)
*vhca_id = MLX5_CAP_GEN(c->mdev, vhca_id);
}
bool mlx5e_channels_get_ptp_rqn(struct mlx5e_channels *chs, u32 *rqn)

6
drivers/net/ethernet/mellanox/mlx5/core/en/channels.h
View File

@ -10,8 +10,10 @@ struct mlx5e_channels;
unsigned int mlx5e_channels_get_num(struct mlx5e_channels *chs);
bool mlx5e_channels_is_xsk(struct mlx5e_channels *chs, unsigned int ix);
void mlx5e_channels_get_regular_rqn(struct mlx5e_channels *chs, unsigned int ix, u32 *rqn);
void mlx5e_channels_get_xsk_rqn(struct mlx5e_channels *chs, unsigned int ix, u32 *rqn);
void mlx5e_channels_get_regular_rqn(struct mlx5e_channels *chs, unsigned int ix, u32 *rqn,
u32 *vhca_id);
void mlx5e_channels_get_xsk_rqn(struct mlx5e_channels *chs, unsigned int ix, u32 *rqn,
u32 *vhca_id);
bool mlx5e_channels_get_ptp_rqn(struct mlx5e_channels *chs, u32 *rqn);
#endif /* __MLX5_EN_CHANNELS_H__ */

48
drivers/net/ethernet/mellanox/mlx5/core/en/monitor_stats.c
View File

@ -20,10 +20,8 @@
#define NUM_REQ_PPCNT_COUNTER_S1 MLX5_CMD_SET_MONITOR_NUM_PPCNT_COUNTER_SET1
#define NUM_REQ_Q_COUNTERS_S1 MLX5_CMD_SET_MONITOR_NUM_Q_COUNTERS_SET1
int mlx5e_monitor_counter_supported(struct mlx5e_priv *priv)
static int mlx5e_monitor_counter_cap(struct mlx5_core_dev *mdev)
{
struct mlx5_core_dev *mdev = priv->mdev;
if (!MLX5_CAP_GEN(mdev, max_num_of_monitor_counters))
return false;
if (MLX5_CAP_PCAM_REG(mdev, ppcnt) &&
@ -36,24 +34,38 @@ int mlx5e_monitor_counter_supported(struct mlx5e_priv *priv)
return true;
}
static void mlx5e_monitor_counter_arm(struct mlx5e_priv *priv)
int mlx5e_monitor_counter_supported(struct mlx5e_priv *priv)
{
struct mlx5_core_dev *pos;
int i;
mlx5_sd_for_each_dev(i, priv->mdev, pos)
if (!mlx5e_monitor_counter_cap(pos))
return false;
return true;
}
static void mlx5e_monitor_counter_arm(struct mlx5_core_dev *mdev)
{
u32 in[MLX5_ST_SZ_DW(arm_monitor_counter_in)] = {};
MLX5_SET(arm_monitor_counter_in, in, opcode,
MLX5_CMD_OP_ARM_MONITOR_COUNTER);
mlx5_cmd_exec_in(priv->mdev, arm_monitor_counter, in);
mlx5_cmd_exec_in(mdev, arm_monitor_counter, in);
}
static void mlx5e_monitor_counters_work(struct work_struct *work)
{
struct mlx5e_priv *priv = container_of(work, struct mlx5e_priv,
monitor_counters_work);
struct mlx5_core_dev *pos;
int i;
mutex_lock(&priv->state_lock);
mlx5e_stats_update_ndo_stats(priv);
mutex_unlock(&priv->state_lock);
mlx5e_monitor_counter_arm(priv);
mlx5_sd_for_each_dev(i, priv->mdev, pos)
mlx5e_monitor_counter_arm(pos);
}
static int mlx5e_monitor_event_handler(struct notifier_block *nb,
@ -97,15 +109,13 @@ static int fill_monitor_counter_q_counter_set1(int cnt, int q_counter, u32 *in)
}
/* check if mlx5e_monitor_counter_supported before calling this function*/
static void mlx5e_set_monitor_counter(struct mlx5e_priv *priv)
static void mlx5e_set_monitor_counter(struct mlx5_core_dev *mdev, int q_counter)
{
struct mlx5_core_dev *mdev = priv->mdev;
int max_num_of_counters = MLX5_CAP_GEN(mdev, max_num_of_monitor_counters);
int num_q_counters = MLX5_CAP_GEN(mdev, num_q_monitor_counters);
int num_ppcnt_counters = !MLX5_CAP_PCAM_REG(mdev, ppcnt) ? 0 :
MLX5_CAP_GEN(mdev, num_ppcnt_monitor_counters);
u32 in[MLX5_ST_SZ_DW(set_monitor_counter_in)] = {};
int q_counter = priv->q_counter;
int cnt = 0;
if (num_ppcnt_counters >= NUM_REQ_PPCNT_COUNTER_S1 &&
@ -127,13 +137,17 @@ static void mlx5e_set_monitor_counter(struct mlx5e_priv *priv)
/* check if mlx5e_monitor_counter_supported before calling this function*/
void mlx5e_monitor_counter_init(struct mlx5e_priv *priv)
{
struct mlx5_core_dev *pos;
int i;
INIT_WORK(&priv->monitor_counters_work, mlx5e_monitor_counters_work);
MLX5_NB_INIT(&priv->monitor_counters_nb, mlx5e_monitor_event_handler,
MONITOR_COUNTER);
mlx5_eq_notifier_register(priv->mdev, &priv->monitor_counters_nb);
mlx5e_set_monitor_counter(priv);
mlx5e_monitor_counter_arm(priv);
mlx5_sd_for_each_dev(i, priv->mdev, pos) {
mlx5_eq_notifier_register(pos, &priv->monitor_counters_nb);
mlx5e_set_monitor_counter(pos, priv->q_counter[i]);
mlx5e_monitor_counter_arm(pos);
}
queue_work(priv->wq, &priv->update_stats_work);
}
@ -141,11 +155,15 @@ void mlx5e_monitor_counter_init(struct mlx5e_priv *priv)
void mlx5e_monitor_counter_cleanup(struct mlx5e_priv *priv)
{
u32 in[MLX5_ST_SZ_DW(set_monitor_counter_in)] = {};
struct mlx5_core_dev *pos;
int i;
MLX5_SET(set_monitor_counter_in, in, opcode,
MLX5_CMD_OP_SET_MONITOR_COUNTER);
mlx5_cmd_exec_in(priv->mdev, set_monitor_counter, in);
mlx5_eq_notifier_unregister(priv->mdev, &priv->monitor_counters_nb);
mlx5_sd_for_each_dev(i, priv->mdev, pos) {
mlx5_cmd_exec_in(pos, set_monitor_counter, in);
mlx5_eq_notifier_unregister(pos, &priv->monitor_counters_nb);
}
cancel_work_sync(&priv->monitor_counters_work);
}

9
drivers/net/ethernet/mellanox/mlx5/core/en/params.c
View File

@ -688,7 +688,7 @@ void mlx5e_build_create_cq_param(struct mlx5e_create_cq_param *ccp, struct mlx5e
.napi = &c->napi,
.ch_stats = c->stats,
.node = cpu_to_node(c->cpu),
.ix = c->ix,
.ix = c->vec_ix,
};
}
@ -959,7 +959,6 @@ static u8 rq_end_pad_mode(struct mlx5_core_dev *mdev, struct mlx5e_params *param
int mlx5e_build_rq_param(struct mlx5_core_dev *mdev,
struct mlx5e_params *params,
struct mlx5e_xsk_param *xsk,
u16 q_counter,
struct mlx5e_rq_param *param)
{
void *rqc = param->rqc;
@ -1021,7 +1020,6 @@ int mlx5e_build_rq_param(struct mlx5_core_dev *mdev,
MLX5_SET(wq, wq, log_wq_stride,
mlx5e_get_rqwq_log_stride(params->rq_wq_type, ndsegs));
MLX5_SET(wq, wq, pd, mdev->mlx5e_res.hw_objs.pdn);
MLX5_SET(rqc, rqc, counter_set_id, q_counter);
MLX5_SET(rqc, rqc, vsd, params->vlan_strip_disable);
MLX5_SET(rqc, rqc, scatter_fcs, params->scatter_fcs_en);
@ -1032,7 +1030,6 @@ int mlx5e_build_rq_param(struct mlx5_core_dev *mdev,
}
void mlx5e_build_drop_rq_param(struct mlx5_core_dev *mdev,
u16 q_counter,
struct mlx5e_rq_param *param)
{
void *rqc = param->rqc;
@ -1041,7 +1038,6 @@ void mlx5e_build_drop_rq_param(struct mlx5_core_dev *mdev,
MLX5_SET(wq, wq, wq_type, MLX5_WQ_TYPE_CYCLIC);
MLX5_SET(wq, wq, log_wq_stride,
mlx5e_get_rqwq_log_stride(MLX5_WQ_TYPE_CYCLIC, 1));
MLX5_SET(rqc, rqc, counter_set_id, q_counter);
param->wq.buf_numa_node = dev_to_node(mlx5_core_dma_dev(mdev));
}
@ -1306,13 +1302,12 @@ void mlx5e_build_xdpsq_param(struct mlx5_core_dev *mdev,
int mlx5e_build_channel_param(struct mlx5_core_dev *mdev,
struct mlx5e_params *params,
u16 q_counter,
struct mlx5e_channel_param *cparam)
{
u8 icosq_log_wq_sz, async_icosq_log_wq_sz;
int err;
err = mlx5e_build_rq_param(mdev, params, NULL, q_counter, &cparam->rq);
err = mlx5e_build_rq_param(mdev, params, NULL, &cparam->rq);
if (err)
return err;

3
drivers/net/ethernet/mellanox/mlx5/core/en/params.h
View File

@ -130,10 +130,8 @@ void mlx5e_build_create_cq_param(struct mlx5e_create_cq_param *ccp, struct mlx5e
int mlx5e_build_rq_param(struct mlx5_core_dev *mdev,
struct mlx5e_params *params,
struct mlx5e_xsk_param *xsk,
u16 q_counter,
struct mlx5e_rq_param *param);
void mlx5e_build_drop_rq_param(struct mlx5_core_dev *mdev,
u16 q_counter,
struct mlx5e_rq_param *param);
void mlx5e_build_sq_param_common(struct mlx5_core_dev *mdev,
struct mlx5e_sq_param *param);
@ -149,7 +147,6 @@ void mlx5e_build_xdpsq_param(struct mlx5_core_dev *mdev,
struct mlx5e_sq_param *param);
int mlx5e_build_channel_param(struct mlx5_core_dev *mdev,
struct mlx5e_params *params,
u16 q_counter,
struct mlx5e_channel_param *cparam);
u16 mlx5e_calc_sq_stop_room(struct mlx5_core_dev *mdev, struct mlx5e_params *params);

12
drivers/net/ethernet/mellanox/mlx5/core/en/ptp.c
View File

@ -646,7 +646,6 @@ static void mlx5e_ptp_build_sq_param(struct mlx5_core_dev *mdev,
static void mlx5e_ptp_build_rq_param(struct mlx5_core_dev *mdev,
struct net_device *netdev,
u16 q_counter,
struct mlx5e_ptp_params *ptp_params)
{
struct mlx5e_rq_param *rq_params = &ptp_params->rq_param;
@ -655,7 +654,7 @@ static void mlx5e_ptp_build_rq_param(struct mlx5_core_dev *mdev,
params->rq_wq_type = MLX5_WQ_TYPE_CYCLIC;
mlx5e_init_rq_type_params(mdev, params);
params->sw_mtu = netdev->max_mtu;
mlx5e_build_rq_param(mdev, params, NULL, q_counter, rq_params);
mlx5e_build_rq_param(mdev, params, NULL, rq_params);
}
static void mlx5e_ptp_build_params(struct mlx5e_ptp *c,
@ -681,7 +680,7 @@ static void mlx5e_ptp_build_params(struct mlx5e_ptp *c,
/* RQ */
if (test_bit(MLX5E_PTP_STATE_RX, c->state)) {
params->vlan_strip_disable = orig->vlan_strip_disable;
mlx5e_ptp_build_rq_param(c->mdev, c->netdev, c->priv->q_counter, cparams);
mlx5e_ptp_build_rq_param(c->mdev, c->netdev, cparams);
}
}
@ -714,13 +713,16 @@ static int mlx5e_ptp_open_rq(struct mlx5e_ptp *c, struct mlx5e_params *params,
struct mlx5e_rq_param *rq_param)
{
int node = dev_to_node(c->mdev->device);
int err;
int err, sd_ix;
u16 q_counter;
err = mlx5e_init_ptp_rq(c, params, &c->rq);
if (err)
return err;
return mlx5e_open_rq(params, rq_param, NULL, node, &c->rq);
sd_ix = mlx5_sd_ch_ix_get_dev_ix(c->mdev, MLX5E_PTP_CHANNEL_IX);
q_counter = c->priv->q_counter[sd_ix];
return mlx5e_open_rq(params, rq_param, NULL, node, q_counter, &c->rq);
}
static int mlx5e_ptp_open_queues(struct mlx5e_ptp *c,

8
drivers/net/ethernet/mellanox/mlx5/core/en/qos.c
View File

@ -122,8 +122,8 @@ int mlx5e_open_qos_sq(struct mlx5e_priv *priv, struct mlx5e_channels *chs,
memset(&param_sq, 0, sizeof(param_sq));
memset(&param_cq, 0, sizeof(param_cq));
mlx5e_build_sq_param(priv->mdev, params, &param_sq);
mlx5e_build_tx_cq_param(priv->mdev, params, &param_cq);
mlx5e_build_sq_param(c->mdev, params, &param_sq);
mlx5e_build_tx_cq_param(c->mdev, params, &param_cq);
err = mlx5e_open_cq(c->mdev, params->tx_cq_moderation, &param_cq, &ccp, &sq->cq);
if (err)
goto err_free_sq;
@ -176,7 +176,7 @@ int mlx5e_activate_qos_sq(void *data, u16 node_qid, u32 hw_id)
*/
smp_wmb();
qos_dbg(priv->mdev, "Activate QoS SQ qid %u\n", node_qid);
qos_dbg(sq->mdev, "Activate QoS SQ qid %u\n", node_qid);
mlx5e_activate_txqsq(sq);
return 0;
@ -190,7 +190,7 @@ void mlx5e_deactivate_qos_sq(struct mlx5e_priv *priv, u16 qid)
if (!sq) /* Handle the case when the SQ failed to open. */
return;
qos_dbg(priv->mdev, "Deactivate QoS SQ qid %u\n", qid);
qos_dbg(sq->mdev, "Deactivate QoS SQ qid %u\n", qid);
mlx5e_deactivate_txqsq(sq);
priv->txq2sq[mlx5e_qid_from_qos(&priv->channels, qid)] = NULL;

4
drivers/net/ethernet/mellanox/mlx5/core/en/reporter_rx.c
View File

@ -294,8 +294,8 @@ static void mlx5e_rx_reporter_diagnose_generic_rq(struct mlx5e_rq *rq,
params = &priv->channels.params;
rq_sz = mlx5e_rqwq_get_size(rq);
real_time = mlx5_is_real_time_rq(priv->mdev);
rq_stride = BIT(mlx5e_mpwqe_get_log_stride_size(priv->mdev, params, NULL));
real_time = mlx5_is_real_time_rq(rq->mdev);
rq_stride = BIT(mlx5e_mpwqe_get_log_stride_size(rq->mdev, params, NULL));
mlx5e_health_fmsg_named_obj_nest_start(fmsg, "RQ");
devlink_fmsg_u8_pair_put(fmsg, "type", params->rq_wq_type);

3
drivers/net/ethernet/mellanox/mlx5/core/en/reporter_tx.c
View File

@ -219,7 +219,6 @@ mlx5e_tx_reporter_build_diagnose_output_sq_common(struct devlink_fmsg *fmsg,
struct mlx5e_txqsq *sq, int tc)
{
bool stopped = netif_xmit_stopped(sq->txq);
struct mlx5e_priv *priv = sq->priv;
u8 state;
int err;
@ -227,7 +226,7 @@ mlx5e_tx_reporter_build_diagnose_output_sq_common(struct devlink_fmsg *fmsg,
devlink_fmsg_u32_pair_put(fmsg, "txq ix", sq->txq_ix);
devlink_fmsg_u32_pair_put(fmsg, "sqn", sq->sqn);
err = mlx5_core_query_sq_state(priv->mdev, sq->sqn, &state);
err = mlx5_core_query_sq_state(sq->mdev, sq->sqn, &state);
if (!err)
devlink_fmsg_u8_pair_put(fmsg, "HW state", state);

125
drivers/net/ethernet/mellanox/mlx5/core/en/rqt.c
View File

@ -4,6 +4,33 @@
#include "rqt.h"
#include <linux/mlx5/transobj.h>
static bool verify_num_vhca_ids(struct mlx5_core_dev *mdev, u32 *vhca_ids,
unsigned int size)
{
unsigned int max_num_vhca_id = MLX5_CAP_GEN_2(mdev, max_rqt_vhca_id);
int i;
/* Verify that all vhca_ids are in range [0, max_num_vhca_ids - 1] */
for (i = 0; i < size; i++)
if (vhca_ids[i] >= max_num_vhca_id)
return false;
return true;
}
static bool rqt_verify_vhca_ids(struct mlx5_core_dev *mdev, u32 *vhca_ids,
unsigned int size)
{
if (!vhca_ids)
return true;
if (!MLX5_CAP_GEN(mdev, cross_vhca_rqt))
return false;
if (!verify_num_vhca_ids(mdev, vhca_ids, size))
return false;
return true;
}
void mlx5e_rss_params_indir_init_uniform(struct mlx5e_rss_params_indir *indir,
unsigned int num_channels)
{
@ -13,19 +40,38 @@ void mlx5e_rss_params_indir_init_uniform(struct mlx5e_rss_params_indir *indir,
indir->table[i] = i % num_channels;
}
static int mlx5e_rqt_init(struct mlx5e_rqt *rqt, struct mlx5_core_dev *mdev,
u16 max_size, u32 *init_rqns, u16 init_size)
static void fill_rqn_list(void *rqtc, u32 *rqns, u32 *vhca_ids, unsigned int size)
{
unsigned int i;
if (vhca_ids) {
MLX5_SET(rqtc, rqtc, rq_vhca_id_format, 1);
for (i = 0; i < size; i++) {
MLX5_SET(rqtc, rqtc, rq_vhca[i].rq_num, rqns[i]);
MLX5_SET(rqtc, rqtc, rq_vhca[i].rq_vhca_id, vhca_ids[i]);
}
} else {
for (i = 0; i < size; i++)
MLX5_SET(rqtc, rqtc, rq_num[i], rqns[i]);
}
}
static int mlx5e_rqt_init(struct mlx5e_rqt *rqt, struct mlx5_core_dev *mdev,
u16 max_size, u32 *init_rqns, u32 *init_vhca_ids, u16 init_size)
{
int entry_sz;
void *rqtc;
int inlen;
int err;
u32 *in;
int i;
if (!rqt_verify_vhca_ids(mdev, init_vhca_ids, init_size))
return -EOPNOTSUPP;
rqt->mdev = mdev;
rqt->size = max_size;
inlen = MLX5_ST_SZ_BYTES(create_rqt_in) + sizeof(u32) * init_size;
entry_sz = init_vhca_ids ? MLX5_ST_SZ_BYTES(rq_vhca) : MLX5_ST_SZ_BYTES(rq_num);
inlen = MLX5_ST_SZ_BYTES(create_rqt_in) + entry_sz * init_size;
in = kvzalloc(inlen, GFP_KERNEL);
if (!in)
return -ENOMEM;
@ -33,10 +79,9 @@ static int mlx5e_rqt_init(struct mlx5e_rqt *rqt, struct mlx5_core_dev *mdev,
rqtc = MLX5_ADDR_OF(create_rqt_in, in, rqt_context);
MLX5_SET(rqtc, rqtc, rqt_max_size, rqt->size);
MLX5_SET(rqtc, rqtc, rqt_actual_size, init_size);
for (i = 0; i < init_size; i++)
MLX5_SET(rqtc, rqtc, rq_num[i], init_rqns[i]);
fill_rqn_list(rqtc, init_rqns, init_vhca_ids, init_size);
err = mlx5_core_create_rqt(rqt->mdev, in, inlen, &rqt->rqtn);
@ -49,7 +94,7 @@ int mlx5e_rqt_init_direct(struct mlx5e_rqt *rqt, struct mlx5_core_dev *mdev,
{
u16 max_size = indir_enabled ? indir_table_size : 1;
return mlx5e_rqt_init(rqt, mdev, max_size, &init_rqn, 1);
return mlx5e_rqt_init(rqt, mdev, max_size, &init_rqn, NULL, 1);
}
static int mlx5e_bits_invert(unsigned long a, int size)
@ -63,7 +108,8 @@ static int mlx5e_bits_invert(unsigned long a, int size)
return inv;
}
static int mlx5e_calc_indir_rqns(u32 *rss_rqns, u32 *rqns, unsigned int num_rqns,
static int mlx5e_calc_indir_rqns(u32 *rss_rqns, u32 *rqns, u32 *rss_vhca_ids, u32 *vhca_ids,
unsigned int num_rqns,
u8 hfunc, struct mlx5e_rss_params_indir *indir)
{
unsigned int i;
@ -82,30 +128,42 @@ static int mlx5e_calc_indir_rqns(u32 *rss_rqns, u32 *rqns, unsigned int num_rqns
*/
return -EINVAL;
rss_rqns[i] = rqns[ix];
if (vhca_ids)
rss_vhca_ids[i] = vhca_ids[ix];
}
return 0;
}
int mlx5e_rqt_init_indir(struct mlx5e_rqt *rqt, struct mlx5_core_dev *mdev,
u32 *rqns, unsigned int num_rqns,
u32 *rqns, u32 *vhca_ids, unsigned int num_rqns,
u8 hfunc, struct mlx5e_rss_params_indir *indir)
{
u32 *rss_rqns;
u32 *rss_rqns, *rss_vhca_ids = NULL;
int err;
rss_rqns = kvmalloc_array(indir->actual_table_size, sizeof(*rss_rqns), GFP_KERNEL);
if (!rss_rqns)
return -ENOMEM;
err = mlx5e_calc_indir_rqns(rss_rqns, rqns, num_rqns, hfunc, indir);
if (vhca_ids) {
rss_vhca_ids = kvmalloc_array(indir->actual_table_size, sizeof(*rss_vhca_ids),
GFP_KERNEL);
if (!rss_vhca_ids) {
kvfree(rss_rqns);
return -ENOMEM;
}
}
err = mlx5e_calc_indir_rqns(rss_rqns, rqns, rss_vhca_ids, vhca_ids, num_rqns, hfunc, indir);
if (err)
goto out;
err = mlx5e_rqt_init(rqt, mdev, indir->max_table_size, rss_rqns,
err = mlx5e_rqt_init(rqt, mdev, indir->max_table_size, rss_rqns, rss_vhca_ids,
indir->actual_table_size);
out:
kvfree(rss_vhca_ids);
kvfree(rss_rqns);
return err;
}
@ -126,15 +184,20 @@ void mlx5e_rqt_destroy(struct mlx5e_rqt *rqt)
mlx5_core_destroy_rqt(rqt->mdev, rqt->rqtn);
}
static int mlx5e_rqt_redirect(struct mlx5e_rqt *rqt, u32 *rqns, unsigned int size)
static int mlx5e_rqt_redirect(struct mlx5e_rqt *rqt, u32 *rqns, u32 *vhca_ids,
unsigned int size)
{
unsigned int i;
int entry_sz;
void *rqtc;
int inlen;
u32 *in;
int err;
inlen = MLX5_ST_SZ_BYTES(modify_rqt_in) + sizeof(u32) * size;
if (!rqt_verify_vhca_ids(rqt->mdev, vhca_ids, size))
return -EINVAL;
entry_sz = vhca_ids ? MLX5_ST_SZ_BYTES(rq_vhca) : MLX5_ST_SZ_BYTES(rq_num);
inlen = MLX5_ST_SZ_BYTES(modify_rqt_in) + entry_sz * size;
in = kvzalloc(inlen, GFP_KERNEL);
if (!in)
return -ENOMEM;
@ -143,8 +206,8 @@ static int mlx5e_rqt_redirect(struct mlx5e_rqt *rqt, u32 *rqns, unsigned int siz
MLX5_SET(modify_rqt_in, in, bitmask.rqn_list, 1);
MLX5_SET(rqtc, rqtc, rqt_actual_size, size);
for (i = 0; i < size; i++)
MLX5_SET(rqtc, rqtc, rq_num[i], rqns[i]);
fill_rqn_list(rqtc, rqns, vhca_ids, size);
err = mlx5_core_modify_rqt(rqt->mdev, rqt->rqtn, in, inlen);
@ -152,17 +215,21 @@ static int mlx5e_rqt_redirect(struct mlx5e_rqt *rqt, u32 *rqns, unsigned int siz
return err;
}
int mlx5e_rqt_redirect_direct(struct mlx5e_rqt *rqt, u32 rqn)
int mlx5e_rqt_redirect_direct(struct mlx5e_rqt *rqt, u32 rqn, u32 *vhca_id)
{
return mlx5e_rqt_redirect(rqt, &rqn, 1);
return mlx5e_rqt_redirect(rqt, &rqn, vhca_id, 1);
}
int mlx5e_rqt_redirect_indir(struct mlx5e_rqt *rqt, u32 *rqns, unsigned int num_rqns,
int mlx5e_rqt_redirect_indir(struct mlx5e_rqt *rqt, u32 *rqns, u32 *vhca_ids,
unsigned int num_rqns,
u8 hfunc, struct mlx5e_rss_params_indir *indir)
{
u32 *rss_rqns;
u32 *rss_rqns, *rss_vhca_ids = NULL;
int err;
if (!rqt_verify_vhca_ids(rqt->mdev, vhca_ids, num_rqns))
return -EINVAL;
if (WARN_ON(rqt->size != indir->max_table_size))
return -EINVAL;
@ -170,13 +237,23 @@ int mlx5e_rqt_redirect_indir(struct mlx5e_rqt *rqt, u32 *rqns, unsigned int num_
if (!rss_rqns)
return -ENOMEM;
err = mlx5e_calc_indir_rqns(rss_rqns, rqns, num_rqns, hfunc, indir);
if (vhca_ids) {
rss_vhca_ids = kvmalloc_array(indir->actual_table_size, sizeof(*rss_vhca_ids),
GFP_KERNEL);
if (!rss_vhca_ids) {
kvfree(rss_rqns);
return -ENOMEM;
}
}
err = mlx5e_calc_indir_rqns(rss_rqns, rqns, rss_vhca_ids, vhca_ids, num_rqns, hfunc, indir);
if (err)
goto out;
err = mlx5e_rqt_redirect(rqt, rss_rqns, indir->actual_table_size);
err = mlx5e_rqt_redirect(rqt, rss_rqns, rss_vhca_ids, indir->actual_table_size);
out:
kvfree(rss_vhca_ids);
kvfree(rss_rqns);
return err;
}

9
drivers/net/ethernet/mellanox/mlx5/core/en/rqt.h
View File

@ -20,7 +20,7 @@ void mlx5e_rss_params_indir_init_uniform(struct mlx5e_rss_params_indir *indir,
unsigned int num_channels);
struct mlx5e_rqt {
struct mlx5_core_dev *mdev;
struct mlx5_core_dev *mdev; /* primary */
u32 rqtn;
u16 size;
};
@ -28,7 +28,7 @@ struct mlx5e_rqt {
int mlx5e_rqt_init_direct(struct mlx5e_rqt *rqt, struct mlx5_core_dev *mdev,
bool indir_enabled, u32 init_rqn, u32 indir_table_size);
int mlx5e_rqt_init_indir(struct mlx5e_rqt *rqt, struct mlx5_core_dev *mdev,
u32 *rqns, unsigned int num_rqns,
u32 *rqns, u32 *vhca_ids, unsigned int num_rqns,
u8 hfunc, struct mlx5e_rss_params_indir *indir);
void mlx5e_rqt_destroy(struct mlx5e_rqt *rqt);
@ -38,8 +38,9 @@ static inline u32 mlx5e_rqt_get_rqtn(struct mlx5e_rqt *rqt)
}
u32 mlx5e_rqt_size(struct mlx5_core_dev *mdev, unsigned int num_channels);
int mlx5e_rqt_redirect_direct(struct mlx5e_rqt *rqt, u32 rqn);
int mlx5e_rqt_redirect_indir(struct mlx5e_rqt *rqt, u32 *rqns, unsigned int num_rqns,
int mlx5e_rqt_redirect_direct(struct mlx5e_rqt *rqt, u32 rqn, u32 *vhca_id);
int mlx5e_rqt_redirect_indir(struct mlx5e_rqt *rqt, u32 *rqns, u32 *vhca_ids,
unsigned int num_rqns,
u8 hfunc, struct mlx5e_rss_params_indir *indir);
#endif /* __MLX5_EN_RQT_H__ */

17
drivers/net/ethernet/mellanox/mlx5/core/en/rss.c
View File

@ -74,7 +74,7 @@ struct mlx5e_rss {
struct mlx5e_tir *tir[MLX5E_NUM_INDIR_TIRS];
struct mlx5e_tir *inner_tir[MLX5E_NUM_INDIR_TIRS];
struct mlx5e_rqt rqt;
struct mlx5_core_dev *mdev;
struct mlx5_core_dev *mdev; /* primary */
u32 drop_rqn;
bool inner_ft_support;
bool enabled;
@ -473,21 +473,22 @@ int mlx5e_rss_obtain_tirn(struct mlx5e_rss *rss,
return 0;
}
static int mlx5e_rss_apply(struct mlx5e_rss *rss, u32 *rqns, unsigned int num_rqns)
static int mlx5e_rss_apply(struct mlx5e_rss *rss, u32 *rqns, u32 *vhca_ids, unsigned int num_rqns)
{
int err;
err = mlx5e_rqt_redirect_indir(&rss->rqt, rqns, num_rqns, rss->hash.hfunc, &rss->indir);
err = mlx5e_rqt_redirect_indir(&rss->rqt, rqns, vhca_ids, num_rqns, rss->hash.hfunc,
&rss->indir);
if (err)
mlx5e_rss_warn(rss->mdev, "Failed to redirect RQT %#x to channels: err = %d\n",
mlx5e_rqt_get_rqtn(&rss->rqt), err);
return err;
}
void mlx5e_rss_enable(struct mlx5e_rss *rss, u32 *rqns, unsigned int num_rqns)
void mlx5e_rss_enable(struct mlx5e_rss *rss, u32 *rqns, u32 *vhca_ids, unsigned int num_rqns)
{
rss->enabled = true;
mlx5e_rss_apply(rss, rqns, num_rqns);
mlx5e_rss_apply(rss, rqns, vhca_ids, num_rqns);
}
void mlx5e_rss_disable(struct mlx5e_rss *rss)
@ -495,7 +496,7 @@ void mlx5e_rss_disable(struct mlx5e_rss *rss)
int err;
rss->enabled = false;
err = mlx5e_rqt_redirect_direct(&rss->rqt, rss->drop_rqn);
err = mlx5e_rqt_redirect_direct(&rss->rqt, rss->drop_rqn, NULL);
if (err)
mlx5e_rss_warn(rss->mdev, "Failed to redirect RQT %#x to drop RQ %#x: err = %d\n",
mlx5e_rqt_get_rqtn(&rss->rqt), rss->drop_rqn, err);
@ -568,7 +569,7 @@ int mlx5e_rss_get_rxfh(struct mlx5e_rss *rss, u32 *indir, u8 *key, u8 *hfunc)
int mlx5e_rss_set_rxfh(struct mlx5e_rss *rss, const u32 *indir,
const u8 *key, const u8 *hfunc,
u32 *rqns, unsigned int num_rqns)
u32 *rqns, u32 *vhca_ids, unsigned int num_rqns)
{
bool changed_indir = false;
bool changed_hash = false;
@ -608,7 +609,7 @@ int mlx5e_rss_set_rxfh(struct mlx5e_rss *rss, const u32 *indir,
}
if (changed_indir && rss->enabled) {
err = mlx5e_rss_apply(rss, rqns, num_rqns);
err = mlx5e_rss_apply(rss, rqns, vhca_ids, num_rqns);
if (err) {
mlx5e_rss_copy(rss, old_rss);
goto out;

4
drivers/net/ethernet/mellanox/mlx5/core/en/rss.h
View File

@ -39,7 +39,7 @@ int mlx5e_rss_obtain_tirn(struct mlx5e_rss *rss,
const struct mlx5e_packet_merge_param *init_pkt_merge_param,
bool inner, u32 *tirn);
void mlx5e_rss_enable(struct mlx5e_rss *rss, u32 *rqns, unsigned int num_rqns);
void mlx5e_rss_enable(struct mlx5e_rss *rss, u32 *rqns, u32 *vhca_ids, unsigned int num_rqns);
void mlx5e_rss_disable(struct mlx5e_rss *rss);
int mlx5e_rss_packet_merge_set_param(struct mlx5e_rss *rss,
@ -47,7 +47,7 @@ int mlx5e_rss_packet_merge_set_param(struct mlx5e_rss *rss,
int mlx5e_rss_get_rxfh(struct mlx5e_rss *rss, u32 *indir, u8 *key, u8 *hfunc);
int mlx5e_rss_set_rxfh(struct mlx5e_rss *rss, const u32 *indir,
const u8 *key, const u8 *hfunc,
u32 *rqns, unsigned int num_rqns);
u32 *rqns, u32 *vhca_ids, unsigned int num_rqns);
struct mlx5e_rss_params_hash mlx5e_rss_get_hash(struct mlx5e_rss *rss);
u8 mlx5e_rss_get_hash_fields(struct mlx5e_rss *rss, enum mlx5_traffic_types tt);
int mlx5e_rss_set_hash_fields(struct mlx5e_rss *rss, enum mlx5_traffic_types tt,

62
drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.c
View File

@ -8,7 +8,7 @@
#define MLX5E_MAX_NUM_RSS 16
struct mlx5e_rx_res {
struct mlx5_core_dev *mdev;
struct mlx5_core_dev *mdev; /* primary */
enum mlx5e_rx_res_features features;
unsigned int max_nch;
u32 drop_rqn;
@ -19,6 +19,7 @@ struct mlx5e_rx_res {
struct mlx5e_rss *rss[MLX5E_MAX_NUM_RSS];
bool rss_active;
u32 *rss_rqns;
u32 *rss_vhca_ids;
unsigned int rss_nch;
struct {
@ -34,6 +35,13 @@ struct mlx5e_rx_res {
/* API for rx_res_rss_* */
static u32 *get_vhca_ids(struct mlx5e_rx_res *res, int offset)
{
bool multi_vhca = res->features & MLX5E_RX_RES_FEATURE_MULTI_VHCA;
return multi_vhca ? res->rss_vhca_ids + offset : NULL;
}
void mlx5e_rx_res_rss_update_num_channels(struct mlx5e_rx_res *res, u32 nch)
{
int i;
@ -85,8 +93,11 @@ int mlx5e_rx_res_rss_init(struct mlx5e_rx_res *res, u32 *rss_idx, unsigned int i
return PTR_ERR(rss);
mlx5e_rss_set_indir_uniform(rss, init_nch);
if (res->rss_active)
mlx5e_rss_enable(rss, res->rss_rqns, res->rss_nch);
if (res->rss_active) {
u32 *vhca_ids = get_vhca_ids(res, 0);
mlx5e_rss_enable(rss, res->rss_rqns, vhca_ids, res->rss_nch);
}
res->rss[i] = rss;
*rss_idx = i;
@ -153,10 +164,12 @@ static void mlx5e_rx_res_rss_enable(struct mlx5e_rx_res *res)
for (i = 0; i < MLX5E_MAX_NUM_RSS; i++) {
struct mlx5e_rss *rss = res->rss[i];
u32 *vhca_ids;
if (!rss)
continue;
mlx5e_rss_enable(rss, res->rss_rqns, res->rss_nch);
vhca_ids = get_vhca_ids(res, 0);
mlx5e_rss_enable(rss, res->rss_rqns, vhca_ids, res->rss_nch);
}
}
@ -200,6 +213,7 @@ int mlx5e_rx_res_rss_get_rxfh(struct mlx5e_rx_res *res, u32 rss_idx,
int mlx5e_rx_res_rss_set_rxfh(struct mlx5e_rx_res *res, u32 rss_idx,
const u32 *indir, const u8 *key, const u8 *hfunc)
{
u32 *vhca_ids = get_vhca_ids(res, 0);
struct mlx5e_rss *rss;
if (rss_idx >= MLX5E_MAX_NUM_RSS)
@ -209,7 +223,8 @@ int mlx5e_rx_res_rss_set_rxfh(struct mlx5e_rx_res *res, u32 rss_idx,
if (!rss)
return -ENOENT;
return mlx5e_rss_set_rxfh(rss, indir, key, hfunc, res->rss_rqns, res->rss_nch);
return mlx5e_rss_set_rxfh(rss, indir, key, hfunc, res->rss_rqns, vhca_ids,
res->rss_nch);
}
int mlx5e_rx_res_rss_get_hash_fields(struct mlx5e_rx_res *res, u32 rss_idx,
@ -280,11 +295,13 @@ struct mlx5e_rss *mlx5e_rx_res_rss_get(struct mlx5e_rx_res *res, u32 rss_idx)
static void mlx5e_rx_res_free(struct mlx5e_rx_res *res)
{
kvfree(res->rss_vhca_ids);
kvfree(res->rss_rqns);
kvfree(res);
}
static struct mlx5e_rx_res *mlx5e_rx_res_alloc(struct mlx5_core_dev *mdev, unsigned int max_nch)
static struct mlx5e_rx_res *mlx5e_rx_res_alloc(struct mlx5_core_dev *mdev, unsigned int max_nch,
bool multi_vhca)
{
struct mlx5e_rx_res *rx_res;
@ -298,6 +315,15 @@ static struct mlx5e_rx_res *mlx5e_rx_res_alloc(struct mlx5_core_dev *mdev, unsig
return NULL;
}
if (multi_vhca) {
rx_res->rss_vhca_ids = kvcalloc(max_nch, sizeof(*rx_res->rss_vhca_ids), GFP_KERNEL);
if (!rx_res->rss_vhca_ids) {
kvfree(rx_res->rss_rqns);
kvfree(rx_res);
return NULL;
}
}
return rx_res;
}
@ -424,10 +450,11 @@ mlx5e_rx_res_create(struct mlx5_core_dev *mdev, enum mlx5e_rx_res_features featu
const struct mlx5e_packet_merge_param *init_pkt_merge_param,
unsigned int init_nch)
{
bool multi_vhca = features & MLX5E_RX_RES_FEATURE_MULTI_VHCA;
struct mlx5e_rx_res *res;
int err;
res = mlx5e_rx_res_alloc(mdev, max_nch);
res = mlx5e_rx_res_alloc(mdev, max_nch, multi_vhca);
if (!res)
return ERR_PTR(-ENOMEM);
@ -504,10 +531,11 @@ static void mlx5e_rx_res_channel_activate_direct(struct mlx5e_rx_res *res,
struct mlx5e_channels *chs,
unsigned int ix)
{
u32 *vhca_id = get_vhca_ids(res, ix);
u32 rqn = res->rss_rqns[ix];
int err;
err = mlx5e_rqt_redirect_direct(&res->channels[ix].direct_rqt, rqn);
err = mlx5e_rqt_redirect_direct(&res->channels[ix].direct_rqt, rqn, vhca_id);
if (err)
mlx5_core_warn(res->mdev, "Failed to redirect direct RQT %#x to RQ %#x (channel %u): err = %d\n",
mlx5e_rqt_get_rqtn(&res->channels[ix].direct_rqt),
@ -519,7 +547,7 @@ static void mlx5e_rx_res_channel_deactivate_direct(struct mlx5e_rx_res *res,
{
int err;
err = mlx5e_rqt_redirect_direct(&res->channels[ix].direct_rqt, res->drop_rqn);
err = mlx5e_rqt_redirect_direct(&res->channels[ix].direct_rqt, res->drop_rqn, NULL);
if (err)
mlx5_core_warn(res->mdev, "Failed to redirect direct RQT %#x to drop RQ %#x (channel %u): err = %d\n",
mlx5e_rqt_get_rqtn(&res->channels[ix].direct_rqt),
@ -534,10 +562,12 @@ void mlx5e_rx_res_channels_activate(struct mlx5e_rx_res *res, struct mlx5e_chann
nch = mlx5e_channels_get_num(chs);
for (ix = 0; ix < chs->num; ix++) {
u32 *vhca_id = get_vhca_ids(res, ix);
if (mlx5e_channels_is_xsk(chs, ix))
mlx5e_channels_get_xsk_rqn(chs, ix, &res->rss_rqns[ix]);
mlx5e_channels_get_xsk_rqn(chs, ix, &res->rss_rqns[ix], vhca_id);
else
mlx5e_channels_get_regular_rqn(chs, ix, &res->rss_rqns[ix]);
mlx5e_channels_get_regular_rqn(chs, ix, &res->rss_rqns[ix], vhca_id);
}
res->rss_nch = chs->num;
@ -554,7 +584,7 @@ void mlx5e_rx_res_channels_activate(struct mlx5e_rx_res *res, struct mlx5e_chann
if (!mlx5e_channels_get_ptp_rqn(chs, &rqn))
rqn = res->drop_rqn;
err = mlx5e_rqt_redirect_direct(&res->ptp.rqt, rqn);
err = mlx5e_rqt_redirect_direct(&res->ptp.rqt, rqn, NULL);
if (err)
mlx5_core_warn(res->mdev, "Failed to redirect direct RQT %#x to RQ %#x (PTP): err = %d\n",
mlx5e_rqt_get_rqtn(&res->ptp.rqt),
@ -573,7 +603,7 @@ void mlx5e_rx_res_channels_deactivate(struct mlx5e_rx_res *res)
mlx5e_rx_res_channel_deactivate_direct(res, ix);
if (res->features & MLX5E_RX_RES_FEATURE_PTP) {
err = mlx5e_rqt_redirect_direct(&res->ptp.rqt, res->drop_rqn);
err = mlx5e_rqt_redirect_direct(&res->ptp.rqt, res->drop_rqn, NULL);
if (err)
mlx5_core_warn(res->mdev, "Failed to redirect direct RQT %#x to drop RQ %#x (PTP): err = %d\n",
mlx5e_rqt_get_rqtn(&res->ptp.rqt),
@ -584,10 +614,12 @@ void mlx5e_rx_res_channels_deactivate(struct mlx5e_rx_res *res)
void mlx5e_rx_res_xsk_update(struct mlx5e_rx_res *res, struct mlx5e_channels *chs,
unsigned int ix, bool xsk)
{
u32 *vhca_id = get_vhca_ids(res, ix);
if (xsk)
mlx5e_channels_get_xsk_rqn(chs, ix, &res->rss_rqns[ix]);
mlx5e_channels_get_xsk_rqn(chs, ix, &res->rss_rqns[ix], vhca_id);
else
mlx5e_channels_get_regular_rqn(chs, ix, &res->rss_rqns[ix]);
mlx5e_channels_get_regular_rqn(chs, ix, &res->rss_rqns[ix], vhca_id);
mlx5e_rx_res_rss_enable(res);

1
drivers/net/ethernet/mellanox/mlx5/core/en/rx_res.h
View File

@ -18,6 +18,7 @@ struct mlx5e_rss_params_hash;
enum mlx5e_rx_res_features {
MLX5E_RX_RES_FEATURE_INNER_FT = BIT(0),
MLX5E_RX_RES_FEATURE_PTP = BIT(1),
MLX5E_RX_RES_FEATURE_MULTI_VHCA = BIT(2),
};
/* Setup */

11
drivers/net/ethernet/mellanox/mlx5/core/en/trap.c
View File

@ -63,10 +63,12 @@ static int mlx5e_open_trap_rq(struct mlx5e_priv *priv, struct mlx5e_trap *t)
struct mlx5e_create_cq_param ccp = {};
struct dim_cq_moder trap_moder = {};
struct mlx5e_rq *rq = &t->rq;
u16 q_counter;
int node;
int err;
node = dev_to_node(mdev->device);
q_counter = priv->q_counter[0];
ccp.netdev = priv->netdev;
ccp.wq = priv->wq;
@ -79,7 +81,7 @@ static int mlx5e_open_trap_rq(struct mlx5e_priv *priv, struct mlx5e_trap *t)
return err;
mlx5e_init_trap_rq(t, &t->params, rq);
err = mlx5e_open_rq(&t->params, rq_param, NULL, node, rq);
err = mlx5e_open_rq(&t->params, rq_param, NULL, node, q_counter, rq);
if (err)
goto err_destroy_cq;
@ -116,15 +118,14 @@ static int mlx5e_create_trap_direct_rq_tir(struct mlx5_core_dev *mdev, struct ml
}
static void mlx5e_build_trap_params(struct mlx5_core_dev *mdev,
int max_mtu, u16 q_counter,
struct mlx5e_trap *t)
int max_mtu, struct mlx5e_trap *t)
{
struct mlx5e_params *params = &t->params;
params->rq_wq_type = MLX5_WQ_TYPE_CYCLIC;
mlx5e_init_rq_type_params(mdev, params);
params->sw_mtu = max_mtu;
mlx5e_build_rq_param(mdev, params, NULL, q_counter, &t->rq_param);
mlx5e_build_rq_param(mdev, params, NULL, &t->rq_param);
}
static struct mlx5e_trap *mlx5e_open_trap(struct mlx5e_priv *priv)
@ -138,7 +139,7 @@ static struct mlx5e_trap *mlx5e_open_trap(struct mlx5e_priv *priv)
if (!t)
return ERR_PTR(-ENOMEM);
mlx5e_build_trap_params(priv->mdev, netdev->max_mtu, priv->q_counter, t);
mlx5e_build_trap_params(priv->mdev, netdev->max_mtu, t);
t->priv = priv;
t->mdev = priv->mdev;

6
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/pool.c
View File

@ -6,10 +6,10 @@
#include "setup.h"
#include "en/params.h"
static int mlx5e_xsk_map_pool(struct mlx5e_priv *priv,
static int mlx5e_xsk_map_pool(struct mlx5_core_dev *mdev,
struct xsk_buff_pool *pool)
{
struct device *dev = mlx5_core_dma_dev(priv->mdev);
struct device *dev = mlx5_core_dma_dev(mdev);
return xsk_pool_dma_map(pool, dev, DMA_ATTR_SKIP_CPU_SYNC);
}
@ -89,7 +89,7 @@ static int mlx5e_xsk_enable_locked(struct mlx5e_priv *priv,
if (unlikely(!mlx5e_xsk_is_pool_sane(pool)))
return -EINVAL;
err = mlx5e_xsk_map_pool(priv, pool);
err = mlx5e_xsk_map_pool(mlx5_sd_ch_ix_get_dev(priv->mdev, ix), pool);
if (unlikely(err))
return err;

8
drivers/net/ethernet/mellanox/mlx5/core/en/xsk/setup.c
View File

@ -49,10 +49,9 @@ bool mlx5e_validate_xsk_param(struct mlx5e_params *params,
static void mlx5e_build_xsk_cparam(struct mlx5_core_dev *mdev,
struct mlx5e_params *params,
struct mlx5e_xsk_param *xsk,
u16 q_counter,
struct mlx5e_channel_param *cparam)
{
mlx5e_build_rq_param(mdev, params, xsk, q_counter, &cparam->rq);
mlx5e_build_rq_param(mdev, params, xsk, &cparam->rq);
mlx5e_build_xdpsq_param(mdev, params, xsk, &cparam->xdp_sq);
}
@ -93,6 +92,7 @@ static int mlx5e_open_xsk_rq(struct mlx5e_channel *c, struct mlx5e_params *param
struct mlx5e_rq_param *rq_params, struct xsk_buff_pool *pool,
struct mlx5e_xsk_param *xsk)
{
u16 q_counter = c->priv->q_counter[c->sd_ix];
struct mlx5e_rq *xskrq = &c->xskrq;
int err;
@ -100,7 +100,7 @@ static int mlx5e_open_xsk_rq(struct mlx5e_channel *c, struct mlx5e_params *param
if (err)
return err;
err = mlx5e_open_rq(params, rq_params, xsk, cpu_to_node(c->cpu), xskrq);
err = mlx5e_open_rq(params, rq_params, xsk, cpu_to_node(c->cpu), q_counter, xskrq);
if (err)
return err;
@ -125,7 +125,7 @@ int mlx5e_open_xsk(struct mlx5e_priv *priv, struct mlx5e_params *params,
if (!cparam)
return -ENOMEM;
mlx5e_build_xsk_cparam(priv->mdev, params, xsk, priv->q_counter, cparam);
mlx5e_build_xsk_cparam(priv->mdev, params, xsk, cparam);
err = mlx5e_open_cq(c->mdev, params->rx_cq_moderation, &cparam->rq.cqp, &ccp,
&c->xskrq.cq);

2
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.c
View File

@ -96,7 +96,7 @@ bool mlx5e_is_ktls_rx(struct mlx5_core_dev *mdev)
{
u8 max_sq_wqebbs = mlx5e_get_max_sq_wqebbs(mdev);
if (is_kdump_kernel() || !MLX5_CAP_GEN(mdev, tls_rx))
if (is_kdump_kernel() || !MLX5_CAP_GEN(mdev, tls_rx) || mlx5_get_sd(mdev))
return false;
/* Check the possibility to post the required ICOSQ WQEs. */

4
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls.h
View File

@ -11,6 +11,7 @@
#ifdef CONFIG_MLX5_EN_TLS
#include "lib/crypto.h"
#include "lib/mlx5.h"
struct mlx5_crypto_dek *mlx5_ktls_create_key(struct mlx5_crypto_dek_pool *dek_pool,
struct tls_crypto_info *crypto_info);
@ -61,7 +62,8 @@ void mlx5e_ktls_rx_resync_destroy_resp_list(struct mlx5e_ktls_resync_resp *resp_
static inline bool mlx5e_is_ktls_tx(struct mlx5_core_dev *mdev)
{
return !is_kdump_kernel() && MLX5_CAP_GEN(mdev, tls_tx);
return !is_kdump_kernel() && MLX5_CAP_GEN(mdev, tls_tx) &&
!mlx5_get_sd(mdev);
}
bool mlx5e_is_ktls_rx(struct mlx5_core_dev *mdev);

6
drivers/net/ethernet/mellanox/mlx5/core/en_accel/ktls_rx.c
View File

@ -267,7 +267,7 @@ resync_post_get_progress_params(struct mlx5e_icosq *sq,
goto err_out;
}
pdev = mlx5_core_dma_dev(sq->channel->priv->mdev);
pdev = mlx5_core_dma_dev(sq->channel->mdev);
buf->dma_addr = dma_map_single(pdev, &buf->progress,
PROGRESS_PARAMS_PADDED_SIZE, DMA_FROM_DEVICE);
if (unlikely(dma_mapping_error(pdev, buf->dma_addr))) {
@ -425,14 +425,12 @@ void mlx5e_ktls_handle_get_psv_completion(struct mlx5e_icosq_wqe_info *wi,
{
struct mlx5e_ktls_rx_resync_buf *buf = wi->tls_get_params.buf;
struct mlx5e_ktls_offload_context_rx *priv_rx;
struct mlx5e_ktls_rx_resync_ctx *resync;
u8 tracker_state, auth_state, *ctx;
struct device *dev;
u32 hw_seq;
priv_rx = buf->priv_rx;
resync = &priv_rx->resync;
dev = mlx5_core_dma_dev(resync->priv->mdev);
dev = mlx5_core_dma_dev(sq->channel->mdev);
if (unlikely(test_bit(MLX5E_PRIV_RX_FLAG_DELETING, priv_rx->flags)))
goto out;

180
drivers/net/ethernet/mellanox/mlx5/core/en_main.c
View File

@ -70,6 +70,7 @@
#include "qos.h"
#include "en/trap.h"
#include "lib/devcom.h"
#include "lib/sd.h"
bool mlx5e_check_fragmented_striding_rq_cap(struct mlx5_core_dev *mdev, u8 page_shift,
enum mlx5e_mpwrq_umr_mode umr_mode)
@ -1024,7 +1025,7 @@ static void mlx5e_free_rq(struct mlx5e_rq *rq)
mlx5_wq_destroy(&rq->wq_ctrl);
}
int mlx5e_create_rq(struct mlx5e_rq *rq, struct mlx5e_rq_param *param)
int mlx5e_create_rq(struct mlx5e_rq *rq, struct mlx5e_rq_param *param, u16 q_counter)
{
struct mlx5_core_dev *mdev = rq->mdev;
u8 ts_format;
@ -1051,6 +1052,7 @@ int mlx5e_create_rq(struct mlx5e_rq *rq, struct mlx5e_rq_param *param)
MLX5_SET(rqc, rqc, cqn, rq->cq.mcq.cqn);
MLX5_SET(rqc, rqc, state, MLX5_RQC_STATE_RST);
MLX5_SET(rqc, rqc, ts_format, ts_format);
MLX5_SET(rqc, rqc, counter_set_id, q_counter);
MLX5_SET(wq, wq, log_wq_pg_sz, rq->wq_ctrl.buf.page_shift -
MLX5_ADAPTER_PAGE_SHIFT);
MLX5_SET64(wq, wq, dbr_addr, rq->wq_ctrl.db.dma);
@ -1274,7 +1276,7 @@ void mlx5e_free_rx_descs(struct mlx5e_rq *rq)
}
int mlx5e_open_rq(struct mlx5e_params *params, struct mlx5e_rq_param *param,
struct mlx5e_xsk_param *xsk, int node,
struct mlx5e_xsk_param *xsk, int node, u16 q_counter,
struct mlx5e_rq *rq)
{
struct mlx5_core_dev *mdev = rq->mdev;
@ -1287,7 +1289,7 @@ int mlx5e_open_rq(struct mlx5e_params *params, struct mlx5e_rq_param *param,
if (err)
return err;
err = mlx5e_create_rq(rq, param);
err = mlx5e_create_rq(rq, param, q_counter);
if (err)
goto err_free_rq;
@ -2335,13 +2337,14 @@ static int mlx5e_set_tx_maxrate(struct net_device *dev, int index, u32 rate)
static int mlx5e_open_rxq_rq(struct mlx5e_channel *c, struct mlx5e_params *params,
struct mlx5e_rq_param *rq_params)
{
u16 q_counter = c->priv->q_counter[c->sd_ix];
int err;
err = mlx5e_init_rxq_rq(c, params, rq_params->xdp_frag_size, &c->rq);
if (err)
return err;
return mlx5e_open_rq(params, rq_params, NULL, cpu_to_node(c->cpu), &c->rq);
return mlx5e_open_rq(params, rq_params, NULL, cpu_to_node(c->cpu), q_counter, &c->rq);
}
static int mlx5e_open_queues(struct mlx5e_channel *c,
@ -2528,14 +2531,20 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
struct xsk_buff_pool *xsk_pool,
struct mlx5e_channel **cp)
{
int cpu = mlx5_comp_vector_get_cpu(priv->mdev, ix);
struct net_device *netdev = priv->netdev;
struct mlx5_core_dev *mdev;
struct mlx5e_xsk_param xsk;
struct mlx5e_channel *c;
unsigned int irq;
int vec_ix;
int cpu;
int err;
err = mlx5_comp_irqn_get(priv->mdev, ix, &irq);
mdev = mlx5_sd_ch_ix_get_dev(priv->mdev, ix);
vec_ix = mlx5_sd_ch_ix_get_vec_ix(mdev, ix);
cpu = mlx5_comp_vector_get_cpu(mdev, vec_ix);
err = mlx5_comp_irqn_get(mdev, vec_ix, &irq);
if (err)
return err;
@ -2548,18 +2557,20 @@ static int mlx5e_open_channel(struct mlx5e_priv *priv, int ix,
return -ENOMEM;
c->priv = priv;
c->mdev = priv->mdev;
c->mdev = mdev;
c->tstamp = &priv->tstamp;
c->ix = ix;
c->vec_ix = vec_ix;
c->sd_ix = mlx5_sd_ch_ix_get_dev_ix(mdev, ix);
c->cpu = cpu;
c->pdev = mlx5_core_dma_dev(priv->mdev);
c->pdev = mlx5_core_dma_dev(mdev);
c->netdev = priv->netdev;
c->mkey_be = cpu_to_be32(priv->mdev->mlx5e_res.hw_objs.mkey);
c->mkey_be = cpu_to_be32(mdev->mlx5e_res.hw_objs.mkey);
c->num_tc = mlx5e_get_dcb_num_tc(params);
c->xdp = !!params->xdp_prog;
c->stats = &priv->channel_stats[ix]->ch;
c->aff_mask = irq_get_effective_affinity_mask(irq);
c->lag_port = mlx5e_enumerate_lag_port(priv->mdev, ix);
c->lag_port = mlx5e_enumerate_lag_port(mdev, ix);
netif_napi_add(netdev, &c->napi, mlx5e_napi_poll);
netif_napi_set_irq(&c->napi, irq);
@ -2654,7 +2665,7 @@ int mlx5e_open_channels(struct mlx5e_priv *priv,
if (!chs->c || !cparam)
goto err_free;
err = mlx5e_build_channel_param(priv->mdev, &chs->params, priv->q_counter, cparam);
err = mlx5e_build_channel_param(priv->mdev, &chs->params, cparam);
if (err)
goto err_free;
@ -2942,15 +2953,18 @@ static MLX5E_DEFINE_PREACTIVATE_WRAPPER_CTX(mlx5e_update_netdev_queues);
static void mlx5e_set_default_xps_cpumasks(struct mlx5e_priv *priv,
struct mlx5e_params *params)
{
struct mlx5_core_dev *mdev = priv->mdev;
int num_comp_vectors, ix, irq;
num_comp_vectors = mlx5_comp_vectors_max(mdev);
int ix;
for (ix = 0; ix < params->num_channels; ix++) {
cpumask_clear(priv->scratchpad.cpumask);
int num_comp_vectors, irq, vec_ix;
struct mlx5_core_dev *mdev;
for (irq = ix; irq < num_comp_vectors; irq += params->num_channels) {
mdev = mlx5_sd_ch_ix_get_dev(priv->mdev, ix);
num_comp_vectors = mlx5_comp_vectors_max(mdev);
cpumask_clear(priv->scratchpad.cpumask);
vec_ix = mlx5_sd_ch_ix_get_vec_ix(mdev, ix);
for (irq = vec_ix; irq < num_comp_vectors; irq += params->num_channels) {
int cpu = mlx5_comp_vector_get_cpu(mdev, irq);
cpumask_set_cpu(cpu, priv->scratchpad.cpumask);
@ -3342,7 +3356,7 @@ int mlx5e_open_drop_rq(struct mlx5e_priv *priv,
struct mlx5e_cq *cq = &drop_rq->cq;
int err;
mlx5e_build_drop_rq_param(mdev, priv->drop_rq_q_counter, &rq_param);
mlx5e_build_drop_rq_param(mdev, &rq_param);
err = mlx5e_alloc_drop_cq(priv, cq, &cq_param);
if (err)
@ -3356,7 +3370,7 @@ int mlx5e_open_drop_rq(struct mlx5e_priv *priv,
if (err)
goto err_destroy_cq;
err = mlx5e_create_rq(drop_rq, &rq_param);
err = mlx5e_create_rq(drop_rq, &rq_param, priv->drop_rq_q_counter);
if (err)
goto err_free_rq;
@ -5271,13 +5285,17 @@ void mlx5e_create_q_counters(struct mlx5e_priv *priv)
u32 out[MLX5_ST_SZ_DW(alloc_q_counter_out)] = {};
u32 in[MLX5_ST_SZ_DW(alloc_q_counter_in)] = {};
struct mlx5_core_dev *mdev = priv->mdev;
int err;
struct mlx5_core_dev *pos;
int err, i;
MLX5_SET(alloc_q_counter_in, in, opcode, MLX5_CMD_OP_ALLOC_Q_COUNTER);
err = mlx5_cmd_exec_inout(mdev, alloc_q_counter, in, out);
if (!err)
priv->q_counter =
MLX5_GET(alloc_q_counter_out, out, counter_set_id);
mlx5_sd_for_each_dev(i, mdev, pos) {
err = mlx5_cmd_exec_inout(pos, alloc_q_counter, in, out);
if (!err)
priv->q_counter[i] =
MLX5_GET(alloc_q_counter_out, out, counter_set_id);
}
err = mlx5_cmd_exec_inout(mdev, alloc_q_counter, in, out);
if (!err)
@ -5288,13 +5306,17 @@ void mlx5e_create_q_counters(struct mlx5e_priv *priv)
void mlx5e_destroy_q_counters(struct mlx5e_priv *priv)
{
u32 in[MLX5_ST_SZ_DW(dealloc_q_counter_in)] = {};
struct mlx5_core_dev *pos;
int i;
MLX5_SET(dealloc_q_counter_in, in, opcode,
MLX5_CMD_OP_DEALLOC_Q_COUNTER);
if (priv->q_counter) {
MLX5_SET(dealloc_q_counter_in, in, counter_set_id,
priv->q_counter);
mlx5_cmd_exec_in(priv->mdev, dealloc_q_counter, in);
mlx5_sd_for_each_dev(i, priv->mdev, pos) {
if (priv->q_counter[i]) {
MLX5_SET(dealloc_q_counter_in, in, counter_set_id,
priv->q_counter[i]);
mlx5_cmd_exec_in(pos, dealloc_q_counter, in);
}
}
if (priv->drop_rq_q_counter) {
@ -5378,6 +5400,8 @@ static int mlx5e_init_nic_rx(struct mlx5e_priv *priv)
features = MLX5E_RX_RES_FEATURE_PTP;
if (mlx5_tunnel_inner_ft_supported(mdev))
features |= MLX5E_RX_RES_FEATURE_INNER_FT;
if (mlx5_get_sd(priv->mdev))
features |= MLX5E_RX_RES_FEATURE_MULTI_VHCA;
priv->rx_res = mlx5e_rx_res_create(priv->mdev, features, priv->max_nch, priv->drop_rq.rqn,
&priv->channels.params.packet_merge,
@ -5987,28 +6011,52 @@ void mlx5e_destroy_netdev(struct mlx5e_priv *priv)
free_netdev(netdev);
}
static int mlx5e_resume(struct auxiliary_device *adev)
static int _mlx5e_resume(struct auxiliary_device *adev)
{
struct mlx5_adev *edev = container_of(adev, struct mlx5_adev, adev);
struct mlx5e_dev *mlx5e_dev = auxiliary_get_drvdata(adev);
struct mlx5e_priv *priv = mlx5e_dev->priv;
struct net_device *netdev = priv->netdev;
struct mlx5_core_dev *mdev = edev->mdev;
int err;
struct mlx5_core_dev *pos, *to;
int err, i;
if (netif_device_present(netdev))
return 0;
err = mlx5e_create_mdev_resources(mdev, true);
mlx5_sd_for_each_dev(i, mdev, pos) {
err = mlx5e_create_mdev_resources(pos, true);
if (err)
goto err_destroy_mdev_res;
}
err = mlx5e_attach_netdev(priv);
if (err)
goto err_destroy_mdev_res;
return 0;
err_destroy_mdev_res:
to = pos;
mlx5_sd_for_each_dev_to(i, mdev, to, pos)
mlx5e_destroy_mdev_resources(pos);
return err;
}
static int mlx5e_resume(struct auxiliary_device *adev)
{
struct mlx5_adev *edev = container_of(adev, struct mlx5_adev, adev);
struct mlx5_core_dev *mdev = edev->mdev;
struct auxiliary_device *actual_adev;
int err;
err = mlx5_sd_init(mdev);
if (err)
return err;
err = mlx5e_attach_netdev(priv);
if (err) {
mlx5e_destroy_mdev_resources(mdev);
return err;
}
actual_adev = mlx5_sd_get_adev(mdev, adev, edev->idx);
if (actual_adev)
return _mlx5e_resume(actual_adev);
return 0;
}
@ -6018,21 +6066,36 @@ static int _mlx5e_suspend(struct auxiliary_device *adev)
struct mlx5e_priv *priv = mlx5e_dev->priv;
struct net_device *netdev = priv->netdev;
struct mlx5_core_dev *mdev = priv->mdev;
struct mlx5_core_dev *pos;
int i;
if (!netif_device_present(netdev)) {
if (test_bit(MLX5E_STATE_DESTROYING, &priv->state))
mlx5e_destroy_mdev_resources(mdev);
mlx5_sd_for_each_dev(i, mdev, pos)
mlx5e_destroy_mdev_resources(pos);
return -ENODEV;
}
mlx5e_detach_netdev(priv);
mlx5e_destroy_mdev_resources(mdev);
mlx5_sd_for_each_dev(i, mdev, pos)
mlx5e_destroy_mdev_resources(pos);
return 0;
}
static int mlx5e_suspend(struct auxiliary_device *adev, pm_message_t state)
{
return _mlx5e_suspend(adev);
struct mlx5_adev *edev = container_of(adev, struct mlx5_adev, adev);
struct mlx5_core_dev *mdev = edev->mdev;
struct auxiliary_device *actual_adev;
int err = 0;
actual_adev = mlx5_sd_get_adev(mdev, adev, edev->idx);
if (actual_adev)
err = _mlx5e_suspend(actual_adev);
mlx5_sd_cleanup(mdev);
return err;
}
static int _mlx5e_probe(struct auxiliary_device *adev)
@ -6078,9 +6141,9 @@ static int _mlx5e_probe(struct auxiliary_device *adev)
goto err_destroy_netdev;
}
err = mlx5e_resume(adev);
err = _mlx5e_resume(adev);
if (err) {
mlx5_core_err(mdev, "mlx5e_resume failed, %d\n", err);
mlx5_core_err(mdev, "_mlx5e_resume failed, %d\n", err);
goto err_profile_cleanup;
}
@ -6111,15 +6174,29 @@ err_devlink_unregister:
static int mlx5e_probe(struct auxiliary_device *adev,
const struct auxiliary_device_id *id)
{
return _mlx5e_probe(adev);
struct mlx5_adev *edev = container_of(adev, struct mlx5_adev, adev);
struct mlx5_core_dev *mdev = edev->mdev;
struct auxiliary_device *actual_adev;
int err;
err = mlx5_sd_init(mdev);
if (err)
return err;
actual_adev = mlx5_sd_get_adev(mdev, adev, edev->idx);
if (actual_adev)
return _mlx5e_probe(actual_adev);
return 0;
}
static void mlx5e_remove(struct auxiliary_device *adev)
static void _mlx5e_remove(struct auxiliary_device *adev)
{
struct mlx5_adev *edev = container_of(adev, struct mlx5_adev, adev);
struct mlx5e_dev *mlx5e_dev = auxiliary_get_drvdata(adev);
struct mlx5e_priv *priv = mlx5e_dev->priv;
struct mlx5_core_dev *mdev = edev->mdev;
mlx5_core_uplink_netdev_set(priv->mdev, NULL);
mlx5_core_uplink_netdev_set(mdev, NULL);
mlx5e_dcbnl_delete_app(priv);
unregister_netdev(priv->netdev);
_mlx5e_suspend(adev);
@ -6129,6 +6206,19 @@ static void mlx5e_remove(struct auxiliary_device *adev)
mlx5e_destroy_devlink(mlx5e_dev);
}
static void mlx5e_remove(struct auxiliary_device *adev)
{
struct mlx5_adev *edev = container_of(adev, struct mlx5_adev, adev);
struct mlx5_core_dev *mdev = edev->mdev;
struct auxiliary_device *actual_adev;
actual_adev = mlx5_sd_get_adev(mdev, adev, edev->idx);
if (actual_adev)
_mlx5e_remove(actual_adev);
mlx5_sd_cleanup(mdev);
}
static const struct auxiliary_device_id mlx5e_id_table[] = {
{ .name = MLX5_ADEV_NAME ".eth", },
{},

39
drivers/net/ethernet/mellanox/mlx5/core/en_stats.c
View File

@ -561,11 +561,23 @@ static const struct counter_desc drop_rq_stats_desc[] = {
#define NUM_Q_COUNTERS ARRAY_SIZE(q_stats_desc)
#define NUM_DROP_RQ_COUNTERS ARRAY_SIZE(drop_rq_stats_desc)
static bool q_counter_any(struct mlx5e_priv *priv)
{
struct mlx5_core_dev *pos;
int i;
mlx5_sd_for_each_dev(i, priv->mdev, pos)
if (priv->q_counter[i++])
return true;
return false;
}
static MLX5E_DECLARE_STATS_GRP_OP_NUM_STATS(qcnt)
{
int num_stats = 0;
if (priv->q_counter)
if (q_counter_any(priv))
num_stats += NUM_Q_COUNTERS;
if (priv->drop_rq_q_counter)
@ -578,7 +590,7 @@ static MLX5E_DECLARE_STATS_GRP_OP_FILL_STRS(qcnt)
{
int i;
for (i = 0; i < NUM_Q_COUNTERS && priv->q_counter; i++)
for (i = 0; i < NUM_Q_COUNTERS && q_counter_any(priv); i++)
strcpy(data + (idx++) * ETH_GSTRING_LEN,
q_stats_desc[i].format);
@ -593,7 +605,7 @@ static MLX5E_DECLARE_STATS_GRP_OP_FILL_STATS(qcnt)
{
int i;
for (i = 0; i < NUM_Q_COUNTERS && priv->q_counter; i++)
for (i = 0; i < NUM_Q_COUNTERS && q_counter_any(priv); i++)
data[idx++] = MLX5E_READ_CTR32_CPU(&priv->stats.qcnt,
q_stats_desc, i);
for (i = 0; i < NUM_DROP_RQ_COUNTERS && priv->drop_rq_q_counter; i++)
@ -607,18 +619,23 @@ static MLX5E_DECLARE_STATS_GRP_OP_UPDATE_STATS(qcnt)
struct mlx5e_qcounter_stats *qcnt = &priv->stats.qcnt;
u32 out[MLX5_ST_SZ_DW(query_q_counter_out)] = {};
u32 in[MLX5_ST_SZ_DW(query_q_counter_in)] = {};
int ret;
struct mlx5_core_dev *pos;
u32 rx_out_of_buffer = 0;
int ret, i;
MLX5_SET(query_q_counter_in, in, opcode, MLX5_CMD_OP_QUERY_Q_COUNTER);
if (priv->q_counter) {
MLX5_SET(query_q_counter_in, in, counter_set_id,
priv->q_counter);
ret = mlx5_cmd_exec_inout(priv->mdev, query_q_counter, in, out);
if (!ret)
qcnt->rx_out_of_buffer = MLX5_GET(query_q_counter_out,
out, out_of_buffer);
mlx5_sd_for_each_dev(i, priv->mdev, pos) {
if (priv->q_counter[i]) {
MLX5_SET(query_q_counter_in, in, counter_set_id,
priv->q_counter[i]);
ret = mlx5_cmd_exec_inout(pos, query_q_counter, in, out);
if (!ret)
rx_out_of_buffer += MLX5_GET(query_q_counter_out,
out, out_of_buffer);
}
}
qcnt->rx_out_of_buffer = rx_out_of_buffer;
if (priv->drop_rq_q_counter) {
MLX5_SET(query_q_counter_in, in, counter_set_id,

4
drivers/net/ethernet/mellanox/mlx5/core/en_tc.c
View File

@ -766,7 +766,7 @@ static int mlx5e_hairpin_create_indirect_rqt(struct mlx5e_hairpin *hp)
return err;
mlx5e_rss_params_indir_init_uniform(&indir, hp->num_channels);
err = mlx5e_rqt_init_indir(&hp->indir_rqt, mdev, hp->pair->rqn, hp->num_channels,
err = mlx5e_rqt_init_indir(&hp->indir_rqt, mdev, hp->pair->rqn, NULL, hp->num_channels,
mlx5e_rx_res_get_current_hash(priv->rx_res).hfunc,
&indir);
@ -1169,7 +1169,7 @@ static int mlx5e_hairpin_flow_add(struct mlx5e_priv *priv,
MLX5_CAP_GEN(priv->mdev, log_min_hairpin_wq_data_sz),
MLX5_CAP_GEN(priv->mdev, log_max_hairpin_wq_data_sz));
params.q_counter = priv->q_counter;
params.q_counter = priv->q_counter[0];
err = devl_param_driverinit_value_get(
devlink, MLX5_DEVLINK_PARAM_ID_HAIRPIN_NUM_QUEUES, &val);
if (err) {

1
drivers/net/ethernet/mellanox/mlx5/core/lib/devcom.h
View File

@ -10,6 +10,7 @@ enum mlx5_devcom_component {
MLX5_DEVCOM_ESW_OFFLOADS,
MLX5_DEVCOM_MPV,
MLX5_DEVCOM_HCA_PORTS,
MLX5_DEVCOM_SD_GROUP,
MLX5_DEVCOM_NUM_COMPONENTS,
};

12
drivers/net/ethernet/mellanox/mlx5/core/lib/mlx5.h
View File

@ -54,4 +54,16 @@ static inline struct net_device *mlx5_uplink_netdev_get(struct mlx5_core_dev *md
{
return mdev->mlx5e_res.uplink_netdev;
}
struct mlx5_sd;
static inline struct mlx5_sd *mlx5_get_sd(struct mlx5_core_dev *dev)
{
return dev->sd;
}
static inline void mlx5_set_sd(struct mlx5_core_dev *dev, struct mlx5_sd *sd)
{
dev->sd = sd;
}
#endif

524
drivers/net/ethernet/mellanox/mlx5/core/lib/sd.c Normal file
View File

@ -0,0 +1,524 @@
// SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB
/* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */
#include "lib/sd.h"
#include "mlx5_core.h"
#include "lib/mlx5.h"
#include "fs_cmd.h"
#include <linux/mlx5/vport.h>
#include <linux/debugfs.h>
#define sd_info(__dev, format, ...) \
dev_info((__dev)->device, "Socket-Direct: " format, ##__VA_ARGS__)
#define sd_warn(__dev, format, ...) \
dev_warn((__dev)->device, "Socket-Direct: " format, ##__VA_ARGS__)
struct mlx5_sd {
u32 group_id;
u8 host_buses;
struct mlx5_devcom_comp_dev *devcom;
struct dentry *dfs;
bool primary;
union {
struct { /* primary */
struct mlx5_core_dev *secondaries[MLX5_SD_MAX_GROUP_SZ - 1];
struct mlx5_flow_table *tx_ft;
};
struct { /* secondary */
struct mlx5_core_dev *primary_dev;
u32 alias_obj_id;
};
};
};
static int mlx5_sd_get_host_buses(struct mlx5_core_dev *dev)
{
struct mlx5_sd *sd = mlx5_get_sd(dev);
if (!sd)
return 1;
return sd->host_buses;
}
static struct mlx5_core_dev *mlx5_sd_get_primary(struct mlx5_core_dev *dev)
{
struct mlx5_sd *sd = mlx5_get_sd(dev);
if (!sd)
return dev;
return sd->primary ? dev : sd->primary_dev;
}
struct mlx5_core_dev *
mlx5_sd_primary_get_peer(struct mlx5_core_dev *primary, int idx)
{
struct mlx5_sd *sd;
if (idx == 0)
return primary;
if (idx >= mlx5_sd_get_host_buses(primary))
return NULL;
sd = mlx5_get_sd(primary);
return sd->secondaries[idx - 1];
}
int mlx5_sd_ch_ix_get_dev_ix(struct mlx5_core_dev *dev, int ch_ix)
{
return ch_ix % mlx5_sd_get_host_buses(dev);
}
int mlx5_sd_ch_ix_get_vec_ix(struct mlx5_core_dev *dev, int ch_ix)
{
return ch_ix / mlx5_sd_get_host_buses(dev);
}
struct mlx5_core_dev *mlx5_sd_ch_ix_get_dev(struct mlx5_core_dev *primary, int ch_ix)
{
int mdev_idx = mlx5_sd_ch_ix_get_dev_ix(primary, ch_ix);
return mlx5_sd_primary_get_peer(primary, mdev_idx);
}
static bool ft_create_alias_supported(struct mlx5_core_dev *dev)
{
u64 obj_allowed = MLX5_CAP_GEN_2_64(dev, allowed_object_for_other_vhca_access);
u32 obj_supp = MLX5_CAP_GEN_2(dev, cross_vhca_object_to_object_supported);
if (!(obj_supp &
MLX5_CROSS_VHCA_OBJ_TO_OBJ_SUPPORTED_LOCAL_FLOW_TABLE_ROOT_TO_REMOTE_FLOW_TABLE))
return false;
if (!(obj_allowed & MLX5_ALLOWED_OBJ_FOR_OTHER_VHCA_ACCESS_FLOW_TABLE))
return false;
return true;
}
static bool mlx5_sd_is_supported(struct mlx5_core_dev *dev, u8 host_buses)
{
/* Feature is currently implemented for PFs only */
if (!mlx5_core_is_pf(dev))
return false;
/* Honor the SW implementation limit */
if (host_buses > MLX5_SD_MAX_GROUP_SZ)
return false;
/* Disconnect secondaries from the network */
if (!MLX5_CAP_GEN(dev, eswitch_manager))
return false;
if (!MLX5_CAP_GEN(dev, silent_mode))
return false;
/* RX steering from primary to secondaries */
if (!MLX5_CAP_GEN(dev, cross_vhca_rqt))
return false;
if (host_buses > MLX5_CAP_GEN_2(dev, max_rqt_vhca_id))
return false;
/* TX steering from secondaries to primary */
if (!ft_create_alias_supported(dev))
return false;
if (!MLX5_CAP_FLOWTABLE_NIC_TX(dev, reset_root_to_default))
return false;
return true;
}
static int mlx5_query_sd(struct mlx5_core_dev *dev, bool *sdm,
u8 *host_buses, u8 *sd_group)
{
u32 out[MLX5_ST_SZ_DW(mpir_reg)];
int err;
err = mlx5_query_mpir_reg(dev, out);
if (err)
return err;
err = mlx5_query_nic_vport_sd_group(dev, sd_group);
if (err)
return err;
*sdm = MLX5_GET(mpir_reg, out, sdm);
*host_buses = MLX5_GET(mpir_reg, out, host_buses);
return 0;
}
static u32 mlx5_sd_group_id(struct mlx5_core_dev *dev, u8 sd_group)
{
return (u32)((MLX5_CAP_GEN(dev, native_port_num) << 8) | sd_group);
}
static int sd_init(struct mlx5_core_dev *dev)
{
u8 host_buses, sd_group;
struct mlx5_sd *sd;
u32 group_id;
bool sdm;
int err;
if (!MLX5_CAP_MCAM_REG(dev, mpir))
return 0;
err = mlx5_query_sd(dev, &sdm, &host_buses, &sd_group);
if (err)
return err;
if (!sdm)
return 0;
if (!sd_group)
return 0;
group_id = mlx5_sd_group_id(dev, sd_group);
if (!mlx5_sd_is_supported(dev, host_buses)) {
sd_warn(dev, "can't support requested netdev combining for group id 0x%x), skipping\n",
group_id);
return 0;
}
sd = kzalloc(sizeof(*sd), GFP_KERNEL);
if (!sd)
return -ENOMEM;
sd->host_buses = host_buses;
sd->group_id = group_id;
mlx5_set_sd(dev, sd);
return 0;
}
static void sd_cleanup(struct mlx5_core_dev *dev)
{
struct mlx5_sd *sd = mlx5_get_sd(dev);
mlx5_set_sd(dev, NULL);
kfree(sd);
}
static int sd_register(struct mlx5_core_dev *dev)
{
struct mlx5_devcom_comp_dev *devcom, *pos;
struct mlx5_core_dev *peer, *primary;
struct mlx5_sd *sd, *primary_sd;
int err, i;
sd = mlx5_get_sd(dev);
devcom = mlx5_devcom_register_component(dev->priv.devc, MLX5_DEVCOM_SD_GROUP,
sd->group_id, NULL, dev);
if (!devcom)
return -ENOMEM;
sd->devcom = devcom;
if (mlx5_devcom_comp_get_size(devcom) != sd->host_buses)
return 0;
mlx5_devcom_comp_lock(devcom);
mlx5_devcom_comp_set_ready(devcom, true);
mlx5_devcom_comp_unlock(devcom);
if (!mlx5_devcom_for_each_peer_begin(devcom)) {
err = -ENODEV;
goto err_devcom_unreg;
}
primary = dev;
mlx5_devcom_for_each_peer_entry(devcom, peer, pos)
if (peer->pdev->bus->number < primary->pdev->bus->number)
primary = peer;
primary_sd = mlx5_get_sd(primary);
primary_sd->primary = true;
i = 0;
/* loop the secondaries */
mlx5_devcom_for_each_peer_entry(primary_sd->devcom, peer, pos) {
struct mlx5_sd *peer_sd = mlx5_get_sd(peer);
primary_sd->secondaries[i++] = peer;
peer_sd->primary = false;
peer_sd->primary_dev = primary;
}
mlx5_devcom_for_each_peer_end(devcom);
return 0;
err_devcom_unreg:
mlx5_devcom_comp_lock(sd->devcom);
mlx5_devcom_comp_set_ready(sd->devcom, false);
mlx5_devcom_comp_unlock(sd->devcom);
mlx5_devcom_unregister_component(sd->devcom);
return err;
}
static void sd_unregister(struct mlx5_core_dev *dev)
{
struct mlx5_sd *sd = mlx5_get_sd(dev);
mlx5_devcom_comp_lock(sd->devcom);
mlx5_devcom_comp_set_ready(sd->devcom, false);
mlx5_devcom_comp_unlock(sd->devcom);
mlx5_devcom_unregister_component(sd->devcom);
}
static int sd_cmd_set_primary(struct mlx5_core_dev *primary, u8 *alias_key)
{
struct mlx5_cmd_allow_other_vhca_access_attr allow_attr = {};
struct mlx5_sd *sd = mlx5_get_sd(primary);
struct mlx5_flow_table_attr ft_attr = {};
struct mlx5_flow_namespace *nic_ns;
struct mlx5_flow_table *ft;
int err;
nic_ns = mlx5_get_flow_namespace(primary, MLX5_FLOW_NAMESPACE_EGRESS);
if (!nic_ns)
return -EOPNOTSUPP;
ft = mlx5_create_flow_table(nic_ns, &ft_attr);
if (IS_ERR(ft)) {
err = PTR_ERR(ft);
return err;
}
sd->tx_ft = ft;
memcpy(allow_attr.access_key, alias_key, ACCESS_KEY_LEN);
allow_attr.obj_type = MLX5_GENERAL_OBJECT_TYPES_FLOW_TABLE_ALIAS;
allow_attr.obj_id = (ft->type << FT_ID_FT_TYPE_OFFSET) | ft->id;
err = mlx5_cmd_allow_other_vhca_access(primary, &allow_attr);
if (err) {
mlx5_core_err(primary, "Failed to allow other vhca access err=%d\n",
err);
mlx5_destroy_flow_table(ft);
return err;
}
return 0;
}
static void sd_cmd_unset_primary(struct mlx5_core_dev *primary)
{
struct mlx5_sd *sd = mlx5_get_sd(primary);
mlx5_destroy_flow_table(sd->tx_ft);
}
static int sd_secondary_create_alias_ft(struct mlx5_core_dev *secondary,
struct mlx5_core_dev *primary,
struct mlx5_flow_table *ft,
u32 *obj_id, u8 *alias_key)
{
u32 aliased_object_id = (ft->type << FT_ID_FT_TYPE_OFFSET) | ft->id;
u16 vhca_id_to_be_accessed = MLX5_CAP_GEN(primary, vhca_id);
struct mlx5_cmd_alias_obj_create_attr alias_attr = {};
int ret;
memcpy(alias_attr.access_key, alias_key, ACCESS_KEY_LEN);
alias_attr.obj_id = aliased_object_id;
alias_attr.obj_type = MLX5_GENERAL_OBJECT_TYPES_FLOW_TABLE_ALIAS;
alias_attr.vhca_id = vhca_id_to_be_accessed;
ret = mlx5_cmd_alias_obj_create(secondary, &alias_attr, obj_id);
if (ret) {
mlx5_core_err(secondary, "Failed to create alias object err=%d\n",
ret);
return ret;
}
return 0;
}
static void sd_secondary_destroy_alias_ft(struct mlx5_core_dev *secondary)
{
struct mlx5_sd *sd = mlx5_get_sd(secondary);
mlx5_cmd_alias_obj_destroy(secondary, sd->alias_obj_id,
MLX5_GENERAL_OBJECT_TYPES_FLOW_TABLE_ALIAS);
}
static int sd_cmd_set_secondary(struct mlx5_core_dev *secondary,
struct mlx5_core_dev *primary,
u8 *alias_key)
{
struct mlx5_sd *primary_sd = mlx5_get_sd(primary);
struct mlx5_sd *sd = mlx5_get_sd(secondary);
int err;
err = mlx5_fs_cmd_set_l2table_entry_silent(secondary, 1);
if (err)
return err;
err = sd_secondary_create_alias_ft(secondary, primary, primary_sd->tx_ft,
&sd->alias_obj_id, alias_key);
if (err)
goto err_unset_silent;
err = mlx5_fs_cmd_set_tx_flow_table_root(secondary, sd->alias_obj_id, false);
if (err)
goto err_destroy_alias_ft;
return 0;
err_destroy_alias_ft:
sd_secondary_destroy_alias_ft(secondary);
err_unset_silent:
mlx5_fs_cmd_set_l2table_entry_silent(secondary, 0);
return err;
}
static void sd_cmd_unset_secondary(struct mlx5_core_dev *secondary)
{
mlx5_fs_cmd_set_tx_flow_table_root(secondary, 0, true);
sd_secondary_destroy_alias_ft(secondary);
mlx5_fs_cmd_set_l2table_entry_silent(secondary, 0);
}
static void sd_print_group(struct mlx5_core_dev *primary)
{
struct mlx5_sd *sd = mlx5_get_sd(primary);
struct mlx5_core_dev *pos;
int i;
sd_info(primary, "group id %#x, primary %s, vhca %#x\n",
sd->group_id, pci_name(primary->pdev),
MLX5_CAP_GEN(primary, vhca_id));
mlx5_sd_for_each_secondary(i, primary, pos)
sd_info(primary, "group id %#x, secondary_%d %s, vhca %#x\n",
sd->group_id, i - 1, pci_name(pos->pdev),
MLX5_CAP_GEN(pos, vhca_id));
}
static ssize_t dev_read(struct file *filp, char __user *buf, size_t count,
loff_t *pos)
{
struct mlx5_core_dev *dev;
char tbuf[32];
int ret;
dev = filp->private_data;
ret = snprintf(tbuf, sizeof(tbuf), "%s vhca %#x\n", pci_name(dev->pdev),
MLX5_CAP_GEN(dev, vhca_id));
return simple_read_from_buffer(buf, count, pos, tbuf, ret);
}
static const struct file_operations dev_fops = {
.owner = THIS_MODULE,
.open = simple_open,
.read = dev_read,
};
int mlx5_sd_init(struct mlx5_core_dev *dev)
{
struct mlx5_core_dev *primary, *pos, *to;
struct mlx5_sd *sd = mlx5_get_sd(dev);
u8 alias_key[ACCESS_KEY_LEN];
int err, i;
err = sd_init(dev);
if (err)
return err;
sd = mlx5_get_sd(dev);
if (!sd)
return 0;
err = sd_register(dev);
if (err)
goto err_sd_cleanup;
if (!mlx5_devcom_comp_is_ready(sd->devcom))
return 0;
primary = mlx5_sd_get_primary(dev);
for (i = 0; i < ACCESS_KEY_LEN; i++)
alias_key[i] = get_random_u8();
err = sd_cmd_set_primary(primary, alias_key);
if (err)
goto err_sd_unregister;
sd->dfs = debugfs_create_dir("multi-pf", mlx5_debugfs_get_dev_root(primary));
debugfs_create_x32("group_id", 0400, sd->dfs, &sd->group_id);
debugfs_create_file("primary", 0400, sd->dfs, primary, &dev_fops);
mlx5_sd_for_each_secondary(i, primary, pos) {
char name[32];
err = sd_cmd_set_secondary(pos, primary, alias_key);
if (err)
goto err_unset_secondaries;
snprintf(name, sizeof(name), "secondary_%d", i - 1);
debugfs_create_file(name, 0400, sd->dfs, pos, &dev_fops);
}
sd_info(primary, "group id %#x, size %d, combined\n",
sd->group_id, mlx5_devcom_comp_get_size(sd->devcom));
sd_print_group(primary);
return 0;
err_unset_secondaries:
to = pos;
mlx5_sd_for_each_secondary_to(i, primary, to, pos)
sd_cmd_unset_secondary(pos);
sd_cmd_unset_primary(primary);
debugfs_remove_recursive(sd->dfs);
err_sd_unregister:
sd_unregister(dev);
err_sd_cleanup:
sd_cleanup(dev);
return err;
}
void mlx5_sd_cleanup(struct mlx5_core_dev *dev)
{
struct mlx5_sd *sd = mlx5_get_sd(dev);
struct mlx5_core_dev *primary, *pos;
int i;
if (!sd)
return;
if (!mlx5_devcom_comp_is_ready(sd->devcom))
goto out;
primary = mlx5_sd_get_primary(dev);
mlx5_sd_for_each_secondary(i, primary, pos)
sd_cmd_unset_secondary(pos);
sd_cmd_unset_primary(primary);
debugfs_remove_recursive(sd->dfs);
sd_info(primary, "group id %#x, uncombined\n", sd->group_id);
out:
sd_unregister(dev);
sd_cleanup(dev);
}
struct auxiliary_device *mlx5_sd_get_adev(struct mlx5_core_dev *dev,
struct auxiliary_device *adev,
int idx)
{
struct mlx5_sd *sd = mlx5_get_sd(dev);
struct mlx5_core_dev *primary;
if (!sd)
return adev;
if (!mlx5_devcom_comp_is_ready(sd->devcom))
return NULL;
primary = mlx5_sd_get_primary(dev);
if (dev == primary)
return adev;
return &primary->priv.adev[idx]->adev;
}

38
drivers/net/ethernet/mellanox/mlx5/core/lib/sd.h Normal file
View File

@ -0,0 +1,38 @@
/* SPDX-License-Identifier: GPL-2.0 OR Linux-OpenIB */
/* Copyright (c) 2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. */
#ifndef __MLX5_LIB_SD_H__
#define __MLX5_LIB_SD_H__
#define MLX5_SD_MAX_GROUP_SZ 2
struct mlx5_sd;
struct mlx5_core_dev *mlx5_sd_primary_get_peer(struct mlx5_core_dev *primary, int idx);
int mlx5_sd_ch_ix_get_dev_ix(struct mlx5_core_dev *dev, int ch_ix);
int mlx5_sd_ch_ix_get_vec_ix(struct mlx5_core_dev *dev, int ch_ix);
struct mlx5_core_dev *mlx5_sd_ch_ix_get_dev(struct mlx5_core_dev *primary, int ch_ix);
struct auxiliary_device *mlx5_sd_get_adev(struct mlx5_core_dev *dev,
struct auxiliary_device *adev,
int idx);
int mlx5_sd_init(struct mlx5_core_dev *dev);
void mlx5_sd_cleanup(struct mlx5_core_dev *dev);
#define mlx5_sd_for_each_dev_from_to(i, primary, ix_from, to, pos) \
for (i = ix_from; \
(pos = mlx5_sd_primary_get_peer(primary, i)) && pos != (to); i++)
#define mlx5_sd_for_each_dev(i, primary, pos) \
mlx5_sd_for_each_dev_from_to(i, primary, 0, NULL, pos)
#define mlx5_sd_for_each_dev_to(i, primary, to, pos) \
mlx5_sd_for_each_dev_from_to(i, primary, 0, to, pos)
#define mlx5_sd_for_each_secondary(i, primary, pos) \
mlx5_sd_for_each_dev_from_to(i, primary, 1, NULL, pos)
#define mlx5_sd_for_each_secondary_to(i, primary, to, pos) \
mlx5_sd_for_each_dev_from_to(i, primary, 1, to, pos)
#endif /* __MLX5_LIB_SD_H__ */

1
include/linux/mlx5/driver.h
View File

@ -823,6 +823,7 @@ struct mlx5_core_dev {
struct blocking_notifier_head macsec_nh;
#endif
u64 num_ipsec_offloads;
struct mlx5_sd *sd;
};
struct mlx5_db {

4
include/linux/mlx5/mlx5_ifc.h
View File

@ -10253,7 +10253,9 @@ struct mlx5_ifc_mcam_access_reg_bits {
u8 mcqi[0x1];
u8 mcqs[0x1];
u8 regs_95_to_87[0x9];
u8 regs_95_to_90[0x6];
u8 mpir[0x1];
u8 regs_88_to_87[0x2];
u8 mpegc[0x1];
u8 mtutc[0x1];
u8 regs_84_to_68[0x11];