Minki/linux
1
0
Fork 1
mirror of https://github.com/torvalds/linux.git synced 2024-11-24 13:11:40 +00:00
Code Issues Packages Projects Releases Wiki Activity

PCI-Express Non-Transparent Bridge Support

Browse Source 
A PCI-Express non-transparent bridge (NTB) is a point-to-point PCIe bus
connecting 2 systems, providing electrical isolation between the two subsystems.
A non-transparent bridge is functionally similar to a transparent bridge except
that both sides of the bridge have their own independent address domains.  The
host on one side of the bridge will not have the visibility of the complete
memory or I/O space on the other side of the bridge.  To communicate across the
non-transparent bridge, each NTB endpoint has one (or more) apertures exposed to
the local system.  Writes to these apertures are mirrored to memory on the
remote system.  Communications can also occur through the use of doorbell
registers that initiate interrupts to the alternate domain, and scratch-pad
registers accessible from both sides.

The NTB device driver is needed to configure these memory windows, doorbell, and
scratch-pad registers as well as use them in such a way as they can be turned
into a viable communication channel to the remote system.  ntb_hw.[ch]
determines the usage model (NTB to NTB or NTB to Root Port) and abstracts away
the underlying hardware to provide access and a common interface to the doorbell
registers, scratch pads, and memory windows.  These hardware interfaces are
exported so that other, non-mainlined kernel drivers can access these.
ntb_transport.[ch] also uses the exported interfaces in ntb_hw.[ch] to setup a
communication channel(s) and provide a reliable way of transferring data from
one side to the other, which it then exports so that "client" drivers can access
them.  These client drivers are used to provide a standard kernel interface
(i.e., Ethernet device) to NTB, such that Linux can transfer data from one
system to the other in a standard way.

Signed-off-by: Jon Mason <jon.mason@intel.com>
Reviewed-by: Nicholas Bellinger <nab@linux-iscsi.org>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>
This commit is contained in:
Jon Mason 2012-11-16 19:27:12 -07:00 committed by Greg Kroah-Hartman
parent ea8a83a4b7
commit fce8a7bb5b
10 changed files with 3012 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
MAINTAINERS
View File

@ -5394,6 +5394,12 @@ S: Maintained
F: Documentation/scsi/NinjaSCSI.txt F: Documentation/scsi/NinjaSCSI.txt
F: drivers/scsi/nsp32* F: drivers/scsi/nsp32*
NTB DRIVER
M: Jon Mason <jon.mason@intel.com>
S: Supported
F: drivers/ntb/
F: include/linux/ntb.h
NTFS FILESYSTEM NTFS FILESYSTEM
M: Anton Altaparmakov <anton@tuxera.com> M: Anton Altaparmakov <anton@tuxera.com>
L: linux-ntfs-dev@lists.sourceforge.net L: linux-ntfs-dev@lists.sourceforge.net

2
drivers/Kconfig
View File

@ -150,6 +150,8 @@ source "drivers/memory/Kconfig"
source "drivers/iio/Kconfig" source "drivers/iio/Kconfig"
source "drivers/ntb/Kconfig"
source "drivers/vme/Kconfig" source "drivers/vme/Kconfig"
source "drivers/pwm/Kconfig" source "drivers/pwm/Kconfig"

1
drivers/Makefile
View File

@ -146,3 +146,4 @@ obj-$(CONFIG_MEMORY) += memory/
obj-$(CONFIG_IIO) += iio/ obj-$(CONFIG_IIO) += iio/
obj-$(CONFIG_VME_BUS) += vme/ obj-$(CONFIG_VME_BUS) += vme/
obj-$(CONFIG_IPACK_BUS) += ipack/ obj-$(CONFIG_IPACK_BUS) += ipack/
obj-$(CONFIG_NTB) += ntb/

13
drivers/ntb/Kconfig Normal file
View File

@ -0,0 +1,13 @@
config NTB
tristate "Intel Non-Transparent Bridge support"
depends on PCI
depends on X86
help
The PCI-E Non-transparent bridge hardware is a point-to-point PCI-E bus
connecting 2 systems. When configured, writes to the device's PCI
mapped memory will be mirrored to a buffer on the remote system. The
ntb Linux driver uses this point-to-point communication as a method to
transfer data from one system to the other.
If unsure, say N.

3
drivers/ntb/Makefile Normal file
View File

@ -0,0 +1,3 @@
obj-$(CONFIG_NTB) += ntb.o
ntb-objs := ntb_hw.o ntb_transport.o

1157
drivers/ntb/ntb_hw.c Normal file
View File

File diff suppressed because it is too large Load Diff

181
drivers/ntb/ntb_hw.h Normal file
View File

@ -0,0 +1,181 @@
/*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* Copyright(c) 2012 Intel Corporation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* BSD LICENSE
*
* Copyright(c) 2012 Intel Corporation. All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copy
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* * Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
* Intel PCIe NTB Linux driver
*
* Contact Information:
* Jon Mason <jon.mason@intel.com>
*/
#define PCI_DEVICE_ID_INTEL_NTB_B2B_JSF 0x3725
#define PCI_DEVICE_ID_INTEL_NTB_CLASSIC_JSF 0x3726
#define PCI_DEVICE_ID_INTEL_NTB_RP_JSF 0x3727
#define PCI_DEVICE_ID_INTEL_NTB_RP_SNB 0x3C08
#define PCI_DEVICE_ID_INTEL_NTB_B2B_SNB 0x3C0D
#define PCI_DEVICE_ID_INTEL_NTB_CLASSIC_SNB 0x3C0E
#define PCI_DEVICE_ID_INTEL_NTB_2ND_SNB 0x3C0F
#define PCI_DEVICE_ID_INTEL_NTB_B2B_BWD 0x0C4E
#define msix_table_size(control) ((control & PCI_MSIX_FLAGS_QSIZE)+1)
#define NTB_BAR_MMIO 0
#define NTB_BAR_23 2
#define NTB_BAR_45 4
#define NTB_BAR_MASK ((1 << NTB_BAR_MMIO) | (1 << NTB_BAR_23) |\
(1 << NTB_BAR_45))
#define NTB_LINK_DOWN 0
#define NTB_LINK_UP 1
#define NTB_HB_TIMEOUT msecs_to_jiffies(1000)
#define NTB_NUM_MW 2
enum ntb_hw_event {
NTB_EVENT_SW_EVENT0 = 0,
NTB_EVENT_SW_EVENT1,
NTB_EVENT_SW_EVENT2,
NTB_EVENT_HW_ERROR,
NTB_EVENT_HW_LINK_UP,
NTB_EVENT_HW_LINK_DOWN,
};
struct ntb_mw {
dma_addr_t phys_addr;
void __iomem *vbase;
resource_size_t bar_sz;
};
struct ntb_db_cb {
void (*callback) (void *data, int db_num);
unsigned int db_num;
void *data;
struct ntb_device *ndev;
};
struct ntb_device {
struct pci_dev *pdev;
struct msix_entry *msix_entries;
void __iomem *reg_base;
struct ntb_mw mw[NTB_NUM_MW];
struct {
unsigned int max_spads;
unsigned int max_db_bits;
unsigned int msix_cnt;
} limits;
struct {
void __iomem *pdb;
void __iomem *pdb_mask;
void __iomem *sdb;
void __iomem *sbar2_xlat;
void __iomem *sbar4_xlat;
void __iomem *spad_write;
void __iomem *spad_read;
void __iomem *lnk_cntl;
void __iomem *lnk_stat;
void __iomem *spci_cmd;
} reg_ofs;
struct ntb_transport *ntb_transport;
void (*event_cb)(void *handle, enum ntb_hw_event event);
struct ntb_db_cb *db_cb;
unsigned char hw_type;
unsigned char conn_type;
unsigned char dev_type;
unsigned char num_msix;
unsigned char bits_per_vector;
unsigned char max_cbs;
unsigned char link_status;
struct delayed_work hb_timer;
unsigned long last_ts;
};
/**
* ntb_hw_link_status() - return the hardware link status
* @ndev: pointer to ntb_device instance
*
* Returns true if the hardware is connected to the remote system
*
* RETURNS: true or false based on the hardware link state
*/
static inline bool ntb_hw_link_status(struct ntb_device *ndev)
{
return ndev->link_status == NTB_LINK_UP;
}
/**
* ntb_query_pdev() - return the pci_dev pointer
* @ndev: pointer to ntb_device instance
*
* Given the ntb pointer return the pci_dev pointerfor the NTB hardware device
*
* RETURNS: a pointer to the ntb pci_dev
*/
static inline struct pci_dev *ntb_query_pdev(struct ntb_device *ndev)
{
return ndev->pdev;
}
struct ntb_device *ntb_register_transport(struct pci_dev *pdev,
void *transport);
void ntb_unregister_transport(struct ntb_device *ndev);
void ntb_set_mw_addr(struct ntb_device *ndev, unsigned int mw, u64 addr);
int ntb_register_db_callback(struct ntb_device *ndev, unsigned int idx,
void *data, void (*db_cb_func) (void *data,
int db_num));
void ntb_unregister_db_callback(struct ntb_device *ndev, unsigned int idx);
int ntb_register_event_callback(struct ntb_device *ndev,
void (*event_cb_func) (void *handle,
unsigned int event));
void ntb_unregister_event_callback(struct ntb_device *ndev);
int ntb_get_max_spads(struct ntb_device *ndev);
int ntb_write_local_spad(struct ntb_device *ndev, unsigned int idx, u32 val);
int ntb_read_local_spad(struct ntb_device *ndev, unsigned int idx, u32 *val);
int ntb_write_remote_spad(struct ntb_device *ndev, unsigned int idx, u32 val);
int ntb_read_remote_spad(struct ntb_device *ndev, unsigned int idx, u32 *val);
void *ntb_get_mw_vbase(struct ntb_device *ndev, unsigned int mw);
resource_size_t ntb_get_mw_size(struct ntb_device *ndev, unsigned int mw);
void ntb_ring_sdb(struct ntb_device *ndev, unsigned int idx);
void *ntb_find_transport(struct pci_dev *pdev);
int ntb_transport_init(struct pci_dev *pdev);
void ntb_transport_free(void *transport);

139
drivers/ntb/ntb_regs.h Normal file
View File

@ -0,0 +1,139 @@
/*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* Copyright(c) 2012 Intel Corporation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* BSD LICENSE
*
* Copyright(c) 2012 Intel Corporation. All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copy
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* * Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
* Intel PCIe NTB Linux driver
*
* Contact Information:
* Jon Mason <jon.mason@intel.com>
*/
#define NTB_LINK_ENABLE 0x0000
#define NTB_LINK_DISABLE 0x0002
#define NTB_LINK_STATUS_ACTIVE 0x2000
#define NTB_LINK_SPEED_MASK 0x000f
#define NTB_LINK_WIDTH_MASK 0x03f0
#define SNB_MSIX_CNT 4
#define SNB_MAX_SPADS 16
#define SNB_MAX_COMPAT_SPADS 8
/* Reserve the uppermost bit for link interrupt */
#define SNB_MAX_DB_BITS 15
#define SNB_DB_BITS_PER_VEC 5
#define SNB_DB_HW_LINK 0x8000
#define SNB_PCICMD_OFFSET 0x0504
#define SNB_DEVCTRL_OFFSET 0x0598
#define SNB_LINK_STATUS_OFFSET 0x01A2
#define SNB_PBAR2LMT_OFFSET 0x0000
#define SNB_PBAR4LMT_OFFSET 0x0008
#define SNB_PBAR2XLAT_OFFSET 0x0010
#define SNB_PBAR4XLAT_OFFSET 0x0018
#define SNB_SBAR2LMT_OFFSET 0x0020
#define SNB_SBAR4LMT_OFFSET 0x0028
#define SNB_SBAR2XLAT_OFFSET 0x0030
#define SNB_SBAR4XLAT_OFFSET 0x0038
#define SNB_SBAR0BASE_OFFSET 0x0040
#define SNB_SBAR2BASE_OFFSET 0x0048
#define SNB_SBAR4BASE_OFFSET 0x0050
#define SNB_NTBCNTL_OFFSET 0x0058
#define SNB_SBDF_OFFSET 0x005C
#define SNB_PDOORBELL_OFFSET 0x0060
#define SNB_PDBMSK_OFFSET 0x0062
#define SNB_SDOORBELL_OFFSET 0x0064
#define SNB_SDBMSK_OFFSET 0x0066
#define SNB_USMEMMISS 0x0070
#define SNB_SPAD_OFFSET 0x0080
#define SNB_SPADSEMA4_OFFSET 0x00c0
#define SNB_WCCNTRL_OFFSET 0x00e0
#define SNB_B2B_SPAD_OFFSET 0x0100
#define SNB_B2B_DOORBELL_OFFSET 0x0140
#define SNB_B2B_XLAT_OFFSET 0x0144
#define BWD_MSIX_CNT 34
#define BWD_MAX_SPADS 16
#define BWD_MAX_COMPAT_SPADS 16
#define BWD_MAX_DB_BITS 34
#define BWD_DB_BITS_PER_VEC 1
#define BWD_PCICMD_OFFSET 0xb004
#define BWD_MBAR23_OFFSET 0xb018
#define BWD_MBAR45_OFFSET 0xb020
#define BWD_DEVCTRL_OFFSET 0xb048
#define BWD_LINK_STATUS_OFFSET 0xb052
#define BWD_SBAR2XLAT_OFFSET 0x0008
#define BWD_SBAR4XLAT_OFFSET 0x0010
#define BWD_PDOORBELL_OFFSET 0x0020
#define BWD_PDBMSK_OFFSET 0x0028
#define BWD_NTBCNTL_OFFSET 0x0060
#define BWD_EBDF_OFFSET 0x0064
#define BWD_SPAD_OFFSET 0x0080
#define BWD_SPADSEMA_OFFSET 0x00c0
#define BWD_STKYSPAD_OFFSET 0x00c4
#define BWD_PBAR2XLAT_OFFSET 0x8008
#define BWD_PBAR4XLAT_OFFSET 0x8010
#define BWD_B2B_DOORBELL_OFFSET 0x8020
#define BWD_B2B_SPAD_OFFSET 0x8080
#define BWD_B2B_SPADSEMA_OFFSET 0x80c0
#define BWD_B2B_STKYSPAD_OFFSET 0x80c4
#define NTB_CNTL_BAR23_SNOOP (1 << 2)
#define NTB_CNTL_BAR45_SNOOP (1 << 6)
#define BWD_CNTL_LINK_DOWN (1 << 16)
#define NTB_PPD_OFFSET 0x00D4
#define SNB_PPD_CONN_TYPE 0x0003
#define SNB_PPD_DEV_TYPE 0x0010
#define BWD_PPD_INIT_LINK 0x0008
#define BWD_PPD_CONN_TYPE 0x0300
#define BWD_PPD_DEV_TYPE 0x1000
#define BWD_PBAR2XLAT_USD_ADDR 0x0000004000000000
#define BWD_PBAR4XLAT_USD_ADDR 0x0000008000000000
#define BWD_MBAR23_USD_ADDR 0x000000410000000C
#define BWD_MBAR45_USD_ADDR 0x000000810000000C
#define BWD_PBAR2XLAT_DSD_ADDR 0x0000004100000000
#define BWD_PBAR4XLAT_DSD_ADDR 0x0000008100000000
#define BWD_MBAR23_DSD_ADDR 0x000000400000000C
#define BWD_MBAR45_DSD_ADDR 0x000000800000000C

1427
drivers/ntb/ntb_transport.c Normal file
View File

File diff suppressed because it is too large Load Diff

83
include/linux/ntb.h Normal file
View File

@ -0,0 +1,83 @@
/*
* This file is provided under a dual BSD/GPLv2 license. When using or
* redistributing this file, you may do so under either license.
*
* GPL LICENSE SUMMARY
*
* Copyright(c) 2012 Intel Corporation. All rights reserved.
*
* This program is free software; you can redistribute it and/or modify
* it under the terms of version 2 of the GNU General Public License as
* published by the Free Software Foundation.
*
* BSD LICENSE
*
* Copyright(c) 2012 Intel Corporation. All rights reserved.
*
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions
* are met:
*
* * Redistributions of source code must retain the above copyright
* notice, this list of conditions and the following disclaimer.
* * Redistributions in binary form must reproduce the above copy
* notice, this list of conditions and the following disclaimer in
* the documentation and/or other materials provided with the
* distribution.
* * Neither the name of Intel Corporation nor the names of its
* contributors may be used to endorse or promote products derived
* from this software without specific prior written permission.
*
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
* "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
* LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
* A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
* OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
* SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
* LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
* DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
* THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
* (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
*
* Intel PCIe NTB Linux driver
*
* Contact Information:
* Jon Mason <jon.mason@intel.com>
*/
struct ntb_transport_qp;
struct ntb_client {
struct device_driver driver;
int (*probe) (struct pci_dev *pdev);
void (*remove) (struct pci_dev *pdev);
};
int ntb_register_client(struct ntb_client *drvr);
void ntb_unregister_client(struct ntb_client *drvr);
int ntb_register_client_dev(char *device_name);
void ntb_unregister_client_dev(char *device_name);
struct ntb_queue_handlers {
void (*rx_handler) (struct ntb_transport_qp *qp, void *qp_data,
void *data, int len);
void (*tx_handler) (struct ntb_transport_qp *qp, void *qp_data,
void *data, int len);
void (*event_handler) (void *data, int status);
};
unsigned char ntb_transport_qp_num(struct ntb_transport_qp *qp);
unsigned int ntb_transport_max_size(struct ntb_transport_qp *qp);
struct ntb_transport_qp *
ntb_transport_create_queue(void *data, struct pci_dev *pdev,
const struct ntb_queue_handlers *handlers);
void ntb_transport_free_queue(struct ntb_transport_qp *qp);
int ntb_transport_rx_enqueue(struct ntb_transport_qp *qp, void *cb, void *data,
unsigned int len);
int ntb_transport_tx_enqueue(struct ntb_transport_qp *qp, void *cb, void *data,
unsigned int len);
void *ntb_transport_rx_remove(struct ntb_transport_qp *qp, unsigned int *len);
void ntb_transport_link_up(struct ntb_transport_qp *qp);
void ntb_transport_link_down(struct ntb_transport_qp *qp);
bool ntb_transport_link_query(struct ntb_transport_qp *qp);