Please don't top post, write shorter lines, and add the odd blank line.
Big blocks of text are hard to read quickly.
From: Petrosyan, Ludwig [mailto:firstname.lastname@example.org]
Yes I agree it has to be started with the write transaction, according of PCIe standard
transaction are address routed, and I agree with Logan:
if in write transaction TLP the endpoint address written in header the TLP should not
touch CPU, the
PCIe Switch has to route it to endpoint.
That depends, IIRC there is a feature for PCIe switches to force them
to send all transactions to the root hub.
This is there so that the host can enforce rules to stop p2p transfers.
It might enabled on the switch you have.
The idea was: in MTCA system there is PCIe Switch on MCH (MTCA crate
HUB) this switch connects CPU to
other Crate Slots, so one port is Upstream and others are Downstream ports, DMA read
from CPU is
usual write on endpoint side, Xilinx DMA core has two registers Destination Address and
device driver to make DMA has to set up these registers,
usually device driver to start DMA read from Board sets Source address as FPGA memory
Destination address is DMA prepared system address,
in case of testing p2p I set Destination address as physical address of other endpoint.
we have so called pcie universal driver: the idea behind is
1. all pcie configuration staff, find enabled BARs, mapping BARs, usual read/write and
(get slot number, get driver version ...) implemented in universal driver and EXPORTed.
2. if some system function in new kernel are changed we change it only in universal parts
support a big number of drivers )
so the universal driver something like PCIe Driver API
3. the universal driver provides read/writ functions so we have the same device access
API for any
PCIe device, we could use the same user application with any PCIe device
now. during BARs finding and mapping universal driver keeps pcie
endpoint physical address in some
internal structures, any top driver may get physical address
of other pcie endpoint by slot number.
in may case during get_resorce the physical address is 0xB2000000, I check lspci -H1
-vvvv -s psie
switch port bus address (the endpoint connected to this port, checked by lspci -H1 -t)
address (0xB200000) is the memory behind bridge,
I want to make p2p writes to offset 0x40000, so I set DMA destination
is something wrong?
You almost certainly need the address that is written into the BAR of the
This could well be different from the physical address that the cpu uses
to write to the endpoint (as well as the cpu virtual address).
lspci lies , run lspci -x (or hexdump config space through /sys/devices)
to see what is actually in the BAR.
 The addresses come from somewhere other than reading the BAR.
If the endpoint resets the BAR lspci will still report the old