SBUS
The SBUS is the SPI protocol for communication between the FPGA and the CPU. SBUS is also the name of the c library we include for accessing this BUS.
 | |
| sbus.c | |
| sbus.h |
Overview
While many CPUs contain some type of parallel interface that we can use for FPGA communication, the Cavium CNS2132 only has an SPI interface. It functions quite well, however some programming considerations must be made with this processor. Since SPI cannot be shared, we have implemented the SBUS which is a locking mechanism to communicate to the 128kb registers in the FPGA. We use this on all of our applications that communicate with the FPGA which includes sdctl, nandctl, xuartctl, daqctl, spiflashctl, dioctl, and ts7500ctl. The SBUS is implemented using standard kernel semaphore locks.
If you use the ctl applications as libraries or connect through the TCP servers they will abstract the use of the SBUS.
Programming Notes
- Best practice will be to lock only when you have to, and unlock as soon as you are done.
- Since the watchdog is fed through this bus, if you hold the lock for more than a few seconds the board will reboot. By default ts7500ctl will feed the watchdog at half of the timeout time.
- Try to avoid any system calls while in a lock. File IO will also be unavailable to SD, XNAND, or SPI as they are all implemented through this lock.
- Instead of locking/unlocking every iteration in a loop, call sbuspreempt which will only unlock if there is another application waiting for the lock. This is much less expensive then releasing and requesting the lock repeatedly.
- Keep in mind the mlockall() call in the sbuslock() function pulls your entire application into memory rather than paging it in as it normally would. If you are using a lot of memory, try to use one of our TCP servers which are already very tiny applications, or create another small application to handle the SBUS calls and use IPC from your main application.
Code Example
This example shows how to read the board ID from syscon.
#include "sbus.h"
#include <stdio.h>
// A program to demonstrate reading the ID register of syscon
int main(void)
{
unsigned short hexID=0;
sbuslock();
hexID = sbus_peek16(0x60); // board ID lives at 0x60 offset 0.
sbusunlock();
printf("This is a TS-%x.\n", hexID);
return 0;
}
This example demonstrates winpeek(), winpoke(), and the intended use of sbuspreempt(). Note some of the code is specific to the TS-4500 (as commented).
#include "sbus.h"
#include <stdio.h>
// The fist two defines are specific to the TS-4500.
#define TS4500_TIMING_REGISTER 0x20000
#define TIMING_VALUE 0x181
// The TS-RLY8 base address is jumper-selectable.
// The one used in testing this code was at 0x140.
#define RLY_BASEADDR 0x140
int main(void)
{
int i = 0; // loop variable
unsigned short pc104ID = 0;
int timing = 0;
// The sbus must be locked before transactions can take place.
sbuslock();
// Set timing for the PC104 bus.
timing = winpeek16(TS4500_TIMING_REGISTER);
winpoke16(TS4500_TIMING_REGISTER, TIMING_VALUE);
// Get the ID of the card at the base address.
// For the TS-RLY8 this should return 0x9b
pc104ID = winpeek8(RLY_BASEADDR);
// Loop through all possible position combinations on the TS-RLY8
for(i = 0; i < 255; i++)
{
winpoke16(RLY_BASEADDR+2,i); // Relay address space starts at base + 2.
// Avoid holding the sbus for too long by using preempt.
sbuspreempt();
usleep(20000);
}
// Return the timing register to what it was before.
winpoke16(TS4500_TIMING_REGISTER, timing);
sbusunlock();
// Output the result of the board ID check.
if(pc104ID > 0)
printf("0x%x returned from sbus_peek8(0x%x)\n",pc104ID, RLY_BASEADDR);
return 0;
}
Determine what is locking the SBUS
Linux has standard tools to interface with semaphore locks. To get a listing of all available semaphore locks, you can use the ipcs command.
ts7500:~# ipcs ------ Shared Memory Segments -------- key shmid owner perms bytes nattch status 0x75000000 0 root 0 4096 7 ------ Semaphore Arrays -------- key semid owner perms nsems 0x75000000 0 root 777 1 0x00000000 32769 www-data 600 1 ------ Message Queues -------- key msqid owner perms used-bytes messages
We use 0x75000000 as the semaphore key for the SBUS. To find what process is using this you would take the semid (0) and run
ipcs -i <semid> -s
For example:
ts7500:~# ipcs -i 0 -s
Semaphore Array semid=0
uid=0 gid=0 cuid=0 cgid=0
mode=0777, access_perms=0777
nsems = 1
otime = Wed Feb 1 23:27:56 2012
ctime = Thu Jan 1 00:00:00 1970
semnum value ncount zcount pid
0 1 0 0 99
ts7500:~# ps 99
PID TTY STAT TIME COMMAND
99 ? S 0:00 xuartctl --server
So in this case, xuartctl (pid 99) is holding the SBUS lock.
SBUS to FPGA Protocol
For the most part, the application and kernel level programmer will not be interested in the following sections as the below information is completely abstracted away from the user by the SBUS C API and the various TS-specific utilities. Even the FPGA programmer is abstracted from the details of this protocol by the spi_sbus.v Verilog module which turns the SBUS SPI traffic into WISHBONE bus cycles.
The SBUS protocol allows for 16 specific 16 bit registers (32-bytes of register space) to be read or written and has provision for the FPGA internal WISHBONE bus cycles to take any amount of time to be ack'ed through the mechanism of SBUS retries. SPI bandwidth efficient burst register reads/writes are also possible when writing/reading continuously to the same address.
The general protocol consists of a 24-bit SPI transaction framed by the SPI CS#. Bits are clocked in on the rising edge and the first MOSI (input) bit signifies whether the bus cycle is a read or write. This is followed by the 4 address bits MSB first. The remaining 19 bits depend on whether the cycle is a read or write. For reads, the WISHBONE bus cycle is started as soon as the last address bit is clocked in.
The 24-bit SPI SBUS data format - bit 23 (WE) is first on wire:
16-bit WISHBONE READ operation
23|22|21|20|19|18|17|16|15|14|13|12|11|10| 9| 8| 7| 6| 5| 4| 3| 2| 1| 0|
--|--|--|--|--|--|--|--|--|--------------------------------------------|
SI: 0|A3|A2|A1|A0| | | | B|<------------should be zero---------------->|
SO: |X2|X1|X0|MSB<----------returned READ data----------->LSB|
16-bit WISHBONE WRITE operation:
23|22|21|20|19|18|17|16|15|14|13|12|11|10| 9| 8| 7| 6| 5| 4| 3| 2| 1| 0|
--|--|--|--|--|-----------------------------------------------|--|--|--|
SI: 1|A3|A2|A1|A0|MSB<-----------WRITE data------------------>LSB|N2|N1|N0|
SO: |Y2|Y1|Y0|
* A3-A0: Address bits (sent by CPU)
* X2-X0: ack bits for reads (sent by FPGA). If any are 1, the following
data is valid. If they are all 0, the WISHBONE cycle did not complete in
time and must be immediately retried.
* Y2-Y0: ack bits for write (sent by FPGA). If any are 1, the write
cycle was completed. If all are 0, the WISHBONE write did not complete
and must be immediately retried.
* B: Burst read. Setting this to 1 starts another WISHBONE read cycle for
the same address. See "SBUS Burst mode" below.
* N2-N0: For burst writes, this represents bits 15-13 of the next burst
write data, otherwise these bits are Dont-Cares. See "SBUS Burst mode"
below.
When the WISHBONE cycle does not assert the wb_ack_i signal in time and the transaction must be retried, it is not necessary to deassert and reassert SPI CS# in between each retry, though it is necessary to start a new bus cycle.
SBUS Burst Mode
After a normal 24-bit SBUS READ/WRITE cycle takes place, it is possible to continue reading or writing to the same WISHBONE register by keeping CS# asserted and continuing the SPI clock 16 bits at a time. However, once a burst is started, there is no provision to allow for an occasional long WISHBONE cycle and all WISHBONE cycles must be able to complete in 14 SPI clock cycles otherwise results are undefined. The only registers used in TS-7500 implementation for bursts are the SPI and SD card data registers.
For burst WRITEs, one issues a normal WRITE cycle as above with possible retries making sure that the 3 LSB (last) bits sent represent the 3 MSB bits (15-13) of the next WRITE data. Subsequent writes take place 16 bits at a time while CS# is held asserted. The burst stops once CS# is deasserted.
For burst READs, one issues a normal 24-bit READ cycle and 0 or more retries, but with the "B" bit set. Once the burst is started, bit 15 must always be set as long as the read burst is to continue. On the very last READ in the burst, the 16-bit data sent by the FPGA on SO should be all zeros.
SBUS Verilog Implementation
The spi_sbus Verilog module WISHBONE clock output is synchronous to the actual SPI clock. This also means when the SPI bus becomes quiet, there are no clocks to the WISHBONE bus.
The sel_i input is used as a byte lane select. On the TS-7500, there are 2 dedicated GPIO pins from the CPU to the FPGA that are used for this. Without the sel_i inputs, it is not possible to have 8-bit reads/writes.
On the TS-7500 implementation, there are 2 extra address lines via GPIO pins on the CPU. This allows a total address space of 4 * 32 bytes == 128 bytes == 64 individual 16-bit registers. For these GPIO lines (and the byte lane selects) it is important that they be setup in advance of the SPI transaction and remain stable during it.
The spi_sbus_resync Verilog module is similar to the spi_sbus module except that it can use an asynchronous WISHBONE clock. This is what is used in the TS-7500 since the common WISHBONE clock is 75Mhz but the SPI clock can be any rate from 32.5Mhz to 50Mhz depending on CPU speed. A side effect of this though is that clock resynchronization overhead can make even the shortest of bus cycles miss the 3 SPI clock window for acknowledgment and force the majority of bus cycles into at least one retry. For this reason, a 2nd WISHBONE master interface is provided for WISHBONE slaves that are okay with using the SPI SCK as their clock -- to use this bus, spiwbm_en_i has to be asserted before the next clock edge after spiwbm_cyc_o, spiwbm_stb_o, and spiwbm_adr_o are asserted and valid.
There is another quirk about the spi_sbus_resync module because burst writes accumulate in a 16-entry write buffer clocked from the SPI clock domain. Flushing of this buffer happens automatically before any subsequent READ or WRITE transaction, but it is possible for there to be writes not yet posted to the asynchronous WISHBONE clock domain at the completion of a WRITE burst. It therefore may be important to follow up any WRITE burst with a READ to insure the last WRITE in the burst completes 100%. This becomes especially significant on the TS-7500 which includes 2 extra GPIO address bits that must remain stable while there are writes pending in the write buffer. For example, it would not be safe to allow users to change those GPIO address bits while there is still pending WRITES in the write buffer.