Broadcom 5718-PG107-R User guide
Broadcom 5718-PG107-R is an essential tool for programmers. It is designed to give users the information they need to effectively use the BCM5718 NetXtreme®/NetLink™ family of products. With this guide, users can learn about the product's features, capabilities, and使用方法. The guide provides comprehensive information on topics such as initialization procedures, send and receive rings, and power management. With its detailed explanations, tables, and figures, this guide is an invaluable resource for anyone looking to get the most out of their Broadcom 5718-PG107-R.
Advertisement
Advertisement
Programmer’s Guide
BCM5718
NetXtreme®/NetLink™ BCM5718 Family
Programmer’s Guide
5300 California Avenue • Irvine, CA 92617 • Phone: 949-926-5000 • Fax: 949-926-5203
5718-PG107-R
July 17, 2013
BCM5718 Programmer’s Guide
Revision History
Revision Date
5718-PG-107-R 07/17/13
Revision History
5718-PG-106-R 06/25/12
Change Description
Updated:
•
•
“Initialization Procedure” on page 140
•
Table 49: “GPIO Usage for Power Management for Broadcom Drivers,” on page 192
•
Table 101: “Multiple Send Ring Mail Boxes,” on page 357
•
“Send BD Ring Host Producer Index Register (offset: 0x5900)” on page 465
•
“Send BD Ring NIC Producer Index Register (offset: 0x5980)” on page 466
•
Table 121: “GbE Port Internal PHY Register Map,” on page 553
•
Table 127: “AUTONEG LINK PARTNER ABILITY,” on page 559
Added:
•
Table 124: “02h: PHY_Identifier_MSB_Register,” on page 558
•
Table 125: “03h: PHY_Identifier_LSB_Register,” on page 558
Updated:
• “Base Address Register 1 (offset: 0x10)” on page 275
• “Base Address Register 2 (offset: 0x14)” on page 275
• “Base Address Register 3 (offset: 0x18)” on page 275
• “Base Address Register 4 (offset: 0x1c)” on page 276
• “Mode Control Register (offset: 0x6800)” on page 475
Added:
• Section 8: “IEEE1588,” on page 152
• “RX TIME STAMP LSB REG [Offset 0X06B0]” on page 163
• “RX TIME STAMP MSB REG [Offset 0x06B4]” on page 163
• “RX PTP SEQUENCE ID REG [Offset 0X06B8]” on page 163
• “RX LOCK TIMER LSB REG [Offset 0x06C0]” on page 164
• “RX LOCK TIMER MSB REG [Offset 0x06C4]” on page 164
• “RX PTP CONTROL REG [Offset 0X06C8]” on page 164
• Section 12: “IO Virtualization (IOV),” on page 264
• “Perfect Match Destination Address Registers” on page 465
• “VRQ Filter Set Registers” on page 461
• “VRQ Mapper Registers” on page 462
• “Base Address Register 5 (offset: 0x20)” on page 276
• “Base Address Register 6 (offset: 0x24)” on page 277
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 2
BCM5718 Programmer’s Guide Revision History
Revision Date
5718-PG-105-R 02/24/12
Change Description
Updated:
• Table 3: “Family Revision Levels,” on page 48
• Table 5: “Flag Fields for a Ring,” on page 70
• Figure 24: “Ring Control Block,” on page 118
• “Summary of Register Settings to Support Jumbo Frames” on page 126
• “Initialization Procedure” on page 136
• “Reading a PHY Register” on page 186
• “Writing a PHY Register” on page 187
• “Subsystem ID/Vendor ID Register (offset: 0x2C)” on page 251
• “DMA Read/Write Control Register (Offset: 0x6c)” on page 258
• “PCI State Register (offset: 0x70)” on page 259
• “Receive BD Standard Producer Ring Index Register (offset: 0x268-
0x26f)” on page 281
• “Transmit MAC Status Register (offset: 0x460)” on page 296
• “Receive MAC Mode Register (offset: 0x468)” on page 297
• “Statistics Registers” on page 319
• “H2B Statistics Registers” on page 320
• “Receive Data and Receive BD Initiator Mode Register (offset: 0x2400)” on page 341
• “Link Speed 10 MB/No Link Power Mode Clock Policy Register (offset:
0x3604)” on page 356
• “Link Speed 100 MB Power Mode Clock Policy Register (offset: 0x3608)” on page 357
• “Link Aware Power Mode Clock Policy Register (offset: 0x3610)” on page 359
• “D0u Clock Policy Register (offset: 0x3614)” on page 360
• “Link Idle Power Mode Clock Policy Register (offset: 0x3618)” on page 360
• “APE CLK Policy Register (offset: 0x361C)” on page 361
• “APE Sleep State Clock Policy Register (offset: 0x3620)” on page 363
• “Clock Speed Override Policy Register (offset: 0x3624)” on page 364
• “Clock Status Register (offset: 0x3630)” on page 367
• “Padring Control Register (offset: 0x3668)” on page 376
• “Receive Coalescing Ticks Register (offset: 0x3C08)” on page 393
• “Send Coalescing Ticks Register (offset: 0x3C0C)” on page 394
• “Receive Max Coalesced BD Count Register (offset: 0x3C10)” on page 395
• “Send Max Coalesced BD Count Register (offset: 0x3C14)” on page 397
• “Status Block Host Address Register (offset: 0x3C38)” on page 400
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 3
BCM5718 Programmer’s Guide
Revision Date
Revision History
Change Description
• “Status Block Base Address Register (offset: 0x3C44)” on page 426
• “BM Hardware Diagnostic 2 Register (offset: 0x4450)” on page 436
• “LSO Read DMA Mode Register (offset: 0x4800)” on page 438
• “LSO Read DMA Reserved Control Register (offset: 0x4900)” on page 445
• “LSO Read DMA Flow Reserved Control Register (offset: 0x4904)” on page 446
• “LSO/Non-LSO/BD Read DMA Corruption Enable Control Register
(offset: 0x4910)” on page 446
• “BD Read DMA Mode Register (Offset: 0x4A00)” on page 449
• “BD READ DMA Reserved Control Register (offset: 0x4A70)” on page 456
• “BD READ DMA Flow Reserved Control Register (offset: 0x4A74)” on page 457
• “BD READ DMA Corruption Enable Control Register (offset: 0x4A78)” on page 457
• “Non_LSO Read DMA Mode Register (offset: 0x4B00)” on page 458
• “Non-LSO Read DMA Reserved Control Register (offset: 0x4B74)” on page 461
• “Non-LSO Read DMA Corruption Enable Control Register (offset:
0X4B7C)” on page 462
• “Write DMA Mode Register (offset: 0x4C00)” on page 464
• “Low Priority Mailboxes” on page 469
• “Interrupt Mailbox 0 Register (offset: 0x5800)” on page 469
• “Other Interrupt Mailbox Register (offset: 0x5808–0x5818)” on page 469
• “General Mailbox Registers 1-8 (offset: 0x5820–0x5824)” on page 469
• “Receive BD Standard Producer Ring Index Register (offset: 0x5868)” on page 470
• “Receive BD Return Ring 0 Consumer Index Register (offset: 0x5880-
0x5887)” on page 470
• “Receive BD Return Ring 0 Consumer Index Register (offset: 0x5880-
0x5887)” on page 470
• “Send BD Ring Consumer Index Register (offset: 0x5900)” on page 471
• “NVM Write Register (offset: 0x7008)” on page 497
• “NVM Address Register (offset: 0x700C)” on page 497
• “NVM Read Register (offset: 0x7010)” on page 498
• “NVM Config 1 Register (offset: 0x7014)” on page 498
• “NVM Access Register (offset: 0x7024)” on page 502
• “00h: MII_Control_Register” on page 513
• “03h: PHY_Identifier_LSB_Register” on page 515
• “04h: Auto_Negot_Advertisement_Register” on page 515
• “09h: 1000Base_T_Control_Register” on page 518
• “10h: PHY_Extended_Control_Register” on page 522
• “18h: Auxiliary Control Register (Shadow Register Selector = “000”)” on page 526
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 4
BCM5718 Programmer’s Guide
Revision Date
Revision History
Change Description
• “18h: Miscellaneous Control Register (Shadow Register Selector =
“111”)” on page 533
• “1Ch: Cabletron LED Register (Shadow Register Selector = “00h”)” on page 538
• “1Ch: Spare Control 4 Register (Shadow Register Selector = “0bh”)” on page 547
• “1Ch: External Serdes Control Register (Shadow Register Selector =
“14h”)” on page 557
• “1Ch: SGMII Slave Register (Shadow Register Selector = “15h”)” on page 559
• “1Ch: Misc 1000-X Control 2 Register (Shadow Register Selector =
“16h”)” on page 561
• “1Ch: Misc 1000-X Control Register (Shadow Register Selector = “17h”)” on page 563
• “1Ch: Auto-Detect SGMII/GBIC Register (Shadow Register Selector =
“18h”)” on page 564
• “1Ch: Auto-Detect Medium Register (Shadow Register Selector =
“1eh”)” on page 572
• “1Ch: Mode Control Register (Shadow Register Selector = “1fh”)” on page 573
Added:
• Table 1: “Register Access Methods,” on page 46
• “Device Reset Procedure” on page 146
• “PHY Loopback Configuration” on page 205
• “PHY Configuration Auto-Negotiation (10/100/1000 Speed with Half and Full Duplex Support)” on page 206
• “MSI-X Capabilities Registers” on page 281
• “PCIe Capabilities Registers” on page 282
• “VRQ Flush Control Register (Offset: 0x2410)” on page 369
• “VRQ Flush Timer Register (offset: 0x2414)” on page 370
• “RDI B2HRX Hardware Debugging Register (offset: 0x2418)” on page 370
• “Receive BD Ring Initiator Local NIC Standard Receive BD Consumer
Index (offset: 0x2474)” on page 373
• “B2HRX Byte-count Statistics Count (offset: 0x24D0)” on page 374
• “B2HRX Unicast Statistics Count (offset: 0x24D4)” on page 374
• “B2HRX Multicast Statistics Count (offset: 0x24D8)” on page 374
• “B2HRX Broadcast Statistics Count (offset: 0x24DC))” on page 374
• “B2HRX Drop Packet Count (offset: 0x24E0)” on page 374
• “B2HRX Drop Packet Byte Count (offset: 0x24E4)” on page 374
• “B2HRX APE Byte-count Statistics Count (offset: 0x24E8)” on page 375
• “B2HRX APE Unicast Statistics Count (offset: 0x24EC)” on page 375
• “B2HRX APE Multicast Statistics Count (offset: 0x24F0)” on page 375
• “B2HRX APE Broadcast Statistics Count (offset: 0x24F4)” on page 375
• “B2HRX APE Drop Packet Count (offset: 0x24F8)” on page 375
• “B2HRX APE Drop Packet Byte Count (offset: 0x24FC)” on page 375
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 5
BCM5718 Programmer’s Guide Revision History
Revision Date
5718-PG104-R 06/29/11
Change Description
• “Receive Max Coalesced BD Count During Interrupt Register (offset:
0x3C18)” on page 424
• “Send Max Coalesced BD Count During Interrupt Register (offset:
0x3C1C)” on page 425
• “NIC Mini Receive BD Consumer Index (offset: 0x3c58)” on page 428
• “Send BD Ring Producer Index Register (offset: 0x5980)” on page 471
• “DMA Completion Mode Register (Offset: 0x6400)” on page 477
• Figure 58: “Copper PHY Register Mapping Table,” on page 511
• Figure 59: “Serdes PHY Register Map,” on page 512
• “Clause 45 Registers” on page 601
• “SerDes PHY Register Definitions” on page 578
• “PHY 0x18 Shadow 0x1 register read Procedure” on page 527
• Added PHY 0x1C Shadow 0x1 register read Procedure information to
“1Ch: Cabletron LED Register (Shadow Register Selector = “00h”)” on page 538
• Added Clause 45 register Dev3 Reg803Eh read Procedure to “Clause 45
Register Dev 3 Reg14h (20d): EEE Capability Register” on page 601
• NIC Ring Addresses information to Memory map tables in
Appendix C: “Device Register and Memory Map,” on page 611
Deleted
• Section 11: Host to/from BMC Pass Through
• Appendix D: Appendix
• Top Level MII Registers
Updated:
• Table 27: “Flag Field Description,” on page 113
• Table 31: “Send Buffer Descriptor Flags,” on page 123
• “Clock Control” on page 191
• Table 47: “Ethernet Controller Power Pins,” on page 191
• “Internal Memory” on page 214
• “ISR Flow” on page 230
• Table 82: “Interrupt-Related Registers,” on page 235
• “Status Register (offset: 0x362C)” on page 391
• “Clock Status Register (offset: 0x3630)” on page 393
• “LSO Read DMA Mode Register (offset: 0x4800)” on page 438
• “NVM Write Register (offset: 0x7008)” on page 497
Added:
• “ Device Closing Procedure” on page 147
• “TX TIME STAMP LSB REG (offset: 0x5C0)” on page 327
• “TX TIME STAMP MSB REG (offset: 0x5C4)” on page 327
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 6
BCM5718 Programmer’s Guide Revision History
Revision
5718-PG103-R
Date
01/26/11
Change Description
Updated:
• Added BCM5720 to Section 1: “Introduction,” on page 49.
• Added BCM5720 to “Introduction” on page 49.
• Added Host to BMC to “Transmit MAC Mode Register (offset: 0x45C)” on page 317.
• Added Host to BMC to “Transmit MAC Lengths Register (offset: 0x464)” on page 319.
• Added Host to BMC to “Mode Control Register (offset: 0x6800)” on page 477.
Added
• “HTX2B Perfect Match[1 – 4] HI Reg (offset: 0x4880, 0x4888, 0x4890,
0x4898)” on page 330.
• “HTX2B Perfect Match[1 – 4] LO Reg (offset: 0x4884, 0x488C, 0x4894,
0x489C)” on page 330.
• “HTX2B Protocol Filter Reg (offset: 0x6D0)” on page 331.
• “HTX2B Global Filter Reg (address: 0x6D4)” on page 333.
• “H2B Statistics Registers” on page 346.
• “HTX2B Statistics” on page 347
• “B2HRX Statistics” on page 347
• “RMU Registers” on page 504
• “RMU_EGRESS_DA1_MATCH[1-8]_REG (offsets: 0x00B0, 0x00B8,
0x00C0, 0x00C8 … 0xE8)” on page 504
• “RMU_EGRESS_DA2_MATCH[1-8]_REG (Offsets 0x00B4, 0x00BC,
0x00C4, 0xCC …0xEC)” on page 504
• “RMU_EGRESS_STATUS_REG (Offset 0x0000)” on page 504
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 7
BCM5718 Programmer’s Guide Revision History
Revision
5718-PG102-R
Date
12/16/10
Change Description
Updated:
• Added BCM5719 to “Introduction” on page 39.
• Added BCM5719 to “Related Documents” on page 39.
• Added BCM5719 to Table 1: “BCM5718 Family Product Features,” on page 40.
• Removed PHY core column and added BCM5717 B0, BCM5718 BO, and
BCM5719 to Table 2: “Family Revision Levels,” on page 42.
• Updated note in “Revision Levels” on page 42.
• Added Memory Arbiter to Figure 1: “Individual Port Functional Block
Diagram,” on page 45.
• Added BCM5719 to “Overview of Features” on page 46.
• Added note about BCM5719 to Figure 2: “High-Level System Functional
Block Diagram,” on page 47.
• Added max ring sizes to “Ring Control Block” on page 99.
• Added BCM5719 toTable 6: “Defined Flags for Send Buffer Descriptors,” on page 67.
• Updated Host Ring Size to Table 7: “Receive Return Rings,” on page 70.
• Corrected typo in Figure 26: “Send Driver Interface,” on page 119.
• Corrected typo in Figure 27: “Receive Producer Interface,” on page 120.
• Corrected typo in Figure 28: “Receive Return Interface,” on page 121.
• Updated Step 36 in “Initialization Procedure” on page 137.
• Added BCM5719 to “Description” on page 168.
• Corrected typo in “PCI Classcode and Revision ID Register (offset:
0x08) — Function 0” on page 255
• Corrected typos in “Power Management Control/Status Register
(offset: 0x4C) — Function 0” on page 261
• Added note to Enable Endian Byte Swap in “Miscellaneous Host Control
Register (offset: 0x68)” on page 264.
• Updated all Indirection Table register descriptions in “RSS Registers” on page 276.
• Added BCM5717 and BCM5718 values to “CPMU Control Register
(offset: 0x3600)” on page 346.
• Added BCM5719 to “Link Aware Power Mode Clock Policy Register
(offset: 0x3610)” on page 349.
• Added BCM5719 to “APE CLK Policy Register (offset: 0x361C)” on page 352.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 8
BCM5718 Programmer’s Guide
Revision Date
Revision History
Change Description
Updated (continued):
• Added BCM5718 to “Clock Speed Override Policy Register (offset:
0x3624) for BCM5718” on page 354
• Added BCM5718 to “Clock Status Register (offset: 0x3630)” on page 358
• Added Reserved for BCM5719 to “PCIE Status Register (offset: 0x3634)” on page 359
• Added BCM5719 to “GPHY Control/Status Register (offset: 0x3638)” on page 360
• Updated introduction to “PCIE Idle Detection De-Bounce Control
Register (offset: 0x364C)” on page 362
• Corrected typo and added BCM5719 to “DLL Lock Timer Register (offset:
0x3654)” on page 364
• Updated Chip ID default value in “CHIP ID Register (offset: 0x3658)” on page 365
• Added BCM5719 to “Padring Control Register (offset: 0x3668)” on page 367
• Added BCM5719 to “Reserved (offset: 0x366C)” on page 368
• Added BCM5719 to “Reserved (offset: 0x367C)” on page 373
• Added BCM5719 to “Read DMA Mode Register (offset: 0x4800)” on page 398
• “LSO Read DMA Corruption Enable Control Register (offset: 0x4910)” on page 413
• Added BCM5719 to “Write DMA Mode Register (offset: 0x4C00)” on page 443
• Added BCM5719 to “MSI Mode Register (offset: 0x6000)” on page 454
Added:
• “Receive BD Standard Producer Ring Index (High Priority Mailbox)
Register (offset: 0x268-0x26f)” on page 276
• “TX Time Stamp LSB Reg (offset: 0x5C0)” on page 282
• “TX Time Stamp MSB Reg (offset: 0x5C4)” on page 283
• “RX Time Stamp LSB Reg (offset 0x06B0)” on page 308
• “RX Time Stamp MSB Reg (offset 0x06B4)” on page 308
• “RX PTP Sequence ID Reg (offset 0x06B8)” on page 308
• “RX Lock Timer LSB Reg (offset 0x6C0)” on page 309
• “RX Lock Timer MSB Reg (offset 0x06C4)” on page 309
• “RX PTP Control Reg (offset: 0x6C8)” on page 310
• “Clock Speed Override Policy Register (offset: 0x3624)” on page 355
• “Clock Status Register (offset: 0x3630)” on page 358
• “Global Mutex Request Register (offset: 0x36F0)” on page 381
• “Global Mutex Grant Register (offset: 0x36F4)” on page 381
• “Temperature Monitor Control Register (offset: 0x36FC)” on page 382
• “BCM5719 Registers” on page 469
Removed:
• “Reserved (offset: 0x378C)”
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 9
Revision
5718-PG101-R
Date
11/12/10
5718-PG100-R 04/13/10
Change Description
Updated:
• Table 1: “BCM5718 Family Product Features,” on page 40
• IP cksum description in “Receive Buffer Descriptors” on page 70
• “Extended RX Buffer Descriptor (BD)” on page 110
• Table title for Table 34: “Jumbo Producer Ring Host Address Low
Register (offset: 0x2444),” on page 123
• Default value and description in Table 36: “Jumbo Producer Ring NIC
Address Register (offset: 0x244C),” on page 123
• NIC ring address values in Table 45: “NIC Ring Addresses,” on page 126
• PCI version in “Description” on page 148
• “EMAC Status Register (offset: 0x404)” on page 279
• “DMA Flag Register for TCP Segmentation (offset: 0xCEC)” on page 321
• “Jumbo Producer Ring NIC Address Register (offset: 0x244C)” on page 337
• “Receive Producer Length/Flags Register (offset: 0x2458)” on page 337
• “Receive Producer Ring NIC Address Register (offset: 0x245C)” on page 338
• “GPHY Strap Register (offset: 0x3664)” on page 366
• “Read DMA Mode Register (offset: 0x4800)” on page 398
• “BCM5718 Family MII Bus PHY Addressing” on page 496
Added:
• Section 8: “Device Control,” on page 137
• Registers 0x00 to 0x3c and 0x48 to 0x64 to “PCI Configuration Registers” on page 254
• Section 14: “Transceiver Registers,” on page 496
Removed:
• References to BCM5724 throughout
• Column from Table 1: “BCM5718 Family Product Features,” on page 40
• Register control mode from “MDI Register Access” on page 187
• MDI Control Register (offset: 0x6844)
Initial release
Broadcom Corporation
5300 California Avenue
Irvine, CA 92617
© 2013 by Broadcom Corporation
All rights reserved
Printed in the U.S.A.
Broadcom®, the pulse logo, Connecting everything®, and the Connecting everything logo are among the trademarks of Broadcom Corporation and/or its affiliates in the United States, certain other countries and/ or the EU. Any other trademarks or trade names mentioned are the property of their respective owners.
BCM5718 Programmer’s Guide Table of Contents
Table of Contents
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 11
BCM5718 Programmer’s Guide Table of Contents
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 12
BCM5718 Programmer’s Guide Table of Contents
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 13
BCM5718 Programmer’s Guide Table of Contents
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 14
BCM5718 Programmer’s Guide Table of Contents
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 15
BCM5718 Programmer’s Guide Table of Contents
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 16
BCM5718 Programmer’s Guide Table of Contents
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 17
BCM5718 Programmer’s Guide Table of Contents
PHY Configuration Auto-Negotiation (10/100/1000 Speed with Half and Full Duplex Support) .207
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 18
BCM5718 Programmer’s Guide Table of Contents
Receive Max Coalesced BD Count During Interrupt Register (Offset 0x3c18) ......................256
Send Max Coalesced BD Count During Interrupt Register (Offset 0x3c1c) ...........................256
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 19
BCM5718 Programmer’s Guide Table of Contents
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 20
BCM5718 Programmer’s Guide Table of Contents
UNDI Receive BD Standard Producer Ring Producer Index Mailbox Register (offset: 0x98–0x9C) .....288
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 21
BCM5718 Programmer’s Guide Table of Contents
Interrupt mail box (High Priority Mailbox) Register
General mail box (High Priority Mailbox)
Reload Statistics mail box (High Priority Mailbox)
Receive BD Standard Producer Ring Index
Receive BD Return Ring 0 Consumer Index
Receive BD Return Ring 1 Consumer Index
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 22
BCM5718 Programmer’s Guide Table of Contents
Receive BD Return Ring 2 Consumer Index
Receive BD Return Ring 3 Consumer Index
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 23
BCM5718 Programmer’s Guide Table of Contents
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 24
BCM5718 Programmer’s Guide Table of Contents
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 25
BCM5718 Programmer’s Guide Table of Contents
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 26
BCM5718 Programmer’s Guide Table of Contents
Receive Diagnostic Data and Receive BD Ring Initiator Local NIC Jumbo Receive BD Consumer Index
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 27
BCM5718 Programmer’s Guide Table of Contents
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 28
BCM5718 Programmer’s Guide Table of Contents
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 29
BCM5718 Programmer’s Guide Table of Contents
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 30
BCM5718 Programmer’s Guide Table of Contents
Non-LSO Read DMA Corruption Enable Control Register
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 31
BCM5718 Programmer’s Guide Table of Contents
VRQ_PERFECT_MATCH[4 - 23]_HIGH_REG (Offsets: 0x5690, 0x5698, 0x56A0 … 0x5728) .................462
VRQ_PERFECT_MATCH[4 - 23]_LOW_REG (Offsets: 0x5694, 0x569C, 0x56A4 … 0x572C)..................463
Receive BD Return Ring 0 Consumer Index Register
Receive BD Return Ring 1 Consumer Index Register
Receive BD Return Ring 2 Consumer Index Register
Receive BD Return Ring 3 Consumer Index Register
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 32
BCM5718 Programmer’s Guide Table of Contents
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 33
BCM5718 Programmer’s Guide Table of Contents
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 34
BCM5718 Programmer’s Guide Table of Contents
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 35
BCM5718 Programmer’s Guide
BROADCOM
®
July 17, 2013 • 5718-PG107-R
Table of Contents
Page 36
BCM5718 Programmer’s Guide Table of Contents
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 37
BCM5718 Programmer’s Guide List of Figures
List of Figures
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 38
BCM5718 Programmer’s Guide List of Figures
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 39
BCM5718 Programmer’s Guide List of Tables
List of Tables
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 40
BCM5718 Programmer’s Guide List of Tables
Table 39: Recommended BCM57XX Ethernet controller Low Watermark Maximum Receive Frames Settings .
141
Table 42: Recommended BCM57XX Ethernet Controller Max Coalesced Frames During Interrupt Counter
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 41
BCM5718 Programmer’s Guide List of Tables
≤ 4] Formats (MSI-X Multivector RSS Mode) .................................................249
≥ N ≤ 16] Format (MSI-X Multivector IOV Mode) .................................................250
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 42
BCM5718 Programmer’s Guide List of Tables
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 43
BCM5718 Programmer’s Guide List of Tables
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 44
BCM5718 Programmer’s Guide
About This Document
Purpose and Audience
This document covers the BCM5718 family of NetXtreme®/NetLink® Ethernet controllers. This family of controllers includes the following devices:
• BCM5717
• BCM5718
• BCM5719
• BCM5720
The document focuses on the registers, control blocks, and software interfaces necessary for host software programming. It is intended to complement the data sheet for the appropriate member of the NetXtreme/
NetLink Ethernet controller family. The errata documentation (see
“Revision Levels” on page 50 ) complements
this document.
Acronyms and Abbreviations
In most cases, acronyms and abbreviations are defined on first use.
For a comprehensive list of acronyms and other terms used in Broadcom documents, go to: http://www.broadcom.com/press/glossary.php
.
Document Conventions
The following conventions may be used in this document:
Convention
Bold
Monospace
< >
[ ]
Description
User input and actions: for example, type exit, click OK, press Alt+C
Code: #include <iostream>
HTML: <td rowspan = 3>
Command line commands and parameters: wl [-l] <command>
Placeholders for required elements: enter your <username> or wl <command>
Indicates optional command-line parameters: wl [-l]
Indicates bit and byte ranges (inclusive): [0:3] or [7:0]
Convention
A/C
RO
RW
Table 1: Register Access Methods
Description
Clear contents after read
Read-Only access, Write has no effect
Full Read and Write access
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 45
BCM5718 Programmer’s Guide
Convention
WO
W1C
W1S
WZO
RW1C
RW1S
RW/S
Table 1: Register Access Methods (Cont.)
Description
Write-Only access, Read returns garbage
Write a value 1 to clear the bit.
Write a value 1 to set the bit
Write 0 only to avoid side effects, Read returns garbage
Readable / Write a 1’b1 value to clear the bit.
Readable / Write a 1’b1 to set the bit
Readable / Writeable but is not affected by any CPU reset or AHB reset.
References
The references in this section may be used in conjunction with this document.
Note: Broadcom provides customer access to technical documentation and software through its
Customer Support Portal (CSP) and Downloads & Support site (see Technical Support ).
For Broadcom documents, replace the “xx” in the document number with the largest number available in the repository to ensure that you have the most current version of the document.
Document (or Item) Name Number Source
Broadcom Items
[1] BCM57XX NetXtreme® Programmer’s Guide:
Programming details for the BCM5700,
BCM5701, BCM5702, BCM5703, BCM5704,
BCM5705, BCM5721, BCM5751, BCM5752,
BCM5714, BCM5715, and BCM57XX devices
[2] x2 PCI Express Dual-Port Gigabit Ethernet
Controller
[3] x2 PCI Express Dual-Port Gigabit Ethernet
Controller
[4] x4 PCI Express® Quad-Port Dual-Media Gigabit
Ethernet Controller
[5] x2 PCI Express® Dual-Port Dual-Media Gigabit
Ethernet Controller
[6] x2 PCI Express® Dual-Port Gigabit Ethernet
Controller Errata (A0 and B0)
[7] BCM5718 Revision A0, B0
[8] BCM5719 A0 Errata
[9] BCM5720 Errata
[10] Self Boot Option
57XX-PG1XX-R
5718-DS0X-R
5719-DS0X-R
5717-ES10X-R
Broadcom CSP
Broadcom CSP
Broadcom CSP
Broadcom CSP
5718-ES10X-R
5719-ES10X-R
Broadcom CSP
Broadcom CSP
5720-ES10X-R Broadcom CSP
5754X_5787X-AN10X-R Broadcom CSP
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 46
BCM5718 Programmer’s Guide
Document (or Item) Name (Cont.)
[11] NetXtreme/NetLink Software Self-Boot NVRAM NetXtreme-AN40X-R
[12] NetXtreme®/NetLink® NVRAM Access NetXtreme-AN50X-R
NetXtreme-AN60X-R [13] NetXtreme/NetLink NVRAM Configuration
Options
[14] NetXtreme/NetLink Shared Memory
Communication
NetXtreme-AN80X-R
Other Items
Number
[15] NC-SI (Network Controller–Sideband Interface)
Specification
–
Source
Broadcom CSP
Broadcom CSP
Broadcom CSP
Broadcom CSP
(http://www.dmtf.org/home)
Technical Support
Broadcom provides customer access to a wide range of information, including technical documentation, schematic diagrams, product bill of materials, PCB layout information, and software updates through its customer support portal ( https://support.broadcom.com
). For a CSP account, contact your Sales or Engineering support representative.
In addition, Broadcom provides other product support through its Downloads & Support site
( http://www.broadcom.com/support/ ).
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 47
BCM5718 Programmer’s Guide Introduction
Section 1: Introduction
The NetXtreme and NetLink family of Media Access Controller (MAC) devices are highly-integrated, single-chip gigabit Ethernet LAN controller solutions for high-performance network applications. These devices integrate the following major functions to provide a single-chip solution for gigabit LAN-on-motherboard (LOM) and network interface card (NIC) applications.
• Triple-speed IEEE 802.3-compliant MAC functionality
• Triple-speed IEEE 802.3-compliant Ethernet PHY transceiver
• PCI Express® (PCIe™) bus interface
• On-chip packet buffer memory
• On-chip RISC processor for custom frame processing
Product Features
Feature
Table 2: BCM5718 Family Product Features
BCM5717
Dual-Port
Copper
Data Management
VLAN tag support (IEEE 802.1Q) Yes
Layer 2 priority encoding (IEEE 802.1p) Yes
Link aggregation (IEEE 802.3ad) Yes
Full-duplex flow control (IEEE 802.3x) Yes
Programmable rules checker for advanced packet filtering and classification
Frame/packet buffer memory
Yes
32 KB RX,
29 KB TX
Yes TCP checksum offload (hardware based) on Tx/Rx over IPv4/IPv6
UDP checksum offload (hardware based) on Tx/Rx over IPv4/IPv6
IP checksum offload on Tx/Rx over
IPv4/IPv6
Hardware TCP segmentation offload over IPv4/IPv6
Yes
Yes
Yes
Jumbo frame support
Receive-side scaling (RSS)
Yes
Yes
UDP Receive-side scaling (UDP RSS) No
BCM5718
Dual-Port
Copper/Serdes
Yes
Yes
Yes
Yes
Yes
32 KB RX,
29 KB TX
Yes
Yes
Yes
Yes
Yes
Yes
No
BCM5719
Quad-Port
Copper/Serdes
Yes
Yes
Yes
Yes
Yes
40 KB RX,
29 KB TX
Yes
Yes
Yes
Yes
Yes
Yes
Yes
BCM5720
Dual-Port
Copper/Serdes
Yes
Yes
Yes
Yes
Yes
40 KB RX,
29 KB TX
Yes
Yes
Yes
Yes
Yes
Yes
Yes
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 48
BCM5718 Programmer’s Guide Product Features
Table 2: BCM5718 Family Product Features (Cont.)
Feature
BCM5717
Dual-Port
Copper
BCM5718
Dual-Port
Copper/Serdes
BCM5719
Quad-Port
Copper/Serdes
BCM5720
Dual-Port
Copper/Serdes
Transmit-side scaling (TSS) Yes
Multiple receive descriptor queues Yes
IOV support (I/O Virtualization) for:
• VMWare® NetQueue
• Microsoft® Virtual Machine Queue
(VMQ)
No
MSI
MSI-X
Function Level Reset
Scatter/gather bus mastering architecture
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes Yes Yes Yes
Yes (5 Vectors) Yes (17 Vectors) Yes (17 Vectors) Yes (17 Vectors)
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Statistics for SNMP MIB II, Ethernet like
MIB, Ethernet MIB (IEEE 802.3z, Clause
30)
Yes Yes Yes Yes
Teaming
No No No No
Yes Yes Yes Yes
Yes Yes Yes Yes
Host Bus Interfaces
PCIe 2.1 x4
LAN Interfaces
Yes Yes Yes Yes
No No Yes No
Integrated Yes Yes Yes Yes
Other Bus Interfaces
NC-SI (version 1.0.0a)
Host 2 BMC
SMBus 2.0 interface
Interface to Flash memory
Interface to Serial EEPROM
Self-Test
Test modes (BIST, SCAN, etc)
JTAG support
Technology
High-performance, low-overhead software/hardware interface
Yes
No
Yes
Yes
Yes
Yes
Yes
Yes
No
Yes
Yes
Yes
Yes
Yes
Yes
No
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes
Yes Yes Yes Yes
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 49
BCM5718 Programmer’s Guide Revision Levels
Table 2: BCM5718 Family Product Features (Cont.)
Feature
High-speed on-chip RISC processors
(one per port)
High-speed on-chip Application
Processor Engine (APE)
Wake-on-LAN (WOL)
Energy Efficient Ethernet (EEE)
Ethernet Audio Video (EAV)
Secure Digital (SD) Card Reader
IEEE 1588 / IEEE 802.1AS Timestamp
Support
On-chip temperature monitor
Integrated Trusted Platform Module
(TPM) Security Engine
Process voltage
CMOS linewidth
Yes
Yes
No
No
No
No
BCM5717
Dual-Port
Copper
Yes
No
No
1.2V
65 nm
BCM5718
Dual-Port
Copper/Serdes
Yes
Yes
Yes
Yes
No
No
No
No
No
1.2V
65 nm
BCM5719
Quad-Port
Copper/Serdes
Yes
Yes
Yes
Yes
No
No
Yes
Yes
No
1.2V
65 nm
BCM5720
Dual-Port
Copper/Serdes
Yes
Yes
Yes
Yes
No
No
Yes
Yes
No
1.2V
65 nm
Revision Levels
for the revision levels of the Ethernet controllers covered by this document. Host software can use the PCI Revision ID and Chip ID information in the PCI configuration registers to determine the revision level of the Ethernet controller on the board, and then load the appropriate workaround described in the errata sheets.
The Broadcom PCI vendor ID is 0x14E4.
Table 3 shows the default values of PCI device IDs. These values may
be modified by firmware in accordance with the manufacturing information supplied in NVRAM (see
Configuration” on page 68 for more details).
Family Member Device ID a
Table 3: Family Revision Levels
Revision
Level
PCI Revision
ID b
Chip ID c
Product ASIC ID d
Errata Sheet e
5717-ES1xx-R
BCM5719
BCM5719
0x1657
0x1657
A0
A1
0x10
0x00
0x01
0xF100xxxx 0X5717100
0xF000xxxx 0X5719000
0XF100XXXX 0X5719100
5717-ES1xx-R
5718-ES1xx-R
5718-ES1xx-R
5719-ES1xx-R
5719-ES1XX-R
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 50
BCM5718 Programmer’s Guide Programming the Ethernet Controllers
Table 3: Family Revision Levels (Cont.)
Family Member Device ID a
BCM5720 0X165F
Revision
Level
A0
PCI Revision
ID b
0x00
Chip ID c
Product ASIC ID
0xF000xxxx 0X5720000 d
Errata Sheet
5720-ES1xx-R e a. See Device ID and Vendor ID Register (Offset: 0x00) — Function 0”, per PCI specification.
b. See “PCI Classcode and Revision ID Register (offset: 0x8) — Function 0” as per the PCI specification. The hardware default value of this register is 0x00. The boot code firmware programs this register with the value as given in the table. c. See
determining chip id.
d. See
“Product ASIC ID (offset: 0xF4)” on page 301 or determining ASIC ID.
e. See the appropriate errata documentation for the errata information and resolutions.
Note: If you are using silicon revision A0 of the BCM5718, you must load a “patch” image
(ap5718.012) into the boot code NVRAM in addition to the usual legacy boot code image common to all NetXtreme/NetLink controllers. This extra patch image, which masquerades as “Management
Firmware”, works around an issue with A0 silicon in which the device may fail to load reliably and execute the primary boot code image. This issue is fixed in BCM5718 B0 and is not an issue in
BCM5719 or BCM5720.
Programming the Ethernet Controllers
ID and Chip ID information in the PCI configuration registers to determine the revision level of the Ethernet controller on the board, and then load the appropriate workarounds described in the errata sheets.
Choice of host access mode determines the mailboxes:
• Host standard mode uses the high-priority mailboxes (see
“Mailbox Registers” on page 93 ).
• Indirect mode uses the low-priority mailboxes (see
“Mailbox Registers” on page 93 ).
The reference documents for Ethernet controller software development include this manual and the errata
based device driver. The Broadcom Linux® driver (a.k.a. “tg3”) is also a very good reference source for writing your own driver.
The programming model for the NetXtreme/NetLink Ethernet controllers does not depend on OS or processor instruction sets. Programmers using Motorola® 68000, Intel® x86, or DEC Alpha host instruction sets can leverage this document to aid in device driver development. Concepts provided in this document are also applicable to device drivers native to any operating system (i.e., DOS, UNIX®, Microsoft®, or Novell®).
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 51
BCM5718 Programmer’s Guide Hardware Architecture
Section 2: Hardware Architecture
Theory of Operation
shows the major functional blocks and interfaces of the Ethernet controllers covered in this document.
Only a single port is illustrated in
Figure 1 . The dual-port controllers in this family of controllers essentially
replicate a second instance of the major areas of functionality shown in the diagram below. The dual-port controllers have only a single PCIe and NVRAM interfaces.
There are two packet flows: MAC-transmit and receive. The device’s DMA engine bus-masters packets from host memory to device local storage, and vice-versa. The host bus interface is compliant with PCIe standards.
The RX MAC moves packets from the integrated PHY into device internal memory. All incoming packets are checked against a set of QOS rules and then categorized. When a packet is transmitted, the TX MAC moves data from device internal memory to the PHY. Both flows operate independently of each other in full-duplex mode.
An on-chip RISC processor is provided for running value-added firmware that can be used for custom frame processing. The on-chip RISC operates independently of all the architectural blocks; essentially, RISC is available for the auxiliary processing of data streams.
Figure 1: Individual Port Functional Block Diagram
Receive
GMII
Receive
MAC
Transmit
GMII
Rx
FIFO
Rx Frame Buffer
Memory/RISC Scratch
Pad Memory
Tx Frame Buffer
Memory
Queue
Memory
Physical Layer
Transceiver Statistics
Transmit
MAC
Tx
FIFO
Rule
Check
Memory
Arbiter
Registers
Frame Buffer
Manager
Send BD RING
Receive BD RING
DMA Descriptor
Config
Read DMA
Read
FIFO
PCIe
PCIe Bus
RISC
Processor
Boot ROM
Write
FIFO
Applications
Processing
Engine
(APE runs firmware such as NC-SI)
PLL
125-MHz Clock
LED Control
Write
DMA
Ring Controllers
Host Coalescing
Queue Management
LED Signals
EEPROM Control
NVRAM
Interface
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 52
BCM5718 Programmer’s Guide Overview of Features
Overview of Features
The BCM5718 family of controllers represents the third generation of Broadcom NetXtreme multi-port gigabit
Ethernet controllers. This family is the successor to the BCM5714/BCM5715 family. The BCM5717, BCM5718, and BCM5720 feature two independent 1 Gb Ethernet ports on the network side. A host computer can communicate with the controller over a single PCIe link. However, the two network controller ports appear as two independent PCIe functions to the host operating system (four functions in the quad-port BCM5719). In essence, the chip consists of a single PCIe interface controller that offers multiple function-level interfaces, and each function-level interface is further attached to an independent DMA engine which in turn feeds an independent Ethernet Media Access Controller (EMAC).
Attached to the other end of each EMAC is the respective 802.3 Ethernet physical media interface. The controller essentially consists of multiple DMA+EMAC logic instances, multiple Ethernet physical interfaces, and a single instance of a PCIe core that is shared by the DMA blocks.
The BCM5718 is available as four SKUs, BCM5717, BCM5718, BCM5719 and BCM5720. These are also referred to as the Dual-Copper SKU and the Dual-Media SKU. For the BCM5718, BCM5719, and BCM5720 part, any of the network ports may be independently configured as a copper-based (1000BASE-T) or as SerDes-based
(1000BASE-X/SGMII) media interfaces. The choice of media interface on each port is configurable via a poweron strapping option.
The BCM5717 part is permanently bonded as a copper (1000BASE-T) only device.
For the Dual-Media SKU (BCM5718 and BCM5719), whenever a port is configured as a SerDes medium, there are two protocol choices: 1000BASE-X or SGMII. This choice can be made by an Auto-Detect feature or by explicit software programming.
The software driver for this device is capable of loading or unloading each network port independently. The
DMA+EMAC associated with each port is also able to acquire different ACPI power states irrespective of the other; however, the power state of the PCIe link may not necessarily follow that of either one or both ports.
This is a significant behavioral difference of a dual-port controllers compared to single-port. The BCM5718 family hardware and firmware is cognizant of this effect and adds the necessary intelligence to handle it. This functionality remains transparent to device driver software.
On top of the basic dual-port network controller functionality, the BCM5718, BCM5719, and BCM5720 also support an advanced feature known as IO Virtualization (IOV).
Network Controller Sideband Interface (NC-SI) pass-through functionality is the Server Management solution offered by the BCM5718 family of controllers. NC-SI pass-through is a part of the Server Management infrastructure. Such technology offers server platform management via a BMC (baseboard management controller) chip. A server platform and, in turn, a BMC is typically administered remotely over the network, but
BMCs are not equipped with direct network connectivity. Here, a network controller chip comes into the picture—a pass-through-capable NIC chip offers a sideband packet interface to the BMC. Over this interface the BMC can send and receive Ethernet packets to and from the network. In essence, the network adapter functionality of the network controller chip is shared by the host computer system and the BMC chip. NC-SI pass-through protocol is an industry standard (DMTF) for a side band interface between the BMC and network controller. The physical and L2 layers of NC-SI are upwardly compliant to the respective sections of the IEEE
802.3 specification, thus allowing the exchange of Ethernet packets between the network port and BMC without any protocol transformation whatsoever.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 53
BCM5718 Programmer’s Guide Overview of Features
In an environment of several network servers, typically no server is dedicated to running only the management console. Host to BMC pass-through functionality identifies host to local BMC-bound packets and routes them internally to the chip. Similarly, BMC to local host-bound packets are also identified and routed internally to the
NIC chip. Host to BMC pass-through functionality is offered by the BCM5720 controller.
In addition to IEEE 802.3 standard size Ethernet frames, the BCM5718 family also supports jumbo frames of sizes up to 9622 bytes.
The BCM5718 family of controllers replaces the traditional PNP-based linear regulator with a more efficient switching regulator. This regulator steps down system supplied 3.3V rail to 1.2V, which it supplies to the chip’s core.
Figure 2: High-Level System Functional Block Diagram
Host
Chipset
PCIe Link x 2 Gen 1 / x 1 Gen 2
BCM5718 Family
Dual Function
PCIe
DMA 0 DMA 1
NVRAM
Device
Mgmt
Engine
EMAC 0 EMAC 1
BMC
RMII
NC-SI
Pass - through
GHPY 0 Serdes 0 GHPY 1 Serdes 1
Port 0 Port 1
Note: BCM5719 has the same general architecture, but four ports (ports 0, 1, 2, 3) and a quadfunction PCIe interface.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 54
BCM5718 Programmer’s Guide
Receive Data Path
RX Engine
The receive engine (see
) activates whenever a packet arrives from the PHY.
Figure 3: Receive Data Path
H ost Jum bo RX Producer Ring
Receive Data Path
RX
Engine
Rx
FIFO
Rules Checker
Fram e
Buffers
N IC Jum bo RX Producer Ring
D M A
Em pty BD
N IC Standard RX Producer Ring
Host Standard RX Producer
Ring
D M A
Em pty BD
Full BD
List
Initiator
DM A
Host RX
Return
Ring
The receive engine performs the following four functions:
• Moves the data from the PHY to an internal FIFO
• Moves the data from the FIFO to NIC internal memory
• Classifies the frame and checks it for rules matches
• Performs the offloaded checksum calculations
RX FIFO
The RX FIFO provides elasticity while data is read from PHY transceiver and written into internal memory. There are no programmable settings for the RX FIFO. This FIFO’s operation is completely transparent to host software.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 55
BCM5718 Programmer’s Guide Receive Data Path
Rules Checker
The rules checker examines frames. After a frame has been examined, the appropriate classification bits are set in the buffer descriptor. The rules checker is part of the RX data path and the frames are classified during data movement to NIC memory. The following frame positions may be established by the rules checker:
• IP Header Start Pointer
• TCP/UDP Header Start Pointer
• Data Start Pointer
RX List Initiator
The RX List Initiator function activates whenever the receive producer index for any of receive buffer descriptor
(BD) rings is written. This value is located in one of the receive BD producer mailboxes. The host software writes to the producer mailbox and causes the RX Initiator function to enqueue an internal data structure/request, which initiates the DMA of one or more new BDs to the NIC. The actual DMAs generated depend on the comparison of the value of the received BD host producer index mailbox, the NIC copy of the received BD consumer index, and the local copy of the received BD producer index.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 56
BCM5718 Programmer’s Guide Transmit Data Path
Transmit Data Path
TX MAC
The Read DMA engine moves packets from host memory into internal NIC memory (see
). When the entire packet is available, the transmit MAC is activated.
Figure 4: Transmit Data Path
Host Send Producer Rings
DMA
NIC Send Ring Cache
Send BD
TX Data
TX Data
TX Data
TX Data
TX Data
TX Data
TX Data
TX Data
Consumer
Index
Update
DMA
Buffer0
Buffer1
Buffer2
Buffer3
Buffer4
Buffer5
Select Send BDs
(SBDs) from Send
Ring
Tx
FIFO
TX
MAC
The transmit MAC is responsible for the following functions:
• Moving data from NIC internal memory into TX FIFO
• Moving data from TX FIFO to PHY
• Checksum substitutions (not calculation)
• Updating statistics
TX FIFO
The TX FIFO provides elasticity while data is moved from device internal memory to PHY. There are no programmable settings for the TX FIFO. This FIFO’s operation is completely transparent to host software.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 57
BCM5718 Programmer’s Guide DMA Read
DMA Read
Read Engine
The DMA read engine (see
Figure 5 ) activates whenever a host read is initiated by the send or receive data
paths.
Figure 5: DMA Read Engine
Buffer Manager
Host Send Buffer
Descriptors
TX
IO
TX
PCS
TX
RMII
TX
GMII
Statistics
NIC BD
Memory
BD Packet#1
TX
MAC
16
Frame
Mod
64
Tx
FIFO
Frame Classify &
Checksum
Calculation
Frame Header #1
Packet Data #1
NIC Buffer
Memory
DMA
Read
FIFO
Host Send Buffer
Memory
The DMA read engine dequeues an internal data structure/request and performs the following functions:
• DMAs the data from the host memory to an internal Read DMA FIFO
• Moves the data from the Read DMA FIFO to NIC internal memory
• Classifies the frame
• Performs checksum calculations
• Copies the VLAN tag field from the DMA descriptor to the frame header
Read FIFO
The read FIFO provides elasticity during data movement from host memory to device local memory. The memory arbiter is a gatekeeper for multiple internal blocks; several portions of the architecture may simultaneously request internal memory. The PCI read FIFO provides a small buffer for the data read from host memory while the Read DMA engine requests internal memory via the memory arbiter. The data is moved out of the read DMA FIFO into device local memory once a memory data path is available. The FIFO isolates the PCI clock domain from the device clock domain. This reduces latency internally and externally on the PCI bus. The
PCIe Read DMA FIFO holds 1024 bytes. The operation of the read DMA FIFO is transparent to host software.
The Read DMA engine makes sure there is enough space in internal Tx Packet Buffer Memory before initiating a DMA request for transfer of Tx packet data from host memory to device internal packet memory.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 58
BCM5718 Programmer’s Guide DMA Write
Buffer Manager
The buffer manager maintains pools of internal memory used by transmit and receive engines. The buffer manager has logic blocks for allocation, free, control, and initialization of internal memory pools. The DMA read
using the DMA Read Engine. The read DMA engine also fetches Rx BDs for the receive data path.
DMA Write
Write Engine
The DMA write engine, as shown in Figure 6
, activates when a host write is initiated by the send or receive data paths.
Figure 6: DMA Write Engine
RX
IO
RX
PCS
RX
RMII
RX
GMII
RX
MAC
Statistics
Frame
Cracker
Checksum
Calculation
Rules
Checker
Frame
Mod
WOL
Filter
NIC
BD Memory
BD Packet #1
Rx
FIFO
Power
Management
NIC
BufferMemory
Buffer Manager
DMA
Write
FIFO
Host Receive Buffer
Descriptor Ring
BD Packet #1
Packet Data #1
Host Receive Buffer
Memory
The DMA write engine dequeues an internal request and performs the following functions:
• Gathers the data from device internal memory into the write DMA FIFO
• DMAs the data to the host memory from the write FIFO
• Performs byte and word swapping
• Interrupts the host using a line or message signaled interrupt
Write FIFO
The write FIFO provides elasticity during data movement from device memory to the host memory. The write
FIFO absorbs small delays created by PCIe bus arbitration. The NetXtreme family uses the write FIFO to buffer data, so internal memory arbitration is efficient. Additionally, the FIFO isolates the PCI clock domain from the device’s clock domain. This reduces latency on the PCI bus during the write operation (wait states are not inserted while data is fetched from internal memory). The operation of the write DMA FIFO is transparent to host software.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 59
BCM5718 Programmer’s Guide LED Control
Buffer Manager
The buffer manager maintains pools of internal memory used in transmit and receive functions. The buffer manager has logic blocks for allocation, free, control, and initialization of internal memory pools. The receive
MAC requests NIC Rx Mbuf memory so inbound frames can be buffered. The read DMA engine requests the device Tx Mbuf memory for buffering the packets from host memory before they are sent out on the wire. The
DMA write engine requests a small amount of internal memory for DMA and interrupt operations. The usage of this internal memory is transparent to host software, and does not affect device/system performance.
LED Control
Refer to section “LED Control” in the applicable data sheet.
Memory Arbiter
The Memory Arbiter (MA) is a gatekeeper for internal memory access. The MA is responsible for decoding the internal memory addresses that correspond to Ethernet controller data structures and control maps. If a functional block faults or traps during access to internal memory, the MA handles the failing condition and reports the error in a status register. In addition to architectural blocks, the MA provides a gateway for the RISC processor to access local memory. The RISC has an MA interface that pipelines up to three access requests. The
MA negotiates local memory access, so all portions of the architecture are provided with fair access to memory resources. The MA prevents starvation and bounds access latency. Host software may enable/disable/reset the
MA, and there are no tunable parameters.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 60
BCM5718 Programmer’s Guide Host Coalescing
Host Coalescing
Host Coalescing Engine
The Host Coalescing Engine is responsible for pacing the rate at which the NIC updates the send and receive ring indices located in host memory space. The completion of a NIC update is reflected through an interrupt on the Ethernet controller INTA pin or a Message Signalled Interrupt (MSI). Although update criteria are calculated separately, all updates occur at once. This is because all of the ring indices are in one status block, and any host update updates all ring indices simultaneously. The Host Coalescing Engine triggers based on a tick and/or a frame counter.
Figure 7: Host Coalescing Engine
Status Block
Status
Memory
Buffer
Manager
DMA
Write
Engine
Write
FIFO
...
PCIe
Interface
Tick
Counter
BD
Counter
Host
Coalescing
Engine
MSI
FIFO
MSI Mailbox
I/O
Driver
IRQ Host
Interrupt
Controller
Host software may configure line IRQ or MSI
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 61
BCM5718 Programmer’s Guide Host Coalescing
A host update occurs whenever one of the following criteria is met:
• The number of BDs consumed for frames received, without updating receive indices on the host, is equal
to or has exceeded the threshold set in the Receive_Max_Coalesced_BD register (see “Receive Max
Coalesced Bd Count Register (Offset: 0x3c10)” on page 255
).
• The number of BDs consumed for transmitting frames, without updating the send indices, on the host is
equal to or has exceeded the threshold set in the Send_Max_Coalesced_BD register (see “Send Max
Coalesced BD Count Register (Offset: 0x3c14)” on page 256
). Updates can occur when the number of BDs
(not frames) meets the thresholds set in the various coalescing registers (see
Processing,” on page 230 for more information).
• The receive coalescing timer has expired, and new frames have been received on any of the receive rings, and a host update has not occurred. The receive coalescing timer is then reset to the value in the
Receive_Coalescing_Ticks register (see
“Send Coalescing Ticks Register (Offset: 0x3c0c)” on page 255 ).
• The send coalescing timer has expired, and new frames have been consumed from any send ring, and a host update has not occurred. The send coalescing timer is then reset to the value in the
Send_Coalescing_Ticks register.
MSI FIFO
This FIFO is eight entries deep and four bits wide. This FIFO is used to send MSIs via the PCI interface. The host coalescing engine uses this FIFO to enqueue requests for the generation of MSI. There are no configurable options for this FIFO and this FIFOs operation is completely transparent to host software.
Status Block
This data structure contains consumer and producer indices/values. Host software reads this control block, to assess hardware updates in the send and receive rings. Two copies of the status block exist. The local copy is
DMAed to host memory by the DMA write engine. Host software does not want to generate PCI transactions to read ring status; rather quicker memory bus transactions are desired. The host coalescing engine enqueues a request to the DMA write engine, so host software gets a refreshed copy of status. The status block is
of the status block.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 62
BCM5718 Programmer’s Guide 10BT/100BTx/1000BASE-T Transceiver
10BT/100BTx/1000BASE-T Transceiver
Auto-Negotiation
The Ethernet controller devices negotiate their mode of operation over the twisted-pair link using the autonegotiation mechanism defined in the IEEE 802.3u and IEEE 802.3ab specifications. Auto-negotiation can be enabled or disabled by hardware or software control. When the auto-negotiation function is enabled, the
Ethernet controllers automatically choose the mode of operation by advertising its abilities and comparing them with those received from its link partner. The Ethernet controllers can be configured to advertise
1000BASE-T full-duplex and/or half-duplex, 100BASE-TX full-duplex and/or half-duplex, and 10BASE-T fullduplex and/or half-duplex. The transceiver negotiates with its link partner and chooses the highest operating speed and duplex that are common between them. Auto-negotiation can be disabled for testing or for forcing
100BASE-TX or 10BASE-T operation, but is always required for normal 1000BASE-T operation.
Automatic MDI Crossover
During auto-negotiation, one end of the link must perform an MDI crossover so that each transceiver’s transmitter is connected to the other receiver. The Ethernet controllers can perform an automatic MDI crossover when the Disable Automatic MDI Crossover bit in the PHY Extended Control register is disabled, thus eliminating the need for crossover cables or cross-wired (MDIX) ports. During auto-negotiation, the Ethernet controllers normally transmit on TRD
±{0} and receive on TRD±{1}. When connected to another device that does not perform the MDI crossover, the Ethernet controller automatically switches its transmitter to TRD ±{1} and its receiver to TRD
±{0} to communicate with the remote device. If two devices that both have MDI crossover capability are connected, an algorithm determines which end performs the crossover function.
During 1000BASE-T operation, the Ethernet controllers swap the transmit symbols on pairs 0 and 1 and pairs 2 and 3 if auto-negotiation completes in the MDI crossover state. The 1000BASE-T receiver automatically detects pair swaps on the receive inputs and aligns the symbols properly within the decoder.
PHY Control
The NetXtreme/NetLink Ethernet controller supports the following physical layer interfaces:
• The MII is used in conjunction with 10/100 Mbps copper Ethernet transceivers.
• GMII supports 1000 Mbps copper Ethernet transceivers.
MII Block
The MII interconnects the MAC and PHY sublayers (as shown in
).
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 63
BCM5718 Programmer’s Guide
Figure 8: Media Independent Interface
2.5 MHz at 10 Mbps
25 MHz at 100 Mbps
Physical Layer
MII 4-bit Data Path
MAC Sublayer
Symbol
Decoder
LED
Control
RX
I/O
RXD /4
RX_CLK1
RX_ER
RX_DV
LED
I/O
RX
I/O
COL
CRS
LNKRDY
Media
Status
I/O
RX Media
Access
Mgmnt
Rx Data
Decapsulation
RX
MAC
PHY Control
Symbol
Encoder
TX
I/O
TXD /4
MII_TXCLK
TX_ER
TX_EN
TX
I/O
TX
Media
Access
Mgmnt
Tx Data
Encapsulation
TX
MAC
2.5 MHz at 10 Mbps
25 MHz at 100 Mbps
4-bit Data Path
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 64
BCM5718 Programmer’s Guide PHY Control
The specifics of MII may be located in section 22 of the IEEE 802.3 specification. RXD[3:0] are the receive data signals; TXD[3:0] are the transmit data signals. MII operates at both 10-Mbps and 100-Mbps wire-speeds.
(Gigabit Ethernet uses the GMII standard.) When MAC and PHY are configured for 10 Mbps operation, the
RX_CLK1 and MII_TXCLK clocks run at 2.5 MHz. Both RX_CLK1 and MII_TXCLK are sourced by the PHY. 100
Mbps wire speed requires RX_CLK1 and MII_TXCLK to provide a 25 MHz reference clock. Receive Data Valid
(RX_DV) is asserted when valid frame data is received; at any point during data reception, the PHY may assert
Receive Error (RX_ER) to indicate a receive error. The MAC will record this error in the statistics block. The MAC may discard a bad RX frame—exception being sniffer/promiscuous modes (see Allow_Bad_Frames bit in MAC mode register). The Transmit Enable (TX_EN) signal is asserted when the MAC presents the PHY with a valid frame for transmission. The MAC may assert TX_ER to indicate the remaining portion of frame is bad. The PHY will insert Bad Code symbols into the remaining portion of the frame. A detected collision in half-duplex mode may be such a scenario where TX_ER is asserted. The PHY will assert COL when a collision is detected. The COL signal is routed to both the RX and TX MACs. The transmit MAC will back off transmission and the RX MAC will throw away partial frames.
The 10 Mbps physical layer uses Differential Manchester encoding on the wire. Manchester encoding uses two encoding levels: 0 and 1. 100 Mbps Ethernet requires MLT-3 waveshaping on the transmission media. MLT-3 uses three encoding levels: – 1, 0, and 1. Both physical signaling protocols are transparent to the MAC sublayer and are digitized by the PHY. The PHY encodes/decodes analog waveforms at its lower edge while the PHY presents digital data at its upper edge (MII).
GMII Block
The GMII is full-duplex (see Figure 9 on page 66
); the send and receive data paths operate independently.
The transmit signals TXD[7:0] create a eight-bit wide data path. The TXD[7:0] signals are synchronized to the reference clockTX_CLK0. The TX_CLK0 clock runs at 125 MHz and is sourced by the MAC sublayer. Transmit
Error (TX_ER) is asserted by the MAC sublayer. The PHY will transmit a bad code until TX_ER is de-asserted by the MAC. TX_ER is driven synchronously with TX_CLK0. The Transmit Enable (TX_EN) indicates that valid data is presented on the TXD lines. The TXD[7:0] data is framed on the rising edge of TX_EN.
The receive data path is also eight bits wide. RXD[7:0] are sourced by the PHY. When valid data is presented to the MAC sublayer, the PHY will also assert Receive Data Valid (RX_DV). The rising edge of RX_DV indicates the beginning of a frame sequence. The PHY drives the reference clock RX_CLK1, so the MAC sublayer can synchronize data sampling on RXD[7:0]. The PHY may assert RX_ER to indicate frame data is invalid; the MAC sublayer must consider frames in progress incomplete.
When the MAC operates in half-duplex mode, a switch or node may transmit a jamming pattern. The PHY will drive the Collision (COL) signal so the MAC may back off transmission and throw away partially received packet(s). The COL signal will also cause the TX MAC to stop the transmission of a packet. The COL signal is not driven for full-duplex operation since collisions are undefined. The PHY will drive Carrier Sense (CRS) as a response to traffic being sent/received. The MAC sublayer can monitor traffic and subsequently drive traffic
LEDs.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 65
BCM5718 Programmer’s Guide PHY Control
Pulse Amplitude Modulated Symbol (PAM5) encoding is leveraged for Gigabit Ethernet wire transmissions.
PAM5 uses five encoding levels: – 2, – 1, 0, 1, and 2. Four symbols are transmitted in parallel on the four twisted-wire pairs. The four symbols create a code group (an eight-bit octet). The process of creating the codegroup is called 4D-PAM5. Essentially, eight data bits are represented by four symbols. Table 40-1 in the IEEE
802.3ab specification shows the data bit to symbol mapping. The code group representation is also referred to as a quartet of quinary symbols {TA, TB, TC, TD}. The modulation rate on the wire is measured at 125 Mbaud.
The resultant bandwidth is calculated by multiplying 125 MHz by eight bits, for
1000 Mbps wire speed.
Figure 9: GMII Block
125-MHz Ref Clock
Physical Layer
GMII
8-bit Data Path
MAC Sublayer
Symbol
Decoder
RX
I/O
RXD /8
RX_CLK1
RX_ER
RX_DV
RX
I/O
RX Media
Access
Mgmnt
Rx Data
Decapsulation
RX
MAC
COL
CRS
Media
Status
I/O
LED
Control
LED
I/O
LNKRDY
Symbol
Encoder
TX
I/O
TXD /8
TX_CLK0
TX_ER
TX_EN
TX
I/O
TX Media
Access
Mgmnt
Tx Data
Encapsulation
TX
MAC
125-MHz Ref Clock 8-bit Data Path
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 66
BCM5718 Programmer’s Guide
MDIO Register Interface
Figure 10 shows the MDI register interface.
Figure 10: MDI Register Interface
Physical Layer MAC Sublayer
Mgmnt
Control
MDI
Register block
Mgmnt
I/O
MDC
MDIO
MDINT
Mgmnt
I/O
Mgmnt
Control
(MII & GMII)
PHY Control
Management Data Clock
The Management Data Clock (MDC) is driven by the MAC sublayer. The PHY will sink this signal to synchronize data transfer on the MDIO signal—MDC is a reference clock. This clock is not functionally associated to either
RX_CLK or TX_CLK. The minimum period for this clock is 400 ns with high and low times having 160 ns duration.
Management Data Input/Output
The Management Data Input/Output (MDIO) signal passes control and status data, between the MAC and PHY sublayers. MDIO is a bidirectional signal, meaning both the PHY and MAC may transfer data. The MAC typically transfers control information and polls status; whereas, the PHY transfers status back to the MAC, using MDIO.
Management Data Interrupt
The integrated Broadcom PHY may be programmed to generate interrupts. A PHY status change initiates a
Management Data Interrupt (MDINT). A MDI mask register allows host software to selectively enable/disable status types, which cause MDINT notification. The PHY will assert INTR until software clears the interrupt.
Reading the status register will clear the interrupt.
Management Register Block
The layout and configuration of MDI register block is device dependent. The MDI register block is the control/ status access point, which host software may read/write. The IEEE 802.3 specification defines a basic register block for MII and GMII; the basic register set contains control and status registers. GMII also exposes an extended register set, used in 1000 Mbps configuration/status. The fundamental point is to understand that the MDC and MDIO signals are used to access the MDI register block.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 67
BCM5718 Programmer’s Guide NVRAM Configuration
Section 3: NVRAM Configuration
Overview
Broadcom NetXtreme and NetLink controllers require the use of an external non-volatile memory (NVRAM) device (Flash or SEEPROM), which contains a boot code program that the controller's on-chip CPU core loads and executes upon release from reset. This external NVRAM device also contains many configuration items that direct the behavior of the controller, enable/disable various management and/or value-add firmware components, etc.
All configuration settings are default-configured in the official release binary image files provided in
Broadcom's CD software releases. However, the settings chosen as default by Broadcom may not be what best suits a particular OEM's application, so some settings may need to be changed by the OEM.
The BCM5718 family supports the following boot code styles:
• “Legacy”—Boot code plus configuration options fully contained in an external 8k byte NVRAM device
• “Self-boot”—Uses a fixed internal ROM'd copy of the boot code program (requires only a very small external NVRAM) for the purpose of housing OEM-programmable configuration items (refer to Application
Note NetXtreme-AN60x-R “NetXtreme/NetLink NVRAM Configuration Options”).
Note: If using self-boot, the design must use a very small (approximately 256 byte) external NVRAM to hold configuration items and self-boot code patches.
Refer to Broadcom Application Note NetXtreme-AN40X-R (NetXtreme®/NetLink™ Software Self-Boot NVRAM
Application Note) for additional detail regarding self-boot NVRAM structure.
Details relating to the legacy style NVRAM organization can be found in NetXtreme/NetLink NVRAM Access
Broadcom application note (Netxtreme-AN50X-R). Some of the topics addressed by this application note include the following:
• Programming NVRAM (sample C code, x86 assembly)
• NVRAM map
• Configuration settings
• Boot code
• Multiple boot agent (MBA), PXE, etc.
Note: NVRAM CRC-32: There are multiple distinct regions contained within the NVRAM map. Each of these regions has its own CRC-32 checksum value associated with it. If any data element contained within a region is modified, then that region's CRC-32 value must also be updated. Details relating to calculating the CRC-32 can be found in Calculating CRC32 Checksums for Broadcom NetLink,
NetXtreme, and NetXtreme II Controllers Broadcom application note (NetXtreme_NetXtremeII-
AN20X-R).
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 68
BCM5718 Programmer’s Guide Self-Boot
Self-Boot
Some NetLink controllers offer a capability known as self-boot. Self-boot allows the controller to use a very small, low-cost, external NVRAM device that contains only a very condensed amount of configuration information, along with any small boot code patches that may be necessary to optimize the functionality of a particular controller.
Details relating to self-boot can be found in Self Boot Option (5754X_5787X-AN10X-R) and NetXtreme/NetLink
Software Self-Boot NVRAM (NetXtreme-AN40X-R) Broadcom application notes.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 69
BCM5718 Programmer’s Guide Common Data Structures
Section 4: Common Data Structures
Theory of Operation
Several device data structures are common to the receive, transmit, and interrupt processing routines. These data structures are hardware-related and are used by device drivers to read and update state information.
Descriptor Rings
In order to send and receive packets, the host and the controller use a series of shared buffer descriptor (BD) rings to communicate information back and forth. Each ring is composed of an array of buffer descriptors that reside in host memory. These buffer descriptors point to either send or receive packet data buffers. The largest amount of data that a single buffer may contain is 65535 (64K-1) bytes (The length field in BD is 16 bits).
Multiple descriptors can be used per packet in order to achieve scatter-gather DMA capabilities.
Note: The maximum number of Send BDs for a single packet is (0.75)*(ring size).
There are three main types of descriptor rings:
• Send Rings
• Receive Producer Rings
• Receive Return Rings
The TX/RX ring base requires an 8 byte alignment. The receive buffer address (recorded in SBD/RBD) cannot cross 4G.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 70
BCM5718 Programmer’s Guide Descriptor Rings
Producer and Consumer Indices
The Producer Index and the Consumer Index control which descriptors are valid for a given ring. Each ring will have its own separate Producer and Consumer Indices. When incremented, the Producer Index can be used to add elements to the ring. Conversely, when incremented, the Consumer Index is used to remove elements from the ring. The difference between the Producer and Consumer Indices mark which descriptors are currently valid in the ring (see
). When the Producer and Consumer Index are equal, the ring is empty. When the producer is one behind the consumer, the ring is considered to be full.
Figure 11: Generic Ring Diagram
The drawing shows a generic host descriptor ring (could be either a send ring or a receive ring), and demonstrates how the consumer and producer indices are used to determine which descriptors in the ring are valid at any given moment in time.
1st
Cons
The delta between the producer and consumer indices is indicated by the shaded areas. These shaded descriptors are considered to be valid
(non-empty) and thus need to be processed.
Prod
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 71
BCM5718 Programmer’s Guide Descriptor Rings
Ring Control Blocks
Each ring (send or receive) has a Ring Control Block (RCB) associated with it. Each RCB has the format shown in
Offset (bytes) 31
0x00
0x04
0x08
0x0c
Table 4: Ring Control Block Format
16 15
Host Ring Address
Max_len
Reserved
Flags
0
The fields are defined as follows:
• The Host Ring Address field contains the 64-bit host address of the first element in the ring. Basically, this field tells the controller precisely where in host memory the ring is located. This field only applies to rings that are located in host memory. The Host_Ring_Address field contains the 64-bit address, in big-endian ordering, of the first Send BD in host memory.
• The Flags field contains bits flags that contain control information about a given ring.
two flags that are defined.
Bits
0
1
Name
Reserved
RCB_FLAG_RING_DISABLED
15:2 STD_MAX_PACKET_SIZE
Table 5: Flag Fields for a Ring
Description
Should be set to 0.
Indicates that the ring is not in use.
Indicates maximum frame size for the ring.
• The Max_len field has a different meaning for different types of rings.
– This field indicates the number elements in the ring.
– The valid values for this field are 32, 64, 128, 256, 512, 1024, 2048, and 4096.
• Max ring sizes supported in the BCM5718 Family are shown below.
– Rx Return: 4096
– Rx Producer: 2048
– Rx Producer Jumbo: 1024
– Tx Producer: 512
• The NIC Ring Address field contains the address where the BD cache is located in the internal NIC address space. This address is only valid for Receive Producer Rings. The Send Rings and Receive Return Rings do not require this field to be populated. The location within the NIC address map for Receive Producer Ring
is provided in “PCI” on page 168 .
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 72
BCM5718 Programmer’s Guide Descriptor Rings
Send Ring Control Blocks
The format of the Send RCB remains unchanged, only 15 more are added as shown in the Table below.
Send Ring# Register Name
Legacy /
1
2
Host Address High
Host Address Low
Max Length/Flag
NIC Address
Host Address High
Host Address Low
Max Length/Flag
NIC Address
3
4
5
….
16
………
Host Address High
Host Address Low
Max Length/Flag
NIC Address
** These are Memory Mapped registers
Table 6: Send RCBs for Multiple Rings
Send Ring RCB
Register Address (**)
0x100
0x104
0x108
0x10C
0x110
0x114
0x118
0x11C
……
0x1F0
0x1F4
0x1F8
0x1FC
Usable in
Legacy Mode,
RSS Mode &
IOV Mode
RSS Mode &
IOV Mode
IOV Mode Only
Note: Address range [0x100 - 0x10F] is the legacy single RCB address and is also used by RCB1.
Receive Ring Control Blocks
Table 7 lists all of the receive RCB Register Addresses.
VRQ #
0 (Default)
Table 7: High Priority Mail Box Registers for VRQ Rings
Register Name
Standard Ring RCB
Register Address
Host Address High 0x2450
Host Address Low 0x2454
Max Length/Flag 0x2458
NIC Address 0x245C
Jumbo Ring RCB
Register Address
0x2440
0x2444
0x2448
0x244C
Return Ring RCB Register
Address (**)
0x200
0x204
0x208
0x20C
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 73
BCM5718 Programmer’s Guide Descriptor Rings
VRQ #
1
2
3
4
.....
16
Table 7: High Priority Mail Box Registers for VRQ Rings (Cont.)
Register Name
Standard Ring RCB
Register Address
Host Address High 0x2510
Host Address Low 0x2514
Max Length/Flag 0x2518
NIC Address 0x251C
Host Address High 0x2530
Host Address Low 0x2534
Max Length/Flag 0x2538
NIC Address 0x253C
Host Address High 0x2550
Host Address Low 0x2554
Max Length/Flag 0x2558
NIC Address 0x255C
Host Address High 0x2570
.....
.....
Host Address High 0x26F0
Host Address Low 0x26F4
Max Length/Flag 0x26F8
NIC Address 0x26FC
Jumbo Ring RCB
Register Address
0x2500
0x2504
0x2508
0x250C
0x2520
0x2524
0x2528
0x252C
0x2540
0x2544
0x2548
0x254C
0x2560
.....
0x26E0
0x26E4
0x26E8
0x26EC
Return Ring RCB Register
Address (**)
0x230
0x234
0x238
0x23C
0x240
.....
0x300
0x304
0x308
0x30C
0x210
0x214
0x218
0x21C
0x220
0x224
0x228
0x22C
Note: [0x2450 .. 0x245C] and [0x2440 .. 0x244C] are legacy RCB addresses and are being assigned to
VRQ Ring# 0.
Note: The Return Ring RCBs are Memory Mapped. Memory Address [0x200 .. 0x23C] are legacy
Return Ring RCB addresses and are being assigned to VRQ Return Ring# 0 through Ring# 3
Send Rings
The controller devices covered in this document support only one host based Send Ring.
The Send Ring Producer Index is incremented by host software to add descriptors to the Send Ring (see
). By adding descriptors to the ring, the device is instructed to transmit packets that are composed of the buffers pointed to by the descriptors. A single transmit packet may be composed of multiple buffers that are pointed to by multiple send descriptors. The maximum number of send descriptors for a single packet is (0.75)*(ring size).
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 74
BCM5718 Programmer’s Guide Descriptor Rings
Figure 12: Transmit Ring Data Structure Architecture Diagram
Transmit Ring Data Scructure is located in the host (as shown below), and the device will keep a local (not shown) copy of the rings.
Data Structures kept on-chip
Host Memory
Host Buffer
1-(64K-1) Bytes
Status Block (80 bytes)
The Status block resides in the NIC memory space and is periodically
DMA'd to the host when the TX/RX coalescing timers expire, or when the
RX/TX max coalesced frames thresholds are met. Software can examine the TX consumer indices in the status block to determine which packets have been sent by the hardware.
Data Structures in the host
Send Buffer Descriptor
Host Address length rsvd for firmware flags
VLAN tag
Send Host BD
Host Send Ring #1
Cons
1st
Prod
Status Block
Status Word unused
RX std cons
Unused
TX Cons
Unused
Unused
RX Prod #1
RX Prod #2
RX Prod #3
RX Prod #4
Ring Control Block
Host Ring Address max_len
NIC Ring Address flags
RCB
Mailbox Registers
TX Host Ring Prod
The mailbox registers reside on-chip starting at offset 0x300.
Each mailbox register is 64 bits wide. Writing the lower 32 bits triggers an event in the HW.
SW updates the TX Host Ring producer index to indicate that there are buffer descriptors ready for the HW to process.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 75
BCM5718 Programmer’s Guide Descriptor Rings
Send Buffer Descriptors
Standard (Not Large Segment Offload)
The format of an individual send buffer descriptor is shown in
.
Offset (Bytes) 31
0x00
0x04
0x08
0x0c
Table 8: Send Buffer Descriptors Format
16 15
Host Address [63:0]
Length [15:0]
Reserved
0
Flags [15:0]
VLAN Tag
The fields are defined as follows:
• The Host Address field contains the 64-bit host address of the buffer that the descriptor points to. A length of 0 indicates that the descriptor does not have a buffer associated with it.
• The Flags field contains bits flags that contain control information for the device for transmitting the
packets. The defined flags are listed in Table 9 .
Table 9: Defined Flags for Send Buffer Descriptors
Bits Name
0
1
2
3
4
5
6
TCP_UDP_CKSUM
IP_CKSUM
PACKET_END a
Description
If set to 1, the controller replaces the TCP/UDP checksum field of TCP/UDP packets with the hardware calculated TCP/UDP checksum for the packet associated with this descriptor.
If set to 1, the controller replaces the IPv4 checksum field of TCP/UDP packets over IPv4 with the hardware calculated IPv4 checksum for the packet associated with this descriptor. This bit should only be set in the descriptor that points to the buffer containing the IPv4 header. It is assumed that the IPv4 header is contained in a single buffer.
If set to 1, the packet ends with the data in the buffer pointed to by this descriptor.
Jumbo Frame
VLAN_TAG a
The driver must set this bit to 1 if the MTU length of the Send Frame is > 1500B.
The MTU length is the Ethernet payload length and excludes Header length (and
Trailer length).
All BDs belonging to a Send Packet must configure this bit identically.
HDRLEN[2] The length of the Ether+IP+TCP Headers to be replicated in each segment arising out of a Large TCP Segment (LSO).
Capture Time Stamp (BCM5719/5720 only) If this bit is 1, this frame’s launch time shall be captured in the TX Time-Stamp Register.
If set to 1, the device inserts an IEEE 802.1Q VLAN tag into the packet. The 16bit TCI (Tag Control Information) field of four byte VLAN tag comes from the
VLAN Tag field in the descriptor.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 76
BCM5718 Programmer’s Guide Descriptor Rings
Table 9: Defined Flags for Send Buffer Descriptors (Cont.)
11
12
13
14
15
Bits
7
8
9
10
Name
COAL_NOW
CPU_PRE_DMA
CPU_POST_DMA
HDRLEN[3]
HDRLEN[4]
HDRLEN[5]
HDRLEN[6]
HDRLEN[7]
DON’T_GEN_CRC a a
Description
If set to 1, the device immediately updates the Send Consumer Index after the buffer associated with this descriptor has been transferred via DMA to NIC memory from host memory. An interrupt may or may not be generated according to the state of the interrupt avoidance mechanisms. If this bit is set to 0, then the Consumer Index is only updated as soon as one of the host interrupt coalescing conditions has been met.
If set to 1, the controller’s internal CPU is required to act upon the packet before the packet is given to the internal Send Data Initiator state machine. Normally this bit should be set to 0.
If set to 1, the controller’s internal CPU is required to act upon the packet before the packet is given to the internal Send Data Completion state machine.
Normally this bit should be set to 0.
–
–
The length of the Ether+IP+TCP Headers (combined) to be replicated in each frame arising out of a Large TCP Segment (LSO). Maximum Header Length is
256B.
–
–
If set to 1, the controller will not append an Ethernet CRC to the end of the frame.
a. Indicates that this bit should be set in all descriptors for a given packet if the desired capability is to be enabled for that packet.
Note: The UDP checksum engine does not span IP fragmented frames.
• The Length field specifies the length of the data buffer. The lengths for the buffers associated with a given packet will add up to the length of the packet.
Note: The Ethernet controller does not validate the value of the Length field and may generate an error on the PCI bus if the Length field has a value of 0. The host driver must ensure that the
Length field is nonzero before enqueueing the BD onto the Send Ring.
• The VLAN Tag field is only valid when the VLAN_TAG bit of Flags field is set. This VLAN Tag field contains the 16-bit VLAN tag that is to be inserted into an IEEE 802.1Q (and IEEE 802.3ac)-compliant packet by the controller. If VLAN tag insertion is desired, this field (and the flag) should be set in the first descriptor for that packet (i.e., the descriptor that points to the buffer that contains the Ethernet header).
Large Segment Offload (LSO) Send BD
See “Large Segment Offload” on page 111 .
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 77
BCM5718 Programmer’s Guide Descriptor Rings
Receive Rings
The Ethernet controllers support two types of Receive Descriptor Rings: Producer Rings and Return Rings (see
Producer ring dedicated to jumbo frame reception. Descriptors in the Producer Rings point to free buffers in the host. When the controller receives a packet and consumes a receive buffer, the controller will modify and write back the descriptor for the consumed buffer into the given Receive Return Ring. Basically the Producer
Rings contain descriptors that point to buffers that the controller is free to use, whereas the Return Rings contain descriptors that the device has used and await processing from host software.
Figure 13: Receive Return Ring Memory Architecture Diagram
Receive Buffer Descriptor
Host Address index type ip chksum error flag reserved opaque len flags tcp_udp_chsum vlan tag
Host Memory
Host Buffer
1-(64K-1) Bytes
Receive Ring #1
Ring Control Block
Host Ring Address max_len
NIC Ring Address flags
RX BD
Prod
1st
RCB #1
Cons
RCB #4
Receive Ring #4
Status Block (80 bytes)
Status block resides in the NIC memory space and is periodically DMA'd to the host based on the host coalescing Timer.
Status Block
RX std cons
Unused
TX Cons
Status Word unused
Unused
Unused
RX Prod #1
RX Prod #2
RX Prod #3
RX Prod #4
The NIC is the Producer of the receive return ring. It increments the internal producer index to add elements to the ring.
Prod
Con
1st
Mailbox Registers
RX Cons #1
RX Cons #2
Note: The receive return rings are stored in host memory.
Receive Producer Ring
The receive producer ring resides in the host and points to empty host receive buffers that will later be filled with received packet data. The controller will internally cache a copy of the producer ring. When the host software driver has a free host receive packet buffer available for incoming packets, it will fill out a receive buffer descriptor and have that descriptor point to the available buffer. Host software will then update the producer index for that receive producer ring to indicate to the controller that there is a newly available receive buffer. After the controller fetches and caches (e.g., consumes) this receive producer descriptor, the controller will update the consumer index of the receive producer ring.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 78
BCM5718 Programmer’s Guide Descriptor Rings
Receive Return Rings
When the NIC receives a packet, it will DMA that packet to a host receive packet data buffer pointed to by the available receive buffer descriptor (see
Section 5: “Receive Data Flow,” on page 89 ). Earlier the NIC will have
received ownership of that data buffer via an update of the producer index of receive producer ring. After the controller does the packet data write DMA, it will DMA a corresponding buffer descriptor into the appropriate receive return ring. The buffer descriptor that is returned in the receive return ring will be slightly modified from the original buffer descriptor that the controller fetched out of the receive producer ring. After the controller has completed the DMA of the receive return ring descriptor, the controller will update its internal copy of the producer index for that particular receive return ring. That new value for that receive return ring producer index will be included in the next status block update that is made to the host. The updated value of receive return ring producer index in status block will be used by host software in determining whether new packets have been received.
Description
Number of Rings
Buffer Descriptor Size (bytes)
Host Ring Size (# of Buffer Descriptors)
Table 10: Receive Return Rings
NIC Cache Size (# of Buffer Descriptors)
BCM5718 Family
4
32
Can be configured for 32 or 64 or 128 or 256 or 512 or
1024 or 208 or 4096
0
Receive Buffer Descriptors
The format of Standard Receive Buffer Descriptors (in both producer ring and return rings) is shown in
.
Table 11: Receive Descriptors Format
150
Host Address
Offset (bytes) 3116
0x00
0x04
0x08
0x0c
0x10
0x14
0x18
0x1C
Index
Type
IP_Cksum
Error_Flags
RSS Hash
Opaque
Length
Flags
TCP_UDP_Cksum
VLAN tag
The fields are defined as follows:
• Host Address — Contains the 64-bit host address of the buffer that the descriptor points to. A length of 0 indicates that the descriptor does not have a buffer associated with it.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 79
BCM5718 Programmer’s Guide Descriptor Rings
• Length — Specifies the length of the data buffer. For Producer Rings this value is set by the host software to correspond to the size of the buffer that is available for a receive packet. Once a packet has been received, the controller will modify this length field to correspond to the length of the packet received. A value of 0 indicates that there is no valid data in the buffer.
• Index — Is set by host software in the descriptors in the producer rings. When the controller uses a given buffer descriptor, it will opaquely pass the Index field for that buffer descriptor through to the corresponding descriptor in the return ring. This index field of the BD in Return Ring is then used by the host software to associate the BD in Return Ring with the BD in Producer Ring that points to the given receive buffer.
• Flags — Contains bits flags that contain control information about a given descriptor. The defined flags are listed in
Bits
15
14
Name
IP Version
13 TCP_UDP_CHECKSUM
12
11
10
9:7
6
5
TCP_UDP_IS_TCP
IP_CHECKSUM
Reserved
FRAME_HAS_ERROR
RSS Hash Type
VLAN_TAG*
Reserved
Table 12: Defined Flags for Receive Buffers
Description
Indicates whether the received IP packet is an IPv6 or IPv4 packet. This bit will be 1 for IPv6 packet and 0 for IPv4 packet.
In producer rings this bit should be set to 0. In return rings this bit will be set to 1 by the controller if the controller calculated that the incoming packet was a TCP packet. If the packet is UDP or a non IP frame, then this bit should be set to 0.
In producer rings this bit should be set to 0. In return rings this bit will be set to 1 by the controller if the controller calculated that the TCP or UDP checksum in the corresponding incoming packet was correct.
In producer rings this bit should be set to 0. In return rings this bit will be set to 1 by the controller if the controller calculated that the IP checksum in the corresponding incoming packet was correct.
–
If set to 1 in a return ring, it indicates that the controller detected an error.
The specific type of error is specified in the Error_Flag field of the receive return descriptor.
Indicates the hash type used in RSS hash calculation for a received packet.
Available hash types are:
• 0 2_TUPLE_IPV4
• 1 4_TUPLE_IPV4
• 2 2_TUPLE_IPV6
• 3 4_TUPLE_IPV6
• 4 Reserved
• 5 Reserved
• 6 Reserved
• 7 Reserved
See
“Receive MAC Mode Register (offset: 0x468)” on page 323
for additional information about enabling the different RSS hash types.
If set to 1 in a return ring, it indicates that the packet associated with this buffer contained a four-byte IEEE 802.1Q VLAN tag. The 16 VLAN ID is stripped from the packet and located in the VLAN tag field in the descriptor.
Should be set to 0.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 80
BCM5718 Programmer’s Guide Descriptor Rings
Bits
4
3
2
Name
Reserved
RSS_Hash Valid
PACKET_END
1:0 Reserved
Table 12: Defined Flags for Receive Buffers (Cont.)
Description
Should be set to 0.
If set to 1, indicates host that the RSS_Hash in Receive BD of return ring is valid.
If set to 1, the packet ends with the data in the buffer pointed to by this descriptor.
Should be set to 0.
• Type — Used internally by the controller. In producer rings it should be set to 0, and in return rings it should be ignored by host software.
• TCP_UDP_Cksum — Holds the TCP/UDP checksum that the controller calculated for all data following the
IP header given the length defined in the IP header. If the Receive No Pseudo-header Checksum bit is set
(see
“Mode Control Register (offset: 0x6800)” on page 472
) to 1, then the pseudo-header checksum value is not added to this value. Otherwise, the TCP_UDP_Cksum field includes the pseudo-header in the controller’s calculation of the TCP or UDP checksum. If the packet is not a TCP or UDP packet, this field has no meaning. Host software should zero this value in the producer ring descriptors. If the host is capable of
TCP or UDP checksum off load, then host software may examine this field in the return rings to determine if the TCP or UDP checksum was correct.
• IP_Cksum — Host software should zero this value in the producer ring descriptors. For Receive Return
Ring descriptors, if using IP checksum offload, the host driver software should rely on the Flags bit
IP_CHECKSUM (Flags bit 12) to determine if the IP checksum in the received packet is correct. This field used to contain the actual IP checksum value but that is not true for the BCM5718 family of controllers.
Only the Flags bit IP_CHECKSUM should be relied on by host driver software as is done by Broadcom drivers.
• VLAN — Only valid when the VLAN_TAG bit is set. This field contains the 16-bit VLAN ID that was extracted from an incoming packet that had an IEEE 802.1Q (and IEEE 802.3ac) -compliant header.
• Error_Flags — Contains bits flags that contain error information about an incoming packet that the descriptor is associated with. The bits in this field are only valid if the FRAME_HAS_ERROR bit is set in the
Flags field in the descriptor. The defined error flags are listed in Table 13 on page 81 .
Bits
7
6
5
Name
31:9 Reserved
8 GIANT_PKT_RCVD
TRUNC_NO_RES
LEN_LESS_64
MAC_ABORT
Table 13: Defined Error Flags for Receive Buffers
Description
Should be set to 0.
If set to 1, the received packet was longer than the maximum packet length value set in the Receive MTU Size register (see
(offset: 0x43C)” on page 317 ). The data in the received packet was truncated at
the length specified in the Receive MTU Size register.
If set to 1, the received packet was truncated because the controller did not have the resources to receive a packet of this length.
If set to 1, the received packet was less than the required 64 bytes in length.
If set to 1, the MAC aborted due to an unspecified internal error while receiving this packet. The packet could be corrupted.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 81
BCM5718 Programmer’s Guide Descriptor Rings
Table 13: Defined Error Flags for Receive Buffers (Cont.)
1
0
Bits Name
4
3
2
Description
ODD_NIBBLE_RX_MII If set to 1, the received packet contained an odd number of nibbles. Thus, packet data could be corrupt.
PHY_DECODE_ERR If set to 1, while receiving this packet the device encountered an unspecified frame decoding error. This packet could be corrupted.
This bit is set for valid packets that are received with a dribble nibble. True alignment errors will be dropped by that MAC and never show up to the driver.
LINK_LOST
COLL_DETECT
BAD_CRC
If set to 1, link was lost while receiving this frame. Therefore, this packet is incomplete.
If set to 1, a collision was encountered while receiving this packet.
If set to 1, the received packet has a bad Ethernet CRC.
• When the RSS Hash Valid flag bit is 1, the RSS Hash field holds the 32-bit RSS hash value calculated for a packet. This field should be ignored when the RSS Hash Valid flag bit is zero.
• The Opaque field is reserved for the host software driver. Any data placed in this field in a producer ring descriptor will be passed through unchanged to the corresponding return ring descriptor.
Additional Ring Information for the BCM5718 Family
Rings in the BCM5718 Family are more complicated than in previous NetXtreme controllers. Before considering ring structure details, first choose the mode of operation:
• Legacy
• RSS
• RSS+TSS
• IOV
Note: RSS (and/or TSS) and IOV are mutually exclusive.
Once the mode of operation is decided, determine if jumbo frames are to be used.
Setting ring sizes involves setting some ring sizes (such as the Rx Producer Rings) in registers and setting other ring sizes (such as the Rx Return Rings) in memory locations (for example, the upper 16 bits of the 32 bit value
bit words:
• host address high
• host address low
• len/flags
• NIC ring address
For example, the RX Return Ring size is set in the high 16 bits of 32bit value in device memory offset 0x208.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 82
BCM5718 Programmer’s Guide Status Block
If using RSS or IOV, there are either 4 or 17 Rx Return and Producer rings. The 17th Rx ring is a default throwaway ring for any Rx traffic that does not map to rings 0-3 (RSS) or 0-15 (IOV). Driver software only pulls valid
Rx BDs from 16 Rx Ret Rings (rings 0-15) in IOV mode.
For Rx Producer Rings, there are only 1 or 2 in non-IOV mode (the second one dedicated to the driver to supply jumbo Rx BDs to the chip, if using jumbo frames). In RSS mode, there are up to 4. In IOV mode, there are up to
16 (17, including the default ring).
The Tx path is simpler. Tx is 1 or 4 or 16 rings (legacy, TSS, or IOV) for giving Send BDs to the chip. Jumbo BDs go onto the same Send ring as non-jumbos. There are no separate jumbo-dedicated Send ring(s) like there are for Rx producer rings.
The max sizes of the various rings is as follow:
• Rx Ret: 4096
• Rx Prod: 2048
• Rx Prod Jumbo: 1024
• Tx Prod: 512
As a maximum use example, there can be an IOV implementation with the following total max ring size/count arrangement:
• 16 Send rings (size=512 each ring)
• 16 Rx Prod rings (size=2048 each ring)
• 16 Rx Prod jumbo rings (size=1024 each ring)
• 17 Rx Ret rings (size=4096 each ring)
Status Block
The Status Block is another shared memory data structure that is located in host memory. The Status Block is
32 bytes in length. Host software will need to allocate 32 bytes of non-paged memory space for the Status Block and set the Status Block Host Address register to point to the host memory physical address reserved for this structure.
The controller will update the Status Block to host memory prior to a host coalescing interrupt or MSI. The frequency of these Status Block updates is determined by the host coalescing logic (see
the frequency of status block updates for a particular application or operating system.
of the controller is regarding its processing of the various send and receive rings. From information in the status block a software driver can determine:
• Whether the Status Block has been recently updated (via a bit in the status word).
• Whether the Link State has changed (via a bit in the status word).
• Whether the controller has recently received a packet and deposited that packet into host memory for a given ring (via the Receive Return Ring Producer Indices).
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 83
BCM5718 Programmer’s Guide Status Block
• Which host receive descriptors that controller has fetched, and it will consume when future packets are received (via the Receive Producer Ring Consumer Indices).
• Whether the controller has recently completed a transmit descriptor buffer DMA for a given ring (via the
Send Ring Consumer Indices).
Status Block Format
Note: Reference registers 0x3C50 – 0x3CC0 for debug access to the various ring indices.
Each MSI-X vector is associated with a status-block structure. A status block is DMAed to the host memory immediately prior to raising a legacy style interrupt (INTx, MSI) or MSI-X interrupt. Status block formats vary depending on RSS and the MSI-X vector number.
The various status block formats are shown in this section.
INTx/MSI — Legacy Mode Status Block Format
INTx and MSI use this status-block format in non-RSS mode.
Table 14: Status Block Format (MSI-X Single-Vector or INTx — RSS Mode)
Offset 31 16 15
0x00 Status Word
0x04 [31:8] Reserved 0x0
0x08 Receive Standard Producer Ring Consumer
Index
0x0C Reserved 0x0
0x10 Send BD Consumer Index
0x14 Reserved 0x0
Reserved 0x0
Reserved 0x0
[7:0]Status Tag
Receive Return Ring Producer Index
Receive Jumbo Producer Ring Consumer Index
0
Status-Block [0] Status Word Format (single-vector RSS):
• Bit [0]: Update bit
• Bit [1]: Link status change
• Bit [2]: Error/attention
• Bits [31:3]: Reserved 0x0
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 84
BCM5718 Programmer’s Guide Status Block
Single-Vector or INTx — RSS Mode Status Block Format
In the single-vector RSS mode, the status-block format used by Vector#0 is shown below.
INTx and MSI also use this status-block format.
Table 15: Status Block Format (MSI-X Single-Vector or INTx — RSS Mode)
Offset 31 16 15
0x00 Status Word
0x04 [31:8] Reserved 0x0
0x08 Receive Standard Producer Ring Consumer
Index
0x0C Receive Return Ring 2 Producer Index
0x10 Send BD Consumer Index
0x14 Reserved 0x0
[7:0]Status Tag
Receive Return Ring 1 Producer Index
Receive Return Ring 3 Producer Index
Receive Return Ring 0 Producer Index
Receive Jumbo Producer Ring Consumer Index
0
Status-Block [0] Status Word Format (single-vector RSS):
• Bit [0]: Update bit
• Bit [1]: Link status change
• Bit [2]: Error/attention
• Bits [31:3]: Reserved — always 0x0
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 85
BCM5718 Programmer’s Guide Status Block
Multivector RSS Mode Status Block Format
There are five slightly different status-block formats used by the multivector RSS mode. Each of these formats associate with their respective vector numbers as shown in the tables below.
Table 16: Status Block [0] Format (MSI-X Multivector RSS Mode)
Offset 31 16 15
0x00 Status Word
0x04 [31:8] Reserved 0x0
0x08 Receive Standard Producer Ring Consumer
Index
0x0C Reserved 0x0
0x10 Send BD Consumer Index
0x14 Reserved 0x0
Reserved 0x0
Reserved 0x0
Reserved 0x0
[7:0]Status Tag
Receive Jumbo Producer Ring Consumer Index
0
Status-Block [0] Status Word Format (multivector RSS):
• Bit [0]: Update bit
• Bit [1]: Link status change
• Bit [2]: Error/attention
• Bits [3]: Reserved — always 0
• Bits [4]: Reserved — always 0
• Bits [5]: Reserved — always 0
• Bits [31:6]: Reserved 0x0
Table 17: Status Blocks [1 thru 4] Formats (MSI-X Multivector RSS Mode)
Offset 31 16 15
0x00 Status Word {Valid for all Status Blocks}
0x04 [31:8] Reserved 0x0
0x08
0x0C
0x10
0x14
Reserved 0x0
Receive Return Ring 2 Producer Index
Valid only for Status Block3 else RSVD 0x0
Reserved 0x0
Reserved 0x0
0
[7:0] Status Tag[n]
{independent for each status blocks}
Receive Return Ring 1 Producer Index
Valid only for Status Block2 else RSVD 0x0
Receive Return Ring 3 Producer Index
Valid only for Status Block4 else RSVD 0x0
Receive Return Ring 0 Producer Index
Valid only for Status Block1 else RSVD 0x0
Reserved 0x0
Status-Block [1 – 4] Status Word Format (multivector RSS):
• Bit [0]: Update bit
• Bit [31:1]: Reserved 0x0
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 86
BCM5718 Programmer’s Guide Status Block
Status Block and INT MailBox Addresses
Each status block may be placed in an independent host memory address (64-bit). Each vector may be acknowledged via associated INT MailBoxes.
0
1
2
3
4
5
Table 18: Status Block Host Addresses and INT MailBox Addresses
RSS Mode
Status Block
#
Legacy
Status Block Host
Address Register
(64-Bit)
INT MailBox
Register
Address
0x3C3C, 0x3C38 0x200 x3C3C, 0x3C38 0x200
0x3D00, 0x3D04 0x208
0x3D08, 0x3D0C 0x210
0x3D10, 0x3D14 0x218
0x3D18, 0x3D1C 0x220
0x3D20, 0x3D24 N/A
Indication
Items
All
Link-Status change
Error/Attention
SBD Ring 1 Cons Index
Std RBD Cons Index
Jmb RBD Cons Index
Rx Return Ring 0
Prod Index
Rx Return Ring 1
Prod Index
Rx Return Ring 2
Prod Index
Rx Return Ring 3
Prod Index
N/A
Comments
Legacy status block
Used by INTx or MSI
Used in all MSI-X modes for
Vector#0
Used only in MSI-X multivector
RSS mode or multivector EAV mode for Vector#1 – Vector#4
Used only in MSI-X multivector
EAV mode for Vector#5
The Status word field contains bit flags that contain error information about the status of the controller. The defined flags are listed in
Table 19: Status Word Flags
Bits Name
0
1
Updated
Description
This bit is always set to 1 each time the status block is updated in the host via
DMA. It is expected that host software clear this bit in the status block each time it examines the status block. This provides the host driver with a mechanism to determine whether the status block has been updated since the last time the driver looked at the status block.
Link State Changed Indicates that link status has changed. This method of determining link change status provides a small performance increase over doing a PIO read of the
Ethernet MAC Status register (see “EMAC Status Register (offset: 0x404)” on page 312 . See
“Wake on LAN Mode/Low-Power” on page 213 for a description
of the PHY setup required when link state changes.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 87
BCM5718 Programmer’s Guide Status Block
Bits
2
Name
Error
Table 19: Status Word Flags (Cont.)
Description
When this bit is asserted by the chip, the following conditions may have occurred. Bit 2 of the status word is the OR of:
• All bits in Flow Attention register (0x3c48) (see “Flow Attention Register
.
• MAC_ATTN—Events from the MAC block (see
(offset: 0x404)” on page 312 .
• DMA_EVENT—Events from the following blocks:
– MSI (see “MSI Status Register (offset: 0x6004)” on page 470
.
– DMA_RD (see
“LSO Read DMA Status Register (offset: 0x4804)” on page 437 .
– DMA_WR (see “Write DMA Status Register (offset: 0x4C04)” on page 453 .
• RXCP_ATTN—Events from RX RISC (see
“RX RISC Status Register (offset:
The Status Block format for these devices is as follows:
• Status Tag — Contains an unique eight-bit tag value in bits 7:0 when the Status Tagged Status mode bit of
) is set to 1. This Status Tag can be returned to the Mailbox 0 register at location 31:24
Mailbox 0 register bits 23:0 are written as 0, the tag field of the Mailbox 0 register is compared with the tag field of the last status block to be DMAed to host. If the tag returned is not equivalent to the tag of the last status block DMAed to the host, the interrupt state is entered.
• Receive Producer Ring Consumer Index — Contains the controller’s current Consumer Index value for the
Receive Producer Ring. This field indicates how many receive descriptors are in the receive producer ring
• Receive Return Rings 0–3 Producer Indices — Contain controller’s current Producer Index value for the each of the Receive Return Rings. When the controller receives a packet and writes that packet data into host memory via DMA, it will increment the Producer Index for the corresponding Receive Return ring.
When a Producer Index is incremented, it is a signal to software that a newly arrived receive packet is ready to be processed.
• Send Ring Consumer Index — Contains controller’s current Consumer Index value for the Send Ring. When the controller completes the read DMA of the host buffer associated with a send BD, the controller will update the Send Ring Consumer Index. This provides the host software with an indication that the controller has buffered this send data and, therefore, the host software may free the buffer that was just consumed by the device.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 88
BCM5718 Programmer’s Guide
Section 5: Receive Data Flow
Receive Data Flow
Introduction
The RX MAC pulls BDs from RX producer rings. The RX BD specifies the location(s) in host memory where packet
data may be written. Figure 14 on page 90 shows the receive buffer descriptor cycle.
All ingress Ethernet frames are classified by the RX rules engine. A class ID is associated to each frame based on QOS rules setup in the RX MAC (see
“Receive Rules Setup and Frame Classification” on page 96
). The Receive
List Placement and Receive List Initiator portions of the MAC architecture move BDs to the RX return rings; the class ID associated to the packet is examined to route the BD to a specific RX return ring.
Once the packet is queued to the RX return ring, the device driver will wait for indication of the same through the status block update and interrupt from the host coalescing engine. The host coalescing engine will update the status block and generate a line interrupt or MSI (see
“Host Coalescing” on page 235 for further details
regarding interrupts) when a specified host coalescence criteria is met. Once the interrupt is generated, the host device driver will service the interrupt. The ISR will determine if new BDs have been completed on the RX
Return Rings. Next, the device driver will indicate to the network protocol that the completed RX packets are available. The network protocol will consume the packets and return physical buffers to the network driver at a later point.
The BDs may then be reused for new RX frames. The device driver must return the BD to an RX producer ring.
For this purpose, the driver should fill out either the opaque field or index field of the Rx BD when inserting/ initializing the BD in an RX Producer ring. When the BD is returned by the device through Return Ring, the opaque or index data field of the BD will be used by the driver to identify the BD in Producer Ring that corresponds to the Returned BD in Return Ring. The device driver will then reinitialize the identified BD in
Producer Ring with a new allocated buffer and replenish the Receive Producer Ring with this BD.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 89
BCM5718 Programmer’s Guide
Figure 14: Receive Buffer Descriptor Cycle
RX
Jumbo
Producer
Ring
RX
Standard
Producer
Ring
Introduction
MAC
DMA Read Engine
Rx MAC
List
Initiator
Local
Memory
Host
Coalescing
Engine
DMA Write Engine
RX Return Packet
Device Driver
Interrupt
Service
Routine
Protocol Interface
(i.e. TCP/IP)
RX Indicate Available
RX
Return
Ring 3
RX
Return
Ring 4
RX
Return
Ring 1
RX
Return
Ring 2
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 90
BCM5718 Programmer’s Guide Receive Producer Ring
Receive Producer Ring
A Receive Producer Ring is an array containing a series of Receive Buffer Descriptors (BD). The Receive Producer
Ring is host-based and a portion of the available buffer descriptors are cached in Ethernet controller internal memory.
A receive producer ring contains a series of buffer descriptors which in turn contain information of host memory locations to where packets are placed by the Ethernet controller at reception.
Setup of Producer Rings Using RCBs
A Ring Control Block (RCB) is used by the host software to set up the shared rings in host memory. In the context of producer ring, the Receive Producer Ring RCB is a set of registers (or internal device memory offsets) used to define the Receive Producer Ring. The host software must initialize the Receive Producer Ring RCB.
Receive Producer Ring RCB—Register Offset 0x2450–0x245f
Other Considerations Relating to Producer Ring Setup
Other registers that affect the producer rings must be initialized by the host software. These registers include the Receive BD Ring Replenish Threshold register, the Receive MTU register, and the Accept Oversized bit (bit
5) in the Receive MAC Mode register.
• Receive BD Producer Ring Replenish Threshold registers:
–
“Standard Receive BD Producer Ring Replenish Threshold Register (offset: 0x2C18)” on page 375
–
“Receive Data Completion Control Registers” on page 373 .
These registers are used for setting the number of BDs that the Ethernet controller can use up before requesting that more BDs be DMAed from a producer ring. In other words, the device does not initiate a
DMA for fetching the Rx BDs until the number of BDs made available to the device by the host is at least the value programmed in this register.
• Receive MTU register ( “Receive MTU Size Register (offset: 0x43C)” on page 317
). This 32-bit register is set to a value that is the maximum size of a packet that the Ethernet controller receives. Any packets above this size is labeled as an oversized packet. The value for this register is typically set to 1518, which is the
Standard Producer Ring RCB typical setting. If Jumbo frames are supported, the MTU would be set to the maximum Jumbo frame size. The BCM5717 does not support jumbo frames.
• Receive MAC Mode register ( “Receive MAC Mode Register (offset: 0x468)” on page 323
). If the Accept
Oversized bit (bit 5) of this register is set, the Ethernet controller accepts packets (of size up to 64 KB) larger than the size specified in the MTU.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 91
BCM5718 Programmer’s Guide Receive Producer Ring
RCB Setup Pseudo Code
An example of setting up receive producer ring RCB:
Content of Pointer_to_RX_PRODUCER_RING_RCB + 0x00 = Host address of standard receive producer ring high 32.
Content of Pointer_to_RX_PRODUCER_RING_RCB + 0x04 = Host address of standard receive producer ring low 32.
Content of Pointer_to_RX_PRODUCER_RING_RCB + 0x0a = No flags.
Content of Pointer_to_RX_PRODUCER_RING_RCB + 0x08 = Max packet size of 1518.
Content of Pointer_to_RX_PRODUCER_RING_RCB + 0x0c = Internal Memory address for device copy of ring.
Figure 15 shows the standard ring RCB for the setup of a host-based standard producer ring.
Receive Buffer Descriptors (BDs) begin on the Receive Producer Ring. The host device driver will populate the
) specifies the largest packet accepted by the RX MAC; packets larger than the Receive MTU are marked oversized and are discarded.
Figure 15: Receive Producer Ring RCB Setup
Offset
0x00
0x04
0x08
0x0c
31 16
Host Ring Address
Max_Len
15 0
Flags
NIC Ring Address
511
512
BD 1
BD 2
BD 3
Std Ring
Standard Producer Ring RCB
Standard Producer Ring
Receive Buffer Descriptors
The Receive Buffer Descriptor is a data structure in host memory. It is the basic element that makes up each receive producer and receive return ring. The format of receive buffer descriptors can be seen in
. A receive buffer descriptor has a 64-bit memory address and may be in any memory alignment and may point to any byte boundary. For performance and CPU efficiency reasons, it is recommended that memory be cache-aligned. The cache line size value is important for the controller to determine when to use the PCI memory write and invalidate command. There are no requirements for memory alignment or cache line integrity for the Ethernet controller.
Unlike send buffer descriptors, the receive buffer descriptors cannot be chained to support multiple fragments.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 92
BCM5718 Programmer’s Guide Receive Producer Ring
Management of Rx Producer Rings with Mailbox Registers and Status
Block
Status Block
The host software manages the producer rings through the Mailbox registers and by using the status block. It does this by writing to the Mail Box registers when a BD is available to DMA to the Ethernet controller and reading the status block to see how many BDs have been consumed by the Ethernet controller. The status block
can be seen in “Status Block” on page 83 .
The status block is controlled and updated by the Ethernet controller. The status block in host memory is constantly updated through a DMA copy by the Ethernet controller from an internal status block. The updates occur at specific intervals and host coalescence conditions that are specified by host software during initialization of the Ethernet controller. The registers for setting the intervals and conditions are in the Host
Ethernet controller DMAs an updated status block to the 32-bit address that is set by the host software in the
Host Coalescing Control registers, 0x3c38.
Among other status, the status block displays the last 16-bit value, BD index that was DMAed to the Ethernet controller from receive producer ring. The Ethernet controller updates these indices as the recipient or consumer of the BD from the producer rings.
Mailbox
The host software is responsible for writing to the Mailbox registers (see
Table 20 on page 93 ) when a BD is
available from the producer rings for use by the Ethernet controller. Host software should use the high-priority mailbox region from 0x200–0x3FF for host standard and the low-priority mailbox region from 0x5800–0x59FF for indirect register access mode.
The Mailbox registers (starting at memory offset 0x200 for host standard and offset 0x5800 for indirect mode) contain the following receive producer index register.
Receive BD Producer Ring Producer Index
• Host standard: memory offset 0x268–0x26F
• Indirect mode: memory offset 0x5868–0x586F
Table 20: Mailbox Registers
Offset
(High-Priority
Offset
(Low-Priority
Mailboxes for Host Mailboxes for
Standard Mode) Indirect Mode) Register
0x200–0x207
0x208–0x20F
0x268–0x26F
0x280–0x287
0x5800–0x5807
0x5808–0x580F
Interrupt Mailbox 0
Interrupt Mailbox 1
Access
RW
RW
0x5868–0x586F Receive BD Standard Producer Ring Producer Index RW
0x5880–0x5887 Receive BD Return Ring 1 Consumer Index RW
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 93
BCM5718 Programmer’s Guide Receive Return Rings
0x288–0x28F
0x290–0x297
0x5888–0x588F
0x5890–0x5897
Table 20: Mailbox Registers (Cont.)
Offset
(High-Priority
Offset
(Low-Priority
Mailboxes for Host Mailboxes for
Standard Mode) Indirect Mode) Register
Receive BD Return Ring 2 Consumer Index
Receive BD Return Ring Consumer Index
Access
RW
RW
The Receive Producer Ring Producer Index register contains the index value of the next buffer descriptor from the producer ring that is available for DMA to the Ethernet controller from the host. When the host software updates the Receive Producer Ring Producer Index, the Ethernet controller is automatically signaled that a new
BD is waiting for DMA. At initialization time, these values must be initialized to zero. These indices are 64-bit wide.
Receive Return Rings
Receive Return Rings (RR) are host-based memory blocks that are used by host software to keep track of the where the Ethernet controller is putting the received packets related receive buffer descriptors. Unlike the producer rings, the return rings reside only in host memory. The Ethernet controller uses the BDs in the NIC memory that are previously copied from the producer rings to use when packets are received from the LAN. It places the BDs that correspond to received packets in the return rings.
Return rings are the exact opposite of producer rings, except that they are not categorized by the maximum length receive packets supported. They are actually categorized by priority or class of received packet. The highest priority return ring is ring 1, and the lowest priority is the last ring (Return Ring 2–Return Ring 4 depending on how many rings are set up by the host software).
The Receive Return Ring RCBs are used to set up return rings in much the same way the Receive Producer Ring
RCB is used to set up the receive producer ring. These RCBs for the return rings are set in the Miscellaneous memory region (SSRAM) at offset 0x200 (this region should not be confused with the register space in the chip). The RCB max_len field is used to indicate the number of buffer descriptor entries in a return ring. If an invalid value is set, the Ethernet controller indicates an attention error in the Flow Attention register.
Figure 13 on page 78 shows receive return rings.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 94
BCM5718 Programmer’s Guide Receive Return Rings
Management of Return Rings with Mailbox Registers and Status Block
The return rings are managed by the host using the Mailbox registers and status block.
When a packet is received from the LAN, the Ethernet controller DMAs the packet to a location in the host, and then DMAs the related BD to a return ring. As the producer of this packet to the host, the Ethernet controller updates the status block producer indices for the related return ring (i.e., return ring 1 to return ring 4 that was
DMAed the BD received packet). These return ring indices can then be read by the host software to determine the last BD index value of a particular ring that has information of the last received packet.
As the consumer of the received packet, the host software must update the return ring consumer indices in
Mailbox registers Receive BD Return Ring 1 Consumer Index (memory offset 0x280–0x287 for host standard and 0x5880–0x5887 for indirect mode) through Receive BD Return Ring 4 Consumer Index.
Host Buffer Allocation
The allocation of memory in the host is dependent on the operating system in which the controller is being used. There are two crucial items:
• The use of non-cached and physically contiguous memory is best for adapter performance.
• Physical memory mapping is required for the controller’s internal copies of logical host memory.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 95
BCM5718 Programmer’s Guide Receive Return Rings
Receive Rules Setup and Frame Classification
The Ethernet controller has a feature that allows for the classification of receive packets based on a set of rules.
The rules are determined by the host software and then input into the Ethernet controller.
A packet can be accepted or rejected based on the rules initialized into two rules register areas. The packets can also be classified into groups of packets of higher to lower priority using the rules registers. This occurs when the packet is directed to a specific return ring. Return rings 1–4 have an inherent priority associated with them. The priority is from lowest ring number to highest ring number; return ring 1 being the highest priority ring and return ring 4 being the lowest. The implementation of priority class is based on how many rings the host software has initialized and made available to the Ethernet controller. As packets arrive, the Ethernet controller may classify each packet based on the rules. When the host services the receive packet, it can service the lower numbered rings first.
A rule can be changed by first disabling it by setting 0 into Enable bit (bit 31) in Receive BD Rules Control register
). Wait about 20 receive clocks (rx_clock) and then reenable it when it is programmed with a new rule. Otherwise, changing the rules dynamically during runtime may cause the rule checker to output erroneous results because the rule checker is a pipelined design and uses the various fields of the rules at different clock cycles.
Receive Rules Configuration Register
the ring where a received packet should be placed into if no rules are met, or if the rules have not been set up.
A value of 0 means the received packet will be discarded. A value of 1–16 specifies a corresponding ring. This ring should be initialized to at least a value of 1 if the rules are not being used to ensure that all received packets will be DMAed to return ring 1.
Bits
31:8
7:3
2:0
Table 21: Receive Rules Configuration Register
Field
Reserved
Specifies the default class (ring) if no rules are matched
Reserved
Access
RO
RW
RO
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 96
BCM5718 Programmer’s Guide Receive Return Rings
Receive List Placement Rules Array
The Receive List Placement Rules Array (memory offset 0x480–0x4ff) is made up of 16 combined element registers. The combined element is actually two 32-bit registers called the Receive BD Rules Control register
) and the Receive BD Rules Value/Mask register (see
Table 24 ). The element can be looked at as a
single 64-bit entity with a Control part and Value/Mask part since they use a single element. Bit 26 of the control part determines how the value/mask part is used. The Receive BD Rules Value/Mask register can be used as either a 32-bit left-justified Value or a 16-bit Mask followed by a 16-bit Value.
Note: Receive rules cannot be used to match VLAN headers because the VLAN tag is stripped from the Ethernet frame before the rule checker runs.
31
E
30
&
29
P1
15 14
Header
13
28
P2
12
27
P3
Table 22: Receive BD Rules Control Register
26
M
25
D
24
Map
23 22 21 20
Reserved
11 10
Class
9 8 7 6 5
19
4
Offset
3
18
2
17
Op
16
1 0
Table 23: Receive List Placement Rules Array (memory offset 0x480–0x4ff)
Bit Name RW Description Default
31
30
29
28
27
26
E
&
P1
P2
P3
M
RW Enable. Enabled if set to 1 –
RW And With Next. This rule and next must both be true to match. The class fields must be the same. A disabled next rule is considered true.
Processor activation bits are specified in the first rule in a series.
–
RW If the rule matches, the processor is activated in the queue descriptor for the Receive List Placement state machine.
–
RW If the rule matches, the processor is activated in the queue descriptor for the Receive Data and Receive BD Initiator state machine.
–
RW If the rule matches, the processor is activated in the queue descriptor for the Receive Data Completion state machine.
–
RW Mask If set, specifies that the value/mask field is split into a 16-bit value and 16-bit mask instead of a 32-bit value.
–
25
24
D
Map
RW Discard Frame if it matches the rule.
RW Map Use the masked value and map it to the class.
23:18 Reserved RW Must be set to zero.
17:16 Op RW Comparison Operator specifies how to determine the match:
• 00 = Equal
• 01 = Not Equal
• 10 = Greater than
• 11 = Less Than
0
–
–
–
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 97
BCM5718 Programmer’s Guide Receive Return Rings
Table 23: Receive List Placement Rules Array (memory offset 0x480–0x4ff) (Cont.)
Bit Name RW Description Default
15:13 Header RW Header Type specifies which header the offset is for:
• 000: Start of Frame (always valid)
• 001: Start of IP Header (if present)
• 010: Start of TCP Header (if present)
• 011: Start of UDP Header (if present)
• 100: Start of Data (always valid, context sensitive)
• 101–111: Reserved
12:8 Class
7:0 Offset
–
RW The class this frame is placed into if the rule matches. 0:4, where 0 means discard. The number of valid classes is the Number of Active Queues divided by the Number of Interrupt Distribution Groups. Ring 1 has the highest priority and Ring 4 has the lowest priority.
–
RW Number of bytes offset specified by the header type.
–
31 30 29
15 14 13
28
12
Table 24: Receive BD Rules Value/Mask Register
27 26 25 24
Mask
23 22 21 20
11 10 9 8
Value
7 6 5 4
19
3
18 17 16
2 1 0
Bit Name
31:16 Mask
15:0 Value
RW Description
–
–
–
–
Default
–
–
Class of Service Example
If either Start of IP Header, Start of TCP Header, or Start of UDP Header is specified, and the frame has no IP,
TCP, or UDP header, respectively, there is no frame match. The full set of rules provides a fairly rich selection and filtering criteria.
Example: If you wanted to set a Class of Service (CoS) of 2 based on the eighth byte in the data portion of an encapsulated IPX frame using Ethernet Type 2 having a value greater than 6, you could set up the rules
.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 98
BCM5718 Programmer’s Guide
Figure 16: Class of Service Example
Rule 1: Control = 0xc400020C
Where: Enable + And with next (chain with next rule)
Mask -Value/Mask is split into two 16-bit values
Comparison Operator –Equal
Header Type – Start of Data
Class -Return Ring 2
Offset -12 bytes from start of frame
Mask/Value
Where:
= 0xffff 8137
Mask – 0xffff
Value - IPX
Rule 2: Control = 0x84028207
Where: Enable
Mask – Value/Mask split into two 16-bit values
Comparison Operator –Greater Than
Header Type – Start of Data
Class -Return Ring 2
Offset – 7 bytes from start of data
Mask/Value
Where:
=0xff00 0600
Mask – 0xff00
Value – 0600
Checksum Calculation
Checksum Calculation
Whether the host software NOS supports checksum offload or not, the Ethernet controller automatically calculates the IP, TCP, and UDP of received packets as described in RFC 791, RFC 793, and RFC 768, respectively.
Which protocol checksum value is produced can be determined by reading the status flag field in the Receive
Return Ring. The valid flag values in the status flag field are IP_CHECKSUM and TCP_UDP_CHECKSUM. When a valid checksum is produced, the values of the checksums are found in the corresponding receive buffer descriptor register. These values should be 0xFFFF for a valid checksum or any other value if the checksum was incorrectly calculated. Assert the Receive No Pseudo-header Checksum bit of the Mode Control register (see
“Mode Control Register (offset: 0x6800)” on page 472
) to not to include Pseudo-header in TCP/UDP checksums.
VLAN Tag Strip
Receiving VLAN-tagged (IEEE 802.1q-compliant) packets are automatically supported by the Ethernet controller. There is no register or setting required to receive packets that are VLAN-tagged. The VLAN tag is automatically stripped from the IEEE 802.1q-compliant packet at reception and then placed in a receive buffer descriptor’s two byte VLAN tag field. The flag field has the BD_FLAGS_VLAN_TAG bit set when a valid VLAN packet is received. After the packet has been serviced by the host software, these fields should be zeroed out.
In the Receive MAC Mode register (offset 0x468–0x46b), the Keep VLAN Tag Diag Mode bit (bit 10) can be set to force the Ethernet controller to not strip the VLAN tag from the packet. This is only for diagnostic purposes.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 99
BCM5718 Programmer’s Guide
shows the frame format with IEEE 802.1Q VLAN tag inserted.
Offset
0:5
6:11
12:13
14:15
16:17
18:1517
Table 25: Frame Format with 802.1Q VLAN Tag Inserted
Description
MAC destination address
MAC source address
Tag Protocol ID (TPID)—0x8100
Tag Control Information (TCI):
• Bit 15:13—IEEE 802.1P priority
• Bit 12—CFI bit
• Bit 11:0—VLAN ID
The original EtherType
Payload
VLAN Tag Strip
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 100
BCM5718 Programmer’s Guide RX Data Flow Diagram
RX Data Flow Diagram
The receive data flow can be summarized in
Figure 17 . The Receive Producer Ring, Receive Buffer Descriptors,
Receive Return Rings, Mailbox registers, and status block registers are the main areas of the receive data flow.
Figure 17: Overview Diagram of RX Flow
Buffer Descriptor points to free RX buffer in host
BD n
Standard and Jumbo Producer Rings
In Host Memory
2
MailBox Registers
Rcv BD Std Producer Ring Index
3
Status Block status word rcv std cons unused unused unused
TX cons #1 unused
RX prod #1
1
4
BCM570X
Family Receive Return Rings in host memory
6
Network
5
Host Memory
BD n
Used Buffer Descriptor points to host memory where packet was copied
The RX flow sequence is as follows:
1. The host software updates a Receive Producer Ring Index in the Mailbox registers.
2. A receive BD or series of BDs with the corresponding index is DMAed to the Ethernet controller from the host-based Receive Producer Ring.
3. The Ethernet controller updates the Receive Consumer Index in the Host Block register and stores copy of the BD.
4. A valid Ethernet packet is received from the network into the device.
5. The Ethernet packet is DMAed to host memory using a BD previously DMAed from a Receive Producer Ring.
6. The BD used for the received packet is DMAed from the Ethernet controller to one of the receive return rings, and the Receive Return Ring Producer Index register in the host status block is updated by the
Ethernet controller.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 101
BCM5718 Programmer’s Guide Receive Side Scaling
The host software must create an array of BD structures in host memory, referred to as a receive producer ring.
Each receive buffer descriptor within a producer ring describes, among other things, the location of a host memory buffer that is used to store the packets received from the network. When the host software (as the producer) updates the mailbox register’s producer ring index that corresponds to the receive producer ring, the Ethernet controller automatically DMAs the BD to itself from the host. When the DMA is completed, the
Ethernet controller (as the consumer) updates the status block’s receive consumer ring index to signal it successfully consumed the BD. The Ethernet controller keeps this BD in internal memory to know where to put a packet that is received from the network.
When a packet is received from the network, a BD gets updated with information regarding the received packet and the packet is DMAed to a location in host memory described by the BD. The Ethernet controller (as the producer) then updates the receive return ring producer index in the Status Block register corresponding to one of host memory’s receive return rings, and DMAs the BD to that receive return ring.
It is the responsibility of the host software to setup, initialize, and manage the data structures in host memory, namely, the receive producer rings and the receive return rings. The producer/consumer indices in the mailbox and status block are read and updated by the host and Ethernet controller for this purpose.
Receive Side Scaling
Overview
RSS is a scalable networking technology that enables receive packet processing to be balanced across multiple processors in the system while maintaining in-order delivery of the data. The RSS enables packets from a single network adapter to be processed in parallel on multiple CPUs/cores while preserving in-order delivery to TCP connections.
Functional Description
The figure below shows the processing of received packets when RSS is enabled. The RSS algorithm is based on a load-sharing algorithm and performs the following steps.
• Computes a hash on the incoming packet to produce a 32-bit Hash result.
• Performs a lookup in the load balancing table (also called indirection table) using the one to seven least significant bits of the Hash result to determine which of the n CPUs are processing the packet, where n is the number of CPUs assigned to process received packets.
• Adds a Base CPU Number to determine the exact CPU that will process the packet.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 102
BCM5718 Programmer’s Guide Receive Side Scaling
Figure 18: RSS Receive Processing Sequence
Base CPU Number
Indirection
Table
CPU
Result
Masked Hash
Result 1-7
Bits
+
Hash
Mask
32-Bit Hash Result
Hash Result
(32 Bits)
Hash
Function
Incoming Packets
Hash Type
The devices implement the above RSS algorithm in hardware except for the step of adding the Base CPU
Number to the value from Indirection Table. If required, the step of adding the Base CPU Number to the CPU
Result can be done in the main Interrupt Service Routine to determine which CPU will process the packet.
RSS Parameters
Hash Function
The default hash function is the Toeplitz hash. No other hash functions are currently supported, so there is no configurable parameter.
Hash Type
The fields that are used to hash across the incoming packet. The 5 devices support all the four hash types given below and the configuration bits for enabling/disabling these hash types are provided in Receive MAC Mode register at offset 0x468. Any combination of these hash types can be enabled:
• Four-tuple of source TCP Port, source IP version 4 (IPv4) address, destination TCP port, and destination
IPv4 address.
• Four-tuple of source TCP Port, source IP version 6 (IPv6) address, destination TCP port, and destination
IPv6 address.
• Two-tuple of source IPv4 address and destination IPv4 address.
• Two-tuple of source IPv6 address and destination IPv6 address.
Note: The BCM5719 and BCM5720 add support for UDP RSS hash functionality, in addition to the legacy TCP RSS hash functionality.
Hash Mask
The RSS Hash Mask bits (bits 22:20 of the Receive MAC Mode register at offset 0x468) allow the configuration of number of hash-result bits that are used to index into the indirection table.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 103
BCM5718 Programmer’s Guide Receive Side Scaling
Indirection Table
The table of CPU numbers used for balancing the receive traffic across multiple processors. The Indirection
Table registers 0–15 at offset 0x630–0x66F are implemented for the required 128 entries of the Indirection
Table. The devices support only four Receive Return Rings so each entry of Indirection Table is implemented as
2 bits.
Secret Hash Key
The hash key that will be used for RSS hash. For the Toeplitz hash, the hash key size is 40 bytes for IPv6 and 16 bytes for IPv4. The host software should program the hash key in hash key registers at offset 0x670 to 0x697.
RSS Initialization
The host protocol stack should configure the above RSS parameters before enabling the RSS engine. The RSS can be enabled by setting the bit-23 of the Receive MAC Mode register at offset 0x468. Normally the RSS parameters except the Indirection Table are static and will be initialized only during device driver initialization.
Though extremely rare, the protocol stack may change the RSS parameters any time. The devices require a reset to change any of the hash type, hash mask, and hash key parameters.
If the hash type flags in Receive MAC Mode register (offset 0x468) enable only one type of hash, then any received packet that does not match the enabled hash type is not hashed. If multiple flags are set, such as If the TCP/IPv4 and IPv4 hash types (bits 17 and 16 of Receive MAC Mode register at offset 0x468) are enabled, then if the packet is not a TCP/IPv4 packet but is an IPv4 packet, the hash is performed on just the IPv4 2-tuple.
Further, for this setting of the hash type flags, if the incoming packet is not an IPv4 packet, then no hash is performed. Because a variety of hash types can be applied on a per-packet basis (including no hash), the hash type is indicated to the host protocol stack on a per-packet basis. If no hash was performed, then none of the hash type flags in the receive BD will be set.
Once RSS is initialized and enabled, data transfer can begin. Over a period of time, the host protocol stack may modify the indirection table to rebalance the processing load. When the indirection table is changed, it is possible for a short period of time (while the current receive descriptor queues are being processed) for packets to be processed on the wrong CPU. This is a normal transient condition and should not be a problem.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 104
BCM5718 Programmer’s Guide Receive Side Scaling
RSS Rx Packet Flow
Each CPU or CPU core in multiprocessor systems is assigned one receive return ring. Only a single interrupt is initiated at a time.
1. As packets arrive, the device parses each packet, calculates the RSS Hash, and derives the CPU number (i.e., receive return ring number) from Indirection Table using the Masked Hash Result as the Indirection Table
Index.
2. The packet data is DMAed to the host memory at the location specified by the receive buffer descriptor
(RBD) of the receive producer ring.
3. Based on the derived CPU number, the device DMAs the used RBDs into appropriate receive return rings in host memory.
4. The device fires the interrupt via MSI, which causes the device driver ISR to run.
5. The ISR disables further interrupts from the device, determines which CPUs have receive packets to be handled and uses inter processor communication mechanisms to start packet receive handlers on CPUs whose return rings have new RBDs.
6. Each CPU processes the new RBDs in it’s receive return ring when its packet handler routine is started by main ISR.
7. Once the main ISR determines that all new RBDs have been processed by the CPUs, it enables the interrupts from the device and exits.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 105
BCM5718 Programmer’s Guide
Section 6: Transmit Data Flow
Transmit Data Flow
Introduction
Send Buffer Descriptors (BDs) begin on the Send Producer rings. The device driver updates the Mailbox to reflect available Send BDs.
• The MAC moves the available Send BDs to device local memory—a cache.
• Next, the MAC selects a BD from the internal cache using priority scheduling.
The physical address, programmed in the Send BD by the host device driver prior to the Mailbox update, contains the host memory location of the TX packet buffer. The MAC reads the address from Send BD and schedules a bus master DMA for reading the packet data from host buffer. The packet data will be moved into device internal buffers from host buffers by Read DMA engine, and all the read buffers of 1 packet are chained together into a cluster. This cluster is then sent to the transmit MAC which sends the packet data to the integrated PHY for transmission on Ethernet media.
The write DMA engine will subsequently update the status block to indicate that the Send BD was consumed.
The host driver normally returns the packet buffers to the NOS/protocol so the next packet can reuse that host physical memory. The send BD is now available for the next TX packet.
Send Rings
The send rings are shared data structures that are used to describe a series of data buffers that will be transferred onto the network. The shared data structure is called the Ring Control Block (RCB), and the entries within a ring for describing the data buffers are called the Send Buffer Descriptors (Send BDs).
Note: The maximum number of Send BDs (buffer descriptors) for a single packet is (0.75)*(ring size).
Associated with each ring are two indices that control its operation. These indices are the producer index and the consumer index, which are not shared between the host software and the Ethernet controller. In the case of send rings, the host software controls the producer index by adding elements (initializing a Send BD) to the ring. Similarly, the Ethernet controller controls the consumer index by removing elements (processing a Send
BD) from the ring.
The host software is responsible for maintaining its producer index and updating it by writing to the send ring producer index mailbox register. The mailbox registers are described in registers
Index Register (offset: 0x5900)” on page 465
through “Send BD Ring Host Producer Index Register (offset:
. The update actually triggers the Ethernet controller to process the send descriptors starting at its consumer index. As a descriptor is processed, the consumer index is incremented, and the new index is reflected in a new status block update. Status block is described in
.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 106
BCM5718 Programmer’s Guide Send Rings
When the producer and consumer indices are equal, the ring is empty. When the producer index is one behind the consumer, the ring is full. Because of this configuration, the producer index always points to an empty slot.
Thus, there will always be at least one empty slot in a ring.
Figure 19 illustrates the relationships between all the components of a send ring.
Figure 19: Relationships Between All Components of a Send Ring
Send
RCB
Consumer
Producer
Send BD 1
Send BD 2
Send BD 3
Send BD 4
Send BD 5
Send BD 6
Send BD 7
Send BD 8
Send BD 512
Buffer
Buffer
Buffer
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 107
BCM5718 Programmer’s Guide Send Rings
Ring Control Block
The Send Ring RCB contains a pointer to the first Send BD in the device and host memory, number of send BDs in the ring, and control flags (see
for a full discussion of the send RCB). All the fields are in big-endian ordering as required by the Ethernet controller. The RCBs of the send rings are located in the device Miscellaneous Memory Region at offset 0x0100.
The devices support a host based send ring. The Send BDs of the host based Send Ring will be bus-mastered from host memory into device local memory. The device driver will program the BDs directly in its memory
space and avoid programmed I/O to the MAC. The Max_Len field in the RCB (see Figure 20
) indicates the maximum number of BDs in the Send Ring. This field can be programmed to either 32, 64, 128, 256, 512, 1024,
2048, or 4096, depending on the type of ring for which the corresponding RCB applies.
Figure 20: Max_Len Field in Ring Control Block
1st Ring Element 512th Ring Element
Host Send Ring Control Block
1st
Offset 32
0x00
0x04
0x08
0x0C
16 15
Host Ring Address
0
Max_Len
Flags
NIC Ring Address
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 108
BCM5718 Programmer’s Guide Send Rings
Host-Based Send Ring
The send buffer descriptors of host based send ring reside in host memory.
The host-based send ring will have up to 512 buffer descriptors, which are periodically and transparently
DMAed to a staging area inside the NIC internal memory where they are waiting to be consumed. The staging area can hold up to 128 entries per-ring, and Ethernet controller tries to keep the staging area full at all times by constantly monitoring the consumer and producer index (the algorithm for accomplishing this is beyond the
illustrates the relationship between the send buffer descriptors in host memory and the staging area in NIC memory.
When the host software initializes new buffer descriptors, its send ring producer index is incremented by the number of descriptors. The new index is then written to the corresponding send ring host producer index
staging area. Eventually, the buffer descriptors are processed, and the data associated with these descriptors is transferred onto the network.
Figure 21: Relationship Between Send Buffer Descriptors
Send Buffer Descriptors in
Host Memory
Send BD 1
Send BD 2
Send BD 3
Send BD 4
Send BD 5
Send BD 6
Send BD 7
Send BD 8
DMA
Consumer
Send BD 512
Producer
Send Buffer Descriptors in
NIC Memory
Send BD 1
Send BD n
Send BD n+1
Send BD n+2
Send BD 128
The Ethernet controller maintains the send ring consumer index, which is incremented as it processes the descriptors. The Ethernet controller informs the host software of its progress by updating the send ring consumer index in the status block. The host software uses the send ring consumer index and its producer index to determine the empty slots in the ring. The Ethernet controller implements an algorithm that periodically DMAs the status block to host memory in an efficient manner.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 109
BCM5718 Programmer’s Guide Checksum Offload
Checksum Offload
As network speed increases, offloading is becoming an important feature, and the ability to offload tasks from the host processor aids in the efficiency of the host and in overall system performance. To achieve a significant performance boost, most operating systems now a days offer a mechanism for the TCP/IP protocol stack to offload checksum calculations to the device.
The host software can configure the Ethernet controller to calculate IP, TCP, and UDP checksum as described in
RFC 791, RFC 793, and RFC 768 respectively. The first step in checksum calculation is determining the start of an IP and UDP datagram and TCP segment within a frame, which could vary depending on whether the frame is tagged (VLAN) or encapsulated with LLC/SNAP header. Then the checksum is computed from the start to the end of the datagram and inserted into the appropriate location in protocol header. Ethernet controller is designed to support checksum calculation on all frame types and also on IP datagram and TCP segments containing options.
For the Ethernet controller to compute the checksum and insert it into the outgoing frame, the host software must set the appropriate control bits in the send buffer descriptors associated with the frame and seed the checksum field with zero or with the pseudo header checksum.
The host software enables IP checksum calculation by setting the IP_CHKSUM bits in all the send buffer descriptors associated with the frame. The Ethernet controller inserts the checksum into the checksum field of the IP header.
To enable TCP or UDP checksum calculation, the host software must set the TCP_UDP_CKSUM bit in all the send buffer descriptors associated with the frame containing the entire UDP datagram or TCP segment. The
TCP and UDP checksum engines do not span IP fragmented frames.
The host software can configure the Ethernet controller to disable TCP or UDP pseudo-header checksum calculation by setting the Mode_Control.Send_No_Pseudo_Header_Checksum bit. When set, the host software must seed the checksum field in the TCP or UDP header with the pseudo-header checksum. If the
Mode_Control.Send_No_Pseudo_Header_Checksum is cleared, the Ethernet controller computes the checksum including the pseudo header and inserts it into the checksum field.
Note: In LSO enabled case, a driver needs to zero the TCP checksum field coming from upper layer of
OS when doing TX CSO. Also, make sure to set bit-27 and bit-28 of 0x4800 when doing TX CSO in both
LSO/Non LSO mode.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 110
BCM5718 Programmer’s Guide Large Segment Offload
Large Segment Offload
In computer networking, large segment offload (LSO) is a technique for increasing outbound throughput of high-bandwidth network connections by reducing host CPU overhead. This is done by queuing up large TCP packets letting the Ethernet controller split them into separate (smaller) TCP packets to be transmitted onto the network. The technique is also called TCP segmentation offload (TSO) or, more generically, LSO.
When large blocks of data are to be sent over a computer network they must be first broken down to smaller segments that can pass through all of the network elements such as routers and switches between the source and destination computers. This process is referred to as segmentation.
For example, a large TCP packet of 64 KB (65,536 bytes) of data is usually segmented into 46 segments of 1448 bytes each before being sent over the network through the Ethernet controller chip. With some intelligence in the controller, the host CPU can hand over a 64k byte TCP packet directly to the controller in a single transmit request and the controller can break the large TCP packet down into smaller segments of 1448 bytes, add the
TCP, IP, and data link layer protocol headers to each segment, and send the resulting frames over the network.
This significantly reduces the work done by the host CPU.
Some Broadcom Ethernet controllers, such as the BCM5718, also support using jumbo sized frames (up to
9,216 bytes) as the individual frame size into which a large offloaded TCP packet is segmented into.
Note: The UDP checksum engine does not span IP fragmented frames.
Note: The Ethernet controller does not validate the value of the Length field and may generate an error on the PCI bus if the Length field has a value of 0. The host driver must ensure that the Length field is nonzero before enqueueing the BD onto the Send Ring.
Note: The BCM5718 family added to ability to use jumbo sized frames simultaneous with LSO.
QuickStart
Follow the steps to enable LSO:
1. Zero TCP checksum field in offloaded packet (leave IP checksum field alone)
2. Set register 0x0C00[3]=1: Enable hardware LSO pre-DMA processing
3. Set register 0x4800[27]=1: Enable hardware processing of LSO IPv4 packets
4. Set register 0x4800[28]=1: Enable hardware processing of LSO IPv6 packets (if desired)
5. Set Send BD Flags[8]=1: CPU pre-DMA
6. Set Send BD Flags[9]=1: CPU post-DMA
7. See LSO Limitations section below
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 111
BCM5718 Programmer’s Guide Large Segment Offload
Note: In Broadcom controllers that have a physically separate isochronous (ISO) TX queue, there is a parallel set of register fields, which mirror that of the normal TX path, for controlling LSO on the ISO
TX path.
LSO-Related Hardware Control Bits
Table 26: Send Data Initiator Mode Register (Offset: 0xC00)
Name Bits Access
Hardware Pre-DMA Enable 3 RW
Default
Value
0
Description
Enable hardware LSO pre-DMA processing
The ISO Send Data Initiator Mode register is applicable only to controllers that have a secondary Tx ISO
(Isochronous) queue.
Name
Table 27: ISO Send Data Initiator Mode Register (Offset: 0xD00)
Bits
Hardware Pre-DMA Enable 3
Access
RW
Default
Value
0
Description
Enable hardware LSO pre-DMA processing
Name
Hardware IPv6 Post-DMA
Processing Enable
Hardware IPv4 Post-DMA
Processing Enable
Table 28: Read DMA Mode Register (offset: 0x4800)
Bits
28
27
Access
RW
RW
Default
Value
1
0
Description
Enable hardware processing of LSO IPv6 packets.
This bit has no effect on Post-DMA processing of
IPv4 packets. This bit when clear disables IPV6
Processing. This bit was not used for controllers before BCM5718.
Enable hardware processing of LSO IPv4 packets.
This bit has no effect on Post-DMA processing of
IPv6 packets. This bit is the TCP Segmentation
Enable bit.
The ISO Read DMA Mode register is applicable only to controllers that have a secondary Tx ISO (Isochronous) queue.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 112
BCM5718 Programmer’s Guide Large Segment Offload
Name
Hardware IPv6 Post-DMA
Processing Enable
Hardware IPv4 Post-DMA
Processing Enable
Table 29: ISO Read DMA Mode Register (Offset: 0x4A00)
Bits
28
27
Access
RW
RW
Default
Value
1
0
Description
Enable hardware processing of LSO IPv6 packets.
This bit has no effect on Post-DMA processing of
IPv4 packets. This bit when clear disables IPV6
Processing. This bit was not used for controllers before BCM5718.
Enable hardware processing of LSO IPv4 packets.
This bit has no effect on Post-DMA processing of
IPv6 packets. This bit is TCP Segmentation Enable bit.
The ISO related registers are to allow for processing of ISO Ethernet Audio Video (EAV)-related TX traffic. A physically separate isochronous TX queue exists in some Broadcom Ethernet controllers to support audio/ video traffic applications, which require very precisely timed isochronous launch of TX packets onto the wire.
Note: LSO using jumbo frames is permissible on some Broadcom controllers (i.e., BCM5718). This is accomplished by appropriately programming the MSS field of the Send BD.
Send Buffer Descriptor
The Send Buffer Descriptor (SBD) diagram is shown below.
Figure 22: Send Buffer Descriptor
31 15
Host Address
Length[15:0]
HdrLen[1:0] MSS[13:0]
Flags
VLAN Tag[15:0]
0
0x0
0x4
0x8
0xC
Host Address
This field is a 64-bit address specifying where the Send Buffer is located in Host memory.
Length[15:0]
This field is the length of the frame or TCP large segment to be transmitted.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 113
BCM5718 Programmer’s Guide Large Segment Offload
VLAN Tag[15:0]
This field is the VLAN Tag to be inserted into the frame if Flags[6] is set to 1.
HdrLen[7:0]
This field is the length of the Ether + IP + TCP Headers to be replicated in each segment arising out of a Large
TCP Segment transmit operation.
The HDRLEN field is split into two fields within the SBD:
• A sub-field within the Flags field at a word offset of 0x08
• Adjacent to the MSS field at an offset of 0x0c
This field is used only for LSO buffers. Its value specifies the combined L3 and L4 header length in 4-byte
DWORDS for TCP/IPv4 or TCP/IPv6 packets. The value includes any option headers for IPv4, any extension headers for IPv6, and any TCP options.
For a TCP/IPv4 packet without IP or TCP options, this field would have a value of 10 (decimal).
For a TCP/IPv6 packet without extension headers or TCP options, this field would have a value of 15 (decimal).
For a TCP/IPv6 packet with a hop-by-hop options extension header of length 8 bytes plus a TCP MSS option (4 bytes), this field would have a value of 18 (decimal).
MSS[13:0]
This field is the size of the TCP segments into which a LSO segment is to be segmented into. Note that the MSS field has been increased for some NetXtreme controllers (i.e., BCM5718) to hold a value specifying a jumbo frame size.
Flags
See the table below.
Table 30: Flag Field Description
2
3
Bit # Flag Name
0
1
TCP/UDP Checksum Offload
Enable
Flag Description
This bit enables calculation of TCP or UDP checksums for IPv4 and IPv6 transmitted packets. The driver will set this bit only if the packet contained within a buffer is TCP or UDP over IPv4 or IPv6.
IP Checksum Offload Enable This bit enables calculation of the IPv4 layer-3 checksum. This bit will be set only for IPv4 packets. The driver will never set it for IPv6 packets.
Packet End
Jumbo Frame
This bit will be set for the last send buffer in a packet.
Driver must set this bit to 1 if the MTU length of the Send Frame is >
1500B. The MTU length is the Ethernet payload length and excludes
Header length (and Trailer length). All BDs belonging to a Send Packet must configure this bit identically.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 114
BCM5718 Programmer’s Guide Large Segment Offload
5
6
Bit # Flag Name
4 HDRLEN[2]
Capture Time Stamp
(BCM5719/5720 only)
VLAN TAG
7
8
9
Coalesce Now
CPU Pre-DMA
CPU Post-DMA
Table 30: Flag Field Description (Cont.)
Flag Description
The length of the Ethernet + IP + TCP Headers to be replicated in each segment arising from Large TCP Segment Offload (LSO) activity.
If this bit is 1, this frame’s launch time shall be captured in the TX Time-
Stamp Register.
When this bit is set, the NIC will insert a VLAN tag in the Ethernet header.
The value for the inserted tag is taken from the VLAN Tag field in the send BD.
If this bit is set, then a status block with an updated send consumer index will be DMA'd to the host as soon as this buffer's data has been
DMA'd from the host. An interrupt may or may not be generated depending on the present state of interrupt avoidance mechanisms.
If this bit is set, then the CPU will be required to act upon the buffer before the send data initiator state machine is kicked off. Alternately, if hardware LSO is enabled and this bit is set in conjunction with CPU Post-
DMA, then this buffer will be treated as part of an LSO segment to be further segmented by hardware.
If this bit is set, then the CPU will be required to act upon the buffer before the send data completion state machine is kicked off. Alternately, if hardware LSO is enabled and this bit is set in conjunction with CPU Pre-
DMA, then this buffer will be treated as part of an LSO segment to be further segmented by hardware.
The length of the Ether + IP + TCP Headers (combined) to be replicated in each frame arising from Large TCP Segment Offload activity (LSO).
Maximum HDRLEN could be 200B.
10
11
12
13
14
15
HDRLEN[3]
HDRLEN[4]
HDRLEN[5]
HDRLEN[6]
HDRLEN[7]
Do Not Generate CRC If set to 1, the controller will not append an Ethernet CRC to the end of the frame.
LSO Limitations
The limitations of the SBD are listed below.
• MSS must not be less than 8 Bytes.
• LSO packet must be a TCP packet.
• IP length field must not be incorrect.
• TCP length field must not be incorrect.
• Total offloaded TCP payload length must be greater than the MSS selected in the SBD.
• For all LSO segments, SBD flag bit 8 and 9 (CPU pre-DMA and CPU post-DMA) must be set.
• LSO packet may not be IEEE 802.3 format with LLC and SNAP headers.
• L2 header must be contained within the very first SBD.
• IP header (IPv4 or IPv6), including IP options, must be contained within a single SBD.
• HdrLen[7:0] field must be correct in SBD.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 115
BCM5718 Programmer’s Guide Large Segment Offload
• DONT_GEN_CRC must not be set in a SBD for an LSO packet.
• SNAP field must not be set in the SBD for an LSO packet.
• TCP Header, including TCP Options, must be contained within a single SBD.
• The total length for all headers (L2/L3/L4) combined, plus options, may not exceed 200 bytes.
Additional LSO Notes
The TCP/UDP and IP checksum offload enable bits in the SBD may be either set or cleared. The hardware still works as expected when LSO is in use (i.e., checksums are calculated/inserted by the hardware since this is a natural requirement for doing LSO).
The driver should zero the TCP checksum field in the offloaded TCP packet, but leave the IP checksum alone.
This requirement may change with newer NetXtreme controllers.
Broadcom drivers enable long burst by default in the Read DMA Mode register (0x4800 bits 17:16 = 11 binary
= 4k byte burst size).
Do not set TxFIFO Underrun Prevention Enable (bit 31) in the Buffer Manager Mode register 0x4400.
Do not set bit 5 (Multiple Segment Enable) in 0xC00 (Send Data Initiator Mode register).
Example TCP-segmentation-related (LSO) register values
Source: Broadcom tg3 Linux driver with BCM5764M NIC
0x4800 08033BFE
0x0C00 0000000A
0x0CF4 50000020
0x0CEC 00040028
0x1008 A0000000
0x0CE8 00000036
0x0CF0 00000000
Read DMA Mode Register
Send Data Initiator Mode Register
Pre-DMA Command Exchange for Segmentation
DMA Flag Register for TCP Segmentation
Pre-DMA Command Exchange for Segmentation
Length/Offset Register for Segmentation
VLAN Tag Register for TCP Segmentation
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 116
BCM5718 Programmer’s Guide Jumbo Frames
Jumbo Frames
The BCM5718 family supports jumbo Ethernet frames in both the receive (RX) and transmit (TX) paths and in conjunction with LSO. The jumbo frame architecture and the software interface is nearly identical to that of the legacy NetXtreme family controllers which also supported jumbo frames. The related jumbo architectural changes for the BCM5718 family are described in this section.
Following are the feature highlights.
The maximum jumbo frame length supported by the BCM5718 family is 9622B, which may be broken down as follows:
• 9600B MTU payload
• 14B Ethernet Header
• 4B Ethernet FCS
• Optional 4B of VLAN Tag
• 9622B maximum size limit applies equally to both TX and RX paths
Transmit Side:
• CRC checksum offload of jumbo frames is permitted.
• TCP and IP checksum offload of jumbo frames is permitted.
• TCP segmentation offload (TSO), a.k.a. large segment offload (LSO), of jumbo frames is permitted.
• Jumbo frames are to be constructed out of standard send buffers.
• There is a single send buffer ring which jumbo and standard frames share. The driver may inter-mix jumbo and standard frames in the send ring without restriction.
• The TX MBUF on-chip memory has be upsized to 22K bytes.
• The behavior of TX jumbo frame processing remains identical to that of standard TX frame processing:
– A TX frame is first completely DMAed into the TX MBUF memory and only then will it be transmitted onto the wire.
– The TX EMAC treats jumbo frames exactly the same as it treats a standard frame, except for being cognizant of the length.
– Full-Duplex and Half-Duplex behavior remains the same.
– Host coalescing timing remains identical to that of standard frames.
– A new jumbo frame flag is introduced in the send buffer descriptor (SBD). The driver must set this flag bit to indicate a jumbo frame (i.e., frame length > 1514 bytes without CRC and VLAN tag fields)
– In Multiple Send Queue mode, all 16 send queues are permitted to post jumbo frames. Thus there are
16 send ring control blocks (RCBs) available in this mode.
Receive Side:
• An additional BD producer ring (the jumbo producer ring) has been introduced.
• The receive side retrieves buffers only from the jumbo producer ring in turn to post a received jumbo frame. Similarly, the receive side only retrieves buffers from the standard producer ring in turn to post standard sized frames.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 117
BCM5718 Programmer’s Guide Jumbo Frames
• The jumbo producer ring is populated with BDs of a format known as the Extended Buffer descriptors. This is different from the standard buffer descriptor format.
• It is important to note that the driver may post extended BDs only in the jumbo producer ring and may post standard BDs only in the standard producer ring. BD formats may not be interchanged.
• The return rings are heterogeneous. That is, the controller returns both standard BDs and extended BDs in the same return ring. Intermixing of both types of BDs can happen without any restriction.
• RX MBUF memory has been upsized to at least 32 KB.
• Both 4-tuple and 2-tuple RSS computation and classification are performed over RX jumbo frames.
• There are 4 return rings as per RSS requirements (this is not affected by jumbo frame support).
• Receive CRC calculation and checking is performed on jumbo frames.
• Hardware calculates TCP, UDP, and IP checksums on jumbo frames.
• In IOV mode (I/O Virtualization), all 17 receive queues (virtual receive queues) must be provided with extended receive BDs. To that end, each VRQ is provided with a dedicated standard receive BD (RBD) ring and a dedicated jumbo RBD Ring.
Other:
• Miscellaneous BD memory has been increased over legacy NetXtreme controllers from 6K (4K receive BD +
1K send BD + 1K gencomm) to 51 KB (17 KB standard receive BD + 17K jumbo receive BD + 16 KB send BD
+ 1K gencomm). Gencomm describes on-chip memory space used for driver to/from boot code/firmware communication.
• The structure of the status block has been updated in order to accommodate jumbo frame related information.
• The controllers memory map has also been updated.
Affected Data Structures
The data structures introduced, updated, or affected due to jumbo frame support are discussed in this section.
In IOV mode, some of these structures are instantiated 17 times in case of receive and 16 times in case of transmit.
Extended RX Buffer Descriptor (BD)
The extended buffer descriptors are only permitted to be posted to the jumbo producer ring. The structure is shown in
Figure 23: “Extended RX Buffer Descriptor,” on page 119 . The main distinction of the extended BD
structure is that it can point to a maximum of four pieces of a scattered receive buffer. Hence the structure contains four host addresses and four length fields.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 118
BCM5718 Programmer’s Guide Jumbo Frames
Figure 23: Extended RX Buffer Descriptor
31
Host Address 1
Host Address 2
15
Host Address 3
Len 1
Len 3
Host Address 0
Len 2
Resvd
Index
Type
IP Checksum
Error Flags
Len 0
Flags
TCP / UDP Checksum
VLAN Tag
RSS Hash
Opaque Data Area
0
0x0
0x4
0x8
0xC
0x10
0x14
0x18
0x1C
0x20
0x24
0x28
0x2C
0x30
0x34
0x38
0x3C
• The Host Address 0 field contains the address of the first buffer in host memory. The host address is in host address format (64-bit).
• The Host Address N field in the Extended Receive Buffer Descriptor contains the address of the Nth piece of the buffer in host memory.
• The Index field is used by the host to keep track of the position of the returned buffer descriptor. This field is passed through opaquely by the controller.
• The Len 0 field is initially set by the host and specifies the length of the first buffer available for receiving data; this length field is set to the length of the data pointed to by Host Address 0. When an extended BD is returned to the receive return ring, the Len 0 field is set to the entire length of the data associated with the buffer descriptor, which is so because the receive return ring contains only a single length field. All BDs posted to the return ring by the controller are of size 32 bytes, whereas extended receive BDs posted to the jumbo producer ring by the driver are of size 64 bytes.
Note: In the case of an extended BD, the host is permitted to make Len 0 > 4 KB and practically even beyond 9.6 KB, such that an entire jumbo frame could be held in a single buffer. In such a scenario, hardware attempts to post an entire jumbo frame in a single buffer designated by Len 0.
Note: Len 0 is not permitted to be 0. Len1, Len2, or Len 3 may be set to 0, but hardware ignores Len2 and Len3 when Len1 = 0. Similarly, hardware ignores Len4 when Len3 = 0.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 119
BCM5718 Programmer’s Guide Jumbo Frames
Note: None of the Len N values may be less than 8 bytes.
• The Len1, Len2 and Len3 fields contain the respective lengths of the remaining three piece of the buffer in host memory.
• The Type field is used by the controller internally and should be ignored and need not be set by the host.
• The Flags bits are used to indicate any special processing that is needed in the buffer. Bits that are not
explicitly defined here must be set to zero. See Table 32 on page 121 .
• For Receive Return Ring descriptors, if using IP checksum offload, the host driver software should rely on the Flags bit IP_CHECKSUM (Flags bit 12) to determine if the IP checksum in the received packet is correct.
This field used to contain the actual IP checksum value, but that is not true for the BCM5718 Family of controllers. Only the Flags bit IP_CHECKSUM should be relied on by host driver software as is done by
Broadcom drivers.
• The TCP/UDP Checksum field is the checksum of all data following the IP header, for the length defined in the IP header. If the Receive No Pseudo-header Checksum bit is set, then the pseudo header checksum is not added to this value. If the bit is set this value includes the pseudo header.
• The Error Flags field contains a bitmask of possible errors. It is only valid if the
BD_FLAGS_FRAME_HAS_ERROR bit (see Table 32 on page 121 ) is set in the Flags field.
Table 31: Receive BD Error Flags
Bit# Error Flag Name Description
16
17
18
BD_ERR_BAD_CRC
BD_ERR_COLL_DETECT
BD_ERR_LINK_LOST_DURING_PKT
This frame has a bad CRC.
This frame had a collision.
The link was lost while this, incomplete frame was being received.
19
20
21
22
23
24
25:31
BD_ERR_PHY_DECODE_ERR The frame had an unspecified frame decoding error.
BD_ERR_ODD_NIBBLED_RCVD_MII The packet arrived with an odd number of nibbles.
BD_ERR_MAC_ABORT
BD_ERR_LEN_LT_64
BD_ERR_TRUNC_NO_RESOURCES
The MAC aborted the packet due to an unspecified internal error.
The MAC received a runt packet.
BD_ERR_GIANT_FRAME_RCVD
Reserved
The MAC could not receive this entire packet due to a lack of internal resources. Note that this bit is not set for frame longer than MTU that have bad CRC.
This frame was longer than the maximum packet length value set in the Receive MTU register. The data is truncated at the length specified in the Receive MTU register. This bit must be set when bit 4 BD_FLAG_IP_FRAG_END is set for the last BD of the last fragment in a train of fragments.
–
• The VLAN Tag field is filled in if the BD_FLAGS_VLAN_TAG bit is set in the flags field. It is the 2 byte VLAN tag that has been extracted from a 802.1Q compliant frame.
• The Reserved field is used internally by the controller. The host should ignore the value of this field.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 120
BCM5718 Programmer’s Guide Jumbo Frames
• The Opaque Data field is reserved for the driver and any data placed here is passed opaquely by the driver from the receive buffer descriptor in the standard or jumbo receive ring to one of the receive return rings.
11
12
13
14
15
Bit #
3
4
5
0
1
2
6
7:8
9
10
Flag Name
Reserved
Reserved
BD_FLAG_END
RSS_HASH_VALID
Reserved
BD_FLAG_JUMBO_RING
BD_FLAG_VLAN_TAG
RSS_HASH_TYPE
Reserved
BD_FLAG_FRAME_HAS_ERROR
–
IP_CHECKSUM
TCP_UDP_CHECKSUM
TCP_UDP_IS_TCP
IPV6_PACKET
Table 32: Receive BD Flags
Flag Description
–
–
The frame ends at the end of the data in this buffer descriptor.
If this bit is 1, then the RSS_HASH_TYPE field is valid. Else the
RSS_HASH_TYPE field is meaningless and must be ignored for this frame.
–
Indicates that this packet came from the Jumbo Receive
Ring, not the Standard Receive Ring (For receive BDs only).
This must be set by the driver, it is just copied through opaquely by the controller firmware.
The frame associated with this buffer descriptor has an
802.1Q VLAN tag associated with it.
Hash type of the receive packet. It indicates which hash_type was used on the receive packet if multiple hash types are defined.
–
An error was detected by the controller. The detected error type is set in the Error_Flag word of the receive buffer descriptor.
–
Indicates that the IP Checksum field is valid.
Indicates that the TCP_UDP Checksum field is valid.
Indicates that this frame has a TCP packet in it.
Indicates that this frame has an IPv6 packet in it.
Receive Jumbo Producer Ring
The jumbo RBD producer ring is reintroduced in the BCM5718 family to support RX jumbo frames. The jumbo
RBD producer ring is structured the same as the standard RBD producer ring, but the primary difference is that only extended BDs are permitted to be posted in the jumbo RBD producer ring. The jumbo RBD producer ring has a unique RCB associated with it.
The jumbo ring is managed by a producer index and a consumer index as in the case with the standard producer ring. Whenever host software adds more BDs to the jumbo producer ring, it writes the updated producer index to the controller via a high-priority mailbox located at the PCIE address range 0x208 – 0x20F.
The producer index register is at 0x3008.
The controller keeps a local copy of the jumbo ring consumer index in register 0x2470. The jumbo ring
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 121
BCM5718 Programmer’s Guide Jumbo Frames
There is also a jumbo ring replenishment threshold register 0x2C1C. The controller DMAs BDs from the jumbo ring in advance and caches them locally in controller memory. The controller initiates DMA of more BDs anytime the local number of cached BDs falls below the threshold programmed in this register.
The maximum number of BDs that can be held in the jumbo ring is programmed in that ring's ring control block
RCB.
Ring Control Blocks
Each host based ring has an RCB associated with it. The same structure applies to all types of host rings, (i.e., receive producer ring, receive return ring, and also send rings). In the case of the BCM5718 family, the RCB structure is enhanced to accommodate jumbo frames. However, the enhancement applies only to the Receive
Standard Producer ring.
The RCB essentially points to the physical address of the host memory where a producer ring is placed. The
BCM5718 family uses two RCBs, one for the Standard Ring and one for the jumbo ring. The jumbo ring RCB is implemented over a set of four registers at the address range 0x2440 – 0x244F. The standard ring RCB register addresses remain the same at 0x2450 – 0x245F. The send ring RCB and return ring RCBs are memory-mapped.
Figure 24: Ring Control Block
31
Host Ring Address
Max Len
NIC Ring Address
16 15
Max Frame Size
2 1 0
Flags
The host ring address is the host address of the first ring element. The host ring address is in host address format. In controller-based send rings, the host address is ignored.
The NIC ring address is the address where a portion of a ring is cached in the controller's miscellaneous BD memory. The driver need not program anything to these fields in most NetXtreme controllers. However, the
BCM5718 family does require the driver to program non-hardware-default values here.
The Max Len field is interpreted differently for different types of rings. In the case of receive producer rings and receive return rings, this field indicates the maximum number of entries the ring can hold. The BCM5718 family imposes constraints for different rings as shown below:
• Receive standard producer ring and send ring: Max Len should be programmed by the host to indicate the maximum number of entries the ring will hold. The allowable values for the Standard Producer Ring Max
Len are 32, 64, 128, 256, 512, 1024, and 2048.
• Receive jumbo producer ring: Max Len should be programmed by the host to indicate the maximum number of entries the ring will hold. The allowable values for the jumbo producer ring Max Len are 32, 64,
128, 256, 512, and 1024.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 122
BCM5718 Programmer’s Guide Jumbo Frames
Receive Return Ring(s)
There are no structural changes to the return rings in the BCM5718 family. There are no functional changes from standard size frames or from a Receive Side Scaling (RSS) perspective. The impact of the jumbo frame feature to the return rings is clarified here:
• The same return ring carries jumbo frames as well as standard size frames. Intermixing can happen without any limitation.
• The total number of return rings remains 4. However, when RSS is disabled, return ring 0 is the only active return ring and all RX frames are returned over Ring 0.
• When an extended buffer descriptor is returned to a receive return ring, the extended portion of the descriptor is truncated by the controller. The offset range 0x00 – 0x01F is the extended portion (see
• Furthermore, when an extended BD is returned in a return ring, the length of the entire RX frame is consolidated in the Len 0 field even if any or all of the scatter buffer pieces of the extended BD were used to place the frame in host memory.
Send Buffer Descriptor
The send buffer descriptor (SBD) has been updated to accommodate LSO over jumbo frames.
illustrates the updated SBD format.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 123
BCM5718 Programmer’s Guide
Figure 25: Send Buffer Descriptor
31 15
Host Address
Length[15:0]
HdrLen[1:0] MSS[13:0]
Flags
VLAN Tag[15:0]
0
0x0
0x4
0x8
0xC
Jumbo Frames
• The Host Address is the 64-bit address where the send buffer is located in host memory.
• The Length[15:0] is the length of the frame or TCP large segment to be transmitted.
• The VLAN TAG[15:0] field is the tag to be inserted in the frame if flags[6] is set to 1.
• The aggregate HDRLEN[7:0] field is the length of the Ethernet+IP+TCP headers to be replicated in each segment arising out of a large TCP segment (LSO). (See Flags also.)
• The MSS[13:0] field is the size of the TCP segments into which a LSO segment is to be chopped up into.
Note that it has been increased to hold the value of a jumbo frame size. The Flags field of SBD is shown in
Table 34: Send Buffer Descriptor Flags
2
3
Bit # Flag Name
0
1
4
5
6
7
TCP/UDP Checksum Offload
Enable
Flag Description
This bit enables calculation of TCP or UDP checksums for IPv4 and IPv6 transmitted packets. The driver sets this bit only if the packet contained within a buffer is TCP or UDP over IPv4 or IPv6.
IP Checksum Offload Enable This bit enables calculation of the IPv4 layer-3 checksum. This bit is set only for IPv4 packets. The driver never sets it for IPv6 packets.
Packet End
Jumbo Frame
This bit is set for the last send buffer in a packet.
Driver must set this bit to 1 if the MTU length of the Send Frame is >
1500B. The MTU length is the Ethernet payload length and excludes header length (and trailer length).
All BDs belonging to a send packet must configure this bit identically.
HDRLEN[2]
Capture Time Stamp
(BCM5719/5720 only)
VLAN TAG
Coalesce Now
The length of the Ethernet+IP+TCP headers to be replicated in each segment arising out of a large TCP segment (LSO).
If this bit is 1, this frame’s launch time shall be captured in the TX Time-
Stamp Register.
When this bit is set, the controller inserts a VLAN tag in the Ethernet header. The value for the inserted tag is taken from the VLAN Tag field in the send BD.
If this bit is set, a status block with an updated send consumer index is
DMA'd to the host as soon as this buffer's data has been DMA'd from the host. An interrupt may or may not be generated depending on the present state of interrupt avoidance mechanisms.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 124
BCM5718 Programmer’s Guide Jumbo Frames
10
11
12
13
14
15
Bit #
8
9
Flag Name
CPU Pre-DMA
CPU Post-DMA
HDRLEN[3]
HDRLEN[4]
HDRLEN[5]
HDRLEN[6]
HDRLEN[7]
Do Not Generate CRC
Table 34: Send Buffer Descriptor Flags (Cont.)
Flag Description
If this bit is set, the CPU is required to act upon the buffer before the send data initiator state machine is kicked off. Alternately, if hardware
LSO is enabled and this bit is set in conjunction with CPU post-DMA, then this buffer is treated as part of an LSO segment to be further segmented by hardware.
If this bit is set, the CPU is required to act upon the buffer before the send data completion state machine is kicked off. Alternately, if hardware LSO is enabled and this bit is set in conjunction with CPU pre-
DMA, then this buffer is treated as part of an LSO segment to be further segmented by hardware.
The length of the Ether+IP+TCP headers (combined) to be replicated in each frame arising out of a large TCP segment (LSO).
Maximum Header Length could be 256B.
If set to 1, the controller will not append an Ethernet CRC to the end of the frame.
Status Block
The status block has been modified in order to accommodate the jumbo producer ring's consumer index.
The status block is a data structure in the host memory. The host driver uses this data structure to trace the packet receive and transmission status and resource usage. Its length is 24 bytes. The driver needs to configure the status block host address register to point to the physical address in host memory for this data structure.
The BCM5718 family will update the status block in host memory (via DMA) prior to a host coalescing interrupt or MSI/MSI-X. The frequency of these status block updates is determined by the host coalescing logic. The two status block update interrupt triggers are RX/TX coalescing timer and RX/TX maximum coalesced frame count threshold.
A new field to indicate the Receive Jumbo Producer Ring Consumer Index is added to the BCM5718 family's
Legacy RSS mode status block. The updated structure of the status block is shown in
Note that there are multiple formats of status blocks.
Table 35: Status Block
Offset 31 16 15
0x00 Status Word
0x04 Reserved 0x0
0x08 Receive Standard Producer Ring Consumer
Index
0x0C Receive Return Ring 2 Producer Index
0x10 Send BD Consumer Index
Status Tag[7:0]
Receive Return Ring 1 Producer Index
Receive Return Ring 3 Producer Index
Receive Return Ring 0 Producer Index
0
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 125
BCM5718 Programmer’s Guide Jumbo Frames
Table 35: Status Block (Cont.)
Offset 31 16 15
0x14 Reserved 0x0 Receive Jumbo Producer Ring Consumer Index
0
The status tag field contains a unique 8-bit tag value in bits 7:0 when the status tagged status mode bit of the miscellaneous Host Control register 0x68 is set to 1. The status tag can be returned to mailbox 0 register 0x200 in field 31:24 by the host driver. When the remaining mailbox 0 register bits 23:0 are written to 0, the tag field of mailbox 0 is compared with the tag field of the last status block to be DMAed into host memory. If the tag returned is not equivalent to the tag of the last status block DMAed to the host, the interrupt state is entered.
Receive return ring 0 is the default return ring. If RSS is disabled all packets are assigned to this default ring.
There is no status block data structure in the controller memory space, but the host can access the current index through the register space.
Misc BD Memory
The Misc BD memory has been increased to 10 KB. The miscellaneous BD memory and the TX MBUF memory are physically the same memory, but a partition is hardwired. The miscellaneous BD memory holds four structures:
• On-chip send BD cache
• On-chip standard receive BD cache
• On-chip jumbo receive BD cache
• Software gencomm area (driver/firmware communication shared memory area)
Device Driver Interface
There are minimal driver interface modifications to support jumbo frames.
Send Interface
The driver essentially does not see any change in the send interface due to the jumbo frame feature. The only difference is that the stack is now allowed to construct larger frames (i.e., up to 9622 bytes long) out of send buffers.
Note: The controller is able to handle a single send buffer of length > 4K bytes and all the way up to
9622 bytes.
As in the case with previous controllers, the driver maintains a buffer descriptor ring (send ring), which allows for the “gather” management of transmit frame data. The production of send side packets by the driver is communicated to the controller via the Send Producer Ring Index Mailbox register. An update of the send BD ring producer index mailbox triggers the controller to begin DMA of the corresponding buffer descriptor. The consumption of send side packets by the controller is communicated back to the driver via the send consumer index, which is returned in the status block and periodically DMAed to the host by the controller.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 126
BCM5718 Programmer’s Guide
A conceptual diagram of the Send Interface is shown in Figure 26
below.
Figure 26: Send Driver Interface
Jumbo Frames
The BCM5718 family supports only host Send Buffer descriptors. The transmit ring is located in the host (as shown below), and the device will keep a local (not shown) copy of the ring.
Host Memory
Host Buffer n ( <= 9622B)
Host Buffer n+1
Data Structures in the host
31
Send Buffer Descriptor
Host Address length rsvd for firmware flags
VLAN tag
0
Send Host BD
Data Structures kept on-chip
Note: The RCB's host ring address field points to the first element of the Ring in the host.
Send Ring Control
Block
Host Send Ring
Cons
1st
RC
B p oin ts to a r ing
in
ho st me mo ry
31
Host Ring Address max_len flags
Unused
0
Prod
Send Host BD
Status Block (24 bytes)
The Status block resides in the NIC memory space and is periodically DMA'd to the host whenever the TX/RX coalescing timers expire, or whenever the RX/TX max coalesced frames thresholds are met. SW can examine the TX consumer indices in the status block to determine which packets have been sent by the HW.
31
Status Block
Status Word
Status Tag
RX Std Cons Rcv Return #1
0
Rcv Return #2
TX Cons Index
Resvd
Rcv Return #3
Rcv Return #0
RX Jumbo Cons
TX Host Ring Prod Index references a specific BD in the ring
63
Mailbox Registers
TX Host Ring Prod Index
0
The mailbox registers reside on-chip starting at offset
0x5800. Each mailbox register is 64 bits wide. Wrting the lower 32 bits, triggers and event in the HW. SW updates the
TX Host Ring producer index to indicate that that there are buffer descriptors ready for the HW to process.
Receive Interface
As mentioned previously, the receive side has added another producer ring called the jumbo producer ring.
This means the driver maintains two buffer descriptor rings (receive producer rings) that provide free data buffer space into which the controller can place received frame data. The production of receive-side buffers by the driver is communicated to the device via the rEceive Producer Ring Index(s) Mailbox registers. An update of a receive BD ring producer index mailbox triggers the device to begin DMA of the corresponding buffer descriptor(s). The consumption of receive side buffers by the device is communicated to the driver via the receive consumer index, which is returned via the status block and periodically DMAed to the host by the controller.
The controller fetches RX BDs in anticipation of RX frames and caches the BDs in the controller. The cached BDs are stored in the Misc BD Memory. There is a set of rules that govern placement of packets in buffers:
• All standard sized frames, that is, frames less than or equal to the Max Frame Size field of the standard
RCB, typically 1522 bytes or less, are placed in buffers retrieved from the RX standard producer ring.
• Hence the host must ensure that buffers pointed to by the standard BDs are at least of that size.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 127
BCM5718 Programmer’s Guide Jumbo Frames
• In case the host creates a standard ring buffer that is smaller than Max Frame Size and an RX frame larger than the buffer size (but <= Max Frame Size) arrives, the controller attempts to place the frame in such a buffer and ends up truncating the frame.
• RX frames larger than Max Frame Size are placed in buffers retrieved from the RX jumbo producer ring.
• Extended buffers placed in the jumbo producer ring must provide an aggregated space of 9622 bytes or higher. Otherwise, jumbo frames might be truncated by the controller during placement.
A conceptual diagram of the Receive Producer Interface is shown in Figure 27
below.
Figure 27: Receive Producer Interface
Data Structures kept on-chip
Note: The RCB's host ring address field points to the first element of the Ring in the host.
Data Structures in the host
The BCM5718 family supports Standard & Extended RX Buffer descriptors. The RX Rings are located in the host (as shown below), and the device will keep a local (not shown) copy of the ring.
Host Memory
Host Buffer n (<= 1522B)
31
Receive Buffer Descriptor
0
Host Address length
…...
flags
…...
RX Std Cons Index references a specific BD in the Std Ring
RX Std Producer Ring
Cons
RX Std Host BD
1st
Prod
Host Buffer n+1
Host Buffer m (<= 9622B)
Status Block (28 bytes)
RX Std Host BD
RX Ext Host BD
RX Jumbo Producer Ring
Cons
1st
Prod
The Status block resides in the NIC memory space and is periodically DMA'd to the host whenever the TX/RX coalescing timers expire, or whenever the RX/TX max coalesced frames thresholds are met. SW can examine the TX consumer indices in the status block to determine which packets have been sent by the HW.
Status Block
31
Status Word
Status Tag
RX Std Cons
Rcv Return #2
Rcv Return #1
Rcv Return #3
0
TX Cons #1
Resvd
Rcv Return #0
RX Jumbo Cons
31
RX Std Ring Control Block
0
Host Ring Address max_len Max_frame_size
NIC Ring Address flags
63
Mailbox Registers
RX Std Ring Prod Index
0
31
RX Jumbo Ring Control
Block
Host Ring Address max_len
NIC Ring Address
0 flags
63
Mailbox Registers
RX Jumbo Ring Prod Index
0
The mailbox registers reside on-chip starting at offset
0x5800. Each mailbox register is 64 bits wide. Wrting the lower 32 bits, triggers and event in the HW. SW updates the
TX Host Ring producer index to indicate that that there are buffer descriptors ready for the HW to process.
The BCM5718 family fills up receive buffers with RX frame data and returns the buffers to the host via receive return rings. The members of a return ring are simply RX buffer descriptors. Both types of descriptors, that is, standard or extended, are returned to the same return ring. As mentioned previously, due to practical reasons, extended buffer descriptors are truncated before posting into a return ring so that the actual size of all BDs posted to the return ring are the same (see
“Receive Return Ring(s)” on page 123
).
To support RSS, there are four return rings in the BCM5718 family. However, when RSS is disabled all RX frames are posted to Ring 0 while the other three rings remain inactive. In any case, standard and jumbo frames may be intermixed in any return ring as the order of placement strictly follows the order of frame reception.
The controller maintains four producer indexes associated with the four return rings. The availability of receiveside packets by the device is communicated to the driver via the receive return ring producer indices, which is delivered via the status block periodically when it is DMAed to the host by the controller.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 128
BCM5718 Programmer’s Guide Jumbo Frames
The consumption of receive return packets by the driver is communicated to the controller via the receive return consumer ring index mailbox register.
A conceptual diagram of the Receive Return Interface is shown in
Figure 28: Receive Return Interface
There are 4 Receive Return Rings in the BCM5718 family when using RSS mode (16 if using IOV mode).
The Return Rings are host memory based – there are cached copies inside the chip.
Data Structures in the host Data Structures kept on-chip
Note: The RCB's host ring address field points to the first element of the Ring in the host.
Host Memory
RX Frame #1 (<= 1522B)
RX Frame #3 (<= 1522B)
RX Fr ame
#2 (<
= 962
2B)
Status Block (28 bytes)
The Status block resides in the NIC memory space and is periodically DMA'd to the host whenever the TX/RX coalescing timers expire, or whenever the RX/TX max coalesced frames thresholds are met. SW can examine the TX consumer indices in the status block to determine which packets have been sent by the HW.
31
Receive Buffer Descriptor
0
Host Address length
…...
flags
…...
RX Std Cons Index references a specific BD in the Std Ring
RX Return Ring 0
Cons
RX Std Host BD
1st
Prod
RX Ext Host BD
RX Std Host BD
Status Block
31
Status Word
Status Tag
RX Std Cons Rcv Return #1
Rcv Return #2
TX Cons #1
Rcv Return #3
Rcv Return #0
Resvd RX Jumbo Cons
0
RX Return Ring 3
Cons
1st
Prod
31
RX Return Ring 0 Control
Block
0
Host Ring Address max_len Max_frame_size
NIC Ring Address flags
Mailbox Registers
63
RX Return Ring 0 Cons Index
0
31
RX Return Ring 3 Control
Block
0
Host Ring Address max_len
NIC Ring Address flags
63
Mailbox Registers
RX Return Ring 3 Cons Index
0
The mailbox registers reside on-chip starting at offset
0x5800. Each mailbox register is 64 bits wide. Wrting the lower 32 bits, triggers and event in the HW. SW updates the
TX Host Ring producer index to indicate that that there are buffer descriptors ready for the HW to process.
Large Segment Offload (LSO/TSO)
LSO may create jumbo frames instead of standard sized frames. This is accomplished by programming the MSS
field of the send BD (see “Send Buffer Descriptor” on page 123 ).
The legacy NetXtreme design has limitations in LSO hardware. In the case of the BCM5718 family, these limitations also exist. Below is the list of such limitations:
• MSS may not be less than 8 bytes
• LSO packet must be a TCP packet
• IP length field must not be incorrect
• TCP length field must not be incorrect
• Total offloaded TCP payload length must be greater than the MSS selected by SBD
• For all LSO segments, the SBD flag bit 8 and 9 (CPU pre-DMA and CPU post-DMA fields) must be set.
• LSO packet may not be IEEE 802.3 format with LLC and SNAP headers
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 129
BCM5718 Programmer’s Guide Jumbo Frames
• The total length of IP header (including IP option for IPv4, extension headers for IPv6) and TCP header
(including TCP option) may not be more than 200 bytes. Note: post_dma_proc can support only up to 2
MBUFs worth of packet header data. 1st Mbuf gives: 128, Mbuf-header (8B), Frame Header field (40B) =
80B; 2nd Mbuf gives: 128, Mbuf Header (8B) = 120B. Hence the total space for all headers, which includes all L2/L3/L4 combined, and options cannot exceed 200 bytes.
• L2 header must be contained within one SBD (the first one)
• IP header (IPv4 or IPv6), including IP Options, must be contained within a single SBD
• Hdrlen[7:0] field must be correct in SBD
• DONT_GEN_CRC field must not be set in a SBD for LSO packet
• SNAP field must not be set in SBD for LSO packet
• TCP header, including TCP Options, must be contained within a single SBD
The Read DMA (RDMA) engine cannot support LSO packets with the above listed attributes. Any such LSO configurations may cause the RDMA engine to lockup and/or exhibit abnormal behavior.
Summary of Register Settings to Support Jumbo Frames
• Standard receive producer ring RCB (Ring Control Block):
0x2450: Host address high [31:0]
0x2454: Host address low [31:0]
0x2458: [31:16]: Standard ring size (power of 2)
[15:2]: std_max_packet_size = 0x5EE (1518 decimal)
[1:0] = 0
0x245C: Standard NIC address = 0x400000
The NIC address is a controller-internal memory address where the controller caches a portion of a ring to achieve faster, higher performance access to buffer descriptors.
• Jumbo receive producer ring RCB:
0x2440: Host address high [31:0]
0x2444: Host address low [31:0]
0x2448: [31:16]: Jumbo ring size (power of 2)
[15:2] = 0
[1:0] = 0
0x244C: Jumbo NIC address 0x00044400
Only the host address and ring size are applicable to the receive return RCB.
• Receive MBUF low water mark 0x4414 = 0x7E (program to this value only when jumbo enabled)
Receive MBUF high water mark 0x4418 = 0xEA (program to this value only when jumbo enabled)
Read DMA watermark register 0x4410 = 0x0
• Standard replenish threshold register 0x2C18 is typically 1/8 of total receive BDs in host memory.
Jumbo replenish threshold register 0x2C1C is typically 1/8 of total Receive BDs in host memory.
• EMAC MTU register 0x43C: Program this register based on max packet size.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 130
BCM5718 Programmer’s Guide Scatter/Gather
Scatter/Gather
Most often, the host software requests the NIC to transmit a frame that spans several physical fragments that are arbitrary in size and buffer alignment. This requires the Ethernet controller to gather all these fragments during a DMA process into a continuous data stream for transmission.
The ability to scatter/gather a frame lessens the restriction on the host software and increases overall system performance.
Example: A TCP/IP protocol stack could preconstruct the MAC and IP headers in separate buffers that are combined with the payload to form a complete frame. Since the header data are fairly constant during a
TCP or UDP session, the stack could use the same header buffers for the next frame.
The Ethernet controller uses a buffer descriptor for describing a physical fragment. There are two types of buffer descriptors; the Receive MAC processes receive buffer descriptors (Receive BD) and the Transmit MAC processes send buffer descriptors (Send BD).
send buffer descriptors.
Figure 29: Scatter Gather of Frame Fragments
Frame Fragments
Frag 3
Frag 2
Frag 5
Frag 4
Frag 1
Send Buffer Descriptors
Send BD 1
Send BD 2
Send BD 3
Send BD 4
Send BD 5
Buffers
Buffer 1
Buffer 2
Buffer 3
Buffer 4
Buffer 5
Tx FIFO Tx MAC
DMA
To transmit a frame, the host software sets up consecutive buffer descriptors in a send ring. Each buffer descriptor describes a physical fragment of a frame. As an example, the above figure illustrates a frame consisting of five fragments that are scattered throughout host memory. Frag1, the first fragment, is at the start of the frame, and Frag5, the last fragment, is at the end of a frame. For each fragment, there is a corresponding buffer descriptor, SendBd1 through SendBd5. These buffer descriptors must be initialized in the send ring in a consecutive order, SendBd1 to SendBd5. The last send buffer descriptor of a frame must have the PACKET_END bit of Send BD Flags field set to indicate the end of a frame.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 131
BCM5718 Programmer’s Guide VLAN Tag Insertion
VLAN Tag Insertion
The Ethernet controller is capable of inserting 802.1Q-compliant VLAN tags into transmitted frames and extracting the VLAN tags from received frames. A frame containing the 802.1Q VLAN tag has the value TPID
(Tag Protocol Identifier) value in the Ethertype field followed by a 16-bit TCI (Tag Control Information) field, which is made up of one CFI bit, 3 802.1P priority bits, and a 12-bit VLAN ID. The original 16-bit Ethertype/
Length field follows the TCI field.
Table 25 on page 100 shows the frame format with 802.1Q VLAN tag inserted.
The Ethernet controller allows the host software to enable or disable tag insertion on a per-packet basis. To send a frame with a VLAN tag, the host software must initialize the first send buffer descriptor of a packet with the VLAN tag value and set the VLAN_TAG bit of Send BD Flags field (see
).
TX Data Flow Diagram
illustrates how a frame, consisting of several fragments, is sent from the host to the NIC and onto the network. For simplicity, the diagram depicts the operation of a single ring.
1. The host software calls a system API to retrieve the three physical fragments of the frame. It initializes the next three send buffer descriptors to point to each fragment. The send buffer descriptors reside in host memory. Internally, the host software maintains the ring’s producer index. In this case, the producer index is incremented by three because there are three fragments.
2. The host software updates the send ring producer index by writing the producer index value to Send Ring
Producer Index Mailbox at offset 0x300 for host standard and offset 0x5900 for indirect mode. The mailbox update triggers the Ethernet controller to process the send buffer descriptors.
3. The send buffer descriptors are DMAed to the ring’s staging area in device memory as indicated in the RCB.
4. The Ethernet controller DMAs the frame (as described in the descriptors) to its internal memory for transmission.
5. Internally, the Ethernet controller maintains the send ring’s consumer index, which is incremented as it processes the descriptors.
).
7. The status block is DMAed to host memory. This DMA is subject to host coalescing, and the NIC may generate an interrupt at this point.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 132
BCM5718 Programmer’s Guide
show the basic driver flow to send a packet.
Figure 30: Transmit Data Flow
Host Memor y
Host Memory
Frame
Buffer 1
Buffer 2
Buffer 3
4
2
1
Send BD 1
Send BD 2
Send BD 3
Send BD 4
Send BD 5
Send BD 6
Send BD 7
Send BD 8
SendBD 512
3
7
Status
Block
TX Data Flow Diagram
Send Producer
Index
Send BD 1
Send BD n
Send BD n+1
Send BD n+2
SendBD 128
Status
Block
Buffer 1
Buffer 2
Buffer 3
6
Send Consumer
Index
5
Tx
Info
Info MAC
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 133
BCM5718 Programmer’s Guide TX Data Flow Diagram
Figure 31: Basic Driver Flow to Send a Packet
OS asks NIC Driver to send a packet
Is the NIC enabled to send packets?
Yes
Examine packet (if necessary) and decide which send ring to use.
No Return appropriate error code to OS.
Get the virtual address and length of the next buffer in packet
Does the NIC have enough free send BD's to send the packet?
Yes
Get the virtual address and length of the first buffer in packet
Make OS call to lock down the virtual buffer in host memory and get the corresponding physical address(es)
Allocate the next available send BD from a free list of pre-allocated send BDs for a given send ring
No Queue packet for later transmission.
Return appropriate status code to OS
Fill out a Send BD with the info (address, length, flags) that corresponds to this physical host fragment
Get the physical address and length of the next physical fragment for the virtual buffer
No
Is this the last physical fragment of the virtual buffer?
Yes
Is this the last virtual buffer for this packet?
Yes
Set BD_FLAG_END bit in send BD
Update the Send Producer Index. This tells the HW that a new packet is ready to be transmitted
No
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 134
BCM5718 Programmer’s Guide Reset
Reset
A hardware reset initiated by the PCI reset signal will initialize all PCI configuration registers and device MAC
values. The content of the device internal memory remains unchanged after warm reset (any reset with the power supplied to the device).
At the end of the reset, the on-chip RX RISC executes a small on chip ROM code. This code loads an executable image contained in an attached NVRAM and referred to as the boot code. This boot code allows at least the following fields to be initialized to different values to support product variations (for additional details, see
Section 3: “NVRAM Configuration,” on page 68 ).
• Vendor ID
• Device ID
• Subsystem Vendor ID
• Subsystem Device ID
• Possible PHY initialization
The boot code may have additional functionality such as PXE that must be acquiesced while the host software is running.
Example: An NDIS driver issues a device reset via the Core Clock Blocks Reset bit (see Table 358 on page 334 ). After the reset is completed, the RX RISC begins executing the boot code as if the power was first applied to the device. However, the NDIS driver must have a mechanism to prevent the PXE driver from running and the boot code must be able to distinguish between a power-on reset and a reset initiated by the host software. The host software and the boot code could implement a reset handshake by using shared memory at offset 0x0b50 as a software mailbox (see
“Firmware Mailbox” on page 219
).
The BCM5718 family of Ethernet controllers supports a boot code mechanism known as “self-boot”. For selfboot the boot code image is stored in internal ROM rather than in an external NVRAM. So there is no loading of a boot code image from external NVRAM when resetting in the self-boot scenario.
However, there may still be a very small external NVRAM device which may contain some configuration items and possibly boot code “patches” to be applied to the ROM'd self-boot boot code. Refer to the following
Broadcom Application Notes for additional self-boot and general NVRAM access information:
• 5754X_5787X-AN10X-R “Self Boot Option”
• NetXtreme-AN40X-R “NetXtreme/NetLink Software Self-Boot NVRAM”
• NetXtreme-AN50X-R “NetXtreme®/NetLink® NVRAM Access”
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 135
BCM5718 Programmer’s Guide MAC Address Setup/Configuration
MAC Address Setup/Configuration
The MAC address registers, starting at offset 0x0410, contain the MAC addresses of the NIC. These registers are usually initialized with a default MAC address extracted from the NIC NVRAM when it is first powered up. The host software may overwrite the default MAC address by writing to the MAC registers with a new MAC address.
illustrates the MAC register format.
The BCM5718 family Ethernet controller allows a NIC to have up to four MAC addresses (offset 0x410–0x42F) that are used for hardware packet reception filtering. However, most host software will initialize the registers of the four MAC addresses to the same MAC address since a NIC usually has only one MAC address.
When flow control is enabled on the Ethernet controller, the MAC Address 0 is used as the source address for sending PAUSE frames (see
“Pause Control Frame” on page 586
).
Register Name
Mac_Address_0
Mac_Address_1
Mac_Address_2
Mac_Address_3
Table 36: Mac Address Registers
Offset
0x0410
0x0414
0x0418
0x041c
0x0420
0x0424
0x0428
0x042c
31 24 23 16
Octet 2
Unused
Octet 3
Unused
Octet 2 Octet 3
Unused
Octet 2 Octet 3
Unused
Octet 2 Octet 3
15
Octet 0
Octet 4
Octet 0
Octet 4
Octet 0
Octet 4
Octet 0
Octet 4
8 7
Octet 1
Octet 5
Octet 1
Octet 5
Octet 1
Octet 5
Octet 1
Octet 5
0
Packet Filtering
Multicast Hash Table Setup/Configuration
The MAC hash registers are used to help discard unwanted multicast packets as they are received from the external media. The destination address is fed into the normal CRC algorithm in order to generate a hash function. The most significant bits of the CRC are then used without any inversion in reverse order to index into a hash table, which is comprised of these MAC hash registers. If the CRC is calculated by shifting right, then the right-most bits of the CRC can be directly used with no additional inversion or bit swapping required. See
“Ethernet CRC Calculation” for more details on the CRC algorithm.
All four MAC hash registers are used so that register 1 bit-32 is the most significant hash table entry and register
4 bit-0 is the least significant hash table entry. This follows the normal big-endian ordering used throughout the
Ethernet controller. Since there are 128 hash table entries, 7 bits are used from the CRC. When hash table is extended to 256 entries, 8 bits from the CRC will be used as hash index.
The MAC hash registers are ignored if the receive MAC is in promiscuous mode.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 136
BCM5718 Programmer’s Guide Packet Filtering
Ethernet CRC Calculation
The Ethernet controller uses the standard 32-bit CRC required by the Ethernet specification as its FCS in all packets. The checksum is the 32-bit remainder of the polynomial division of the data taken as a bit stream of polynomial coefficients and a predefined constant, which also represents binary polynomial coefficients. The checksum is optionally appended most-significant bit first to a packet, which is to be sent down the wire. At the receiving side, the division is repeated on the entire packet including the CRC checksum. The remainder is compared to a known constant. For details on the mathematical basis for CRC checksums, see Tanenbaum's
Computer Networks, Third Edition, c1996.
The 32-bit CRC polynomial divisor is shown below: x32 + x26 + x23 + x22 + x16 + x12 + x11 + x10 + x8 + x7 + x5 + x4 + x2 + x + 1
Generating CRC
The following steps describe a method to calculate the CRC with the resulting 32-bit quantity having reversed bit order (i.e., most significant bit x31 of the remainder is right-most bit). The data should be treated as a stream of bytes. Set remainder to 0xFFFFFFFF. For each bit of data starting with least-significant bit of each byte:
1. If right-most bit (bit-0) of the current remainder XORed with the data bit equal 1, then remainder = (remainder shifted right one bit) XOR 0xEDB88320, else remainder = (remainder shifted right one bit).
2. Invert remainder such that remainder = ~remainder.
Remainder is CRC checksum.
Right-most byte is the most significant and is to be sent first.
Swap bytes of CRC if big-endian byte ordering is desired.
Checking CRC
The following steps describe a method to check a stream of bytes, which has a CRC appended .
1. Set remainder to 0xFFFFFFFF.
2. For each bit of data starting with least-significant bit of each byte:
If right-most bit (bit-0) of the current remainder XORed with the data bit equal 1, then remainder = (remainder shifted right one bit) XOR 0xEDB88320, else remainder = (remainder shifted right one bit).
3. Remainder should equal magic value 0xDEBB20E3 if CRC is correct.
Initializing the MAC Hash Registers
The 128-bit multicast hash table is treated as a single object occupying four Ethernet controller registers
controller. Thus, the most significant 32-bit of the 128-bit value resides in Mac_Hash_Register_0 at offset
0x0470 and the least significant 32-bit resides in Mac_Hash_Register_3 at offset 0x047c.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 137
BCM5718 Programmer’s Guide Packet Filtering
Host software can enable the reception of all multicast frames including broadcast frames by setting all four multicast hash registers to 0xFFFFFFFF.
Register Name
Mac_Hash_Register_0
Mac_Hash_Register_1
Mac_Hash_Register_2
Mac_Hash_Register_3
Table 37: Multicast Hash Table Registers
Offset
0x0470
0x0474
0x0478
0x047c
Description
Most significant 32-bit of the 128-bit hash table
Bits 64:93 of the 128-bit hash table
Bits 32:63 of the 128-bit hash table
Least significant 32-bit of the 128-bit hash table
The following C code fragment illustrates how to initialize the multicast hash table registers. The code fragment computes the indices into hash table from a given list of multicast addresses and initializes the multicast hash registers.
Unsigned long HashReg[4];
Unsigned long j, McEntryCnt;
Unsigned char McTable[32][6];
// Initialize the McTable here.
McEntryCnt = 32;
// List of multicast addresses to accept.
// Initialize the multicast table registers.
HashReg[0] = 0;
HashReg[1] = 0;
// Mac_Hash_Regsiter_0 at offset 0x0470.
// Mac_Hash_Register_1 at offset 0x0474.
HashReg[2] = 0;
HashReg[3] = 0; for(j = 0; j < McEntryCnt; j++)
{
// Mac_Hash_Register_2 at offset 0x0478.
// Mac_Hash_Register_3 at offset 0x047c.
unsigned long RegIndex; unsigned long Bitpos; unsigned long Crc32;
Crc32 = ComputeCrc32(McTable[j], 6);
// The most significant 7 bits of the CRC32 (no inversion),
// are used to index into one of the possible 128 bit positions.
Bitpos = ~Crc32 & 0x7f;
// Hash register index.
RegIndex = (Bitpos & 0x60) >> 5;
// Bit to turn on within a hash register.
Bitpos &= 0x1f;
// Enable the multicast bit.
HashReg[RegIndex] |= (1 << Bitpos);
}
The following C routine computes the Ethernet CRC32 value from a given byte stream. The routine is called from the above code fragment.
// Routine for generating CRC32.
unsigned long
ComputeCrc32( unsigned char *pBuffer, // Buffer containing the byte stream.
unsigned long BufferSize) // Size of the buffer.
{ unsigned long Reg; unsigned long Tmp; unsigned long j, k;
Reg = 0xffffffff;
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 138
BCM5718 Programmer’s Guide Packet Filtering
} for(j = 0; j < BufferSize; j++)
{
Reg ^= pBuffer[j]; for(k = 0; k < 8; k++)
{
Tmp = Reg & 0x01;
Reg >>= 1; if(Tmp)
{
}
} return ~Reg;
Reg ^= 0xedb88320;
}
Promiscuous Mode Setup/Configuration
The host software may enable promiscuous mode by setting the Promiscous_Mode bit (bit 8) of the
Receive_MAC_Mode register (offset 0x468). The Promiscous_Mode bit defaults to disabled after reset, and host software must explicitly set this bit for promiscuous mode. In promiscuous mode of operation, the
Ethernet controller accepts all incoming frames that are not filtered by the active receive rules regardless of the destination MAC address. In other words, the Ethernet controller operating in promiscuous mode ignores multicast and MAC address filtering (
“Multicast Hash Table Setup/Configuration” on page 136 and “MAC
Address Setup/Configuration” on page 136 ), but applies Receive Rules.
Broadcast Setup/Configuration
The host software may configure the Ethernet controller to discard the received broadcast frames by using two receive rules as defined below. The Ethernet controller parses all incoming frames according to these receive rules and discards those frames that have a broadcast destination address (see
“Receive Rules Setup and Frame
Classification” on page 96 for more details on setting up the receive rules).
The following is a sample of the two receive rules for discarding broadcast frames.
Rule1 Control: 0xc2000000
Rule2 Control: 0x86000004
Rule1 Mask/Value: 0xffffffff
Rule2 Mask/Value: 0xffffffff
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 139
BCM5718 Programmer’s Guide
Section 7: Device Control
Device Control
Initialization Procedure
This section describes the initialization procedure for the MAC portion of the NetXtreme family of devices.
1. Call the device reset procedure. (see
“Device Reset Procedure” on page 147 )
2. Enable/Disable any required bug fixes. Refer to the applicable Errata document for information on any errata that must be worked around by enabling/disabling the control bits of chip bug fixes if any are applicable.
3. Optionally, enable Tagged Status mode by setting the Enable_Tagged_Status_Mode bit of the
).
Note: For additional information on Tagged Status mode see
“Section 11: “Interrupt Processing,” on page 230”
.
4. Clear the driver status block memory region by writing zeros to the host memory region where the status block will be direct memory accessed (DMA) (see
).
5. Configure the DMA Write Water Mark in the DMA Read/Write Control register (see
Control Register (Offset: 0x6C)” on page 285
), as follows.
• If the Max Payload Size of PCIe Device Control register is 128 bytes, set the DMA write water mark bits
(bits19-21) of DMA Read/Write Control register to 011b (for a water mark of 128 bytes).
• if the Max Payload Size is 256 bytes or more, set the DMA write water mark bits (bits19-21) of DMA
Read/Write Control register to 111b (for a water mark of 256 bytes).
6. Set 0x6c[1:0] = 00b to allow 64 byte cache alignment for DMA writes on 5719.
7. Optionally, set DMA byte swapping by setting the Byte_Swap_Non-Frame_Data, Byte_Swap_Data and
frame data, when acting as a PCI DMA master.
8. Configure the host-based send ring by setting the Host_Send_BDs bit in the General Mode Control register
(see
“Mode Control Register (offset: 0x6800)” on page 472
).
9. Now that the Indicate Driver is ready to receive traffic, set the Host_Stack_Up bit in the General Mode
Control register (see
“Mode Control Register (offset: 0x6800)” on page 472
).
10. Configure TCP/UDP pseudo header checksum offloading.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 140
BCM5718 Programmer’s Guide Initialization Procedure
This step is relevant when TCP/UDP checksum calculations are offloaded to the device. The device driver may optionally disable receive and transmit pseudo header checksum calculations by the device by setting the Receive_No_PseudoHeader_Checksum and Send_No_PseudoHeader_Checksum bits in the General
Mode Control register (see “Mode Control Register (Offset 0x6800)” ). If the
Send_No_PsuedoHeader_Checksum bit is set, the host software must seed the correct pseudo header checksum value in TCP/UDP checksum field. Similarly, if the Receive_No_PsuedoHeader_Checksum bit is set, the device driver must calculate the pseudo header checksum and add it to the TCP/UDP checksum field of the received packet.
11. Configure MAC Mbuf memory pool watermarks (
“MAC RX MBUF Low Watermark Register (offset: 0x4414)” on page 432
, and
“Read DMA MBUF High Watermark Register (offset: 0x4418)” on page 432 ). Broadcom
has run hardware simulations on the Mbuf usage and strongly recommends the settings shown in
.
These settings/values establish proper operation for 10/100/1000 speeds.
Table 38: Recommended BCM57XX Ethernet Controller Memory Pool Watermark Settings
Register Standard Ethernet Frames
MAC RX Mbuf Low Watermark (0x4414)
Mbuf High Watermark (0x4418)
0x2A
0xA0
Note: The Low WaterMark Max Receive Frames register (0x504) specifies the number of good frames to receive after RxMbuf Low Watermark has been reached. The driver software must make sure that the MAC
RxMbuf Low WaterMark is greater than the number of Mbufs required for receiving the number of frames as specified in 0x504. The first Mbuf in the Mbuf chain of a frame will have 80 bytes of packet data while each of the subsequent Mbufs [except the last Mbuf] will have 120 bytes for packet data. The last Mbuf in the chain will have the rest of the packet data which can be up to 120 bytes.
12. Configure flow control behavior when the Rx Mbuf low watermark level has been reached (see
).
Table 39: Recommended BCM57XX Ethernet controller Low Watermark Maximum Receive Frames Settings
Register
Low Water Mark Maximum Receive Frames
(0x504)
Recommended Value
1
13. Enable the buffer manager by setting the Enable and Attn_Enable bits in the Buffer Manager Mode register
(see
“Buffer Manager Mode Register (offset: 0x4400)” on page 430 ). The buffer manager handles the
internal allocation of memory resources for send and receive traffic.
14. Set the BD Ring Replenish threshold for the RX Producer Ring. The threshold values indicate the number of buffer descriptors that must be indicated by the host software before a DMA is initiated to fetch additional receive descriptors in order to replenish used receive descriptors. The recommended configuration value
for the standard receive BD Ring replenish threshold is 0x19 (see “Standard Receive BD Producer Ring
Replenish Threshold Register (offset: 0x2C18)” on page 375
)
15. Initialize the standard receive Buffer Ring. Host software must write the Ring Control Block structure (see
“Ring Control Block” on page 108 ) to the standard receive BD Ring RCB register (see
Ring RCB Registers” on page 368”
). Host software must initialize the host physical memory address based on allocation routines specific to the OS.
16. Initialize the Max_len/Flags Receive Ring RCB register (0x2458).
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 141
BCM5718 Programmer’s Guide Initialization Procedure
Note: Beginning with the BCM57785 and BCM5718 families of controllers, field 15:2 no longer specifies the maximum expected size of a receive frame (it was formerly “Reserved”).
Note: For the BCM5718 family, do not initialize the Receive Producer Ring NIC Address register
(offset: 0x245C).
17. Initialize Receive BD Standard Producer Ring Index (0x26C) to zero.
18. Initialize the Standard Ring Replenish Watermark register (offset: 0x2d00). The recommended value is
0x20. If supporting Jumbo frames, also initialize 0x2d04 (the recommended value is 0x10). See
for more information about supporting Jumbo frames.
19. Initialize the Send Ring Producer Index registers by clearing (that is, set to zero) the Send BD Ring Host
Producer index (see “Send BD Ring Host Producer Index Register (offset: 0x300–0x307)” on page 309
).
20. Disable unused Receive Return Rings. Host software must write the RCB_FLAG_RING_DISABLED bit to the flags field of the ring control blocks of all unused Receive Return Rings.
21. Initialize Receive Return Rings. The Receive Return Ring RCBs are located in the Miscellaneous Memory region from 0x200 to 0x2FF. Host software must initialize the host physical memory address based on allocation routines specific to the OS. The Max_Len field indicates the ring size and it can be configured to either 32 or 64 or 128 or 256 or 512 or 1024, 2048, or 4096 depending on which ring's RCB is being initialized.
22. Initialize the Receive Producer Ring mailbox registers. The driver must write the value 0x00000000 (clear) to the low 32 bits of the Receive BD Standard Producer Ring Index mailbox (see
Producer Ring Index Register (offset: 0x5868)” on page 464 ).
Note: The host software must ensure that on systems that support more than 4 GB of physical memory, Send Rings, Receive Return Rings, Producer Rings, and packet buffers are not allocated across the 4 GB memory boundary. For example, if the starting memory address of the Standard
Receive Buffer Ring is below 4 GB and the ending address is above 4 GB, a read DMA PCI host address overflow error may be generated (see
“LSO Read DMA Status Register (offset: 0x4804)” on page 437 ).
The Standard RX Producer threshold value should be set very low. Some operating systems may run short of memory resources and the number of BDs that are made available will decrease proportionally.
The maximum number of Send BDs for a single packet is (0.75)*(ring size).
23. Configure the MAC unicast address. See “MAC Address Setup/Configuration” on page 136 for a full
description of unicast MAC address initialization.
24. Configure random backoff seed for transmit. See the Ethernet Transmit Random Backoff register (see
“Ethernet Transmit Random Backoff Register (offset: 0x438)” on page 317 ). Broadcom recommends using
the following algorithm:
Seed = (MAC_ADDR[0] + MAC_ADDR[1] + MAC_ADDR[2] + MAC_ADDR[3] + MAC_ADDR[4] +
MAC_ADDR[5]) & 0x3FF
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 142
BCM5718 Programmer’s Guide Initialization Procedure
25. Configure the Message Transfer Unit MTU size. The MTU sets the upper boundary on RX packet size; packets larger than the MTU are marked oversized and discarded by the RX MAC. The MTU bit field in the
Receive MTU Size register (see “Receive MTU Size Register (offset: 0x43C)” on page 317 ) must be
configured before RX traffic is accepted. Host software must account for the following variables when calculating the MTU:
• VLAN TAG
• CRC
• Jumbo Frames Enabled
26. Configure the Inter-Packet Gap (IPG) for transmit by setting the Transmit MAC Lengths register (see
IPG_CRS_Length, IPG_Length, and Slot_Time_Length. The value 0x2620 should be written into this register.
Note: An incorrectly configured IPG introduces far end receive errors on the MAC’s link partner.
27. Configure the default RX return ring for non-matched packets. The MAC has a rules checker, and packets do not always have a positive match. For this situation, host software must specify a default ring, where the
RX packet is placed. The bit field is located in the Receive Rules Configuration register (see
Configuration Register (offset: 0x500)” on page 328
).
List Placement Configuration Register (offset: 0x2010)” on page 360
) allows host software to initialize QOS rules checking. For example, a value of 0x181 (as used by Broadcom drivers) breaks down as follows:
• One interrupt distribution list
• Sixteen active lists
• One bad frames class
29. Write the Receive List Placement Statistics mask. Broadcom drivers write a value of 0x7BFFFF (24 bits) to
Register (offset: 0x2018)” on page 361
).
30. Enable RX statistics by asserting the Statistics_Enable bit in the Receive List Placement Control register (see
“Receive List Placement Statistics Control Register (offset: 0x2014)” on page 361 ).
31. Enable the Send Data Initiator mask by writing 0xFFFFFF (24 bits) to the Send Data Initiator Enable Mask
register (see “Send Data Initiator Statistics Mask Register (offset: 0xC0C)” on page 348”
).
32. Enable TX statistics by asserting the Statistics_Enable and Faster_Statistics_Update bits in the Send Data
Initiator Control register (0x0C08)
33. Disable the host coalescing engine by writing 0x0000 to the Host Coalescing Mode register (see
Coalescing Mode Register (offset: 0x3C00)” on page 415
). Software needs to disable the host coalescing engine before configuring its parameters.
Coalescing Mode Register (offset: 0x3C00)” on page 415
) and poll for 0x0000.
35. Configure the host coalescing tick count. The Receive Coalescing Ticks and Send Coalescing Ticks registers specify the number of clock ticks elapsed before an interrupt is driven (see
Register (offset: 0x3C08)” on page 416”
and “Send Coalescing Ticks Register (offset: 0x3C0C)” on page 417 ).
The clock begins ticking after RX/TX activity. Broadcom recommends the settings shown in
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 143
BCM5718 Programmer’s Guide Initialization Procedure
Table 40: Recommended BCM57XX Ethernet Controller Host Coalescing Tick Counter Settings
Register
Receive Coalescing Ticks(0x3C08)
Send Coalescing Ticks(0x3C0C)
Recommended Value
0x48
0x14
36. Configure the host coalescing BD count. The Receive Max Coalesced BD and Send Max Coalesced BD registers specify the number of frames processed before an interrupt is driven (see
BD Count Register (offset: 0x3C10)” on page 418
and
“Send Max Coalesced BD Count Register (offset:
). Broadcom recommends the settings shown in Table 41 .
Table 41: Recommended BCM57XX Ethernet Controller Host Coalescing Frame Counter Settings
Register
Receive Max Coalesced Frames(0x3C10)
Send Max Coalesced Frames(0x3C14)
Recommended Value
0x05
0x35
37. Configure the max-coalesced frames during interrupt counter. While host software processes interrupts,
this value is used. Broadcom recommends the settings shown in Table 42
.
Table 42: Recommended BCM57XX Ethernet Controller Max Coalesced Frames During Interrupt Counter
Settings
Register
Receive Max Coalesced Frames During
Interrupt(0x3C20)
Send Max Coalesced Frames During
Interrupt(0x3C24)
Recommended Value
0x05
0x05
38. Initialize host status block address. Host software must write a physical address to the Status Block Host
Address register, which is the location where the MAC must DMA status data (see “Status Block Host
Address Register (offset: 0x3C38)” on page 423
). This register accepts a 64-bit value in register 0x3C38 (high order 32 bits) and 0x3C3C (low order 32 bits).
39. Enable the host coalescing engine (0x3C00 bit 1).
40. Enable the receive BD completion functional block by setting the Enable and Attn_Enable bits in the
).
41. Enable the receive list placement functional block by setting the Enable bit in the Receive List Placement
Mode register (see
“Receive List Placement Mode Register (offset: 0x2000)” on page 359 .
42. Enable DMA engines by setting the Enable_FHDE, Enable_RDE, and Enable_TDE bits in the Ethernet Mac
Mode register (see
“EMAC Mode Register (offset: 0x400)” on page 310 ).
43. Enable and clear statistics by setting the Clear_TX_Statistics, Enable_TX_Statistics, Clear_RX_Statistics, and
).
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 144
BCM5718 Programmer’s Guide Initialization Procedure
44. Delay 40 microseconds.
45. Configure the General Miscellaneous Local Control register (see
“Miscellaneous Local Control Register
any of the attention bits in the CPU event register are asserted.
46. Delay 100 microseconds.
).
The following bits are asserted:
• Enable (starts the functional block)
• Write_DMA_PCI_Target_Abort_Attention_Enable
• Write_DMA_PCI_Master_Abort_Attention_Enable
• Write_DMA_PCI_Parity_Attention_Enable
• Write_DMA_PCI_Host_Address_Overflow_Attention_Enable
• Write_DMA_PCI_FIFO_Overerrun_Attention_Enable
• Write_DMA_PCI_FIFO_Underrun_Attention_Enable
• Write_DMA_PCI_FIFO_Overwrite_Attention_Enable
• Write_DMA_Local_Memory_Read_Longer_Than_DMA_Length
) to enable the host coalescence block fix that configures the device to send out status block update before the interrupt message.
49. Delay 40 microseconds.
50. Set register 0x4900[2] = 1 to prevent DMA overruns for BD read DMA engine. This enables a hardware function which limits all BD fetches to 256 bytes or less.
51. Configure the Read DMA Mode register (see
“LSO Read DMA Mode Register (offset: 0x4800)” on page 435 ).
The following bits are asserted:
• Enable—start functional block
• Read_DMA_PCI_Target_Abort
• Read_DMA_PCI_Master_Abort
• Read_DMA_PCI_Parity_Error
• Read_DMA_PCI_Host_Overflow_Error
• Read_DMA_PCI_FIFO_Overrun_Error
• Read_DMA_PCI_FIFO_Underrun_Error
• Read_DMA_PCI_FIFO_Overread_Error
• Read_DMA_Local_Memory_Write_Longer_Than_DMA_Length
52. Delay 40 microseconds.
53. Set 0x4800[24] = 0 to Allows multiple outstanding read requests from the non-LSO read DMA engine.
54. Set 0x4800[17:16] = 11b to Allows 4KB burst length reads for Jumbo/LSO network frames.
55. Set 0x4910[19:18] = 11b to allow 4KB burst length reads for non-LSO (i.e. standard) network frames.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 145
BCM5718 Programmer’s Guide Initialization Procedure
56. Enable the receive data completion functional block by setting the Enable and Attn_Enable bits in the
Receive Data Completion Mode register (see
“Receive Data Completion Mode Register (offset: 0x2800)” on page 373 ).
57. Enable the send data completion functional block by setting the Enable bit in the Send Data Completion
Mode register (see
“Send Data Completion Mode Register (offset: 0x1000)” on page 352
).
58. Enable the send BD completion functional block by setting the Enable and Attn_Enable bits in the Send BD
Completion Mode register (see “Send BD Completion Control Registers” on page 358
).
59. Enable the receive data and BD initiator functional block by setting the Enable and Illegal_Return_Ring_Size bits in the Receive Data and Receive BD Initiator Mode register (see
“Receive Data and Receive BD Initiator
Mode Register (offset: 0x2400)” on page 364
).
60. Enable the send data initiator functional block. Set the Enable bit in the Send Data Initiator Mode register
(see
“Send Data Initiator Mode Register (offset: 0xC00)” on page 347 ).
61. Enable the send BD initiator functional block by setting the Enable and Attn_Enable bits in the Send BD
Initiator Mode register (see
“Send BD Initiator Mode Register (offset: 0x1800)” on page 355 ).
62. Enable the send BD selector functional block by setting the Enable and Attn_Enable bits in the Send BD
Selector Mode register (see
“Send BD Ring Selector Mode Register (offset: 0x1400)” on page 353 ).
63. Enable the transmit MAC by setting the Enable bit and the Enable_Bad_TxMBUF_Lockup_fix bit in the
Transmit MAC Mode register (see
“Transmit MAC Mode Register (offset: 0x45C)” on page 320 ).
Optionally, the software may set the Enable_Flow_Control to enable 802.3x flow control.
64. Delay 100 microseconds.
65. Enable the receive MAC. Set the Enable bit in the Receive MAC Mode register (see
Register (offset: 0x468)” on page 323 ). Optionally, the software may set the following bits:
• Enable_Flow_Control bit – Enable 802.3x flow control
• Accept_oversized bit – Ignore RX MTU up to 64K maximum size
• Promiscuous_Mode bit – Accept all packets regardless of destination address
• No_CRC_Check bit – RX MAC will not check Ethernet CRC
• Various Hash Enable bits – if using RSS mode (Receive Side Scaling)
66. Delay 10 microseconds.
67. Setup the LED Control Register (0x40C). The Broadcom driver uses a value of 0x800 when initializing this register.
68. Activate link and enable MAC functional blocks by setting the Link_Status bit in the MI Status register (see
“MII Status Register (offset: 0x450)” on page 319 ) to generate a link attention.
69. Optionally, disable auto-polling on the management interface (see
“MII Mode Register (offset: 0x454)” on page 319
).
70. Set Low Watermark Maximum Receive Frame register (offset: 0x504) to a value of 1 for the BCM5717 and
BCM 5718 family of controllers.
71. Configure D0 power state in PMSCR (see “Power Management” on page 188
). Note that the PMCSR register is reset to 0x00 after chip reset. Software may optionally reconfigure this register if the device is being moved from D3 hot/cold.
72. Setup the physical layer and restart auto-negotiation. For details on PHY auto-negotiation, refer to the applicable PHY data sheet.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 146
BCM5718 Programmer’s Guide Device Reset Procedure
73. Setup multicast filters. See “Packet Filtering” on page 136
for details on multicast filter setup.
74. Enable the Host Interrupt: a. Set the Clear_Interrupt bit in the Miscellaneous Host Control register (offset: 0x68) to clear the interrupt and Clear the Mask_Interrupt bit in the Miscellaneous Host Control register (offset: 0x68) to unmask the interrupt.
b. Set the interrupt mail box register (offset: 0x200) to 0.
Device Reset Procedure
1. Write the T3_MAGIC_NUMBER (0x4B657654 = KevT) to the device memory at offset 0xB50 to notify the bootcode that the reset is a warm reset (driver initiated core_clocks reset).
2. Save PCI command register 0x4 before chip reset (GRC_MISC_CFG core clock reset will clear the memory enable bit in PCI register 0x4, so we save relevant registers here)
3. Clear the Fast Boot Program Counter register (Offset 0x6894) and enable the Memory Arbiter (see
“Memory Arbiter Mode Register (offset: 0x4000)” on page 428
).
4. Initialize the Miscellaneous Host Control register (see
“Miscellaneous Host Control Register (offset: 0x68)” on page 283
). a. Set the Enable_Endian_Word_Swap bit in the Miscellaneous Host Control register, when the host processor architecture is little-endian. Set the Enable_Endian_Word_Swap bit and the
Enable_Endian_Byte_Swap bit in the Miscellaneous Host Control register, when the host processor architecture is big-endian. b. Enable the indirect register pairs by setting the Enable_Indirect_Access bit in the Miscellaneous Host
Control register (see
). c. Enable the PCI State register to allow the device driver read/write access by setting the
Enable_PCI_State_Register bit in the Miscellaneous Host Control register.
5. Reset the core clocks by setting the CORE_Clock_Blocks-Reset bit in the General Control Miscellaneous
Configuration Register (offset: 0x6804)” on page 474
).
6. Wait for the core-clock reset to complete. The core clock reset disables indirect mode and flat/standard modes ¡X software cannot poll the core-clock reset bit to deassert, since the local memory interface is disabled by the reset. Delay a minimum of 1 millisecond before continuing the initialization sequence.
7. Enable MAC memory space decode and bus mastering by setting the Bus_Master and Memory_Space bits
8. Enable the MAC memory arbiter by setting the Enable bit in the Memory Arbiter Mode register (see
“Memory Arbiter Mode Register (offset: 0x4000)” on page 428
).
9. Initialize the Miscellaneous Host Control register (see
“Miscellaneous Host Control Register (offset: 0x68)” on page 283
).
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 147
BCM5718 Programmer’s Guide Device Closing Procedure a. Set the Enable_Endian_Word_Swap bit in the Miscellaneous Host Control register, when the host processor architecture is little-endian. Set the Enable_Endian_Word_Swap bit and the
Enable_Endian_Byte_Swap bit in the Miscellaneous Host Control register, when the host processor architecture is big-endian. b. Enable the indirect register pairs by setting the Enable_Indirect_Access bit in the Miscellaneous Host
Control register (see
). c. Enable the PCI State register to allow the device driver read/write access by setting the
Enable_PCI_State_Register bit in the Miscellaneous Host Control register.
10. Set Word_Swap_Data, Byte_Swap_Data, and win the General Mode Control register when the host processor architecture is little endian. Optionally, set Byte_Swap_Non_Frame_Data in the General Mode
Control register when the host processor architecture is big endian, (see
“Mode Control Register (offset:
).
11. Poll for bootcode completion. The device driver must poll the general communication memory at 0xB50 for the one is complement of the T3_MAGIC_NUMBER (that is, 0xB49A89AB). The bootcode should complete initialization within 1000 ms for Flash devices and 10000 ms for SEEPROM devices.
Device Closing Procedure
This section describes the device close procedure for the MAC portion of the NetXtreme family of devices.
1. Disable Host Interrupt.
a. Set the Mask_Interrupt bit in the Miscellaneous Host Control register (offset: 0x68) to disable interrupt b. Set the interrupt mail box register (offset: 0x200)
2. Tell Firmware the driver are shutting down and doing prereset.
Note: Do this only if ASF enabled.
a. Write the 0x2 to the device memory (offset: 0xB78) to PAUSE the firmware b. Set the RX-CPU Event register (offset: 0x6810) bit14 (SW Event 7) 7 bit and wait for SW bit14 (SW Event
7) to be clear c. Write MAGIC Number 0x4B657654 to the device memory (offset: 0xB50) d. Write 0x2 to the device memory (offset 0xc04)
3. Disable all the receiver blocks.
a. Clear the Enable bit in the Receive MAC Mode register (offset: 0x468) b. Clear the Enable bit in the Receive BD Initiator Mode register (offset: 0x2C00) c. Clear the Enable bit in the Receive List Placement Mode register (offset: 0x2000) d. Clear the Enable bit in the Receive BD Initiator Mode register (offset: 0x2400) e. Clear the Enable bit in the Receive Data Completion Mode Register (offset: 0x2800) f. Clear the Enable bit in the Receive BD Completion Mode Register (offset: 0x3000)
4. Disable all the transmit blocks.
a. Clear the Enable bit in the Send BD Selector Mode register (offset: 0x1400) b. Clear the Enable bit in the Send BD Initiator Mode Register (offset: 0x1800) c. Clear the Enable bit in the Send Data Initiator Mode Register (offset: 0xC00)
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 148
BCM5718 Programmer’s Guide Energy Efficient Ethernet™ d. Clear the Enable bit in the Read DMA Mode Register (offset: 0x4800) e. Clear the Enable bit in the Send Data Completion Mode Register (offset: 0x1000) f. Clear the Enable bit in the Send BD Initiator Mode Register (offset: 0x1C00)
5. Shut down all of the memory managers and related state machines.
a. Clear the Enable bit in the Host Coalescing Mode Register (offset: 0x3C00) b. Clear the Enable bit in the Write DMA Mode Register (offset: 0x4C00) c. Reset the FTQ by setting 0xfffffff to the FTQ Reset Register (offset: 0x5C00) d. Set FTQ Reset Register 0x5C00 to 0
6. Reset the controller. Call Device Reset Procedure (see “Device Reset Procedure” on page 147 )
7. Instruct the firmware do the following post reset:
Note: Do this only if ASF enabled.
a. write 0x2 to the device memory (offset 0xc04) b. write 0x80000002 to the device memory (offset 0xc04)
8. Free the RX/TX Ring list and buffers.
9. Set PCI power state to D3hot.
Energy Efficient Ethernet™
The BCM5718 family of controllers supports the IEEE specification for Energy Efficient Ethernet (EEE)
(IEEE 802.3az-2010). The algorithm below describes how to initialize and enable EEE mode in the BCM5718 family of controllers.
/*
* Controller EEE initialization
*/
// Disable LPI requests val = reg_read(0x36B0); val &= 0xFFFFFF7F; reg_write(0x36B0, val);
// Setup PHY DSP for EEE mii_write(0x18, 0x0C00); mii_write(0x17, 0x4022); mii_write(0x15, 0x017B); mii_write(0x18, 0x0400); if (enable EEE advertisement)
{
// Enable EEE advertisement for 100Base-TX and 1000Base-T modes
mii_write(0x0D, 0x0007);
mii_write(0x0E, 0x003C);
mii_write(0x0D, 0x4007);
mii_write(0x0E, 0x0006);
val = mii_read(0x0E);
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 149
BCM5718 Programmer’s Guide
//Enable MAC control of LPI
reg_write(0x36BC,0x01000004);
reg_write(0x36D0,0x000001F8);
reg_write(0x36B0,0x00100348);
// Set EEE timer debounce values
reg_write(0x36B4,0x07ff07ff);
reg_write(0x36B8,0x07ff07ff);
} else
{
// Disable EEE advertisement for 100Base-TX and 1000Base-T modes
mii_write(0x0D, 0x0007);
mii_write(0x0E, 0x003C);
mii_write(0x0D, 0x4007);
mii_write(0x0E, 0x0000);
val = mii_read(0x0E);
// Disable MAC control of LPI
reg_write(0x36B0,reg_read(0x36B0) &= 0x00100000);
}
Energy Efficient Ethernet™
/*
* Link status interrupt handler
*/
// Check for PHY link status if ((mii_read(0x11) & 0x100) == 0x100)
{
// Check for 1000mb link
if ((mii_read(0x19) & 0x700) == 0x700)
{
// Set EEE LPI exit timing for 1000mb link speed
reg_write(0x36d0, 0x19d);
}
// Check for 100mb link
elseif ((mii_read(0x19) & 0x500) == 0x500)
{
// Set EEE LPI exit timing for 100mb link speed
reg_write(0x36d0, 0x384);
}
//delay 1000 milliseconds
ms_delay(1000);
// Read PHY’s EEE negotiation status
mii_write(0x0d, 7);
mii_write(0x0e, 0x803e);
mii_write(0x0d, 0x4007);
val = mii_read(0x0e);
// Enable EEE LPI request if EEE negotiated
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 150
BCM5718 Programmer’s Guide Energy Efficient Ethernet™
if (4 == val || 2 == val)
{
reg_write(0x0x36b0, reg_read(0x36b0) | 0x80);
}
break;
} else
{
// Disable LPI requests
val = reg_read(0x36B0);
val &= 0xFFFFFF7F;
reg_write(0x36B0, val);
}
After the controller is fully initialized, the following algorithm may be used to verify EEE link status: i=0 while(i < 100)
{
if ((mii_read(0x11) & 0x100) == 0x100)
{
if ((mii_read(0x19) & 0x700) == 0x700)
{
if (((mii_read(0xA) & 0x7000) == 0x7000) ||
((mii_read(0xA) & 0x3000) == 0x3000)))
{
//link negotiated to gigabit master or slave
reg_write(0x36d0, 0x19d);
}
}
elseif ((mii_read(0x19) & 0x500) == 0x500)
{
//link negotiated to 100Mbps
reg_write(0x36d0, 0x384);
}
ms_delay(1000);//delay 1000 milliseconds
//Assert LPI
reg_write(0x36b0,(reg_read(0x36b0) | 0x80))
break;
}
else
{
ms_delay(500);//delay 500 milliseconds
i++;
}
} if (i >= 100)
{
Link_not_detected();
}
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 151
BCM5718 Programmer’s Guide IEEE1588
Section 8: IEEE1588
IEEE1588 Time Sync Introduction
IEEE1588 and IEEE802.1AS are two protocols designed to perform time synchronization over Ethernet networks. IEEE1588 Precision Time Protocol (PTP) provides a UDP packet based time synchronization mechanism, while IEEE802.1AS extends PTP to operate directly over the L2 or Ethernet layer. Time precision offered by PTP is in the sub-microsecond range and, in fact, can get down to a Nanosecond.
NetXtreme Time Sync Assist
NetXtreme architecture provides a set of hardware (HW) features to assist IEEE1588 and IEEE802.1AS traffic over Ethernet. Both 1588 and 802.1AS are complex protocols from the perspective of statefulness and are best implemented by a software module resident in the host computer. However, since these protocols demand a sub-microsecond, in some cases nanosecond grade precision, long latencies encountered by host software do not permit these protocols to be implemented in software (Host CPU) in their entirety. The goal of NetXtreme architecture is to provide the cheapest and simplest set of HW hooks so that a service grade PTP profile may be enabled in the host server computer.
The assists that NetXtreme provides are:
• A 64-bit Counter clocked by the 125Mhz DLL clock to serve as the Precision Clock - Thus 8ns is the maximum precision offered. This clock is known as the EAV Reference Clock.
• A 64-bit Transmit Time Stamp Register - Software shall mark certain types of PTP message Packets in turn for TX hardware to capture launch time of that packet.
• A 64-bit Receive Time Stamp Register - RX hardware shall crack and identify certain types of PTP packets as configured by host SW. The reception time of such a packet shall be recorded in this register. Host SW shall retain the control to choose which PTP message Types get time stamped.
This feature, Time Sync Assist, may be enabled by setting a Mode bit in the chip.
Coexistence
The Time Sync Assist feature coexists with the following NetXtreme features:
• VLAN Tagging - in-band or out-of-band
• SNAP/LLC Framing - transmit and receive.
• Transmit LSO - note that PTP packets cannot be TCP segments.
• Transmit IP, UDP, TCP checksum offload - IP/UDP Transmit Checksum offload is applicable to UDP PTP packets, provided the packets are appropriately formatted by the software protocol engine, that is, 2B
UDP padding etc must be taken care by software.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 152
BCM5718 Programmer’s Guide NetXtreme Time Sync Assist
PTP Link Delay Measurement
At the completion of the Delay Request/Response exchange, the delay requester uses four timestamps (t1, t2, t3, t4) to compute the link delay. The link delay is computed as the average of the two one-way delays using the following formula:
T delay = [(t2 - t1) + (t4 - t3)] / 4
Assume that both nodes contain a NetXtreme Time-Synch Capable NIC. Table 43
describes the roles the PTP software component and the Time-Synch capable NIC hardware play during the above exchange.
Table 43: PTP Link Delay Measure Roles
Delay Requester Delay Responder
Host Software
1 TX PTP Delay Request packet- mark capture →
2
NIC Hardware
Capture TX stamp (t1) →
NIC Hardware Host Software
3a
→ Receive and iden fy
PTP delay request packet, then capture RX Time
Stamp (t2)
→ Receive packet, Read
RX Time Stamp Reg.
Capture TX Time Stamp
(t3) ←
TX PTP Delay Response packet with embedded
(t2) value — mark capture ←
4 ← Receive Packet, Read
RX Time Stamp Reg
Receive and identify PTP
Delay Response packet, then capture RX Time
Stamp (t4) ←
3b Receive packet ← (pass through) TX PTP Delay Response
Follow-up packet with embedded (t3) value
5 Collect t1, t2, t3, and t4 to compute T delay.
PTP Time Synchronization Messaging
The slave device then uses the link delay (Tdelay) and the Sync Message timestamps (t1, t2) to calculate the time offset it needs to compensate its local clock by using the following equation:
Tslave-offset = t2 - t1 - Tdelay
Assume that both nodes contain a NetXtreme Time-Synch Capable NIC. Table 44
describes the roles the PTP software component and Time-Synch capable NIC hardware play during above exchange.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 153
BCM5718 Programmer’s Guide Hardware Description
PTP Master Node
Host Software
1 TX PTP Sync packet— mark capture →
2
Table 44: PTP Time Synchronization Messaging Roles
PTP Slave Node
NIC Hardware
Capture TX stamp (t1) →
NIC Hardware
3
4
TX PTP Sync Follow-up packet with embedded
(t1) value →
Host Software
→ Receive and iden fy
PTP sync packet, then capture RX Time Stamp
(t2)
→ (pass through)
→ Receive packet, Read
RX Time Stamp Reg.
→ Receive PTP Follow-up packet
Collect t1, t2, and Tdelay to compute the Slave
Offset value
Hardware Description
The following items describe the included hardware:
Clock Hardware:
• EAV Reference Clock - this clock is neither the Master-clock nor the Slave-clock, but a specific clock. All time related hardware services are provided with respect to this clock.
• Reference Time Capture off an external GPIO Trigger
• A Time Watchdog which drives a GPIO output pin upon meeting a programmable time value
• A divided EAV Reference Clock output
Transmit Time Sync hardware:
• A 64-bit TX Time Stamp Register.
• A bit defined in Send BD for indication of TX Time capture
Receive Time Sync hardware:
• Programmable Receive Frame Cracker.
• A 64-bit RX Time Stamp Register + A 16-bit RX PTP Sequence ID Register
• An RX Time Stamp Lock Timer
Each of above items is described separately in the following sections.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 154
BCM5718 Programmer’s Guide Hardware Description
EAV Reference Clock/Counter
The EAV Reference Clock (Count) is a 64-bit free-running clock-tick counter with a 1ns time base. All Real-Time sensitive services needed by EAV are in reference to this counter. In the 5719/5720 chip, this counter is clocked by a 125Mhz free running DLL. Therefore, the precision of this counter is limited to 8ns for this generation of
NIC. This implies that the LSB 3-bits of EAV Reference Count shall always be "000" in this chip. This counter will emerge with a 0 value from Power on Reset and always count upwards. After reaching the full count, the counter will roll-over to 0 and count up again. These two counters are the only static attributes of this clock.
The rest are programmable.
Time referencing, Time-stamping, Time-sampling etc. all real-time interfacing of the chip hardware with the AV
Device Driver shall be performed in reference to this counter.
The following are the instrumentations available for the EAV Reference Counter:
1. Reset Control: The Driver has the option to configure the EAV Reference Counter to be reset in response to the following events: a. GRC Reset (MAC-core soft reset) and PCIE FLR (if supported) b. PCIe Link Reset c. Network Link transi on from UP→DOWN d. Network Link transi on from DOWN→UP
2. Explicit Driver Controls: a. Driver may reset the counter.
b. Driver may stop the counter.
c. Driver may re-start the counter after being stopped.
d. Driver may over-write the counter value by PIO write.
3. The driver may read the value of the free-running counter anytime. For that purpose, a 32-bit Register pair is provided.
4. The driver may take a snap-shot of this counter.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 155
BCM5718 Programmer’s Guide EAV Reference Corrector
EAV Reference Corrector
The 64-bit of EAV Ref Count described in the previous section is the integer porting of the counter. This is the part visible to the system for time stamping purposes. There is also a 24-bit correction field associated with the
EAV Reference Count. This field is used by the host system to periodically add or subtract a minimum resolution value from the EAV Reference Count, in case of 5719/5720. That value happens to be 8 nanoseconds. This feature is mainly used to compensate for the local crystal/PLL drifts. The add/subtract operation is performed entirely in hardware. Software is responsible only for programming the EAV Reference Corrector field appropriately.
The Corrector features a Correction-Value and a Correction-Sense sub-field. Host software (in this case the software PTP module) must compute the correction amount and then load the Correction-Value/Sense fields accordingly. The programmed value could be anything from 0x000001 through 0xFFFFFF. A value 0x0 is NOT permitted. The Correction-Value is added to a 24-bit accumulator every EAV Reference Clock tick (accumulator is re-initiated to 0 at POR reset and every time the Corrector register is updated). Therefore, the accumulator's value will increase with every EAV Ref clock cycle until it overflows. In the EAV Reference Clock tick in which the accumulator overflows, the hardware reloads the accumulator with the last programmed Correction-
Value. On the same clock cycle, based on the programmed Correction-Sense, the hardware either adds or subtracts 8 ns from EAV Reference Count. Therefore, the host software controls the sense, value, and period of an automatic hardware based correction. Although software may re-program the EAV Reference Corrector frequently, be cautious that a large negative correction-value may result in out of order time-stamps.
Time Watchdogs
A Time Watchdog is a 64-bit register which may be programmed by host software to a specific time value in the future. When the value of the EAV Reference Count equates the value of the Time Watchdog, the1588_GPIO[n] output pin is toggled. There are two such Time Watchdogs which work independently.
Divided EAV Reference Clock Output
An additional PLL channel output has been routed to an external pin. This output clock is edge synchronous to the EAV Reference clock, although there could be a < 5ns routing delay present. A register is available in the
CPMU block in which the clock-divisor value may be programmed by host software. The available output frequency range is 125MHz through 4.8 MHz. See the Flash Clock Policy Register (0x366C).
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 156
BCM5718 Programmer’s Guide Transmit Time Stamping Service
Transmit Time Stamping Service
The software is able to mark certain types of PTP packets (Delay Request, Synch etc) and the NIC hardware captures the time when such a packet's first byte, or SOF, is launched for transmission. Ideally, the time capture is the instant of the actual SOF launch on the wire, but due to practical limitations, the capture will take place at the time the SOF appears on GMII interface among EMAC and GPHY. The propagation delay through the
GPHY is virtually fixed, and hence, could easily be accounted for by system software.The captured time is in reference to the EAV Reference Count. There is only a single 64-bit register, the TX Time Stamp Register, for this purpose. The precision of this stamp is 1ns. System software may read the value of this register anytime to get the time-stamp of last sent PTP packet. Reference the following table of send ring SBD for IEEE1588.
Table 45: Send Ring SBD Flags
2
3
Bit # Flag Name
0
1
4
5
6
Flag Description
TCP/UDP Checksum Offload Enable This bit enables calculation of TCP or UDP checksums for IPv4 and
IP Checksum Offload enable
Packet End
Jumbo Frame
HDRLEN[2]
Capture Time Stamp
VLAN TAG
IPv6 transmitted packets. The driver will set this bit only if the packet contained within a buffer is TCP or UDP over IPv4 or IPv6.
This bit enables calculation of the IPv4 layer-3 checksum. This bit will be set only for IPv4 packets. The driver will never be set for
IPv6 packets.
This bit will be set for the last send buffer in a packet.
Driver must set this bit to 1 if the MTU length of the Send Frame is > 1500B. The MTU length is the Ethernet payload length and excludes Header length (and Trailer length). All BDs belonging to a
Send Packet must configure this bit identically.
The length of the Ether+IP+TCP Headers to be replicated in each segment arising out of a Large TCP Segment (LSO).
If this bit is 1, this frame’s launch time shall be captured in the TX
Time-Stamp Register.
When this bit is set, the NIC will insert a VLAN tag in the Ethernet header. The value for the inserted tag is taken from the VLAN Tag field in the send BD.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 157
BCM5718 Programmer’s Guide Receive Time Stamp and Sequence ID Registers
Receive Time Stamp and Sequence ID Registers
The time of reception of chosen PTP frames shall be recorded by hardware in the RX Time Stamp Register. The place of time capture is the GMII interface, the point of capture is arrival of the SOF octet and the reference of time is the EAV Reference Count. Thus, the time of appearance of the SOF octet of every RX frame on the internal GMII interface needs to be recorded by hardware and held it in a temporary register or a temporary
FIFO. After reception of a complete frame, upon examining the RX Frame cracker results, hardware shall decide if that particular Frame had the Host SW desirable PTP message type and only then it will transfer its acquired time stamp into the RX Time Stamp Register. Along with the RX Time Stamp, hardware shall also capture the respective PTP packets sequenceID field. Hardware shall also mark a status field in a packet's Receive BD indicating if the packet was a PTP packet and also if RX Time Stamp was captured for the packet.
Bit # Flag Name
1:0 PTP Packet Type
2
3
BD_FLAG_END
RSS_HASH_VALID
Table 46: Receive Return Ring RBD Flags
Flag Description
00: PTP Delay Request Packet
01: PTP Delay Response Packet
10: PTP Sync Packet
11: All other type of PTP Packets
The frame ends at the end of the data in this buffer descriptor.
If this bit is 1, then the RSS_HASH_TYPE field is valid. Else the
RSS_HASH_TYPE field is meaningless and must be ignored for this frame.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 158
BCM5718 Programmer’s Guide Receive Time Stamp and Sequence ID Registers
Bit # Flag Name
4
5
6
7
8
PTP Status[0]
BD_FLAG_JUMBO_RING
BD_FLAG_VLAN_TAG
RSS_HASH_TYPE
Table 46: Receive Return Ring RBD Flags (Cont.)
9 PTP Status[1]
10 BD_FLAG_FRAME_HAS_ERROR
Flag Description
Bit{[9][4]} makes up this 2-bit field
00: Not a PTP packet
01: PTP v1 (UDP) packet Time Stamped
10: PTP v2 (L2) packet Time Stamped
11: PTP packet, but not time stamped
Indicates that this packet came from the Jumbo Receive Ring, not the Standard Receive Ring (for receive BDs only). This must be set by the driver, it is just copied through opaquely by the NMIC firmware.
The frame associated with this buffer descriptor has an 802.1Q
VLAN tag associated with it.
Hash type of the receive packet. It indicates which hash_type was used on the receive packet if multiple hash type are defined.
See bit [4]
An error was detected by the NIC. The detected error type is set in the Error_Flag word of the receive buffer descriptor.
11 Reserved
12 IP_CHECKSUM
13 TCP_UDP_CHECKSUM
14 TCP_UDP_IS_TCP
15 IPV6_PACKET
Indicates that the IP Checksum field is valid.
Indicates that the TCP_UDP_Checksum field is valid.
Indicates that this frame has a TCP packet in it.
Indicates that this frame has an IPv6 packet in it.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 159
BCM5718 Programmer’s Guide Time Sync Registers
Time Sync Registers
GRC MODE REG [0x6800]
Name Bits Access Default Value Description
Legacy 31:20
Time Sync Mode Enable 19 RW
Legacy 20:0
0 Write 1 to this bit to enable Time Sync Mode.
EAV REF COUNT CAPTURE LSB REG [Offset 0x6900]
The MSB and LSB registers are interface to the actual EAV Ref Counter hardware. The Counter value may be read via this pair anytime and even be overwritten by this pair anytime. While reading the pair, the hardware
Counter does not stop, only its value is latched in this pair.
The only two legal sequences of accessing this pair is Read-LSB followed by Read-MSB and Write-LSB followed by Write-MSB.
Name
EAV Reference Count
[lower half]
Reserved
Bits Access Default Value Description
31:3 RW
2:0 RO
UUUU
000
LSB of the EAV Reference Count – Reading this LSB latches a Count in this pair until the time the corresponding MSB is read.
Writing to this LSB latches the value in this pair and the subsequent write to the MSB transfers the 64bit value to EAV Ref Counter and counting immediately resumes from there.
[2:0] shall always be 000.
EAV REF COUNT CAPTURE MSB REG [Offset 0x6904]
Name
EAV Reference Count
[Upper half]
Bits Access Default Value Description
31:0 RW UUUU MSB of the EAV Reference Count – See the pairing
LSB register.
Reading this register returns the MSB of the 64- bit
EAV Ref count previously latched by performing an
LSB read.
Writing to this MSB transfers the 64-bit value, this plus previously latched LSB, to EAV Ref Counter and counting immediately resumes from there.
Back to back writes to this MSB has no effect.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 160
BCM5718 Programmer’s Guide Time Sync Registers
EAV REF CLOCK CONTROL REG [Offset 0x6908]
This register controls the EAV Reference Counter and the TimeSync related GPIO pins. Each MAC’s 1588 HW owns a dedicated TimeSync_GPIO pin which may be connected to any of its four Snap-shot/WatchDog HW logic. If a MAC needs to use more pins beyond its TimeSync_GPIO pin, it may use any or all of the four
APE_GPIO[3:0] pins – note that these pins are shared among APE HW and four MAC-1588 HW. Thus a platform must design-in these pins and have individual BootCode or FW configure this register and APEGPIO register accordingly.
Note: HW behavior shall be indeterminate in case of conflicting or duplicate assignment of GPIO pins to the same resource. A platform design MUST allocate its dedicated TimeSync_GPIO pin first before using any pin from APE_GPIO shared pool (we are talking PCB/Hardware design here).
Name
TimeSync_GPIO
Mapping
Bits Access Default Value Description
Reserved 31:30 RO
APE_GPIO[3] Mapping 29:27 RW
APE_GPIO[2] Mapping 26:24 RW
APE_GPIO[1] Mapping 23:21 RW
APE_GPIO[0] Mapping 20:18 RW
17:16 RW
00
000
000
000
000
00
–
Same as below
Same as below
Same as below
An APE_GPIO[n] pin is mapped to 1588 input/ output via this field:
000 => Do not use APE_GPIO[n] pin
001 => Reserved
010 => Reserved
011 => Reserved
100 => Use as Snap-Shot[0] Input Trigger
101 => Use as Snap-Shot[1] Input Trigger
110 => Use as Time Watchdog[0] Output
111 => Use as Time Watchdog[1] Output
The MAC/Port dedicated TimeSync_GPIO pin is mapped via this field:
00 => Use as Snap-Shot[0] Input Trigger
01 => Use as Snap-Shot[1] Input Trigger
10 => Use as Time Watchdog[0] Output
11 => Use as Time Watchdog[1] Output
Reserved
Reset on Network Link
Down -> Up
Reset on Network Link
Up -> Down
15:12 RO
11
10
RW
RW
Reset on GRC Reset and
PCIe FLR
9
Reset on PCIe reset
Reserved
RW
8 RW
7:3 RO
0x0
0
0
0
0
0x0
Reset on GRC Reset pulse
Reset on de-asserting edge of PCIe Reset
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 161
BCM5718 Programmer’s Guide Time Sync Registers
Name Bits Access Default Value Description
Resume EAV Ref Count 2
Stop EAV Ref Count 1
Reset EAV Ref Count 0
W1C
RW
W1C
0
0
0
EAV REF-COUNT SNAP-SHOT LSB[0] REG [Offset 0X6910]
This LSB and MSB pair captures the EAV Reference Count when externally triggered by the desired TimeSync/
APE_GPIO pin – a toggle serves a trigger. The only legal sequence of accessing this pair is Read-LSB followed by
Read-MSB.
Name
EAV Reference Count
Snap-shot [lower half]
Bits Access Default Value Description
31:0 RO U LSB of the EAV Reference Count as snapshotted by
TimeSync/APE_GPIO
[2:0] shall always be 000
EAV REF-COUNT SNAP-SHOT MSB[0] REG [Offset 0X6914]
Name
EAV Reference Count
Snap-shot [Upper half]
Bits Access Default Value Description
31:0 RO U MSB of the EAV Reference Count as snapshotted by
TimeSync/APE_GPIO
EAV REF CORRECTOR REG [Offset 0x6928]
Name
Correction Enable
Correction Sense
Reserved
Correction Value
Bits Access Default Value Description
31 RW
30 RW
29:24 RO
23:0 RW
0
0
0x00
0x000001
Write a 1 to Enable the Correction feature.
If 0, the correction is an addition
If 1, the correction is a subtraction
Reserved
This value is accumulated in an accumulator every
EAV REF CLK tick until it overflows – upon which the
EAV REF COUNT is corrected by a 8ns unit and the acc is reloaded.
A 0x0 value is not permissible.
TX TIME STAMP LSB REG [Offset 0x05C0]
This LSB and MSB pair captures time-stamp of transmit packets when marked to do so. The only legal sequence of accessing this pair is Read-LSB followed by Read-MSB. Once the LSB is read, the pair attains a frozen state, and is unfrozen immediately after a subsequent MSB read – if another TX packet passes to the wire between these two reads, it’s time stamp capture request is ignored by hardware.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 162
BCM5718 Programmer’s Guide Time Sync Registers
Name
TX Time Stamp [lower half]
Bits Access Default Value Description
31:0 RO U LSB of the TX Time Stamp – Reading this LSB freezes the time stamp and is only unfrozen when the corresponding MSB is read.
TX TIME STAMP MSB REG [Offset 0X05C4]
Name
TX Time Stamp [Upper half]
Bits Access Default Value Description
31:0 RO U MSB of the TX Time Stamp – Reading this MSB unfreezes the time stamp which was earlier frozen by the corresponding LSB read.
RX TIME STAMP LSB REG [Offset 0X06B0]
Name
RX Time Stamp [lower half]
Bits Access Default Value Description
31:0 RO U LSB of the RX Time Stamp.
RX TIME STAMP MSB REG [Offset 0x06B4]
This MSB and LSB pair captures the time-stamp (63-bits) of received PTP packets when qualified to do so. HW
. The MSB and LSB registers may be accessed in the order of LSB first and MSB second – else the Valid / Interlock bit would not serve its purpose.
Name
RX Time Stamp Valid /
Interlock
RX Time Stamp [Upper half]
Bits Access Default Value Description
31 RO
30:0 RO
U
U
This bit is set by hardware in conjunction with posting a new value in the Time Stamp MSB and LSB fields. When this bit is 1 and SW executes a read to the RX TIME STAMP MSB Register, hardware shall clear this bit – the reset behavior of this bit is influenced by RX PTP CONTROL Register’s [RX TIME
STAMP INTERLOCK POLICY] field.
MSB of the RX Time Stamp.
RX PTP SEQUENCE ID REG [Offset 0X06B8]
Name
Reserved
Bits Access Default Value Description
31:16 RO 0x0000
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 163
BCM5718 Programmer’s Guide Time Sync Registers
Name
PTP Sequence ID
Bits Access Default Value Description
15:0 RO 0x0000 This field reflects the Sequence ID extracted from the PTP packet which was last Time Stamped by RX hardware
Note: Hardware shall blindly capture 2 octets from offset 30 of each time-stamped PTP packet – not all message types carry SequenceId, so use this value with discretion.
RX LOCK TIMER LSB REG [Offset 0x06C0]
Name
Lock Time LSB Value
Bits Access Default Value Description
31:0 RW 0x0000 See RX Lock Timer MSB Register.
RX LOCK TIMER MSB REG [Offset 0x06C4]
See “RX PTP CONTROL REG [Offset 0X06C8]” on page 164
Name Bits Access Default Value Description
Reserved 31:16 RO
Lock Time MSB Value 15:0 RW
0x0
0x00 This, concatenated with Lock Time LSB value, constitutes a 48-bit Lock Timer value. Precision of the value is 1ns – which equates to the precision of the EAV Reference Count.
RX PTP CONTROL REG [Offset 0X06C8]
Name
Reserved
RX Time Stamp Interlock
Policy
Bits Access Default Value Description
31:28 RO
27:26 RW
0x00
01 This field determines how the successive RX Time
Stamps are retained in the RX TIME STAMP Register, given that there is a chance of over-write.
00 => HW freely over-writes RX Time Stamp values.
01 => Interlock Mode – HW does not over-write unless SW reads a posted RX Time Stamp value.
10 => Lock Timer Mode - HW does not over-write unless SW reads a posted RX Time Stamp value OR the Lock Timer has expired.
11 => Reserved
Note: Changing the state of this field while receiving PTP traffic may result in abrupt HW behavior. This field is best configured statically.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 164
BCM5718 Programmer’s Guide Time Sync Registers
Name
PTP RX Time Stamping
Enable bit-map
Bits Access Default Value Description
25:23 RW 00 Write a 1 to enable – all permutations allowed.
[25] RX Time Stamp PTP V1 - L4 packets
[24] RX Time Stamp PTP V2 - L4 packets
[23] RX Time Stamp PTP V2 - L2 packets
See field [15:0]
Reserved 22:16 RO
Stamp-able PTP Message
Type bit-map
15:0 RW
0x00
0x0000 This field, in conjunction with field [25:23] qualifies a particular PTP message type packet to be time stamped by hardware. Write a 1 in a bit position to enable time stamping of the corresponding message type and write a 0 in a bit position to disable time stamping of the corresponding message type:
[00] => Sync Event
[01] => Delay_Req
[02] => Pdelay_Req
[03] => Pdelay_Resp
[04 – 07] => N/A
[08] => Follow_Up
[09] => Delay_Resp
[10] => Pdelay_Resp_Follow_Up
[11] => Announce
[12] => Signaling
[13] => Management
[14 – 15] => N/A
Note: Not all message Types are available in PTP V1.
TX TIME WATCHDOG LSB[0] REG [Offset 0x6918]
Name
Watchdog LSB Value
Bits Access Default Value Description
31:0 RW 0x0000 See TX Time Watchdog MSB[0] Register.
TX TIME WATCHDOG MSB[0] REG [Offset 0x691C]
Name
Enable Lock Timer
Bits Access Default Value Description
31 RW 0 Write a 1 to enable Time Watchdog[0].
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 165
BCM5718 Programmer’s Guide Time Sync Registers
Name Bits Access
Watchdog MSB Value 30:0 RW
Default Value Description
0x00 This, concatenated with the Watchdog[0] LSB value, constitutes a 63-bit value. Precision of the value is
1ns – which equates to the precision of the EAV
Reference Count.
If bit[31] ==1, the desired TimeSync/APE_GPIO shall toggle as soon as EAV Reference Count[62:0] increments to match this 63-bit value.
Note: Setting this time value back in time will produce no toggle.
TX TIME WATCHDOG LSB[1] REG [Offset 0x6920]
Name
Watchdog LSB Value
Bits Access Default Value Description
31:0 RW 0x0000 See TX Time Watchdog MSB[1] Register
TX TIME WATCHDOG MSB[1] REG [Offset 0x6924]
Name Bits Access Default Value Description
Enable Lock Timer 31 RW
Watchdog MSB Value 30:0 RW
0
0x00
Write a 1 to enable Time Watchdog[0].
This, concatenated with the Watchdog[0] LSB value, constitutes a 63-bit value. Precision of the value is
1ns – which equates to the precision of the EAV
Reference Count.
If bit[31] ==1, the desired TimeSync/APE_GPIO shall toggle as soon as EAV Reference Count[62:0] increments to match this 63-bit value.
Note: Setting this time value back in time will produce no toggle.
EAV REF-COUNT SNAP-SHOT LSB[1] REG [Offset 0X6930]
This LSB and MSB pair captures the EAV Reference Count when externally triggered by the desired TimeSync/
APE_GPIO pin – a toggle serves a trigger. The only legal sequence of accessing this pair is Read-LSB followed by
Read-MSB.
Name
EAV Reference Count
Snap-shot [lower half]
Bits Access Default Value Description
31:0 RO U LSB of the EAV Reference Count as snapshotted by
TimeSync/APE_GPIO [2:0] shall always be 000.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 166
BCM5718 Programmer’s Guide Time Sync Registers
EAV REF-COUNT SNAP-SHOT MSB[1] REG [Offset 0X6934]
Name
EAV Reference Count
Snap-shot [Upper half]
Bits Access Default Value Description
31:0 RO U MSB of the EAV Reference Count as snapshotted by
TimeSync/APE_GPIO.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 167
BCM5718 Programmer’s Guide PCI
Section 9: PCI
Configuration Space
Description
PCI, PCI-X, and PCIe devices must implement sixteen 32-bit PCI registers. These registers are required for a device to have PCI compliance. The format and layout of these registers is defined in the PCI 2.2 specification.
Capability registers provide system BIOS and Operating Systems visibility into a set of optional features, which devices may implement. Although the capability registers are not required, the structure and mechanism for chaining auxiliary capabilities is defined in the PCI specification. Both software and BIOS must implement algorithms to fetch and program capabilities fields accordingly. Refer to section 6.7 of the PCI SIIG 2.2 specification. Additional PCI configuration space may be used for device-specific registers. However, devicespecific registers are not exposed to system software, according to a specification/standard. System software cannot probe device specific registers without a predetermined understanding of the device and its functionality. In summary, three types of PCI configuration space registers may be exposed by any particular device:
• Required
• Optional capabilities
• Device specific
Note: The BCM5718 is PCIe v1.1 compliant.
Network devices implement large quantities of registers, and these registers could consume huge amounts of
PCI configuration space. PCI configuration access is not very efficient, on a performance basis.
Example: Intel x86 architectures use two I/O mapped I/O addresses 0xCF8 and 0xCFC for host-based access to PCI configuration space. Should a host device driver access these I/O addresses on every device read/ write, CPU overhead would grow greatly. Generally, host device drivers should not use PCI configuration space for standard I/O and control programming. There is one special case—Universal Network Device
Interface (UNDI) drivers. UNDI drivers may not have access to host memory mapped registers when operating in real-mode; thus, an indirect mode of access is necessary. The Ethernet controller implements a PCI indirect mode for memory, registers, and mailboxes access. A specific example of a device driver, which uses indirect mode, is the Preboot Execution (PXE) driver. PXE drivers may be stored in either option
ROMs or directly in the system BIOS.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 168
BCM5718 Programmer’s Guide Configuration Space
Most host device drivers use register blocks, which are mapped into host memory. Memory Mapped I/O is an efficient mechanism for PCI devices to use system resources. The type and extent of this memory mapping depends upon the MAC’s configuration (see the operational characteristics subsection). A typical PCI device will decode a range of physical (bus) addresses, which do not conflict with physical memory or other PCI devices. Each device on the PCI bus will request a range of physical memory, and the PnP BIOS will assign mutually exclusive resources to that device. The size and range of resource is based upon each device’s hardwired programming of the BAR. The Ethernet controller implements two modes of memory mapped I/O—
Standard and Flat. I/O mapped I/O is not supported by the Ethernet controller, and there are no I/O space registers.
Note: The PCI BAR 0 register is only reset to 0 after a hard reset, otherwise it maintains its value over
GRC and PCI resets.
Two programmable blocks expose Ethernet controller functionality to host software. The first is a register block. The second is a memory block. The register and memory blocks map into address spaces based on processor context. For example, the Ethernet controller has an on-chip RISC processor. This RISC processor will have an internal view of the register and memory blocks. This view is one large contiguous and addressable range, where the register block maps starting at offset 0xC0000000. Conversely, host processors have two entirely different views. When the Ethernet controller is configured in standard mode, the register block is mapped into a 64K host memory range. The host processor must use a memory window or indirect mode to access the memory block. It is fundamental to understand that the register and memory blocks are not necessarily tied together. The PCI mode and processor context all affect how software views both blocks (see
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 169
BCM5718 Programmer’s Guide
Figure 32: Local Contexts
Register Block
Ethernet
Controller
Local Memory
Block
Configuration Space
0xC0000000
0xC0000100
0xC0000200 1
0xC0000400 1
0xC0008000
PCI Configuration
Space Shadow
Rsvd
Priority Mailboxes 1
Registers
Memory Window
0xC0010000
0xC0030000
0xC0038000
Rsvd
Rx CPU
Scratchpad
Reserved
ROM
Internal
Memory
1. The high-priority mailboxes are at offset 0x200 through 0x3FF for host standard, and the low-priority mailboxes are at
offset 0x5800 through 0x59FF for indirect mode.
2. The local memory addresses in this diagram apply to standard and internal views only. Refer to the section on memory maps
and pool configuration.
The following components are involved in Ethernet controller configuration space mapping:
• Base Address registers
• Standard mode map mode
• Flat memory map mode
• Indirect access mode
• Configuration space header
• Host memory
• MAC registers
0x00000000
2
0x00020000
2
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 170
BCM5718 Programmer’s Guide Configuration Space
• MAC local memory
Functional Overview
PCI Configuration Space Registers
The Ethernet controller configuration space can be broken into two regions: Header and Device Specific.
controller. Reserved fields in PCI configuration registers will always return zero.
PCI Required Header Region
The bit-7 of the Header Type register (Offset 0x0E) in the PCI Required Header Region is used to identify whether the device is a single function device or multifunction device.
Figure 33: Header Type Register 0xE
DeviceFunctions:
Single Function = 0
Multi-Function = 1
Func
[7] [6]
Header Layout
[0]
Note: BIOS programmers should take special care to read bit_7 in PCI Header Type register (Offset
0x0E) before scanning the Ethernet controller PCI configuration space.
Single function PCI devices may decode access to non-implemented device functions in two ways, per Section
3.2.2.3.4 of the PCI 2.2 specification:
• A single function device may optionally respond to all function numbers as the same.
• May decode the function number field and respond only to function 0.
The Ethernet controller single function chips follow the stated technique #1— BIOS code scanning multifunctions get a target response from function(s) 1–7, but these functions are essentially shadows of function 0. Software that programs to function(s) 1–7 is remapped to function 0.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 171
BCM5718 Programmer’s Guide Configuration Space
The header region is required by the PCI 2.2 specification. These registers must be implemented. The capabilities registers are optional; however, they must adhere to section 6.7 of the PCI SIIG 2.2 specification.
Each capability has a unique ID, which is well-defined. The capabilities are chained using the Next Caps field, in the capability register. The last capability will have a Next Caps field, which is zeroed.
The Device-Specific registers are shown in
.
Table 47: Device Specific Registers
Register Cross Reference
Miscellaneous Host Control
PCI State
Register Base Address
Memory Base Address
“Miscellaneous Host Control Register (offset: 0x68)” on page 283 .
“PCI State Register (offset: 0x70)” on page 286 .
“Register Base Register (offset: 0x78)” on page 287
.
“Memory Base Register (offset: 0x7C)” on page 287
.
Register Data
Memory Window Data
“Register Data Register (offset: 0x80)” on page 287
“Memory Data Register (offset: 0x84)” on page 288
.
UNDI Receive BD Standard Producer Ring
Producer Index Mailbox
“UNDI Receive BD Standard Producer Ring Producer Index Mailbox
Register (offset: 0x98–0x9C)” on page 288 .
UNDI Receive Return Ring Consumer
Index Mailbox
“UNDI Receive Return Ring Consumer Index Register (offset: 0x88–
UNDI Send BD Producer Index Mailbox
“UNDI Send BD Producer Index Mailbox Register (offset: 0x90–
.
Indirect Mode
Host software may use Indirect mode to access the Ethernet controller resources, without using Memory
Mapped I/O. Indirect mode shadows MAC resources to PCI configuration space registers. These shadow registers can be read/written by system software through PCI configuration space registers. The Ethernet controller Indirect mode registers expose the following MAC resources:
• Registers
• Local Memory
• Mailboxes
Indirect mode access can be used in conjunction with Standard mode PCI access. Indirect mode has no interdependency on other PCI access modes and is a mode in itself.
Note: Host software must assert the Indirect_Mode_Access bit in the Miscellaneous Host Control
register (see “Miscellaneous Host Control Register (offset: 0x68)” on page 283 ) to enable indirect
mode.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 172
BCM5718 Programmer’s Guide Configuration Space
Indirect Register Access
Two PCI configuration space register pairs give host software access to the Ethernet controller register block.
The Register_Base_Address register creates a position in the MAC register block. Valid positions range from
0x0000–0x8000 and 0x30000–0x38800 ranges. Access to the register block from 0x8000–0x30000, should be avoided and is not necessary. The Flat and Standard modes do map a memory window into the 0x8000–0xFFFF ranges; however, the Memory Indirection register pair provides a more efficient mechanism to access the
Ethernet controller memory block. The Register_Data register allows host software to read/write, from the indirection position. The Register_Base_Address register can be perceived as creating a cursor/pointer into the register block. The Register_Data register allows host software to read/write to the location, specified by the
Register_Base_Address. This register pair accesses the Ethernet controller register block (see
).
Note: If indirect register access is performed using memory write cycles (i.e., by accessing the
Register_Base_Address and Register_Data registers through memory mapped by the PCI BAR register), as opposed to PCI configuration write cycles, the host software must insert a read command to the Register_Base_Address register between two consecutive writes to the
Register_Base_Address and Register_Data registers.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 173
BCM5718 Programmer’s Guide
PCI Configuration Space
Figure 34: Register Indirect Access
Register Indirect Access
BCM57XX Ethernet Controller
Register Block
0x00000000
0x00000400
Address may be located anywhere
Configuration Space
BCM57XX
Ethernet
Controller
Registers Address may be located anywhere
0x00008000
BusX
DeviceY
Function Z
Register Base Address
Register Data Register
Not Accessible
via Register
Indirect Mode
0x00038000
0x00038800
Rx CPU
ROM
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 174
BCM5718 Programmer’s Guide Configuration Space
Indirect Memory Access
Memory indirect mode operates in the same fashion to register indirect mode. There is a PCI configuration space register pair, which is used to access the Ethernet controller memory block. The
Memory_Window_Base_Address register positions a pointer/cursor in the local memory block. Unlike the
Register_Base_Address register, the Memory_Window_Base_Address register may position at any valid offset.
Access to ranges 0x00000–0x1FFFF is allowable. The Memory_Window_Data register is the read/write porthole for host software, using the previously positioned pointer/cursor. This register pair accesses the
Ethernet controller local memory block (see Figure 35 on page 176 ).
Note: If Indirect Memory Access is performed using memory write cycles (i.e., by accessing the
Memory_Window_Base_Address and Memory_Window_Data registers through memory mapped by the PCI BAR register), as opposed to PCI configuration write cycles, the host software must insert a read command to the Memory_Window_Base_Address register between two consecutive writes to the
Memory_Window_Base_Address and Memory_Window_Data registers.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 175
BCM5718 Programmer’s Guide
PCI Configuration Space
Figure 35: Indirect Memory Access
Indirect Memory Access
BCM57XX Ethernet Controller
Memory Block
0x00000000
Configuration Space
Address may be located anywhere
Internal
Memory
BusX
DeviceY
Function Z
Memory Base Address
Memory Data Register
0x00020000
Address may be located anywhere
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 176
BCM5718 Programmer’s Guide Configuration Space
UNDI Mailbox Access
The UNDI mailboxes are shadows of Ethernet controller mailbox registers. All mailboxes reside in the Ethernet controller register block, not memory block. Unlike register and memory indirect access, the UNDI Mailboxes shadows are mapped 1:1 to a Ethernet controller register; these shadow registers do not have an address register.
• The UNDI_RX_BD_Standard_Ring_Producer_Index_Mailbox register shadows a mailbox located at offset
0x5868 (see “Receive BD Return Ring 3 Consumer Index Register (offset: 0x5898-0x589F)” on page 465
), in the Ethernet controller register block. Any index update (write) to the
UNDI_RX_BD_Standard_Ring_Producer_Index_Mailbox will advance the standard producer ring index; software signals hardware that an RX buffer descriptor is available.
• The UNDI_RX_BD_Return_Ring_Consumer_Index_Mailbox register corresponds to a mailbox located at offset 0x5880 (see
“Receive BD Standard Producer Ring Index Register (offset: 0x5868)” on page 464 ). A
update (write) to this register indicates that host software has consumed a RX buffer descriptor(s); return rings contain filled Enet frames, from the receive MAC.
• Finally, the UNDI_TX_BD_Host_Producer_Mailbox register maps to register offset 0x5900 () in the
Ethernet controller register block. Host software writes to this register when Ethernet frame(s) are ready to be transmitted. Host software writes the index of buffer descriptor, which is ready for transmission.
Notice that all these UNDI shadows are the first or primary ring and not all the rings are shadowed into PCI configuration space. For example, Receive Return rings 2–16 do not have shadow registers. UNDI drivers only require a minimal set of registers to provide basic network connectivity. Functionality is the most important consideration. Fifteen additional receive return rings would extend the size of the Device Specific portion of the PCI Configuration Space registers.
The UNDI shadow registers alias three registers in the Ethernet controller register block (see
).
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 177
BCM5718 Programmer’s Guide
Figure 36: Low-Priority Mailbox Access for Indirect Mode
Indirect Mailbox Access
Configuration Space
PCI Configuration Space BCM57XX Register Block
0x00000000
0x00000400
BusX
DeviceY
Function Z
Low Priority
Mailbox Region
Not Aliased
UNDI Rx BD Std Ring
Producer Index Mailbox
UNDI Rx BD Return Ring
Consumer Index Mailbox
UNDI Tx BD NIC
Producer Index Mailbox
0x00005800
0x00005868
0x0000586F
0x0005880
0x000588F
0x0005980
0x00005987
0x00005BFF
Rx BD Std Ring
Producer Index
Rx BD Return Ring 1
Consumer Index
Tx BD Ring 1
NIC Producer Index
Not Aliased
0x00008000
BCM57XX
MAC
Registers
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 178
BCM5718 Programmer’s Guide Configuration Space
Standard Mode
Standard mode is the most useful memory mapped I/O view provided by the Ethernet controller (see
). 64K of host memory space must be made available. The PnP BIOS or OS will program BAR0 and
BAR1 with a base address where the 64K address region may be decoded. The BAR registers point to the beginning of the host memory mapped regions where Ethernet controller can be accessed.
Figure 37: Standard Memory Mapped I/O Mode
PCI Configuration Space Host Memory Address Space
Physical
Memory
Access to this memory region is remapped by the hardware.
BusX
DeviceY
Function Z
BAR0
BAR1
BAR + 0x00000000
BAR + 0x000000FF
BAR + 0x000001FF
BAR + 0x000003FF
PCI Configuration
Space Registers
(Shadow Copy)
Reserved
High Priority
Mailboxes
Registers
BAR + 0x00007FFF 64K
Memory
Window
BAR + 0x0000FFFF
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 179
BCM5718 Programmer’s Guide Configuration Space
The Ethernet controller resources listed in the following are decoded in the 64K address block.
Offset
0x00000000–0x000000ff
0x00000200–0x000003ff
0x00000400–0x00007fff
0x00008000–0x0000ffff
Table 48: PCI Address Map Standard View
Name
PCI Configuration space
High-Priority Mailboxes
Ethernet controller registers
Memory Window
Size
256 bytes
512 bytes
31 KB
32 KB
32K is partitioned for MAC control registers and 32K available for a memory access window. Range 0x0000–
0x00FF is a complete shadow of the PCI configuration space registers—host software can also read/write to the Ethernet controller’s PCI configuration space registers via the host memory map. Host software may use the shadow registers to change PCI register contents and avoid PCI configuration cycles (transactions). Again, using the host memory map is slightly more efficient. The MAC’s control/status registers are mapped from
0x0400–0x8000. See
Section 13: “Ethernet Controller Register Definitions,” on page 270 for complete register
and bit definitions. Finally, the memory window range is 0x8000–0xFFFF. This 32K window is set in the PCI
Configuration space using the Memory_Window_Base_Address register (see Figure 38
). Bits 23:15 set the window aperture and bits 14:2 are effectively ignored/masked off. Bits 14–2 are relevant when host software uses memory indirection and the Memory_Window_Data register.
Figure 38: Memory Window Base Address Register
Window aprerature set by bits 23-15
These bits are ignored - 32K window position.
Rsvd
[31:24]
Window
[23:15]
XXXX (Dont Care)
[14:2]
Rsvd
[1:0]
shows how the 32K window can float in the Ethernet controller’s local memory. The window aligns on 32K boundaries.
Example: The memory window may start on the following addresses: 0x8000, 0x10000, and 0x18000. The window aperture may be positioned in the internal memory range 0x00000000 to 0x0001FFFF. When host software reads/writes to PCI_BAR + 32K + OFFSET in the host memory space, the Ethernet controller translates this read/write access to Memory_Window_Base_Address + OFFSET. Host software must not read/write from any address greater than PCI_BAR + 64K, since this memory space is not decoded by the
Ethernet controller. Such an access may be decoded by another device, or simply go unclaimed on the PCI bus.
Figure 39 on page 181 shows the relationship between the Memory_Window_Base_Address register
and the Memory Window.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 180
BCM5718 Programmer’s Guide Configuration Space
PCI Configuration Space
Figure 39: Standard Mode Memory Window
Standard Mode
Memory Window
Host Memory Address Space Local Memory Address Space
0x00000000 0x00000000
BusX
DeviceY
Function Z
Mem Wnd
Base Addr
Physical
Memory
32K
Window may be located anywhere
Host software may access the BCM57XX
Ethernet Controller local memory using this window
Registers
32K
Memory
Window
32K
Window may be located anywhere
Internal
32K
0x00020000
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 181
BCM5718 Programmer’s Guide
PCI Configuration Space
BusX
DeviceY
Function Z
BAR0
BAR1
0x00000000
0x00000200
0x00000400
0x00008000
0x00100000
0x00110000
0x00130000
0x00180000
0x001C0000
0x01000000
Host Memory Address Space
Physical
Memory
PCI Cfg Space Registers
(Shadow Copy)
Reserved
High Priority Mailboxes
Registers
Memory
Window
Reserved
IRQ Mailbox 0-3
Reserved
General Mailbox 1-8
Reserved
Rx BD Producer Index
Reserved
Rx BD Return Ring 1-4
Consumer Index
Reserved
Tx BD Ring
Host Producer Index
Reserved
Tx BD Ring
NIC Producer Index
Reserved
Configuration Space
32 MB
Memory
0x01FFFFFF
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 182
BCM5718 Programmer’s Guide Configuration Space
0x00000000
0x00000200
0x00000400
0x00008000
0x00100000
0x00110000
0x00130000
0x00140000
0x00180000
0x001C0000
0x01000000
Figure 40: Techniques for Accessing Ethernet Controller Local Memory
Host Memory Address Space
Local Memory Address Space
Physical
Memory
0x00000000
Device Internal
Memory PCI Cfg Space Registers
(Shadow Copy)
Reserved
High Priority Mailboxes
Registers
Memory
Window
Reserved
IRQ Mailbox 0-3
Reserved
General Mailbox 1-8
Reserved
Rx BD Send
Producer Index
Reserved
Rx BD Return Ring 1-16
Consumer Index
Reserved
Tx BD Ring 1-16
Host Producer Index
Reserved
Tx BD Ring 1-16
NIC Producer Index
Reserved
Memory Window
Window may be located anywhere
Window may be located anywhere
Memory Window
Device
External
Memory
Memory
0x01FFFFFF 0x00020000
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 183
BCM5718 Programmer’s Guide Configuration Space
Memory Mapped I/O Registers
The following Ethernet controller registers are used in the mode configuration of the PCI memory-mapped I/O.
PCI Command Register
The PCI_Command register is 16-bits wide (see
Figure 41 ). The Ethernet controller does not have I/O mapped
I/O. The I/O_Space bit is de-asserted by hardware. The Ethernet controller does support
Memory_Mapped_Memory and hardware will assert the Memory_Space bit. Both these bits are read-only and are usually read by the PnP BIOS/OS. The BIOS/OS examines these bits to assign non-conflicting resources to
PCI devices.
Figure 41: PCI Command Register
I/O Space
Read Only
Always = 0
Memory Space
Read/Write
Rsvd
[15:10]
Fast
Back 2
Back
[9]
Sys
Err
[8]
Step
Ctrl
[7]
Parity
Error
VGA
Snoop
Write
Invld
Spec
Cycle
Bus
Mast
Mem I/O
[6] [5] [4] [3] [2] [1] [0]
PCI State Register
The PCI_State register is 32-bits wide. Operating mode is set with the Flat_View bit in the PCI_State register.
When the Flat_View bit is asserted, the Ethernet controller decodes a 32M of block host memory. When the
Flat_View bit is de-asserted, the Ethernet controller decodes a 64K block of host memory.
PCI Base Address Register
The PCI_Base_Address Register (BAR) specifies the location of a Ethernet controller memory mapped I/O block.
The Ethernet controller mode configuration (Flat vs. Standard) affects how the BAR is setup (see
).
• Bits 4–31 in the PCI_Base_Address register are selectively programmable based on the amount of host memory requested. The PnP BIOS/OS will use an algorithm to test the BAR bits and determine the amount of physical memory requested.
• The Memory_Space_Indicator bit designates whether the BAR is memory or I/O mapped. The Ethernet controller hard codes the Memory_Space_Indicator bit to zero (de-asserted).
• The Location/Type bits specify locations in host memory space where a device can decode physical addresses. The Ethernet controller memory mapped I/O range may be placed anywhere in 64-bit address space (Type = 10).
• The Ethernet controller deasserts the Prefetchable bit to indicate that the memory range should not be cached.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 184
BCM5718 Programmer’s Guide Configuration Space
Figure 42: PCI Base Address Register
Binary Weighted Value: the 1st programmable bit (ascending) indicates requested block size.
Prefetchable:
0 = Disabled
1 = Enabled
Location:
00 = Anywhere
01 = Below 1 MB
10 = Anywhere in 64-bit Addr Space
11 = Reserved
Memory Space Indicator:
I/O = 1
Memory = 0
Base Address
[31:4]
P
[3]
Type
[2:1]
M
[0]
The Ethernet controller 64K memory mapped I/O block is determined by the first programmable bit in the BAR.
When the MAC is configured in standard mode, the mask 0xFFFF0000 identifies the BAR bits, which are programmable. Bit 16 is the first bit encountered in the scan upward, which is programmable; bits 0–3 are
returned to the OS/BIOS during resource allocation.
Figure 43: PCI Base Address Register Bits Read in Standard Mode
Binary Weighted Value:
0x00010000 = 64K
X's are don't cares
Ignored:
Bits 0-3
XXXX XXXX XXXX XXX1 0000 0000 0000
[31:4]
0
[3]
11
[2:1]
0
[0]
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 185
BCM5718 Programmer’s Guide Bus Interface
Bus Interface
Description
The read/write DMA engines both drive the PCIe interface. Typically, each DMA engine alternates bursts to the
PCIe bus, and both interfaces may have outstanding transactions on the PCI bus. The BCM5718 family architecture identifies two channels—a read DMA channel and a write DMA channel. Each channel corresponds to the appropriate DMA engine (see
). The configuration of the DMA engines and the PCI interface is discussed in this section.
Figure 44: Read and Write Channels of DMA Engine
Read Channel Write Channel
DMA Write
Engine
Write FIFO
PCI Interface
DMA Read
Engine
Read FIFO
PCIe Bus
The following architectural components are involved in the configuration of the PCI/DMA interface:
• DMA read engine
• DMA write engine
• DMA read FIFO
• DMA write FIFO
• PCIe interface
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 186
BCM5718 Programmer’s Guide Expansion ROM
• PCI state register
• DMA read/write register
Operational Characteristics
Read/Write DMA Engines
Software must enable the bus master DMA bit for the Ethernet controller. The Ethernet controller is a busmastering device and the PCI interface requires that the Bus_Master enable bit be set by either the BIOS or host device driver. The bus master is the PCI transaction initiator. A PCI target will claim the transaction driven by the bus master. The Bus_Master enable bit is located in the PCI configuration space Command register and this bit is read/write. The bit defaults to cleared/disabled after device reset.
The read and write DMA channels use FIFOs to buffer small amounts of PCI bus data. The FIFOs provide elasticity for data movement between internal memory and the PCI interface. Host software may configure
DMA watermarks—values where PCI activity is enabled/disabled.
When enqueued data is less than the watermark value, PCI bus transactions are inhibited. The DMA channel will wait until the FIFO fills above the threshold before initiating PCI transactions. Host software may configure the DMA_Write_Watermark bit fields to set the activity threshold in the write FIFO. The
DMA_Write_Watermark bit field is read/write and is also located in the DMA Read/Write register. The write watermark registers default to zero after power-on reset.
Expansion ROM
Description
The expansion ROM on the Ethernet controller is intended for implementation of PXE (Preboot Execution
Environment). The devices support expansion ROM of up to 16 MB.
Operational Characteristics
By default, the Expansion ROM is disabled and the firmware has to explicitly enable this feature by setting
this bit is enabled, the boot code firmware handles the Expansion ROM accesses of the device.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 187
BCM5718 Programmer’s Guide Power Management
BIOS
The BIOS detects if a PCI device supports Expansion ROM or not by writing the value 0xFFFFFFFE to
Expansion_ROM_Base_Register (register 0x30 of PCI configuration). The BIOS then reads back from this register. If the value is nonzero, then this PCI device supports Expansion ROM; otherwise, it does not. The
Ethernet controller returns a nonzero value appropriate for the expansion ROM size selected in NVRAM (see
Section 4: “Common Data Structures,” on page 70
) when Expansion ROM is enabled
(PCI_State.PCI_Expansion_ROM_Desired bit is set to 1). On the other hand, if the
PCI_Expansion_ROM_Desired bit cleared, then the Ethernet controller returns a value of 0x00000000. This indicates to the BIOS that no Expansion ROM is supported.
If a PCI device supports Expansion ROM, the BIOS will assign a Expansion Base address to the device. It then checks for a valid ROM header (0x55 0xAA as first two bytes, and so forth) and checksum. If the ROM header and image are valid, the BIOS will copy the Expansion ROM image to HOST’s Upper Memory Block (UMB) and invoke the initializing entry point.
Preboot Execution Environment
Preboot Execution Environment (PXE) is implemented as an Expansion ROM in the NIC implementation. In the
LOM implementation, PXE normally resides in the system BIOS. In the NIC implementation, PXE image is stored in the NVRAM. Upon power on reset of the Ethernet controller, the RX RISC will load the boot code from the
NVRAM into RX RISC scratch pad and execute. This boot code will program the device with programmable manufacturing information (such as MAC address, PCI vendor ID/device ID, etc.). If PXE is enabled, the boot code responds to the Expansion ROM accesses of system BIOS.
Boot code is executed when the Ethernet controller is reset via PCI Reset or S/W device reset. PXE initialization should only be necessary after a PCI reset. The boot code differentiates PCI Reset and driver initiated software reset by checking content in Internal Memory at 0xb50. If the content is 0x4B657654, then the reset is due to driver initiated software reset. Therefore, the device driver has to initialize 0xb50 with 0x4B657654 before issuing a S/W device reset.
Power Management
Description
The Ethernet controller is compliant with the PCI v2.0 (PCI v2.1 for BCM5719) power management specification. The MAC is programmable to two ACPI states: D0 and D3. The D0 state is a full power, operational mode—all the MAC core functions run at the highest clocking frequency, and components are fully functional.
The MAC may be either initialized or un-initialized in the D0 ACPI state. An un-initialized D0 state is entered through a device reset or PME event; the MAC functional blocks are not started and initialized. Host software must reset/initialize hardware blocks to transition the device to a D0 initialized (active) state. The D0 active state places the device into a full power/operational mode. Receive and transmit data paths are fully operational, and the PCI block is initialized for bus mastering DMA.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 188
BCM5718 Programmer’s Guide Power Management
Host device drivers do not differentiate between D3 hot and D3 cold states. ACPI-compliant device drivers are unloaded and quiescent in the D3 state and PCI slot power state is transparent. When the MAC is in D3 hot state, PCI slot power (3.3V or 5.0V) is available to power the PCI I/O pins. The PCI configuration and memory space may be accessed in D3 hot state. The core clock must remain enabled, so the MAC can respond to PCI configuration and memory transactions. The Disable_Core_Clock bit, in the PCI Clock Control register enables/ disables clocking in the core clock domain. A D3 cold state provides only the PCI Vaux supply—PCI slot power is not present. The MAC will consume a maximum of 375 mA in a D3 cold power management mode.
The following functional blocks are integral to MAC power management:
• PMSCR register
• PCI Clock Control register
• Miscellaneous Control register
• WOL
• PCI Vaux Supply
• PCI Slot Power Supply
• GPIO
Operational Characteristics
specific usage; however, Broadcom encourages both software and hardware engineers to follow the Broadcom design guidelines and application notes. NIC and LOM designs use external board level logic to switch power regulators for D3 ACPI mode.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 189
BCM5718 Programmer’s Guide
D0
Active
Power Management
Figure 45: Power State Transition Diagram
Device driver sets
PMCSR register to power state D3 and
GPIO0, 1, and 2
Other I/O functions and core logic powered from PCI Vaux
D3 Hot
PCI I/O Pins Powered from PCI Slot Supply
Device driver initializes state machines, memory arbiter, and so on
PME Asserted or
Software reset
Device driver sets
PMCSR register to power state D3 and
GPIO0, 1, and 2
PCI Slot
(3.3v or 5.0v) Power
Removed
PCI I/O Pins Powered from PCI Vaux Supply
D0
Unitialized
D3 Cold
(PME
Enabled)
PME Asserted or
Hardware reset
Device State D0 (Uninitialized)
The D0 state is entered after a PCI reset or device (software) reset. The assertion of PME causes the PCI bridge to drive RST. The MAC hardware blocks are not initialized in this state.
Example: The RX engine, TX engine, multicast filter, and memory arbiter are all uninitialized. All the MAC functional blocks are powered.
Device State D0 (Active)
Host software has initialized the MAC hardware blocks. The RX and TX data paths are ready to send/receive
Ethernet packets. The PCI block is available to DMA packets to host memory. This is a full power ACPI
Hash Registers” on page 137 ) to move the MAC into a D0 active state.
When the BCM5718 family NetXtreme devices detect that main power is lost and it is still in the D0 state, it will reset itself to the D3 (Cold) state and then operate in 10/100 mode, like the OOB WOL state.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 190
BCM5718 Programmer’s Guide Power Management
Device State D3 (Hot)
The MAC’s configuration space and memory mapped I/O blocks are accessible in D3 hot state. The PCI I/O drivers are still powered by slot power in this state. However, host software has switched the MAC to use PCI
Vaux for VDD_CORE and VDD_IO. GPIO pins 0, 1, and 2 are configured before the transition to this state. The
RX RISC processor clock has been stopped in this state. The core clock remains active so PCI transactions may be processed by the MAC. This is a low-power state where some key components have been powered down.
The physical layer auto-advertises 10 Mbps capability in this state, and link is set to 10 Mbps half-duplex or fullduplex. The PHY is configured for WOL mode. WOL pattern filters are initialized and active; the MAC will process Magic Packets™. The host chipset implements the power management policy for the PCI bus; the MAC driver does not influence the PCI Vaux or Slot power supply.
Note: The drivers should use configuration cycles (not the memory write cycles) to write to the
PMCSR register at offset 0x4C for putting the device in D3 Hot state.
Device State D3 (Cold)
The MAC is completely powered by PCI Vaux in D3 cold. PCI configuration space and memory mapped I/O are not available. The only portion of the MAC active is the WOL pattern and Magic Packet filters
1
. The MAC will assert a PME in this state and indicate to the host bridge that a wake up event has occurred. The host bridge will normally provide PCI Slot power and then reset the device. GPIO pins 0, 1, and 2 are configured the same as D3 hot. Host software does not differentiate between D3 hot/cold. The MAC and PHY will not consume more than 375 mA in this mode. The integrated PHY must negotiate for 10 Mbps half/duplex speed. The PHY WOL mode is configured.
Note: The PCIe devices support the PCIe power management which is compatible with PCI bus power management.
Wake on LAN
See “Wake on LAN Mode/Low-Power” on page 213 .
1. Magic Packet™ is a registered trade mark of AMD.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 191
BCM5718 Programmer’s Guide Power Management
GPIO
The use of GPIO pins for power management is design-specific, though Broadcom-delivered drivers use GPIO pins in the manner listed in
Table 49 . This usage is only applicable when the Ethernet controller is configured
for a NIC design; it is not applicable to LAN-on-Motherboard (LOM) designs.
Function
VAux
VMain
Table 49: GPIO Usage for Power Management for Broadcom Drivers a
Description
Sequence for switching to VAux
Sequence for switching to VMain
GPIO0
1 x
0
1 x x
GPIO1
1
1
1
1
0
1
GPIO2
1
1
0 x x x a. x= Don’t Care
Power Supply in D3 State
shows the power supply to various power pins on the Ethernet controller, and it is assumed that host software has switched power regulators using GPIO pins 0, 1, and 2.
Supply Pins
VDDIO
VDDC
Table 50: Ethernet Controller Power Pins
D0 Normal
PCIe Slot Vmain
PCIe Slot Vmain
D3 Hot, D3 Cold
PCIe Slot Vaux
PCIe Slot Vaux
Clock Control
Certain functional blocks in the MAC architecture should be powered down before a transition to D3 ACPI state. MAC clock generators/PLLs drive transistor level logic, which switch states on every clock pulse.
Transistor level switching consumes power (mW). Software should selectively disable clocking to non-essential functional blocks. Software must set the Enable_Clock_Control_Register bit in the Miscellaneous Host Control register (see
“Miscellaneous Host Control Register (offset: 0x68)” on page 283
); the assertion of this bit allows host software to configure the PCI clock control register. The following clock bits should be configured in the
PCI Clock Control register:
• RX RISC clock disable
• Select alternate clock—the 133 MHz PLL is not used as reference clock.
Note: For the 57818 family chip, the clock control register (offset: 0x68) does not need to be configured.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 192
BCM5718 Programmer’s Guide Power Management
Device ACPI Transitions
Host software must program the Power Management Control/Status (PMSCR) register to transition the device
Note: The D1 and D2 configurations are not supported in the Ethernet controller. The D1 and D2 bit configurations are available for applications, where D1 and D2 states are introduced for board level designs—the bits provide flexibility to the application. The Broadcom reference NIC/LOM designs do not use D1 and D2 states; therefore, host software should avoid setting these states. Before the Mac is moved into the D3 state, the clocks and GPIO must be configured (see above sections).
The PME signal is enabled in the PMSCR by asserting the PME_Enable bit. Device drivers/BIOS may also read the PME_Status bit to determine whether the event has been driven; PME_Status is a write to clear bit. The type and supported power management features for the Ethernet controller are reported in the Power
Management Capabilities (PMC) register. System software and BIOS may read this register to enumerate and detect the power management features supported by the NIC/LOM. For example, the Ethernet controller can assert PME from both D3 hot and cold states. The PME_Support bit field in the PMC register will reflect this capability.
Disable Device Through BIOS
The Ethernet controllers can be disabled (that is, placed in Low Power IDDQ mode) through BIOS by writing the value of DEADDEADh to shared memory location of B50h. This eliminates the need for BIOS to execute the device specific procedure for disabling the MAC device. The BIOS must do the following steps to disable the device.
1. Config cycle, write 88h to location 68h.
2. Config cycle, write 0B50h to location 7Ch.
3. Config cycle, write DEADDEADh to location 84h.
Note: The BIOS should first place the controller into the D3 power state prior to writing the
0xDEADDEAD signature value.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 193
BCM5718 Programmer’s Guide Endian Control (Byte and Word Swapping)
Endian Control (Byte and Word Swapping)
Note: Setting register 0x68 bit 2 (Enable Endian Byte Swap) causes PCI configuration reads to the following registers to become swapped:
• 0x40
• 0x68 - 0x9C
• 0xF4 - 0xFF
This is different behavior from previous NetXtreme controllers. Reference BCM5718 Family errata relating to byte swap control for additional information.
Background
There are two basic formats for storing data in memory—little-endian and big-endian. The endianess of a system is determined by how multibyte quantities are stored in memory. A big-endian architecture stores the most significant byte at the lowest address offset while little-endian architecture stores the least significant byte at the lowest address offset.
For example, the 32-bit hex value 0x12345678 would be stored in memory as shown in the following table.
Address
Big Endian
Little Endian
00
12
78
Table 51: Endian Example
01
34
56
02
56
34
03
78
12
Another method of viewing how this data would be stored is shown in the following tables.
Storage Byte
Data Contents
00
12
Table 52: Storage of Big-Endian Data
01
34
02
56
03
78
Storage Byte
Data Contents
03
12
Table 53: Storage of Little-Endian Data
02
34
01
56
00
78
Examples of big-endian platforms include SGI Irix, IBM RS6000, and SUN.
Examples of little-endian platforms include Intel x86 and DEC Alpha.
PCI assumes a little-endian memory model. PCI configuration registers are organized so that the least significant portion of the data is assigned to the lower address.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 194
BCM5718 Programmer’s Guide Endian Control (Byte and Word Swapping)
Architecture
The Ethernet controller is internally a big-endian machine, and its internal processors are big-endian devices.
The Ethernet controller stores data internally in big-endian format using a 64-bit memory subsystem.
However, many hosts (e.g., x86 systems) use the little-endian format, and the PCI bus uses the little-endian format. Therefore the Ethernet controller has a number of byte swapping options that may be configured by software so that Little or Big Endian hosts can interface as seamlessly as possible with Ethernet controller over
PCI. The Ethernet controller has the following bits that control byte and word swapping:
• Enable Endian Word Swap (bit 3, Miscellaneous Host Control register (offset 0x68 into PCI Config register,
swapping when accessing the Ethernet controller via the PCI target interface.
• Enable Endian Byte Swap (bit 2, Miscellaneous Host Control register (offset 0x68 into PCI Config register, see
“Miscellaneous Host Control Register (offset: 0x68)” on page 283 ). If 1, this register enables byte
swapping (within a 32-bit word) when accessing the Ethernet controller via the PCI target interface.
• Word Swap Data (bit 5, Mode Control register (offset 0x6800 into the Ethernet controller registers). If 1, this register enables word swapping of frame data when it comes across the bus.
• Byte Swap Data (bit 4, Mode Control register (offset 0x6800 into the Ethernet controller registers). If 1, this register enables byte swapping of frame data when it comes across the bus.
• Word Swap Non-Frame Data (bit 2, Mode Control register (offset 0x6800 into the Ethernet controller registers). If 1, this register enables word swapping of non frame data (i.e., buffer descriptors, statistics, etc.) when it comes across the bus.
• Byte Swap Non-Frame Data (bit 1, Mode Control register (offset 0x6800 into the Ethernet controller registers). If 1, this register enables byte swapping of non frame data (i.e., buffer descriptors, statistics, etc.) when it comes across the bus.
The setting of the above swapping bits will affect the order of how data is represented when it is transferred across PCI. Since byte swapping is a confusing subject, examples will be shown that reflect how each byte swapping bit works
Enable Endian Word Swap and Enable Endian Byte Swap Bits
The Enable Endian Word Swap, and Enable Endian Byte Swap bits affect whether words or bytes are swapped during target PCI accesses. Thus, these bits affect the byte order when the host is directly reading/writing to registers or control structures that are physically located on the Ethernet controller. These bits do not affect the byte ordering of packet data or other structures that are mastered (DMAed) by the Ethernet controller.
When the Ethernet controller is accessed via PCI (which is little endian) as a PCI target, the Ethernet controller must implicitly map those accesses to internal structures that use a 64-bit Big Endian architecture. In the default case where no swap bits are set the Ethernet controller maps PCI data to internal structures shown in
.
Internal Byte #
Internal Bit #
Example Content
MSB
0
63
88
1
89
2
48 47
8A
3
8B
4
32 31
8C
5
8D
6
16 15
8E
LSB
7
8F
0
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 195
BCM5718 Programmer’s Guide Endian Control (Byte and Word Swapping)
PCI Byte #
PCI Bit #
Example Content
MSB
7
63
88
6
89
5
48 47
8A
4
8B
3
32 31
8C
2
8D
1
16 15
8E
LSB
0
8F
0
Figure 46: Default Translation (No Swapping) on 64-Bit PCI
Internal Byte OrderingPCI Byte Ordering
31 16 15 0
0x00 88 89 8A 8B
0x04 8C 8D 8E 8F
31
8C
88
16
8D
89
15
8E
8A
0
8F
8B
0x00
0x04
Figure 47: Default Translation (No Swapping) on 32-bit PCI
As illustrated above, because the Ethernet controller uses an internal 64-bit big endian architecture, it will map
(by default) the most significant byte of an 8-byte (64-bit) internal quantity to the most significant byte on a
64-bit PCI bus. This works nicely for quantities (fields) that are 64 bits in size (e.g., a host physical address).
However, this can be confusing for quantities that are 32 bits in size. Without Word Swapping enabled, the host could easily access the wrong 32-bit quantity when making a 32-bit access.
Take, for example, a Ring Control Block (RCB). RCBs are on-chip structures and read/written by the host via PCI target accesses. The table below shows the big-endian layout of an on-chip RCB:
Byte #
Bit # 31
MSB
MSB
0
Table 54: RCB (Big Endian 32-Bit Format)
1 2
16 15
Host Ring Address
MAX_Len LSB
NIC Ring Address
Flags
3
0
LSB
0x00
0x04
0x08
0x0C
If Word Swapping is not enabled, and the host made a 32-bit read request to address 0x08, the four bytes of data returned on the PCI bus would actually be the NIC Ring Address rather than the Max_Len and Flags fields.
This initially might seem counter-intuitive, but is explained in
Figure 47 on page 196 . Therefore, if a software
driver running on an x86 host (Little Endian) referenced on-chip data structures as they are defined in the
Ethernet controller data sheet, the driver should set the Enable Endian Word Swap bit. By setting this bit, the translation would be as follows:
Internal Byte OrderingPCI Byte Ordering
31 16 15 0
0x00 88 89 8A 8B
31
88
16
89
15
8A
0
8B 0x00
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 196
BCM5718 Programmer’s Guide Endian Control (Byte and Word Swapping)
0x04 8C 8D 8E 8F 8C 8D 8E 8F 0x04
Figure 48: Word Swap Enable Translation on 32-Bit PCI (No Byte Swap)
The only side effect for a little endian host that sets the Enable Endian Word Swap bit would be that the driver would have to perform an additional word swap on any 64-bit fields (e.g., a 64-bit physical address) that were given to the driver by the Network Operating System (NOS).
Little-endian hosts will not want to set the Enable Endian Byte Swap bit for target accesses. This bit is intended to be used by big endian systems that needed PCI data (little endian) translated back to big endian format.
Note: Some big endian systems automatically do this depending on the architecture of the host’s PCI to memory interface.
The following figures show the translation of data when the Enable Endian Byte Swap bit is set:
0x00
Internal Byte Ordering PCI Byte Ordering
31 16 15 0
88
0x04 8C
89
8D
8A
8E
8B
8F
31
8F
8B
16
8E
8A
15
8D
89
0
8C
88
0x00 88
0x04 8C
Figure 49: Byte Swap Enable Translation on 32-Bit PCI (No Word Swap)
Internal Byte Ordering PCI Byte Ordering
31 16 15 0
89
8D
8A
8E
8B
8F
31
8B
8F
16
8A
8E
15
89
8D
0
88
8C
0x00
0x04
0x00
0x04
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 197
BCM5718 Programmer’s Guide Endian Control (Byte and Word Swapping)
Word Swap Data and Byte Swap Data Bits
The Word Swap Data, and Byte Swap Data bits effect how packet data is ordered on the PCI bus. These only affect how packet data is ordered, and do not affect non-frame data (i.e., buffer descriptors, statistics block, etc.). In other words, these bits effect how data is transferred to/from host send/receive buffers.
Example: If Ethernet controller were to receive a packet that had the following byte order:
01 02 03 04 05 06 07 08 09 0A 0B 0C 0D 0E 0F 10
D1 D2 D3 D4 D5 D6 S1 S2 S3 S4 S5 S6 T1 T2 IP1 IP2
Where:
• D1–D6 consists of the packet’s destination address (Byte D0 is the first byte on the wire);
• S1–S6 is the source address;
• T1–T2 is the Ethernet type/length field;
• IP1–IP2 are the first two bytes of the IP header which immediately follow the type/length field.
The packet would be stored internally in big endian format:
B0
63–56
D1
S3
B1
55–48
D2
S4
Table 55: Big-Endian Internal Packet Data Format
B2
47–40
D3
S5
B3
39–32
D4
S6
B4
31–24
D5
T1
B5
23–16
D6
T2
B6
15–8
S1
IP1
B7
7–0
S2
IP2
However, when the data gets transferred across PCI, there could be confusion about the correct byte ordering because PCI is Little Endian whereas Ethernet controller is a Big Endian device. So, in order to provide flexibility for different host processor/memory architectures, Ethernet controller can order this data on PCI in four different ways depending on the settings of the Word Swap Data, and Byte Swap Data bits. The following figures illustrate how data would appear on the PCI AD[63:0] pins depending on the settings of those swap bits:
Word Swap Data = 0, and Byte Swap Data = 0
B7
63–56
D1
S3
7B6
55–48
D2
S4
Table 56: 64-Bit PCI Bus (WSD = 0, BSD = 0)
B5
47–40
D3
S5
B4
39–32
D4
S6
B3
31–24
D5
T1
B2
23–16
D6
T2
B1
15–8
S1
IP1
B0
7–0
S2
IP2
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 198
BCM5718 Programmer’s Guide Endian Control (Byte and Word Swapping)
B3
31–24
D5
D1
T1
S3
Table 57: 32-Bit PCI Bus (WSD = 0, BSD = 0)
B2
23–16
D6
D2
T2
S4
B1
15–8
S1
D3
IP1
S5
Word Swap Data = 0, and Byte Swap Data = 1
B7
63–56
D4
S6
B6
55–48
D3
S5
Table 58: 64-Bit PCI Bus (WSD = 0, BSD = 1)
B5
47–40
D2
S4
B4
39–32
D1
S3
B3
31–24
S2
IP2
B2
23–16
S1
IP1
B3
31–24
S2
D4
IP2
S6
Table 59: 32-Bit PCI Bus (WSD = 0, BSD = 1)
B2
23–16
S1
D3
IP1
S5
B1
15–8
D6
D2
T2
S4
Word Swap Data = 1, and Byte Swap Data = 0
B7
63–56
D5
T1
B6
55–48
D6
T2
Table 60: 64-Bit PCI Bus (WSD = 1, BSD = 0)
B5
47–40
S1
IP1
B4
39–32
S2
IP2
B3
31–24
D1
S3
B2
23–16
D2
S4
B1
15–8
D6
T2
B1
15–8
D3
S5
B0
7–0
D5
D1
T1
S3
B0
7–0
S2
D4
IP2
S6
B0
7–0
D4
S6
B0
7–0
D5
T1
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 199
BCM5718 Programmer’s Guide Endian Control (Byte and Word Swapping)
B3
31–24
D1
D5
S3
T1
Table 61: 32-Bit PCI Bus (WSD = 1, BSD = 0)
B2
23–16
D2
D6
S4
T2
B1
15–8
D3
S1
S5
IP1
Word Swap Data = 1, and Byte Swap Data = 1
B7
63–56
S2
IP2
B6
55–48
S1
IP1
Table 62: 64-Bit PCI Bus (WSD = 1, BSD = 1)
B5
47–40
D6
T2
B4
39–32
D5
T1
B3
31–24
D4
S6
B2
23–16
D3
S5
B0
7–0
D4
S2
S6
IP2
B1
15–8
D2
S4
B0
7–0
D1
S3
B3
31–24
D4
S2
S6
IP2
Table 63: 32-Bit PCI Bus (WSD = 1, BSD = 1)
B2
23–16
D3
S1
S5
IP1
B1
15–8
D2
D6
S4
T2
B0
7–0
D1
D5
S3
T1
So, for a little-endian (e.g., x86) host, software should set both the Word Swap Data, and Byte Swap Data bits.
This is because a little endian host will expect the first byte on the wire (byte D1) to be placed into memory at the least significant (starting) address of the packet data.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 200
BCM5718 Programmer’s Guide Endian Control (Byte and Word Swapping)
Word Swap Non-Frame Data and Byte Swap Non-Frame Data Bits
The Word Swap Non-Frame Data, and Byte Swap Non-Frame Data bits affect the byte ordering of certain shared memory data structures (buffer descriptors, statistics block, etc.) when those structures are transferred across PCI.
The following table shows as example of how a Send Buffer Descriptor is stored internally in the Ethernet controller.
Byte # 0
Bit # 63
MSB
Table 64: Send Buffer Descriptor (Big-Endian 64-Bit format)
1 2
48 47
3 4
32 31
Host Address
5
16
6
15
7
0
LSB 0x00
LSB Flags Reserved VLAN 0x08
Since the Ethernet controller uses a 64-bit memory subsystem, the above diagram is shown in 64-bit format.
Furthermore, the table shows both the internal byte offset for each field and the bit position for each byte.
Note: This may seem confusing because big-endian notation normally has the bit positions incrementing from left to right. However, in this case, the bit positions are relevant because they correspond to the bit positions on PCI (AD[63:0]) if neither of the non-frame data swap bits are set.
For clarification, the following table shows the same structure in 32-bit format.
Byte #
Bit # 31
MSB
MSB
Table 65: Send Buffer Descriptor (Big-Endian 32-Bit format)
0
Length
Reserved
1
LSB
2
16 15
Host Address
Flags
VLAN
3
0
LSB
0x00
0x04
0x08
0x0C
To provide flexibility for different host processor/memory architectures, the Ethernet controller can order the data in memory in four different ways depending on the settings of the Word Swap Non-Frame Data and Byte
Swap Non-Frame Data bits. The following tables show how data will appear depending on the settings of those swap bits:
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 201
BCM5718 Programmer’s Guide Endian Control (Byte and Word Swapping)
Word Swap Non-Frame Data = 0 and Byte Swap Non-Frame Data = 0
This would require the software to use the following little-endian data structure on the host:
Byte #
Bit #
Table 66: Send Buffer Descriptor (Little-Endian 32-Bit format) with No Swapping
3 2 1 0
31 16 15
Host Address
0
LSB
MSB
MSB
Reserved
Length LSB
VLAN
Flags
0x00
0x04
0x08
0x0C
In this case, the data structure takes on a slightly new format because the words have been swapped.
Word Swap Non-Frame Data = 1 and Byte Swap Non-Frame Data = 0
This requires the software to use the following little-endian data structure on the host:
Byte #
Bit #
Table 67: Send Buffer Descriptor (Little-Endian 32-Bit format) with Word Swapping
3 2 1 0
31
MSB
16 15
Host Address
0
LSB
MSB Length
Reserved
LSB Flags
VLAN
0x00
0x04
0x08
0x0C
The disadvantage of this approach is if the host operating system supported a 64-bit data type for a physical address, the host device driver would have to swap the two 32-bit words that comprise the 64-bit address that the host operating system used.
Word Swap Non-Frame Data = 0 and Byte Swap Non-Frame Data = 1
This requires the software to use the following big-endian data structure on the host:
Byte #
Bit #
Table 68: Send Buffer Descriptor (Big-Endian 32-bit format) with Byte Swapping
0 1 2 3
31
MSB
16 15
Host Address
0
LSB
MSB
Reserved
Length LSB
VLAN
Flags
0x00
0x04
0x08
0x0C
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 202
BCM5718 Programmer’s Guide Endian Control (Byte and Word Swapping)
Word Swap Non-Frame Data = 1 and Byte Swap Non-Frame Data = 1
This requires the software to use the following big-endian data structure on the host:
Table 69: Send Buffer Descriptor (Big-Endian 32-bit format) with Word and Byte Swapping
Byte #
Bit # 31
MSB
0 1
16 15
Host Address
2 3
0
MSB Length
Reserved
LSB Flags
VLAN
LSB
0x00
0x04
0x08
0x0C
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 203
BCM5718 Programmer’s Guide Ethernet Link Configuration
Section 10: Ethernet Link Configuration
Overview
The Ethernet controller supports multiple link operating modes. It can operate at multiple link speeds:
10 Mbps, 100 Mbps, or 1000 Mbps. It can also operate at half-duplex (IEE 802.3 CSMA/CD) or full-duplex. The
MAC is compliant with IEEE 802.3, IEEE 802.3u, IEEE 802.3x, and IEEE 802.3z specifications.
GMII/MII
The Gigabit Media Independent Interface (GMII) is normally used to interface the controller to a transceiver that supports Gigabit Ethernet over copper wiring (1000BASE-T). The Media Independent Interface (MII) is used to interface the controller to a transceiver that is capable of 10/100 Mbps Ethernet. Depending upon the link speed, driver software will need to configure the Ethernet controller to operate in either GMII mode or MII mode.
Note: The integrated PHY transceiver of the BCM5718 has a has a fixed PHY address value of 1.
Configuring the Ethernet Controller for GMII and MII Modes
Configuring the Ethernet controller to operate in GMII or MII mode is simple. During initialization, software should configure the Ethernet_MAC_Mode.Port_Mode bits to a value that corresponds to the correct interface speed (01b for MII, 10b for GMII).
Configuring How MAC Detects Link Up/Down
The Ethernet controller has the ability to determine if the Ethernet link is up or down. The link will be down if the Ethernet cable is not properly attached at both ends of the network. Link will be up only if the cable is properly attached and the devices at both ends of the cable recognize that link has been established. The device cannot successfully transfer packets on the link unless it determines that it has a valid link up.
The controller involves using the Ethernet controller’s LNKRDY input signal. This method allows the Ethernet controller to determine the link status based on the link status output from integrated PHY connected to
LNKRDY input of MAC. The Transmit_MAC_Status.Link Up bit (see
“Transmit MAC Status Register (offset:
enable this method by clearing the MII_Mode.Port_Polling bit (bit-4 of offset 0x454).
The link state of the Ethernet controller can also be forced by disabling both the auto-polling function and the
LNKRDY signal and forcing the link status by directly writing to the MII_Status.Link_Status bit.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 204
BCM5718 Programmer’s Guide GMII/MII
Link Status Change Indications
It is advantageous for host software to know when the status of the Ethernet link has changed. To generate an interrupt to the host when link status changes, software should set the
) and the Mode_Control.Interrupt_on_MAC_Attention bit (see “Mode Control Register (offset:
). With this configuration, the Ethernet_MAC_Status.Link_State_Changed bit and
Link_State_Changed bit in the status block (see
“Status Block” on page 83 ) will be set when the link has
changed state.
Configuring the GMII/MII PHY
GMII/MII transceivers (PHYs) contain registers that a software driver can manipulate to change parameters in the PHY. These parameters include the link speed or duplex that the PHY is currently running at, or the speed/ duplex options that the PHY advertises during the auto-negotiation process. NIC device drivers will typically access PHY registers during the driver initialization process to configure the PHYs speed/duplex or to examine the results of the auto-negotiation process.
The integrated PHY registers are accessed via a process called MDIO. The integrated PHY is connected to the
Ethernet controller through an internal MDIO bus (MDIO and MDC pins). Software accesses PHY’s registers via
MDIO through the Ethernet controller’s MII_Communication register. The following example code describes accessing the PHY registers through the MII_Communication registers of the Ethernet controller.
Reading a PHY Register
// Setup the value that we are going to write to MI Communication register
// Set bit 27 to indicate a PHY read.
// Set bit 29 to indicate the start of a MDIO transaction
Value32 = ((PhyAddress << 21) | (PhyRegOffset << 16) | 0x28000000)
// Write value to MI communication register
MII_Communication_Register = Value32
// Now read back MI Communication register until the start bit
// has been cleared or we have timed out (>5000 reads)
Loopcount = 5000
While (LoopCount > 0)
Begin
Value32 = MII_Communication_Register
If (!(Value32 | 0x20000000)) then BREAK loop
Else Loopcount--
End
// Print message if error
If (Value32 | 0x20000000) then
Begin
// It a debug case–cannot read PHY
Procedure (Print Error Message)
Value32 = 0
End
// Now return the value that we read (lower 16 bits of reg)
Return (Value32 & 0xffff)
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 205
BCM5718 Programmer’s Guide
Writing a PHY Register
// Setup the value that we are going to write to MI Communication register
// Set bit 26 set to indicate a PHY write.
// Set bit 29 to indicate the start of a MDIO transaction
// The lower 16 bits equal the value we want to write to the PHY register
Value32 = ((PhyAddress << 21) | (PhyRegOffset << 16) | RegValue | 0x24000000)
// Write value to MI communication register
MII_Communication_Register = Value32
// Now read back MI Communication register until the start bit
// has been cleared or we have timed out (>5000 reads)
Loopcount = 5000
While (LoopCount > 0)
Begin
Value32 = MII_Communication_Register
If (!(Value32 | 0x20000000)) then BREAK loop
Else Loopcount--
End
// Print message if error
If (Value32 | 0x20000000) then
Begin
// It a debug case – can’t write PHY
Procedure (Print Error Message)
Value32 = 0
End
// If auto-polling is enabled, turn it back on
PHY Loopback Configuration
External PHY Loopback
10Base-T
- Write 0x0100 to PHY register 00h // Force 10Base-T, full duplex
- Write 0x1000 to PHY register 1Eh // Force link (required for 10Base-T)
- Set MAC register 0x400[3:2] = 01b // Set MII
- Set MAC register 0x400[1] = 0 // Force full-duplex operation
100Base-TX
- Write 0x2100 to PHY register 00h // Force 100Base-TX, full duplex
- Write 0x1000 to PHY register 1Eh // Force link (required for 100Base-TX)
- Set MAC register 0x400[3:2] = 01b // Set MII
- Set MAC register 0x400[1] = 0 // Force full-duplex operation
1000Base-T
- Write 0x1B00 to PHY register 09h // Enable 1000Base-T master mode
- Write 0x8400 to PHY register 18h // Enable external loopback mode
- Write 0x1000 to PHY register 1Eh // Force link (optional for 1000Base-T)
- Write 0x0140 to PHY register 00h // Force 1000Base-T operation
- Set MAC register 0x400[3:2] = 10b // Set GMII
- Set MAC register 0x400[1] = 0 // Force full-duplex operation
Internal PHY Loopback
---------------------
10Base-T
- Write 0x4100 to PHY register 00h // Force 10Base-T, full duplex, internal loopback
- Write 0x1000 to PHY register 1Eh // Force link (required for 100Base-TX)
GMII/MII
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 206
BCM5718 Programmer’s Guide GMII/MII
- Set MAC register 0x400[3:2] = 01b // Set MII
- Set MAC register 0x400[1] = 0 // Force full-duplex operation
100Base-TX
- Write 0x6100 to PHY register 00h // Force 100Base-TX, full duplex, internal loopback
- Write 0x1000 to PHY register 1Eh // Force link (required for 100Base-TX)
- Set MAC register 0x400[3:2] = 01b // Set MII
- Set MAC register 0x400[1] = 0 // Force full-duplex operation
1000Base-T
- Write 0x4140 to PHY register 00h // Force 1000Base-T, full duplex, internal loopback
- Write 0x1000 to PHY register 1Eh // Force link (optional for 1000Base-T)
- Set MAC register 0x400[3:2] = 10b // Set GMII
- Set MAC register 0x400[1] = 0 // Force full-duplex operation
PHY Configuration Auto-Negotiation (10/100/1000 Speed with Half and Full
Duplex Support)
Basic PHY pseudo-code:
• Enable 10/100/1000 PHY loopback mode
• Enable/disable Auto-MDI crossover
• Enable auto-negotiation
• Forced 10/100/1000 link speeds
-------------------
PHY Reset Procedure
------------------uint16_t val16;
// Initiate PHY reset by setting 0x00[15] = 1 phy_write(0x00, 0x8000);
// Wait up to 100ms for 0x00[15] = 0 for (int i = 0; i < 100; i++) {
val16 = phy_read(0x00);
if ((val16 & 0x8000) == 0)
break;
// Delay for 1ms
delay_us(1000);
}
----------------------
PHY Loopback Procedure
----------------------
// Force link down by enabling loopback phy_write(0x00, (1 << 14));
// Wait up to 15ms for link to drop for (int i = 0; i < 15000; i++) {
if ((phy_read(0x01) & (1 << 4) == 0)
break;
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 207
BCM5718 Programmer’s Guide
// Wait 10us
delay_us(10);
}
----------------------------------------------------
Configure Auto-MDIX (Cable pair swapping for 10/100)
----------------------------------------------------
// Read Misc Control Register (PHY 18h, Shadow 7) phy_write(0x18, 0x7007); val16 = phy_read(0x18)
//Set Force Auto MDIX Mode (Phy 18h, Shadow 7, bit 9) val16 |= (1 << 9);
// Write Misc control Register (set write enable bit) phy_write(0x18, val16 | (1 << 15));
// Enable Auto MDIX Crossover (Phy 10h, bit 14) val16 = phy_read(0x10); val16 &= ~(1 << 14); phy_write(0x10, val16);
---------------------------------------------------------------------
Autonegotiation (10/100/1000 speed with half and full duplex support)
--------------------------------------------------------------------uint16_t gig = 0, anar = 0;
// Reset PHY
// Enable auto-MDIX
// Force loopback
// Select pause and asymmetric pause advertisement for Ethernet anar = (1 << 11) | (1 << 10) | (00001b << 0);
// Select 10/100 half/full duplex advertisement anar |= (1 << 8) | (1 << 7) | (1 << 6) | (1 << 5);
// Enable 10/100 autoneg advertisement phy_write(0x04, anar);
// Select 1000Mb full-duplex operation gig = (1 << 9);
// Advertise full-duplex operation phy_write(0x09, gig);
// Enable and restart autonoegotiation phy_write(0x00, ((1 << 12) | (1 << 9)));
--------------------
10Base-T Half-Duplex
--------------------
BROADCOM
®
July 17, 2013 • 5718-PG107-R
GMII/MII
Page 208
BCM5718 Programmer’s Guide uint16_t bmcr = 0;
// Reset PHY
// Enable auto-MDIX
// Set link speed to 10Mb bmcr = (0 << 6) | (0 << 13);
// Set half-duplex operation bmcr |= (0 << 8);
// Force loopback
// Disable 1000Mb autoneg advertisement phy_write(0x09, 0);
// Disable 10/100 autoneg advertisement phy_write(0x04, (00001b << 0));
// Write forced link speed phy_write(0x00, bmcr);
--------------------
10Base-T Full-Duplex
-------------------uint16_t bmcr = 0;
// Reset PHY
// Enable auto-MDIX
// Set link speed to 10Mb bmcr = (0 << 6) | (1 << 13);
// Set full-duplex operation bmcr |= (1 << 8);
// Force loopback
// Disable 1000Mb autoneg advertisement phy_write(0x09, 0);
// Disable 10/100 autoneg advertisement phy_write(0x04, (00001b << 0));
// Write forced link speed phy_write(0x00, bmcr);
----------------------
100Base-TX Half-Duplex
---------------------uint16_t bmcr = 0;
BROADCOM
®
July 17, 2013 • 5718-PG107-R
GMII/MII
Page 209
BCM5718 Programmer’s Guide
// Reset PHY
// Enable auto-MDIX
// Set link speed to 10Mb bmcr = (0 << 6) | (1 << 13);
// Set half-duplex operation bmcr |= (0 << 8);
// Force loopback
// Disable 1000Mb autoneg advertisement phy_write(0x09, 0);
// Disable 10/100 autoneg advertisement phy_write(0x04, (00001b << 0));
// Write forced link speed phy_write(0x00, bmcr);
----------------------
100Base-TX Full-Duplex
---------------------uint16_t bmcr = 0;
// Reset PHY
// Enable auto-MDIX
// Set link speed to 10Mb bmcr = (0 << 6) | (1 << 13);
// Set full-duplex operation bmcr |= (1 << 8);
// Force loopback
// Disable 1000Mb autoneg advertisement phy_write(0x09, 0);
// Disable 10/100 autoneg advertisement phy_write(0x04, (00001b << 0));
// Write forced link speed phy_write(0x00, bmcr);
----------------------
1000Base-T Half-Duplex
----------------------
- Half duplex operation not supported at 1000Mb per 802.3 specification
----------------------
1000Base-T Full-Duplex
----------------------
- Forced speed not supported, must use autoneg but only advertise 1000Mb speed
BROADCOM
®
July 17, 2013 • 5718-PG107-R
GMII/MII
Page 210
BCM5718 Programmer’s Guide uint16_t bmcr = 0, gig = 0;
// Reset PHY
// Enable auto-MDIX
// Set link speed to 1000Mb bmcr = (1 << 6) | (0 << 13);
// Select full-duplex operation gig = (1 << 9);
// Force loopback
// Disable 10/100 autoneg advertisement phy_write(0x04, (00001b << 0));
// Advertise full-duplex operation phy_write(0x09, gig);
// Enable and restart autonoegotiation phy_write(0x00, (bmcr | (1 << 12) | (1 << 9)));
MDI Register Access
MDI Register Access
Configuring physical devices and querying the status of physical devices are done via the MDIO interface (MDC and MDIO).
Note: This procedure is PHY-independent. The MAC access to the PHY is the same for the entire
NetXtreme family.
There are two modes in which the internal MII Management interface signals (MDC/MDIO) can be controlled for communication with the internal transceiver registers. These modes are as follows:
• Autopolling mode. Enabled by setting the Enable bit in the MAC Ethernet MI Mode register. The device will poll for the link status bit in the transceiver.
• Command Control. Writing to the MI Communications register directly to either read or write the transceiver registers.
Autopolling mode has the lower priority and it will be stalled any time there is an active operation through the
MI Communications register.
Operational Characteristics
The interface between the MAC and physical devices is with the two signals of:
• MDIO clock (MDC)
• Bidirectional serial data (MDIO)
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 211
BCM5718 Programmer’s Guide MDI Register Access
The details of the MDIO interface can be found in the IEEE 802.3 specification.
Access Method
The MAC provides the auto-access method to access the Physical Device register via the MDIO interface.
Auto-Access Method
The Ethernet controller has a built-in interface to access physical device registers without having to control
MDC and MDIO pins by software/firmware. It provides an easy way to access the physical device register.
To use this mode, MDI_Control_Register.MDI_Select has to be cleared to 0. The MII_Communication_Register is used to access physical device.
Note: Programmers must be careful to wait for the start _busy bit to clear. Writing to the MI
Communication register prior to the completion of a previous MDI access will yield unpredictable
MDI data. The previous access will not complete successfully.
For example, to read a 16-bit PHY register at offset 0x2 of a PHY device which is strapped to PHY address 1, perform the following steps:
1. MII_Communication_Register.Register_Address is set to 0x2.
2. MII_Communication_Register.PHY_Addr is set to 1.
3. MII_Communication_Register.Command is set to 0x2.
4. MII_Communication_Register.Start_Busy is set to 1.
5. Poll Until MII_Communication_Register.Start_Busy is cleared to 0.
6. MII_Communication_Register.Transaction_Data contains 16-bit data of the PHY register.
See “Configuring the GMII/MII PHY” on page 205
for example code.
To write a value of 0x1000 into 16-bit PHY register at offset 0x0 of a PHY device which is strapped to PHY address 1, perform the following steps:
1. MII_Communication_Register.Register_Address is set to 0x0.
2. MII_Communication_Register.PHY_Addr is set to 1.
3. MII_Communication_Register.Command is set to 0x1.
4. MII_Communication_Register.Transaction_Data is set to 0x1000
5. MII_Communication_Register.Start_Busy is set to 1.
6. Poll Until MII_Communication_Register.Start_Busy is cleared to 0.
See “Configuring the GMII/MII PHY” on page 205
for example code.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 212
BCM5718 Programmer’s Guide Wake on LAN Mode/Low-Power
Wake on LAN Mode/Low-Power
Description
The Ethernet controller uses the ACPI D3 hot/cold (low-power) state to conserve energy. The OS power management policy notifies device drivers to initiate power management transitions. The device driver should move the MAC into the D3 hot/cold power state—a response to the power management request. While the
Ethernet controller is in a D3 state, the RX MAC will filter incoming packets. The RX MAC compares incoming traffic for Interesting Packet pattern matches. The Ethernet controller asserts the PCI PME signal, when a positive WOL packet comparison is made. The PME signal notifies the Operating System and host device driver to transition the MAC into the D0 (high power) state.
WOL mode is a combination of PHY and MAC configurations. Both the PHY and MAC must be configured correctly to enable Broadcom’s WOL technology. The Ethernet controller provides WOL pattern filters for 10/
100 wire speeds.
The Ethernet controller supports both Interesting Packet pattern matching the AMD Magic Packet proprietary technology for WOL. The WOL support for the AMD Magic Packet format does not require host software to configure a pattern filter. The Magic Packet comparison is made in hardware and is enabled through a register interface. The AMD Magic Packet can be either broadcast or directed, and must contain the receiver's MAC address at least six times (repeating) in the packet. The Magic Packet wake-up is configured different from pattern match wake-up.
The following components are involved in WOL operation:
• Internal memory
• WOL Pattern Pointer register
• WOL Pattern Configuration register
• WOL streams
• Pattern data structure
• GPIO
• Firmware mailbox
• PHY auto-negotiation
• Ethernet controller power management
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 213
BCM5718 Programmer’s Guide Wake on LAN Mode/Low-Power
Functional Overview
The Ethernet controller is capable of WOL in 10/100 Mbps for copper-based controllers.
Note: When configured for WOL in 1000 Mbps mode, the Ethernet controller draws more than the
375 mA allowed by the PCI specification.
transmit engine is disabled and its FIFO is free for use. The TDE fetches data from the memory arbiter starting at a location specified in the WOL_Pattern_Pointer register. The WOL pattern checker pulls data off the TX FIFO for packet comparisons. The RX MAC will move incoming frame(s) to the pattern checker, and the remaining
RX data path is not utilized. A state machine controls the Magic Packet comparisons. The WOL state machine will move out of an Idle state, when ACPI power management is enabled. The WOL state machine will clear the
TX FIFO and Match register. The Match register indicates a positive Magic Packet comparison(s) on a stream.
In 10/100 Mbit mode, data is received once every four clock cycles. The pattern checker compares the first three patterns in the first cycle, the second three patterns in the second cycle, and the third three patterns in the last cycle. It is idle during the fourth cycle. In gigabit mode, the pattern checker gets three pattern words from the FIFO at one time.
Figure 50: WOL Functional Block Diagram
Tx FIFO TDE
Memory
Arbiter
Pattern
Data
Internal
Memory
Pattern
Checker
Power
Managment
RX
IO
RX
PCS
RX
RMII
RX
GMII
Rx MAC
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 214
BCM5718 Programmer’s Guide Wake on LAN Mode/Low-Power
Operational Characteristics
Internal Memory
The WOL pattern must be stored in the Ethernet controller miscellaneous memory region. All memory locations require the host software to reinitialize the WOL pattern before each D0 to D3 transition. The RX/TX
MAC places packets into this internal memory and the WOL pattern is overwritten during normal operation.
When the Ethernet controller operates in D0 state, internal data structures use the same memory location as the WOL pattern. Host software should reinitialize the WOL pattern before each WOL sleep transition.
Table 70 shows the required memory regions for the WOL pattern.
Internal Address Range
0x8000–0x8FFF
Table 70: Required Memory Regions for WOL Pattern
Size
8 KB
Name
Miscellaneous Memory Region
For 5718 Family chip, the address range starts from 0x20000 (ASF Disable) / 0x24000 (ASF Enable).
WOL Pattern Pointer Register
The WOL_Pattern_Pointer specifies a location within Ethernet controller address space where the pattern buffers reside (see
“WOL Pattern Pointer Register (offset: 0x430)” on page 317 for the register definition). The
internal memory subsection discusses how host programmers can choose an address range. The
WOL_Pattern_Pointer register uses a pointer value, not an internal memory location. The pointer value is calculated by dividing an internal memory location by the value 8. Do not program the WOL_Pattern_Pointer register with the actual internal memory location. Rather, host software must first convert the base address to a pointer value. Here are example conversion from memory base to pointer values:
• 0x0000 (Misc Memory)/8 = 0x00 (required value)
• 0x400 (base addr)/8 = 0x80 (pointer value)
• 0x8000 (base addr)/8 = 0x1000 (pointer value)
• 0xF000 (base addr)/8 = 0x1E00 (pointer value)
WOL Pattern Configuration Register
The WOL_Pattern_Configuration register contains two programmable data fields. Both fields use different units of measurement, so the host programmer should be careful (see
“WOL Pattern Configuration Register
for the register definition). This register is used to position and extract data from
RX Ethernet frames.
• Offset Field—The Offset field in the WOL_Pattern_Configuration register specifies a position in RX
Ethernet frame(s), where comparisons for WOL patterns should begin. This register uses a unit of measurement specified in terms of 2-byte chunks. Software should not program this field with a byte value, but should first normalize to a 2-byte unit. Hardware cannot begin WOL comparisons on odd byte alignments (i.e., 3,5,7,9 offsets). Host software must begin all pattern matching on even byte boundaries
(i.e., 2,4,6,8 offsets). The 2 bytes per unit forces even byte alignment. For example:
– 0x14 (byte offset)/2 = 0x0A (register ready)
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 215
BCM5718 Programmer’s Guide Wake on LAN Mode/Low-Power
– 0x28 (byte offset)/2 = 0x14 (register ready)
– 0xFC (byte offset)/2 = 0x7E (register ready)
• Length Field—The Length field in the WOL_Pattern_Configuration register specifies the number of clock cycles required to compare a variable number of bytes, in the RX stream. The Length field uses a unit of measurement specified in terms of memory arbiter clock cycles. Software should not program this field with a byte value. The Length field should be programmed with the maximum number of clocks required to compare the largest pattern size for the nine streams (10/100 mode only).
Note: The Ethernet controller only supports one pattern stream at gigabit wire speed, so the length field will always be the largest pattern size.
The programmer must use the following equation to calculate the number of clock cycles required to match patterns at 10/100 wire-speed: (Length/2) * 3 MA clocks. The equation breaks down as follows:
– Determine the number of bytes in the RX Ethernet frame to compare. This value is a byte length.
– The WOL pattern checker can compare two bytes simultaneously. Divide length by two bytes and round up to nearest integer value.
– The Ethernet controller compares 2 bytes every three memory arbiter (MA) clock cycles. Multiply
(Length/2) by three clock cycles.
• The following are example clock cycle calculations:
– Data stream length = 25 bytes
– 25 bytes/2 = 12.5 byte-pairs
– Round(12.5) = 13 byte-pairs
– 13 byte-pairs * 3 clocks/byte-pairs = 39 clocks (register ready)
– Data stream length = 83 bytes
– 83 bytes/2 = 41.5 byte-pairs
– Round(41.5) = 42 byte-pairs
– 42 byte-pairs * 3 clocks/byte-pair = 126 clocks (register ready)
WOL Streams
A stream is a comparison operation on RX frame(s). When the MAC is running at 10/100 Mbps wire speed, nine different patterns can be compared against the RX frame(s). The Ethernet controller moves RX frame(s) into nine parallel comparators, and the frame is matched simultaneously. The MAC is capable of filtering nine different patterns in 10/100 modes. The WOL pattern checker breaks frames into 2-byte pairs, so all nine comparators can begin matching data. In
, three Ethernet frames are compared against the nine available patterns.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 216
BCM5718 Programmer’s Guide Wake on LAN Mode/Low-Power
Figure 51: Comparing Ethernet Frames Against Available Patterns (10/100 Ethernet WOL)
10/100 Ethernet WOL
Frame 0 Skip C0, C1,C2,C3
Frame 1 Skip B0, B1, B2, B3, B4, B5
A0,A1,A2,A3,A4,A5 Frame 2 Skip
ACPI Offset
Field
ACPI Mbuf Pointer Register
Base Addr/8
ACPI Length Offset Register
Length Offset
Control Stream
1
Stream
2
Stream
3
A0, A1 B0, B1 C0, C1
A2,A3 B2, B3 C2, C3
A4,A5 B4, B5 00,00
Patterns
C0 S1 S2 S3 C0 S4 S5 S6 C0 S7 S8 S9
ACPI length field is the max pattern size
Nine pattern streams for simultaneousWOL compare
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 217
BCM5718 Programmer’s Guide Wake on LAN Mode/Low-Power
Pattern Data Structure
The maximum number of entries in either 10/100 or 1000 mode is 128. The Ethernet controller cannot process a pattern that requires more than 128 entries. The size of an entry will vary based on 10/100 or 1000 Mbps mode. Additionally, all unused rows must be initialized with zeros. The WOL hardware cannot process an entry
unless unused rows and rules have been zeroed out (see Figure 52
).
Figure 52: Unused Rows and Rules Must Be Initialized with Zeros
Control Stream 1 Stream 2 Stream 3
Unused rules are initialized with 0
USED
00,00 00,00 00,00 00,00
00,00 00,00 00,00 00,00
Frame patterns are stored as data structures in memory. A control word is always present in a 64 bit entry/row.
The control word describes proceeding data fields in the entry.
In 10/100 Mbps mode, one WOL entry requires three 64-bit wide rows (see
). The total length of an
The remaining 48-bits contains 2-byte rules. The 2-byte rules are distributed across three streams: S, S+1, and
S+2. The next row’s 2-byte rules will correspond to three more streams: S+3, S+4, and S+5. Both
the packet stream are compared.
63
CTRL012
CTRL345
CTRL678
Table 71: 10/100 Mbps Mode Frame Patterns Memory
48 47 32 31 16 15
S0D0
S3D0
S6D0
S1D0
S4D0
S7D0
S2D0
S5D0
S8D0
0
Bits Field
Table 72: Frame Control Field for 10/100 Mbps Mode
Description
63:62 Reserved
61
60
S0 High Byte Enable
S0 Low Byte Enable
Enable S0 higher byte for comparison
Enable S0 lower byte for comparison
Access
RW
RW
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 218
BCM5718 Programmer’s Guide Wake on LAN Mode/Low-Power
Bits
59
58
57
56
55:51
50
49
48
Table 72: Frame Control Field for 10/100 Mbps Mode (Cont.)
Field
S1 High Byte Enable
S1 Low Byte Enable
S2 High Byte Enable
S2 Low Byte Enable
Reserved
S0 Done
S1 Done
S2 Done
Description
Enable S1 higher byte for comparison
Enable S1 lower byte for comparison
Enable S2 higher byte for comparison
Enable S2 lower byte for comparison
–
End of S0 Stream
End of S1 Stream
End of S2 Stream
Access
–
RW
RW
RW
RW
RW
RW
RW
shows an example of how 10/100 Mbps frame data is split up in the pattern data structure. Eight streams are compared simultaneously with three 64-bit rows comprising one WOL entry. Rows 0–2 compare frame data0 against eight rules. Rows 3–5 compare frame data1 against the next eight rules. Rows 6–9 compare data2 against the final eight rules. The eight rules may be uniquely defined for all three WOL entries.
Data[63:48]
Control Bits
Control Bits
Control Bits
Control Bits
Control Bits
Control Bits
Control Bits
Control Bits
Control Bits
Table 73: Example of Splitting 10/100 Mbps Frame Data in Pattern Data Structure
Data[47:32]
Stream 0 data 0
Stream 3 data 0
Stream 6 data 0
Stream 0 data 1
Stream 3 data 1
Stream 6 data 1
Stream 0 data 2
Stream 3 data 2
Stream 6 data 2
Data[31:16]
Stream 1 data 0
Stream 4 data 0
Stream 7 data 0
Stream 1 data 1
Stream 4 data 1
Stream 7 data 1
Stream 1 data 2
Stream 4 data 2
Stream 7 data 2
Data[15:0]
Stream 2 data 0
Stream 5 data 0
Stream 8 data 0
Stream 2 data 1
Stream 5 data 1
Stream 8 data 1
Stream 2 data 2
Stream 5 data 2
Stream 8 data 2
Firmware Mailbox
When the Ethernet controller initializes (the firmware boot code is loaded from NVRAM when the chip powers on or when reset completes), the boot code checks the T3_FIRMWARE_MAILBOX in shared memory. When the
T3_MAGIC_NUM signature (0x4B657654) is present, the boot code does not issue a hard reset to the PHY. This is especially important in WOL mode since the PHY should not be reset.
Before the host software issues a reset to the Ethernet controller, it must write the T3_MAGIC_NUM to the shared memory address T3_FIRMWARE_MAILBOX (0xb50). This address is a software mailbox, which boot code polls before it resets the PHY. The boot code will acknowledge the signature by writing the one’s complement of the T3_MAGIC_NUM back into the T3_FIRMWARE_MAILBOX. If the T3_MAGIC_NUM is present, the boot code will not reset the PHY. After resetting the Ethernet controller, host software should poll for the one’s complement of the T3_MAGIC_NUM before it proceeds, otherwise, boot code initialization may interfere with the host software initialization.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 219
BCM5718 Programmer’s Guide Wake on LAN Mode/Low-Power
If the host software will be controlling the WOL configuration, it should write the DRV_WOL_SIGNATURE
(0x474c0000) to the shared memory address DRV_WOL_MAILBOX (0xd30) so that the boot code will not take over the WOL initialization. If the DRV_WOL_SIGNATURE is not present, and WOL has been enabled, the boot code will assume that the host software is a legacy driver and skip the WOL initialization. If WOL is disabled, the boot code will take over the WOL initialization based on the NVRAM configuration.
Name
T3_FIRMWARE_MAILBOX
DRV_WOL_MAILBOX
Table 74: Firmware Mailbox Initialization
Address
0x0B50
0xd30
Recommended Value
0x4B657654
0x474c0000
PHY Auto-Negotiation
is required if the NIC must be placed into a D3 cold state. Half- or full-duplex operation is acceptable. Software must modify auto-advertise configurations in the PHY’s MDI registers. The link partner will read advertisement settings to find a highest common capability. Since WOL requires 10 Mbps wire speed, the two PHYs will effectively auto-negotiate for half- or full-duplex connection.
Table 75: Recommended Settings for PHY Auto-Negotiation
Register
Auto_Negotation_Advertisement
Auto_Negotation_Advertisement
Auto_Negotation_Advertisement
Auto_Negotation_Advertisement
1000BASE-T_Control
1000BASE-T_Control
Bit
10_BASE_TX_Half_Duplex
10_BASE_TX_Full_Duplex
100_BASE_TX_Half_Duplex
100_BASE_TX_Full_Duplex
1000_BASE_TX_Half_Duplex
1000_BASE_TX_Full_Duplex
Recommended Value
Enable
Enable
Disable
Disable
Disable
Disable
Power Management
operation, so its clock can be disabled. The MAC has an internal phase-locked loop that clocks internal logic at
133 MHz. Software must select an alternate clocking source and then disable this PLL.
Register
PCI Clock_Control
PCI Clock_Control
PCI Clock_Control
Table 76: WOL Mode Clock Inputs
Bit
RX RISC_Clock_Disable
Select_Alternate_Clock
PLL133
Recommended Value
Set the bit to 1
Set the bit to 1
Set the bit to 1
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 220
BCM5718 Programmer’s Guide Wake on LAN Mode/Low-Power
The settings shown in Table 77
enable Magic Packet detection logic in the MAC. These setting also enable the
MAC to assert PME on the PCI bus. The RX MAC should maintain the multicast and broadcast settings that were previously configured by the NOS. The Microsoft® power management specification states:
“Only a frame that passes the device’s MAC, broadcast, or multicast address filter and matches on the previously loaded sample patterns will cause the wake-up signal to be asserted.”
The ACPI_Power-on bit needs to be set for pattern match, but not for Magic Packet recognition. The Magic
Packet detection mechanism is separate from the pattern match mechanism. Host software may configure
WOL using four filter permutations:
• Pattern match WOL disabled. Magic Packet disabled.
• Pattern match WOL enabled. Magic Packet disabled.
• Pattern match WOL disabled. Magic Packet enabled.
• Pattern match WOL enabled. Magic Packet enabled.
Table 77: Magic Packet Detection Logic Enable
Register
PCI Power_Management_Control/Status
PCI Power_Management_Control/Status
Ethernet_MAC_Mode
Ethernet_MAC_Mode
Bit(s)
PME_Enable
Power_State
ACPI_Power-On
Magic_Packet_Detection
Recommended Value
Enable
0x03
See above
See above
Integrated MACs
Table 78 lists the WOL mode control registers in the Ethernet controllers.
Register
WOL_Pattern_
Pointer
WOL_Pattern_
Configuration
Table 78: Integrated MAC WOL Mode Control Registers
Bit(s) Name
All
Length
Offset
Description Cross Reference
This register points to an internal memory location. Programmers should calculate pointer value by dividing a base address by 8.
Register (offset: 0x430)” on page 317
.
The number of memory arbiter clock cycles needed to read X bytes in the RX stream/frame.
Register (offset: 0x434)” on page 317
.
The number of bytes into the RX stream/frame to begin the pattern comparison.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 221
BCM5718 Programmer’s Guide Wake on LAN Mode/Low-Power
Table 78: Integrated MAC WOL Mode Control Registers (Cont.)
Register
Ethernet_MAC_
Mode
Bit(s) Name Description Cross Reference
Port_Mode This bit field specifies the type of interface the Ethernet controller port is currently using: MII, GMII, or none.
(offset: 0x400)” on page 310 .
Magic_Packet_
Detection
Enable WOL pattern filtering.
Promiscuous_mode All frames are forwarded, without any filtering, when this bit is enabled.
PCI Clock_Control RX RISC_Clock_Disable Disable the clock to the receive
CPU.
Alternate_Clock_Sour ce
PLL133
Use an alternate clock as a reference, rather than the PLL 133.
Disable the 133 MHz phase-locked loop.
Clock Control Register as per
PCI specifications.
Misc Local Control Misc_Pin_0_Output
Misc_Pin_0_Output_
Enable
GPIO pin 0.
When asserted, MAC drives pin output.
Misc_Pin_1_Output GPIO pin 1.
Misc_Pin_1_Output_
Enable
When asserted, MAC drives pin output.
Misc_Pin_2_Output GPIO pin 2.
Misc_Pin_2_Output_
Enable
When asserted, MAC drives pin output.
Power Management
Control/Status
PME_Enable
Power_State
Enable the Ethernet controller to assert PME on PCI bus.
Set the ACPI power state: D0, D3.
WOL Data Flow Diagram
The Ethernet controller and PHY are both configured for WOL mode. The process is as follows:
1. Clear the PME_Status bit in the Power Management Control/Status Register (offset: 0x4C) (see
Management Control/Status Register (offset: 0x4C)” on page 280
). This bit must be cleared, so the PME interrupt is not immediately generated once the NIC is moved to the D3 state. The bit could be asserted from a previous D3–D0 transition.
2. Set the Mask_PCI_Interrupt_Output bit in the Miscellaneous_Host_Control register (see
not generate interrupts during the WOL configuration of the PHY. The device driver’s ISR may attempt to reset and reconfigure the PHY as part of an error recovery code path.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 222
BCM5718 Programmer’s Guide Wake on LAN Mode/Low-Power
3. If host software must place the NIC into D3 cold state, the following step is necessary. Set the
10_Base_TX_Half_Duplex and 10_BASE_TX_Full_Duplex Capability bits, in the Auto-Negotiation
Advertisement Register. Clear the 100_BASE_TX_Full_Half_Duplex and 100_BASE_TX_Full_Duplex
Capability bits, in the Auto-Negotiation Advertisement Register. Clear the 1000_BASE_TX_Half_Duplex and
1000_BASE_TX_Full_Duplex Capability bits, in the 1000BASE-T Control Register. The link partner will now only be able to auto-negotiate for 10 Mbps speed full/half-duplex.
4. Set the Restart_Auto_Negotiation bit in the MII Control Register. The integrated PHY and link partner will now reconfigure for 10 Mbps wire speed. Essentially, 10 Mbps link must be auto-negotiated or forced.
5. Disable the FHDE, RDE, TDE bits of the “EMAC Mode Register (offset: 0x400)” on page 310
”, and on-chip
RISCs.
6. Host software must write the signature 0x4B657654 to internal memory address 0x0B50. Check for one’s complement of 0x4B657654.
7. Enable the Wake_On_LAN bit in the AUXILIARY Control Register.
8. For Interesting Packet WOL Only: Set up the Interesting Packet pattern in Ethernet controller local memory.
9. For Interesting Packet WOL Only: Write a pointer value to the
“WOL Pattern Pointer Register (offset:
. This register uses a normalized pointer value, not a device base address. The value written to this register is BCM5700_BASE_ADDR/8. The base address must be a specific location in local memory: 0x8000, 0xC000, or 0xD000. The choice of memory location depends upon other MAC configurations, and the selection is not arbitrary.
10. For Interesting Packet WOL Only: Write the Offset field in the
“WOL Pattern Configuration Register (offset:
. The WOL pattern checker will position into received frames on two-byte intervals.
The pattern checker compares two bytes in parallel, so host software should program the offset field accordingly. Host software may perceive this unit as OFFSET_BYTE/2 units.
. The length value is specified in terms of Memory Arbiter clock cycles, not bytes/ words/dwords. A comprehensive discussion of how the clock cycles are calculated will be presented.
12. Set the Port_Mode field in the
“EMAC Mode Register (offset: 0x400)” on page 310
to GMII mode. These bits enable the GMII between the MAC and internal PHY.
. This bit will enable logic for D3 hot/cold transitions to D0 ACPI state. The MAC will also be capable of asserting PME on the PCI bus.
(offset: 0x400)” on page 310 . The WOL logic will compare RX frames for Magic Packet patterns.
disabled.
16. Set the Enable_Alternate_Clock bit in the PCI Clock_Control register. The Ethernet controller’s 133 MHz
Phase Locked Loop (PLL) no longer clocks internal logic and an alternate clock reference is used. Set the PLL
LowPowerClock bit while keeping the Enable_Alternate_Clock bit set. Wait at least 27 µs and then clear the
Enable_Alternate_Clock bit. The Ethernet controller’s PLL is then switched to its lower power consumption mode.
17. In NIC applications, switch from VMAIN to VAUX in order to prevent a GRC reset. Set the required GPIOs of
Ethernet controller if any of them are used for switching the power from VMAIN to VAUX.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 223
BCM5718 Programmer’s Guide Flow Control
” and put it in promiscuous mode by setting the Promiscuous Mode bit of
(offset: 0x468)” on page 323 .
19. Enable the PME bit in the PCI Power Management Control/Status Register (offset: 0x4C) (see
wake up the system. Set the Power_State bits to D3 in the PCI Power Management Control/Status Register
(offset: 0x4C).
Flow Control
Description
The Ethernet controller supports IEEE 802.3x flow control. Flow control is a switched Ethernet capability, where link partners may pause traffic. The 802.3x flow control specifies that a MAC sublayer may transmit pause frames. The pause frames instruct the MAC’s link partner to wait a specified amount of time, before sending additional frames. This delay provides the MAC time to free packet buffers. Conversely, the MAC sublayer must also accept/receive pause frames. Flow control is used by switches and bridges to prevent clients of dissimilar speeds from exhausting switching packet buffers. Clients and servers may use flow control for similar reasons.
A very important requirement is that both link partners must share a full-duplex connection for flow control to be enabled. IEEE 802.3x flow control does not operate on a half-duplex connection. More information on flow control can be found in
Appendix A: “Flow Control,” on page 582 .
The following architectural blocks are integral to flow control:
• Transmit MAC
• Receive MAC
• Statistics Block
• PHY Auto-negotiation
• PHY Auto-Advertise
Operational Characteristics
The Ethernet controller implements pause functionality using Xon and Xoff states. The MAC will extract a pause quantum from a pause control frame. Then, the MAC will configure its internal timer with the pause_time specified by the link partner. Frames that are currently in the transmit engine will be completed before the transmit engine is inhibited. The MAC has moved flow control into a Xoff state once the transmit engine is inhibited. Note that the transmit engine is not completely disabled since the IEEE 802.3 specification stipulates that MAC control frames should not be paused.
One of the following conditions moves the Ethernet controller into an Xon state:
• Link partner sends a pause frame with pause_time = 0.
• Internal pause timer expires.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 224
BCM5718 Programmer’s Guide Flow Control
Transmit MAC
The transmit MAC is responsible for sending flow control frames. Software enables the transmit MAC to send
transmit MAC will not generate flow control frames. The MAC_RX_MBUF_Low_Water_Mark register value triggers PAUSE frames to be transmitted when a threshold value is passed. Software may alter the watermark to tune system performance.
Table 79: Transmit MAC Watermark Recommendation
Register
MAC_RX_MBUF_Low_Water_Mark
Recommended Value
24
As soon as PAUSE frame is transmitted, any incoming packet can be dropped, and the ifInDiscard counter in statistics will increase. When packet size is small (64 bytes) with 1000 Mbps, more frames can be dropped. Even if the PAUSE frame is transmitted, Pause frames cannot inhibit MAC control frames.
Low Water Mark Maximum Receive Frames register (see “Low Watermark Maximum Receive Frame Register
) control the number of good frames to receive after the RX MBUF Low Water
Mark has been reached. After the RX MAC receives this number of frames, it will drop subsequent incoming frames until the MBUF High Water Mark is reached.
The IEEE 802.3 pause control frame contains a pause_time field. The Ethernet controller inserts a time quanta into the pause_time field. Software should set the Enable_Long_Pause bit in the Transmit_MAC_Mode register to configure long pause quanta. Clearing the Enable_Long_Pause bit will default the pause_time back to the shorter quanta.
Table 80 shows the pause quanta based on the Enable_Long_Pause bit setting.
Enable_Long_Pause Bit
DISABLED (0)
ENABLED (1)
Table 80: Pause Quanta
Pause_Time
0x1FFF
0xFFFF
Receive MAC
The Ethernet controller receive MAC’s link partner may want to inhibit frame transmission until upstream resources become available. The receive MAC must be configured to accept IEEE 802.3x pause frames (see
). Software should set the Enable_Flow_Control bit in the Receive_MAC_Mode_Control register to enable automatic processing of flow control frames. If software clears the Enable_Flow_Control bit, IEEE
802.3x pause frames will be discarded. The Keep_Pause bit in the Receive_MAC_Mode_Control register will instruct the RX engine to forward pause frames to host memory. Software may be interested in setting this bit for debugging or promiscuous/sniffer configurations. Passing pause frames to the host will increase DMA and protocol processing and consume available host buffers. The receive MAC will filter pause control frames when the Keep_Pause bit is disabled.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 225
BCM5718 Programmer’s Guide Flow Control
Table 81: Keep_Pause Recommended Value
Register.Bit
Receive_MAC_Mode_Control.Keep_Pause
Recommended Value
DISABLED
Statistics Block
discussed in this section. Xon/Xoff statistical counters are related to internal Ethernet controller flow control states. Xon is associated to transmit enabled state and Xoff is associated to transmit disabled state. These Xon/
Xoff states are not part of the IEEE 802.3 specification; the Ethernet controller uses Xon/Xoff to manage flow control state and transitions. The Xon/Xoff statistics provide programmers with a high level of granularity for the measurement of Ethernet controller flow control performance in a LAN (see
Appendix A: “Flow Control,” on page 582 ).
Table 82: Statistic Block
Statistic Description xoffStateEntered This counter is bumped under the following conditions:
• IEEE 802.3 MAC flow control pause frame received with valid CRC.
• (Pause_time > 0) The link partner requests transmission inhibit.
The counter increments independently of the enabled/disabled state of
Receive_MAC_Mode_Control.Flow_Enabled.
xonPauseFramesReceived This counter is incremented under the following conditions:
• IEEE 802.3 MAC flow control pause frame received with valid CRC.
• (Pause_time == 0) The link partner no longer requires the device family to pause/wait/delay outgoing packets.
The counter increments independently of the enabled/disabled state of
Receive_MAC_Mode_Control.Flow_Enabled.
xoffPauseFramesReceived This counter is incremented under the following conditions:
• IEEE 802.3 MAC flow control pause frame received with valid CRC.
• (Pause_time > 0) The link partner requires the BCM5718 family to pause/ wait/delay outgoing packets.
The counter increments independently of the enabled/disabled state of
Receive_MAC_Mode_Control.Flow_Enabled.
outXon This counter is incremented under the following conditions:
• Transmit_MAC_Mode_Control.Flow_Enabled bit is set.
• (MAC_RX_MBUF_Low_Water_Mark > Threshold Value MAC resources are available.
• (pause_time == 0) 802.3 MAC flow control frame is sent.
outXoff This counter is incremented under the following conditions:
• Transmit_MAC_Mode_Control.Flow_Enabled bit is set.
• (MAC_RX_MBUF_Low_Water_Mark < Threshold Value) MAC resources are running low and a pause is desired.
• (pause_time > 0) IEEE 802.3 MAC flow control frame is sent.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 226
BCM5718 Programmer’s Guide Flow Control
PHY Auto-Negotiation
The PHY encodes flow control capability into Fast Link Pulse (FLPs) bursts. Link partners will extract encoded flow control capability from FLPs and then create a Link Code Word (LCW). The LCW is a message, which contains a selector and technology ability field. The technology ability field contains a bit called
Pause_Operation_for Full_Duplex_Link (A5). Refer to Annex 28-B of the IEEE 802.3 specifications. The A5 bit signifies that a link partner has implemented pause functionality. If both link partners support autonegotiation, they will further exchange data regarding flow control, using the next page bit in the LCW.
Auto-advertise is integrally tied to auto-negotiation. If link partner does not support pause functionality, the
PHY Auto_Negotation_Link_Partner_Ability_Register does not set the Pause_Capable bit. The Ethernet controller should not send pause frames to this link partner since flow control is not implemented or disabled.
The Ethernet controller can still accept pause frames, but sending a pause frame does not yield a preferred result.
Integrated MACs
lists the flow control registers in the Ethernet controllers.
Table 83: Integrated MAC Flow Control Registers
Register Bit(s) Name Description Cross Reference
Receive MAC Mode Enable_Flow_Control Enable automatic processing of IEEE
802.3 flow control frames.
See
Register (offset: 0x468)” on page 323
.
Transmit MAC
Mode
Enable_Flow_Control Enable automatic processing of IEEE
802.3 flow control frames.
See
Register (offset: 0x45C)” on page 320
.
MAC_RX_MBUF_
Low_Water_Mark
All 32 bits The number of internal buffers that must be available before the RX engine can accept a frame from the wire.
Threshold value for initiating flow control.
See
Register (offset: 0x504)” on page 329
.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 227
BCM5718 Programmer’s Guide Flow Control
{
Flow Control Initialization Pseudocode
//Check the Link State
If (MII_Status_Reg.Link_Status == TRUE) Then
//Check PHY status register for full-duplex configuration
If (MII_Aux_Status_Reg.Auto_Neg_HCD ==
(1000_FULL_DUPLEX Or 100_FULL_DUPLEX Or 10_FULL_DUPLEX) ) Then
{
//Check if USER has forced either auto-negotiation or auto-advertise
If ( (Driver_Auto_Neg_Variable == ENABLED) And
(Driver_Auto_Advertise_Variable != FORCED_SPEED_DUPLEX )) Then
{
// Probe Phy control registers for advertised flow control info
// Expected abilities should match the configured abilities. Expected abilities
// are based on the IEEE 803.3ab flow control subsection.
If ( (Auto_Neg_Advertise_Reg.Asymetric_Pause != 802.3ab_Table_28B-3 ) And
(Auto_Neg_Advertise_Reg.Pause_Capable != 802.3ab_Table_28B-3 ) ) Then
{
//The current advertised state does not match 802.3 specifications
Driver_ Link__link_state = LINK_STATUS_DOWN
}
Else
{
If (Auto_Neg_Advertise_Reg.Pause_Capable == ENABLED)
{
If ( Auto_Neg_Advertise_Reg.Asymetric_Pause == ENABLED) ) Then
{
If (Auto_Neg_Link_Partner_Ability_Reg.Pause_Capable == ENABLED) Then
{
Driver_Flow_Capability = FLOW_CONTROL_TRANSMIT_PAUSE \
| FLOW_CONTROL_RECEIVE_PAUSE
}
Else If (Auto_Neg_Link_Partner_Ability_Reg.Asymetric_Pause == \
ENABLED) Then
{
Driver_Flow_Capability = FLOW_CONTROL_RECEIVE_PAUSE
}
Else
{
Driver_Flow_Capability = NONE
}
}
//The local physical layer was not configured to advertise Asymmetric pause
Else
{
If (Auto_Neg_Link_Partner_Ability_Reg.Pause_Capable == ENABLED) Then
{
Driver_Flow_Capability = FLOW_CONTROL_TRANSMIT_PAUSE \
| FLOW_CONTROL_RECEIVE_PAUSE
}
Else
{
Driver_Flow_Capability = NONE
}
}
}
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 228
BCM5718 Programmer’s Guide Flow Control
// The local physical layer was not configured to advertise Pause capability
Else If (Auto_Neg_Advertise_Reg.Asymetric_Pause == ENABLED) Then
{
If (Auto_Neg_Link_Partner_Ability_Reg.Pause_Capable == ENABLED) Then
{
Driver_Flow_Capability = FLOW_CONTROL_TRANSMIT_PAUSE
}
Else
{
Driver_Flow_Capability = NONE
}
}
} //Link Status is up
} // Auto negotiation was not disabled && Speed Duplex was not forced
Else
{
// The use forced speed/duplex, so the partner's flow control capabilities are
// indeterminate - software cannot use the Link_Partner_Abitity
// registers.
Driver_Flow_Capability= DISABLED
}
} //The current link is full-duplex at 10/100/1000 wire speeds
Else
{
//Full-Duplex mode is not available or forced half-duplex
//Flow control is not available in half-duplex mode.
Driver_Flow_Capability = NONE
}
//Configure MAC Flow Control Registers if ( Driver_Flow_Capability & FLOW_CONTROL_RECEIVE_PAUSE )
{
Receive_MAC_Mode_Control_Register.Enable_Flow_Control = ENABLED
} if ( Driver_Flow_Capability & FLOW_CONTROL_TRANSMIT_PAUSE ) Then
{
Transmit_MAC_Mode_Control_Register.Enable_Flow_Control = ENABLED
}
} // Link is up on the local PHY
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 229
BCM5718 Programmer’s Guide
Section 11: Interrupt Processing
Interrupt Processing
NetXtreme Legacy Interrupt Model
For reference, this section reviews the legacy NetXtreme interrupt model.
When the controller completes a transmit or a receive event, it updates a status block in host memory. This status block contains information that tells the host which transmit buffers have been DMAed by the controller, and which receive buffer descriptors (Rx BDs) have been consumed by a newly received packet. Normally, host software checks this status block whenever an interrupt is generated. In addition, host software could also poll the status block to determine whether it had been updated by the hardware since the last time it read the
.
Whenever the controller updates the status block, it decides whether to assert the interrupt line (INTA#). If MSI were enabled, the controller would DMA the MSI data DWORD instead of asserting a line interrupt (or, in the case of PCIe, instead of sending an assert interrupt message).
The controller has interrupt avoidance mechanisms (“host interrupt coalescing”) that allow the host to instruct the controller not to generate an interrupt every time it writes a status block into host memory. In addition, it has mechanisms that allow host software to control when and how often the status block is updated in host memory. Since the status block updates and interrupt generation need not happen one-to-one, the following mechanism is in place to communicate to host software if the status block was updated since it was last read by the host — essentially avoiding race conditions:
1. The controller DMAs the status block into host memory before a line interrupt or MSI is generated.
2. The host interrupt service routine (ISR) reads an “update bit” at the top of the status block and checks whether this bit is set to 1.
3. When set to 1, the update bit of the status block indicates to host that the status block has been refreshed by the controller.
4. The ISR must then write a zero to clear/deassert this bit to dirty the status block, and then the ISR may proceed to read the updated producer/consumer index pointers, etc.
5. If the update bit is not set to 1, the interrupt may be considered as spurious and the ISR may wish to abort.
This mechanism allows host software to determine if the status block has been updated. Due to various platform-dependent asynchronous timing issues, an ISR may occasionally see stale status block data. In this case, the ISR may either spin and wait for the status block DMA to complete and explicitly flush the status block, or just wait for the next line interrupt.
Offset
0x00
0x04
3116
Table 84: NetXtreme Legacy Status Block Format
150
Status Word
[31:8] Reserved 0x0 [7:0]Status Tag
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 230
BCM5718 Programmer’s Guide NetXtreme Legacy Interrupt Model
Offset
0x08
0x0C
0x10
0x14
3116
Table 84: NetXtreme Legacy Status Block Format (Cont.)
150
Receive Return Ring 1 Producer Index Receive Standard Producer Ring Consumer
Index
Receive Return Ring 2 Producer Index
Send BD Consumer Index
Reserved 0x0
Receive Return Ring 3 Producer Index
Receive Return Ring 0 Producer Index
Receive Jumbo Producer Ring Consumer Index
Legacy Status Word Format:
• Bit [0]: Update-Bit
• Bit [1]: Link Status Change
• Bit [2]: Error/Attention
• Bit [3]: Resvd–always 0
• Bit [4]: Resvd–always 0
• Bit [5]: Resvd–always 0
• Bits [31:6]: Reserved 0x0
ISR Flow
The basic flow of an ISR is as follows:
1. Acknowledge interrupt. Write a nonzero value (i.e., value = 1) to interrupt mailbox 0 (see Interrupt Mailbox
0 Register 0x200) for host standard and flat modes, and other Interrupt Mailbox Registers 0x5800-0x40 for indirect mode, to indicate to the controller that the driver is currently processing the interrupt. This step temporarily disables further device interrupts.
2. Read and save the value of the status tag field of the status block.
3. Claim the interrupt. Determine if action is required. Read the updated bit of the status word. If the update bit is asserted then the controller has updated the status block.
4. Clear the update bit of the status word in the status block. This indicates that the host driver either has or will touch the status block. If a during interrupt event occurs, the host driver can examine the update bit later to determine if a fresh status block has been moved to host memory space since.
5. Check for Rx traffic indications: a. Loop through enabled Rx Return Rings (0 to 3).
b. Check for differences between Rx return ring producer indices (status block) and Rx return ring consumer indices (value written to mailbox on previous call) as this indicates the number of Rx frames to process from the Rx return ring.
c. Process the Rx packets.
d. Update the Rx return ring consumer indices in each mailbox.
6. Check for Tx completions:
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 231
BCM5718 Programmer’s Guide NetXtreme Legacy Interrupt Model a. Check for differences between previous consumer index (tracked by host driver) and current consumer index in the status block. These are the Tx BDs which can be made available again to the next send operation.
b. Update the previous consumer index (i.e., next call) to the value of the status block consumer index.
7. Compare the current value of the status tag to the saved value of the status tag. Flush the status block (i.e., flush status blocks cached by intermediate PCI bridges).
8. Check the update bit in the status word of the status block. If the update bit is asserted, then new data has been DMAed to the host. a. Repeat steps 5 and 6.
9. Check error bit in status word. The driver may check the state machine/FTQ status registers for various attentions.
10. Reenable interrupts. When the “status tagged” status mode bit of the miscellaneous host control register
(see “Miscellaneous Host Control Register (offset: 0x68)” on page 283
) is set to 1, write the saved status tag to the upper 8 bits of Interrupt Mailbox 0, and 0 to the remaining bits (23 down to 0) to indicate that the
ISR is finished processing Rx and Tx. Otherwise write 0 to Interrupt Mailbox 0 register. This step also clears existing interrupts.
Legacy Status TAGGING Mode
This mode is enabled by setting the “status tagged” mode bit of the miscellaneous host control register (0x68).
When enabled, a unique 8-bit tag value is inserted into the status block status tag at location [7:0]. The status tag can be returned to the [31:24] field of the INT mailbox register by the driver. When the mailbox register field [23:0] is written with a zero value, the tag field of the mailbox register is compared with the tag field of the last status block to be DMAed to the host. If the tag returned is not equivalent to the tag of the first status block DMAed, then the controller triggers another interrupt immediately.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 232
BCM5718 Programmer’s Guide Basic Driver Interrupt Processing Flow
Basic Driver Interrupt Processing Flow
Flowchart for Servicing an Interrupt
The following figure shows the basic driver interrupt service routine flow.
Figure 53: Basic Driver Interrupt Service Routine Flow
NIC encounters an Interrupt event and asserts in IN TA# line to interrupt host
H ost OS receives Interrupt and calls NIC driver's ISR
Driver reads the "status w ord" field in the
570X's "Status Block" (located in host m em ory)
Tell the OS "N ot m y interrupt", and
R ETU RN (this is im portant for interruptsharing environm ents)
N o
Is the "U pdated bit" set in the Status
Block?
IS R C ode
D P C C ode
Process any Link Status change events.
Yes
D river writes a value of '"1" into the 570X's
Interrupt M ailbox 0 register. W hile this register contains a nonzero value, it prevents future interrupt assertions from the 570X.
D river clears the "updated" bit in the Status
Block.
D river claim s the Interrupt and schedules a callback to handle the interrupt processing
(m any OSes due this via a low er priority thread). Allternatively, the driver could directly invoke the interrupt processing code.
Process any received packets (receive interrupts).
No m ore work to do. Exit the D PC
Process any com pleted transm its (transm it interrupts).
Enable Interrupts by w riting the Interrupt
M ailbox 0 register to '0'.
Read Interrupt M ailbox 0 in order to flush any posted writes in the PCI chipset.
There is m ore w ork to do. Force an interrupt by setting bit 2 in the M isc Local Control
Regsiter (offset 0x6808)
No
R ead the,
Is the "U pdated bit" set in the Status Block?
Yes
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 233
BCM5718 Programmer’s Guide Basic Driver Interrupt Processing Flow
Interrupt Procedure
1. Acknowledge interrupt. Write a nonzero value (i.e., value = 1) to the interrupt mailbox 0 (see registers
0x270–0x304) to indicate that the driver is currently processing the interrupt. This step disables device interrupts except during interrupt feature.
2. Read and save the value of the Status Tag field of the Status Block (see
3. Claim interrupt. Determine if the Ethernet controller action is required. Read the Updated bit of the status word. If the Updated bit is asserted, then the host coalescing engine has updated the status block.
4. Clear the Updated bit of the status word. This indicates that the host driver either has or will touch the status block. If a during interrupt event is driven, the host driver can examine the Updated bit to determine if a fresh status block has been moved to host memory space.
5. Check for RX traffic.
• Loop through enabled RX Return Rings (0 to 3).
• Check for difference between RX Return Ring Producer index (Status block) and RX Return Ring
Consumer index (value written to mailbox on previous call) are the number of frames to process for RX
Return Ring.
• Process the packet.
• Update the RX Return Ring consumer pointer in each mailbox for new RX frames.
6. Check for TX completes.
• Loop through enabled TX Send Rings.
• Check for difference between previous consumer index (software kept) and current consumer index in the status block. These are the TX BDs which can be made available to next send operation.
• Update the previous consumer index (i.e., next call) to the value of the status block consumer index.
7. Compare the current value of the Status Tag to the saved value of the Status Tag. Flush status block (i.e., force update of status blocks cached by PCI bridge).
• Read interrupt mailbox (see “Interrupt Mailbox 0 (High Priority Mailbox) Register (offset: 0x200-207) for host standard and “Interrupt Mailbox 0 Register (offset: 0x5800)” for indirect mode).
• Check the Updated bit in the status word located in the status block. If the Updated bit is asserted, then new data has been DMAed to the host. Repeat steps 5 and 6.
8. Check the Error bit in status word (optional). The driver may check the state machine/FTQ status registers for various attentions.
9. Enable interrupts. When Status Tagged Status mode bit of the Miscellaneous Host Control register (see
“Miscellaneous Host Control Register (offset: 0x68)” on page 283
) is set to 1, then write the saved Status
Tag to the upper 8 bits of Interrupt Mailbox 0, and 0 to the remaining bits (23 down to 0) to indicate that the ISR is done processing RX/TX. Otherwise, write 0 to Interrupt Mailbox 0 register. This step also clears existing interrupts.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 234
BCM5718 Programmer’s Guide Host Coalescing
Host Coalescing
Interrupt coalescing (or interrupt moderation) is a common technique used by NIC vendors to increase the performance of NICs. High-level descriptions of the benefits of interrupt coalescing can be found at:
• http://www.microsoft.com/HWDEV/devdes/optinic.htm
• http://support.microsoft.com/support/kb/articles/Q170/6/43.ASP
• http://msdn.microsoft.com/library/books/serverdg/networkadapterrequirements.htm
Description
The Ethernet controller supports the concept of host coalescing. Host coalescing controls when status information is returned to the host, and when interrupts are generated. The Ethernet controller provides a number of software configurable registers that control when/how it updates the host with status information and how often it asserts an interrupt.
When the Ethernet controller has completed transmit or receive events, it updates a Status block in host memory. This status block contains information that tells the host which transmit buffers have been DMAed by the NIC, and which receive Buffer Descriptors (BDs) have been consumed by a newly arrived received packet.
Normally, the host will check this status block when an interrupt is generated. In addition, the host could also poll the status block to determine whether or not it had been updated by the hardware since the last time the host had read the status block (this is called during interrupt processing).
When the NIC updates the status block, it will make a decision about whether to assert the interrupt line (INTA) or not. The Ethernet controller has special interrupt avoidance mechanisms that allow the host to tell the NIC not to generate an interrupt when it writes a status block back to the host. In addition, there are also mechanisms that allow host software to control when and how often the status block is updated.
Example: The host could configure the NIC to only update status block after it receives two packets, as opposed to one packet. These mechanisms are documented in more detail to follow.
Operational Characteristics
The Ethernet controller DMAs the status block to host memory before a line interrupt or MSI is generated. The host ISR reads the update bit at the top of the status block and checks whether this bit is set to 1 or not. When set to 1, the updated bit of status block indicates the host that the status block has been refreshed by the MAC.
The ISR must then write to clear/de-assert this bit to dirty the status block, and then the ISR may proceed to read the updated producer/consumer index pointers. This mechanism allows host system software to determine if the status block has been updated. Due to various asynchronous timing issues (dependent upon platform) the ISR may occasionally see stale data. The ISR may either spin and wait for the status block DMA to complete and explicitly flush the status block or just wait for the next line interrupt.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 235
BCM5718 Programmer’s Guide Host Coalescing
Registers
The Ethernet controller supports a variety of registers that affect status block updates and interrupt generation
Table 85: Interrupt-Related Registers
Register Cross Reference
Miscellaneous Host Control register.
The two bits of this register that are related to interrupts are:
• Mask PCI Interrupt Output (aka Mask Interrupt) bit
• Clear Interrupt INTA bit
“Miscellaneous Host Control Register (offset: 0x68)” on page 283
.
Miscellaneous Local Control register.
The two bits of this register that are related to interrupts are:
• Set Interrupt bit
• Clear Interrupt bit
“Miscellaneous Local Control Register (offset:
Interrupt Mailbox 0 register
Receive Coalescing Ticks register
Send Coalescing Ticks register
Receive Max Coalesced BD Count register
Send Max Coalesced BD Count register
“Interrupt Mailbox 0 Register (offset: 0x5800)” on page 463
.
“Receive Coalescing Ticks Register (offset: 0x3C08)” on page 416
.
“Send Coalescing Ticks Register (offset: 0x3C0C)” on page 417
.
“Receive Max Coalesced BD Count Register (offset:
“Send Max Coalesced BD Count Register (offset:
Mode Control register.
• Interrupt on Flow Attention (Bit 28) causes a host interrupt when an enabled flow attention occurs
• Interrupt on DMA Attention (Bit 27) causes a host interrupt when an enabled DMA attention occurs
“Mode Control Register (offset: 0x6800)” on page 472
• Interrupt on MAC Attention (Bit 26) causes a host interrupt when an enabled MAC attention occurs
• Interrupt on RX RISC Attention (Bit 25) causes a host interrupt when an enabled RX-RISC attention occurs
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 236
BCM5718 Programmer’s Guide MSI
MSI
PCI Specification 2.2 defines a new mechanism for a device to request services by its device driver. It is called
Message Signaled Interrupt (MSI). MSI will eventually deprecate the traditional interrupt mechanism. In MSI, device DMAs a specified DWORD data to a specified host address if it needs to request services by its device driver. The MSI state machine can be enabled/disabled by setting/resetting the Enable bit of MSI Mode register
(offset 0x6000). By default, this bit is set to 1 indicating that the MSI state machine is enabled. The main advantages of MSI generation versus using a traditional interrupt are as follows:
• Eliminates the need for interrupt signal trace on the PCI device.
• Eliminates the need to perform a dummy read from the device by the device driver in its interrupt service routine. The dummy read is done at the beginning of ISR to force all posted memory writes to be flushed to the host memory.
Traditional Interrupt Scheme
A simplified block diagram showing traditional interrupt scheme is depicted in
.
Figure 54: Traditional Interrupt Scheme
BCM5718 Ethernet Controller
BCM5700
Interrupt A
PCI Bus
Memory
Host Bus
CPU
Interrupt
To clarify second issue in traditional interrupt scheme, an example is given. The Ethernet controller receives one or more packets from the networks. The Ethernet controller does the following:
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 237
BCM5718 Programmer’s Guide MSI
• DMAs data of received packets to the host.
• DMAs receive buffer descriptors to Receive Return Ring in the host memory.
• DMAs status block to the host memory.
• Generates an interrupt to request its device driver for processing.
The writes are posted and are actually performed at some later time by the PCI host bridge. When interrupt service routine of device driver is executed, the driver reads the status block from the host memory and finds that status block does not contain latest index information if the writes for status block are not performed by the PCI host bridge yet. The scheme to resolve this problem is to do a dummy read of the Ethernet controller in the beginning of the interrupt service routine. The dummy read has to traverse the same bridge that memory writes from the Ethernet controller have to traverse to get to the host memory. The ordering rules for bridges dictate that the bridge must flush its posted write buffers before permitting a read to traverse the bridge. As a result, writes for status block are flushed to the host memory by the bridge before dummy read cycle is completed.
Message Signaled Interrupt
A simplified block diagram showing a possible MSI scheme is depicted in
Figure 55: Message-Signaled Interrupt Scheme
BCM5718 Ethernet
PCI Bus
Bridge
Host Memory
Host Memory
Host Bus
Interrupt
Controller
Controller
Interrupt
CPU
CPU
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 238
BCM5718 Programmer’s Guide MSI
Similar example in traditional interrupt scheme is used again here to illustrate MSI concept. The Ethernet controller receives one or more packets from the networks. The Ethernet controller does the following:
• DMAs data of received packets to the host.
• DMAs receive buffer descriptors to receive return ring in the host memory.
• DMAs status block to the host memory.
• Writes specified DWORD data to specified host address.
In this mode, the Ethernet controller writes DWORD data to specified host address instead of generating an interrupt. The specified data and address are configurable. The specified address is typically a memory-mapped
IO port within the PCI host bridge. The PCI host bridge is the gateway to the main memory controller. This means that the DWORD data write (MSI message) to PCI host bridge is in the posted write buffers and was posted after the writes for the status block update. It is the rule that PCI host bridge must perform posted writes in the same order that they were received. This means that by the time MSI message arrives at the PCI host bridge, the status block has already been posted to the host memory. Upon receipt of the MSI message write, the PCI host bridge generates the interrupt request to the processor. Interrupt service routine of the device driver is invoked. It is not necessary to do a dummy read because updated status block is already in the host memory.
PCI Configuration Registers
Operating system/system software can configure the specified DWORD data and specified 64-bit host address for the device with MSI_DATA (Offset 0x64) and MSI_Address register (Offset 0x5c), respectively.
MSI Address
This is a 64-bit field. MSI address at offset 0x5c and 0x60 should be programmed with the low-order and highorder bits of the 64-bit physical address. If the host only supports 32-bit physical address, the high-order address should be programmed with zeros.
MSI Data
This is a 16-bit field. The least significant three bits can be modified by the Ethernet controller when it writes
MSI message to host. The DWORD data for the MSI message is depicted as shown in Figure 56 .
Figure 56: MSI Data FIeld
31 16 15 0
The BCM5700 MAC can only modify the three LSBs
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 239
BCM5718 Programmer’s Guide MSI
The Ethernet controller can support up to eight message types, and these MSI messages can be generated by either of the two sources of:
• Host coalescing engine
• Firmware
Host Coalescing Engine
After the host coalescing engine updates the status block on the host (due to receive indication, transmit completion, and so on), it either generates an interrupt or writes a MSI message if MSI is enabled. The least significant 3-bits of the MSI message originating from host coalescing block is configurable and can be configured by programming bits 4, 5, and 6 of the Host_Coalesing_Mode register. The default of these bits is zeros.
Firmware
The Ethernet controller provides a way for firmware executed by RX RISC to generate MSI messages. Firmware can generate MSI messages by using MSI_FIFO_Access register (Offset 0x6008). For example, if firmware wants to generate an MSI message with least significant 3-bit as 0x2, it will write 2 to MSI_FIFO_Access register. It also needs to verify that the MSI message is written successfully by reading back MSI_FIFO_Access Overflow. If this bit is zero, then the MSI message is encoded successfully and will be sent to HOST. Otherwise, the message is not encoded.
Note: Without any special firmware supporting multiple MSIs, the device can generate only 1 MSI message even though the device requests for 8 MSI messages through Multiple Message Capable field (bits 3:1) of Message Control register (offset 0x5A). The least significant 3-bits of the MSI message generated by the device are always taken from bits 6:4 of Host_Coalesing_Mode register
(offset 0x3C00).
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 240
BCM5718 Programmer’s Guide MSI-X
MSI-X
The BCM5718 family introduces PCIe™-compliant message signal interrupts (MSI-X), supporting a maximum of
17 vectors (BCM5718).
This is accomplished without disturbing the legacy interrupt system model. The introduction of interrupt vectoring provides per-receive-queue indication and per-transmit-queue completion to the device driver. The maximum number of MSI-X vectors attainable is indirectly determined by certain controller modes of operation.
The existence of multiple receive queues (Rx return rings) in the BCM5718 family stems either from the Receive
Side Scaling (RSS) feature or from the I/O Virtualization (IOV) feature.
Note: RSS and IOV are mutually exclusive — only one of these features may be enabled in the
BCM5718 family at any time.
When both RSS and IOV are disabled, all of the received packets are posted in the default receive queue (Rx
Return Ring 0). This default Rx return ring is sometimes referred to as “the Receive Return Ring”. The number of transmit queues is also limited to one in this use case.
In RSS enabled mode, receive traffic classification (also known as sorting) is done purely based on the RSS hash lookup table. Four RSS receive queues and a single transmit queue exist in this mode for the controller.
Therefore, up to 5 MSI-X vectors are offered in RSS mode.
In IOV mode, receive traffic classification is performed by VRQ filters. Sorted Rx traffic is routed among 17
Virtual Receive queues (VRQs). The transmit side offers up to 16 transmit queues. Therefore, while in IOV mode, there is an opportunity to map packet indication and completions into 17 MSI-X vectors.
The MSI-X specification allows a device to advertise the availability of a chosen number of vectors in its PCIe capabilities list; however, it does not specify a mechanism for an operating system (OS) to negotiate the number of vectors down to the number it would actually want to use. So, one simple way for an OS to allocate fewer vectors than a device has asked for is to fill out all of the vector table entries advertised by a device. In such an event, the device would first need to identify the situation, and then reassign or regroup the internal interrupt sources into the limited number of table entries allocated by the OS. But, the device hardware would not possess all of the information necessary to accomplish either task. This means that device driver involvement is required. In the BCM5718 family, it has been assumed that, in most cases, an OS would allocate the requested number of vectors. A deviation from that would be considered an exception; in that case, because support for MSI-X has already been advertised to the BIOS, there would be no way to fall back to INTx mode. This situation could be handled by forcing everything into MSI-X Vector#0 and not using other vectors, disregarding however many of the originally requested vectors the OS actually allocated.
Thus, the BCM5718 family offers two vector modes within the MSI-X mode:
• Single Vector mode
• Multivector mode
Each of these two modes in turn offers the following submodes:
• Single Vector mode (Restrict to Vector#0)
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 241
BCM5718 Programmer’s Guide MSI-X
– Single Vector RSS mode
– Single Vector IOV mode
• Multivector mode
– Multivector RSS mode (5 Vectors Requested)
– Multivector IOV mode (17 Vectors Requested)
The idea is that during boot up, if either RSS or IOV mode is to be enabled, the driver would program the controller into the appropriate Multivector submode first. Subsequently, if the OS does not allocate the requested number of MSI-X vectors (five for RSS and 17 for IOV), then the driver must reprogram the controller into the appropriate Single Vector submode. All of this must be performed before the driver enables the EMAC to receive or transmit traffic. Once traffic is started, the MSI-X Vector mode must not be reprogrammed. The controller will behave unpredictably if that is done.
Single Vector mode or Multivector mode may be chosen by the driver by programming register bit 0x6000[7].
Such submodes are derived by the controller from the appropriate RSS, IOV and Multiple Send queue mode settings. All permissible combinations are shown in the table below.
Note: IOV mode enables 16 VRQs but does not necessarily enable multiple Tx queues, which may be enabled independent of IOV-mode selection. Hence the send completion, when Single Send queue is chosen, will always be indicated on Vector#0 while Multiqueue Send Completions will be paired with
Rx queue indications in the respective vectors.
Table 86: MSI-X Vector Mode Selection
MSI-X Vector Mode
Multivector Enable
0x6000[7]
Single Vector —
RSS Mode
Single TXQ 0x0
Multiple TXQ 0x0
Single Vector —
IOV Mode
Single TXQ 0x0
Multiple TXQ 0x0
Multivector —
RSS Mode
Single TXQ
Multiple TXQ
0x1
0x1
Multivector —
IOV Mode
Single TXQ 0x1
Multiple TXQ 0x1
RSS Mode Enable
0x468[23]
IOV Mode Enable
0x6800[8]
Multi TXQ Enable
0x1800[5]
0x1
0x1
0x0
0x0
0x1
0x1
0x0
0x0
0x0
0x0
0x1
0x1
0x0
0x0
0x1
0x1
0x0
0x1
0x0
0x1
0x0
0x1
0x0
0x1
Vector allocation in each mode is shown below:
Single Vector Mode:
– RSS mode: Vector#0–Aggregate of the following:
– Rx Return Ring Indication (Active only when RSS is Disabled)
– Send Completion
– Link Status Change
– Error/Attention
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 242
BCM5718 Programmer’s Guide
– Rx Return Ring3 through Ring0 Indications
• IOV mode: Vector#0 — Aggregate of the following:
– All 16 Send queue Completions
– All 17 VRQ Indications
– VRQ Active Bit-Map
– Link Status Change
– Error/Attention
Multivector Mode:
• RSS (5 Vector) mode: Vector allocation would be as follows:
– Vector#0–Aggregate of the following:
- Send Completion (Only in Single TXQ mode)
- Link Status Change
- Error/Attention
– Vector#1–RSS Return Ring 0 Indication/TXQ 1 Completion (in Multi TXQ mode)
– Vector#2–RSS Return Ring 1 Indication/TXQ 2 Completion (in Multi TXQ mode)
– Vector#3–RSS Return Ring 2 Indication/TXQ 3 Completion (in Multi TXQ mode)
– Vector#4–RSS Return Ring 3 Indication/TXQ 4 Completion (in Multi TXQ mode)
• IOV (17 Vector) mode: Vector allocation would be as follows
– Vector#0 — Aggregate of the following:
- VRQ0 Indication (Default VRQ)
- Send Completion (Only in Single TXQ mode)
- Link Status Change
- Error/Attention
- VRQ Active Bit-Map
– Vector#1 — Aggregate of the following:
- VRQ1 Indication
- Transmit queue 1 Completion (in Multi TXQ mode)
- Link Status Change
– Vector#2–Aggregate of the following
- VRQ2 Indication
- Transmit queue 2 Completion (in Multi TXQ mode)
.
- Link Status Change
.
.
– Vector#16–Aggregate of the following
- VRQ16 Indication
- Transmit queue 16 Completion (in Multi TXQ mode)
- Link Status Change
BROADCOM
®
July 17, 2013 • 5718-PG107-R
MSI-X
Page 243
BCM5718 Programmer’s Guide MSI-X
In the case of Multivector selection, it is not mandatory to enable all four RSS queues or enable all VRQs/send queues. So, in the case of Multivector, RSS vector#1 through vector#4 are useful only in conjunction with the respective number of RSS queues or CPU# enable. For example, if only three receive queues are receiving traffic, and the Multivector mode is selected by the driver, then even though vectors 0, 1, 2, and 3 remain active, vector#4 never triggers. Hence, in Multivector RSS mode the receive queues are hard-mapped 1:1 with
MSI-X vectors 1 through 4. Therefore it is not permissible to regroup receive queues into vectors in an arbitrary manner. This policy also applies to Multivector IOV mode.
Consequently, Multivector mode must only be selected in conjunction with the enabling of either RSS or IOV.
Note: There is no known use case for enabling Multiple Tx queues (TSS) without enabling either RSS or IOV. Therefore, doing so is not permitted.
In Single Vector mode, all receive traffic is indicated via Vector#0. In RSS mode, all four receive queue indications, and all four Send Completions are thus grouped together in Vector#0. While in IOV mode, all 17 Rx queues and all 16 Tx queues are grouped into Vector#0. Also, this mode must be used when RSS and IOV are both disabled (but MSI-X is enabled, although it is not anticipated that doing this has any real-world usefulness). There could be instances when the OS enables RSS but allocates only one MSI-X vector to our device. The Broadcom driver would resort to choosing Single Vector mode in such a scenario.
Note: When MSI-X is disabled, INTx or MSI continue to function as in legacy device implementations.
The data structures used in INTx and MSI mode are identical to those in Single Vector MSI-X mode.
The BCM5718 family offers two product SKUs. The MSI-X table size requested by each differs:
• BCM5717: This controller advertises MSI-X table size = 5.
• BCM5718: This controller advertises an MSI-X table size = 17 by default; however, an NVRAM
Configuration option is be provided to limit the Table Size to 5 if desired.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 244
BCM5718 Programmer’s Guide MSI-X Plumbing
MSI-X Plumbing
The basic interrupt/ISR model for NetXtreme controllers remains largely unchanged even with the introduction of vectored interrupts in the form of MSI-X. However, there are four basic design changes:
• Replication of status blocks, mailboxes, etc. This essentially scales the legacy model from one vector to five or 17 vectors.
• Implement MSI-X capability structures in PCIe configuration space.
• Implement PCI-mandated MSI-X data structures within each MAC core, including new base address register (BAR) decoder, MSI-X table, pending bit array (PBA) structure, etc.
• Add per Rx queue host coalescing attributes.
Each item is discussed in detail in the following sections.
Replication of Status Blocks and INT Mailboxes
As shown in
, all four Rx Return Consumer Indices are indicated in the legacy Status Block.
When MSI-X Multivector RSS mode is enabled, it leads to logically five interrupt vectors. Each of these vectors is bound to its own status block; thus, there are five different status blocks, numbered Status-Block0 through
Status-Block4. Each of these five vectors is also bound to its own INT Mail-Box registers.
Each Status Block has a fixed location in the Host Memory. Hence, there are five Status Block Host Address
shows all five Status-Block Host Address Registers and INT Mail-Box Registers.
Similarly, in case of MSI-X Multivector IOV mode, there are 17 sets of Status Blocks and MailBox registers.
Table 87: MIS-X Status-Block and Mail Box Addresses
IOV Mode
Status
Block
Number
Status Block
Host
Address
Register
(64-bit)
INT Mail
Box
Register
Address Indication Items
Legacy 0x3C3C,
0x3C38
0x200 (*) ALL
RSS Mode
INT Mail
Box
Register
Address Indication Items
0x200(*) ALL
Comments
Legacy Status Block.
Used by INTx or MSI.
Also used in MSI-X
Single-Vector RSS mode or IOV mode
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 245
BCM5718 Programmer’s Guide MSI-X Plumbing
Table 87: MIS-X Status-Block and Mail Box Addresses (Cont.)
Status
Block
Number
Status Block
Host
Address
Register
(64-bit)
0
1
2
3
4
5
6
7
0x3C3C,
0x3C38
0x3D00,
0x3D04
0x3D08,
0x3D0C
0x3D10,
0x3D14
0x3D18,
0x3D1C
0x3D20,
0x3D24
0x3D28,
0x3D2C
0x3D30,
0x3D34
IOV Mode
INT Mail
Box
Register
Address Indication Items
RSS Mode
INT Mail
Box
Register
Address Indication Items Comments
0x200 VRQ 0 RR Prod
Index
Error/Attention
Single SBD Cons
Index
Link Status
RBD 0 Cons
Indexes
VRQ Active Bit-
Map
0x200 (*) Link-Status change
Error/Attention
Single SBD Cons Index
StdRBD Cons Indx
JmbRBD Cons Indx
0x208 VRQ1 RR Prod
Index
SBD1 Cons Index
RBD 1 Cons Index
0x208 (*) Rx Return Ring 0 Prod Index
SBD1 Cons Index
0x210 VRQ2 RR Prod
Index
SBD2 Cons Index
RBD 2 Cons Index
0x210 (*) Rx Return Ring 1 Prod Index
SBD2 Cons Index
Used only in MSI-X
Multivector RSS mode or
Multivector IOV mode for
Vector#0 -- Vector#2.
0x218 VRQ3 RR Prod
Index
SBD3 Cons Index
RBD 3 Cons Index
0x218 (*) Rx Return Ring 2 Prod Index
SBD3 Cons Index
Used only in MSI-X
Multivector RSS mode or Multivector
IOV mode
0x220 VRQ4 RR Prod
Index
0x220 (*) Rx Return Ring 3 Prod Index
SBD4 Cons Index for
Vector#3 -- Vector#4.
SBD4 Cons Index
‘RBD 4 Cons
Index
N/A 0x228 VRQ5 RR Prod
Index
SBD5 Cons Index
RBD 5 Cons Index
N/A
0x22C VRQ6 RR Prod
Index
SBD6 Cons Index
RBD 6 Cons Index
N/A N/A
Used in MSI-X
Multivector IOV mode for
Vector#5 --
Vector#16
0x230 VRQ7 RR Prod
Index
SBD7 Cons Index
RBD 7 Cons
Indexes
N/A N/A
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 246
BCM5718 Programmer’s Guide MSI-X Plumbing
Table 87: MIS-X Status-Block and Mail Box Addresses (Cont.)
IOV Mode
Status
Block
Number
Status Block
Host
Address
Register
(64-bit)
INT Mail
Box
Register
Address Indication Items
…….
..…..
……..
……..
……..
……..
…….
……..
16 0x3D78,
0x3D7C
0x254 VRQ16 RR Prod
Index
SBD16 Cons
Index
RBD 16 Cons
Index
N/A
RSS Mode
INT Mail
Box
Register
Address Indication Items
N/A N/A
N/A
Comments
–
–
Each replicated Status Block has its own format; however, each of the new status block formats is extrapolated from the legacy status block format.
Note: Although High Priority INT Mail Boxes are DWORD (32-bit) registers, the original four were placed on QWORD (64-bit) boundaries for legacy PCI 64-bit target access purposes. Broadcom’s PCIe does not allow 64-bit target accesses. Thus, the new Mailboxes are being placed 32-bits apart; however, we are keeping the original four addresses intact.
Single-Vector RSS Mode Status Block Format
shows the Status-Block format used by Vector#0 in the Single-Vector RSS mode. Note that first 24B of this structure is identical to the legacy Status Block.
Table 88: Status Block Format (MSI-X Single-Vector RSS Mode)
Offset 3116
0x00
0x04
0x08
0x0C
0x10
0x14
0x18
150
Status Word
[31:8] Reserved 0x0 [7:0]Status Tag
Receive Standard Producer Ring Consumer Index Receive Return Ring 1 Producer Index
Receive Return Ring 2 Producer Index
Send BD [1] Consumer Index
(Acts as Single Send queue Cons Index)
Receive Return Ring 3 Producer Index
Receive Jumbo Producer Ring Consumer Index
Send BD 2 Consumer Index (IF Multi SendQ)
Send BD 3 Consumer Index (IF Multi SendQ)
Receive Return Ring 0 Producer Index
Send BD 4 Consumer Index (IF Multi SendQ)
Status-Block [0] Status Word Format (Single-Vector RSS):
• Bit [0]: Update-Bit
• Bit [1]: Link Status Change
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 247
BCM5718 Programmer’s Guide MSI-X Plumbing
• Bit [2]: Error/Attention
• Bits [31:4]: Reserved 0x0
Single-Vector IOV Mode Status Block Format
In Single-Vector IOV mode, all events are reported in the same Status Block used by Vector#0.
Table 89: Status Block format (MSI-X Single-Vector IOV Mode)
0x48
0x4C
0x50
0x54
0x58
……..
0x8C
0x0C
0x10
0x14
0x18
0x1C
0x20
0x24
……
Offset 3116
0x00
0x04
0x08
Status Word
[31:8] Reserved 0x0
Single Send BD Consumer Index (IF Single Send queue ELSE 0x0)
Send BD 1 Consumer Index (IF Multi SendQ)
Send BD 2 Consumer Index (IF Multi SendQ)
Send BD 3 Consumer Index (IF Multi SendQ)
Send BD 4 Consumer Index (IF Multi SendQ)
Send BD 5 Consumer Index (IF Multi SendQ)
Send BD 6 Consumer Index (IF Multi SendQ)
Send BD 7 Consumer Index (IF Multi SendQ)
…………………………
150
Send BD 16 Consumer Index (IF Multi SendQ) Receive Return Ring 16 Producer Index
Standard RBD Ring 0 Consumer Index Jumbo RBD Ring 0 Consumer Index
Standard RBD Ring 1 Consumer Index
Standard RBD Ring 2 Consumer Index
Jumbo RBD Ring 1 Consumer Index
Jumbo RBD Ring 2 Consumer Index
Standard RBD Ring 3 Consumer Index
…………………………
Standard RBD Ring 16 Consumer Index
……………………….
[7:0]Status Tag
Receive Return Ring 0 Producer Index
Receive Return Ring 1 Producer Index
Receive Return Ring 2 Producer Index
Receive Return Ring 3 Producer Index
Receive Return Ring 4 Producer Index
Receive Return Ring 5 Producer Index
Receive Return Ring 6 Producer Index
Receive Return Ring 7 Producer Index
Jumbo RBD Ring 3 Consumer Index
……………………….
Jumbo RBD Ring 16 Consumer Index
Status-Block Status Word Format (Single-Vector IOV):
• Bit [0]: Update-Bit
• Bit [1]: Link Status Change
• Bit [2]: Error/Attention
• Bits[5:3]: Resvd–always 0x0
• Bit [6]: Change in VRQ Active Bit Map
• Bits [14:7]: Resvd–always 0x0
• Bit[15]: VRQ Active Bit-Map[16]
• Bits [31:16]: VRQ Active Bit-Map[15:0]
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 248
BCM5718 Programmer’s Guide MSI-X Plumbing
Multivector RSS Mode Status Block Format
There are five slightly different Status-Block formats used by the Multivector RSS mode–each of these formats associate with their respective vector numbers as, shown in
Table 90: Status Block [0] Format (MSI-X Multivector RSS Mode] offset 3116
0x00
0x04
0x08
0x0C
0x10
0x14
150
Status Word
[31:8] Reserved 0x0
Receive Standard Producer Ring Consumer Index Reserved 0x0
Reserved 0x0 Reserved 0x0
Reserved 0x0 Single Send BD Consumer Index (IF Single Send queue ELSE 0x0)
Reserved 0x0
[7:0] Status Tag
Receive Jumbo Producer Ring Consumer Index
Status-Block [0] Status Word Format (Multivector RSS):
• Bit [0]: Update-Bit
• Bit [1]: Link Status Change
• Bit [2]: Error/Attention
• Bit [3]: Resvd–always 0
• Bit [4]: Resvd–always 0
• Bit [5]: Resvd–always 0
• Bits [31:6]: Reserved 0x0
Table 91: Status Block [1 N ≤ 4] Formats (MSI-X Multivector RSS Mode) offset 3116
0x00
0x04
0x08
0x0C
0x10
0x14
Status Word {Valid for all Status Blocks}
[31:8] Reserved 0x0
Reserved 0x0
150
[7:0] Status Tag[n]
{Independent for each
Status Blocks}
Receive Return Ring 1 Producer Index {Valid only for Status Block2 else Rsvd 0x0}
Receive Return Ring 2 Producer Index {Valid only for Status Block3 else Rsvd 0x0}
Receive Return Ring 3 Producer Index {Valid only for Status Block4 else Rsvd 0x0}
Send BD [1 ≤ N ≤ 4] Consumer Index (IF Multi
Send queue Enabled, ELSE 0x0)
Receive Return Ring 0 Producer Index {Valid only for Status Block1 else Rsvd 0x0}
Reserved 0x0 Reserved 0x0
Status-Block [1–4] Status Word Format (Multivector RSS):
• Bit [0]: Update-Bit
• Bit [31:1]: Reserved 0x0
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 249
BCM5718 Programmer’s Guide MSI-X Plumbing
Multivector IOV Mode Status Block Format
Two slightly varying Status Block formats are used in this mode: one format for Status Block#0 and the other
for Status Block#1 through Status Block#16, as shown in Table 92
.
Table 92: Status Block [0] Format (MSI-X Multivector IOV Mode) offset 3116
0x00
0x04
0x08
0x0C
0x10
0x14
150
Status Word
[31:8] Reserved 0x0
RBD0 Standard Producer Ring Consumer Index Reserved 0x0
Reserved 0x0 Reserved 0x0
[7:0] Status Tag
Receive Return Ring 0 Producer Index Single Send BD Consumer Index (IF Single Send queue ELSE 0x0)
Reserved 0x0 RBD0 Jumbo Producer Ring Consumer Index
Status-Block [0] Status Word Format (Multivector IOV):
• Bit [0]: Update-Bit
• Bit [1]: Link Status Change
• Bit [2]: Error/Attention
• Bits[5:3]: Resvd–always 0x0
• Bit [6]: Change in VRQ Active Bit Map
• Bits [14:7]: Resvd–always 0x0
• Bit[15]: VRQ Active Bit-Map[16]
• Bits [31:16]: VRQ Active Bit-Map[15:0]
Table 93: Status Block [1 ≥ N ≤ 16] Format (MSI-X Multivector IOV Mode) offset 3116
0x00
0x04
0x08
0x0C
0x10
0x14
150
Status Word
[31:8] Reserved 0x0
RBD[N] Standard Producer Ring Consumer Index Reserved 0x0
Reserved 0x0 Reserved 0x0
[7:0] Status Tag
Receive Return Ring [N] Producer Index Send BD [N] Consumer Index (IF Multiple Send queue ELSE 0x0)
Reserved 0x0 RBD [N] Jumbo Producer Ring Consumer Index
Status-Block [1–16] Status Word Format (Multivector IOV):
• Bit [0]: Update-Bit
• Bit [1]: Link Status Change
• Bits [31:1]: Resvd–always 0x0
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 250
BCM5718 Programmer’s Guide MSI-X Plumbing
MSI-X Capability Structure
MSI-X requires that its device-resident data structures or registers be addresses over BARs, as opposed to MSI, which declares its device structure addresses directly in related configuration registers. The are certain restrictions related to the BARs:
• If a BAR is shared with MSI-X for other device purposes, the MSI-X region must be isolated within a 4 KB naturally aligned region. Or, the MSI-X structures may be placed in their own captive BARs.
• MSI-X has been placed in it own set of BARs: BAR4 and BAR5 (64-bit).
• Each PCIe Function of the BCM5718 family, that is Function0 and Function1, advertises availability of their
BAR4 and BAR5 as MSI-X BARs. This is done via the MSI-X Capability structure.
The MSI-X Capability structure is implemented inside the EP-RC core. It points to two structures that must be implemented inside a device: the MSI-X Table and a Pending Bit Array (PBA). There is also a Message Control
Register. The Capability structure is shown in
3116
Table 94: MSI-X Capability Structure
158
Message Control Reg Next Pointer
MSI-X Table Offset
PBA Offset
73
Capability ID
20
–
Table BIR
PBA BIR
The BIR bits point to the BAR registers that a device function uses to base the respective data structures. In the
BCM5718 family, both BIRs have a hard-wired value of 0x4, which implies BAR4 and BAR5.
Note: BAR4 and BAR5 must support 8/16- and 32-bit accesses from the host. However, unaligned
16/32-bit access support are not required.
Note: PCI 2.3 specification notes that “For all accesses to MSI-X Table and MSI-X PBA fields, software must use aligned full DWORD or aligned full QWORD transactions; otherwise, the result is undefined”.
MSI-X Data Structures
Full specification of MSI-X is available in PCI Specification rev. 2.3; it is not repeated here. The MSI-X Table hosted by BCM5718 family is described here:
• This structure is placed at offset 0x0 pointed by BAR4 and BAR5. The content of the MSI-X Table structure is shown in
• Depending on whether IOV mode is selected, the PCIe core advertises either a 17-entry MSI-X table or a
5-entry MSI-X table. The advertisement choice is made only once following POR and cannot be changed afterwards (see the Table Size field). Selection of Single-Vector/Multivector mode does not affect Table
Size.
• This table comprises multiple 4 DWORD-long entries and each entry corresponds to one MSI-X vector.
Thus, in the BCM5718 family, there is a maximum of 17 such entries.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 251
BCM5718 Programmer’s Guide MSI-X Plumbing
• Each entry consists of four fields:
– The Message Address High and Message Address Low fields points to a 64-bit Host Address where the corresponding vector message must be posted.
– Message Data contains a 32-bit vector data. Every time the BCM5718 family wants to send an interrupt message corresponding to this vector, it writes the value provided by this field into the address pointed by Message Address field(s).
– Host software is allowed to replicate the same physical host address into multiple entries; that amounts to interrupt vector aliasing. The BCM5718 family can handle such aliasing.
– Only one bit, bit[0] of the Vector Control field, is implemented by PCI/PCIe. This is the (per vector)
Mask bit. When this bit is 1, the device function must not send a corresponding interrupt vector message to the host. Instead, the function must set the corresponding Pending bit in the PBA. When this bit is 1 and a 0 is written to it, a device must schedule an interrupt vector in case one was already
“Pending”.
• The PBA Structure in the BCM5718 family is only 4 DWORDs or 128 bits wide, out of which bits [16:0] are useful, while bits [127:17] are reserved for future use. Each PBA bit index corresponds to the respective
MSI-X vector# — that is why only 17 bits are implemented in the BCM5718 family. The PBA is placed at the
offset 0x120 relative to the addresses pointed by BAR4 and BAR5. See Table 95
.
• There is room for expansion of up to 18 MSI-X vectors in future NetXtreme controllers.
• In the Message Control Register, only three fields are important to the respective MAC Core:
– MSI-X Enable: This is the feature enable/disable bit. The MAC Core must dynamically snoop CFG writes to this bit in order to determine if MSI-X gets enabled or not. When MSI-X is enabled, Line Interrupt
Message and MSI Message must be gated off by MAC Core. (Though PCIe allows host software to enable MSI and MSI-X concurrently, albeit erratically, MSI is preempted in such a scenario.)
– Function Mask: This bit acts like a device function-wide vector mask. When this bit is 1, all vectors in the function, i.e., 0 through 16 in the BCM5718 family must be masked. If any interrupt vector event occurs while Function Mask is 1, the corresponding Pending bit in PBA must be set. When this bit is 0, the per-vector Mask bits found in each Table Entry determine whether a vector is masked or not.
– Table Size: This field may declare a value of either 5 or 17. This is accomplished by Boot Code programming the appropriate Private Register of the PCIe core.
depicts the implemented address regions of BAR4 and BAR5 in the BCM5718 family. Only two structures are present: the MSI-X Table and the Pending Bit Array in the offsets shown in the table. Any host accesses to the non-implemented addresses in these BARs must be gracefully handled by the controller; i.e.,
Reads return 0x0 and Writes have no effect whatsoever on the MAC.
Table 95: MSI-X Table and PBA Structures in BCM5718 Family
MSI-X Table
Entry#
DW3 Content
(32-bit)
DW2 Content
(32-bit)
DW1 Content
(32-bit)
DW0 Content
(32-bit)
BAR4 and BAR5
Offset
N/A Reserved Bits [127:17] (PBA) Pending Bits[16:0]
Reserved Entry ………….
………….
………….
………….
16
15
……
……
Vec#16 Control
Vec#15 Control
……
……
Msg#16 Data
Msg#15 Data
……
……
Msg#16 Addr H
Msg#15 Addr H
……
……
Msg#16 Addr L
Msg#15 Addr L
……
……
0x120
0x110
0x100
0xF0
……
……
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 252
BCM5718 Programmer’s Guide MSI-X Plumbing
2
1
4
3
0
MSI-X Table
Entry#
Table 95: MSI-X Table and PBA Structures in BCM5718 Family (Cont.)
DW3 Content
(32-bit)
DW2 Content
(32-bit)
Vec#4 Control Msg#4 Data
Vec#3 Control Msg#3 Data
Vec#2 Control Msg#2 Data
Vec#1 Control Msg#1 Data
Vec#0 Control Msg#0 Data
DW1 Content
(32-bit)
DW0 Content
(32-bit)
BAR4 and BAR5
Offset
Msg#4 Addr H Msg#4 Addr L 0x40
Msg#3 Addr H Msg#3 Addr L 0x30
Msg#2 Addr H Msg#2 Addr L 0x20
Msg#1 Addr H Msg#1 Addr L 0x10
Msg#0 Addr H Msg#0 Addr L 0x00
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 253
BCM5718 Programmer’s Guide MSI-X Plumbing
MSI-X Cognizant Host Coalescing
After sending a complete Transmit packet to the wire, the controller updates the Send BD Ring consumer index internally. Similarly, after moving each full Received packet to the host memory, the controller updates its internal copy of the Rx Return Ring producer index. The controller notifies the host software or the network stack of such consumer and producer index updates by DMAing the Status Block and then sending an interrupt to the Host CPU.
Host software does not necessarily need to get indication for every packet received from the network, nor does it need to get completion for every packet sent to the wire. Heavy interrupt processing would degrade host CPU performance. However, the stack must communicate with application processes in a timely manner to keep up with network bandwidth demanded by the application. Thus, host software must have control in regulating —
however coarsely — when and how often it gets interrupts from the controller. The controller, in turn, accumulates update events. When the accumulation reaches a threshold value, as configured by the host software, the controller sends an interrupt. This scheme is known as interrupt coalescing or host coalescing.
Legacy NetXtreme architecture already offers a set of Host Coalescing (HC) Parameters.
Legacy Host Coalescing Parameters
This section lists the legacy set of host coalescing parameters that NetXtreme already offered prior to BCM5718 family.
Receive Coalescing Ticks Register (Offset: 0x3c08)
The value in this register can be used to control how often the status block is updated (and how often interrupts are generated) due to receiving packets. The value in this register controls how many ticks, in units of 1 µs each, get loaded in an internal receive tick timer register. The timer is reset to the value of this register and starts counting down after every status block update (regardless of the reason for the status block update). The timer is reset only after status block updates, and is not reset after any given packet is received. When the timer reaches 0, it is considered to be in the expired state. When the counter is in the expired state, a status block update will occur if a packet had been received and copied to host memory (via DMA) since the last status block update.
This register must be initialized by host software. A value of 0 in this register disables the receive tick coalescing logic. In this case, status block updates occur for receive events only if the Receive Max Coalesced BD value is reached. Status block updates for other reasons (e.g., transmit events) also include any updates to the receive indices. By setting the value in this register to a high number, a software device driver can reduce the number of status block updates and interrupts that occur due to receiving packets. This generally increases performance in hosts that are under a high degree of stress and whose RISCs are saturated due to handling a large number of interrupts from the network controller. For host environments where receive interrupt latency must be very low, and the host is not close to saturation, it is recommended that this register be set to 1.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 254
BCM5718 Programmer’s Guide MSI-X Plumbing
Send Coalescing Ticks Register (Offset: 0x3c0c)
The value in this register can be used to control how often the status block is updated (and how often interrupts are generated) according to the completion of transmit events. The value in this register controls how many ticks, in units of 1 µs each, get loaded in an internal transmit tick timer register. The timer is reset to the value of this register and starts counting down after every status block update (regardless of the reason for the status block update). The timer is reset only after status block updates, and is not reset after a transmit event completes. When the timer reaches 0, it is considered to be in the expired state. When the counter is in the expired state, a status block update occurs if a transmit event has occurred since the last status block update.
In this case, a transmit event is defined by an update to one of the device's Send BD Consumer Indices. A Send
Consumer Index increments whenever the data associated with a particular packet has been successfully moved (via DMA) across the bus, rather than when the packet is actually transmitted over the Ethernet wire.
This register must be initialized by host software. A value of 0 in this register disables the transmit tick coalescing logic. In this case, status block updates occur for transmit events only if the Send Max Coalesced BD value is reached, or if the BD_FLAG_COAL_NOW bit is set in a send BD. Status block updates for other reasons
(e.g., receive events) also include any updates to the send indices. By setting the value in this register to a high number, a software device driver can reduce the number of status block updates and interrupts that occur due to transmit completions. This generally increases performance in hosts that do not require their send buffers to be freed quickly. For host environments that do require their send buffers to be recovered quickly, it is recommended that this register be set to 0.
Receive Max Coalesced Bd Count Register (Offset: 0x3c10)
This register contains the maximum number of receive return ring BDs that must filled in by the device before the device updates the status block due to a receive event. Whenever the device completes the reception of a packet, it fills in a receive return ring BD, and then increments an internal receive coalesce BD counter. When this internal counter reaches the value in this register, a status block update occurs. This counter is reset (i.e., zeroed) whenever a status block update occurs regardless of the reason for the status block update. This register must be initialized by host software. A value of 0 in this register disables the receive max BD coalescing logic. In this case, status block updates occur for receive packets only via the Receive Coalescing Ticks mechanism. Status block updates for other reasons (e.g., transmit events) also include any updates to the receive indices. For simplicity, if a host wanted to get a status block update for every received packet, the host driver should just set this register to a value of 1. On the other hand, by setting the value in this register to a high number, a software device driver can reduce the number of status block updates and interrupts that occur due to receiving packets. This can increase performance in hosts that are under a high degree of stress and whose RISCs are saturated due to handling a large number of interrupts from the network controller. However, in lower traffic environments, there is no guarantee that consecutive packets will be received in a timely manner. Therefore, for those environments, it is recommended that the Receive Coalescing Ticks register are used to make sure that status block updates due to receiving packets are not delayed for an infinite amount of time.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 255
BCM5718 Programmer’s Guide MSI-X Plumbing
Send Max Coalesced BD Count Register (Offset: 0x3c14)
This register contains the maximum number of send BDs that must be processed by the device before the device updates the status block due to the transmission of packets. Whenever the device completes the DMA of transmit packet buffer, it increments an internal send coalesce BD counter. When this internal counter reaches the value in this register, a status block update occurs. This counter is reset (i.e., zeroed) whenever a status block update occurs, regardless of the reason for the status block update. This register must be initialized by host software. A value of 0 in this register disables the send max BD coalescing logic. In this case, status block updates occur for receive packets only via the Send Coalescing Ticks mechanism. Status block updates for other reasons (e.g., receive events) also include any updates to the send indices. For simplicity, if a host wants to get a status block update for every transmitted packet, the host driver could set this register to a value of 1.
However, by setting the value in this register to a high number, a software device driver can reduce the number of status block updates and interrupts that occur due to transmitting packets. This can increase performance in hosts that are under a high degree of stress and whose RISCs are saturated due to handling a large number of interrupts from the network controller. However, in lower traffic environments, there is no guarantee that consecutive packets will be transmitted in a timely manner. Therefore, for those environments, it is recommended that the Send Coalescing Ticks register is used to ensure that status block updates due to transmitting packets are not delayed for an infinite amount of time.
Receive Max Coalesced BD Count During Interrupt Register (Offset 0x3c18)
This register has the same attribute as 0x3C10 except that this parameter is active only during the During
Interrupt state, which is the state during which the ISR has acknowledged an interrupt by writing a nonzero value to the MailBox register and thus the interrupt is in a masked state. If this parameter triggers (while in
During Interrupt state), the controller will DMA the latest Status Block to the host memory, but the interrupt will remain deasserted.
Send Max Coalesced BD Count During Interrupt Register (Offset 0x3c1c)
This register has the same attribute as 0x3C14 except that this parameter is active only during the During
Interrupt state, which is the state during which the ISR has acknowledged an interrupt by writing a nonzero value to the MailBox register and thus the interrupt is in a masked state. If this parameter triggers (while in
During Interrupt state), the controller DMAs the latest Status Block to the host memory, but the interrupt remains deasserted.
BCM5718 Family Host Coalescing Parameter Sets
The legacy HC Parameter registers do not offer granularity in terms of individual Rx Return queues or individual
Tx queues. Instead, in the legacy implementation, all Transmit/Return Rings are grouped together for metering.
The BCM5718 family offers HC parameters or control on a per-Rx-and-Tx-queue basis when and only when
MSI-X Multivector mode is chosen. To that end, 16 more sets of Host Coalescing Parameter registers are added.
Each such HC Parameter Set comprises of the following registers:
• Receive [n] Coalescing Ticks Register
• Send [n] Coalescing Ticks Register
• Receive [n] Max Coalesced BD Count Register
• Send [n] Max Coalesced BD Count Register
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 256
BCM5718 Programmer’s Guide MSI-X Plumbing
• Receive [n] Max Coalesced BD Count During Interrupt Register
• Send [n] Max Coalesced BD Count During Interrupt Register
Where, n ranges from 1 through 16. The legacy HC Parameter registers are now called HC Parameter Set [0].
Each of these new sets have the same behavior or attribute as defined in
“Receive Coalescing Ticks Register
and “Receive Max Coalesced Bd Count Register (Offset: 0x3c10)” on page 255
, except one aspect: when MSI-X Multivector mode is enabled, each of these sets associate with its respective
Status-Block[n]. However, when either MSI-X is disabled or MSI-X Single-Vector mode is enabled, none of these additional sets would exist. Note that, when MSI-X Single-Vector mode is enabled, even though 5 (or 17) vectors are advertised, only Vector#0 remains active.
Note: To further clarify, in Legacy INTx mode, MSI mode, or MSI-X Single Vector mode, all Transmit and Receive queues are metered collectively by HC Parameter Set [0]. Parameter Sets [1–16] do not exist. Only in Multivector MSI-X mode do HC Parameter Sets [1–16] come into existence.
RCTR[0]
SCTR[0]
RMCBCR[0]
SMCBCR[0]
RMCBCDIR[0]
SMCBCDIR[0]
RCTR[1]
SCTR[1]
RMCBCR[1]
SMCBCR[1]
RMCBCDIR[1]
SMCBCDIR[1]
RCTR[2]
SCTR[2]
RMCBCR[2]
SMCBCR[2]
RMCBCDIR[2]
SMCBCDIR[2]
summarizes the existence of HC Parameter Sets, and their association to Status-Blocks. An “----” indicates that this register does not exist in this particular mode.
Valid
HC
Parameter
Register
Set
Table 96: MSI-X Host Coalescing Parameters
Invokes
Status
Block#
IOV-Mode +
Multiple TXQ
(Netqueue/ VMQ+TSS)
HC
Parameter
Registers
Address
Indication
Items
0
1
2
0x3C08
------
0x3C10
------
0x3C18
------
0x3D80
0x3D84
0x3D88
0x3D8C
0x3D90
0x3D94
0x3D98
0x3D9C
0x3DA0
0x3DA4
0x3DA8
0x3DAC
VRQ 0
LinkStat
Errors
VRQ Map
VRQ 1
TXQ 1
VRQ 2
TXQ 2
IOV-Mode +
Single TXQ
(VMQ)
HC
Parameter
Registers
Address
Indication
Items
0x3C08
0x3C0C
0x3C10
0x3C14
0x3C20
0x3C24
0x3D80
-----
0x3D88
-----
0x3D90
-----
0x3D98
----
0x3DA0
-----
0x3DA8
-----
VRQ 0
TX
LinkStat
Errors
VRQ Map
VRQ 1
VRQ 2
RSS-Mode +
Multiple TXQ
(TSS)
HC
Parameter
Registers
Address
Indication
Items
-----
-----
-----
-----
-----
-----
0x3D80
0x3D84
0x3D88
0x3D8C
0x3D90
0x3D94
0x3D98
0x3D9C
0x3DA0
0x3DA4
0x3DA8
0x3DAC
LinkStat
Errors
RSS 0
TXQ 1
RSS 1
TXQ 2
RSS-Mode +
Single TXQ
(Legacy)
HC
Parameter
Registers
Address
Indication
Item
-----
0x3C0C
-----
0x3C14
-----
0x3C24
0x3D80
-----
0x3D88
-----
0x3D90
-----
0x3D98
----
0x3DA0
-----
0x3DA8
-----
TX
LinkStat
Errors
RSS 0
RSS 1
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 257
BCM5718 Programmer’s Guide MSI-X Plumbing
Valid
HC
Parameter
Register
Set
RCTR[3]
SCTR[3]
RMCBCR[3]
SMCBCR[3]
RMCBCDIR[3]
SMCBCDIR[3]
RCTR[4]
SCTR[4]
RMCBCR[4]
SMCBCR[4]
RMCBCDIR[4]
SMCBCDIR[4]
RCTR[5]
SCTR[5]
RMCBCR[5]
SMCBCR[5]
RMCBCDIR[5]
SMCBCDIR[5]
RCTR[6]
SCTR[6]
RMCBCR[6]
SMCBCR[6]
RMCBCDIR[6]
SMCBCDIR[6]
…….
…….
RCTR[16]
SCTR[16]
RMCBCR[16]
SMCBCR[16]
RMCBCDIR[16]
SMCBCDIR[16]
Table 96: MSI-X Host Coalescing Parameters (Cont.)
IOV-Mode +
Multiple TXQ
Invokes
Status
Block#
(Netqueue/ VMQ+TSS)
HC
Parameter
Registers
Address
Indication
Items
3
4
5
6
…….
…….
0x3DE0
0x3DE4
0x3DE8
0x3DEC
0x3DF0
0x3DF4
0x3EF8
0x3EFC
0x3E00
0x3E04
0x3E08
0x3E0C
…….
…….
0x3DB0
0x3DB4
0x3DB8
0x3DBC
0x3DC0
0x3DC4
0x3DC8
0x3DCC
0x3DD0
0x3DD4
0x3DD8
0x3DDC
VRQ 3
TXQ 3
VRQ 4
TXQ 4
VRQ 5
TXQ 5
VRQ 6
TXQ 6
…….
…….
IOV-Mode +
Single TXQ
(VMQ)
HC
Parameter
Registers
Address
Indication
Items
0x3DE0
-----
0x3DE8
-----
0x3DE0
-----
0x3DE8
-------
0x3DF0
-------
0x3DF8
-------
…….
…….
0x3DB0
-----
0x3DB8
-----
0x3DC0
-----
0X3DC8
-----
0x3DD0
-----
0x3DD8
-----
VRQ 3
VRQ 4
VRQ 5
VRQ 6
…….
…….
16 0x3EE8
0x3EEC
0x3EF0
0x3EF4
0x3EF8
0x3EFC
VRQ 16
TXQ 16
0x3EE8
-------
0x3EF0
-------
0x3EF8
-------
VRQ 16
RSS-Mode +
Multiple TXQ
(TSS)
HC
Parameter
Registers
Address
Indication
Items
0x3DB0
0x3DB4
0x3DB8
0x3DBC
0x3DC0
0x3DC4
0x3DC8
0x3DCC
0x3DD0
0x3DD4
0x3DD8
0x3DDC
RSS 2
TXQ 3
RSS 3
TXQ 4
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
RSS-Mode +
Single TXQ
(Legacy)
HC
Parameter
Registers
Address
Indication
Item
0x3DB0
-----
0x3DB8
-----
0x3DC0
-----
0X3DC8
-----
0x3DD0
-----
0x3DD8
-----
RSS 2
RSS 3
N/A
N/A
N/A
N/A
N/A
N/A
N/A
N/A
Abbreviations:
RCTR[n]
SCTR[n]
RMCBCR[n]
SMCBCR[n]
RMCBCDIR[n]
RECEIVE COALESCING TICKS REGISTER[n]
SEND COALESCING TICKS REGISTER[n]
RECEIVE MAX COALESCED BD COUNT REGISTER[n]
SEND MAX COALESCED BD COUNT REGISTER[n]
RECEIVE MAX COALESCED BD COUNT DURING INTERRUPT REGISTER[n]
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 258
BCM5718 Programmer’s Guide MSI-X Plumbing
SMCBCDIR[n]
0x55AA
SEND MAX COALESCED BD COUNT DURING INTERRUPT REGISTER[n]
New Registers in BCM5718 family (legend)
MSI-X One Shot Mode
The BCM5718 family introduces a new method of MSI-X acknowledgement known as the One Shot mode.
When this mode is set, an ISR is asked to skip the interrupt acknowledgement step in which it would otherwise write a nonzero value to the respective INT MailBox. This is sensible because MSI-X vector messages are equivalent to edge-triggered non-shared interrupt events; therefore, there is no need for ISR to explicitly acknowledge the event.
One Shot mode is enabled by default and could be disabled by writing a 1 to the register bit 0x6000[5]. the One
Shot mode setting has no effect on the Line Interrupt or MSI modes.
The controller hardware stores a nonzero value to an INT Mailbox as soon as a respective MSI-X Message DMA is completed at the EP-RC (PCIe Core) interface.
Coalesce Now or Forced Update
There is a Coalesce Now bit in the legacy Host Coalescing block, 0x3C00[3]. If set, the Host Coalescing block updates the Status Block immediately and sends an interrupt to host. This bit is self-clearing.
In the BCM5718 family, this bit retains the same functionality and associates with Status-Block0 and Vector#0 when MSI-X is enabled. Moreover, 16 additional Coalesce Now bits replicate the same function associated to vector numbers 1 through 16. Below are the definitions of the bits.
• 0x3C00[3]: Coalesce Now (When INTx or MSI Enabled)
• 0x3C00[3]: Coalesce vector#0 Now (When MSI-X Enabled)
• 0x3C00[13]: Coalesce vector#1 Now (When MSI-X Enabled and Multivector mode Enabled)
• 0x3C00[14]: Coalesce vector#2 Now (When MSI-X Enabled and Multivector mode Enabled)
• 0x3C00[15]: Coalesce vector#3 Now (When MSI-X Enabled and Multivector mode Enabled)
• 0x3C00[16]: Coalesce vector#4 Now (When MSI-X Enabled and Multivector mode Enabled)
• 0x3C00[17]: Coalesce vector#5 Now (When MSI-X Enabled and Multivector mode Enabled)
• ………
• 0x3C00[28]: Coalesce vector#16 Now (When MSI-X Enabled and Multivector mode Enabled)
Misc Coalescing Controls
There are a few Host Coalescing controls in the legacy NetXtreme design in the HOST COALESCING MODE
REGISTER (0x3C00) and HOST CONTROL REGISTER (0x68). Some of these controls apply equally to the newly added HC parameters or MSI-X feature in general, and some do not apply equally. Such controls are listed here for clarity:
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 259
BCM5718 Programmer’s Guide MSI-X Plumbing
Broadcom Tagged Status Mode (0x68[9])
Enabled by setting the Status Tagged Status Mode bit of the Miscellaneous Host Control register. When enabled, a unique eight-bit tag value is inserted into the Status Block Status Tag at location 7:0. The Status Tag can be returned to the Mailbox 0 register at location 31:24 by the host driver. When the Mailbox 0 register field
23:0 is written with a zero value, the tag field of the Mailbox 0 register is compared with the tag field of the last
Status Block to be DMAed to the host. If the tag returned is not equivalent to the tag of the first Status Block
DMAed, the interrupt status is entered. This bit, 0x68[7], applies to all 17 MSI-X vectors.
Clear Interrupt, Mask Interrupt, Mask Mode (0x68[0], 0x68[1], 0x68[8])
These bits have no effect on MSI-X operations.
Clear Ticks On Rx Bd Events Mode (0x3c00[9])
Enabled by setting the Clear Ticks mode on Rx bit of the Host Coalescing Mode register. When enabled, the counters initialize to the idle state and begin counting only after a receive BD event is detected. This register bit also applies to all newly created RCTR Registers.
No Interrupt On Force Update (0x3c00[11])
Enabled by setting the No Interrupt on Force bit of the Host Coalescing Mode register. After enabling this bit, subsequent writes to the Coalesce Now bit(s) of the Host Coalescing Mode register cause status block update(s) without the corresponding interrupt event. This bit applies to all MSI-X vectors and respective Status
Blocks.
No Interrupt On DMAD Force (0x3c00[12])
Enabled by setting the No Interrupt on DMAD force bit of the Host Coalescing Mode register. When enabled, the BD_FLAG_COAL_NOW bit of the buffer descriptor may be set to force a status block update without a corresponding interrupt. This feature is associated to Send BDs only; hence, it applies to Vector#0 in MSI-X mode when Multiple Send Queues are not enabled.
Register Transfer Level (RTL) Note: The HC RTL honors the Coal_Now Flag coming from both Send or Receive flow-through queues (FTQs). Rx FTQ never requests it, as the RBDs do not support any such flag.
Do Not Interrupt On Receives (0x6800[14])
If set, an interrupt is not generated upon a Receive Return Ring producer update. This bit applies equally to vector#0 through vector#16 in Multivector mode.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 260
BCM5718 Programmer’s Guide MSI-X Plumbing
End of Receive Stream Interrupt
A new kind of forced interrupt is being introduced in the BCM5718 family. The End of Rx Stream Interrupt attempts to sense the end of a receive burst and, if it does, it fires an interrupt/MSI-X instantaneously.
After completing the DMA of every Return BD to the host memory, a hardware FSM checks if the Rx-MBUF is empty (discounting the effects of pre-allocation). If it is empty, hardware starts counting down a count value.
While the countdown is in progress, if another Rx packet starts to pour into the Rx-MBUF, the FSM goes back to idle. However if no other Rx packet arrives, it allows the counter to go down to zero, at which point the FSM triggers an interrupt/MSI-X. The counter basically debounces effects of IPG or short gaps among packets within a burst.
This feature may be enabled or disabled by a register bit. The countdown preload value is also programmable.
When enabled in conjunction with Multivector MSI-X mode, there is a programmable option to fire either
Vector#0 only or all Vectors.
Host Coalescing Mode Register (Offset 0x3c00)
Access
Default
Value Name
As defined in Legacy
End of Rx Stream Detector
Fires ALL MSI-X Vectors
Bits
31
30 RW 0x0
Enable End of Rx Stream
Interrupt
Coalesce Now MSI-X Vector#
[16–1]
29
28:13
As defined in Legacy 12:0
RW
WC
0x0
0x0
DESCRIPTION
Write 1 to fire ALL MSI-X Vectors when an
End of Rx Stream is detected.
Write 0 to fire only MSI-X Vector#0 when an End of Rx Stream is detected.
Write 1 to enable the End of Rx Stream
Interrupt
Individual Coalesce Now bits associated with MSI-X Vector# 16 through 1.
These bits are self-clearing.
End Stream Debounce Register (Offset 0x3cd4)
Access
Default
Value Name
As defined in Legacy
Reserved
Bits
31
30:16 RO 0 –
DESCRIPTION
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 261
BCM5718 Programmer’s Guide
Name Bits
End of Rx Stream Debounce
Count
15:0
MSI-X Plumbing
Access
RW
Default
Value
0x000F
DESCRIPTION
This field is meaningful only when
0x3C00[29] is 1.
After completing the DMA of every Return
BD to the Host memory, a hardware FSM checks if the Rx-MBUF is empty
(discounting the effects of pre-allocation).
If it is, hardware starts counting down a count value programmed by this field.
While the count down is in progress, if another Rx packet starts to pour into the
Rx-MBUF, the FSM goes back to idle.
However if no other Rx packet arrives, it allows the counter to go down to zero, at which point the FSM triggers an interrupt/
MSI-X. The counter basically de-bounces effects of IPG or short gaps among packets within a burst.
The counter counts in Core-Clocks.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 262
BCM5718 Programmer’s Guide Other Configuration Controls
Other Configuration Controls
Broadcom Mask Mode
Enabled by setting the Mask_Interrupt_Mode bit (bit 8) of the Miscellaneous Host Control register (see
Miscellaneous Host Control register will mask (deassert) the INTA signal at the pin, but it will not clear the interrupt state and it will not latch the INTA value. Clearing the mask bit will enable the interrupt state to propagate to the INTA signal. Note that the During Interrupt Coalescence registers are only used when the
Mailbox 0 is set.
Broadcom Tagged Status Mode
Enabled by setting the Status Tagged Status mode bit of the Miscellaneous Host Control register (see
“Miscellaneous Host Control Register (offset: 0x68)” on page 283
). When enabled, a unique eight-bit tag value will be inserted into the Status Block Status Tag at location 7:0. The Status Tag can be returned to the Mailbox
0 register at location 31:24 by the host driver. When the Mailbox 0 register field 23:0 is written with a zero value, the tag field of the Mailbox 0 register is compared with the tag field of the last Status Block to be DMAed to the host. If the tag returned is not equivalent to the tag of the first Status Block DMAed, the interrupt status is entered.
Clear Ticks on BD Events Mode
Enabled by setting the Clear Ticks mode on RX or the Clear Ticks mode on TX bits of the Host Coalescing Mode register (see
“Miscellaneous Host Control Register (offset: 0x68)” on page 283 ). When enabled, the counters
initialize to the idle state and begin counting only after a receive or transmit BD event is detected.
No Interrupt on Force Update
Mode Register (offset: 0x3C00)” on page 415
). When enabled, writing the Force update bit of the Host
Coalescing Mode register will cause a status block update without a corresponding interrupt event.
No Interrupt on DMAD Force
buffer descriptor may be set to force a status block update without a corresponding interrupt.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 263
BCM5718 Programmer’s Guide IO Virtualization (IOV)
Section 12: IO Virtualization (IOV)
IO Virtualization or IOV is a solution to transparently share a physical IO device among the child/guest OS images in a virtualized environment. The NetXtreme (BCM5718/19/20) offers a cost efficient set of IO
Virtualization features. The following list provides the feature highlights:
• Receive Side:
• RX Traffic sorting over multiple RX queues
• Hardware assisted RX Packet replication in Driver
• Universal VLAN stripping (Not per queue based)
• 24x Perfect Match Addresses Filters
• One 128-bit MultiCast Addresses Hash Filter
• 32-Element/31-Set Programmable Protocol Filter (VLAN, TCP etc)
• 16 Receive Queues + 1 Default Queue + 1 Drop Queue
• 17 Standard Receive Producer Rings
• 17 Jumbo Receive Producer Rings
• 17 Return Rings
• Per Queue synchronization with Driver
• RX Packet Header Data Split and copy - for VMQ
Note: No RSS support in IOV Mode.
• Transmit Side:
• 16 Transmit Queues for use by NetQueue or NDIS-TSS
• Per Frame Round Robin and Weighted RR scheduling in Hardware
• General:
• IOV Mode is a static configuration
• 18 MSI-X vectors - 1 per TX/RX Queue Pair
• Per Queue nominal Statistics in Hardware
• No RX Bandwidth Limitation or TX Traffic/Rate shaping
Note: IOV shall operate in a distinct and a static chip-mode. This mode is called the IOV Mode and the chip would need to be configured during boot-up. Once configured, the mode is irreversible unless a hard-reset is asserted to the chip. During boot-up, if the IOV-Mode is not chosen, the chip will operate in the Legacy Mode which is akin to the operation of previous NetXtreme devices.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 264
BCM5718 Programmer’s Guide Data Structure and Register Changes for IOV
Data Structure and Register Changes for IOV
The following changes have been made to support IOV:
Mail Box Register Changes
The following mailbox register changes have been made:
• 16 more Standard RBD Producer Index Mail Boxes have been added.
• 16 more Jumbo RBD Producer Index Mail Boxes have been added.
• 13 more Receive Return Ring Consumer Index Mail Boxes have been added.
• 15 more Host Send Producer Index Mail Boxes have been added.
Receive Mail Box Register Changes
16 Standard RBD Producers, 16 Jumbo RBD Producers, and 13 Receive Return Ring Consumer mail boxes have
been added to the High Priority Mail Box region (see “RX Mail Box Registers for VRQ” on page 461
).
Send Mail Box Register Changes
15 Host Send Producer Index registers have been added to the High Priority Mail Box region (see
).
Ring Control Block Changes
The following ring control block changes have been made:
• 16 more Standard Producer RCBs added
• 16 more Jumbo Producer RCBs added
• 13 more Return Ring RCBs added
• 15 more Send RCBs added
Receive Ring Control Blocks (see “Receive Ring Control Blocks” on page 73 ).
Send Ring Control Blocks (see
“Send Ring Control Blocks” on page 73 ).
VRQ Statistics
• VRQ Receive Statistics
• VRQ Transmit Statistics
EMAC collects basic statistics on an individual generic VRQ basis. Though the accumulation mechanism remains the same, i.e., these registers are Clear-On-Read, these statistic registers are independent of the
aggregated EMAC statistics registers for LAN and APE (see “VRQ Statistics” on page 457 ).
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 265
BCM5718 Programmer’s Guide Data Structure and Register Changes for IOV
MSI-X Vectors Changes
The NetXtreme I offers two vector modes within the MSI-X mode for IOV:
• Single Vector IOV mode (Restrict to Vector#0)
• Multivector IOV mode (17 Vectors Requested)
For more information, see
Register Changes
The following register changes have been made:
• GRC-MODE-REG (Offset 0x6800): Additional bits have been added to this existing register (see “Mode
Control Register (offset: 0x6800)” on page 472 ).
Data and Receive BD Initiator Mode Register (offset: 0x2400)” on page 364
).
• Standard Replenish LWM Register (Offset: 0x2D00): This register is meaningful in both Legacy and IOV
Modes (see “Standard Replenish LWM Register (offset 0x2D00)” on page 375 ).
• Jumbo Replenish LWM Register (Offset 0x2D04): This register is meaningful in both Legacy and IOV Modes
(see
“Jumbo Replenish LWM Register (offset 0x2D04)” on page 376 ).
Limit Register (Offset 0x2D08)” on page 377
).
Status Register (offset: 0x240C)” on page 366 ).
• VRQ Flush Control Register (Offset: 0x2410): Additional bits have been added to this existing register (see
“VRQ Flush Control Register (Offset: 0x2410)” on page 366 ).
• VRQ Flush Timer Register (Offset: 0x2414): The description has been updated on this existing register (see
“VRQ Flush Timer Register (offset: 0x2414)” on page 367 ).
• HC Parameter Set Reset Register (Offset: 0x3C28): This parameter should be placed in this HC block (see
“HC Parameter Set Reset Register (Offset: 0x3C28)” on page 423 ).
• Perfect Match Destination Address Registers: Twenty additional Perfect Match Destination Address
Registers are added in RX-EMAC for VRQ Filtering purposes (see
“Perfect Match Destination Address
).
• SEND_BD_INITIATOR_MODE_REG (Offset 0x1800) Additional bits have been added to this existing register
(see
“Send BD Initiator Mode Register (offset: 0x1800)” on page 355 ).
• SEND_BD_FETCH_THRESHOLD_REG (Offset: 0x1850) This is a newly added register (see “Send BD Fetch
Threshold Register (offset: 0x1850)” on page 357 ).
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 266
BCM5718 Programmer’s Guide IOV - Receive Side
IOV - Receive Side
The chosen Receive-Side solution to IOV acceleration is to scale into multiple Receive Indication Queues. These queues are called Virtual Receive Queues or VRQ as defined for NetXtreme internal use. VRQ generally refers to an implementation of both VMQ as well as NetQueue. Each VRQ is provided with dedicated host buffer memory resources. The reason behind this requirement involves memory protection issues with the host platform and OS. Each such queue is also provided with programmable RX traffic filter resources called VRQ
Filters. The task of RX-EMAC is to sort the RX traffic designated for multiple VRQs based on VRQ Filter settings.
RX-EMAC marks each RX packet with a VRQ number after placing it into the RX-MBUF. Before initiating a RX packet DMA To the host memory, the DMA engine shall inspect each packet's Frame-Header to determine which VRQ it belongs to. It then draws Receive Buffers only from the respective VRQ's host buffer resources and subsequently proceeds to DMA the packet to those buffers. After placing the frame in a VRQ designated host buffer, the controller designates the RX packet selectively to the respective owner (a child/guest OS) of the VRQ and not to any other OS images. Therefore, it implies that each VRQ shall also be associated with an independent receive indication queue, which is nothing but an independent Receive Return Ring (refer to
for additional information.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 267
BCM5718 Programmer’s Guide
Post
RX
Buffers
Jumbo RX BD
Producer Rings x17
Standard RX BD
Producer Rings x17
Figure 57: IOV Receive Flow
Host Memory
RX Buffers
IOV - Transmit Side
Return
BD Rings x17 sBD Prod Index
Ring #00 sBD Prod Index
Ring #16 jBD Prod Index
Ring #00 jBD Prod Index
Ring #16
Producer
MailBoxes
17 + 17
RBD Fetch
Req
RDMA
Engine
RBD
Data
Per
Queue
RBD
Cache
Frame
Header VRQ #
PCIe Link
o f
Ch ain
M
BU
Fs
RX-MBUF
Packet data
Write
Frame-Header
Return BD
Write
[ VRQ # ]
WDMA
Engine
MB
Poin ter luste r
Receive
Queue
Sawtooth
Chip
Per-Q
Stats
VRQ Filters
RX EMAC
RX
Wires
Drop-VRQ
(Discard
Packets )
IOV - Transmit Side
Transmit enhancement for IOV involves adding multiple Host Send Rings or effectively adding multiple transmit queues. At present, Feb 2009, only Netqueue is capable of utilizing multiple TX queues whilst VMQ is not.
Although, the Transmit Side Scaling (TSS) feature of NDIS 6.x is capable of using multiple transmit queues. This is the reason this feature, namely 16 Send Rings, could be enabled in BCM5718/19/20 irrespective of the IOV-
Mode settings.
BCM5718/19/20 shall implement a limited set of capabilities in this regard:
• A maximum of 16 Send Rings (SBD Rings)
• Multiple Send Rings could be enumerated only in conjunction to either IOV-Mode or RSS Mode
• All 16 Send Rings may be enabled in conjunction of IOV Mode
• Up to 4 Send Rings may be enabled in conjunction of RSS Mode
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 268
BCM5718 Programmer’s Guide IOV - Transmit Side
• Only Host based Rings supported (NIC based Rings de-featured early on)
• Maximum Ring size of 512 each
• 16 High Priority Send Producer Index Mailboxes - one per ring
• 16 Send Ring Control Blocks (RCB)
• 1KB of private on-chip SBD cache per Ring - Total 16KB
• Introduction of a SBD Fetch Threshold
• Send BD Format unchanged in Multi Ring mode.
• No Rate / Traffic Shaping algorithm offered
• Very basic round-robin packet by packet arbitration among 16 Send Rings
• All Send Offload features, namely LSO & Checksum-Offload shall continue to function in all rings without any behavior change.
• Minimal set of per Send Queue EMAC Statistics
The basic Send interface with the Device Driver remains unchanged, only the number of Send Rings are scaled up to 16 from 1. Thus there shall be 16x Send Producer BD Index Mailboxes and 16x Ring Control Block
Registers in controller.
SBDs are fetched from the Host memory and are stored in SBD-cache memory internal to the chip. Though such a cache memory could be shared by all 16 Rings, for the sake of simplicity we chose to assign each Send Ring a private partition of the SBD cache - physically it is a single SRAM, but is divided into 16 equal address regions.
Each such address range shall serve as a private SBD cache to a Send Ring. The total size of the SBD cache is thus 16 KB.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 269
BCM5718 Programmer’s Guide Ethernet Controller Register Definitions
Section 13: Ethernet Controller Register
Definitions
BCM5718 Family Register MAP
The BCM5718 family's internal register map is shown in
Block Name Block Range
PCIe CFG 0x0000–0x03FF
EMAC 0x0400–0x07FF
EMAC STAT
SDI
SDC 0x1000–0x13FF
SBDS
SBDI
SBDC
RQP
RDI
RDC
RBDI
RBDC
CMPU
0x0800–0x0BFF
0x0C00–0x0FFF
0x1400–0x17FF
0x1800–0x1BFF
0x1C00–0x1FFF
0x2000–0x23FF
0x2400–0x27FF
0x2800–0x2BFF
0x2C00–0x2FFF
0x3000–0x33FF
0x3400–0x37FF
Table 97: BCM5718 Family Register Map
Subblock Range
0x0000–0x03FF
0x0400–0x06FF
0x0700–0x7FF
0x0800–0x08FF
0x0900–0x0BFF
0x0C00–0x0CF7
0x0CF8–0x0FFF
0x1000–0x100B
0x100C–0x13FF
0x1400–0x147F
0x1480–0x17FF
0x1800–0x1847
0x1848–0x1BFF
0x1C00–0x1C03
0x1C04–0x1FFF
0x2000–0x2258
0x2259–0x23FF
0x2400–0x24C3
0x24C0–0x27FF
0x2800–0x2803
0x2804–0x2BFF
0x2C00–0x2C1B
0x2C1C–0x2FFF
0x3000–0x300F
0x3010–0x33FF
0x3400–0x35FF
0x3600–0x3687
Description
PCIe Configuration Register Shadow
EMAC
Unused
EMAC Statistics
Unused
Send Data Initiator
Unused
Send data Completion
Unused
SBDS Registers
Unused
Send BD Initiator
Unused
Send BD Completion
Unused
Receive List Placement
Unused
Receive Data Initiator
Unused
Receive Data Completion
Unused
Receive BD Initiator
Unused
Receive BD Completion
Unused
Central Power Management Unit
Unused
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 270
BCM5718 Programmer’s Guide BCM5718 Family Register MAP
Table 97: BCM5718 Family Register Map (Cont.)
Block Name Block Range
DBU 0x3800–0x3BFF
HC
MA
BM
DMAR
DMAW
RX-CPU
Open Block
MB
FTQ
MSI
CFG Port
0x3C00–0x3FFF
0x4000–0x43FF
0x4400–0x47FF
0x4800–0x4BFF
0x4C00–0x4FFF
0x5000–0x53FF
0x5400–0x57FF
0x5800–0x5BFF
0x5C00–0x5FFF
0x6000–0x63FF
0x6400–0x67FF
Subblock Range
0x3800–0x3817
0x3900–0x3907
0x3908–0x3BFF
0x3C00–0x3CC3
0x3CC4–0x3FFF
0x4000–0x400F
0x4010–0x43FF
0x4400–0x445B
0x445C–0x47FF
0x4800–0x4A13
0x4A14–0x4BFF
0x4C00–0x4C07
0x4C08–0x4FFF
0x5000–0x5037
0x5038–0x53FF
–
0x5800–0x5903
0x5903–0x5BFF
0x5C00–0x5CFF
0x5D00–0x5FFF
0x6000–0x6007
0x6008–0x63FF
0x6400–0x67FF
GRC
Rserved
NVM
UART
TL-DL-PL Port
0x6800–0x6BFF
0x6C00–0x6FFF
0x7000–0x73FF
0x7800–0x7BFF
0x7C00–0x7FFF
0x6800–0x681B
0x6834–0x6843
0x6890–0x68A8
0x68B4–0x68DF
0x6C00–6C37
0x6C38–0x6FFF
0x7000–0x703B
0x703C–0x73FF
0x7800–0x781F
0x7820–0x7BFF
0x7C00–0x7FFF
Description
Debug Unit (UART)
Chip Debug
Unused
Host Coalescing
Unused
Memory Arbiter
Unused
Buffer manager
Unused
DMA Read
Unused
DMA Write
Unused
RX CPU
Unused
–
Low Priority Mail Box
Unused
Flow Through Queue
Unused
Message Signaled Interrupt
Unused
PCIe Core Private Registers Access to
Configuration Space
Misc Host Control
SEEPROM
Misc Control
Unused
Unused
Unused
Non Volatile memory (Flash Controller)
Unused
Debug UART Modem
Unused
PCIe Core Private Register Access to TL, DL
& PL
******
APE 0x10000–0x18FFF
MF 0x19000–0x193FF Management Filters
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 271
BCM5718 Programmer’s Guide PCI Configuration Registers
PCI Configuration Registers
Device ID and Vendor ID Register (offset: 0x00)
This register is reset by hard Reset.
Name
Device ID
Vendor ID
Bits Access Default Value Description
31:16 RO
15:0 RO
–
0x14E4
Default for BCM
Default for BCM5718 (LAN Function 0): 0x1656
–
5717 (LAN Function 0): 0x1655
Status and Command Register (offset: 0x04)
This register is reset by PCIE Reset.
Name
Detected Parity
Error
Signaled System
Error
Bits
31
30
Received Master
Abort
Received Target
Abort
Signaled Target
Abort
29
28
27
DEVSEL Timing 26:25
Master Data Parity
Error
24
Fast Back-to-back capable
Reserved
23
22
66 MHz Capable 21
Capabilities List 20
Access
Default
Value
RW2C 0x0
RW2C
RW2C
RW2C
RW2C
RO
RW2C
RO
0x0
0x0
0x0
0x0
0x0
0x0
0x0
Description
When this bit is set, it indicates that the function has received a poisoned TLP
This bit is set when a function sends an ERR_FATAL or
ERR_NONFATAL message and the SERR enable bit in the command register is set
This bit is set when a requester receives a completion with UR completion status
This bit is set when a requester receives a completion with completer abort completion status.
This bit is set when a function acting as a completer terminates a request by issuing Completer abort completion status to the requester
Does not apply to PCIE
The master data parity error bit is set by a requester if the parity error enable bit is set in its command register and either of the following 2 conditions occur. If the requester receives a poisoned completion if the requester poisons a write request If the parity Error enable bit is cleared, the master data parity error status bit is never set
Does not apply to PCIE.
RO
RO
RO
0x0
0x0
0x1
These bits are reserved and tied low per the PCI specification.
Does not apply to PCIE
This bit is tied high to indicate that the device supports a capability list. The list starts at address 0x40.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 272
BCM5718 Programmer’s Guide PCI Configuration Registers
Name Bits
Interrupt Status 19
Reserved 18:16
Reserved 15:11
Interrupt Disable 10
Fast Back-to-back
Enable
System Error
Enable
Stepping Control
Memory Space
I/O Space
9
8
7
Parity Error Enable 6
VGA Palette Snoop 5
Memory Write and
Invalidate
Special Cycles
Bus Master
4
3
2
1
0
RO
RW
RO
RO
RO
RW
Access
RO
RO
RO
RW
RO
RW
RW
RO
0x0
0x0
0x0
0x0
Default
Value
0x0
0x0
0x00
0x0
0x0
Description
Indicates this device generated an interrupt
These bits are reserved and tied low per the PCIE specification.
These bits are reserved and tied low per the PCIE specification.
When this bit is set, function is not permitted to generate
IntX interrupt messages (deasserted) regardless of any internal chip logic. Setting this bit has no effect on the
INT_STATUS bit below. Writing this bit to 0 will un-mask the interrupt and let it run normally.
Does not apply to PCIE
0x0
0x0
0x0
0x0
0x0
When set, this bit enables the non fatal and fatal errors detected by the function to be reported to the Root
Complex. The function reports such errors to the Root
Complex if it is enabled to do so either through this bit or though PCI express specific bits in DCR
Does not apply to PCIE
This bit enables the write to the Master data parity error status bit. If this bit is cleared, the master data parity error status bit will never be set.
Does not apply to PCIE
Does not apply to PCIE
Does not apply to PCIE
This bit controls the enabling of the bus master activity by this device. When low, it disables an Endpoint function from issuing memory or IO requests. Also disables the ability to issue MSI messages.
This bit controls the enabling of the memory space. When disabled, memory transactions targeting this device return completion with UR status
This bit indicates that the device does not support I/O space access because it is zero and can not be modified.
IO transactions targeting this device return completion with UR status.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 273
BCM5718 Programmer’s Guide PCI Configuration Registers
PCI Classcode and Revision ID Register (offset: 0x08)
This register is reset by Hard Reset.
Name
PCI Classcode
Revision ID — Alllayer Revision ID
Bits
31:8
7:4
Revision ID —
Metal Revision ID
3:0
Access
RO
RO
FW-RW
Host-RO
Default
Value Description
0x020000 Default for (LAN Function 0): 0x020000
ASIC Rev
Input
This field will be updated automatically by hardware based on the External All Layer Revision ID. For example, this field will contain a value of 0x0 after hard reset for BCM5718 A0 silicon. Software shall use this field only to display the Device Silicon Revision ID for application where the user/customer needs to know the Device Silicon Revision ID. One such application is the B57DIAG Device Banner. Furthermore, Software
(Boot Code/Driver/B57DIAG) shall NOT use this field in determining Bug Fixes. It should only use the Internal
Revision ID, bits 31:24 and bits 19:16 from Register 68, for that purpose.
• 0x0 for A steps
• 0x1 for B steps
• 0x2 for C steps
ASIC Rev
Input
This field will be updated automatically by hardware based on the Metal Revision ID.
• 0x0 for metal 0 step
• 0x1 for metal 1 step
• 0x2 for metal 2 step
BIST, Header Type, Latency Timer, Cache Line Size Register (offset:
0x0C)
Name
BIST
Header Type
Latency Timer
Bits Access
31:24 RO
23:16 RO
15:8 RO
Default
Value
0x0
0x80
0x0
Description
The 8-bit BIST register is used to initiate and report the results of any Built-In-Self-Test. This value can be written by firmware through the PCI register space
BIST register to modify the read value to the host.
The 8-bit Header Type register identifies both the layout of bytes 10h through 3Fh of the Configuration space, as well as whether this adapter contains multiple functions. A value of 0x80 indicates a multifunction device (Type 0) using the format specified in the PCI specification, while a value of 0x0 indicates a single function Type 0 device.
This register does not apply to PCI express and must be hardwired to zero
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 274
BCM5718 Programmer’s Guide PCI Configuration Registers
Name Bits
Cache Line Size 7:0
Access
RO
Default
Value
0x0
Description
This field is implemented by PCIE device as a read/ write field for legacy compatibility purposes.
Base Address Register 1 (offset: 0x10)
Name
Address
Prefetch
Type
Space
Bits
31:4
3
2:1
0
Access
RW
Default
Value
0
RO
RO
RO
0x1
0x2
0
Description
These bits set the address within a 32-bit address space that will be card will respond in. These bits may be combined with the bits in BAR_2 to create a full 64 bit address decode. Only the bits that address blocks bigger than the setting in the
BAR1_SIZE value are RW. All lower bits are RO with a value of zero. This value is sticky and only reset by HARD Reset.
This bit indicates that the area mapped by BAR_1 may be pre-fetched or cached by the system without side effects. Bit can be programmed from shadow register. Path = i_cfg_func.i_cfg_private.
These bits indicate that BAR_1 may be programmed to map this adapter to anywhere in the 64-bit address space. Path = i_cfg_func.i_cfg_private.
This bit indicates that BAR_1 maps a memory space and is always read as 0. Path = i_cfg_func.i_cfg_private.
Base Address Register 2 (offset: 0x14)
Name
Address
Bits
31:0
Access
RW
Default
Value Description
0 These bits set the address upper 32-bit address space. These bits may be combined with the bits in BAR_1 to create a full
64 bit address decode. These bits must be set to zero for the card to respond to single address cycle requests. This value is sticky and only reset by HARD Reset.
Base Address Register 3 (offset: 0x18)
Name
Address
Bits
31:4
Access
RW
Default
Value
0
Description
These bits set the address within a 32-bit address space that will be card will respond in. These bits may be combined with the bits in BAR_4 to create a full 64 bit address decode. Only the bits that address blocks bigger than the setting in the
BAR2_SIZE value are RW. All lower bits are RO with a value of zero. This value is sticky and only reset by HARD Reset.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 275
BCM5718 Programmer’s Guide PCI Configuration Registers
Name
Prefetch
Type
Space
Bits
3
2:1
0
Access
RO
RO
RO
Default
Value
0x1
0x2
0
Description
This bit indicates that the area mapped by BAR_2 may be prefetched or cached by the system without side effects. Path = i_cfg_func.i_cfg_private.
These bits indicate that BAR_2 may be programmed to map this adapter to anywhere in the 64-bit address space. Path = i_cfg_func.i_cfg_private.
This bit indicates that BAR_2 maps a memory space and is always read as 0. Path = i_cfg_func.i_cfg_private.
Base Address Register 4 (offset: 0x1c)
Name
Address
Bits Access
31:0 RW
Default
Value
0
Description
These bits set the address upper 32-bit address space. These bits may be combined with the bits in BAR_2 to create a full
64 bit address decode. These bits must be set to zero for the card to respond to single address cycle requests. This value is sticky and only reset by HARD Reset.
Base Address Register 5 (offset: 0x20)
The 32-bit BAR_5 register programs the 3rd base address for the memory space mapped by the card onto the
PCI bus. This register can be combined with BAR_4 to make a 64-bit address for supporting Dual Address cycles systems.This register is not needed by Xinan and is expected to be disabled. The register is used by Everest which requires a 2nd BAR. Path = i_cfg_func.i_cfg_public.i_cfg_dec.
Name
Address
Prefetch
Type
Space
Bits Access
31:4 RW
Default
Value
0
3
2:1
0
RO
RO
RO
0x1
0x2
0
Description
These bits set the address within a 32-bit address space that will be card will respond in. These bits may be combined with the bits in BAR_6 to create a full 64 bit address decode. Only the bits that address blocks bigger than the setting in the
BAR3_SIZE value are RW. All lower bits are RO with a value of zero. This value is sticky and only reset by HARD Reset.
This bit indicates that the area mapped by BAR_3 may be pre-fetched or cached by the system without side effects.
Path = i_cfg_func.i_cfg_private.
These bits indicate that BAR_3 may be programmed to map this adapter to anywhere in the 64-bit address space. Path = i_cfg_func.i_cfg_private.
This bit indicates that BAR_3 maps a memory space and is always read as 0. Path = i_cfg_func.i_cfg_private
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 276
BCM5718 Programmer’s Guide PCI Configuration Registers
Base Address Register 6 (offset: 0x24)
The 32-bit BAR_4 register programs the upper half of the 3nd base address for the memory space mapped by the card onto the PCI bus.
Name
Address
Bits Access
31:0 RW
Default
Value
0
Description
These bits set the address upper 32-bit address space These bits may be combined with the bits in BAR_5 to create a full
64 bit address decode. These bits must be set to zero for the card to respond to single address cycle requests. This value is sticky and only reset by HARD Reset. Path = i_cfg_func.i_cfg_public.i_cfg_dec.
Cardbus CIS Pointer Register (offset: 0x28)
This register is reset by Hard Reset.
Name
Cardbus CIS
Pointer
Bits Access
31:0 RO
Default
Value
0x0
Description
N/A for PCIE Device
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 277
BCM5718 Programmer’s Guide PCI Configuration Registers
Subsystem ID/Vendor ID Register (offset: 0x2C)
This register is reset by Hard Reset.
Name Bits Access
Subsystem Device
ID
31:16 RO
Subsystem
Vendor ID
15:0 RO
Default
Value
–
0x14E4
Description
Default for BCM5717 (LAN Function 0): 0x1655
Default for BCM5719 (LAN Function 0): 0x1657
Default for BCM5718 (LAN Function 0): 0x1656
Default for BCM5720 (LAN Function 0): 0x165F
Identifies board manufacturer
Expansion ROM Base Address Register (offset: 0x30)
This register is reset by PCIE Reset. It becomes aN RW register if bit 5 of PCI State Register is set.
Name
ROM Base
Address
ROM Size indication
Reserved
Bits
31:24
23:11
Access
RW
RW
Expansion ROM
Enable
0
10:1 RO
RW
Default
Value
0xXXXX
0x00
0x000
0x0
Description
These bits indicate the address of the Expansion ROM area.
These bits indicate the size of the Expansion ROM area or the address of it. The boundary form RO bits to RW bits is controlled by the EXP_ROM_SIZE bits.
These bits indicate that the Expansion ROM area is at least 2k bytes. They always read as zero. P
This bit indicates that the Expansion ROM BAR is valid when set to one. If it is zero, the expansion BAR should not be programmed or used. This bit will only be RW if it is enabled by the EXP_ROM_ENA bit which defaults to 0.
Capabilities Pointer Register (offset: 0x34)
Name
Reserved
Capabilities pointer
Bits Access
31:8 RO
7:0 RO
Default
Value
0x0
0x48
Description
Unused
The 8-bit Capabilities Pointer register specifies an offset in the PCI address space of a linked list of new capabilities. The capabilities are PCI-X, PCI Power
Management, Vital Product Data (VPD), and Message
Signaled Interrupts (MSI) is supported.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 278
BCM5718 Programmer’s Guide PCI Configuration Registers
Interrupt Register (offset: 0x3C)
Name Bits Access
MAXIMUM_LATENC
Y
31:24 RO
MIN_GRANT
Interrupt Pin
23:16
15:8
RO
RO
Default
Value
0x00
0x00
0x01
Interrupt Line 7:0 RW 0x00
Description
Hardwired to zero
Hardwired to zero
Indicates which interrupt pin this device uses:
0: no Interrupt
1: Use Interrupt A
2: Use Interrupt B
3: Use Interrupt C
4: Use Interrupt D
Identifies interrupt routing information
INT Mailbox Register (offset: 0x40–0x44)
Name
Indirect Interrupt mail box
Bits Access
63:0 RW
Default
Value
0
Description
Interrupt Mailbox
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 279
BCM5718 Programmer’s Guide PCI Configuration Registers
Power Management Capability Register (offset: 0x48)
This register is reset by hard Reset.
Name
PME Support
D2 Support
D1 Support 25 RO
FW-RW
Aux Current 24:22 RO
FW-RW
DSI 21 RO
0x0
0x0
0x0
Reserved
PME Clock
Version
PM Next
Capabilities
PM Capability
ID
Bits
31:27
26
20
19
18:16
15:8
7:0
Access
RO
RO
RO
RO
RO
RO
RO
Default
Value
0x08 if no aux
0x18 if aux
0x0
0x0
0x0
0x3
0x58
0x01
Description
Indicates the power states in which the device may assert
PME. A 0 for any bit indicates that the device is not capable of asserting the PME pin signal while in that power state.
Bit 27: PME can be asserted from D0
Bit 28: PME can be asserted from D1
Bit 29: PME can be asserted from D2
Bit 30: PME can be asserted from D3H
Bit 31: PME can be asserted from D3C (default depends on the presence of Aux power)
Indicates whether the device supports the D2 PM state. This device does not support D2; hardwired to 0
Indicates whether the device supports the D1 PM state. This device does not support D1
This device supports the data register for reporting Aux
Current requirements so this field is N/A.
Indicates that the device requires device specific initialization
(beyond PCI configuration header) before the generic class device driver is able to use it. This device hardwires this bit to
0 indicating that DSI is not necessary
–
Indicates that the device relies on the presence of the PCI clock for PME operation. This device does not require the PCI clock to generate PME. Therefore, the bit is hardwired to 0
A value of 011b indicates that this function complies with revision 1.2 of the PCI PM specification.
Points to the next capabilities block which is Broadcom
Vendor Specific Capability Header
Identifies this item as Power management capabilities
Power Management Control/Status Register (offset: 0x4C)
This register is reset by Hard Reset.
Name Bits Access
PM Data 31:24 RO
FW-RW
Reserved 23:16 RO
Default
Value
0x00
0x00
Description
Contains the power management data indicated by the Data
Select field in PMCSR
–
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 280
BCM5718 Programmer’s Guide PCI Configuration Registers
Name Bits
PME Status 15
Data Scale
Data Select
Reserved
No Soft
Reset
Reserved
14:13
12:9
PME Enable 8
7:4
3
2
Power State 1:0
Access
Default
Value
RW2C 0x0
RO
RW
RW
RO
RO
RO
RW
0x1
0x0
0x1
0x00
0x1
0x0
0x0
Description
This bit is set when the device asserts the WAKE signal independent of the PME enable bit. Writing 1 this bit will clear it and cause the device to stop asserting WAKE
Indicates the scaling factor that is used when interpreting the value of the data register (offset 7 in PM capability space). The device hardwires this value to 1 to indicate a scale of 1x
Indicates which data is to be reported via the Data register (offset
7 in PM capability space)
Enables the device to generate PME when this bit is set to 1.
When 0, PME generation is disabled
–
No_Soft_Reset
When set (1), this bit indicates that devices transitioning from
D3hot to D0 because of PowerState commands do not perform an internal reset. Configuration Context is preserved. Upon transition from the D3hot to the D0 Initialized state, no additional operating system intervention is required to preserve
Configuration Context beyond writing the PowerState bits.
When clear (0), devices do perform an internal reset upon transitioning from D3hot to D0 via software control of the
PowerState bits. Configuration Context is lost when performing the soft reset. Upon transition from the D3hot to the D0 state, full reinitialization sequence is needed to return the device to D0
Initialized.
Regardless of this bit, devices that transition from D3hot to D0 by a system or bus segment reset will return to the device state D0
Uninitialized with only PME context preserved if PME is supported and enabled.
–
Indicates the current power state of the device when read.
When written, it sets the device into the specified power state
00: D0 - Select D0
01: D1 - Select D1
10: D2 - Select D2
11: D3-Hot - Select D3
These bits may be used by the system to set the power state. The register is implemented as two banks of two bits each. Can be written from both configuration space and from the PCI register space as the PM_STATE bits. When written from the PCI bus, only values of 0 and 3 are accepted. This is the register returned on reads of this register from configuration space. The second bank catches all writes values. The value of the second register is returned when the PM_STATE bits are read from register space.
The idea of these registers is to a) Provide compatible operation to 5701 b) Allow implementation of other power states though firmware.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 281
BCM5718 Programmer’s Guide PCI Configuration Registers
MSI Capability Header (offset: 0x58)
The device driver is prohibited from writing to this register.
Name
MSI Control
MSI_PVMASK_
CAPABLE
64-bit Address
Capable
Bits Access
31:25 RO
24 RO
Default
Value
0x00
0
23 RO 1
Multiple Message
Enable
Multiple Message
Capable
MSI Enable
Next Capability
Pointer
MSI capability ID
22:20
19:17
16
15:8
7:0
RW
RO
RW
RO
RO
0x0
0x3
0
A0
0x5
Description
Reserved
This bit indicates if the function supports per vector masking. This value comes from the MSI_PV_MASK_CAP bit in the register space.
Hardwired
Advertise 64-bit address capable
This bit indicates that the chip is capable of generating 64 bit MSI messages.
These bits indicate the number of message that the chip is configured (allowed) to generate. Number of allocated message:
0 1 Chip is set to generate 1 message
1 2 Chip is set to generate 2 messages
2 4 Chip is set to generate 4 messages
3 8 Chip is set to generate 8 messages
4 16 Chip is set to generate 16 messages
5 32 Chip is set to generate 32 messages
These bits indicate the number of messages that the chip is capable of generating. This value comes from the bit in the register space. Number of requested message:
0 1 Chip is set to generate 1 message
1 2 Chip is set to generate 2 messages
2 4 Chip is set to generate 4 messages
3 8 Chip is set to generate 8 messages
4 16 Chip is set to generate 16 messages
5 32 Chip is set to generate 32 messages
When this bit is set, the chip will generate MSI cycles to indicate interrupts instead of asserting the INTA# pin.
When this bit is zero, the INTA# pin will be used.
This value continues the PCI capability chain. It's value specified an offset in the PCI address space of the next capability. The read-only value of this register is controlled by the CAP_ENA register in the PCI register space.
The 8-bit MSI Capability ID is set to 5 to indicate that the next 8 bytes are a Message Signaled Interrupt capability block.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 282
BCM5718 Programmer’s Guide PCI Configuration Registers
MSI Lower Address Register (offset: 0x5C)
Name
MSI Lower
Address
Reserved
Bits
31:2
1:0
Access
RW
RO
Default
Value Description
Unknown MSI Lower Address
0 –
MSI Upper Address Register (offset: 0x60)
Name
MSI Upper
Address
Bits
31:0
Access
RW
Default
Value Description
Unknown MSI Upper Address
MSI Data Register (offset: 0x64)
Name
MSI Data
Bits
15:0
Access
RW
Default
Value Description
Unknown MSI Data
Miscellaneous Host Control Register (offset: 0x68)
Name
ASIC Rev ID
Enable TLP Minor Error
Tolerance
Log Header Overflow
Bits Access
31:28 R
27:24
23:16
15
14
R
R
RW
RW
Default
Value Description
Product ID input
0xF: Indication that BCM5718 family follows new PRODUCT/REV ID mapping
ASIC Rev
Input
External All Layer Revision ID.
These bits will reflect in offset 8-bit mapping description:
0x0: A
0x1: B
0x2: C
ASIC Rev
Input
0
0
Metal Rev Number
0x0: 0
0x1: 1
0x2: 2
Set this bit to enable TLP minor error tolerance
(ATTR/TC/LOCK command)
Set this bit to enable log header due to overflow
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 283
BCM5718 Programmer’s Guide PCI Configuration Registers
Name
Boundary check
Byte enable Rule Check
Interrupt Check
RCB Check
Enable Tagged Status Mode
Mask Interrupt Mode
Enable indirect access 7
Enable Register Word Swap 6
Enable Clock Control register read/write capability
5
Enable PCI State register read/ write capability
4
Enable Endian Word Swap
Enable Endian Byte Swap
Mask Interrupt
Clear Interrupt
Bits Access
13 RW
Default
Value
0
12
11
10
9
8
3
2
1
0
RW
RW
RW
RW
RW
RW
RW
RW
RW
RW
RW
RW
WO
0
0
0
0
0
0
0
0
0
0
0
0
0
Description
Set this bit to enable crossing 4 KB boundary check
Set this bit to enable the byte enable rule check
Set this bit to enable the interrupt check
Set this bit to enable RCB check
When set, an unique 8-bit tag value will be inserted into the Status block status tag
When set, the interrupt is masked. However, the internal interrupt state (host coalescing event) will not be cleared
Set this bit to enable indirect addressing mode
Set this bit to enable word swapping when accessing registers through the PCI target device
Set this bit enable clock control register read/ write capability, otherwise, the clock control register is read only
Set this bit to enable PCI state register read/ write capability, otherwise the register is read only
Set this bit to enable endian word swapping when accessing through PCIE target interface
Set this bit to enable endian byte swapping when accessing through PCIE target interface.
Note: Setting register 0x68 bit 2 (Enable Endian
Byte Swap) causes PCI configuration reads to the following registers to become swapped:
• 0x40
• 0x68 - 0x9C
• 0xF4 - 0xFF
This is different behavior from previous
NetXtreme controllers. Reference BCM5718
Family errata relating to byte swap control for additional information.
Setting this bit will mask future interrupt events from being generated. Setting this bit will not clear or deassert the internal interrupt state, nor will it deassert the external interrupt state.
Setting this bit will clear interrupt as long as the mask interrupt bit is not set. If mask interrupt bit is set, then writing 1 to this bit will not deassert interrupt, however, it will clear the internal unmasked interrupt state, so if the interrupt is later unmasked, the interrupt will deassert.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 284
BCM5718 Programmer’s Guide PCI Configuration Registers
DMA Read/Write Control Register (Offset: 0x6C)
Name
Reserved
Reserved
DMA write watermark
Reserved
Enable MSIX fix
Reserved
Bits Access
31:29 RW
28:2 RO
21:19 RW
18:8
7
DMA read MRRS for slow speed 6:4
3:2
Disable_64B_cache_alignment 1
Disable_32B_cache_alignment 0
RW
RW
RW
RW
RW
RW
0
0
0
0
0
Default
Value
0x0
0x00
7
0
Description
–
–
Watermark for DMA write.
0: 32B
1: 64B
2: 96B
3:128B
4:160B
5:192B
6:224B
7:256B
–
Enable msi/msix legacy fix.
This setting is for 10/100M Ethernet DMA read
MRRS. The pcie_core will accord to this value to be max DMA read length. This configuration has no effect for GIGA mode.
0: 1024B
1: 128B
2: 256B
3: 512B
4:512+256B
5:1024+512B
6:2048B
7:4096B
–
Disable 64B cache alignment for DMA write to
Host memory
Disable 32B cache alignment for DMA write to
Host memory
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 285
BCM5718 Programmer’s Guide PCI Configuration Registers
PCI State Register (offset: 0x70)
This register is reset by PCIE Reset.
Name
Reserved
Generate reset plus
APE Program Space Write
Enable
APE Shared Memory Write
Enable
APE Control Register Write
Enable
Config Retry
Reserved
Max PCI Target Retry
Flat View
VPD Available
PCI Expansion ROM Retry
PCI Expansion ROM Desired
Reserved
Bits Access
31:20 RO
19 W1
Read 0
18 RW
17
16
15
14:12
11:9
8
7
6
5
4:0
RW
RW
RO
RO
RW
RW
RO
RW
RW
RO
Default
Value Description
0x0000 –
0 For func 1 write 1 generates 10 clock wide reset pulse reads always 0 for func 0 reserved
0
0
0
When this bit is set the APE program space may be written.
When this bit is set the APE shared memory region may be written.
When this bit is set the APE control registers may be written.
0x1
On Hard reset
0x0
0x1
When asserted, forces all config access to be retried.
0x0
0x0
0x0
0x0
XXX
–
Indicates the number of PCI clock cycles before
Retry occurs, in multiple of 8. At reset, this field is set to 001 N/A in PCIE
Asserted if the Base Address register presents a
32 MB PCI Address map flat view, otherwise, indicates a 64 KB PCI Address map in standard view
This bit reads as 1 if the VPD region of the NVRAM can be accessed by the host
Comes from GRC 6808
Force PCI Retry for accesses to Expansion ROM region if enabled
Enable PCI ROM base address register to be visible to the PCI host
–
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 286
BCM5718 Programmer’s Guide PCI Configuration Registers
Reset Counters Initial Values Register (offset: 0x74)
Name
Reset Counter 5 Register
(PCI CLK Core Syn Reset)
Reset Counter 4 Register
(Hot Reset)
Reset Counter 3 Register
(GRC Reset)
Reset Counter 2 Register
(Perst Reset)
Reset Counter 1 Register
(LinkDown Reset)
Bits Access
Default
Value
31:28 Host RW Any
27:24 Host RW Any
Description
Keep tracks of the number of Core Syn Reset that are synchronized in the PCI Clock Domain
Keep tracks of the number of Hot Reset events.
23:16 Host RW Any
15:8 Host RW Any
7:0 Host RW Any
Keep tracks of the number of GRC Reset.
Keep tracks of the number of Perst events.
Keep tracks of the number of LinkDown Reset events.
Register Base Register (offset: 0x78)
Name
Reserved
Register Base Register
Reserved
Bits Access
31:18 RO
17:2 RW
0
X
Default
Value
1:0 RO 0
Description
–
Local controller memory address of a register than can be written or read by writing to the register data register
–
Memory Base Register (offset: 0x7C)
Name
Reserved
Memory Base Register
Reserved
Bits Access
31:24 RO
23:2 RW
0
X
Default
Value
1:0 RO 0
Description
–
Local controller memory address of the NIC memory region that can be accessed via Memory
Window data register
–
Register Data Register (offset: 0x80)
Name
Register Data Register
Bits Access
31:0 RW
Default
Value
X
Description
Register Data at the location pointed by the
Register Base Register
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 287
BCM5718 Programmer’s Guide PCI Configuration Registers
Memory Data Register (offset: 0x84)
Name
Memory Base Register
Bits Access
31:0 RW
Default
Value
X
Description
Memory value at the location pointed by the
Memory Base Register
UNDI Receive Return Ring Consumer Index Register (offset: 0x88–
0x8C)
Name Bits Access
UNDI Receive Return C_Idx 63:0 RW
Default
Value
0
Description
UNDI Receive Return Ring Consumer Index
Mailbox
UNDI Send BD Producer Index Mailbox Register (offset: 0x90–0x94)
Name
UNDI Send BD NIC P_Idx
Bits Access
63:0 RW
Default
Value
0
Description
UNDI Send BD NIC Producer Index Mailbox
UNDI Receive BD Standard Producer Ring Producer Index Mailbox
Register (offset: 0x98–0x9C)
Name
UNDI Receive BD Standard
Ring Producer Index
Bits Access
63:0 RW
Default
Value
0
Description
UNDI Receive BD Std. Ring Producer Index
Mailbox
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 288
BCM5718 Programmer’s Guide PCI Configuration Registers
MSI-X Capabilities Registers
MSI-X Capability Header Register (offset: 0xA0)
Name
MSIX_ENABLE
FUNC_MASK
RESERVED
TABLE_SIZE
MSIX_NEXT_CAP_PTR
MSIX_CAP_ID
Bits Access
31 RW
Default
Value
0
30
29:27
26:16
15:8
7:0
RW
RO
RO
RO
RO
0
0
0
0xac
0x11
Description
If 1 and the MSI enable bit in the MSI message control register is 0, the function is permitted to use MSIX request service and profited from using
INTx# messages.
Path = i_cfg_func.i_cfg_public.i_cfg_msi_cap
If 1, all of the vectors associated with the function are masked regardless of their per vector Mask bit.
Path = i_cfg_func.i_cfg_public.i_cfg_msi_cap
Reserved
System sw reads this field to determine the MSI-
X table size N, which is encoded as N-1
Path = i_cfg_func.i_cfg_private
This value continues the PCI capability chain. It's value specified as offset in the PCI address space of the next capability. The read-only value of this register is controlled by the CAP_ENA register in the PCI register space.
Path = i_cfg_func.i_cfg_public.i_cfg_ep_reg
Capability ID for MSIX Path = cfg_defs
MSIX_TBL_OFF_BIR – 0xa4
Name
TABLE_OFFSET
TABLE_BIR
Bits Access
31:3 RO
2:0 RO
0
0
Default
Value Description
Path = i_cfg_func.i_cfg_private
Indicates which one of functions BAR is used to map MSI-X table into memory space.
Path = i_cfg_func.i_cfg_private
MSIX_PBA_BIR_OFF – 0xa8
Name
TABLE_OFFSET
TABLE_BIR
Bits Access
31:3 RO
2:0 RO
0
0
Default
Value Description
Path = i_cfg_func.i_cfg_private
Indicates which one of functions BAR is used to map MSI-X table into memory space.
Path = i_cfg_func.i_cfg_private
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 289
BCM5718 Programmer’s Guide PCI Configuration Registers
PCIe Capabilities Registers
PCIE_CAPABILITY – 0xac
Name unused0
MSG_NUM
SLOT_IMPLEMENTED
TYPE
VER
PCIE_NEXT_CAP_PTR
PCIE_CAP_ID
Bits Access
31:30 RO
29:25 RO
0
0
Default
Value
24
23:20
19:16
15:8
7:0
RO
RO
RO
RO
RO
0
0
0x2
0
0x10
Description
–
Interrupt Message Number: Indicate which MSI/
MSI-X vector is used for the interrupt message generated in association with any of the status bits of this capability structure. For MSI, the value in this register indicates the offset between the base Message Data and the interrupt message that is generated. For MSI-X, the value in this register indicates which MSI-X Table entry is used to generate the interrupt message. The entry must be one of the first 32 entries even if the function implements more than 32 entries.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
Slot Implemented. This register is not supported.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
Slot Implemented. This register is not supported.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
Capability Version. PCI Express Capability structure version number. These bits are hardwired to 2h. Path= cfg_defs
This registers contains the pointer to the next PCI capability structure.
Path= i_cfg_func.i_cfg_public.i_cfg_rd_mux
This register contains the PCIExpress Capability
ID. Path= i_cfg_func.i_cfg_public.i_cfg_rd_mux
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 290
BCM5718 Programmer’s Guide PCI Configuration Registers
DEVICE_CAPABILITY – 0xb0
Name Bits Access unused3 31:29 RO
FLR_CAP_SUPPORTED 28 RO
0
0
Default
Value
CAPTURED_SLOT_PWR_SCAL
E
ROLE_BASED_ERR_RPT unused1 14:12
L1_ACCEPTABLE_LATENCY
L0S_ACCEPTABLE_LATENCY
EXTENDED_TAG_SUPPORT
27:26
CAPTURED_SLOT_PWR_VAL 25:18 unused2 17:16
15
11:9
8:6
5
RO
RO
RO
RO
RO
RO
RO
RO
0
0
0
0x1
0
0x6
0x6
0
Description
–
FLR capability is advertized when flr_supported bit in private device_capability register space is set.
Specifies the scale used for the Slot Power Limit
Value. It is set by the Set_Slot_Power_Limit
Message. This field is not set for Root ports
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
Specifies the upper limit on power supplied by slot. It is set by the Set_Slot_Power_Limit
Message. This field is not set for Root ports.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
–
Indicate device is conforming to the ECN, PCI
Express Base Specification, Revision 1.1., or subsequent PCI Express Base Specification revisions. Path= i_cfg_func.i_cfg_private
–
Endpoint L1 Acceptable Latency. These bits are programmable through register space. The bits should be 0 for Root ports.
Path= i_cfg_func.i_cfg_private
Endpoint L0s Acceptable Latency. These bits are programmable through register space. The value should be 0 for root ports.
Path= i_cfg_func.i_cfg_private
Extended Tag Field Support. This bit is programmable through register space. This capability is not currently supported.
Path= i_cfg_func.i_cfg_private unused0 4:3
MAX_PL_SIZE_SUPPORTED 2:0
RO
RO
0
0x1 Max Payload Size Supported. These bits are programmable from the register space and default value is based on define in version.v file.
Path= i_cfg_func.i_cfg_private
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 291
BCM5718 Programmer’s Guide PCI Configuration Registers
DEVICE_STATUS_CONTROL – 0xb4
Name
Unused1
NP_TRANSACTION_PEND
AUX_PWR_DET
UNSUP_REQ_DET
FATAL_ERR_DET
NON_FATAL_ERR_DET
CORR_ERR_DET
FLR_INITIATED
MAX_READ_REQ_SIZ
NO_SNOOP_ENABLE
AUX_PWR_PM_ENA
Unused0 9
EXTENDED_TAG_EN
MAX_PAYLOAD_SIZE
Bits Access
31:22 RO
21 RO
0
0
Default
Value
20
19
18
17
16
15
14:12
11
10
8
7:5
RELAX_ORDERING_ENABLE 4
RO
WC
WC
WC
WC
RW
RW
RW
RW
RO
RO
RW
RW
0x1
0
0
0
0
0
0
0x1
0x1
0
0
0
0x1
Description
–
This is bit is read back a 1, whenever a non-posted request initiated by PCIE core is pending to be completed. Path= i_tl_top
This bit is the current state of the VAUX_PRSNT pin of the device. When it is '1', it is indicating that part needs VAUX and detects the VAUX is present.
Path= input to pcie_vaux_pipe
UnSupported Request Detected.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap.
Fatal Error Detected.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap.
Non-Fatal Error Detected.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap.
Correctable Error Detected.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap.
Initiate Function Level reset. This bit is writeable only if flr_supported bit in private device_capability register is set. A write of 1 to this bit initiates Function Level Reset. The value read by s/w from this bit is always 0.
Maximum Read Request Size. Depending on the spec, internal logic uses either the min or the max of the value of the two functions.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap.
Enable No Snoop. When this bit is set to 1, PCIE initiates a read request with the No Snoop bit in the attribute field set for the transactions that request the No Snoop attribute.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap.
This bit when set enables device to draw aux power independent of PME AUX power
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap.
–
Extended Tag Field Enable. This capability is not supported.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap.
Max Payload Size. Depending on the spec, internal logic uses either the min or the max of the value of the two functions.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap.
Ordering
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 292
BCM5718 Programmer’s Guide PCI Configuration Registers
Name
U_REQ_REPORT_EN
FATAL_ERR_REPORT_EN
NFATAL_ERR_REPORT_EN
CORR_ERR_REPORT_EN
Bits Access
3 RW
Default
Value
0:pr
2
1
0
RW
RW
RW
0:pr
0:pr
0:pr
Description
Unsupported Request Reporting Enable.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap.
Fatal Error Reporting Enable.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap.
Non-Fatal Error Reporting Enable.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap.
Correctable Error Reporting Enable.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap.
LINK_CAPABILITY – 0xb8
Name
PORT_NUMBER
Unused0 23:22
LINK_BW_NOTIFY
DL_ACTIVE_REP
SUR_DWN_ERR_REP
CLK_PWR_MGMT
Bits Access
31:24 RO
Default
Value
0
21
20
19
18
RO
RO
RO
RO
RO
0
0
0
0
0x1
Description
PCIE Port Number. These bits are programmable through register. Path= i_cfg_func.i_cfg_private
–
Link Bandwidth Notification Capability: RC: A value of 1b indicates support for the Link
Bandwidth Notification status and interrupt mechanisms. This capability is required for all
Root Ports and Switch Downstream Ports supporting Links wider than x1 and/or multiple
Link speeds. RC: Field is implemented. EP: Not supported and hardwired to 0.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
Data Link Layer Link Active Reporting Capable:
RC: this bit must be hardwired to 1b if the component supports the optional capability of reporting the DL_Active state of the Data Link
Control and Management State Machine. RC:
Implemented (RW) for RC. Default to 0. EP: Not supported and hardwired to 0.
Path= i_cfg_func.i_cfg_private
Surprise Down Error Reporting Capable: RC: this bit must be set if the component supports the optional capability of detecting and reporting a
Surprise Down error condition. RC: Not supported and hardwired to 0. EP: Not supported and hardwired to 0.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
Clock Power Management. These bits are programmable through register. The feature itself has to be enabled in version.v
Path= i_cfg_func.i_cfg_private
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 293
BCM5718 Programmer’s Guide PCI Configuration Registers
Name
L1_EXIT_LAT
L0S_EXIT_LAT
Bits Access
17:15 RO
Default
Value
0x2
Description
L1 Exit Latency. These bits are programmable through register space.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
Depending on whether device is in common clock mode or not, the value reflected by these bits is one of the following.
14:12 RO 0x5
Value Name Description
1
2
255
1_2
2_4
–
L1 exit latency of 1 us to 2 us.
L1 exit latency of 2 us to 4 us. end_of_table
L0s Exit Latency. These bits are programmable through register space.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
Depending on whether device is in common clock mode or not, the value reflected by these bits is one of the following.
Value Name Description
4
5
0_1
1_2
L0s exit latency of 512 ns to 1 us.
L0s exit latency of 1 us to 2 us.
255 – end_of_table
ASPM Support. These bits are programmable through reg space.
Path= i_cfg_func.i_cfg_private
ASPM_SUPT 11:10 RO 0x3
Value
2
3
0
1
255
Name
RES_0
L0S
RES_2
L0S_L1
–
Description
Reserved
L0s entry supported
Reserved
L0s and L1 supported end_of_table
Path= i_cfg_func.i_cfg_private Value used by internal logic is the smaller of the value programmed for each function
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 294
BCM5718 Programmer’s Guide
Name Bits
MAX_LINK_SPEED 3:0
Access
RO
Default
Value
0x1
Description
Value
0
1
255
Name
64
128
–
PCI Configuration Registers
Description
64 Bytes
128 Bytes end_of_table
LINK_STATUS_CONTROL – 0xbc
Name
Unused3
DL_ACTIVE
SLOT_CLK_CONFIG
LINK_TRAINING
Unused2
NEG_LINK_WIDTH
NEG_LINK_SPEED
Unused1
LINK_BW_INT_EN
LINK_BW_MGMT_INT_EN
Bits Access
31:30 RO
29 RO
0
0
Default
Value
28
27
26
25:20
19:16
15:12
11
10
RO
RO
RO
RO
RO
RO
RO
RO
0x1
0
0
0
0
0
0
0
Description
–
Data Link Layer Link Active: returns a 1b to indicate the DL_Active state, 0b otherwise. Not implemented and hardwire to 0.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
Slot Clock configuration. This bit is read-only by host, but read/write via backdoor CS bus.
Path= i_cfg_func.i_cfg_private
EP: This bit is N/A and is hardwired to 0.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
–
Negotiated Link Width. These bits indicate the negotiated link width of the PCI Express link.
Path= i_pl_top.i_pl_ltssm
Link Speed. These bits indicate the negotiated link speed of the PCI Express link.
Path= i_pl_top.i_pl_ltssm
–
Link Autonomous Bandwidth Interrupt Enable:
When Set, this bit enables the generation of an interrupt to indicate that the Link Autonomous
Bandwidth Status bit has been Set. RC: Not implemented and hardwired to 0. EP: N/A and hardwired to 0
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
Link Bandwidth Management Interrupt Enable: when Set, this bit enables the generation of an interrupt to indicate that the Link Bandwidth
Management Status bit has been Set. RC: N/A and hardwired to 0. EP: Not implemented and hardwired to 0.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 295
BCM5718 Programmer’s Guide PCI Configuration Registers
Name
HW_AUTO_WIDTH_DIS
EN_CLK_PW_MGMT
LINK_CR_EXT_SYNC
LINK_CR_COMMON_CLK
CFG_PSM_RETRAIN_LINK
CFG_PSM_LINK_DISABLE
RCB
Bits Access
9 RO
Default
Value
0
8
7
6
5
4
3
RW
RW
RW
RO
RO
RW
0
0
0
0
0
0
Description
Hardware Autonomous Width Disable: When Set, this bit disables hardware from changing the Link width for reasons other than attempting to correct unreliable Link operation by reducing Link width. Other functions are reserved. RC: Not applicable and hardwire to 0 EP: If supported, only apply to function0. Not implemented and hardwire to 0. Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
Enable Clock Power Management: RC: N/A and hardwired to 0. EP: When this bit is set, the device is permitted to use CLKREQ# signal to power management. Feature is enabled through version.v define
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
Extended Synch. This bit when set forces the transmission of 4096 FTS ordered sets in the L0s state followed by a single SKP ordered set prior to entering the L0 state, and the transmission of
1024 TS1 ordered sets in the L1 state prior to entering the Recovery state. Value used by logic is resolved to 1 if either function has this bit set.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
Common Clock Configuration. Value used by logic is resolved to 1 only if both functions (when enabled) have this bit set. Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
Requesting PHY to retrain the link. This bit is only applicable to RC. So for EP it is read only bit. Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
Requesting PHY to disable the link. This bit is only applicable to RC. So for EP it is read only bit.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
Read Completion Boundary.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
Unused0 2 RO 0
Value Name
0
1
255
64
128
–
–
Description
64 Bytes
128 Bytes end_of_table
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 296
BCM5718 Programmer’s Guide PCI Configuration Registers
Name
ASPM_CTRL
Bits Access
1:0 RW
Default
Value
0
Description
ASPM Control. Value used by logic is dependent on the value of this bit for each enabled function and also on the programmed powerstate of each function.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
SLOT_CAPABILITY – 0xc0
Name Bits Access
PHYSICAL_SLOT_NUMBER 31:19 RO
UNUSED 18:17 RO
SLOT_POWER_LIMIT_SCALE 16:15 RO
SLOT_POWER_LIMIT_VALUE 14:7 RO
UNUSED_2 6:0 RO
0
0
0
0
0
Default
Value Description
Not implemented
SLOT_CONTROL_STATUS – 0xc4
Name Bits Access
SLOT_STATUS 31:23 RO
PRESENCE_DETECT 22 RO
UNUSED_1 21:16 RO
SLOT_CONTROL 15:0 RO
0
0
0
0
Default
Value Description
ROOT_CAP_CONTROL – 0xc8
This register is not applicable for EP and hardwired to 0.
Name
Unused
Bits Access
31:0 RO
Default
Value
0
Description
–
ROOT_STATUS – 0xcc
This register is not applicable for EP and hardwired to 0.
Name
Unused
Bits Access
31:0 RO
Default
Value
0
Description
–
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 297
BCM5718 Programmer’s Guide PCI Configuration Registers
DEVICE_CAPABILITY_2 – 0xd0
Name Bits Access
Unused1 31:12 RO
LTR_MECHANISM_SUPPORTED 11 RO
0
0
Default
Value
Unused0
CMPL_TIMEOUT_DISABL_
SUPPORTED
CMPL_TIMEOUT_RANGES_
SUPPORTED
10:5
4
3:0
RO
RO
RO
0
0x1
0xf
Description
–
Latency Tolerance Reporting Mechanism
Supported, Programmable through register space. This field will read 1, when bit 5 of ext_cap_ena field in private register space is set.
–
Completion Timeout Disable Supported,
Programmable through register space
Path= i_cfg_func.i_cfg_private
Completion Timeout Ranges Supported.
Programmable through register space
Path= i_cfg_func.i_cfg_private
Value
15
255
Name
ABCD
–
Description
Ranges A, B, C, and D end_of_table
DEVICE_STATUS_CONTROL2 – 0xd4
Name
DEVICE_STATUS_2
Unused
LTR_MECHANISM_ENABLE
IDO_CPL_ENABLE
IDO_REQ_ENABLE
Unused0
Bits Access
31:16 RO
Default
Value
0
15:11
10
9
8
7:5
RO
RW
RW
RW
RO
0
0
0
0
0
Description
Placeholder for Gen2
Path= i_cfg_func.i_cfg_public.i_cfg_rd_mux
–
Latency Tolerance Reporting Mechanism
Enable, This field is writeable, when bit 5 of ext_cap_ena field in private register space is set.
This bit is RW only in function 0 and is RsvdP for all other functions.
IDO Completion Enable, This field is writeable, when bit ido_supported bit of private device_capability_2 register is set. When this bit is set, function is permitted to set ID based
Ordering Attribute of Completions it returns.
IDO Request Enable, This field is writeable, when bit ido_supported bit of private device_capability_2 register is set. When this bit is set, function is permitted to set ID based
Ordering Attribute of Requests it initiates.
–
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 298
BCM5718 Programmer’s Guide PCI Configuration Registers
Name Bits
CMPL_TIMEOUT_DISABLE 4
Access
RW
Default
Value
0
Description
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
CMPL_TIMEOUT_VALUE 3:0 RW 0
Value
4
5
6
2
3
0
1
8
255
Name
50MS
100US
10MS
55MS
210MS
900MS
3_5S
13S
64S
–
Description
50ms
100us
10ms
55ms
210ms
900ms
3.5s
13s
64s end_of_table
LINK_CAPABILITY_2 – 0xd8
Name
LINK_CAPABILITY_2
Bits Access
31:0 RO
Default
Value
0
Description
Placeholder for Gen2 Path= i_cfg_func.i_cfg_private
LINK_STATUS_CONTROL_2 – 0xdc
This register will be Read only by default, and will read all 0's to allow compliance with PCIE spec 1.1. To enable this register, reset comply_pcie_1_1 bit in the register space to 0.
Name Bits Access
LINK_STATUS_2
CURR_DEEMPH_LEVEL
31:17 RO
16 RO
Unused0 15:13 RO
CFG_COMPLIANCE_DEEMPH 12 RW
0
0
0
0
Default
Value
CFG_COMPLIANCE_SOS 11
CFG_ENTER_MOD_
COMPLIANCE
10
RW
RW
0
0
Description
Placeholder for Gen2 curr_deemph_level Path = pl_top
–
De-emphasis.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
Enter Modified Compliance.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 299
BCM5718 Programmer’s Guide PCI Configuration Registers
Name
CFG_TX_MARGIN
SEL_DEEMPHASIS
Bits Access
9:7 RW
Default
Value
0
Description
Controls the value of non de-emphasized voltage level at the TX pins. Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap Value used by logic is resolved to the smaller binary value, if two functions have different values.
6
HW_AUTO_SPEED_DISABLE 5
ENTER_COMPLIANCE 4
RW
RO
RW
0
0
0
Value Name Description
0 000
1
2
3
6
7
4
5
255
001
010
011
100
101
110
111
–
800 – 1200 mV for full swing and 400 – 600 mV for half swing.
Values will be monotonic with non zero Slope
Values will be monotonic with non zero Slope
200 – 400 mV for full swing and 100 – 200 mV for half swing
Reserved
Reserved
Reserved
Reserved end_of_table
When link is operating at Gen2 rates, this bit selects the level of de-emphasis. Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap Value used by logic is resolved to 1 if either function has this bit set.
Value
0
1
255
Name Description
0
1
–
–
–
–
Not Supported and hardwired to 0.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
S/W instructs link to enter compliance mode.
Value used by internal logic is set when either function has this bit enabled.
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 300
BCM5718 Programmer’s Guide PCI Configuration Registers
Name
TARGET_LINK_SPEED
Bits Access
3:0 RW
Default
Value
0x1
Description
Upper limit of link speed
Path= i_cfg_func.i_cfg_public.i_cfg_exp_cap
Value
0
1
255
Name Description
2_5
5_0
–
2.5 Gbps
5.0 Gbps end_of_table
SLOT_CAPABILITY_2 – 0xe0
Name Bits Access
SLOT_CAPABILITY_2 31:0 RO
Default
Value
0
Description
SLOT_STATUS_CONTROL_2 – 0xe4
Name Bits Access
SLOT_STATUS_2 31:16 RO
SLOT_CONTROL_2 15:0 RO
0
0
Default
Value Description
Not implemented
Product ASIC ID (offset: 0xF4)
Name
Product ASIC ID
Bits Access
31:0 RO
Default
Value Description
For 5717 B0, value is 0x05717100:
• 5718 A0, value is 0x05717000
• 5718 B0, value is 0x05717100
• 5719 A0, value is 0x05719000
• 5719 A1, value is 0x05719100
• 5720 AO, value is 0x05720000
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 301
BCM5718 Programmer’s Guide PCI Configuration Registers
Advanced Error Reporting Enhanced Capability Header (offset: 0x100)
Name
PCI Express Extended
Capability ID
Capability Version
Next Capability Offset
Bits Access
15:0 RO
19:16
31:20
RO
RO
Default
Value Description
0x0001 Extended Capability ID for the Advanced Error
Reporting Capability is 0001h
0x1
0x13C
–
Pointer to the Virtual Channel Capability
Structure
Uncorrectable Error Status Register (offset: 0x104)
This register is reset by Hard Reset.
Name
Reserved
Unsupported Request Error
Status
ECRC Error Status
Malformed TLP Status
Receiver Overflow Status
19
18
17
Bits Access
31:21 RO
20
0
RW1CS 0
Default
Value
RW1CS 0
RW1CS 0
RW1CS 0
Unexpected Completion
Status
Completer Abort Status
Completion Timeout Status
Flow control Protocol Error
Status
Poisoned TLP Status
Reserved
Data Link Protocol Error
Status
Reserved
Training Error Status
16
15
14
13
RW1CS 0
RW1CS 0
RW1CS 0
RW1CS 0
12 RW1CS 0
11:5 RO 0
4 RW1CS 0
3:0
0
RO 0
RW1CS 0
Description
–
This bit is set when an unsupported request error occurs.
This bit is set when an ECRC error occurs
This bit is set when a Malformed TLP error occurs.
This bit is set when a Receiver Overflow error occurs.
This bit is set when an Unexpected Completion error occurs.
This bit is set when a completer Abort error occurs.
This bit is set when completion timeout error occurs.
This bit is set when a Flow control protocol error occurs.
This bit is set when a Poisoned TLP error occurs
–
This bit is set when a Data Link Protocol error occurs.
–
This bit is set when a training error occurs.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 302
BCM5718 Programmer’s Guide PCI Configuration Registers
Uncorrectable Error Mask Register (offset: 0x108)
This register is reset by Hard Reset.
Name Bits Access
Reserved
Unsupported Request Error
Mask
31:21
20
ECRC Error Mask
Malformed TLP Mask
19
18
Receiver Overflow Mask 17
Unexpected Completion Mask 16
RO
RWS
RWS
RWS
RWS
RWS
0
0
0
0
0
0
Default
Value
Completer Abort Mask
Completion Timeout Mask
Flow Control Protocol Error
Mask
Poisoned TLP Mask
Reserved
12 RWS
11:5 RO
Data Link Protocol Error Mask 4
Reserved 3:1
Training Error Mask
15
14
13
0
RWS
RWS
RWS
RWS
RO
RWS
0
0
0
0
0
0
0
0
Description
–
Setting this bit will mask Unsupported Request
Error.
Setting this bit will mask ECRC error.
Setting this bit will mask Malformed TLP error.
Setting this bit will mask Receiver overflow error.
Setting this bit will mask unexpected completion error.
Setting this bit will mask completer abort error
Setting this bit will mask completion timeout error.
Setting this bit will mask flow control protocol error.
Setting this bit will mask poisoned TLP error.
–
Setting this bit will mask data link protocol error.
–
Setting this bit will mask training error.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 303
BCM5718 Programmer’s Guide PCI Configuration Registers
Uncorrectable Error Severity Register (offset: 0x10C)
This register is reset by Hard Reset.
Name
Reserved
Unsupported Request Error
Severity
Bits Access
31:21 RO
20 RWS
0
0
Default
Value
ECRC Error Severity
Malformed TLP Severity
Receiver Overflow Error
Severity
Unexpected completion Error
Severity
Completer Abort Error
Severity
Completion Timeout Error
Severity
Flow control Protocol Error
Severity
Poisoned TLP Severity
Reserved
13
Surprise down error severity 5
Data Link Protocol Error
Severity
Reserved
Training Error Severity
19
18
17
16
15
14
12
11:4
4
3:1
0
RWS
RWS
RWS
RWS
RWS
RWS
RWS
RWS
RO
RO
RWS
RO
RWS
0
1
1
0
0
0
1
0
0
1
1
0
1
Description
–
This bit controls the severity
0 = nonfatal
1 = fatal
This bit controls the severity
0 = nonfatal
1 = fatal
This bit controls the severity
0 = nonfatal
1 = fatal
This bit controls the severity
0 = nonfatal
1 = fatal
This bit controls the severity
0 = nonfatal
1 = fatal
This bit controls the severity
0 = nonfatal
1 = fatal
This bit controls the severity
0 = nonfatal
1 = fatal
This bit controls the severity
0 = nonfatal
1 = fatal
This bit controls the severity
0 = nonfatal
1 = fatal
–
Pcie 1.1 spec page 409
This bit controls the severity
0 = nonfatal
1 = fatal
–
This bit controls the severity
0 = nonfatal
1 = fatal
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 304
BCM5718 Programmer’s Guide PCI Configuration Registers
Correctable Error Status Register (offset: 0x110)
Name Bits Access
Reserved 31:14 RO
Advisory Nonfatal Error Status 13 RO
0
0
Default
Value
Replay Timer Timeout Status 12
Reserved 11:9
REPLAY_NUM Rollover Status 8
Bad DLLP Status
Bad TLP Status
Reserved
Receiver Error Status
7
6
5:1
0
RW1CS 0
RO 0
RW1CS 0
RW1CS 0
RW1CS 0
RO 0
RW1CS 0
Description
–
This bit is set when an Advisory Nonfatal error occurs.
This bit is set when a Replay Timer Timeout error occurs.
–
This bit is set when a REPLAY_NUM Rollover error occurs.
This bit is set when a Bad DLLP error occurs.
This bit is set when a Bad TLP error occurs.
–
This bit is set when a Receiver error occurs.
Correctable Error Mask Register (offset: 0x114)
Name Bits Access
Reserved 31:14 RO
Advisory Nonfatal Error Mask 13 RWS
Replay Timer Timeout Mask 12 RWS
0
1
0
Default
Value
Reserved
Bad DLLP Mask
Bad TLP Mask
Reserved
Receiver Error Mask
11:9
REPLAY_NUM Rollover Mask 8
7
6
5:1
0
RO
RWS
RWS
RWS
RO
RWS
0
0
0
0
0
0
Description
–
Setting this bit masks Advisory Nonfatal errors.
Setting this bit masks Replay Timer Timeout errors.
–
Setting this bit masks REPLAY_NUM Rollover errors.
Setting this bit masks Bad DLLP errors.
Setting this bit masks Bad TLP errors.
–
Setting this bit masks Receiver errors.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 305
BCM5718 Programmer’s Guide PCI Configuration Registers
Advanced Error Capabilities and Control Register (offset: 0x118)
Name
Reserved
ECRC Check Enable
ECRC Check Capable
ECRC Generation Enable
ECRC Generation Capable
First Error Pointer
6
5
Bits Access
31:9 RO
8 RWS
7 RO
0
0
1
Default
Value
4:0
RWS
RO
ROS
0
1
0
Description
–
Setting this bit will enable ECRC checking.
When this bit is set, it indicates that this device supports ECRC checking.
Setting this bit will enable ECRC generation.
When this bit is set, it indicates that this device supports ECRC generation.
This value indicates the bit position within the
“Uncorrectable Error Status Register” 0x104.
Header Log Register (offset: 0x11C)
Name
Header Byte 0
Header Byte 1
Header Byte 2
Header Byte 3
Bits Access
31:24 ROS
Default
Value
–
23:16
15:8
7:0
ROS
ROS
ROS
–
–
–
Description
The TLP header of the transaction that has incurred a failure.
The TLP header of the transaction that has incurred a failure.
The TLP header of the transaction that has incurred a failure.
The TLP header of the transaction that has incurred a failure.
Header Log Register (offset: 0x120)
Name
Header Byte 4
Header Byte 5
Header Byte 6
Header Byte 7
Bits Access
31:24 ROS
Default
Value
–
23:16
15:8
7:0
ROS
ROS
ROS
–
–
–
Description
The TLP header of the transaction that has incurred a failure.
The TLP header of the transaction that has incurred a failure.
The TLP header of the transaction that has incurred a failure.
The TLP header of the transaction that has incurred a failure.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 306
BCM5718 Programmer’s Guide PCI Configuration Registers
Header Log Register (offset: 0x124)
Name
Header Byte 8
Header Byte 9
Header Byte 10
Header Byte 11
Bits Access
31:24 ROS
Default
Value
–
23:16
15:8
7:0
ROS
ROS
ROS
–
–
–
Description
The TLP header of the transaction that has incurred a failure.
The TLP header of the transaction that has incurred a failure.
The TLP header of the transaction that has incurred a failure.
The TLP header of the transaction that has incurred a failure.
Header Log Register (offset: 0x128)
Name
Header Byte 12
Header Byte 13
Header Byte 14
Header Byte 15
Bits Access
31:24 ROS
Default
Value
–
23:16
15:8
7:0
ROS
ROS
ROS
–
–
–
Description
The TLP header of the transaction that has incurred a failure.
The TLP header of the transaction that has incurred a failure.
The TLP header of the transaction that has incurred a failure.
The TLP header of the transaction that has incurred a failure.
Interrupt mail box (High Priority Mailbox) Register
(offset: 0x200 - 0x21c)
This mailbox serves two functions. When the host writes it, the interrupt (IntA) is cleared. It is also used by the
Host Coalescing engine to determine if the host is in the interrupt handler. If it is non-zero this indicates the host is in the interrupt handler. If it is zero this indicates the host is not in the interrupt handler. The Host
Coalescing engine uses this information to determine which set of coalescing parameters it should use.
General mail box (High Priority Mailbox)
Register (offset: 0x220 - 0x25c)
Reload Statistics mail box (High Priority Mailbox)
Register (offset: 0x260 - 0x264)
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 307
BCM5718 Programmer’s Guide High Priority Mailbox Registers
High Priority Mailbox Registers
All registers reset are core reset unless specified.
Note: When performing a 32 bit access the higher addresses should be accessed last.
Receive BD Standard Producer Ring Index
Register (offset: 0x268-0x26F)
Name
Received BD standard
Producer Ring Index
Bits Access
7:0 RW
Default
Value
0
Description
The Receive BD standard Producer Ring Index register contains the index of the next buffer descriptor for the standard producer ring that will be produced in the host for the NIC to DMA into
NIC memory. Host software writes this register whenever it updates the standard producer ring.
This register must be initialized to 0.
Receive BD Jumbo Producer Ring Index Register (offset: 0x270)
Name
Received BD jumbo Producer
Ring Index
Table 98: Receive BD Jumbo Producer Ring Index Register (offset: 0x270)
Bits
7:0
Access
RW
Default
Value
0
Description
The Receive BD Extended Producer Ring Index register contains the index of the next buffer descriptor for the extended producer ring that will be produced in the host for the controller to DMA into controller memory. Host software writes this register whenever it updates the extended producer ring. This register must be initialized to
0.
Receive BD Return Ring 0 Consumer Index
Register (offset: 0x280–0x287)
The Receive BD Return Ring 0 Consumer Index Register contains the index of the last buffer descriptor for
Receive Return Ring 0 that has been consumed. Host software writes this register whenever it updates the return ring 1. This register must be initialized to 0.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 308
BCM5718 Programmer’s Guide RX Mail Box Registers for VRQ
Receive BD Return Ring 1 Consumer Index
Register (offset: 0x288–0x28F)
The Receive BD Return Ring 1 Consumer Index Register contains the index of the last buffer descriptor for
Receive Return Ring 1 that has been consumed. Host software writes this register whenever it updates the return ring 1. This register must be initialized to 0.
Receive BD Return Ring 2 Consumer Index
Register (offset: 0x290–0x297)
The Receive BD Return Ring 2 Consumer Index Register contains the index of the last buffer descriptor for
Receive Return Ring 2 that has been consumed. Host software writes this register whenever it updates the return ring 2. This register must be initialized to 0.
Receive BD Return Ring 3 Consumer Index
Register (offset: 0x298–0x29F)
The Receive BD Return Ring 3 Consumer Index Register contains the index of the last buffer descriptor for
Receive Return Ring 3 that has been consumed. Host software writes this register whenever it updates the return ring 3. This register must be initialized to 0.
Send BD Ring Host Producer Index Register (offset: 0x300–0x307)
The Send BD Ring Host Producer Index Register contains the index of the next buffer descriptor for a given standard (non-jumbo) send ring that will be produced in the host for the NIC to DMA into NIC memory. Host software writes this register whenever it updates the given send ring. This register must be initialized to 0.
RX Mail Box Registers for VRQ
A set of new High Priority Mail Box Registers have been introduced. The register addresses are as shown in
below:
Table 99: High Priority Mail Box Registers for VRQ Rings
1
2
3
VRQ #
RX Jumbo Ring
Producer Index
RX Standard Ring
Producer Index
RX Return Ring
Consumer Index Comments
0 0x270
0x2D4
0x2D8
0x2DC
UNDI - 0x98
HP - 0x268
0x354
0x358
0x35C
NIC Diagnostic
RX Return Ring
Producer Index
UNDI - 0x88
HP - 0x280
0x288 (64-bit Regs)
64-bit Registers 0x3C80
0x3C84
0x290 (64-bit Regs) The new Registers are 32-bit 0x3C88
0x298 (64-bit Regs) 0x3C8C
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 309
BCM5718 Programmer’s Guide Ethernet MAC (EMAC) Registers
Table 99: High Priority Mail Box Registers for VRQ Rings (Cont.)
VRQ #
RX Jumbo Ring
Producer Index
RX Standard Ring
Producer Index
RX Return Ring
Consumer Index Comments
6
7
4
5
0x2E0
0x2E4
0x2E8
0x2EC
8
9
0x2F0
0x2F4
10 0x2F8
11 0x2FC
12 0x340
13 0x344
14 0x348
15 0x34C
16 0x350
0x360
0x364
0x368
0x36C
0x370
0x374
0x378
0x37C
0x380
0x384
0x388
0x38C
0x390
0x2A0
0x2A4
0x2A8
0x2AC
0x2B0
0x2B4
0x2B8
0x2BC
0x2C0
0x2C4
0x2C8
0x2CC
0x2D0
NIC Diagnostic
RX Return Ring
Producer Index
0x3C90
0x3C94
0x3C98
0x3C9C
0x3CA0
0x3CA4
The new Registers are 32-bit 0x3CA8
0x3CAC
0x3CB0
0x3CB4
0x3CB8
0x3CBC
0x3CC4
Additional Notes:
• 0x268 - 0x298, 0x88, 0x98 are legacy registers
• NIC Diagnostic Standard RBD Consumer Index [1 - 16] == 0x3F40 - 0x3F7C
• NIC Diagnostic Jumbo RBD Consumer Index [1 - 16] == 0x3F80 - 0x3FFC
Ethernet MAC (EMAC) Registers
All registers reset are core reset unless specified.
EMAC Mode Register (offset: 0x400)
Name Bits Access
Reserved 31:30 RO
Mac loop back mode control 29 RW
0
1
Default
Value
Enable APE TX path
Enable APE RX path
28
27
RW
RW
0
0
Description
–
1: gate off outgoing TX data path when emac loopback mode is enabled.
0: TX data will show up in normal functional path as well as MAC loopback path.
This bit must be written a 1 for the EMAC to transmit APE packets.
This bit must be written a 1 for APE subsystem to receive packets from the EMAC
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 310
BCM5718 Programmer’s Guide Ethernet MAC (EMAC) Registers
Name
Free Running ACPI
Halt Interesting packet PME
Keep Frame in WOL
Enable FHDE
Enable RDE
Enable TCE
Reserved
ACPI Power-on Enable
Magic Packet Detection Enable 18
Send Config Command
Flush TX statistics
Clear TX statistics
Enable TX Statistics
Flush RX Statistics
Clear RX Statistics
Enable RX Statistics
Reserved
Max Defer
Enable TX Bursting
Tagged MAC Control
Reserved
Loopback mode
Bits Access
26 RW
Default
Value
0
25
24
23
22
21
20
19
17
16
15
14
13
12
11
10
9
8
7
6:5
4
RW
RW
RW
RW
RW
RO
RW
RW
RW
RW
RW
RW
RW
RW
RW
RO
RW
RW
RW
RO
RW
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
0
Description
When this bit is set, the ACPI state machine will continue running when a match is found. When this bit is clear, the ACPI state machine will halt when a match is found.
When this bit is set, the WOL signal will not be asserted on an interesting packet match.
–
Enable receive Frame Header DMA engine. Must be set for normal operation.
Enable RDMA engine.
Must be set for normal operation.
Enable Transmit DMA engine.
–
Enable Wake on LAN filters when in powerdown mode
Enable Magic Packet detection
Send config commands when in TBI mode
Write transmit statistics to external memory.
This bit is self-clearing.
Clear transmit statistics internal RAM.
This bit is self-clearing.
Enable transmit statistics external updates
Write receive statistics to external memory.
This bit is self-clearing.
Clear receive statistics internal RAM.
This bit is self-clearing.
Enable receive statistics external updates.
–
Enable Max Deferral checking statistic.
Enable transmit bursting in gigabit half-duplex mode.
Allow the MAC to receive tagged MAC control packets.
–
When set, an internal loopback path is enabled from the transmit MAC to the receive MAC. This bit is provided for diagnostic purposes only.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 311
BCM5718 Programmer’s Guide Ethernet MAC (EMAC) Registers
Name
Port Mode
Half-duplex
Global Reset
Bits Access
3:2 RW
Default
Value
01
1
0
RW
RW
0
0
Description
11: Obsolete (Was TBI)
10: 1000BT(Copper) GMII or Serdes 1000 mode
01: 10/100BT(Copper) MII or Serdes 10/100 mode
00: None
When set, the MII/GMII interface is configured to operate in half-duplex mode and the CSMA/
CD state machines in the MAC are set to halfduplex mode.
When this bit is set to 1, the MAC state machine is reset.
This is a self-clearing bit.
EMAC Status Register (offset: 0x404)
Name
Reserved
Reserved.
Interesting packet PME
Attention
TX Statistic Overrun
RX Statistic Overrun
ODI Error
AP Error
MII Interrupt
MII Completion
Reserved
Bits Access
31:30 RO
29 RO
28 W2C
0
0
0
Default
Value
27
26
25
24
23
22
21:13
W2C
W2C
RO
RO
RO
W2C
RO
0
0
0
0
0
0
0
Description
–
–
When this bit is set, the WOL signal is asserted when an interesting packet is detected.
Transmit Statistics block has overrun.
Generates an attention when enabled.
Receive Statistics block has overrun.
Generates an attention when enabled.
Output Data Interface block has an overrun or underrun.
Will generate attention when enabled.
Clear this attention using the Transmit Status register.
Auto-polling interface needs service.
Generates an attention when enabled. Clear this attention using the Auto-polling Status register
Management interface is signaling an interrupt
Generates an attention when enabled.
Management interface transaction has completed.
Generates an attention when enabled.
–
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 312
BCM5718 Programmer’s Guide Ethernet MAC (EMAC) Registers
Name
Link State Changed
Reserved
Bits Access
12 RO
Default
Value
0
11:0 RO 0
Description
Set when the link state has changed
Generates an attention when enabled by bit 12 of the EMAC Event Enable register.
Clear this attention by writing 1 to Sync Changed
(bit 4) and Config changed (bit 3).
–
EMAC Event Enable Register (offset: 0x408)
Name
Reserved
Enable TX Offload Error
Interrupt
Interesting packet PME
Attention Enable
TX Statistics Overrun
RX Statistics Overrun
ODI Error
AP Error
MII Interrupt
MII Completion
Reserved
Link State Changed
Reserved
Bits Access
31:30 RO
29 RW
0
0
Default
Value
28
27
26
25
24
23
22
21:13
12
11:0
RW
RW
RW
RW
RW
RW
RW
RO
RW
RO
0
0
0
0
0
0
0
0
0
0
Description
–
Enables or unmasks the interrupt associated with Reg 0x404[29].
When this bit is set, an attention will be asserted on an interesting packet match.
Enable attention when transmit statistics block has overrun.
Enable attention when receive statistics block has overrun.
Enable attention when an output data interface block has an overrun or underrun.
Enable attention when the auto-polling interface has an error.
Enable attention when the Management
Interface is signaling an interrupt.
Enable attention when the Management
Interface transaction has completed.
–
Enable attention when the link has changed state.
–
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 313
BCM5718 Programmer’s Guide Ethernet MAC (EMAC) Registers
LED Control Register (offset: 0x40C)
Name
Override Blink Rate
Blink Period
Reserved
Reserved
Shared Traffic/Link LED mode 14
MAC Mode
LED Mod
Traffic LED Status
10 Mbps LED Status
Bits Access
31 RW
Default
Value
1
30:19
18:16
15
13
12:11
10
9
RW
RO
RO
RW
RW
RW
RO
RO
000001
000000
0
0
1
0
01
0
0
Description
–
–
If set, the blink rate for the Traffic LED is determined by the Blink Period field (bit 30 to bit
9). This bit is rest to 1.
If not set, the blink rate assumes a Blink Period of 0x040, corresponding to approximately
15.9Hz.
Specifies the period of each blink cycle (on+off) for Traffic LED in milliseconds. Must be a nonzero value.
This 12-bit field is reset to 0x040, giving a default blink period of approximately 15.9Hz.
When this bit is set, the Link LED is solid green when there is a link and blinks when there is traffic.
(The LED_MODE field must be set to 00 before enabling this bit).
When this bit is set, the traffic LED blinks only when traffic is addressed for the device (The
LED_MODE field must be set to 00 before enabling this bit).
–
–
00: MAC mode–LED signal is in active low (on) when link is established and is in high (off) when link is not established.
01: PHY mode 1–LED signal is in active low (on) when link is established and is in tristate (off) when link is not established
• LINKLEDB = Link 10 (open drain)
• SPD100LEDB = Link 100 (open drain)
• SPD1000LEDB = Link10000 (open drain)
• TRAFFICLEDB = PHY RCVLED or PHY XMTLED
10: PHY mode 2–LED signal is in active low (on) when link is established and is in high (off) when link is not established.
• LINKLEDB = Link 10
• SPD100LEDB = Link 100 and valid data or idle
• SPD1000LEDB = Link10000 and valid data or idle
• TRAFFICLEDB = PHY RCVLED or PHY XMTLED
11: Same as PHY mode 1
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 314
BCM5718 Programmer’s Guide Ethernet MAC (EMAC) Registers
Name
100 Mbps LED Status
1000 Mbps LED Status
Traffic LED
Blink Traffic LED
Override Traffic LED
10 Mbps LED
100 Mbps LED
1000 Mbps LED
Override Link LEDs
Bits Access
8
7
6
RO
RO
RW
0
0
0
Default
Value
5
4
3
2
1
0
RW
RW
RW
RW
RW
RW
0
0
0
0
0
0
Description
–
–
If set along with the Override Traffic bit, the
Traffic LED is turned on. If the Blink Traffic LED bit is also set, the LED will blink with blink rate specified in Override Blink Rate (bit 31) and Blink
Period (bits 30–19) fields.
If set along with the Override Traffic bit and
Traffic LED bit, the Traffic LED will blink with the blink rate specified in Override Blink Rate (bit 31) and Blink Period (bits 30–19) fields.
If set, overrides hardware control of the Traffic
LED.
The Traffic LED will then be controlled via bit 6 and bit 5.
If set along with the LED Override bit, turns on the 10 Mbps LED.
If set along with the LED Override bit, turns on the 100 Mbps LED.
If set along with the LED Override bit, turns on the 1000 Mbps LED.
If set, overrides hardware control of the three link LEDs. The LEDs will be controlled via bits 3–
1.
EMAC MAC Addresses 0 High Register (offset: 0x410)
Name
Reserved
MAC Address High
Bits Access
31:16 RO
15:0 RW
0
0
Default
Value Description
–
Upper 2-bytes of this node's MAC address.
EMAC MAC Addresses 0 Low Register (offset: 0x414)
Name
MAC Address Low
Bits Access
31:0 RW
Default
Value
0
Description
Lower 4-byte of this node's MAC address.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 315
BCM5718 Programmer’s Guide Ethernet MAC (EMAC) Registers
EMAC MAC Addresses 1 High Register (offset: 0x418)
Name
Reserved
MAC Address High
Bits Access
31:16 RO
15:0 RW
0
0
Default
Value Description
–
Upper 2-bytes of this node's MAC address.
EMAC MAC Addresses 1 Low Register (offset: 0x41C)
Name
MAC Address Low
Bits Access
31:0 RW
Default
Value
0
Description
Lower 4-byte of this node's MAC address.
EMAC MAC Addresses 2 High Register (offset: 0x420)
Name
Reserved
MAC Address High
Bits Access
31:16 RO
15:0 RW
0
0
Default
Value Description
–
Upper 2-bytes of this node's MAC address.
EMAC MAC Addresses 2 Low Register (offset: 0x424)
Name
MAC Address Low
Bits Access
31:0 RW
Default
Value
0
Description
Lower 4-byte of this node's MAC address.
EMAC MAC Addresses 3 High Register (offset: 0x428)
Name
Reserved
MAC Address High
Bits Access
31:16 RO
15:0 RW
0
0
Default
Value Description
–
Upper 2-bytes of this node's MAC address.
EMAC MAC Addresses 3 Low Register (offset: 0x42C)
Name
MAC Address Low
Bits Access
31:0 RW
Default
Value
0
Description
Lower 4-byte of this node's MAC address.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 316
BCM5718 Programmer’s Guide Ethernet MAC (EMAC) Registers
WOL Pattern Pointer Register (offset: 0x430)
Name
Reserved
ACPI Pointer
Bits Access
31:9 RO
8:0 RW
0
0
Default
Value Description
–
Specifies the offset into the 6 KB BD memory for frame comparison. (Bits 3:0 are ignored to align the memory address to a natural 128-bit boundary).
WOL Pattern Configuration Register (offset: 0x434)
Name
Reserved
ACPI Offset
Reserved
ACPI Length
Bits Access
31:28 RO
27:16 RW
0
0
Default
Value
15:10
9:0
RO
RW
0
0
Description
–
Offset of a frame where the pattern comparison starts.
–
Specifies the total number of 64-bit double words inside the MISC_BD memory that are valid for ACPI.
For GMII, it should have a value of 2, 4, 6,…
For MII, it should have a value of 3, 6, 9, …
Ethernet Transmit Random Backoff Register (offset: 0x438)
Name
Reserved
Random Backoff Seed
Bits Access
31:10 RO
9:0 RW
0
0
Default
Value Description
–
For half-duplex, initialize with any nonzero seed.
Receive MTU Size Register (offset: 0x43C)
Name
Reserved
MTU
Bits Access
31:16 RO
15:0 RW
Default
Value
0
05F2h
Description
–
2-byte field which is the largest size frame that will be accepted without being marked as oversize.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 317
BCM5718 Programmer’s Guide Ethernet MAC (EMAC) Registers
Gigabit PCS Test Register (offset: 0x440)
Name
Reserved
Bits Access
31:0 RO
Default
Value
0
Description
–
Transmit 1000BASE-X Auto-Negotiation Register (offset: 0x444)
Name
Reserved
Bits Access
31:0 RO
Default
Value
0
Description
–
Receive 1000BASE-X Auto-Negotiation Register (offset: 0x448)
Name
Reserved
Bits Access
31:0 RO
Default
Value
0
Description
–
MII Communication Register (offset: 0x44C)
Name
Reserved
Start/Busy
Read Failed
Command
PHY Addr
Register Address
Bits Access
31:30 RO
29 RW
0
0
Default
Value
28
27:26
25:21
20:16
RO
RW
RW
RW
0
0
0
0
Description
–
Set this bit to start a transaction.
While it is high, it indicates that the current transaction is still ongoing.
If enabled, generates an attention via EMAC
Status Register MI Completion bit (bit 22).
When set, the transceiver device did not driver the bus during the attempted read transaction.
Valid after the Start/Busy bit is cleared.
These bits specify the transaction type:
11: Undefined
10: Read command
01: Write command
00: Undefined
0x8: SGMII Serdes Port0.
0x9: SGMII Serdes Port1.
as strapped: External PHY Port0 as strapped: External PHY Port1
Address of the register to be read or written.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 318
BCM5718 Programmer’s Guide Ethernet MAC (EMAC) Registers
Name
Transaction Data
Bits Access
15:0 RW
Default
Value
0
Description
When configured for a write command, the data stored at this location is written to the PHY at the specified PHY and register address.
During a read command, the data returned by the PHY is stored at this location.
MII Status Register (offset: 0x450)
Name
Reserved
Mode 10 Mbps
Link Status
Bits Access
31:2 RO
1 RW
0
0
Default
Value
0 RW 0
Description
–
When read, a value of 1 indicates the transceiver device is operating in 10 Mbps mode
The bit will generate an attention if enabled.
Indicates status of the link on the transceiver device.
When read, a value of 1 indicates the transceiver is linked
MII Mode Register (offset: 0x454)
Name
Reserved
MII Clock Count
Constant MDIO/MDC clock speed.
Reserved
PHY Address
Port polling
Reserved
Bits Access
31:21 RO
20:16 RW
Default
Value
0
0Ch
15
14:10
9:5
4
3
RW
RO
RW
RW
RO
0
0
1
0
0
Description
–
Counter to divide CORE_CLK (62.5 MHz) to generate the MI clock.
The formula is:
MI Clock = CORE_CLK/2/(MI Clock Count + 1).
Enable ~500Khz constant MII management interface (MDIO/MDC) frequency regardless core clock frequency.
1: Enable
0: Disable
–
This field specifies the PHY Address.
Set to enable autopolling of the transceiver link information from the MII management interface.
If cleared, the device will obtain the link status information from the state of the LINKRDY input signal.
–
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 319
BCM5718 Programmer’s Guide Ethernet MAC (EMAC) Registers
Name
Auto_control
Use Short Preamble
Fast_Clock
Bits Access
2
1
0
RW
RW
RW
0
1
0
Default
Value Description
–
Use short preamble while polling, if set.
–
Autopolling Status Register (offset: 0x458)
Name
Reserved
Auto-polling Error
Bits Access
31:1 RO
0 W2C
0
0
Default
Value Description
–
Indicates an autopolling error occurred, if set.
Transmit MAC Mode Register (offset: 0x45C)
Name
RR Weight
Transmit FTQ Arbitration
Mode
Bits Access
31:27 RW
Default
Value
00000
26:24 RW 000
Description
This field may be programmed to assign a weight to the “Weighted Round Robin” arbitration mode. This field is applicable only when the appropriate arbitration mode is chosen, i.e.,
[19:17] of this register is equal to “001”.
This field determines the arbitration mode of the TCE block among LAN traffic and APE traffic as below:
000–Simple Round Robin
001–Weighted Round Robin
010–Shut off APE transmit stream
011–Shut off LAN transmit stream
1xx–Reserved for future use
Caution: This field must remain static following boot.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 320
BCM5718 Programmer’s Guide Ethernet MAC (EMAC) Registers
Name
HTX2B Count Down Mode
HTX2B Programmable Jumbo
Frame Length
Reserved
TX-MBUF Burst Size
Do not insert GCM/GMAC IV
Do not drop if packet found malformed
Do not drop if SA found in RX direction
22
15
14
Do not drop if unsupported
IPV6 extension found or IPv4 option found
Do not drop if SA invalid
13
12
Do not drop if AH/ESP Header not found
11
Bits Access
23 RW
Default
Value
0
21
20:17
16
RW
–
RW
RW
RW
RW
RW
RW
RW
0
–
0000
0
0
0
0
0
0
Description
When this bit is 0:
If there is an HTX2B packet to be enqueued to the HTX2B, FIFO. However, if the FIFO does not have sufficient space, TCE drops the copy of the
HTX2B-bound packet and moves on.
When this bit is 1:
If there is an HTX2B packet to be enqueued to the HTX2B, FIFO. However, if the FIFO does not appear to have sufficient space at the first attempt, TCE counts down a few clocks, then makes a second attempt. If the FIFO still does not have sufficient space, TCE simply drops the copy of the HTX2B-bound packet and moves on.
The number of clocks to count-down is programmable at register 0x464[31:24].
When this bit is 0:
Hardware automatically applies preset lengths to detect jumbo frames for HTX2B.
When this bit is 1:
The jumbo frame length for HTX2B is programmable via 0x464[23:16].
–
This filed determines the size of the MA read performed by TCE.
0000 => burst-size 16
0001–01111 => reserved
1000 => burst-size 8
1001 => burst-size 9
………
1111 => burst-size 15
If this bit is 0, an IV is generated by the chip and inserted in an offloaded TX ESP or TX AH packet in GCM or GMAC Cipher mode.
If this bit is 1, an IV is not inserted. Instead, the
IV is extracted from the TX frame.
These bit are there for debug purposes.
Normally an offloaded TX packet that does not adhere to BCM5718 family limitations or does not associate with a valid SA, unless the bit corresponding to the error symptom is set to 1 here; in that case the particular error is overlooked and the packet is transmitted in clear text.
When such a packet is dropped in the chip, an interrupt is generated.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 321
BCM5718 Programmer’s Guide Ethernet MAC (EMAC) Registers
Name
Enable TX AH Offload
Enable TX ESP Offload
TxMBUF corruption lockup fix enable
8
Link Aware Enable 7
Enable Long Pause
Enable Big Backoff
Enable Flow Control
Reserved
Enable TCE
Reset
Bits Access
10 RW
Default
Value
0
9
6
5
1
0
4
3:2
RW
RW
RW
RW
RW
RW
RO
RW
RW
0
0
0
0
0
0
0
1
0
Description
A value 1 enables the TX AH offload feature.
When 0, offloaded AH packet gets dropped. This value must be static.
A value 1 enables the TX ESP offload feature.
When 0, offloaded ESP packet gets dropped.
This value must be static.
When set, TXMBUF corruption lockup fix is enabled.
When set, transmission of packets by the MAC is enabled only when link is up.
When set, the Pause time value set in the transmitted PAUSE frames is 0xFFFF.
The default value for PAUSE time is 0x1FFF
MAC will use larger than normal back-off algorithm.
MAC will send 802.3x flow control frames.
–
Used to be enable TDE in legacy–same purpose.
When this bit is set to 1, the Transmit MAC state machine will be reset.
This is a self-clearing bit.
Transmit MAC Status Register (offset: 0x460)
Name
Reserved
Consumer Index
ODI Overrun
ODI Underrun
Link Up
Sent XON
Sent XOFF
RX Currently XOFFed
3
2
5
4
1
0
Bits Access
31:6 RO
22:11 RO
Default
Value
0
UUUU
W2C
W2C
RO
W2C
W2C
RO
0
0
0
0
0
0
Description
–
The Consumer Index of the erring packet is reported by this field.
Output data interface has overrun.
Output data interface has underrun.
Link is up, if set.
An XON flow control frame was sent.
An XOFF flow control frame was sent.
Received stopped due to flow control.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 322
BCM5718 Programmer’s Guide Ethernet MAC (EMAC) Registers
Transmit MAC Lengths Register (offset: 0x464)
Name Bits Access
HTX2B Count Down Value 31:24 RW
HTX2B Jumbo Frame Length 23:16 RW
Default
Value
0x1
0x0
Description
HT2XB Count Down Value
This value + 1500 is used by hardware as the maximum standard frame length for HTX2B. A frame with a length larger than that is a jumbo frame for HTX2B.
The length is the effective length of a composed
L2 frame as seen from the wire, including the L2 header, L2 payload, and the FCS (CRC) field.
Reserved
IPG CRS Length
15:14 RO
13:12 RW
0
0
IPG Length
Slot Time Length
11:8
7:0
RW
RW
0
0
When multiplied by 2, this field indicates the number of bytes from the end of the interpacket gap (IPG) during which incoming carrier is ignored.
When multiplied by 2, this field indicates the number of bytes in the entire IPG.
When multiplied by 2, this field indicates the number of bytes in the slot time.
Receive MAC Mode Register (offset: 0x468)
Name
FIX EMAC drops first packet on false carrier event
Bits Access
31 RW
Default
Value
0
Reserved
Disable 802.3 length check fix for VLAN Tag frames
30
29
Reset Management Filter Set 28
Enable RX AH Offload
Enable RX ESP Offload
APE promiscuous mode enable
27
26
25
RO
RW
WO
RW
RW
RW
0
0
0
0
0
0
Description
This bit when set disables the fix where EMAC drops first packet on False Carrier Event.
1: Disable Fix
0: Enable Fix
–
If clear, 802.3 length check takes VLAN length into account properly.
Writing a 1 to this field generates a pulse to reset all Management Filter registers.
A value 1 enables the RX AH offload feature.
When 0, in coming packets are not CAMed for offload consideration.
A value 1 enables the RX ESP offload feature.
When 0, in coming packets are not CAMed for offload consideration.
When set, no source address or MC hashing checking will be performed on incoming frames on APE filter path.
All frames will be accepted and subject to
Management filter actions.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 323
BCM5718 Programmer’s Guide Ethernet MAC (EMAC) Registers
Name
IPV6 Enable
RSS_enable
RSS Hash Mask Bits
RSS TCP/IPV6 Hash Enable
RSS IPV6 Hash Enable
RSS TCP/IPV4 Hash Enable
RSS IPV4 Hash Enable
Reserved
FIX EMAC drops a packets if incoming DA partial match both perfect and Pause
Multicast address
Filter Broadcast
Keep VLAN Tag Diag mode
No CRC Check
Promiscuous mode
Length Check
Bits Access
24 RW
Default
Value
0
23
22:20
19
18
17
16
RW
RW
RW
RW
RW
RW
15:13 RO
12
11
10
9
8
7
RW
RW
RW
RW
RW
RW
1
0x7
0
0
0
0
0
0
0
0
0
0
0
Description
1: Enable IPv6 RX
0: Disable IPv6 RX which includes IPv6 packet parsing, checksum offload and IPv6 RSS
1: Enable RSS function.
0: Disable RSS function. FHDE will ignore the
RSS_valid from Frame Cracker and set RSS_valid to be 0 in frame descriptor of each packet.
These bits specify the number of hash bits that are used to offset into the indirection table. A value of one specifies that only bit 0 of the hash is used to offset into the indirection table (so only the first two entries of the table are utilized.) A value of seven specifies that bits 6:0 of the hash are used to offset into the indirection table. A value of zero will result in undefined behavior and should not be programmed.
When this bit is set, 4-tuple hashes are enabled for TCP over IPV6 packets. This bit should be set to 0 if IPv6 RX is disabled.
When this bit is set, 2-tuple hashes are enabled for IPV6 packets. This bit should be set to 0 if
IPv6 RX is disabled.
When this bit is set, 4-tuple hashes are enabled for TCP over IPV4 packets.
When this bit is set, 2-tuple hashes are enabled for IPV4 packets.
–
This bit disables the fix that EMAC stage fsm drops a packet if incoming packet's DA has a partial match in both perfect match address and
Pause Multicast address.
When set, reception of broadcast frames is disabled
If set, forces Receive MAC to keep the VLAN tag in the frame.
This is for debugging purpose only and should be reset during normal operation
When set, no CRC check by receive MAC on incoming frames.
Also, allows the reception of packets received with RXERR on MII/GMII.
When set, no source address or MC hashing checking will be performed on incoming frames.
All frames will be accepted.
If set, 802.2 length checking is done on LLC frames.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 324
BCM5718 Programmer’s Guide
Name
Accept Runts
Reserved
Keep Pause
Reserved
Enable Flow Control
Enable
Reset
Ethernet MAC (EMAC) Registers
Bits Access
4
3
6
5
2
RW
RO
RW
RO
RW
0
0
0
0
0
Default
Value
1
0
RW
RW
0
0
Description
If set, MAC accepts packets less than 64 bytes.
–
If set, MAC forwards pause frame to host buffer.
–
Enable automatic processing of 802.3x flow control frames.
This bit is orthogonal to the Keep Pause bit.
This bit controls whether the Receive MAC state machine is active or not.
When set to 0, it completes the current operation and cleanly halts. Until it is completely halted, it remains 1 when read.
When this bit is set to 1, the Receive MAC state machine will be reset.
This is a self-clearing bit.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 325
BCM5718 Programmer’s Guide Ethernet MAC (EMAC) Registers
Receive MAC Status Register (offset: 0x46C)
Name
Reserved
RX FIFO Overrun
XON received
XOFF received
Remote Transmitter XOFFed
Bits Access
31:4 RO
3 W2C
2 W2C
0
0
0
Default
Value
1
0
W2C
RO
0
0
Description
–
RX FIFO has encountered an overrun condition.
MAC control frame with the PAUSE opcode was received with PAUSE TIME field set to zero.
The bit is sticky and must be written to clear.
MAC control frame with the PAUSE opcode was received with PAUSE TIME field set to nonzero.
The bit is sticky and must be written to clear.
A previously received XOFF timer has not expired yet.
MAC Hash Register 0 (offset: 0x470)
Name
Hash value
Bits Access
31:0 RW
Default
Value
0
Description
Hash value for multicast destination address matching.
MAC Hash Register 1 (offset: 0x474)
Name
Hash value
Bits Access
31:0 RW
Default
Value
0
Description
Hash value for multicast destination address matching.
MAC Hash Register 2 (offset: 0x478)
Name
Hash value
Bits Access
31:0 RW
Default
Value
0
Description
Hash value for multicast destination address matching.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 326
BCM5718 Programmer’s Guide Ethernet MAC (EMAC) Registers
MAC Hash Register 3 (offset: 0x47C)
Name
Hash value
Bits Access
31:0 RW
Default
Value
0
Description
Hash value for multicast destination address matching.
Receive Rules Control Registers (offset: 0x480 + 8*N)
The BCM5718 family employs eight receive rules (N = 0 to 7).
Name
Enable
And With Next
Activate Processor 1
Activate Processor 2
Activate Processor 3
Mask
Discard
Map
Reserved for future use
Comparison Operator
Header Type
Bits Access
31
30
RW
RW
0
0
Default
Value
29
28
27
26
25
24
23:18
17:16
15:13
RW
RW
RW
RW
RW
RW
RW
RW
RW
0
0
0
0
0
0
0
0
0
Description
Corresponding Rule is enabled when set.
This rule and next must both be true to match.
The class fields must be the same.
A disabled next rule is considered true.
Processor activation bits are specified in the first rule in series.
If the rule matches, the processor is activated in the queue descriptor for the Receive Queue
Placement state machine.
If the rule matches, the processor is activated in the queue descriptor for the Receive Queue
Placement state machine.
If the rule matches, the processor is activated in the queue descriptor for the Receive Queue
Placement state machine.
IF set, specifies that the value/mask field is split into a 16-bit mask instead of a 32bit value.
Discard frame if it matches the rule.
Use the masked value and map it to the class.
–
Specifies how to determine the match:
00: Equal
01: Not Equal
10: Greater Than
11: Less Than
Specifies which header the offset is for:
000: Start of Frame (always valid)
001: Start of IP Header (if present)
010: Start of TCP Header (if present)
011: Start of UDP Header (if present)
100: Start of Data (always valid, context sensitive)
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 327
BCM5718 Programmer’s Guide Ethernet MAC (EMAC) Registers
Name
Class
Offset
Bits Access
12:8 RW
Default
Value
0
7:0 RW 0
Description
The class this frame is place into if the rule matches. 0-16 where 0 means discard.
The number of valid classes is the number of active queues divided by the Number of
Interrupt Distribution Groups.
Ring 1 has the highest priority.
Number of bytes offset specified by the header type.
Receive Rules Value/Mask Registers (offset: 0x484 + 8*N)
The BCM5718 family employs eight receive rules (N = 0 to 7).
Name
Mask/Value
Value
Bits Access
31:16 RW
Default
Value
0
15:0 RW 0
Description
For each bit set, the corresponding bit in the value field is ignored during the rule match process.
If bit 26 of the corresponding rule control register is set, the field is used as an additional
16-bit value for rule comparison.
This field specifies a 16-bit value for rule comparison.
Receive Rules Configuration Register (offset: 0x500)
Name
Reserved
No Rules Matches Default
Class
Reserved
Bits Access
31:6 RO
5:3 RW
0
0
Default
Value
2:0 RO 0
Description
–
Specifies the default class of service for the frame if no rules are matched.
A value of 1 is the highest priority.
A value of zero will cause the frame to be discarded.
–
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 328
BCM5718 Programmer’s Guide Ethernet MAC (EMAC) Registers
Low Watermark Maximum Receive Frame Register (offset: 0x504)
Name
Reserved
TXFIFO Almost Empty
Threshold
Low Watermark Max Receive
Frames
Bits Access
31:21 RO
20:16 RW
Default
Value
0
0xC
15:0 RW 0
Description
–
When the remaining entries of TXFIFO are less than this threshold, TXFIFO_almost_empty will be asserted. This value is used in conjunction with Buffer Manager Mode register bit31 to prevent EMAC TXFIFO underrun.
Specifies the number of good frames to receive after RX MBUF Low Watermark has been reached. After the RX MAC receives this number of frames, it will drop subsequent incoming frames until the MBUF High Watermark is reached.
Default to zero (i.e., drop frames ones RX MBUF
Low Watermark is reached).
APE_PERFECT_MATCH[1–4]_HIGH_REG (Offsets 0x540, 0x548, 0x550,
0x558)
There are total 4 Perfect (Destination Address) Match registers dedicated to APE in RX-MAC. These registers hold the higher 2 octets of the matching address.
Name
Reserved
MAC High Address
Bits Access
31:29 RO
15:0 RW
Default
Value Description
000 –
0x0000 Upper 2-bytes of APE's [1–4]th unicast address.
APE_PERFECT_MATCH[1–4]_LOW_REG (Offsets 0x544, 0x54C, 0x554,
0x55C)
There are total 4 Perfect (Destination Address) Match registers dedicated to APE in RX-MAC. These registers hold the lower 4 octets of the matching address.
Name
MAC Low Address
Bits Access
31:0 RW
Default
Value Description
0x0000 Lower 4-bytes of APE's [1–4]th unicast address.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 329
BCM5718 Programmer’s Guide Ethernet MAC (EMAC) Registers
SGMII Control Register (offset: 0x5B0)
Name
Reserved
Bits Access
– –
Default
Value
–
Description
–
SGMII Status Register (offset: 0x5B4)
This register reflects various status of the respective SGMII port when enabled.
Name
LP AutoNeg Capability
Reserved
External CRS Detect
PCS CRS Detect
Media Selection mode
Pause TX
Pause RX
AutoNeg Next Page RX
Speed_100
Speed_1000
Duplex Status
Link Status
AutoNeg Completion
Bits Access
31:16 RO
Default
Value
0
15:11 RO
10
5
4
7
6
9
8
3
2
1
0
RO
RO
RO
RO
RO
RO
RO
RO
RO
RO
RO
0
0
0
0
0
0
0
0
0
0
0
0
Description
Link partner advertised auto-negotiation abilities.
–
External PHY's CRS output
Internal PCS blocks CRS output
1: SGMII/1000BaseX mode selected for this port
0: Copper Media Selected for this port
1: enable pause TX
1: enable pause RX
1: next auto-negotiation page received
The SGMII Link currently operable at 100mbps data speed.
The SGMII Link currently operable at 1 Gbps data speed.
1: The Link currently is in Full Duplex mode.
0: The Link is currently in Half Duplex mode.
1: Link is up.
0: Link is down.
Auto-negotiation process has completed.
HTX2B Perfect Match[1 – 4] HI Reg (offset: 0x4880, 0x4888, 0x4890,
0x4898)
There are four Perfect (Destination Address) Match registers in DMAR for HTX2B. These registers hold the higher two octets of the matching address.
Name
Reserved
MAC High Address
Bits Access
31:29 RO
15:0 RW
Default
Value Description
000 Reserved.
0x0000 Upper 2-bytes Destination Address.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 330
BCM5718 Programmer’s Guide Ethernet MAC (EMAC) Registers
HTX2B Perfect Match[1 – 4] LO Reg (offset: 0x4884, 0x488C, 0x4894,
0x489C)
There are four Perfect (Destination Address) Match registers in DMAR for HTX2B. These registers hold the lower two octets of the matching address.
Name
MAC Low Address
Bits Access
31:0 RW
Default
Value Description
0x0000 Lower 4-bytes of Destination Address.
HTX2B Protocol Filter Reg (offset: 0x6D0)
This register resides in TCE.
Name Bits Access
Unused
Duplicate – DHCPv6 Relay and
Server multicast
Duplicate – IPv6 Router
Advertisement
Duplicate – IPv6 Neighbor
Advertisement
Duplicate – NetBios Packet
Duplicate – DHCP Server
Packet
31:27
26
25
24
23
22
RO
RW
RW
RW
RW
RW
Duplicate – DHCP Client Packet 21
Duplicate – ARP Packet 20
19 Duplicate – HTX2B Perfect
Match Address[3]
Duplicate – HTX2B Perfect
Match Address[2]
18
17 Duplicate – HTX2B Perfect
Match Address[1]
Duplicate – HTX2B Perfect
Match Address[0]
Unused
16
15:11
RW
RW
RW
RW
RW
RW
RO
0
0
Default
Value
0x0
0
0
0
0
0
0
0
0
0
0x0
Description
Write a 1 to each bit to enable the respective
HTX2B protocol filter duplication.
Note: The respective Protocol Filter Enable bit must be 1 for a Duplication to be meaningful.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 331
BCM5718 Programmer’s Guide Ethernet MAC (EMAC) Registers
Name
Enable – DHCPv6 Relay and
Server multicast
Enable – IPv6 Router
Advertisement
Bits Access
10
9
Enable – IPv6 Neighbor
Advertisement
8
Enable – NetBios Packet 7
Enable – DHCP Server Packet 6
Enable – DHCP Client Packet 5
Enable – ARP Packet 4
3 Enable – HTX2B Perfect Match
Address[3]
Enable – HTX2B Perfect Match
Address[2]
Enable – HTX2B Perfect Match
Address[1]
2
1
Enable – HTX2B Perfect Match
Address[0]
0
RW
RW
RW
RW
RW
RW
RW
RW
RW
RW
RW
0
0
0
0
0
0
0
0
0
0
Default
Value
0
Description
Write a 1 to each bit to enable the respective
HTX2B protocol filter.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 332
BCM5718 Programmer’s Guide RSS Registers
HTX2B Global Filter Reg (address: 0x6D4)
This register resides in TCE.
Name Bits Access
Unused 31:2 RU
HTX2B Broadcast Filter Enable 1 RW
HTX2B Multicast Filter Enable 0 RW
Default
Value Description
0xUUUU Unused.
0 Write 1 to replicate and route all Host Send broadcast packets to APE. This bit overrides any conflicting Protocol Filter setting.
0 Write 1 to replicate and route all Host Send multicast packets to APE. This bit overrides any conflicting Protocol Filter setting.
RSS Registers
All registers reset are core reset unless specified.
Indirection Table Register 0 (offset: 0x630)
Name table_entry0 table_entry1 table_entry2 table_entry3 table_entry4 table_entry5 table_entry6 table_entry7
Bits Access
31:28 RW
27:24 RW
23:20 RW
19:16 RW
15:12 RW
11:8 RW
7:4
3:0
RW
RW
0
0
0
0
0
0
0
0
Default
Value Description
The RSS_ring value for entry 0.
The RSS_ring value for entry 1.
The RSS_ring value for entry 2.
The RSS_ring value for entry 3.
The RSS_ring value for entry 4.
The RSS_ring value for entry 5.
The RSS_ring value for entry 6.
The RSS_ring value for entry 7.
Indirection Table Register 2 (offset: 0x634)
Name table_entry8 table_entry9 table_entry10 table_entry11 table_entry12
Bits Access
31:28 RW
27:24 RW
23:20 RW
19:16 RW
15:12 RW
0
0
0
0
0
Default
Value Description
The RSS_ring value for entry 8.
The RSS_ring value for entry 9.
The RSS_ring value for entry 10.
The RSS_ring value for entry 11.
The RSS_ring value for entry 12.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 333
BCM5718 Programmer’s Guide
Name table_entry13 table_entry14 table_entry15
Bits Access
11:8 RW
7:4 RW
3:0 RW
0
0
0
Default
Value Description
The RSS_ring value for entry 13.
The RSS_ring value for entry 14.
The RSS_ring value for entry 15.
Indirection Table Register 3 (offset: 0x638)
Name table_entry16 table_entry17 table_entry18 table_entry19 table_entry20 table_entry21 table_entry22 table_entry23
Bits Access
31:28 RW
27:24 RW
23:20 RW
19:16 RW
15:12 RW
11:8 RW
7:4
3:0
RW
RW
0
0
0
0
0
0
0
0
Default
Value Description
The RSS_ring value for entry 16.
The RSS_ring value for entry 17.
The RSS_ring value for entry 18.
The RSS_ring value for entry 19.
The RSS_ring value for entry 20.
The RSS_ring value for entry 21.
The RSS_ring value for entry 22.
The RSS_ring value for entry 23.
Indirection Table Register 4 (offset: 0x63C)
Name table_entry24 table_entry25 table_entry26 table_entry27 table_entry28 table_entry29 table_entry30 table_entry31
Bits Access
31:28 RW
27:24 RW
23:20 RW
19:16 RW
15:12 RW
11:8 RW
7:4
3:0
RW
RW
0
0
0
0
0
0
0
0
Default
Value Description
The RSS_ring value for entry 24.
The RSS_ring value for entry 25.
The RSS_ring value for entry 26.
The RSS_ring value for entry 27.
The RSS_ring value for entry 28.
The RSS_ring value for entry 29.
The RSS_ring value for entry 30.
The RSS_ring value for entry 31.
RSS Registers
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 334
BCM5718 Programmer’s Guide
Indirection Table Register 5 (offset: 0x640)
Name table_entry32 table_entry33 table_entry34 table_entry35 table_entry36 table_entry37 table_entry38 table_entry39
Bits Access
31:28 RW
27:24 RW
23:20 RW
19:16 RW
15:12 RW
11:8 RW
7:4
3:0
RW
RW
0
0
0
0
0
0
0
0
Default
Value Description
The RSS_ring value for entry 32.
The RSS_ring value for entry 33.
The RSS_ring value for entry 34.
The RSS_ring value for entry 35.
The RSS_ring value for entry 36.
The RSS_ring value for entry 37.
The RSS_ring value for entry 38.
The RSS_ring value for entry 39.
Indirection Table Register 6 (offset: 0x644)
Name table_entry40 table_entry41 table_entry42 table_entry43 table_entry44 table_entry45 table_entry46 table_entry47
Bits Access
31:28 RW
27:24 RW
23:20 RW
19:16 RW
15:12 RW
11:8 RW
7:4
3:0
RW
RW
0
0
0
0
0
0
0
0
Default
Value Description
The RSS_ring value for entry 40.
The RSS_ring value for entry 41.
The RSS_ring value for entry 42.
The RSS_ring value for entry 43.
The RSS_ring value for entry 44.
The RSS_ring value for entry 45.
The RSS_ring value for entry 46.
The RSS_ring value for entry 47.
Indirection Table Register 8 (offset: 0x648)
Name table_entry48 table_entry49 table_entry50 table_entry51 table_entry52 table_entry53 table_entry54 table_entry55
Bits Access
31:28 RW
27:24 RW
23:20 RW
19:16 RW
15:12 RW
11:8 RW
7:4
3:0
RW
RW
0
0
0
0
0
0
0
0
Default
Value Description
The RSS_ring value for entry 48.
The RSS_ring value for entry 49.
The RSS_ring value for entry 50.
The RSS_ring value for entry 51.
The RSS_ring value for entry 52.
The RSS_ring value for entry 53.
The RSS_ring value for entry 54.
The RSS_ring value for entry 55.
RSS Registers
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 335
BCM5718 Programmer’s Guide
Indirection Table Register 8 (offset: 0x64C)
Name table_entry56 table_entry57 table_entry58 table_entry59 table_entry60 table_entry61 table_entry62 table_entry63
Bits Access
31:28 RW
27:24 RW
23:20 RW
19:16 RW
15:12 RW
11:8 RW
7:4
3:0
RW
RW
0
0
0
0
0
0
0
0
Default
Value Description
The RSS_ring value for entry 56.
The RSS_ring value for entry 57.
The RSS_ring value for entry 58.
The RSS_ring value for entry 59.
The RSS_ring value for entry 60.
The RSS_ring value for entry 61.
The RSS_ring value for entry 62.
The RSS_ring value for entry 63.
Indirection Table Register 9 (offset: 0x650)
Name table_entry64 table_entry65 table_entry66 table_entry67 table_entry68 table_entry69 table_entry70 table_entry71
Bits Access
31:28 RW
27:24 RW
23:20 RW
19:16 RW
15:12 RW
11:8 RW
7:4
3:0
RW
RW
0
0
0
0
0
0
0
0
Default
Value Description
The RSS_ring value for entry 64.
The RSS_ring value for entry 65.
The RSS_ring value for entry 66.
The RSS_ring value for entry 67.
The RSS_ring value for entry 68.
The RSS_ring value for entry 69.
The RSS_ring value for entry 70.
The RSS_ring value for entry 71.
Indirection Table Register 10 (offset: 0x654)
Name table_entry72 table_entry73 table_entry74 table_entry75 table_entry76 table_entry77 table_entry78 table_entry79
Bits Access
31:28 RW
27:24 RW
23:20 RW
19:16 RW
15:12 RW
11:8 RW
7:4
3:0
RW
RW
0
0
0
0
0
0
0
0
Default
Value Description
The RSS_ring value for entry 72.
The RSS_ring value for entry 73.
The RSS_ring value for entry 74.
The RSS_ring value for entry 75.
The RSS_ring value for entry 76.
The RSS_ring value for entry 77.
The RSS_ring value for entry 78.
The RSS_ring value for entry 79.
RSS Registers
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 336
BCM5718 Programmer’s Guide
Indirection Table Register 11 (offset: 0x658)
Name table_entry80 table_entry81 table_entry82 table_entry83 table_entry84 table_entry85 table_entry86 table_entry87
Bits Access
31:28 RW
27:24 RW
23:20 RW
19:16 RW
15:12 RW
11:8 RW
7:4
3:0
RW
RW
0
0
0
0
0
0
0
0
Default
Value Description
The RSS_ring value for entry 80.
The RSS_ring value for entry 81.
The RSS_ring value for entry 82.
The RSS_ring value for entry 83.
The RSS_ring value for entry 84.
The RSS_ring value for entry 85.
The RSS_ring value for entry 86.
The RSS_ring value for entry 87.
Indirection Table Register 12 (offset: 0x65C)
Name table_entry88 table_entry89 table_entry90 table_entry91 table_entry92 table_entry93 table_entry94 table_entry95
Bits Access
31:28 RW
27:24 RW
23:20 RW
19:16 RW
15:12 RW
11:8 RW
7:4
3:0
RW
RW
0
0
0
0
0
0
0
0
Default
Value Description
The RSS_ring value for entry 88.
The RSS_ring value for entry 89.
The RSS_ring value for entry 90.
The RSS_ring value for entry 91.
The RSS_ring value for entry 92.
The RSS_ring value for entry 93.
The RSS_ring value for entry 94.
The RSS_ring value for entry 95.
Indirection Table Register 12 (offset: 0x660)
Name table_entry96 table_entry97 table_entry98 table_entry99 table_entry100 table_entry101 table_entry102 table_entry103
Bits Access
31:28 RW
27:24 RW
23:20 RW
19:16 RW
15:12 RW
11:8 RW
7:4
3:0
RW
RW
0
0
0
0
0
0
0
0
Default
Value Description
The RSS_ring value for entry 96.
The RSS_ring value for entry 97.
The RSS_ring value for entry 98.
The RSS_ring value for entry 99.
The RSS_ring value for entry 100.
The RSS_ring value for entry 101.
The RSS_ring value for entry 102.
The RSS_ring value for entry 103.
RSS Registers
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 337
BCM5718 Programmer’s Guide
Indirection Table Register 13 (offset: 0x664)
Name table_entry104 table_entry105 table_entry106 table_entry107 table_entry108 table_entry109 table_entry110 table_entry111
Bits Access
31:28 RW
27:24 RW
23:20 RW
19:16 RW
15:12 RW
11:8 RW
7:4
3:0
RW
RW
0
0
0
0
0
0
0
0
Default
Value Description
The RSS_ring value for entry 104.
The RSS_ring value for entry 105.
The RSS_ring value for entry 106.
The RSS_ring value for entry 99.
The RSS_ring value for entry 100.
The RSS_ring value for entry 101.
The RSS_ring value for entry 102.
The RSS_ring value for entry 103.
Indirection Table Register 14 (offset: 0x668)
Name table_entry112 table_entry113 table_entry114 table_entry115 table_entry116 table_entry117 table_entry118 table_entry119
Bits Access
31:28 RW
27:24 RW
23:20 RW
19:16 RW
15:12 RW
11:8 RW
7:4
3:0
RW
RW
0
0
0
0
0
0
0
0
Default
Value Description
The RSS_ring value for entry 112.
The RSS_ring value for entry 113.
The RSS_ring value for entry 114.
The RSS_ring value for entry 115.
The RSS_ring value for entry 116.
The RSS_ring value for entry 117.
The RSS_ring value for entry 118.
The RSS_ring value for entry 119.
Indirection Table Register 15 (offset: 0x66C)
Name table_entry120 table_entry121 table_entry122 table_entry123 table_entry124 table_entry125 table_entry126 table_entry127
Bits Access
31:28 RW
27:24 RW
23:20 RW
19:16 RW
15:12 RW
11:8 RW
7:4
3:0
RW
RW
0
0
0
0
0
0
0
0
Default
Value Description
The RSS_ring value for entry 120.
The RSS_ring value for entry 121.
The RSS_ring value for entry 122.
The RSS_ring value for entry 123.
The RSS_ring value for entry 124.
The RSS_ring value for entry 125.
The RSS_ring value for entry 126.
The RSS_ring value for entry 127.
RSS Registers
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 338
BCM5718 Programmer’s Guide RSS Registers
Hash Key Register 0 (offset: 0x670)
Name
Hash_key[7:0]
Hash_key[15:8]
Hash_key[23:16]
Hash_key[31:24]
Bits Access
31:24 RW
Default
Value
0
23:16
15:8
7:0
RW
RW
RW
0
0
0
Description
The first byte of the hash_key. Bit31 is the first bit of the hash_key. It's the big endian format.
The 2nd byte of the hash_key. The bits are in the big endian format.
The 3rd byte of the hash_key. The bits are in the big endian format.
The 4th byte of the hash_key. The bits are in the big endian format.
Hash Key Registers 1–8 (offset: 0x674–0x693)
The rest of Hash Keys for 5th through 36th bytes. They follow the same format as above.
Hash Key Register 9 (offset: 0x694)
Name
Hash_key[295:288]
Hash_key[303:296]
Hash_key[311:304]
Hash_key[319:312]
Bits Access
31:24 RW
Default
Value
0
23:16
15:8
7:0
RW
RW
RW
0
0
0
Description
The 37th byte of the hash_key. The bits are in the big endian format.
The 38th byte of the hash_key. The bits are in the big endian format.
The 39th byte of the hash_key. The bits are in the big endian format.
The 40th byte of the hash_key. The bits are in the big endian format.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 339
BCM5718 Programmer’s Guide RSS Registers
Receive MAC Programmable IPv6 Extension Header Register (offset:
0x6A0)
Name
Programmable Extension
Header Type #2 Enable
Programmable Extension
Header Type #1 Enable
Reserved
Programmable Extension
Header Type #2
Programmable Extension
Header Type #1
Bits Access
31 RW
Default
Value
0
30
29:16
15:8
7:0
RW
RO
RW
RW
0
0
0
0
Description
This bit enables programmable extension header #1. If this bit is clear, then the value programmed in bits [15:8] of this register will be ignored. If this bit is set, then extension headers will be checked for a type matching the value in bits [15:8]. This bit should be set to 0 if IPv6 RX is disabled.
This bit enables programmable extension header #1. If this bit is clear, then the value programmed in bits [7:0] of this register will be ignored. If this bit is set, then extension headers will be checked for a type matching the value in bits [7:0]. This bit should be set to 0 if IPv6 RX is disabled.
–
These bits contain the programmable extension header value for programmable header #2.
These bits contain the programmable extension header value for programmable header #1.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 340
BCM5718 Programmer’s Guide Statistics Registers
Statistics Registers
Note: Access method for these registers (A/C and 32bit).
Transmit MAC Static Counters ifHCOutOctets (offset: 0x800)
The number of octets transmitted out of the interface, including frame characters.
etherStatsCollisions (offset: 0x808)
The number of collisions experienced.
outXonSent (offset: 0x80C)
Sent Xon.
outXoffSent (offset: 0x810)
Sent Xoff.
dot3StatsInternalMacTransmitErrors (offset: 0x818)
A count of frames for which transmission on a particular interface fails due to an internal MAC sublayer transmit error.
dot3StatsSingleCollisionFrames (offset: 0x81C)
A count of successfully transmitted frames on a particular interface for which transmission is inhibited by exactly one collision.
dot3StatsMultipleCollisionFrames (offset: 0x820)
A count of successfully transmitted frames on a particular interface for which transmission is inhibited by more than one collision.
dot3StatsDeferredTransmissions (offset: 0x824)
A count of frames for which the first transmission attempt on a particular interface is delayed because of the medium is busy.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 341
BCM5718 Programmer’s Guide Statistics Registers dot3StatsExcessiveTransmissions (offset: 0x82C)
A count of frames for which transmission on a particular interface fails due to excessive collisions.
dot3StatsLateCollisions (offset: 0x830)
The number of times that a collision is detected on a particular interface later than 512 bit-times into the transmission of a packet.
iHCOutUcastPkts (offset: 0x86C)
The number of packets that higher-level protocols requested be transmitted, and that were not addressed to a multicast or broadcast address at this sublayer, including those that were discarded or not sent.
iHCOutMulticastPkts (offset: 0x870)
The number of packets that higher-level protocols requested be transmitted, and that were addressed to a multicast address at this sublayer, including those that were discarded or not sent.
iHCOutBroadcastPkts (offset: 0x874)
The number of packets that higher-level protocols requested be transmitted, and that were addressed to a broadcast address at this sublayer, including those that were discarded or not sent.
ifCRSERRORS (offset: 0x878)
The number of packets that have CRS errors.
iOUTDISCARDS (offset: 0x87C)
The number of packets that are discarded.
H2B Statistics Registers
There are two sets of H2B Statistics registers – though both sets are accessible from Host as well as APE, each side shall own a set. Set1 is owned by Host Driver and Set2 is owned by APE. HTX2B Stats Registers resides in
TX-MAC and the B2HRX Stats Registers reside in RDI. A counter is automatically reset to 0 by HW upon a Read
Operation by software. All counters roll over.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 342
BCM5718 Programmer’s Guide Statistics Registers
HTX2B Statistics
The HTX2B statistics reside in Transmit MAC. A counter is automatically reset to 0 by hardware upon a Read
Operation. All counters roll over.
Statistics Name
HTX2B Octets
HTX2B UCAST PKTS
HTX2B MCAST PKTS
HTX2B BCAST PKTS
HTX2B DROP PKTS
HTX2B DROP OCTETS
Set1 Address Set2 Address Size
0x700
0x704
0x708
0x70C
0x710
0x720
0x724
0x728
0x72C
0x730
32
32
32
32
32
0x714 0x734 32
Description
Total number of H2B octets.
Number of H2B unicast frames.
Number of H2B multicast.
Number of H2B broadcast frames.
Number of H2B frames dropped by
EMAC.
Total number of H2B octets dropped by
EMAC.
B2HRX Statistics
The B2HRX statistics reside in RDI.
Statistics Name
B2HRX OCTETS
B2HRX UCAST PKTS
B2HRX MCAST PKTS
B2HRX BCAST PKT
B2HRX DROP PKTS
B2HRX DROP OCTETS
Set1 Address Set2 Address Size
0x24D0
0x24D4
0x24D8
0x24DC
0x24E0
0x24E8
0x24EC
0x24F0
0x24F4
0x24F8
32
32
32
32
32
0x24E4 0x24FC 32
Description
Total number of H2B octets.
Number of B2H unicast frames.
Number of B2H multicast frames.
Number of B2H broadcast frames.
Number of B2H frames dropped by
RDI.
Total number of B2H octets dropped by RDI.
Receive MAC Static Counters ifHCInOctets (offset: 0x880)
The number of octets received on the interface, including frame characters.
ifHCINOctets_bad (offset: 0x884)
The number of bad octets received on the interface.
etherStatsFragments (offset: 0x888)
A frame size that is less than 64 bytes with a bad FCS.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 343
BCM5718 Programmer’s Guide Statistics Registers ifHCInUcastPkts (offset: 0x88C)
The number of packets delivered by this sublayer to a higher sublayer, which were not addressed to a multicast or broadcast address at this sublayer.
ifHCInMulticastPkts (offset: 0x890)
The number of packets delivered by this sublayer to a higher sublayer, which were addressed to a multicast address at this sublayer.
ifHCInBroadcastPkts (offset: 0x894)
The number of packets delivered by this sublayer to a higher sublayer, which were addressed to a broadcast address at this sublayer.
dot3StatsFCSErrors (offset: 0x898)
A count of frames received on a particular interface that are an integral number of octets in length and do not pass the FCS check.
dot3StatsAlignmentErrors (offset: 0x89C)
A count of frames received on a particular interface that are not an integral number of octets in length and do not pass the FCS check.
xonPauseFrameReceived (offset: 0x8A0)
MAC control frames with pause command and length equal to zero.
xoffPauseFrameReceived (offset: 0x8A4)
MAC control frames with pause command and length greater than zero.
macControlFramesRecevied (offset: 0x8A8)
MAC control frames with no pause command.
xoffStateEntered (offset: 0x8AC)
Transmitting is disabled.
dot3StatsFramesTooLongs (offset: 0x8B0)
A count of frames received on a particular interface that exceeds the maximum permitted frame size.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 344
BCM5718 Programmer’s Guide Statistics Registers etherStatsJabbers (offset: 0x8B4)
Frames exceed jabber time.
etherStatsUndersizePkts (offset: 0x8B8)
Frames with a size less than 64 bytes.
Ifnomorerxbd:0x224C
The number of times the NIC overran the Receive Buffer Descriptors.
Ifindiscard:0x2250
The number of inbound packets selected to be discarded (even though an error was not detected) to prevent the packets from being delivered to a higher layer protocol.
Ifinerror:0x2254
The number of inbound packets containing errors that prevented the packets from being delivered to a higherlayer protocol.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 345
BCM5718 Programmer’s Guide Statistics Registers
APE_NETWORK_STATS_REGS (Offsets 0x900–0x9BC)
These are duplicate network statistics registers meant for APE's private use. All of these registers accumulate their respective event-counts and the counts self clear to 0 when a GRC read accesses is performed on a register.
Statistics Name
Transmit Statistics
IFHCOUTOCTETS
ETHERSTATSCOL
OUTXONSENT
OUTXOFFSENT
MACXMITERRORS
SINGLECOL
MULTICOL
DEFERREDXMIT
EXCESSIVECOL
LATECOL
IFHCOUTUCAST
IFHCOUTMCAST
IFHCOUTBCAST
CRSERRORS
IFOUTDISCARDS
Receive Statistics
IFHCINOCTETS_GOOD
IFHCINOCTETS_BAD
ETHERSTATSFRAGMENTS
IFHCINUCASTPKTS
IFHCINMULTICASTPKTS
IFHCINBROADCASTPKTS
DOT3STATSFCSERRORS
DOT3STATSALIGNMENTERRORS
XONPAUSEFRAMESRECEIVED
XOFFPAUSEFRAMESRECEIVED
MACCONTROLFRAMESRECEIVED
XOFFSTATEENTERED
DOT3STATSFRAMESTOOLONG
ETHERSTATSJABBERS
ETHERSTATSUNDERSIZEPKTS
Address Size
0x0980 32
0x0984 32
0x0988 16
0x098C 32
0x0990 32
0x0994 32
0x0998 16
0x099C 16
0x09A0 16
0x09A4 16
0x09A8 16
0x09AC 16
0x09B0 16
0x09B4 16
0x09B8 16
0x900h 32
0x908h 16
0x90Ch 16
0x910h 16
0x0918 16
0x091C 16
0x0920 16
0x0924 16
0x092C 16
0x0930 16
0x096C 32
0x0970 32
0x0974 32
0x0978 16
0x097C 16
Description
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 346
BCM5718 Programmer’s Guide Send Data Initiator Registers
Send Data Initiator Registers
All registers reset are core reset unless specified.
Send Data Initiator Mode Register (offset: 0xC00)
Name Bits Access
Reserved 31:9 RO
Incorrect BD flag fix disable 8 RW
0
0
Default
Value
Reserved
Multiple Segment Enable
Pre-DMA Debug
Hardware Pre-DMA Enable
7:6
5
4
3
Stats Overflow Attn Enable 2
Enable
Reset
1
0
RO
RW
RW
RW
RW
RW
RW
0
0
0
0
0
1
0
Description
–
Disable fix for SDI sends incorrect last BD flag to
SBDS.
–
Enable RDMA to read multisegment (up to four segments) in one DMA request during TCP segmentation.
When this bit is set, Send Data Initiator state machine will be halted if the pre-DMA bit of the
Send BD is set.
Enable hardware LSO pre-DMA processing.
Enable attention for statistics overflow.
This bit controls whether the Send Data Initiator state machine is active or not. When set to 0, it completes the current operation and cleanly halts. Until it is completely halted, it remains one when read.
When this bit is set to 1, Send Data Initiator state machine is reset.
This is a self-clearing bit.
Send Data Initiator Status Register (offset: 0xC04)
Name
Reserved
Stats Overflow Attention
Reserved
Bits Access
31:3 RO
2 RO
0
0
Default
Value
1:0 RO 0
Description
–
A statistics managed by Send Data Initiator has overflowed.
–
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 347
BCM5718 Programmer’s Guide Send Data Initiator Registers
Send Data Initiator Statistics Control Register (offset: 0xC08)
Name
Reserved
Zap Statistic
Flush Statistic
Statistics Clear
Faster Update
Statistics Enable
1
0
Bits Access
3
2
31:5 RO
4 RW
RW
RW
0
0
0
0
Default
Value
RW
RW
0
0
Description
–
–
–
If set, resets local statistics counters to zero.
Clears only masked statistics.
Self-clearing when done.
–
When set, allows the local statistics counters to increment. When reset, counters hold their values until next update to NIC memory.
Enables only masked statistics.
Send Data Initiator Statistics Mask Register (offset: 0xC0C)
Name
Reserved
Counters Enable Mask
Bits Access
31:1 RO
0 RW
0
0
Default
Value Description
–
Controls whether Class of Service 0 statistics can be updated, cleared, or flushed.
Send Data Initiator Statistics Increment Mask Register (offset: 0xC10)
Name
Reserved
Counters Increment Mask
Reserved
Counters Increment Mask
Bits Access
31:24 RO
23:19 WO
0
0
Default
Value
18:16
15:0
RO
WO
0
0
Description
–
Writing 1 to the bit position forces the corresponding statistics counters to increment by 1. Not affected by Statistics Enable Mask.
Bits 16:23 correspond to Set Send Producer
Index, Status Updated, Interrupts, Avoided
Interrupts, Send Threshold Hit respectively.
–
Writing 1 to the bit position forces the corresponding statistics counters to increment by 1. Not affected by Statistics Enable Mask.
Bits 15:0 correspond to statistics for Class of
Service 16:1
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 348
BCM5718 Programmer’s Guide TCP Segmentation Control Registers
Local Statistics Register (offset: 0xC80–0xCDF)
Name
Reserved
Counter Value
Bits Access
31:10 RO
9:0 RO
0
0
Default
Value Description
–
The current counter value for statistics kept by the Send Data Initiator.
TCP Segmentation Control Registers
All registers reset are core reset unless specified.
Lower Host Address Register for TCP Segmentation (offset: 0xCE0)
Name
Lower Host Address
Bits Access
31:0 RW
Default
Value
0
Description
Specifies the lower 32bits of the starting address in host memory where the transmit data buffer resides.
Upper Host Address Register for TCP Segmentation (offset: 0xCE4)
Name
Upper Host Address
Bits Access
31:0 RW
Default
Value
0
Description
Specifies the upper 32bits of the starting address in host memory where the transmit data buffer resides.
Length/Offset Register for TCP Segmentation (offset: 0xCE8)
Name
Reserved
MBUF Offset
Length
Bits Access
31:23 RO
22:16 RW
0
0
Default
Value
15:0 RW 0
Description
–
MBUF offset.
It specifies the offset of the first TXMBUF at where DMA starts putting data. The valid value is between 48 and 128.
Specifies the length of data to be transmitted.
Although firmware can specify up to 64 KB, it should not attempt to program more than 8 KB because it would exceed the size of TXMBUF.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 349
BCM5718 Programmer’s Guide TCP Segmentation Control Registers
DMA Flag Register for TCP Segmentation (offset: 0xCEC)
Name
Reserved
Reserved
MBUF offset valid
Last Fragment
No Word Swap
Status_dma
MAC source address Select
MAC source address insertion 13
TCP/UDP checksum enable
IP Checksum enable
12
11
Force RAW checksum enable 10
Data_only
Header
VLAN Tag Present
Force Interrupt
Last BD in Frame
Coalesce Now mbuf
Bits Access
31:26 RO
25:20 RW
19 RW
–
–
–
Default
Value
18
17
16
15:14
9
8
7
6
5
4
3
RW
RW
RW
RW
RW
RW
RW
RW
RO
RW
RW
RW
RW
RW
RW
–
–
–
–
–
–
–
–
–
–
–
–
–
–
–
Description
–
–
MBUF offset valid.
When this bit is set, the RDMA engine will DMA the data into the TXMBUF starting at an offset specified in the Length/Offset register.
Last fragment. This bit is passed transparently to the SDC.
When this bit is set, the SDC will inform the HC to increment the Send Ring Consumer Index.
• The bit is always set by hardware if no firmware assisted TCP segmentation occurs.
• Otherwise, firmware sets it at the end of fragmentation.
No Word Swap.
Set to disable endian word swap on data from
PCIE bus.
–
This 2-bit field determines which of the four
MAC addresses should be inserted into the frame.
Indicates that the predetermined source address is inserted into the Ethernet header of the frame.
TCP/UDP Checksum enable.
IP checksum enable.
Force RAW checksum enable.
–
–
VLAN Tag present.
Indicates that the VLAN tag should be copied to the Frame Header by the DMA engine.
Following the completion of this DMA, a host interrupt is generated.
Last BD in frame.
Pass through Send Buffer Descriptor flag.
–
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 350
BCM5718 Programmer’s Guide TCP Segmentation Control Registers
Name
Invoke Processor
Don't Generate CRC
No Byte Swap
Bits Access
2 RW
Default
Value
–
1
0
RW
RW
–
–
Description
Clears the Pass bit of the entry queued to the
SDCQ, so that SDC will invoke CPU.
• If the packet is created by hardware, this bit will be the same as bit 9 of the flag field in
Send BD.
• If the packet is created by firmware, it will be up to CPU whether it needs to post-process the data.
Do not generate CRC.
Pass through Send Buffer Descriptor flag.
Set to disable endian byte swap on data from
PCIE bus.
VLAN Tag Register for TCP Segmentation (offset: 0xCF0)
Name
Reserved
VLAN Tag
Bits Access
31:16 RO
15:0 RW
0
0
Default
Value Description
–
VLAN Tag to be inserted into the Frame Header if bit 7 of DMA Flags register is set.
Pre-DMA Command Exchange Register for TCP Segmentation (offset:
0xCF4)
Name
READY
PASS Status
SKIP Status
Unsupported_Mss Status
Reserved
BD Index
Bits Access
31 RW
Default
Value
0
30
29
28
27:7
6:0
RO
RW
RO
RO
RO
1
0
0
0
0
Description
The CPU sets this bit to tell SDI that DMA address, length, flags, and VLAN tag are valid and request is read to go.
The CPU polls this bit to be clear for the completion of request.
0xCF4.31 is writable only if 0xCE8.15:0 is nonzero.
If this bit is set to 0, the CPU will be responsible for processing the buffer descriptor
The CPU sets this bit to 1 to inform the SDI that the TCP Segmentation is completed, and the
BD_Index can be incremented.
–
–
The internal current buffer descriptor pointer that the hardware/firmware is servicing.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 351
BCM5718 Programmer’s Guide Send Data Completion Control Registers
Send Data Completion Control Registers
All registers reset are core reset unless specified.
Send Data Completion Mode Register (offset: 0x1000)
Name
Reserved
Disable Delayed Host
Coalescing
Reserved
Enable
Reset
Bits Access
31:5 RO
4 RW
0
0
Default
Value
3:2
1
0
RO
RW
RW
0
1
0
Description
–
A value 1 disables the Delayed HC feature introduced in BCM5718 family.
–
This bit controls whether Send Data Completion state machine is active or not. When set to 0, it completes the current operation and cleanly halts. Until it is completely halted, it remains one when read.
When this is set to 1, the Send Data Completion state machine is reset.
This is a self-clearing bit.
Pre-DMA Command Exchange Register for TCP Segmentation (offset:
0x1008)
Name
PASS
SKIP
End of Fragmentation
Reserved
Head TXMBUF pointer
Tail TXMBUF pointer
Bits Access
31 RW
Default
Value
1
30
29
28:12
11:6
5:0
RW
RW
RO
RW
RW
0
1
0
0
0
Description
If this bit is set to 0, the CPU will be invoked to process TXMBUF data.
It is same as SDCQ bit 143.
CPU Sets this bit to 1 to inform the SDC that the post-processing is completed and hardware can resume operation.
If this bit is set to 1, SDC will request the HC to increment Send Ring Consumer Index when CPU sets the SKIP bit.
It is same as SDCQ bit 12.
–
Head TXMBUF Pointer.
They are same as SDCQ bits 11:6.
Tail TXMBUF Pointer.
They are same as SDCQ bits 5:0
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 352
BCM5718 Programmer’s Guide Send BD Selector Control Registers
Send BD Selector Control Registers
All registers reset are core reset unless specified.
Send BD Ring Selector Mode Register (offset: 0x1400)
Name Bits Access
Reserved 31:4 RO
SBD consumer index fix disable 3 RW
0
0
Default
Value
Attention Enable
Enable
Reset
2
1
0
RW
RW
RW
0
0
0
Description
–
Disable for sbd consumer index does not rollover for ring sizes 32,64,128, 256.
When this bit is set to 1, an internal attention is generated when an error occurs.
This bit controls whether Send BD Ring Selector state machine is active or not.
When set to 0, it completes the current operation and cleanly halts. Until it is completely halted, it remains 1 when read.
When this is set to 1, the Send BD Ring Selector
State machine is reset.
This is a self clearing bit.
Send BD Ring Selector Status Register (offset: 0x1404)
Name
Reserved
Error
Reserved
Bits Access
31:3 RO
2 RO
1:0 RO
0
0
0
Default
Value Description
–
Send BD Ring Selector error status.
–
Send BD Ring Selector Hardware Diagnostics Register (offset: 0x1408)
Name
Reserved
Bits Access
31:0 RO
Default
Value
0
Description
–
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 353
BCM5718 Programmer’s Guide Send BD Selector Control Registers
Send BD Ring Selector Local NIC Send BD Consumer Index Register
(offset: 0x1440–0x147C)
Name
Reserved
Index
Bits Access
31:9 RO
8:0 RO
Default
Value
0
Description
–
These bits contain the current NIC send BD index.
Each register applies to:
Name
0x1440-0x1443 Send BD Ring Selector Local NIC
0x1444-0x1447 Send BD Ring Selector Local NIC
0x1448-0x144b Send BD Ring Selector Local NIC
0x144c-0x144f Send BD Ring Selector Local NIC
0x1450-0x1453 Send BD Ring Selector Local NIC
0x1454-0x1457 Send BD Ring Selector Local NIC
0x1458-0x145b Send BD Ring Selector Local NIC
0x145c-0x145f Send BD Ring Selector Local NIC
0x1460-0x1463 Send BD Ring Selector Local NIC
0x1464-0x1467 Send BD Ring Selector Local NIC
0x1468-0x146b Send BD Ring Selector Local NIC
0x146c-0x146f Send BD Ring Selector Local NIC
0x1470-0x1473 Send BD Ring Selector Local NIC
0x1474-0x1477 Send BD Ring Selector Local NIC
0x1478-0x147b Send BD Ring Selector Local NIC
0x147c-0x147f Send BD Ring Selector Local NIC
Description
Send BD 1 Consumer Index RO.
Send BD 2 Consumer Index RO.
Send BD 3 Consumer Index RO.
Send BD 4 Consumer Index RO.
Send BD 5 Consumer Index RO.
Send BD 6 Consumer Index RO.
Send BD 7 Consumer Index RO.
Send BD 8 Consumer Index RO.
Send BD 9 Consumer Index RO.
Send BD 10 Consumer Index RO.
Send BD 11 Consumer Index RO.
Send BD 12 Consumer Index RO.
Send BD 13 Consumer Index RO.
Send BD 14 Consumer Index RO.
Send BD 15 Consumer Index RO.
Send BD 16 Consumer Index RO.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 354
BCM5718 Programmer’s Guide Send BD Initiator Control Registers
Send BD Initiator Control Registers
All registers reset are core reset unless specified.
Send BD Initiator Mode Register (offset: 0x1800)
Name
Reserved
Multiple Send Ring mode
Pass_bit status
Sbdi_rupd_enable
Attention Enable
Enable
Reset
3
2
4:0
4
Bits Access
31.6
31:5 RO
5 RW
0
0
Default
Value
1
0
RW
RW
RW
RW
RW
0
0
0
1
0
Description
Unchanged
–
Write a 1 to enable 16 Send Rings.
Write a 0 to limit to a Single Send Ring.
Unchanged
Always return 1 when read.
–
When this bit is set to 1, an internal attention is generated when an error occurs.
This bit controls whether the Send BD Initiator state machine is active or not. When set to 0, it completes the current operation and cleanly halts.
Until it is completely halted, it remains 1 when read.
When this is set to 1, the Send BD Initiator State machine is reset.
This is a self clearing bit.
Send BD Initiator Status Register (offset: 0x1804)
Name
Reserved
Error
Reserved
Bits Access
31:3 RO
2 RO
1:0 RO
0
0
0
Default
Value Description
–
Send BD Initiator Error.
–
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 355
BCM5718 Programmer’s Guide Send BD Initiator Control Registers
Send BD Diagnostic Initiator Local NIC BD N Producer Index Registers
(offset: 0x1808–0x1844)
This set of registers is used to keep track of the current DMAs queued to move send BDs from the host to the
NIC.
Each register applies to:
Send BD Initiator Local NIC
0x1808-0x180b
0x180c-0x180f
0x1810-0x1813
0x1814-0x1817
0x1818-0x181b
0x181c-0x181f
0x1820-0x1823
0x1824-0x1827
0x1828-0x182b
0x182c-0x182f
0x1830-0x1833
0x1834-0x1837
0x1838-0x183b
0x183c-0x183f
0x1840-0x1843
0x1844-0x1847
Table 100: Send BD Diagnostic Initiator
Send BD Producer Index
Send BD 1 Producer Index RO.
Send BD 2 Producer Index RO.
Send BD 3 Producer Index RO.
Send BD 4 Producer Index RO.
Send BD 5 Producer Index RO.
Send BD 6 Producer Index RO.
Send BD 7 Producer Index RO.
Send BD 8 Producer Index RO.
Send BD 9 Producer Index RO.
Send BD 10 Producer Index RO.
Send BD 11 Producer Index RO.
Send BD 12 Producer Index RO.
Send BD 13 Producer Index RO.
Send BD 14 Producer Index RO.
Send BD 15 Producer Index RO.
Send BD 16 Producer Index RO.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 356
BCM5718 Programmer’s Guide Send BD Initiator Control Registers
Send BD Fetch Threshold Register (offset: 0x1850)
Name
Reserved
Send BD Fetch Threshold
Bits Access
31:6 RO
5:0 RW
Default
Value
0x00
0x1F
Description
Unused
The value programmed in this field sets a cap to the number of SBDs that would be fetched by a single DMA transaction (although there are other factors which might further limit the DMA size).
This parameter uniformly applies to all 16 Send
Rings.
This parameter is meaningful only in the
Multiple Send Ring mode. 0x1800[5] == 1
14
15
16
10
11
12
13
8
9
6
7
4
5
2
3
Send Mail Box Registers
15 more Host Send Producer Index registers are added to the High Priority Mail Box region as shown in the
Table below. Similarly, 15 more NIC Send Producer Index registers are also added in the GRC space.
Send Ring #
Legacy /1
Table 101: Multiple Send Ring Mail Boxes
Host Send Ring
Producer Index
(High Priority Mail Box)
0x304
NIC Send Ring
Producer Index
0x5980
NIC Diagnostic
Send Ring
Consumer Index
0x3CC0
Usable in
Legacy Mode,
RSS Mode and
IOV Mode
0x320
0x32C
0x328
0x334
0x330
0x33C
0x338
0x300
0x30C
0x308
0x314
0x310
0x31C
0x318
0x324
0x5984
0x5988
0x598C
0x5990
0x5994
0x5998
0x599C
0x59A0
0x59A4
0x59A8
0x59AC
0x59B0
0x59B4
0x59B8
0x59BC
0x3F24
0x3F28
0x3F2C
0x3F30
0x3F34
0x3F38
0x3F3C
0x3F04
0x3F08
0x3F0C
0x3F10
0x3F14
0x3F18
0x3F1C
0x3F20
RSS Mode and
IOV Mode
IOV Mode Only
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 357
BCM5718 Programmer’s Guide Send BD Completion Control Registers
Send BD Completion Control Registers
All registers reset are core reset unless specified.
Send BD Completion Mode Register (offset: 0x1C00)
Name
Reserved
Attention Enable
Enable
Until it is completely halted, it remains 1 when read
Reset 0
This is a self clearing bit
Bits Access
31:3 RO
2 RW
0
0
Default
Value
1 RW 1
Description
–
When this bit is set to 1, an internal attention is generated when an error occurs.
This bit controls whether the Send BD
Completion state machine is active or not. When set to 0, it completes the current operation and cleanly halts.
–
RW 0 When this is set to 1, the Send BD Completion
State machine is reset.
–
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 358
BCM5718 Programmer’s Guide Receive List Placement Registers
Receive List Placement Registers
All registers reset are core reset unless specified.
Receive List Placement Mode Register (offset: 0x2000)
Name Bits Access
Reserved
Stats Overflow Attention
Enable
Mapping out of Range
Attention Enable
3
Class Zero Attention Enable 2
Enable 1
31:5 RO
4 RW
RW
RW
RW
–
–
1
0
–
Default
Value
Reset 0 RW 0
Description
–
Enable attention for statistics overflow.
Enable attention for mapping out of range error.
Enable attention for zero class field.
This bit controls whether the Receive List
Placement state machine is active or not. When set to 0, it completes the current operation and cleanly halts. Until it is completely halted, it remains one when read.
When this bit is set to 1, the Receive List
Placement state machine is reset.
This is a self clearing bit.
Receive List Placement Status Register (offset: 0x2004)
Name
Reserved
Stats Overflow Attention
Mapping out of Range
Attention
Class Zero Attention
Reserved
Bits Access
31:5 RO
4 RO
0
–
Default
Value
3
2
1:0
RO
RO
RO
–
–
0
Description
–
A statistics managed by Receive List Placement has overflowed.
Class of service mapping is out of the range of the active queue number.
Class field extracted from frame descriptor is zero.
–
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 359
BCM5718 Programmer’s Guide Receive List Placement Registers
Receive Selector Non-Empty Bits Register (offset: 0x200C)
This 32-bit register is used by the RISCs to quickly determine the status of the receive selector. Bit 0 refers to receive selector list 1. Bit 15 refers to receive selector list 16. If this register is nonzero the receive selector nonempty bit is set in the RXCPU event register.
Name
Reserved
List non-empty bits
Bits Access
31:16 RO
15:0 RO
0
–
Default
Value Description
–
If set, the bit indicates that the associated list is not empty (that is the counter is nonzero).
Receive List Placement Configuration Register (offset: 0x2010)
Name Bits Access
Reserved
Default Interrupt Distribution
Queue
31:15 RO
14:13 RW
0
0
Default
Value
Bad Frames Class
Number of Active Lists
Number of Lists per
Distribution Group
12:8
7:3
2:0
RO
RW
RW
1
0
0
Description
–
Default interrupt distribution queue. Number within a class of service group when the frame has errors, is truncated, or is a non-IP frame.
Default class for error or truncated frames.
These frames are placed in this class of service group when the Allow Bad Frame bit (bit 11) is set in the Mode Control Register.
The total number of active receive lists. The value must be between 1 and 16. This value must be an integer multiple of the Number of
Lists per Distribution Group value.
Specifies the number of lists per interrupt distribution group. This register must always be a power of 2. For example, if the system wants four classes of service and four interrupt distribution lists per class of service, this value is set to four and the Number of Active Lists value is set to 16.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 360
BCM5718 Programmer’s Guide Receive List Placement Registers
Receive List Placement Statistics Control Register (offset: 0x2014)
Name
Reserved
Statistics Clear
Reserved
Statistics Enable
1
0
Bits Access
31:3 RO
2 RW
0
0
Default
Value
RO
RW
0
0
Description
–
When set, resets local statistics counters to zero.
Clears only masked statistics.
Self-clearing when done.
–
When set, allow the local statistics counters to increment.
When reset, counters hold their values until the next update to the NIC memory.
Enables only masked statistics.
Receive List Placement Statistics Enable Mask Register (offset:
0x2018)
Name
Reserved
RSS_Priority
RC Return Ring Enable
CPU MACTQ Priority Disable
Reserved
Disable MACTQ Double Ack issue fix
Reserved
Bits Access
31:26 RO
25 RW
Default
Value
0
0x0
24
23
22:19
18
17:2
RW
RW
RO
RW
RO
0x0
0x0
0
1
N/A
Description
–
This bit enables the receive packet to choose receive return ring in terms of RSS hash value instead of RC class when both RSS and RC rules are matched. Default is to give priority to RC.
1: Enable receive packet to use RC rule class as return ring number if RC rule is matched. This bit will be used in conjunction with bit25 to derive the final receive return ring.
0: Disable receive packet to use RC rule class as return ring number. Receive packet only uses
RSS hash to select the receive return ring. If no any RSS hash types are applied, the default ring
0 will be used.
1: Disable CPU priority over SDC when arbitrating the MACTQ write requests.
0: Enable CPU priority over SDC when arbitrating the MACTQ write requests.
–
Disable MACTQ double ack issue fix.
1: Disabled
0: Enabled
–
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 361
BCM5718 Programmer’s Guide Receive List Placement Registers
Name Bits
Disable ASF Lockup Issue Fix 1
Access
RW
Default
Value
1
Reserved 0 RO 0
Description
Disable ASF Lockup Fix.
1: Disabled
0: Enabled
–
Receive List Placement Statistics Increment Mask Register (offset:
0x201C)
Name
Reserved
Counters Increment Mask
Reserved
Counters Increment Mask
Bits Access
31:22 RO
21:16 WO
0
0
Default
Value
15:1
0
RO
WO
0
0
Description
–
Writing a 1 to a Counters Increment Mask bit forces the corresponding statistics counter to increment by 1. Not affected by Statistics Enable
Mask.
Bits 16-21 correspond to statistics for Drop due to filter, DMA Write Queue Full, DMA High
Priority Write Queue Full, No More Receive BD, ifInDiscards, and ifInErrors.
–
Writing a 1 to a Counters Increment Mask bit forces the corresponding statistics counter to increment by 1. Not affected by Statistics Enable
Mask.
Bit 0 corresponds to statistics Class of Service 1.
Receive Selector List Head & Tail Pointers (offset: 0x2100)
The 16 receive selector lists head and tail pointers are MBUF cluster pointers. The selector list head pointer is the MBUF cluster pointer of the first frame queued in the associated selector list. Similarly, the selector list tail pointer is the MBUF cluster pointer of the last frame queued in that selector list.
Receive Selector List 1 Count Registers (Offset: 0x2108)
These registers indicate how many frames are currently queued to the associated selector list.
Receive Selector List 2 Count Register (offset: 0x2118)
Receive Selector List 3 Count Registers (offset: 0x2128)
Receive Selector List 4 Count Registers (offset: 0x2138)
Receive Selector List 5 Count Registers (offset: 0x2148)
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 362
BCM5718 Programmer’s Guide
Receive Selector List 6 Count Registers (offset: 0x2158)
Receive Selector List 7 Count Registers (offset: 0x2168)
Receive Selector List 8 Count Registers (offset: 0x2178)
Receive Selector List 9 Count Registers (offset: 0x2188)
Receive Selector List 10 Count Registers (offset: 0x2198)
Receive Selector List 11 Count Registers (offset: 0x21a8)
Receive Selector List 12 Count Registers (offset: 0x21b8)
Receive Selector List 13 Count Registers (offset: 0x21c8)
Receive Selector List 14 Count Registers (offset: 0x21d8)
Receive Selector List 15 Count Registers (offset: 0x21e8)
Receive Selector List 16 Count Registers (offset: 0x21f8)
Receive List Placement Registers
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 363
BCM5718 Programmer’s Guide Receive Data and Receive BD Initiator Control Registers
Receive Data and Receive BD Initiator Control Registers
All registers reset are core reset unless specified.
Receive Data and Receive BD Initiator Mode Register (offset: 0x2400)
Name
Reserved
Legacy
Large RX Ring Sizes
Reserved
No BD Discard Policy
Bits Access
31:17 RO
31:14
16 RW
0
0
0
Default
Value Description
–
Defined by Legacy
When this bit is 1, following are the maximum allowable Receive Ring sizes:
Standard Producer Ring == 2048
Jumbo Producer Ring == 1024
Receive Return Ring == 4096
15
14:13
RO
RW
0
00
When this bit is 0, following are the maximum allowable Receive Ring sizes:
Standard Producer Ring == 512
Jumbo Producer Ring == 256
Receive Return Ring == 1024
–
This field decides what to do with a packet in
RXMBUF belonging to a generic VRQ, when not a single BD is available in the VRQ's BD Cache:
00 => The RDI shall discard an RX packet belonging to a VRQ, if no corresponding RX BD is available in the VRQ's BD Cache
01 => RDI shall place the RX packet in the Default
VRQ instead
10 => RDI shall wait for a BD to become available in the VRQ's BD cache, no matter how long it takes.
11 => Reserved for future use.
Ignored in non-IOV mode
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 364
BCM5718 Programmer’s Guide Receive Data and Receive BD Initiator Control Registers
Name Bits Access
HDS Look-Ahead Boundary 12:6 RW
Default
Value
0x00
HDS Enable
Legacy
Illegal Return Ring Size
Frame Size is too large to fit into one Receive BD
Reserved
Enable
Reset
5
4:0
4
3
2
1
0
RW
RW
RW
RO
RW
RW
0
–
–
0
1
0
Description
This field indicates the size, in 1B increments, of the Header that the chip would replicate when
HDS feature is enabled (meaningful only if bit[5]
== 1)
0x00 => 128B
0x01 => 1B
0x02 => 2B
….
….
0x7F => 127B
Ignored in non-IOV mode.
When this bit is written 1, the Header Data Split feature is enabled. This bit is meaningful only when IOV mode is enabled (0x400[5] == 1)
Defined by Legacy.
Enables illegal return ring size attention.
Enables frame size is too large to fit into one
Receive BD attention.
–
This bit controls whether the Receive Data and
Receive BD Initiator state machine is active or not. When set to 0, it completes the current operation and cleanly halts. Until it is completely halted, it remains one when read.
When this bit is set to 1, the Receive Data and
Receive BD Initiator state machine is reset. This is a self-clearing bit.
Receive Data and Receive BD Initiator Status Register (offset: 0x2404)
Name
Reserved
Illegal Return Ring Size
Frame size is too large to fit into one Receive BD
Jumbo Frame Enable
3
2
Reserved
Bits Access
31:5 RO
4 RO
0
–
Default
Value
1:0
RO
RW
RO
–
0
0
Description
–
One of the return rings contains illegal ring size
(e.g., only contains 1024 entries).
The received frame size is too big for the selected Receive BD.
Enable Jumbo Receive BD is needed and Jumbo
Receive BD ring is disabled attention.
–
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 365
BCM5718 Programmer’s Guide Receive Data and Receive BD Initiator Control Registers
VRQ Status Register (offset: 0x240C)
Name
Legacy
VRQ Active Bit-Map
Bits Access
31:17 RU
16:0 RW
Default
Value
0x0
0x0
Description
Unused
A position of the Bit-Map shows a 1, in case the
VRQ is currently enabled or recently disabled and is at the verge of being flushed, and thus there could possibly be associated traffic inside the chip.
A bit-position shows 0, when a VRQ is completely disabled and there is no traffic whatsoever associated with it either inside the chip or in the PCIe link between the chip and the
Root-Complex.
VRQ Flush Control Register (Offset: 0x2410)
Name Bits Access
Default
Value
S/W flush IOV vector 31:15 RW
Reserved 14:9 RW
S/W flush reset request 8
0x0
0x0
RW/SC 0x0
Reserved
VRQ hardware flush drop enable
VRQ status update and interrupt enable
VRQ hardware flush reset enable
7:4
3
2
1
RW
RW
RW
RW
0x0
0x0
0x0
0x0
Description
S/W Flush vector [16:0].
Reserved
Software writing 1 to this bit to force an internal index reset request. Bit-8 combining with
[31:15] bit-map indicates which IOV index group to be reset. The function is identical to bit-1 hardware flush reset. This bit is self-clear.
Reserved
1: Enable hardware drop packet function immediately when the IOV enable bits are disabled from driver. This bit will force RDI and
WDMA engines to place the incoming packets into drop queue when the IOV enable bit is disabled in EMAC for that particular IOV child process.
1: Enable hardware triggered host status interrupt and status block update request.
RDI engine will generate a single pulse request signal to HC alone with current IOV status vector value for status block update.
1: Enable hardware local index reset feature when VRQ flush timer expires.
RDI engine will generate a single pulse clear request along with current IOV status [16:0] vector to various modules to clear local maintained index values for that particular IOV process.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 366
BCM5718 Programmer’s Guide Receive Data and Receive BD Initiator Control Registers
Name Bits
VRQ hardware Flush enable 0
Access
RW
Default
Value
0x0
Description
This value will be loaded into a count-down counter triggered by a falling-edge of any bit of
VRQ Enable register (0x560). The count-down is based on internal CORE_CLK. The purpose of this counter is to allow hardware to drain out certain pending DMA requests within WDMA pipeline and PCIE core. Once the timer expires, hardware shall invoke the Automatic VRQ Flush Procedure.
A value 0x0 in this register effectively zeroes this timer count.
VRQ Flush Timer Register (offset: 0x2414)
Name
IOV Flush Timer
Bits Access
31:0 RW
Default
Value
0x0
Description
This value will be loaded into a count-down counter triggered by a falling-edge of any bit of
VRQ Enable register (0x560). The count-down is based on internal CORE_CLK. The purpose of this counter is to allow hardware to drain out certain pending DMA requests within WDMA pipeline and PCIE core. Once the timer expires, hardware shall invoke the Automatic VRQ Flush Procedure.
A value 0x0 in this register effectively zeroes this timer count.
RDI B2HRX Hardware Debugging Register (offset: 0x2418)
Name
Legacy
Bits Access
31:0 RO
Default
Value
0x0
Description
RDI internal B2HRX status.
Jumbo Producer Ring Host Address High Register (offset: 0x2440)
Name
Host Address High
Bits Access
31:0 RW
Default
Value
0
Description
The host ring address is the host address of the first ring element.
The host ring address is in host address format.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 367
BCM5718 Programmer’s Guide Receive Data and Receive BD Initiator Control Registers
Jumbo Producer Ring Host Address Low Register (offset: 0x2444)
Name
Host Address Low
Bits Access
31:0 RW
Default
Value
0
Description
The host ring address is the host address of the first ring element.
The host ring address is in host address format.
Jumbo Producer Length/Flags Register (offset: 0x2448)
Name
Max Length
Reserved
Disable Ring
Reserved
Bits Access
31:16 RW
Default
Value
0
15:2
1
0
RO
RW
RO
0
0
0
Description
Specifies the number of entries for Jumbo ring based on bit-mask (Supported values are
32,64,128,256,512,1024).
–
Set to disable the use of the ring.
–
Jumbo Producer Ring NIC Address Register (offset: 0x244C)
Name
NIC Address
Bits Access
31:0 RW
Default
Value Description
0x000444
00
The NIC ring address is the NIC address of the first ring element.
Note: For the BCM5718 family, do not initialize this register; leave as default.
Standard Receive BD Ring RCB Registers
Receive Producer Ring Host Address High Register (offset: 0x2450)
Name
Host Address High
Bits Access
31:0 RW
Default
Value
0
Description
The host ring address is the host address of the first ring element. The host ring address is in host address format.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 368
BCM5718 Programmer’s Guide Receive Data and Receive BD Initiator Control Registers
Receive Producer Ring Host Address Low Register (offset: 0x2454)
Name
Host Address Low
Bits Access
31:0 RW
Default
Value
0
Description
The host ring address is the host address of the first ring element. The host ring address is in host address format.
Receive Producer Length/Flags Register (offset: 0x2458)
Name
Max Length
Reserved
Disable Ring
Reserved
Bits Access
31:16 RW
Default
Value
0
15:2
1
0
RO
RW
RO
0
0
0
Description
Programmable ring size:
• 2048
• 1024
• 512
• 256
• 128
• 64
• 32
Maximum RX frame size.
1 = Ring Disabled
0 = Ring Enabled
Reserved
Receive Producer Ring NIC Address Register (offset: 0x245C)
Name
NIC Address
Bits Access
31:0 RW
Default
Value
0x000400
00
Description
The NIC ring address is the NIC address of the first ring element.
Note: For the BCM5718 family, do not initialize this register; leave as default.
Receive Diagnostic Data and Receive BD Ring Initiator Local NIC Jumbo
Receive BD Consumer Index (offset: 0x2470)
This set of registers keeps track of the current DMAs queued to move receive data from the controller to the host. The receive data and receive BD initiator maintains the state of the indices by keeping two local copies, a copy of the controller's return ring producer index and a copy of the controller's receive BD consumer index.
The local return ring producer index is set to the value placed in the DMA descriptor. The local controller receive return consumer index is also set to the value placed in the DMA descriptor.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 369
BCM5718 Programmer’s Guide Receive Data and Receive BD Initiator Control Registers
Receive BD Ring Initiator Local NIC Standard Receive BD Consumer
Index (offset: 0x2474)
This set of registers keeps track of the current DMAs queued to move receive data from the NIC to the host.
The Receive Data and Receive BD Initiator maintains the state of the indices by keeping two local copies, a copy of the NIC’s return ring producer index, and a copy of the NIC’s receive BD consumer index. The local return ring producer index is set to the value placed in the DMA descriptor. The local NIC receive return consumer index is also set to the value placed in the DMA descriptor.
Receive Data and Receive BD Initiator Hardware Diagnostic Register
(offset: 0x24C0)
Name
Diagnostics
Bits Access
31:0 RO
Default
Value
0
Description
Hardware Diagnostics
B2HRX Byte-count Statistics Count (offset: 0x24D0)
Name
B2HRX OCTETS
Bits Access
31:0 CORW 0
Default
Value Description
Host B2HRX total packet byte count.
B2HRX Unicast Statistics Count (offset: 0x24D4)
Name
B2HRX UCAST PKT
Bits Access
31:0 CORW 0
Default
Value Description
Host B2HRX unicast packet count.
B2HRX Multicast Statistics Count (offset: 0x24D8)
Name
B2HRX MCAST PKT
Bits Access
31:0 CORW 0
Default
Value Description
Host B2HRX multicast packet count.
B2HRX Broadcast Statistics Count (offset: 0x24DC)
Name
B2HRX BCAST PKT
Bits Access
31:0 CORW 0
Default
Value Description
Host B2HRX broadcast packet count.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 370
BCM5718 Programmer’s Guide Receive Data and Receive BD Initiator Control Registers
B2HRX Drop Packet Count (offset: 0x24E0)
Name
B2HRX DROP PKT
Bits Access
31:0 CORW 0
Default
Value Description
Host B2HRX packet drop count due to empty
RBD.
B2HRX Drop Packet Byte Count (offset: 0x24E4)
Name
B2HRX DROP OCTETS
Bits Access
31:0 CORW 0
Default
Value Description
Host B2HRX packet drop byte count due to empty RBD.
B2HRX APE Byte-count Statistics Count (offset: 0x24E8)
Name
B2HRX OCTETS
Bits Access
31:0 CORW 0
Default
Value Description
APE B2HRX total packet byte count.
B2HRX APE Unicast Statistics Count (offset: 0x24EC)
Name
B2HRX UCAST PKT
Bits Access
31:0 CORW 0
Default
Value Description
APE B2HRX unicast packet count.
B2HRX APE Multicast Statistics Count (offset: 0x24F0)
Name
B2HRX MCAST PKT
Bits Access
31:0 CORW 0
Default
Value Description
APE B2HRX multicast packet count.
B2HRX APE Broadcast Statistics Count (offset: 0x24F4)
Name
B2HRX BCAST
Bits Access
31:0 CORW 0
Default
Value Description
APE B2HRX broadcast packet count.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 371
BCM5718 Programmer’s Guide Receive Data and Receive BD Initiator Control Registers
B2HRX APE Drop Packet Count (offset: 0x24F8)
Name
B2HRX DROP PKT
Bits Access
31:0 CORW 0
Default
Value Description
APE B2HRX packet drop count due to empty
RBD.
B2HRX APE Drop Packet Byte Count (offset: 0x24FC)
Name
B2HRX DROP OCTETS
Bits Access
31:0 CORW 0
Default
Value Description
APE B2HRX packet drop byte count due to empty
RBD.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 372
BCM5718 Programmer’s Guide Receive Data Completion Control Registers
Receive Data Completion Control Registers
All registers reset are core reset unless specified.
Receive Data Completion Mode Register (offset: 0x2800)
Name
Reserved
Attention Enable
Enable
Reset
Bits Access
31:3 RO
2 RW
0
0
Default
Value
1
0
RW
RW
1
0
Description
–
When this bit is set to 1, an internal attention is generated when an error occurs.
This bit controls whether the Receive Data
Completion state machine is active or not. When set to 0, it completes the current operation and cleanly halts. Until it is completely halted, it remains one when read.
When this bit is set to 1, the Receive Data
Completion state machine is reset. This is a selfclearing bit.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 373
BCM5718 Programmer’s Guide Receive BD Initiator Control Registers
Receive BD Initiator Control Registers
All registers reset are core reset unless specified.
Receive BD Initiator Mode Register (offset: 0x2C00)
Name
Reserved
Receive BDs available on a disabled Receive BD Ring
Enable
Enable
Reset
Bits Access
31:3 RO
2 RW
0
0
Default
Value Description
–
Attention enable for Receive BDs available on a disabled Receive BD ring.
1
0
RW
RW
1
0
This bit controls whether the Receive BD
Initiator state machine is active or not. When set to 0, it completes the current operation and cleanly halts. Until it is completely halted, it remains one when read.
When this bit is set to 1, the Receive BD Initiator state machine is reset. This is a self-clearing bit.
Receive BD Initiator Status Register (offset: 0x2C04)
Name
Reserved
Receive BDs available on a disabled Receive BD Ring status
Reserved
Bits Access
31:3 RO
2 RO
0
0
Default
Value Description
–
Host requests to DMA Receive BDs to a disabled ring.
1:0 RO 0 –
Receive BD Initiator Local NIC Jumbo Receive BD Producer Index
(offset: 0x2C08)
Name Bits Access
Reserved
Local Received BD jumbo
Producer Ring requested Index
31:8 RO
7:0 RO
0
0
Default
Value Description
–
Current Jumbo Received BD Index requested by
RBDI for BD fetching. Notice that, this index is different from MB producer index and also different from the index indicated by RBDC.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 374
BCM5718 Programmer’s Guide Receive BD Initiator Control Registers
Receive BD Initiator Local NIC Receive BD Producer Index Register
(offset: 0x2C0C–0x2C13)
This set of registers is used to keep track of the current DMAs queued to move receive BDs from the host to the NIC.
Standard Receive BD Producer Ring Replenish Threshold Register
(offset: 0x2C18)
Name
Reserved
BD Number
Bits Access
31:10 RO
9:0 RW
0
0
Default
Value Description
–
Number of buffer descriptors indicated by the receive producer index for the DMA engine to initiate a transfer of buffer descriptors for replenishing the ring.
Jumbo Receive BD Producer Ring Replenish Threshold Register
(offset: 0x2C1C)
All registers reset are core reset unless specified.
Name
Reserved
BD Number
Bits Access
31:10 RO
9:0 RW
0
0
Default
Value Description
–
Number of buffer descriptors indicated by the receive producer index for the DMA engine to initiate a transfer of buffer descriptors for replenishing the ring.
Standard Replenish LWM Register (offset 0x2D00)
Name
Legacy
Bits Access
31:8 RU
Default
Value
0x0
Description
Unused
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 375
BCM5718 Programmer’s Guide Receive BD Initiator Control Registers
Name Bits
Replenish Low Water Mark 7:0
Access
RW
Default
Value
0x0
Description
This field must be programmed with the desired
LWM for Standard RBD cache/memory. The value equally applies to all VRQs. Whenever the number of BDs in a VRQ’s BD memory falls below this (LWM+1), a BD fetch is request triggered.
Recommended Settings:
• Non-IOV mode:
STD LWM: 32. (1/4 of total RBDs)
• IOV mode:
STD LWM: 16. (1/2 of total RBDs)
Jumbo Replenish LWM Register (offset 0x2D04)
Name Bits Access
Legacy 31:8 RU
Replenish Low Water Mark 7:0 RW
Default
Value
0x0
0x0
Description
Unused
This field must be programmed with the desired
LWM for Jumbo RBD cache/memory. The value equally applies to all VRQs. Whenever the number of BDs in a VRQ’s BD memory falls below this (LWM+1), a BD fetch is request triggered.
Recommended Settings:
• Non-IOV mode:
JMB LWM: 16. (1/4 of total RBDs)
• IOV mode:
JMB LWM: 4. (1/4 of total RBDs)
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 376
BCM5718 Programmer’s Guide Receive BD Completion Control Registers
BD Fetch Limit Register (Offset 0x2D08)
This register is meaningful only in the IOV-Mode.
Name
Legacy
BD Fetch Limit
Table 102: BD Fetch Limit Register (Offset 0x2D08)
Bits
4:0
Access
31:5 RU
RW
Default
Value
0x0
0x1F
Description
Unused
The number of BDs fetched by a single DMA request shall be the lesser of the following:
• Space available in the respective BD cache.
• Standard or Jumbo Replenish Threshold.
• Number of BDs made available in the Host memory based Ring.
• Programmed Value of this Field.
Receive BD Completion Control Registers
All registers reset are core reset unless specified.
Receive BD Completion Mode Register (offset: 0x3000)
Name
Reserved
Attention Enable
Enable
Reset
Bits Access
31:3 RO
2 RW
0
0
Default
Value
1
0
RW
RW
1
0
Description
–
When this bit is set to 1, an internal attention is generated when an error occurs.
This bit controls whether the Receive BD
Completion state machine is active or not. When set to 0, it completes the current operation and cleanly halts. Until it is completely halted, it remains one when read.
When this bit is set to 1, the Receive BD
Completion state machine is reset. This is a selfclearing bit.
Receive BD Completion Status Register (offset: 0x3004)
Name
Reserved
Error
Bits Access
31:3 RO
2 RO
0
0
Default
Value Description
–
Receive BD Completion Error Status.
BROADCOM
®
July 17, 2013 • 5718-PG107-R Page 377
BCM5718 Programmer’s Guide Central Power Management Unit (CPMU) Registers
Name
Reserved
Bits Access
1:0 RO
Default
Value
0
Description
–
NIC Jumbo Receive BD Producer Index Register (offset: 0x3008)
All registers reset are core reset unless specified.
Name
Reserved
NIC Jumbo Receive BD
Producer Index
Bits Access
31:8 RO
7:0 RW
0
–
Default
Value Description
–
Current Jumbo Received BD have been fetched by RDMA module and are available for incoming
RX packets.
NIC Standard Receive BD Producer Index Register (offset: 0x300C)
Name
Reserved
NIC Standard Receive BD
Producer Index
Bits Access
31:9 RO
8:0 RW
0
–
Default
Value Description
–
–
Central Power Management Unit (CPMU) Registers
CPMU Control Register (offset: 0x3600)
This register is reset by POR Reset except for Powerdown bit (bit #2)
Name
Reserved
Software controlled GPHY
Force DLL on
Enable GPHY powerdown in
D0u (this feature is not used in
BCM5718 family)
Reserved
Bits Access
31:29 DC