Broadcom Stingray PS225 Quick Start Manual

Broadcom Stingray PS225, a high-performance SmartNIC adapter, offers exceptional network and compute capabilities in a compact design. With dual 25GbE SFP28 ports, it delivers blazing-fast data transfer speeds. The PS225 is powered by an 8-core ARMv8 processor running at 3.0 GHz, providing unparalleled processing power for demanding network applications. It also features up to 16GB of DDR4 memory, ensuring smooth handling of complex tasks.

PDF

ஆவணம்

விளம்பரம்

Stingray™ PS225

Quick Start Guide

User Guide

Broadcom

Confidential

Broadcom Confidential 5880X-PS225-UG101

December 12, 2018

Broadcom, the pulse logo, Connecting everything, Stingray, TruFlow, Avago Technologies, Avago, and the A logo are among the trademarks of Broadcom and/or its affiliates in the United States, certain other countries, and/or the EU.

The term “Broadcom” refers to Broadcom Inc. and/or its subsidiaries. For more information, please visit www.broadcom.com

Broadcom reserves the right to make changes without further notice to any products or data herein to improve reliability, function, or design. Information furnished by Broadcom is believed to be accurate and reliable. However, Broadcom does not assume any liability arising out of the application or use of this information, nor the application or use of any product or circuit described herein, neither does it convey any license under its patent rights nor the rights of others.

Broadcom

Confidential

Quick Start Guide

PS225

User Guide

1 Introduction

..................................................................................................................................................................4

1.1 Block Diagram......................................................................................................................................................4

1.2 Board Features ....................................................................................................................................................5

1.3 Package Contents................................................................................................................................................6

1.4 PS225 Card Variants ...........................................................................................................................................7

2 Checklist

.......................................................................................................................................................................8

3 Initial Board Setup and Connectivity

.......................................................................................................................10

3.1 PS225 Connector Locations ..............................................................................................................................10

3.2 Installing the PS225 ...........................................................................................................................................10

3.3 Setting Up the Serial Console (Optional) ...........................................................................................................11

3.4 Accessing the PS225 from an x86 Host ............................................................................................................12

3.4.1 Installing L2 Driver on the x86 Host.........................................................................................................12

3.4.2 Assigning an IP Address to the First New Host Interface ........................................................................13

3.4.3 Establishing an SSH Connection to the PS225 .......................................................................................16

4 Upgrading the PS225 Software

.................................................................................................................................17

5 Setting Up a Linux Distribution rootfs on the PS225

..............................................................................................19

5.1 Running Ubuntu on a PS225 .............................................................................................................................19

Confidential

Revision History ............................................................................................................................................................ 30

Broadcom

Broadcom Confidential 5880X-PS225-UG101

PS225

User Guide Quick Start Guide

1 Introduction

Broadcom ® PS225 SmartNIC adapters, based on Broadcom's latest data center SoC technology, offer groundbreaking network and compute power in a compact package. The PS225 adapters are PCI Express low-profile, half-length adapters, and provide two 25-Gigabit Ethernet (25GbE) ports along with eight 64-bit ARMv8 processors running at an unmatched

3.0 GHz clock rate. The adapters are available in a range of on-board DDR4 memory sizes. The PS225-H04, PS225-H08, and PS225-H16 adapters include 4 GB, 8 GB, and 16 GB of DDR4 memory, respectively.

1.1 Block Diagram

Figure 1 shows the main functional blocks of the PS225 SmartNIC adapter cards.

Figure 1: PS225 Block Diagram

PCIe Edge Connector

SMBus VPD

FRU

8L SPI

8 MB eMMC

16 GB

PCIe 3.0

BCM58802H

CPU

Subsystem

L3$

Accelerators Broadcom

Ethernet 25GbE SerDes

DDR4

Ch. 0

DDR4

Ch. 1

72b

DDR x16

DDR

DDR x16 x16

DDR x16

NC-SI 20-pin connector

UART 3.5 mm connector

SFP28 connector

Broadcom Confidential 5880X-PS225-UG101

PS225

User Guide Quick Start Guide



1.2 Board Features

PCIe Interface:

– x8 PCI Express 3.0 compliant

– Single-Root I/O Virtualization (SR-IOV) with up to 128 VFs

– Function Level Reset (FLR) support

Processing Subsystem:

– 8-core ARMv8 A72 64-bit processor subsystem at 3.0 GHz

– 16 MB total cache (8 MB L2 + 8 MB L3)

– Two channels (64-bit + ECC) of DDR4 memory at 2,400 MT/s

Hardware Accelerators:

– TruFlow™ configurable flow accelerator engine

– Line-rate Crypto engine with single-pass hashing and encryption

– RAID 5/6 engine

Network Interface:

– Dual-port SFP28 pluggable media interface, supporting 25GbE or 10GbE optical transceiver or direct-attach copper

(DAC) cables



– RDMA over Converged Ethernet (RoCE) v1 and v2

– Advanced congestion avoidance

– Virtual network termination – VXLAN, NVGRE, Geneve, GRE

– Multiqueue, NetQueue, and VMQ

– Tunnel-aware stateless offloads:



IPv4 and IPv6



TCP, UDP, and IP checksum

Large send offload (LSO)



Large receive offload (LRO)

TCP segmentation offload (TSO)

Receive-side scaling (RSS)

Transmit-side scaling (TSS)

Confidential



VLAN insertion/removal

– DCB support: PFC, ETS, QCN, DCBx

– Jumbo frames (up to 9 KB)

– Network boot (PXE, UEFI)

Security and Manageability:

– Secure Boot

– Secure Key Storage

– ARM PKA Engine

Broadcom

– NC-SI over RMB (via separate connector)

Form Factor:

– PCI Express CEM specification for half-height, half-length adapters

– I/O brackets available in both low profile and full height

Broadcom Confidential 5880X-PS225-UG101

PS225

User Guide

1.3 Package Contents

The PS225 package contains the following components:



PS225 adapter with low-profile bracket installed

Full-height bracket

The PS225 card with a low-profile bracket installed is shown in Figure 2 .

Figure 2: PS225 with Low-Profile Bracket

Quick Start Guide

Broadcom Confidential

Broadcom

Confidential

5880X-PS225-UG101

PS225

User Guide Quick Start Guide

1.4 PS225 Card Variants

The PS225 card comes in the following three variants based on the amount of DDR memory:



PS225-H04 (4 GB DDR4, part number BCM958802A8024C/BCM958802A8044C)

PS225-H08 (8 GB DDR4, part number BCM958802A8028C/BCM958802A8048C)

PS225-H16 (16 GB DDR4, part number BCM958802A8021C)

The back of the card contains a sticker that indicates the board type. Figure 3 shows an example of the 4G card label on a preproduction adapter.

Figure 3: Example Label

Broadcom Confidential

Broadcom

Confidential

5880X-PS225-UG101

PS225

User Guide Quick Start Guide

2 Checklist

The following list of major steps must be completed to complete the setup of the PS225:

1. Install the PS225 card in a host machine (see Initial Board Setup and Connectivity ).

2. Obtain the documentation listed in Table 1 .

Table 1: List of Reference Documentation

Document name Document number Description

SmartNIC Configuration Guide

5880X-PS225-AN1Xx Application note for configuring OVS+DPDK on the

PS225.

BCM5880X High Performance Ethernet System on Chip

5880X-DS1Xx BCM5880X Data Sheet

BCM5880X Hardware Design Guide

BCM5880X SmartNIC Solution

PS225 SmartNIC Adapters Dual-Port 25 Gb/s PCI

Express

5880X-DG1Xx

5880X-UG3Xx

Hardware design guide

BCM5880X SmartNIC solution

5880X-PS225-DS1Xx PS225 Advanced Data Sheet

3. Install the latest binary/image package for the PS225. Upgrade if the PS225 card received has an old image.

NOTE:

To check the image version information, refer to docSAFE (the Broadcom documentation portal) to download the binary package required for the specific card type.

Package name format : NXS_SN-<release type>_<release version>_<card type>_bin.tar.gz

Example: NXS_SN-BM-Beta_1.1.8.0_H08GB_bin.tar

4. Become familiar with the following PS225 basic information:

– PS225 software version:

Confidential

On PS225 Stingray (MAIA) console: root@bcm958802a8048c:~# uname -a

Linux bcm958802a8048c 4.14.79+g5bb1623 #1 SMP Tue Nov 27 01:48:50 UTC 2018 aarch64 aarch64 aarch64 GNU/Linux

– PS225 board/card type (8 GB, 4 GB, 16 GB):

On PS225 Stingray (MAIA) console: root@bcm958802a8048c:~# hostname bcm958802a8048c

NOTE:

Broadcom

Broadcom Confidential 5880X-PS225-UG101

PS225

User Guide

Supported Nitro profile configurations in the PS225 released image package are as follows: stingray_4p.pkg

stingray_2p.pkg

stingray_1p.pkg

BCM958802A8048C.pkg

bcm958802a8048_8+8_pf_rdma.pkg

bcm958802a8048_2xAN.pkg

bcm958802a8048_2xAN_8+5_pf_rep.pkg

bcm958802a8048_2xAN_8+4_pf_rep.pkg

bcm958802a8048_2xAN_8+4_pf.pkg

bcm958802a8048_2xAN_8+3_pf.pkg

bcm958802a8048_2xAN_8+2_baremetal.pkg

bcm958802a8048_2x25g.pkg

bcm958802a8048_2x25g_8+5_pf_rep.pkg

bcm958802a8048_2x25g_8+4_pf_rep.pkg

bcm958802a8048_2x25g_8+4_pf.pkg

bcm958802a8048_2x25g_8+3_pf.pkg

bcm958802a8048_2x25g_8+2_baremetal.pkg

bcm958802a8048_2x10g.pkg

bcm958802a8048_2x10g_8+5_pf_rep.pkg

bcm958802a8048_2x10g_8+4_pf_rep.pkg

bcm958802a8048_2x10g_8+4_pf.pkg

bcm958802a8048_2x10g_8+3_pf.pkg

bcm958802a8048_2x10g_8+2_baremetal.pkg

bcm958802a8048_1xAN.pkg

bcm958802a8048_1x25g.pkg

bcm958802a8048_1x10g.pkg

Confidential

Quick Start Guide

Broadcom

Broadcom Confidential 5880X-PS225-UG101

PS225

User Guide

3 Initial Board Setup and Connectivity

This section describes the initial board setup and connectivity.

Quick Start Guide

3.1 PS225 Connector Locations

There are two SFP28 ports and a UART console port for the ARM CPU subsystem located on the I/O panel of the board

(see Figure 4 ).

Figure 4: PS225 Connector Locations

ARM A72

UART Console

Port 0 Port 1

NOTE:

Confidential and can be accessed using a widely available USB-to-TTL cable of type TTL-232R-3V3-AJ (based on FTDI chip).

3.2 Installing the PS225

To install the PS225:

1. Power down the host system and remove the AC power.

2. Insert the PS225 card into a PCIe 3.0 slot (at least x8 size) in the host system.

3. Connect one end of the SFP cable to port 0 or port 1 on the PS225 (see Figure 5 ). Broadcom

Figure 5 ).

NOTE:

Both Ethernet ports must be connected to link partners operating at the same speed: either both 25GbE or both

10GbE. The ports do not support mixing 25GbE and 10GbE links.

5. Optionally: a. Connect the other port of the PS225 card to an Ethernet switch (see Figure 5 ).

b. Insert the 3.5 mm plug of a USB serial cable to the 3.5 mm socket on the front panel (see Figure 5 ).

Broadcom Confidential 5880X-PS225-UG101

Quick Start Guide

PS225

User Guide

Figure 5: Connecting the PS225 Cables

3.3 Setting Up the Serial Console (Optional)

To setup the optional serial console:

1. Insert one end of the 3.5 mm jack of the provided cable to the card and the other end of the cable (USB-A) into a PC with an installed terminal emulator program.

2. Install the appropriate UART-USB driver:

– For a Windows OS, the driver for the UART-USB transceiver is downloaded from the FTDI driver download site

( 2.12.28 , http://www.ftdichip.com/Drivers/CDM/CDM21228_Setup.zip

– For a Linux OS, the device is automatically enumerated as one of the USB tty (

/dev/ttyUSBx

– If another operating system is used, see the FTDI website for available drivers.

– Baud rate: 115200

Confidential

– Data: 8-bit

– No parity

– Stop bit

– No flow control

A typical Linux setup for Minicom is shown in the following screen capture:

Broadcom

Broadcom Confidential 5880X-PS225-UG101

PS225

User Guide Quick Start Guide

3.4 Accessing the PS225 from an x86 Host

NOTE:

The instructions in this document assume that the GA 1.0.2 (or later) SmartNIC software release is preloaded on the PS225. Contact Broadcom support regarding the software version that was delivered on the card. If a card is delivered with older software, contact Broadcom support for upgrade procedures. The upgrade procedure described in this document does not apply to releases earlier than GA 1.0.2.

When the PS225 boots up for the very first time, it assigns a fixed IP address of 192.168.1.10 to the first PF on the ARM side.

The x86 host can access this IP address after the following actions:

1. Install appropriate Layer 2 (L2) bnxt driver.

2. Assign IP address on the new x86 network interface.

3. Verify that the interface is up and check the link status ( ip link show

or ifconfig

4. Establish an SSH connection from the x86 host to the SSH server running on the PS225 ARM Linux.

NOTE:

The initial IP address of the first interface on the PS225 card (192.168.1.10) can be changed by modifying

/etc/ network/interfaces

using either the serial console or after the first successful ssh to the card. The change in

/etc/network/interfaces

takes effect on the next reboot of the card. If the IP address is changed during the active ssh session, ensure that the ssh reconnect is to the new IP address.

3.4.1 Installing L2 Driver on the x86 Host

The L2 driver source code can be provided as part of the SmartNIC release or as a separate release package. Before running the following script, install the packages in advance that have dependency with L2 Driver: sudo apt-get install cmake libpci-dev

Confidential

1. If the driver is provided as part of the release package, run the provided install script.

Example:

./poky/images/bcm958802a8021/x86-host-install.sh

This script creates a subdirectory brcm-nxt-l2-drv

where the driver is compiled. The current version of the install script intentionally omits installing the driver as part of the kernel. An insmod

must be used instead of modprobe

, but prior to using insmod,

the dependency modules must be installed.

Example: sudo modprobe ptp sudo modprobe vxlan

Broadcom sudo modprobe devlink

# from the top-level directory above poky: sudo insmod brcm-nxt-l2-drv/git/main/Cumulus/drivers/linux/v3/bnxt_en.ko

Broadcom Confidential 5880X-PS225-UG101

PS225

User Guide Quick Start Guide

2. If the driver is provided separately, it must be extracted and compiled for the kernel running on the x86 host and installed.

Example commands are provided below:

NOTE:

It is assumed that the delivered L2 source code is in the tarball brcm-nxt-l2-drv.tar.gz

tar zxf brcm-nxt-l2-drv.tar.gz

cd git/main/Cumulus/drivers/linux/v3/ make sudo make install

The driver name is bnxt_en.ko

and the last command installs it in the system in the

/lib/modules/<kernelversion>/updates/

directory.

3. Install the driver using the following command: sudo modprobe bnxt_en

3.4.2 Assigning an IP Address to the First New Host Interface

After a modprobe

or insmod

of the bnxt_en

driver, the new network interfaces show up on the x86 host. The number of interfaces/PFs that show up on the x86 host and separately on the ARM side is Nitro firmware-dependent. An example for x86 is shown below and the interfaces are highlighted

NOTE:

The exact names may differ from system to system and depend on the OS being used and the physical slot where the card is installed.

$ ip link show

1000

link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

2: enp0s31f6: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000

link/ether b0:6e:bf:30:44:65 brd ff:ff:ff:ff:ff:ff

19: enp1s0f0 : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

link/ether 00:10:18:de:06:08 brd ff:ff:ff:ff:ff:ff

20: enp1s0f1d1 : <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP mode DEFAULT group default qlen 1000

link/ether 00:10:18:de:06:09 brd ff:ff:ff:ff:ff:ff

21: enp1s0f2 : <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000

link/ether 00:10:18:de:06:0a brd ff:ff:ff:ff:ff:ff

22: enp1s0f3d1 : <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000 default qlen 1000

Broadcom

23: enp1s0f4 : <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group

link/ether 00:10:18:de:06:0c brd ff:ff:ff:ff:ff:ff

24: enp1s0f5d1 : <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000

link/ether 00:10:18:de:06:0d brd ff:ff:ff:ff:ff:ff

25: enp1s0f6 : <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000

link/ether 00:10:18:de:06:0e brd ff:ff:ff:ff:ff:ff

26: enp1s0f7d1 : <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000

link/ether 00:10:18:de:06:0f brd ff:ff:ff:ff:ff:ff

Broadcom Confidential 5880X-PS225-UG101

PS225

User Guide Quick Start Guide

1. Assign an IP address on the 192.168.1.x subnet to the interface for function 0 (in the previous example, it is enp1s0f0).

Example: sudo ifconfig enp1s0f0 192.168.1.20 up

# OR sudo ip addr add 192.168.1.20/24 dev enp1s0f0

2. Check the new IP address with either an ifconfig

PS225) can now be pinged.

or ip addr show

command. The PS225 (the ARM host on the

Example:

$ ping 192.168.1.10

PING 192.168.1.10 (192.168.1.10) 56(84) bytes of data.

64 bytes from 192.168.1.10: icmp_seq=1 ttl=64 time=0.131 ms

64 bytes from 192.168.1.10: icmp_seq=2 ttl=64 time=0.046 ms

...

Figure 6 shows the connectivity of the first interface on the x86 host and the first interface on the ARM side. 192.168.1.20 is assigned to enp1s0f0 (on the x86 host) and 192.168.1.10 to enP8p1s0f0np0 on the ARM host on the PS225. These two interfaces are connected through Nitro (red arrow).

NOTE:

As shown in Figure 6 , each interface can send packets out on the 25GbE port (blue arrow for the x86 enp1s0f0

Figure 6: Interface Connectivity

Confidential

Host CPUs

enp1s0f0 (PF8)

PCIe x8

Broadcom

NIC

192.168.1.10 is always assigned to enp8p1s0f0np0 interface on new PS225 boards. Change

/etc/network/interfaces as appropriate for your setup.

NIC

PF0 PF1

A72 CPUs

A72(MAIA) cores to ports

Host(x86) to ports

Host(x86) to A72(MAIA) cores

Broadcom Confidential 5880X-PS225-UG101

PS225

User Guide Quick Start Guide

When sending packets to the external ports, the even-numbered interfaces on the x86 host (for example, enp1s0f0, enp1s0f2, enp1s0f4, and enp1s0f6) are connected to external port 0. This also applies to the even-numbered interfaces on the ARM side (enP8p1s0f0np0, enP8p1s0f2np0, enP8p1s0f4np0, and enP8p1s0f6np0). The odd-numbered interfaces on both the x86 host side and the ARM side are connected to external port 1.

Use the following steps to permanently assign a static IP address to the interface:



For Debian-based Linux OS (for example, Ubuntu) set a static IP address in

/etc/network/interface

with the following commands: auto enp1s0f0 iface enp1s0f0 inet static

address 192.168.1.20

netmask 255.255.255.0



For CentOS, create a new file using the following commands:

/etc/sysconfig/network-scripts/ifcfg-ens4f0

NOTE:

The previous example is provided for an interface named ens4f0.

The interface name may be different on different systems.

Add the following content:

TYPE="Ethernet"

PROXY_METHOD="none"

BROWSER_ONLY="no"

BOOTPROTO="static"

DEFROUTE="yes"

NAME=" ens4f0 "

UUID="

660e94eb-a5b4-4975-8139-388675307f7d

DEVICE=" ens4f0 "

ONBOOT="no"

IPADDR=192.168.1.20

NETMASK=255.255.255.0

GATEWAY=192.168.1.20

Confidential

The content can be completely reused, but the highlighted

items must be replaced. The variables NAME and DEVICE contain the interface name that was determined above. The variable UUID contains the ID of the interface, which can be obtained by running uuidgen ens4f0

Broadcom

Broadcom Confidential 5880X-PS225-UG101

PS225

User Guide Quick Start Guide

3.4.3 Establishing an SSH Connection to the PS225

To establish an SSH connection to the PS225:

Assuming that the PS225 is available (through ping), the x86 host can access the ARM host on the PS225 by running ssh using the following command: ssh [email protected]

No password is required. The following command line prompt displays:

$ ssh [email protected]

Last login: Fri Feb 16 22:19:21 2018 root@bcm958802a8048c:~#

During the first run, it is recommended to execute uname -a

to verify that the latest kernel available for the PS225 is being used (verify the latest kernel with the Broadcom support team).

Confidential

Broadcom

Broadcom Confidential 5880X-PS225-UG101

PS225

User Guide Quick Start Guide

4 Upgrading the PS225 Software

The upgrade procedure for PS225 relies on an external TFTP or web server. Normally this server is the machine that hosts the PS225, but it can also be an external server on the network. The binary image must be placed in a subdirectory on the

TFTP/web server.

To upgrade the PS225 software:

1. From the host SSH to the PS225 using the following command: ssh [email protected]

2. Run the update script with the following commands: root@bcm958802a8048c:~# update-me.sh -s <serverip> -d <dir> -i <image> -t <transport>

-s <serverip> IPv4 address of http or tftp server

-d <dir> Directory under the top-level TFTP or HTTP server directory

-i <image> Image can take one of the following values (or maybe newer ones) all Copies all files, dtb, fip, kernel, nitro and rootfs dtb Copies dtb only fip Copies fip only kernel Copies fip kernel only nitro Copies nitro firmware only nonitro Copies all but nitro noroot Copies fip, dtb and kernel, nitro, no rootfs rootfs Copies rootfs only

-t <transport> Transport (tftp or http) (default tftp)

Confidential

-r <yes/no/off> Reboot (yes) or Poweroff (off) or nothing after upgrade (default no)

-f <file> file to download and run (default updateme.sh)

Example: root@bcm958802a8048c:~# update-me.sh -s 192.168.1.20 -d nic -i all -t tftp

3. Reboot the PS225 once the image update has completed. The reboot command must be applied on the ARM side.

NOTE:

If the reboot or poweroff command is applied on the x86 host after the PS225 update, ensure that the L2 driver bnxt_en

is removed ( sudo modprobe -r bnxt_en

command on the x86 host side) so that the resources on the host are free.

Broadcom

Broadcom Confidential 5880X-PS225-UG101

PS225

User Guide

Figure 7: Upgrading the PS225 Software

TFTP server or

HTTP server

192.168.1.20

Host CPUs

enp1s0f0 (PF8)

PCIe x8

NIC

PF0

NIC

PF1

update-me.sh –s 192.168.1.20 –d

A72 CPUs

nic –i all –t tftp

Execute on the A72(MAIA console)

Confidential

Quick Start Guide

Broadcom

Broadcom Confidential 5880X-PS225-UG101

PS225

User Guide

5 Setting Up a Linux Distribution rootfs on the PS225

This section describes how to set up a Linux distribution rootfs on the PS225.

Quick Start Guide

5.1 Running Ubuntu on a PS225

This section describes basic Ubuntu 16.04 setup on a PS225 board. It is assumed that an x86 host with Ubuntu 16.04 is being used. With small changes, it should apply to other variants as well.

NOTE:

Follow Upgrading the PS225 Software and successfully boot to Linux from the most recent SmartNIC release on the PS225.

5.1.1 Setting Up Ubuntu on a PS225

To set up Ubuntu on a PS225:

1. SSH to the PS225 card or use the serial console to access the PS225 Linux shell.

2. Create a new ext4 partition on the eMMC card. Ensure that the existing partitions used for UEFI (and yocto) are not deleted.

3. Run fdisk -l to verify the current partitions on

/dev/mmcblk0

If mmcblk0p5 is already created and it is 2 GB, skip Step 3 .

NOTE: mmcblk0p5 may have already been created during installation of the SmartNIC release.

NOTE:

In future releases, the partition layout may change. Check the release documentation before proceeding with

Step 4 and Step 5 .

4. Use the fdisk

command to create

/dev/mmcblk0p5

. Skip this step if it has already been completed. Refer to the following example command:

# boot normally, so from Yocto built kernel and RootFS root@bcm958802a8048c:~# fdisk /dev/mmcblk0

type 'n' to create a new partition

used defaults (will created mmcblk0p5)

typed 'w' to write changes

5. Format the partition using the following command: Broadcom

NOTE:

This step takes several minutes.

6. Mount the newly created partition to a directory of choice (for example, /mnt) using the following commands: root@bcm958802a8048c:~# mkdir /mnt/ubuntu root@bcm958802a8048c:~# mount /dev/mmcblk0p5 /mnt/ubuntu

7. Download the latest 16.04 ARM64 image from Ubuntu to the x86 host and untar it on the PS225. See the following commands:

Broadcom Confidential 5880X-PS225-UG101

PS225

User Guide Quick Start Guide

NOTE:

Ensure that there is connectivity with the host (follow the

PS225 Quick Start Guide

in Table 1 for additional details).

a. On the host, download the Ubuntu image with the following commands: byang@byang-PowerEdge-2950:~$ wget http://cloud-images.ubuntu.com/releases/16.04/release/ubuntu-

16.04-server-cloudimg-arm64-root.tar.xz b. Ensure connectivity between host and PS225 with the following commands (see the

PS225 Quick Start Guid e in

Table 1 ): byang@byang-PowerEdge-2950:~$ sudo insmod ~/temp/bnxt_en-1.8.29/bnxt_en.ko byang@byang-PowerEdge-2950:~$ sudo ifconfig enp8s0f0 192.168.1.20 up ; ping 192.168.1.10

Pinging 192.168.1.10 with 32 bytes of data:

Reply from 192.168.1.10: bytes=32 time=1ms TTL=128

Ping statistics for 192.168.1.10:

Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),

Approximate round trip times in milli-seconds:

Minimum = 1ms, Maximum = 1ms, Average = 1ms c. On the PS225, scp the Ubuntu image with the following command: root@bcm958802a8048c:~#

Confidential root@bcm958802a8048c:~# date -s "14 MAR 2018 20:00:00" root@bcm958802a8048c:~# cd /mnt/ubuntu root@bcm958802a8048c: /mnt/ubuntu# tar --numeric-owner -xf /mnt/ubuntu/ubuntu-16.04-servercloudimg-arm64-root.tar.xz

8. The new rootfs is extracted in the specified partition (in the example above

/dev/mmcblk0p5

). chroot

to the device to set up several items before the kernel boots that rootfs for the first time. This step is important, otherwise the Ubuntu

OS does not boot properly. See the following commands: a. Copy the bnxt* modules from the yocto filesystem to the ubuntu rootfs with the following command: Broadcom root@bcm958802a8048c: /mnt/ubuntu# chroot /mnt/ubuntu c. Remove cloud init from this prebuilt image (since it interferes with normal boot) with the following command:

\u@\h:\w$ dpkg --purge cloud-init d. Add getty for ttyS0 with the following commands:

\u@\h:\w$ cp /etc/init/ttyAMA0.conf /etc/init/ttyS0.conf

Broadcom Confidential 5880X-PS225-UG101

Quick Start Guide

PS225

User Guide

\u@\h:\w$ sed 's/ttyAMA0/ttyS0/g' -i /etc/init/ttyS0.conf

e. Set up the user lab with the following commands:

\u@\h:\w$ adduser lab f. Enter the user name and password information.

g. Add the user lab to admin and sudo groups with the following commands:

\u@\h:\w$ addgroup lab admin

\u@\h:\w$ addgroup lab sudo

\u@\h:\w$ exit

9. Unmount the device, reboot, and stop at the UEFI prompt: root@bcm958802a8048c: /mnt/ubuntu# cd /mnt root@bcm958802a8048c: /mnt# umount /mnt/ubuntu root@bcm958802a8048c: /mnt# reboot

10. The system is now ready to boot the new Ubuntu rootfs, however, the new rootfs to be used by kernel must be specified.

The UEFI must use Ubuntu rootfs for boot. The UEFI then sets up Linux args automatically. See the following commands:

Press any key within 5 seconds

AUTOBOOT SKIPPED

Shell> gpt list blk0

Shell> select rootfs-ordinal 5

Shell> select

Shell> startup

Confidential

The login prompt and username/password that was set up above is now available. This may take several minutes to get to the login prompt. See the following commands:

NOTE:

It is possible to switch back to yocto Linux by performing the following using the UEFI.

Shell> select rootfs-ordinal 3

Shell> startup

11. Acquire network connectivity with host using the following commands:

NOTE:

Broadcom a. On the PS225 console, (optionally) get rid of the warning "sudo: unable to resolve host ubuntu: Connection refused" by editing /etc/hosts, and add the following line (in blue): lab@ubuntu:/$ sudo nano /etc/hosts

127.0.0.1 localhost

127.0.1.1 ubuntu b. Continue to load the driver and setup the network interface as follows: lab@ubuntu:~$ sudo depmod -a # apply only at the very first boot

Broadcom Confidential 5880X-PS225-UG101

Quick Start Guide

PS225

User Guide lab@ubuntu:~$ sudo modprobe bnxt_en lab@ubuntu:~$ ifconfig -a

Note the name of the interface slot that is plugged into the PS225. lab@ubuntu:/$ sudo ifconfig enP8p1s0f0np0 up 192.168.1.10 c. On the host, insmod the bnxt_en.ko module. See the PS225 Quick Start Guide to find out where to get/build this module for the x86 host. The following command operates on the example setup. byang@byang-PowerEdge-2950:~$ sudo insmod ~/temp/bnxt_en-1.8.29/bnxt_en.ko byang@byang-PowerEdge-2950:~$ sudo ifconfig enp8s0f0 192.168.1.20 up ; ping 192.168.1.10

d. Access between the host and PS225 should be now available.

12. Allow the PS225 to reach the Internet, so the user can use apt install to retrieve software from the Internet. We configure the host to forward using the following commands:

NOTE:

If your PS225's SFP28 port is connected to the Internet (for example, via a switch and DHCP server), then there is no need for host to forward Internet traffic. Skip the following steps.

NOTE:

"eno2" is the regular Ethernet interface that the host uses to access the Internet while enp8s0f0 is the first PF towards the PS225 card. On different machines these could be different. Adjust the commands based on the specific setup.

a. On the example host, the result from ifconfig is as follows: byang@byang-PowerEdge-2950:

Confidential

inet6 addr: fe80::116e:caa:13f3:1dbf/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:23392 errors:0 dropped:0 overruns:0 frame:0

TX packets:16471 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:2509414 (2.5 MB) TX bytes:2926607 (2.9 MB) enp8s0f0 Link encap:Ethernet HWaddr 00:10:18:ad:05:08

inet addr:192.168.1.20 Bcast:192.168.1.255 Mask:255.255.255.0

inet6 addr: fe80::210:18ff:fead:508/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:1661 errors:0 dropped:0 overruns:0 frame:0

Broadcom enp8s0f2 Link encap:Ethernet HWaddr 00:10:18:ad:05:0a

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:1183 errors:0 dropped:0 overruns:0 frame:0

TX packets:1321 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:381443 (381.4 KB) TX bytes:271592 (271.5 KB) enp8s0f4 Link encap:Ethernet HWaddr 00:10:18:ad:05:0c

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:1186 errors:0 dropped:0 overruns:0 frame:0

Broadcom Confidential 5880X-PS225-UG101

PS225

User Guide Quick Start Guide

TX packets:1307 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:382936 (382.9 KB) TX bytes:267866 (267.8 KB) enp8s0f6 Link encap:Ethernet HWaddr 00:10:18:ad:05:0e

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:1174 errors:0 dropped:0 overruns:0 frame:0

TX packets:1315 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:378828 (378.8 KB) TX bytes:270156 (270.1 KB) enp8s0f1d1 Link encap:Ethernet HWaddr 00:10:18:ad:05:09

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:189 errors:0 dropped:0 overruns:0 frame:0

TX packets:1312 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:65394 (65.3 KB) TX bytes:270596 (270.5 KB) enp8s0f3d1 Link encap:Ethernet HWaddr 00:10:18:ad:05:0b

UP BROADCAST MULTICAST MTU:1500 Metric:1

RX packets:437 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:151202 (151.2 KB) TX bytes:0 (0.0 B) enp8s0f5d1 Link encap:Ethernet HWaddr 00:10:18:ad:05:0d

UP BROADCAST MULTICAST MTU:1500 Metric:1

RX packets:437 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

Confidential enp8s0f7d1 Link encap:Ethernet HWaddr 00:10:18:ad:05:0f

UP BROADCAST MULTICAST MTU:1500 Metric:1

RX packets:437 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:151202 (151.2 KB) TX bytes:0 (0.0 B) lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

inet6 addr: ::1/128 Scope:Host

UP LOOPBACK RUNNING MTU:65536 Metric:1

RX packets:2002 errors:0 dropped:0 overruns:0 frame:0 Broadcom b. On the host (assuming that it is Ubuntu 16), ensure that it is changed based on the distribution. c. Edit

/etc/sysctl.conf

and uncomment the following line: net.ipv4.ip_forward=1 byang@byang-PowerEdge-2950:~$ sudo iptables -t nat -A POSTROUTING -o eno2 -j MASQUERADE byang@byang-PowerEdge-2950:~$ sudo iptables -A FORWARD -i eno2 -o enp8s0f0 -m state --state

RELATED,ESTABLISHED -j ACCEPT byang@byang-PowerEdge-2950:~$ sudo iptables -A FORWARD -i enp8s0f0 -o eno2 -j ACCEPT d. On PS225, configure default gateway and DNS server with the following commands:

Broadcom Confidential 5880X-PS225-UG101

PS225

User Guide Quick Start Guide lab@ubuntu:/$ sudo route add default gw 192.168.1.20

lab@ubuntu:/$ sudo nano /etc/resolvconf/resolv.conf.d/head e. Add the following line to /etc/resolvconf/resolv.conf.d/head: nameserver 8.8.8.8 lab@ubuntu:/$ sudo resolvconf -u f. Test it by pinging Google: lab@ubuntu:/$ ping google.com g. Install something useful with the apt command: lab@ubuntu:/$ sudo apt update

Install gcc, for example. lab@ubuntu:/$ sudo apt install gcc lab@ubuntu:~$ sudo su root@ubuntu:~$ sed 's/PasswordAuthentication/# PasswordAuthentication/g' -i /etc/ssh/sshd_config root@ubuntu:/home/lab# dpkg-reconfigure openssh-server h. On the host, SSH from the host to the PS225 using the following command: byang@byang-PowerEdge-2950:~$ ssh [email protected]

Confidential

Broadcom

Broadcom Confidential 5880X-PS225-UG101

PS225

User Guide Quick Start Guide

5.2 Running CentOS on a PS225

This section describes basic CentOS 7 setup on a PS225 board. It is assumed that an x86 host with Ubuntu 16.04 is being used. With small changes, it should apply to other variants as well.

NOTE:

Follow Upgrading the PS225 Software and successfully boot to Linux from the most recent SmartNIC release on the PS225.

5.2.1 Setting Up CentOS on the PS225

To set up CentOS on the PS225:

1. SSH to the PS225 card or use the serial console to access the PS225 Linux shell.

2. Create a new ext4 partition on the eMMC card.

CAUTION!

Ensure that the existing partitions used for UEFI (and Yocto) are not deleted.

If mmcblk0p5 is already created and it is 2 GB, skip Step 3 . mmcblk0p5 is already created during the installation of GA

1.0.3.

NOTE:

In future releases, the partition layout may change. Check the release documentation before proceeding with

Step 4 and Step 5 .

3. Run fdisk -l

to verify the current partitions on

/dev/mmcblk0

4. Create

/dev/mmcblk0p5

with the following fdisk

command. Skip this step if this has already been done.

root@bcm958802a8048c:~# fdisk /dev/mmcblk0

type 'n' to create a new partition

used defaults (will created mmcblk0p5)

typed 'w' to write changes

Confidential

5. Format the partition with the following command: root@bcm958802a8048c:~# mkfs.ext4 /dev/mmcblk0p5

NOTE:

This step can take several minutes to complete.

6. Mount the newly created partition to a directory of choice with the following command (for example. /mnt): root@bcm958802a8048c:~# mkdir /mnt/centos Broadcom connectivity with the host (see the PS225 Quick Start Guide for additional details).

a. Download the image with the following command: byang@byang-PowerEdge-2950: ~$wget http://mirror.centos.org/altarch/7/isos/aarch64/CentOS-7aarch64-rootfs-7.4.1708.tar.xz b. Ensure connectivity with the host and PS225 with the following commands: byang@byang-PowerEdge-2950:~$ sudo insmod ~/temp/bnxt_en-1.8.29/bnxt_en.ko byang@byang-PowerEdge-2950:~$ sudo ifconfig enp8s0f0 192.168.1.20 up ; ping 192.168.1.10

c. On the PS225, SCP the CentOS image with the following command: cd /mnt/centos

Broadcom Confidential 5880X-PS225-UG101

PS225

User Guide Quick Start Guide root@bcm958802a8048c:/mnt/centos# scp [email protected]:~/CentOS-7-aarch64-rootfs-

7.4.1708.tar.xz

root@bcm958802a8048c:/mnt/centos# tar --numeric-owner -xf CentOS-7-aarch64-rootfs-

7.4.1708.tar.xz

d. Free up space with the following command: root@bcm958802a8048c:/mnt/centos# rm CentOS-7-aarch64-rootfs-7.4.1708.tar.xz

8. With the new rootfs extracted in the specified partition (in the example,

/dev/mmcblk0p5

), chroot to the device to set up the device before the kernel boots that rootfs for the first time. This step is important, otherwise the Ubuntu OS will not boot properly.

a. Copy the bnxt* modules from the Yocto filesystem to the ubuntu rootfs with the following command: cp -r /lib/modules/`uname -r` lib/modules b. Switch to CentOS's filesystem through chroot with the following commands: chroot /mnt/centos adduser lab passwd lab gpasswd -a lab wheel exit cd umount /mnt/centos reboot

9. Unmount the device and stop at the UEFI prompt:

NOTE:

Confidential

Press any key within 5 seconds

AUTOBOOT SKIPPED

Shell> gpt list blk0

Shell> select rootfs-ordinal 5

Shell> select

Shell> startup

At this point, it is possible to reach the login prompt and log in with the username/password that was created in the previous steps. It may take a few minutes to reach the login prompt.

NOTE:

Broadcom startup

10. Provide connectivity to the host using the following steps: a. Install modules copied in Step 8 , Step a .

NOTE:

If the insmod'ed bnxt_en.ko module in Yocto Linux has been previously installed, power cycle the system and try again.

Broadcom Confidential 5880X-PS225-UG101

Quick Start Guide

PS225

User Guide b. On the PS225 console, run the following commands:

[lab@localhost ~] $ sudo depmod -a # apply only at the very first boot

[lab@localhost ~] $ sudo modprobe bnxt_en.

11. On the PS225, edit

/etc/sysconfig/network-scripts/ifcfg-enP8p1s0f0np0

and add the following lines:

DEVICE= enP8p1s0f0np0

GATEWAY=192.168.1.20

IPADDR=192.168.1.10

NETMASK=255.255.255.0

DNS1=8.8.8.8

[lab@localhost ~] $ sudo ifup enP8p1s0f0np0

[lab@localhost ~] $ sudo ip addr show a. On the host, insmod to bnxt_en.ko

module using the following commands (see the PS225 Quick Start Guide for information on building this module on a x86 host): sudo insmod ~/temp/bnxt_en-1.8.29/bnxt_en.ko sudo ifconfig enp8s0f0 192.168.1.20 up ; ping 192.168.1.10

b. The ping to the PS225 board should be successful. SSH and SCP between the host and the PS225 with the following command:

# ssh [email protected]

12. All the PS225s must be able to access the Internet so that the yum install software from the Internet.

command can be used to retrieve the

NOTE:

The results of ifconfig on the host are as follows:

Confidential

NOTE:

Eno2 is the regular Ethernet interface that the host uses to access the Internet while enp8s0f0 is the first PF towards the PS225 card. On different systems these items could be different. Adjust these items based on your setup.

byang@byang-PowerEdge-2950: ~/temp/bnxt_en-1.8.29$ ifconfig eno2 Link encap:Ethernet HWaddr 00:22:19:91:7c:e4

inet addr:10.67.93.199 Bcast:10.67.93.255 Mask:255.255.254.0

inet6 addr: fe80::116e:caa:13f3:1dbf/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 Broadcom

RX bytes:2509414 (2.5 MB) TX bytes:2926607 (2.9 MB) enp8s0f0 Link encap:Ethernet HWaddr 00:10:18:ad:05:08

inet addr:192.168.1.20 Bcast:192.168.1.255 Mask:255.255.255.0

inet6 addr: fe80::210:18ff:fead:508/64 Scope:Link

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:1661 errors:0 dropped:0 overruns:0 frame:0

TX packets:344 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:481235 (481.2 KB) TX bytes:32437 (32.4 KB)

Broadcom Confidential 5880X-PS225-UG101

PS225

User Guide enp8s0f2 Link encap:Ethernet HWaddr 00:10:18:ad:05:0a

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:1183 errors:0 dropped:0 overruns:0 frame:0

TX packets:1321 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:381443 (381.4 KB) TX bytes:271592 (271.5 KB) enp8s0f4 Link encap:Ethernet HWaddr 00:10:18:ad:05:0c

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:1186 errors:0 dropped:0 overruns:0 frame:0

TX packets:1307 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:382936 (382.9 KB) TX bytes:267866 (267.8 KB) enp8s0f6 Link encap:Ethernet HWaddr 00:10:18:ad:05:0e

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:1174 errors:0 dropped:0 overruns:0 frame:0

TX packets:1315 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:378828 (378.8 KB) TX bytes:270156 (270.1 KB) enp8s0f1d1 Link encap:Ethernet HWaddr 00:10:18:ad:05:09

UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1

RX packets:189 errors:0 dropped:0 overruns:0 frame:0

TX packets:1312 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:65394 (65.3 KB) TX bytes:270596 (270.5 KB) enp8s0f3d1 Link encap:Ethernet HWaddr 00:10:18:ad:05:0b Confidential

collisions:0 txqueuelen:1000

RX bytes:151202 (151.2 KB) TX bytes:0 (0.0 B) enp8s0f5d1 Link encap:Ethernet HWaddr 00:10:18:ad:05:0d

UP BROADCAST MULTICAST MTU:1500 Metric:1

RX packets:437 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:151202 (151.2 KB) TX bytes:0 (0.0 B) enp8s0f7d1 Link encap:Ethernet HWaddr 00:10:18:ad:05:0f

UP BROADCAST MULTICAST MTU:1500 Metric:1 Broadcom

RX bytes:151202 (151.2 KB) TX bytes:0 (0.0 B) lo Link encap:Local Loopback

inet addr:127.0.0.1 Mask:255.0.0.0

inet6 addr: ::1/128 Scope:Host

UP LOOPBACK RUNNING MTU:65536 Metric:1

RX packets:2002 errors:0 dropped:0 overruns:0 frame:0

TX packets:2002 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:170071 (170.0 KB) TX bytes:170071 (170.0 KB)

Quick Start Guide

Broadcom Confidential 5880X-PS225-UG101

PS225

User Guide

13. On the host (assume it is Ubuntu 16), ensure that the change is based on your distribution.

14. Edit

/etc/sysctl.conf and uncomment the following line:

Quick Start Guide net.ipv4.ip_forward=1 byang@byang-PowerEdge-2950:~$ sudo iptables -t nat -A POSTROUTING -o eno2 -j MASQUERADE byang@byang-PowerEdge-2950:~$ sudo iptables -A FORWARD -i eno2 -o enp8s0f0 -m state --state

RELATED,ESTABLISHED -j ACCEPT byang@byang-PowerEdge-2950:~$ sudo iptables -A FORWARD -i enp8s0f0 -o eno2 -j ACCEPT

15. Test the configuration by accessing www.google.com.

lab@ubuntu:/$ ping google.com

Broadcom

Confidential

Broadcom Confidential 5880X-PS225-UG101

Quick Start Guide

PS225

User Guide

Revision History

5880X-PS225-UG101; December 12, 2018

Updated:



Upgrading the PS225 Software

5880X-PS225-UG100; October 4, 2018

Initial release.

Broadcom Confidential

Broadcom

Confidential

5880X-PS225-UG101

Broadcom

Confidential

வணக்கம்! Broadcom Stingray PS225 Quick Start Manual உடன் உங்களுக்கு உதவ சிறப்பாகப் பயிற்றுவிக்கப்பட்ட AI சாட்போட் நான். நான் ஆவணத்தை முழுமையாக மதிப்பாய்வு செய்துள்ளேன், மேலும் உங்களுக்குத் தேவையான சரியான தகவலைக் கண்டறிய அல்லது உள்ளடக்கத்தை தெளிவான மற்றும் எளிமையான சொற்களில் விளக்க உதவ முடியும். குறிப்பிட்ட அம்சங்கள், சரிசெய்தல் படிகள் அல்லது பொதுவான பயன்பாடு குறித்த வழிகாட்டுதலை நீங்கள் தேடுகிறீர்களானால், உங்கள் கேள்விகளைக் கேட்க தயங்கவும். உங்கள் கவலைகள் அல்லது தேவைகள் பற்றி நீங்கள் எவ்வளவு அதிகமாக விவரங்களை வழங்குகிறீர்களோ, அவ்வளவு துல்லியமாகவும் эффективноவும் நான் உங்களுக்கு உதவ முடியும்!