Installation

By default, the switch comes pre-installed with ONIE, which "defines an open 'install environment' for bare metal network switches." However, PXE installation is also possible. This document describes both methods.

Note: Installing using PXE uninstalls ONIE, which might be required by other operating systems.

Table of Contents

1. Getting the Installation Files
2. Default Credentials
3. Installing with ONIE
   1. Preparing the Installation
   2. Performing the Installation
   3. Uninstalling
4. Installing with PXE
   1. Preparing the Installation
   2. Performing the Installation
5. Customizing the Installation
6. Post Installation Updates
7. Installing ONIE
   1. Using PXE
   2. Using a USB Drive
   3. Using kexec
8. Querying ASIC Revision
   1. Using devlink
   2. Using mstflint

Getting the Installation Files

For Fedora, an installation tarball is available here. The following files are included in the provided tarball:

• The operating system ONIE installer, if using ONIE - Fedora-ONIE-installer.bin
• Two auto-install files (kickstart) for ONIE and PXE - install-onie.ks and install-pxe.ks
• Recent Fedora ISO

Note that the Fedora ISO image is there to ease the installation process; It is an unmodified copy of the ISO that can be downloaded directly from Fedora.

Default Credentials

User: root Password: switchdevftw

The root password can be easily changed by editing the rootpw command in the kickstart file. It is highly recommended to do so!

Installing with ONIE

The ONIE project describes a variety of installation options. This section describes installation using an HTTP server accessible through the management port on the switch.

Preparing the Installation

The extracted contents of the ISO file, the ONIE installer and the kickstart file (e.g. install-onie.ks), should all be accessible through an HTTP server.

To extract the contents from the ISO file, you must first mount it. Run:

$ mount -o loop,ro -t iso9660 /path/file.iso /path/iso/directory

Then copy the contents over to the HTTP server directory. Run:

$ cp -r /path/iso/directory /path/http/server

Finally, edit the auto-install file to point to the HTTP server. In the kickstart file, edit the url command to point to the extracted ISO directory.

Note: The ONIE installer makes a few assumptions about the location and name of the auto-install. In the provided package, it is assumed that the kickstart file is named install-onie.ks and that it is located in a directory called ks, placed next to the ONIE installer. For example:

├── ks
│   ├── install-onie.ks
├── Fedora-ONIE-installer.bin

Performing the Installation

Reboot the switch and select ONIE: Resuce in GRUB menu as the discovery phase is unnecessary:

 +----------------------------------------------------------------------------+
 | ONIE: Install OS                                                           |
 |*ONIE: Rescue                                                               |
 | ONIE: Uninstall OS                                                         |
 | ONIE: Update ONIE                                                          |
 | ONIE: Embed ONIE                                                           |
 |                                                                            |

In the ONIE prompt run the following:

ONIE:/ # onie-nos-install http://server/Fedora-ONIE-installer.bin

From this point on, the installation is completely automatic. Once it is complete, the switch reboots into the newly installed system.

Uninstalling

Reboot the switch and select ONIE in the GRUB menu:

      Fedora (4.8.13-100.fc23.x86_64) 23 (Twenty Three)
      Fedora (0-rescue-851011758d7ea341a944f10920a2faa5) 23 (Twenty Three)
      ONIE

Then select ONIE: Uninstall OS. This removes the partitions used by the OS, but keeps ONIE installed, so that a new OS can be installed using it.

Installing with PXE

The instructions below assume that a configured PXE server is already available on the network and that the installation files are available over HTTP. However, other methods are possible.

Preparing the Installation

As with the ONIE installation, the extracted contents of the ISO file and the kickstart file (e.g. install-pxe.ks) should all be accessible through an HTTP server.

To extract the contents from the ISO file, first mount it. Run:

$ mount -o loop,ro -t iso9660 /path/file.iso /path/iso/directory

And copy the contents over to the HTTP server directory. Run:

$ cp -r /path/iso/directory /path/http/server

In addition, the kernel executable and the initrd image should be copied over to the PXE server. Run:

$ cp /path/iso/directory/images/pxeboot/{initrd.img,vmlinuz} /tftpboot/directory/

Next, edit the auto-install file to point to the HTTP server:

• In the kickstart file, edit the url command to point to the extracted ISO directory.
• add the following lines to the file /tftpboot/pxelinux.cfg/default on the PXE server:

LABEL MY_LABEL
        kernel directory/vmlinuz
        append initrd=directory/initrd.img inst.ks=http://server/path/install-pxe.ks console=ttyS0,115200n8 modprobe.blacklist=mlxsw_spectrum

Performing the Installation

Make sure network boot is enabled in the BIOS (Ctrl+b) and reboot the switch. Once up, select the recently configured PXE label (e.g. MY_LABEL) and the automated installation should begin.

Once done, the switch reboots into the newly installed system.

Customizing the Installation

As explained above, the installation is automated using auto-install files. These files can be easily edited to provide different answers during the installation, and to select different packages to install by default. Please refer to the official RedHat kickstart documentation for more info.

Creating an ONIE-compatible installer from an ISO image is explained in ONIE's Github repository, in case one wants to create its own installer.

Post Installation Updates

Once Fedora is installed, it is advised to upgrade all packages to the latest available by using:

$ dnf upgrade

Notice this would also upgrade the kernel image (assuming a newer one is available) and a reboot would be required following the command's completion.

You can find more information here or in the DNF man page.

At this point it is also possible to use DNF and install packages of interest that are not part of the ISO. Some notable packages worthy of mentioning are: iproute-tc gcc teamd bridge-utils

This can be achieved by using:

$ dnf install <package>

Installing ONIE

To install ONIE on the switch you might need to access the BIOS (Ctrl+b) and change the boot order. If the BIOS is password-protected, please contact Mellanox global support.

For both methods described below you need the ONIE installation files which are available here. Version r5.0.1410 is used as an example.

Further information regarding ONIE installation can be found in ONIE's official documentation located here.

Using PXE

Copy the following files to the appropriate place on the PXE server:

mlnx_x86-r5.0.1410.vmlinuz
recovery-x86_64-mlnx_x86-r5.0.1410.initrd

Create the following PXE label on the server:

LABEL ONIE
	kernel mlnx_x86-r5.0.1410.vmlinuz
	append initrd=recovery-x86_64-mlnx_x86-r5.0.1410.initrd console=ttyS0,115200n8 ramdisk_size=1048576 root=/dev/ram boot_reason=embed install_url=file:///lib/onie/onie-updater boot_env=recovery

Make sure the switch is configured to boot from PXE, reboot it and choose the ONIE label in the PXE menu.

Using a USB Drive

Copy the ISO file (e.g. onie-recovery-x86_64-mlnx_x86-r5.0.1410.iso) to a USB drive. On Linux systems this can be done using the dd utility. On Windows, an application such as this one is needed.

Connect the USB drive to the switch, make sure the switch is configured to boot from the USB drive and reboot the switch. When the ONIE menu is presented choose the ONIE: Embed ONIE option to install ONIE on the switch.

Using kexec

If you are currently running Linux with kexec tools, booting into the recovery image directly from the current operating system might also be possible.

Download the ISO file to the switch, and issue the following commands:

mkdir -p /mnt/cdrom
mount -o loop onie-recovery-x86_64-mlnx_x86-r5.0.1410.iso /mnt/cdrom
kexec -l /mnt/cdrom/vmlinuz --initrd=/mnt/cdrom/initrd.xz --command-line "console=tty0 console=ttyS0,115200n8 boot_env=recovery boot_reason=embed install_url=file:///lib/onie/onie-updater"
kexec -e

This should reinstall ONIE.

Querying ASIC Revision

It is possible to query the ASIC revision using mstflint or devlink.

Using devlink

To query the ASIC revision, run:

$ devlink dev info pci/0000:03:00.0
pci/0000:03:00.0:
  driver mlxsw_spectrum
  versions:
      fixed:
        hw.revision A0
        fw.psid MT_2750110033
      running:
        fw.version 13.2000.1122

Where pci/0000:03:00.0 is the PCI address of the device. In the above example the ASIC revision is A0.

This option is supported from kernel 5.1 and corresponding iproute2 version.

Using mstflint

In Fedora it can be installed using dnf:

$ dnf install mstflint

To query the ASIC revision, run:

$ mstflint -d 01:00.0 hw query | grep HwRevId
  HwRevId               0x0

Where 01:00.0 is the PCI address of the device. In the above example the ASIC revision is A0. In case the ASIC revision is A1 the output will be:

  HwRevId               0x1