Installation

The operating system shipped with the switch is either a Fedora Remix with a Stable kernel, which tracks the stable tree or an out-of-the-box Ubuntu. Refer to the Installing a New Kernel page for more information regarding kernel versioning. It is possible to use other Linux distributions, including unmodified Fedora, which is available here.

By default, the switch comes pre-installed with ONIE, which "defines an open 'install environment' for bare metal network switches." However, PXE installation is also possible. This document describes both methods.

Note: Installing with PXE uninstalls ONIE, which might be required by other operating systems.

Table of Contents

1. Getting the Installation Files
2. Default Credentials
3. Installing with ONIE
   1. Preparing the Installation
   2. Performing the Installation
   3. Uninstalling
4. Installing with PXE
   1. Preparing the Installation
   2. Performing the Installation
5. Post Installation Steps (Ubuntu only)
6. Customizing the Installation
7. Installing ONIE
   1. Using PXE
   2. Using a USB drive
   3. Using kexec
8. Troubleshooting
   1. Automatic Module Loading

Getting the Installation Files

Both the Fedora Remix and Ubuntu installation tarballs are available here. The following files are included in the provided Fedora Remix / Ubuntu tarballs:

• The operating system ISO file
• The operating system ONIE installer, if using ONIE
• An auto-install file (kickstart or preseed for Fedora Remix and Ubuntu, respectively).
• A matching firmware file

Default Credentials

User: root Password: switchdevftw

The root password can be easily changed by editing the rootpw command in the kickstart file or by editing the passwd/root-password command in the preseed file. It is highly recommended to do so!

Installing with ONIE

The ONIE project describes a variety of installation options. This section describes installation using an HTTP server accessible through the management port on the switch.

Preparing the Installation

The extracted contents of the ISO file, the ONIE installer and the kickstart file (e.g. install-onie.ks), should all be accessible through an HTTP server.

To extract the contents from the ISO file, you must first mount it. Run:

$ mount -o loop,ro -t iso9660 /path/file.iso /path/iso/directory

Then copy the contents over to the HTTP server directory:

$ cp -r /path/iso/directory /path/http/server

Finally, edit the auto-install file to point to the HTTP server:

• In the kickstart file, edit the url command to point to the extracted ISO directory.
• In the preseed file, edit the mirror/http/hostname and mirror/http/directory values to point to the HTTP server and the directory within the server, respectively.

Note: The ONIE installer assumes some things about the location and name of the auto-install. In the provided Fedora Remix ONIE package, it is assumed that the kickstart file is named install-onie.ks and that it is located in a directory called ks, placed next to the ONIE installer. For example:

├── ks
│   ├── install-onie.ks
├── mlxsw-4.6.0-0.rc7.git1.1-ONIE-installer.bin

And in the provided Ubuntu ONIE package, it is assumed that the file is named preseed.cfg and is located in directory called preseed placed next to the ONIE installer. For example:

├── preseed
│   ├── preseed.cfg
├── mlxsw-UBUNTU16-ONIE-installer.bin

Performing the Installation

Reboot the switch and select ONIE: Resuce in GRUB menu as the discovery phase is unnecessary:

 +----------------------------------------------------------------------------+
 | ONIE: Install OS                                                           |
 |*ONIE: Rescue                                                               |
 | ONIE: Uninstall OS                                                         |
 | ONIE: Update ONIE                                                          |
 | ONIE: Embed ONIE                                                           |
 |                                                                            |

In ONIE prompt type:

ONIE:/ # onie-nos-install http://server/provided-ONIE-installer.bin

From this point on, the installation is completely automatic. Once it is complete, the switch reboots into the newly installed system.

Uninstalling

Reboot the switch and select ONIE in the GRUB menu:

      Generic (4.6.0-0.rc7.git1.1.fc25.x86_64) 23 (Generic)
      Generic (0-rescue-1c9ab09884c946649fa3e1688040abde) 23 (Generic)
      ONIE

Then select ONIE: Uninstall OS. This removes the partitions used by the OS, but keeps ONIE installed, so that a new OS can be installed using it.

Installing with PXE

The instructions below assume that a configured PXE server is already available on the network and that the installation files are available over HTTP. However, other methods are possible.

Preparing the Installation

As with the ONIE installation, the extracted contents of the ISO file and the kickstart file (e.g. install-pxe.ks) should all be accessible via an HTTP server.

To extract the contents from the ISO file, first mount it. Run:

$ mount -o loop,ro -t iso9660 /path/file.iso /path/iso/directory

And copy the contents over to the HTTP server directory. Run:

$ cp -r /path/iso/directory /path/http/server

In addition, the kernel executable and the initrd image should be copied over to the PXE server. Run:

$ cp /path/iso/directory/images/pxeboot/{initrd.img,vmlinuz} /tftpboot/directory/

Next, edit the auto-install file to point to the HTTP server:

• In the kickstart file, edit the url command to point to the extracted ISO directory.
• In the preseed file, edit the mirror/http/hostname and mirror/http/directory values to point to the HTTP server and the directory within the server, respectively.

Finally, add the following lines to the /tftpboot/pxelinux.cfg/default file on the PXE server:

LABEL MY_LABEL
        kernel directory/vmlinuz
        append initrd=directory/initrd.img inst.ks=http://server/path/install-pxe.ks console=ttyS0,115200n8 modprobe.blacklist=mlxsw_pci

In case using Fedora Remix, or for Ubuntu:

LABEL MY_LABEL
        kernel directory/vmlinuz
        append initrd=directory/initrd.gz console=ttyS0,115200n8 auto=true priority=critical preseed/url=http://server/path/preseed.cfg netcfg/choose_interface=auto modprobe.blacklist=mlxsw_pci

Performing the Installation

Make sure network boot is enabled in the BIOS (Ctrl+b) and reboot the switch. Once up, select the recently configured PXE label (e.g. MY_LABEL) and the automated installation will then begin.

Once done, the switch will reboot into the newly installed system.

Post Installation Steps (Ubuntu only)

After the Ubuntu installation is complete, it's recommended to add official Ubuntu repositories from which packages can be obtained. See this page for more details.

The Ubuntu provided kernel and iproute2 packages might be older than the latest stable upstream counterparts. Therefore, if new features are required, it's recommended to update these packages. Please refer to Installing a New Kernel page for more information.

Customizing the Installation

As explained above, the installation is automated using auto-install files. These files can be easily edited to provide different answers during the installation, and to select different packages to install by default. Please refer to the official RedHat kickstart documentation and Ubuntu preseed documentation for more info.

For Fedora, it is possible to generate different installation ISOs with a different package selection by using mock and pungi as explained in Fedora's wiki.

Creating an ONIE-compatible installer from an ISO image is explained in ONIE's Github repository.

Installing ONIE

To install ONIE on the switch you might need to access the BIOS (Ctrl+b) and change the boot order. In case the BIOS is password-protected, please contact Mellanox global support.

For both methods described below you will need the ONIE installation files which are available here. Version r5.0.1410 is used as an example.

Further information regarding ONIE installation can be found in ONIE's official documentation located here.

Using PXE

Copy the following files to the appropriate place on the PXE server:

mlnx_x86-r5.0.1410.vmlinuz
recovery-x86_64-mlnx_x86-r5.0.1410.initrd

Create the following PXE label on the server:

LABEL ONIE_r5.0.1410
	KERNEL /mlnx_x86-r5.0.1410.vmlinuz
	APPEND /recovery-x86_64-mlnx_x86-r5.0.1410.initrd panic=10 noexec=off console=ttyS0,115200n8 ramdisk_size=1048576 root=/dev/ram boot_reason=embed install_url=file:///lib/onie/onie-updater boot_env=recovery acpi_enforce_resources=no nmi_watchdog=0

Make sure the switch is configured to boot from PXE, reboot it and choose the ONIE_r5.0.1410 label in the PXE menu.

Using a USB drive

Copy the ISO file (e.g., onie-recovery-x86_64-mlnx_x86-r5.0.1410.iso) to a USB drive. On Linux systems this can be done using the dd utility. On Windows, an application such as this one is needed.

Connect the USB drive to the switch, make sure the switch is configured to boot from the USB drive and reboot the switch. When the ONIE menu is presented choose the ONIE: Embed ONIE option to install ONIE on the switch.

Using kexec

In case you are currently running Linux with kexec tools, booting into the recovery image directly from the current operating system might also be possible.

Download the ISO file to the switch, and issue the following commands:

mkdir -p /mnt/cdrom
mount -o loop onie-recovery-x86_64-mlnx_x86-r5.0.1410.iso /mnt/cdrom
kexec -l /mnt/cdrom/vmlinuz --initrd=/mnt/cdrom/initrd.xz --command-line "console=tty0 console=ttyS0,115200n8 boot_env=recovery boot_reason=embed install_url=file:///lib/onie/onie-updater"
kexec -e

This should now reinstall ONIE.

Troubleshooting

Automatic Module Loading

If the driver is not loaded following a reboot, then it's possible the driver isn't included in the initramfs. To verify that, run:

$ lsinitrd /boot/initramfs-$(uname -r).img | grep mlxsw
drwxr-xr-x   2 root     root            0 Feb  3 22:28 usr/lib/modules/4.7.0-0.rc4.git2.2.fc25.x86_64/kernel/drivers/net/ethernet/mellanox/mlxsw
-rw-r--r--   1 root     root        20132 Feb  3 22:28 usr/lib/modules/4.7.0-0.rc4.git2.2.fc25.x86_64/kernel/drivers/net/ethernet/mellanox/mlxsw/mlxsw_core.ko.xz
-rw-r--r--   1 root     root        55976 Feb  3 22:28 usr/lib/modules/4.7.0-0.rc4.git2.2.fc25.x86_64/kernel/drivers/net/ethernet/mellanox/mlxsw/mlxsw_spectrum.ko.xz
-rw-r--r--   1 root     root        18560 Feb  3 22:28 usr/lib/modules/4.7.0-0.rc4.git2.2.fc25.x86_64/kernel/drivers/net/ethernet/mellanox/mlxsw/mlxsw_switchx2.ko.xz

If the output isn't similar to the above, then add the missing drivers to the initramfs. Assuming mlxsw_spectrum is missing:

$ dracut -f --add-drivers mlxsw_spectrum /boot/initramfs-$(uname -r).img