To start an instance with SR-IOV networking capabilities, you must attach a “Direct” SR-IOV port to the instance. This can be accomplished in two ways:

Method 1 (Attach After Launch):

Start an instance in the normal manner (e.g. on your project’s local network)
Once the instance is started, attach an interface in the instance options menu
1. https://vmaccel.atlassian.net/wiki/spaces/docs/pages/44630041

Note that with this method, the user must ensure that the instance is scheduled on a node that supports SR-IOV. This is usually accomplished by selecting a certain flavor which is mapped to SR-IOV nodes exclusively. This may vary on a project-by-project basis

Method 2 (Attach During Launch):

Start an instance, but select a network port during launch instead of selecting a network:

This method is useful because it eliminates the need to ensure the instance is scheduled on an SR-IOV node, as the scheduler will automatically handle this due to the SR-IOV port being requested at launch. This gives the user the freedom to select whichever flavor they desire (even non-SR-IOV flavors), which is not possible with method 1. However, note that some SR-IOV nodes have restrictions on which flavors are allowed to run on them, so certain combinations of SR-IOV port / flavor requests may fail to schedule.

Notes on Routes and Public IPs:

Once the instance is started with an SRIOV port, if the instance is attached to multiple networks/interfaces, additional OS configuration steps may be necessary in order to utilize the network.

By default, the network which the instance is launched on will obtain a default route via DHCP. If the instance is only attached to one network/interface, then everything will work automatically. However, any additional ports/interfaces which are attached to the instance after launch will not get routes via DHCP - they will only get their IP information. This ensures that adding new interfaces to an existing instance will not disrupt existing networks.

In some cases it is desired to set routes for the newly attached interfaces. In these cases, the user must manually add the routes inside the VM from the command line and ensure the priority (metric) is set appropriately to achieve their desired result.

These default settings can sometimes cause confusion. Consider the following example:

A user wishes to start an instance which meets the following requirements:

Has an SRIOV port
Has external network access via a Public IP

This requires that the instance has 2 interfaces. The user might construct this instance using either Method 1 or Method 2.

For Method 1, the user might launch the instance on a project_local network. Once the instance is launched, they would attach the SRIOV port.

For Method 2, the user would launch the instance on the SRIOV port, and then they would attach an interface after launch on either a project_local network (with a floating IP) or an external (Public IP) network.

If the user launches with Method 1, then the project_local network will be set as the default route, and the SRIOV port will only handle traffic destined for that specific SRIOV subnet. In most cases, this is what the user wants, and no additional configuration is required

If the user launches with Method 2, then the SRIOV network is set as the default route. This results in most traffic being sent out the SRIOV interface instead of the external interface. In this case, the Public IP would not work, ssh would be unavailable, and pings would not respond. In order to use the Public IP as intended, the user needs to set the external interface as the default route instead of the SRIOV port

There are many permutations of possible network/interface configurations which a user can construct by attaching different interfaces to a VM at different times. Because of this, it is impossible to have a single default solution which accommodates them all automatically. We encourage users to consider their desired network behavior and ensure their routes and other appropriate settings are set correctly inside the VM

Note: Security Groups cannot be applied to SR-IOV ports. Firewall rules for these ports must be managed at the OS level