User talk:Udaykumar.kaja/sandbox

SR-IOV
Insert non-formatted text here

Introduction
This article documents the configuration used to configure SR-IOV in OSP 8. Note these configs were pushed via Ansible, not via director. We are probably not going to worry about templatizing this deployment as SR-IOV support will be native in heat/director starting in OSP 10.

Compute Node Configuration
This section will outline the changes needed to configure SR-IOV on each Compute Node.

Bios Configuration on Compute Nodes
First you will need to ssh to the drac of each Compute Node.

Now, type the command below to enter racadm command line.


 * 1) racadm

Type the command below to enable SRIOV [Key=BIOS.Setup.1-1#IntegratedDevices] RAC1017: Successfully modified the object value and the change is in       pending state. To apply modified value, create a configuration job and reboot the system. To create the commit and reboot jobs, use "jobqueue" command. For more information about the "jobqueue" command, see RACADM help.
 * 1) racadm set BIOS.IntegratedDevices.SriovGlobalEnable Enabled

Type the command below to verify


 * 1) racadm>>get BIOS.IntegratedDevices.SriovGlobalEnable

racadm get BIOS.IntegratedDevices.SriovGlobalEnable [Key=BIOS.Setup.1-1#IntegratedDevices] SriovGlobalEnable=Enabled

If the server already has an OS, reboot it to make these settings stick. If it doesn’t, use these racadm commands to power cycle
 * 1) racadm serveraction powerdown
 * 2) racadm serveraction powerup​

Grub Configuration on Compute Nodes
Add "intel_iommu=on" to the GRUB_CMDLINE_LINUX line as shown below

[root@tpavcpnfvcomp0 ~]# cat /etc/default/grub GRUB_TIMEOUT=5 GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)" GRUB_DEFAULT=saved GRUB_DISABLE_SUBMENU=true GRUB_TERMINAL_OUTPUT="console" GRUB_CMDLINE_LINUX="console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet intel_iommu=on" GRUB_DISABLE_RECOVERY="true" audit=1

First, make a backup of /etc/default/grub
 * 1) cp -p /etc/default/grub /etc/default/grub/.$(date +%F_%R)

Edit the line below.

"GRUB_CMDLINE_LINUX=\"console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet\"

Change it to this.

"GRUB_CMDLINE_LINUX=\"console=tty0 console=ttyS0,115200n8 crashkernel=auto rhgb quiet intel_iommu=on\"

Now, rebuild GRUB config as shown below.
 * 1) grub2-mkconfig -o /boot/grub2/grub.cfg

Specify the number of VFs to Create in rc.local
Add the following line to /etc/rc.d/rc.local, adjusting for the device name and #VFs: echo 32 > /sys/class/net/ /device/sriov_numvfs For example:
 * 1) echo 32 > /sys/class/net/p1p1/device/sriov_numvfs
 * 2) echo 32 > /sys/class/net/p3p1/device/sriov_numvfs

Also,ensure the correct selinux context is restored.


 * 1) restorecon -R -v /etc/rc.d/rc.local

Ensure that /etc/rc.d/rc.local is executable.
 * 1) chmod +x /etc/rc.d/rc.local

Whitelist PCI devices nova-compute (Compute)
Tell nova-compute which pci devices are allowed to be passed through. Edit the file /etc/nova/nova.conf:

[default] pci_passthrough_whitelist = [{"devname":"p1p1","physical_network":"sriov_p1p1"},{"devname":"p3p1","physical_network":"sriov_p3p1"}]

This tells nova that all VFs belonging to the physical interface, "p1p1", are allowed to be passed through to VMs and belong to the neutron provider network "sriov_p1p1" and all VFs belonging to the physical interface, "p3p1", are allowed to be passed through for the network "sriov_p3p1".

Restart nova compute with service nova-compute restart to let the changes take effect.

Install and Enable Neutron Sriov-Agent (Compute)
Install the following rpm.


 * 1) yum -y install openstack-neutron-sriov-nic-agent

Now, on each compute node edit the file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini:

[securitygroup] firewall_driver = neutron.agent.firewall.NoopFirewallDriver

[sriov_nic] physical_device_mappings = sriov_p1p1:p1p1,sriov_p3p1:p3p1

Now enable and start the nic agent.


 * 1) systemctl enable neutron-sriov-nic-agent.service && systemctl start neutron-sriov-nic-agent.service

Controller Node Configuration
Perform the following steps on each Controller Node. Note we will modify Nova and Neutron config files.

Neutron-Server changes in /etc/neutron/plugins/ml2/ml2_conf.ini(Controller)
The following changes take place in the file /etc/neutron/plugins/ml2/ml2_conf.ini

Add sriovnicswitch as mechanism driver.

mechanism_drivers =openvswitch,bsn_ml2,sriovnicswitch

Set type_drivers to vlan as shown below.

type_drivers = vlan

Set tenant_network_types to vlan

tenant_network_types = vlan

Set flat_networks as shown below where "sriov_p1p1" and "sriov_p3p1" are the networks we are going to create

flat_networks =datacentre,sriov_p1p1,sriov_p3p1

Add VLAN ranges for the SRIOV networks to the network_vlan_ranges line as shown below.

network_vlan_ranges =datacentre:3500:3749,datacentre:112:112,sriov_p1p1:3750:3999,sriov_p3p1:3750:3999

Neutron-Server changes in /etc/neutron/plugins/ml2/ml2_conf_sriov.ini(Controller)
The change below needs to be made in /etc/neutron/plugins/ml2/ml2_conf_sriov.ini on each controller

Update the /etc/neutron/plugins/ml2/ml2_conf_sriov.ini on each controller. In our case,the vendor_id is 8086 and the product_id is 10ed.

supported_pci_vendor_devs = 8086:10ed

Modify Nuetron-Server Startup
Edit "usr/lib/systemd/system/neutron-server.service". Here we add "--config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini" to the ExecStart line. See example below.

[root@tpavcpnfvctrl0 system]# cat neutron-server.service [Unit] Description=OpenStack Neutron Server After=syslog.target network.target

[Service] Type=notify User=neutron ExecStart=/usr/bin/neutron-server --config-file /usr/share/neutron/neutron-dist.conf --config-dir /usr/share/neutron/server --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugin.ini --config-file /etc/neutron/plugins/ml2/ml2_conf_sriov.ini --config-dir /etc/neutron/conf.d/common --config-dir /etc/neutron/conf.d/neutron-server --log-file /var/log/neutron/server.log PrivateTmp=true NotifyAccess=all KillMode=process

[Install] WantedBy=multi-user.target

Restart neutron on the controllers via PCS


 * 1) pcs resource restart neutron-server-clone

Configure nova-scheduler (Controller)
On every controller node running nova-scheduler add PCIDeviceScheduler to the scheduler_default_filters parameter and add a new line for scheduler_available_filters parameter under the [default] section in /etc/nova/nova.conf:

[DEFAULT] scheduler_default_filters = RetryFilter, AvailabilityZoneFilter, RamFilter, ComputeFilter, ComputeCapabilitiesFilter, ImagePropertiesFilter, ServerGroupAntiAffinityFilter, ServerGroupAffinityFilter, PciPassthroughFilter scheduler_available_filters = nova.scheduler.filters.all_filters scheduler_available_filters = nova.scheduler.filters.pci_passthrough_filter.PciPassthroughFilter

Now restart nova-scheduler via Pacemaker.


 * 1) pcs resource restart openstack-nova-scheduler-clone

Creating SR-IOV Networks
[root@controller1 ~(keystone_admin)]# neutron net-create sriov_net1 --provider:physical_network=sriov1 --provider:network_type=flat --shared

A dummy subnet needs to be configured unless fixed IPs are assigned when a port is created. An RFE was created to address this issue.

[root@controller1 ~(keystone_admin)]# neutron subnet-create --disable-dhcp --allocation-pool start=172.16.10.2,end=172.16.10.254 sriov_net1 172.16.10.0/24

Verify VFs have been reported to nova
[root@controller1 ~]# mysql nova -e 'select compute_node_id,count(id) from pci_devices group by compute_node_id'; +-+---+ +-+---+ +-+---+
 * compute_node_id | count(id) |
 * 2 |   	68 |
 * 5 |   	68 |
 * 10 |   	68 |
 * 13 |   	68 |
 * 14 |   	68 |
 * 17 |   	68 |
 * 20 |   	68 |
 * 23 |   	68 |
 * 26 |   	68 |
 * 29 |   	68 |
 * 32 |   	68 |
 * 35 |   	68 |
 * 38 |   	68 |
 * 41 |   	68 |
 * 44 |   	68 |
 * 47 |   	68 |

[root@controller1 ~(keystone_admin)]# neutron port-create sriov_net1 --binding:vnic-type direct

Created a new port: +---+-+ +---+-+ +---+-+ [root@controller1 ~(keystone_admin)]# nova boot --flavor 2 --image rhel-7 --nic port-id=857d2a71-d061-4bcc-9224-0cf188206efc test-sriov +--+---+ +--+---+ +--+---+
 * Field            	| Value                                                                           	|
 * admin_state_up   	| True                                                                            	|
 * allowed_address_pairs |                                                                                	|
 * binding:host_id  	|                                                                                 	|
 * binding:profile  	| {}                                                                              	|
 * binding:vif_details  | {}                                                                              	|
 * binding:vif_type 	| unbound                                                                         	|
 * binding:vnic_type 	| direct                                                                         	|
 * device_id        	|                                                                                 	|
 * device_owner     	|                                                                                 	|
 * fixed_ips        	| {"subnet_id": "d1c32d39-51a9-41c8-a086-d754339643d2", "ip_address": "192.168.0.31"} |
 * id               	| 857d2a71-d061-4bcc-9224-0cf188206efc                                            	|
 * mac_address      	| fa:16:3e:b2:bb:fb                                                               	|
 * name             	|                                                                                 	|
 * network_id       	| 91808ab0-becc-4ece-99ad-828a0411db4e                                            	|
 * security_groups  	| 57ca2aee-084b-4a45-8df1-0f47681d1dcd                                            	|
 * status           	| DOWN                                                                            	|
 * tenant_id        	| a6d5dd72b6334ae387cbb033484795bd                                                	|
 * Property                        	| Value                                     	|
 * OS-DCF:diskConfig               	| MANUAL                                    	|
 * OS-EXT-AZ:availability_zone     	|                                           	|
 * OS-EXT-SRV-ATTR:host            	| -                                         	|
 * OS-EXT-SRV-ATTR:hypervisor_hostname  | -                                         	|
 * OS-EXT-SRV-ATTR:instance_name   	| instance-00000064                         	|
 * OS-EXT-STS:power_state          	| 0                                         	|
 * OS-EXT-STS:task_state           	| scheduling                                	|
 * OS-EXT-STS:vm_state             	| building                                  	|
 * OS-SRV-USG:launched_at          	| -                                         	|
 * OS-SRV-USG:terminated_at        	| -                                         	|
 * accessIPv4                      	|                                           	|
 * accessIPv6                      	|                                           	|
 * adminPass                       	| cAaikGjCTB6n                              	|
 * config_drive                    	|                                           	|
 * created                         	| 2015-07-04T13:49:49Z                      	|
 * flavor                          	| m1.small (2)                              	|
 * hostId                          	|                                           	|
 * id                              	| b5e9e187-1dc1-4dce-b86a-8bca593b0645      	|
 * image                           	| rhel-7 (65df9890-1388-4bc7-b4ec-15ffb53a6f6e) |
 * key_name                        	| -                                         	|
 * metadata                        	| {}                                        	|
 * name                            	| test-sriov                                	|
 * os-extended-volumes:volumes_attached | []                                        	|
 * progress                        	| 0                                         	|
 * security_groups                 	| default                                   	|
 * status                          	| BUILD                                     	|
 * tenant_id                       	| a6d5dd72b6334ae387cbb033484795bd          	|
 * updated                         	| 2015-07-04T13:49:49Z                      	|
 * user_id                         	| 09b6d193497a4ab88ee0dadc3804a235          	|

Reference
http://docs.openstack.org/liberty/networking-guide/adv-config-sriov.html