USB Passthrough with Red Hat OpenStack Platform 13
Some OpenStack users users would like to attach USB devices to OpenStack instances for security or legacy applications.
For example, a security application which run inside an OpenStack instance could require access to a Java card from an USB Gemalto eToken:
The nova API doesn’t have the feature to add USB passthrough to an instance.
A workaround is available with PCI passthrough. PCI passthrough can be used to access to dedicated PCI USB cards or motherboard USB controller of the compute node.
For this test, I bought a simple Inateck HT4006 2-Port PCI-E USB 3.0 Express Card ($16.99):
https://www.amazon.com/Inateck-Express-Controller-Internal-Connector/dp/B00JFR2H64
This card was added to an HP Micro server (compute node):
For this test, we will not use a Java card but an USB Flash drive SanDisk Extreme 32GB:
https://www.amazon.com/SanDisk-Extreme-Flash-Drive-SDCZ80-064G-G46/dp/B00DZPUOU8?th=1
This test has been done with Red Hat OpenStack Platform 13 (Queens) on 3 physical servers: one director, one controller and one compute node.
PCI passthrough is supported since Red Hat OpenStack Platform 6.
Documentation
Red Hat OpenStack platform 13 installation
Community documentation for PCI passthrough
Requirements
Before installing OpenStack and before plugging the USB PCI card, you can install a plain RHEL on one compute node and check the PCI and USB status.
[stack@z0 ~]$ sudo yum install -y pciutils usbutils
[stack@z0 ~]$ sudo lspci | grep -i usb
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
01:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (rev 02)
04:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)
Check USB devices:
[stack@z0 ~]$ sudo lsusb
Bus 002 Device 003: ID 0424:2660 Standard Microsystems Corp. Hub
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 005 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 004 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 003: ID 0518:0001 EzKEY Corp. USB to PS2 Adaptor v1.09
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Shutdown the node.
Plug the USB cards to check the changes.
Boot the compute node.\
A new USB controller is available:
[stack@z0 ~]$ sudo lspci -nn | grep Fresco
07:00.0 USB controller [0c03]: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100] (rev 10)
We can grab the vendor id: 1b73 and the product id: 1100 from this command.
Ensure SR-IOV and VT-d are enabled in BIOS. \
Red Hat OpenStack setup
Two main options are used to setup PCI passthrough: pci_passthrough_whitelist and alias.
pci_passthrough_whitelist is related to the hardwares installed on the compute hosts.
alias is related to the API, it provides a way for users to request hardwares (option to use only in the controllers)
We will cover only the specific PCI passthrough templates.
To setup the compute node, create a template compute-params.yaml:
[stack@lab-director ~]$ cat << EOF > /home/stack/templates/compute-params.yaml
parameter_defaults:
NovaPCIPassthrough:
- vendor_id: '1b73'
product_id: '1100'
EOF
Create a template for the controllers:
[stack@lab-director ~]$ cat << EOF > /home/stack/templates/controller-params.yaml
parameter_defaults:
ControllerExtraConfig:
nova::api::pci_alias:
- name: a1
vendor_id: '1b73'
product_id: '1100'
EOF
Enable IOMMU in Linux by adding intel_iommu=on to the kernel parameters. Create a template using GRUB for the first boot:
[stack@lab-director ~]$ cat << EOF > /home/stack/templates/first-boot-env.yaml
resource_registry:
OS::TripleO::NodeUserData: /home/stack/templates/firstboot/first-boot.yaml
[stack@lab-director ~]$ cat << EOF > /home/stack/templates/firstboot/first-boot.yaml
heat_template_version: 2014-10-16
resources:
userdata:
type: OS::Heat::MultipartMime
properties:
parts:
- config: {get_resource: compute_kernel_args}
# Logs can be checked on /var/log/cloud-init.log on the overcloud node
compute_kernel_args:
type: OS::Heat::SoftwareConfig
properties:
config: |
#!/bin/bash
set -x
# Set grub parameters
if hostname | grep compute >/dev/null
then
sed -i.orig 's/quiet"$/quiet intel_iommu=on iommu=pt"/' /etc/default/grub
grub2-mkconfig -o /etc/grub2.cfg
systemctl stop os-collect-config.service
/sbin/reboot
fi
outputs:
OS::stack_id:
value: {get_resource: userdata}
EOF
Include these three templates into the deploy command and launch the deployment:
[stack@lab-director ~]$ cat << EOF > ~/overcloud-deploy.sh
#!/bin/bash
time openstack overcloud deploy \
--templates /usr/share/openstack-tripleo-heat-templates \
-e /home/stack/templates/node-info.yaml \
-e /home/stack/templates/custom-domain.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/templates/network-environment.yaml \
-e /home/stack/templates/first-boot-env.yaml \
-e /home/stack/templates/overcloud_images.yaml \
-e /home/stack/templates/controller-params.yaml \
-e /home/stack/templates/compute-params.yaml \
--timeout 60
EOF
Deploy:
[stack@lab-director ~]$ ~/overcloud-deploy.sh
...
2018-06-18 23:27:55Z [overcloud.AllNodesDeploySteps.ControllerPostConfig]: CREATE_COMPLETE state changed
2018-06-18 23:27:55Z [overcloud.AllNodesDeploySteps.BlockStoragePostConfig]: CREATE_COMPLETE state changed
2018-06-18 23:27:55Z [overcloud.AllNodesDeploySteps.ComputePostConfig]: CREATE_COMPLETE state changed
2018-06-18 23:27:55Z [overcloud.AllNodesDeploySteps.ObjectStoragePostConfig]: CREATE_IN_PROGRESS state changed
2018-06-18 23:27:56Z [overcloud.AllNodesDeploySteps.CephStoragePostConfig]: CREATE_IN_PROGRESS state changed
2018-06-18 23:27:56Z [overcloud.AllNodesDeploySteps.ObjectStoragePostConfig]: CREATE_COMPLETE state changed
2018-06-18 23:27:56Z [overcloud.AllNodesDeploySteps.CephStoragePostConfig]: CREATE_COMPLETE state changed
2018-06-18 23:27:56Z [overcloud.AllNodesDeploySteps]: CREATE_COMPLETE Stack CREATE completed successfully
2018-06-18 23:27:56Z [overcloud.AllNodesDeploySteps]: CREATE_COMPLETE state changed
2018-06-18 23:27:56Z [overcloud]: CREATE_COMPLETE Stack CREATE completed successfully
Stack overcloud CREATE_COMPLETE
Host 172.16.5.55 not found in /home/stack/.ssh/known_hosts
Started Mistral Workflow tripleo.deployment.v1.get_horizon_url. Execution ID: 1f3f9fe8-9470-4f49-8c9c-bc773ed10e3f
Overcloud Endpoint: http://172.16.5.55:5000/
Overcloud Horizon Dashboard URL: http://172.16.5.55:80/dashboard
Overcloud rc file: /home/stack/overcloudrc
Overcloud Deployed
(undercloud) [stack@lab-director ~]$
Check configuration generated by director
Check nova configuration generated on the compute nodes:
[root@overcloud-compute-0 ~]$ sudo docker inspect nova_compute | grep MergedDir
"MergedDir": "/var/lib/docker/overlay2/74e4b55ed5d046506ad5469a54cf36378634d858c70368dd089b67b0347c4d52/merged",
[root@overcloud-compute-0 ~]$ sudo grep 1b73 /var/lib/docker/overlay2/74e4b55ed5d046506ad5469a54cf36378634d858c70368dd089b67b0347c4d52/merged/etc/nova/nova.conf
passthrough_whitelist={"product_id":"1100","vendor_id":"1b73"}
Check nova configuration on the controller node:
[heat-admin@overcloud-controller-0 ~]$ sudo docker inspect nova_api | grep MergedDir
"MergedDir": "/var/lib/docker/overlay2/480028668bd319f8b48e950234343d021f5dd4057717b667ddf61b3e79fc95d3/merged",
[heat-admin@overcloud-controller-0 ~]$ sudo grep 1b73 /var/lib/docker/overlay2/480028668bd319f8b48e950234343d021f5dd4057717b667ddf61b3e79fc95d3/merged/etc/nova/nova.conf
alias={"name":"a1","product_id":"1100","vendor_id":"1b73"}
Validate DMAR and IOMMU logs in the compute nodes:
[root@overcloud-compute-0 heat-admin]# dmesg | grep -e DMAR -e IOMMU
[ 0.000000] ACPI: DMAR 00000000f1de4a80 003B4 (v01 HP ProLiant 00000001 \xffffffd2? 0000162E)
[ 0.000000] DMAR: IOMMU enabled
[ 0.028740] DMAR: Host address width 39
[ 0.028742] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[ 0.028748] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap c9008020660262 ecap f010da
[ 0.028749] DMAR: RMRR base: 0x000000f1ffd000 end: 0x000000f1ffffff
[ 0.028751] DMAR: RMRR base: 0x000000f1ff6000 end: 0x000000f1ffcfff
[ 0.028752] DMAR: RMRR base: 0x000000f1f93000 end: 0x000000f1f94fff
[ 0.028753] DMAR: RMRR base: 0x000000f1f8f000 end: 0x000000f1f92fff
[ 0.028754] DMAR: RMRR base: 0x000000f1f7f000 end: 0x000000f1f8efff
[ 0.028755] DMAR: RMRR base: 0x000000f1f7e000 end: 0x000000f1f7efff
[ 0.028756] DMAR: RMRR base: 0x000000000f4000 end: 0x000000000f4fff
[ 0.028757] DMAR: RMRR base: 0x000000000e8000 end: 0x000000000e8fff
[ 0.028758] DMAR: RMRR base: 0x000000f1dee000 end: 0x000000f1deefff
[ 0.028761] DMAR-IR: IOAPIC id 8 under DRHD base 0xfed90000 IOMMU 0
[ 0.028762] DMAR-IR: HPET id 0 under DRHD base 0xfed90000
[ 0.028763] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[ 0.028763] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[ 0.028894] DMAR-IR: Enabled IRQ remapping in xapic mode
[ 0.690271] DMAR: No ATSR found
[ 0.694262] DMAR: dmar0: Using Queued invalidation
[ 0.694429] DMAR: Hardware identity mapping for device 0000:00:00.0
[ 0.694431] DMAR: Hardware identity mapping for device 0000:00:01.0
[ 0.694433] DMAR: Hardware identity mapping for device 0000:00:06.0
[ 0.694435] DMAR: Hardware identity mapping for device 0000:00:1a.0
[ 0.694436] DMAR: Hardware identity mapping for device 0000:00:1c.0
[ 0.694438] DMAR: Hardware identity mapping for device 0000:00:1c.4
[ 0.694439] DMAR: Hardware identity mapping for device 0000:00:1c.6
[ 0.694441] DMAR: Hardware identity mapping for device 0000:00:1c.7
[ 0.694443] DMAR: Hardware identity mapping for device 0000:00:1d.0
[ 0.694445] DMAR: Hardware identity mapping for device 0000:00:1f.0
[ 0.694450] DMAR: Hardware identity mapping for device 0000:07:00.0
[ 0.694456] DMAR: Hardware identity mapping for device 0000:04:00.0
[ 0.694461] DMAR: Hardware identity mapping for device 0000:01:00.1
[ 0.694463] DMAR: Hardware identity mapping for device 0000:01:00.4
[ 0.694464] DMAR: Setting RMRR:
[ 0.694617] DMAR: Setting identity map for device 0000:01:00.0 [0xf1dee000 - 0xf1deefff]
[ 0.694775] DMAR: Setting identity map for device 0000:01:00.2 [0xf1dee000 - 0xf1deefff]
[ 0.694784] DMAR: Ignoring identity map for HW passthrough device 0000:01:00.4 [0xf1dee000 - 0xf1deefff]
[ 0.694931] DMAR: Setting identity map for device 0000:00:1f.2 [0xe8000 - 0xe8fff]
[ 0.694939] DMAR: Ignoring identity map for HW passthrough device 0000:07:00.0 [0xe8000 - 0xe8fff]
[ 0.695088] DMAR: Setting identity map for device 0000:03:00.0 [0xe8000 - 0xe8fff]
[ 0.695242] DMAR: Setting identity map for device 0000:03:00.1 [0xe8000 - 0xe8fff]
[ 0.695250] DMAR: Ignoring identity map for HW passthrough device 0000:04:00.0 [0xe8000 - 0xe8fff]
[ 0.695251] DMAR: Setting identity map for device 0000:01:00.0 [0xe8000 - 0xe8fff]
[ 0.695259] DMAR: Setting identity map for device 0000:01:00.2 [0xe8000 - 0xe8fff]
[ 0.695267] DMAR: Setting identity map for device 0000:00:1f.2 [0xf4000 - 0xf4fff]
[ 0.695269] DMAR: Ignoring identity map for HW passthrough device 0000:07:00.0 [0xf4000 - 0xf4fff]
[ 0.695271] DMAR: Setting identity map for device 0000:03:00.0 [0xf4000 - 0xf4fff]
[ 0.695272] DMAR: Setting identity map for device 0000:03:00.1 [0xf4000 - 0xf4fff]
[ 0.695274] DMAR: Ignoring identity map for HW passthrough device 0000:04:00.0 [0xf4000 - 0xf4fff]
[ 0.695275] DMAR: Setting identity map for device 0000:01:00.0 [0xf4000 - 0xf4fff]
[ 0.695276] DMAR: Setting identity map for device 0000:01:00.2 [0xf4000 - 0xf4fff]
[ 0.695278] DMAR: Setting identity map for device 0000:00:1f.2 [0xf1f7e000 - 0xf1f7efff]
[ 0.695285] DMAR: Ignoring identity map for HW passthrough device 0000:07:00.0 [0xf1f7e000 - 0xf1f7efff]
[ 0.695286] DMAR: Setting identity map for device 0000:03:00.0 [0xf1f7e000 - 0xf1f7efff]
[ 0.695294] DMAR: Setting identity map for device 0000:03:00.1 [0xf1f7e000 - 0xf1f7efff]
[ 0.695301] DMAR: Ignoring identity map for HW passthrough device 0000:04:00.0 [0xf1f7e000 - 0xf1f7efff]
[ 0.695302] DMAR: Setting identity map for device 0000:01:00.0 [0xf1f7e000 - 0xf1f7efff]
[ 0.695307] DMAR: Setting identity map for device 0000:01:00.2 [0xf1f7e000 - 0xf1f7efff]
[ 0.695312] DMAR: Setting identity map for device 0000:00:1f.2 [0xf1f7f000 - 0xf1f8efff]
[ 0.695313] DMAR: Ignoring identity map for HW passthrough device 0000:07:00.0 [0xf1f7f000 - 0xf1f8efff]
[ 0.695315] DMAR: Setting identity map for device 0000:03:00.0 [0xf1f7f000 - 0xf1f8efff]
[ 0.695316] DMAR: Setting identity map for device 0000:03:00.1 [0xf1f7f000 - 0xf1f8efff]
[ 0.695318] DMAR: Ignoring identity map for HW passthrough device 0000:04:00.0 [0xf1f7f000 - 0xf1f8efff]
[ 0.695319] DMAR: Setting identity map for device 0000:01:00.0 [0xf1f7f000 - 0xf1f8efff]
[ 0.695321] DMAR: Setting identity map for device 0000:01:00.2 [0xf1f7f000 - 0xf1f8efff]
[ 0.695323] DMAR: Setting identity map for device 0000:00:1f.2 [0xf1f8f000 - 0xf1f92fff]
[ 0.695325] DMAR: Ignoring identity map for HW passthrough device 0000:07:00.0 [0xf1f8f000 - 0xf1f92fff]
[ 0.695326] DMAR: Setting identity map for device 0000:03:00.0 [0xf1f8f000 - 0xf1f92fff]
[ 0.695327] DMAR: Setting identity map for device 0000:03:00.1 [0xf1f8f000 - 0xf1f92fff]
[ 0.695329] DMAR: Ignoring identity map for HW passthrough device 0000:04:00.0 [0xf1f8f000 - 0xf1f92fff]
[ 0.695330] DMAR: Setting identity map for device 0000:01:00.0 [0xf1f8f000 - 0xf1f92fff]
[ 0.695332] DMAR: Setting identity map for device 0000:01:00.2 [0xf1f8f000 - 0xf1f92fff]
[ 0.695333] DMAR: Setting identity map for device 0000:00:1f.2 [0xf1f93000 - 0xf1f94fff]
[ 0.695335] DMAR: Ignoring identity map for HW passthrough device 0000:07:00.0 [0xf1f93000 - 0xf1f94fff]
[ 0.695336] DMAR: Setting identity map for device 0000:03:00.0 [0xf1f93000 - 0xf1f94fff]
[ 0.695337] DMAR: Setting identity map for device 0000:03:00.1 [0xf1f93000 - 0xf1f94fff]
[ 0.695339] DMAR: Ignoring identity map for HW passthrough device 0000:04:00.0 [0xf1f93000 - 0xf1f94fff]
[ 0.695340] DMAR: Setting identity map for device 0000:01:00.0 [0xf1f93000 - 0xf1f94fff]
[ 0.695342] DMAR: Setting identity map for device 0000:01:00.2 [0xf1f93000 - 0xf1f94fff]
[ 0.695344] DMAR: Setting identity map for device 0000:01:00.0 [0xf1ff6000 - 0xf1ffcfff]
[ 0.695345] DMAR: Setting identity map for device 0000:01:00.2 [0xf1ff6000 - 0xf1ffcfff]
[ 0.695347] DMAR: Ignoring identity map for HW passthrough device 0000:01:00.4 [0xf1ff6000 - 0xf1ffcfff]
[ 0.695348] DMAR: Ignoring identity map for HW passthrough device 0000:00:1a.0 [0xf1ffd000 - 0xf1ffffff]
[ 0.695349] DMAR: Ignoring identity map for HW passthrough device 0000:00:1d.0 [0xf1ffd000 - 0xf1ffffff]
[ 0.695352] DMAR: Prepare 0-16MiB unity mapping for LPC
[ 0.695353] DMAR: Ignoring identity map for HW passthrough device 0000:00:1f.0 [0x0 - 0xffffff]
[ 0.695354] DMAR: Intel(R) Virtualization Technology for Directed I/O
[ 0.742301] DMAR: 32bit 0000:00:1a.0 uses non-identity mapping
[ 0.742445] DMAR: Setting identity map for device 0000:00:1a.0 [0xf1ffd000 - 0xf1ffffff]
[ 0.751812] DMAR: 32bit 0000:00:1d.0 uses non-identity mapping
[ 0.751956] DMAR: Setting identity map for device 0000:00:1d.0 [0xf1ffd000 - 0xf1ffffff]
[ 0.761819] DMAR: 32bit 0000:01:00.4 uses non-identity mapping
[ 0.761961] DMAR: Setting identity map for device 0000:01:00.4 [0xf1dee000 - 0xf1deefff]
[ 0.761970] DMAR: Setting identity map for device 0000:01:00.4 [0xf1ff6000 - 0xf1ffcfff]
Testing
(overcloud) [stack@lab-director ~]$ source overcloudrc_myproject
Create a flavor without PCI Passtrough USB controller access:
(overcloud) [stack@lab-director ~]$ openstack flavor create --ram 2048 --disk 10 --vcpus 1 m1.small
+----------------------------+--------------------------------------+
| Field | Value |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| disk | 10 |
| id | 03ad81da-d93c-42d0-b099-c48b550a5507 |
| name | m1.small |
| os-flavor-access:is_public | True |
| properties | |
| ram | 20148 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 1 |
+----------------------------+--------------------------------------+
Create a flavour with PCI Passtrough USB controller access:
(overcloud) [stack@lab-director ~]$ openstack flavor create --ram 2048 --disk 10 --vcpus 1 m1.small-usb
+----------------------------+--------------------------------------+
| Field | Value |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| disk | 10 |
| id | acd58d32-d34d-4687-b6bb-9c088c3f7913 |
| name | m1.small-usb |
| os-flavor-access:is_public | True |
| properties | |
| ram | 20148 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 1 |
+----------------------------+--------------------------------------+
Update the USB passthrough flavor:
(overcloud) [stack@lab-director ~]$ openstack flavor show m1.small-usb
+----------------------------+--------------------------------------+
| Field | Value |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| access_project_ids | None |
| disk | 10 |
| id | ac7e9fdd-e8d8-47d8-9585-e7d466302d19 |
| name | m1.small-usb |
| os-flavor-access:is_public | True |
| properties | |
| ram | 2048 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 1 |
+----------------------------+--------------------------------------+
(overcloud) [stack@lab-director ~]$ openstack flavor set m1.small-usb --property "pci_passthrough:alias"="a1:1"
(overcloud) [stack@lab-director ~]$ openstack flavor show m1.small-usb
+----------------------------+--------------------------------------+
| Field | Value |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled | False |
| OS-FLV-EXT-DATA:ephemeral | 0 |
| access_project_ids | None |
| disk | 10 |
| id | ac7e9fdd-e8d8-47d8-9585-e7d466302d19 |
| name | m1.small-usb |
| os-flavor-access:is_public | True |
| properties | pci_passthrough:alias='a1:1' |
| ram | 2048 |
| rxtx_factor | 1.0 |
| swap | |
| vcpus | 1 |
+----------------------------+--------------------------------------+
Create an instance with PCI passthrough:
(overcloud) [stack@lab-director ~]$ openstack server create --flavor m1.small-usb --image rhel75 --security-group web --nic net-id=internal0 --key-name lambda instance0
+-------------------------------------+-----------------------------------------------------+
| Field | Value |
+-------------------------------------+-----------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | None |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| OS-EXT-SRV-ATTR:instance_name | |
| OS-EXT-STS:power_state | NOSTATE |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| adminPass | nkKJ2BP4d7Fw |
| config_drive | |
| created | 2018-06-29T11:18:40Z |
| flavor | m1.small-usb (ac7e9fdd-e8d8-47d8-9585-e7d466302d19) |
| hostId | |
| id | d865e835-f009-4e8b-b582-0b49e30c77c6 |
| image | rhel75 (8ff93daa-5088-45c9-a3e1-61ab55867b63) |
| key_name | lambda |
| name | instance0 |
| progress | 0 |
| project_id | 6163570b88e54038897a97e7be1f12d0 |
| properties | |
| security_groups | name='a2bfd621-6872-4d98-ac33-12c481b3378c' |
| status | BUILD |
| updated | 2018-06-29T11:18:40Z |
| user_id | 602cf7eb3cbd43b19b665e9c3a11af38 |
| volumes_attached | |
+-------------------------------------+-----------------------------------------------------+
List instances:
(overcloud) [stack@lab-director ~]$ openstack server list
+--------------------------------------+-----------+--------+------------------------------------+--------+--------------+
| ID | Name | Status | Networks | Image | Flavor |
+--------------------------------------+-----------+--------+------------------------------------+--------+--------------+
| d865e835-f009-4e8b-b582-0b49e30c77c6 | instance0 | ACTIVE | internal0=172.31.0.4, 172.16.5.155 | rhel75 | m1.small-usb |
+--------------------------------------+-----------+--------+------------------------------------+--------+--------------+
Attach a floating Ip to the instance:
(overcloud) [stack@lab-director ~]$ openstack server add floating ip instance0 172.16.5.155
Test to ping the instance:
[root@overcloud-controller-0 ~]# ping 172.16.5.155
PING 172.16.5.155 (172.16.5.155) 56(84) bytes of data.
64 bytes from 172.16.5.155: icmp_seq=1 ttl=63 time=1.41 ms
64 bytes from 172.16.5.155: icmp_seq=2 ttl=63 time=2.08 ms
64 bytes from 172.16.5.155: icmp_seq=3 ttl=63 time=2.19 ms
64 bytes from 172.16.5.155: icmp_seq=4 ttl=63 time=1.78 ms
SSH to the instance:
[root@overcloud-controller-0 ~]# ssh cloud-user@172.16.5.155
The authenticity of host '172.16.5.155 (172.16.5.155)' can't be established.
ECDSA key fingerprint is SHA256:F0nJbJrJOsDH8oQvV0k7b0+uKVSnLJ8+HRfLI7qkcgI.
ECDSA key fingerprint is MD5:7d:ea:00:c6:b0:7e:b3:7b:b1:0e:5d:ef:2c:3d:fd:b4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.16.5.155' (ECDSA) to the list of known hosts.
[cloud-user@instance0 ~]$ uptime
07:23:40 up 4 min, 1 user, load average: 0.05, 0.13, 0.06
Install pciutils and usbutils tools in the instance:
[cloud-user@instance0 ~]$ yum install -y pciutils libusbx usbutils
Check PCI status:
[root@instance0 ~]# lspci -nn
00:00.0 Host bridge [0600]: Intel Corporation 440FX - 82441FX PMC [Natoma] [8086:1237] (rev 02)
00:01.0 ISA bridge [0601]: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] [8086:7000]
00:01.1 IDE interface [0101]: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] [8086:7010]
00:01.2 USB controller [0c03]: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] [8086:7020] (rev 01)
00:01.3 Bridge [0680]: Intel Corporation 82371AB/EB/MB PIIX4 ACPI [8086:7113] (rev 03)
00:02.0 VGA compatible controller [0300]: Cirrus Logic GD 5446 [1013:00b8]
00:03.0 Ethernet controller [0200]: Red Hat, Inc. Virtio network device [1af4:1000]
00:04.0 SCSI storage controller [0100]: Red Hat, Inc. Virtio block device [1af4:1001]
00:05.0 USB controller [0c03]: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100] (rev 10)
00:06.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon [1af4:1002]
Check USB status:
[root@instance0 tmp]# lsusb
Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Check USB hubs:
[root@instance0 tmp]# lsusb -v | grep -E '\<(Bus|iProduct|bDeviceClass|bDeviceProtocol)' 2>/dev/null
Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
bDeviceClass 9 Hub
bDeviceProtocol 3
iProduct 2 xHCI Host Controller
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
bDeviceClass 9 Hub
bDeviceProtocol 1 Single TT
iProduct 2 xHCI Host Controller
Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
bDeviceClass 0 (Defined at Interface level)
bDeviceProtocol 0
iProduct 3 QEMU USB Tablet
(Bus Powered)
bDeviceClass 0 (Defined at Interface level)
bDeviceProtocol 0
(Bus Powered)
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
bDeviceClass 9 Hub
bDeviceProtocol 0 Full speed (or root) hub
iProduct 2 UHCI Host Controller
Plug the USB Sandisk key inside the compute node.
We can see a Sandisk key in the USB device list:
[root@instance0 ~]# lsusb
Bus 003 Device 003: ID 0781:5580 SanDisk Corp. SDCZ80 Flash Drive
Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
The device is seen with the fdisk command inside the instance:
[root@instance0 tmp]# fdisk -l /dev/sda
Disk /dev/sda: 31.4 GB, 31406948352 bytes, 61341696 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00000000
Device Boot Start End Blocks Id System
/dev/sda1 * 2 61341695 30670847 b W95 FAT32
[root@instance0 tmp]# fdisk -l | grep Disk
Disk /dev/vda: 100.7 GB, 10737418240 bytes, 20971520 sectors
Disk label type: dos
Disk identifier: 0x000ac719
Disk /dev/sda: 31.4 GB, 31406948352 bytes, 61341696 sectors
Disk label type: dos
Disk identifier: 0x00000000
Mount the USB key inside the instance:
[root@instance0 tmp]# mount -t vfat /dev/sda1 /mnt/sda1/
[root@instance0 tmp]# df -h /mnt/sda1
Filesystem Size Used Avail Use% Mounted on
/dev/sda1 30G 1.6M 30G 1% /mnt/sda1
Write a file:
cat << EOF > /mnt/sda1/test_file_osp_13_2018.txt
This is a text written for a test
EOF
Check the file:
[root@instance0 ~]# ls /mnt/sda1/
test_file_osp_13_2018.txt
Unmount the key:
[root@instance0 ~]# umount /mnt/sda1/
On the compute host, no USB device are listed:
[root@overcloud-compute-0 tmp]# lsusb
Bus 002 Device 003: ID 0424:2660 Standard Microsystems Corp. Hub
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 007 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 006 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
But still PCI device visible:
[root@overcloud-compute-0 tmp]# lspci -nn | grep Fresco
07:00.0 USB controller [0c03]: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100] (rev 10)
With admin rights, we can use OS-EXT-SRV-ATTR to find the libvirt name of the instance:
(overcloud) [stack@lab-director ~]$ openstack server show instance0 | grep instance_name
| OS-EXT-SRV-ATTR:instance_name | instance-00000005 -
On the compute node we can find the running instance:
[heat-admin@overcloud-compute-0 ~]$ sudo virsh list
Id Name State
----------------------------------------------------
1 instance-00000005 running
vfio prepared by OpenStack can be seen in the libvirt configuration:
[heat-admin@overcloud-compute-0 tmp]$ sudo virsh dumpxml instance-00000005 | xmllint --xpath '//hostdev' -
<hostdev mode="subsystem" type="pci" managed="yes">
<driver name="vfio"/>
<source>
<address domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
</source>
<alias name="hostdev0"/>
<address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/>
</hostdev>
To compare check the default instance configuration, create an instance without PCI Passtrough USB controller access:
(overcloud) [stack@lab-director images]$ openstack server create --flavor m1.small --image rhel75 --security-group web --nic net-id=internal0 --key-name lambda instance0
+-------------------------------------+-------------------------------------------------+
| Field | Value |
+-------------------------------------+-------------------------------------------------+
| OS-DCF:diskConfig | MANUAL |
| OS-EXT-AZ:availability_zone | |
| OS-EXT-SRV-ATTR:host | None |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None |
| OS-EXT-SRV-ATTR:instance_name | |
| OS-EXT-STS:power_state | NOSTATE |
| OS-EXT-STS:task_state | scheduling |
| OS-EXT-STS:vm_state | building |
| OS-SRV-USG:launched_at | None |
| OS-SRV-USG:terminated_at | None |
| accessIPv4 | |
| accessIPv6 | |
| addresses | |
| adminPass | FJ7KaJ6cLuc7 |
| config_drive | |
| created | 2018-06-28T19:53:14Z |
| flavor | m1.small (8733812d-eacd-4a8d-9bcb-c920e9f583da) |
| hostId | |
| id | 89a8b073-876f-4a77-9389-8537c7ce5c6a |
| image | rhel75 (63b8eac0-095e-4d6a-b137-93f6b72e5c81) |
| key_name | lambda |
| name | instance0 |
| progress | 0 |
| project_id | c9d4227db1f54805881e81843abe7555 |
| properties | |
| security_groups | name='6f203816-7543-4218-bdbc-f0be9be6436f' |
| status | BUILD |
| updated | 2018-06-28T19:53:14Z |
| user_id | 9a6ad6a8e3c54f24b5eef74c3b2f4eb4 |
| volumes_attached | |
+-------------------------------------+-------------------------------------------------+
Check default PCI devices without USB controller:
[root@instance0 tmp]# lspci -nn
00:00.0 Host bridge [0600]: Intel Corporation 440FX - 82441FX PMC [Natoma] [8086:1237] (rev 02)
00:01.0 ISA bridge [0601]: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] [8086:7000]
00:01.1 IDE interface [0101]: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] [8086:7010]
00:01.2 USB controller [0c03]: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] [8086:7020] (rev 01)
00:01.3 Bridge [0680]: Intel Corporation 82371AB/EB/MB PIIX4 ACPI [8086:7113] (rev 03)
00:02.0 VGA compatible controller [0300]: Cirrus Logic GD 5446 [1013:00b8]
00:03.0 Ethernet controller [0200]: Red Hat, Inc. Virtio network device [1af4:1000]
00:04.0 SCSI storage controller [0100]: Red Hat, Inc. Virtio block device [1af4:1001]
00:05.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon [1af4:1002]