Erwan Gallen
Erwan Gallen

Aug 14, 2018 18 min read

USB Passthrough with Red Hat OpenStack Platform 13

thumbnail for this post

Some OpenStack users users would like to attach USB devices to OpenStack instances for security or legacy applications.

For example, a security application which run inside an OpenStack instance could require access to a Java card from an USB Gemalto eToken:

Gemalto eToken

The nova API doesn’t have the feature to add USB passthrough to an instance.

A workaround is available with PCI passthrough. PCI passthrough can be used to access to dedicated PCI USB cards or motherboard USB controller of the compute node.

For this test, I bought a simple Inateck HT4006 2-Port PCI-E USB 3.0 Express Card ($16.99):
https://www.amazon.com/Inateck-Express-Controller-Internal-Connector/dp/B00JFR2H64 Inateck KT4006

This card was added to an HP Micro server (compute node):
HPE ProLiant MicroServer Gen10 Inateck KT4006 inside the MicroServer

For this test, we will not use a Java card but an USB Flash drive SanDisk Extreme 32GB:
https://www.amazon.com/SanDisk-Extreme-Flash-Drive-SDCZ80-064G-G46/dp/B00DZPUOU8?th=1

This test has been done with Red Hat OpenStack Platform 13 (Queens) on 3 physical servers: one director, one controller and one compute node.
PCI passthrough is supported since Red Hat OpenStack Platform 6.

Documentation

Red Hat OpenStack platform 13 installation

Community documentation for PCI passthrough

Requirements

Before installing OpenStack and before plugging the USB PCI card, you can install a plain RHEL on one compute node and check the PCI and USB status.

[stack@z0 ~]$ sudo yum install -y pciutils usbutils
[stack@z0 ~]$ sudo lspci | grep -i usb
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
01:00.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller (rev 02)
04:00.0 USB controller: Renesas Technology Corp. uPD720201 USB 3.0 Host Controller (rev 03)

Check USB devices:

[stack@z0 ~]$ sudo lsusb
Bus 002 Device 003: ID 0424:2660 Standard Microsystems Corp. Hub
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 005 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 004 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 003: ID 0518:0001 EzKEY Corp. USB to PS2 Adaptor v1.09
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

Shutdown the node.
Plug the USB cards to check the changes.
Boot the compute node.

A new USB controller is available:

[stack@z0 ~]$ sudo lspci -nn | grep Fresco
07:00.0 USB controller [0c03]: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100] (rev 10)

We can grab the vendor id: 1b73 and the product id: 1100 from this command.

Ensure SR-IOV and VT-d are enabled in BIOS.

Red Hat OpenStack setup

Two main options are used to setup PCI passthrough: pci_passthrough_whitelist and alias.

pci_passthrough_whitelist is related to the hardwares installed on the compute hosts.
alias is related to the API, it provides a way for users to request hardwares (option to use only in the controllers)

We will cover only the specific PCI passthrough templates.

To setup the compute node, create a template compute-params.yaml:

[stack@lab-director ~]$ cat << EOF > /home/stack/templates/compute-params.yaml 
parameter_defaults:
  NovaPCIPassthrough:
    - vendor_id: '1b73'
      product_id: '1100'
EOF

Create a template for the controllers:

[stack@lab-director ~]$ cat << EOF > /home/stack/templates/controller-params.yaml 
parameter_defaults:
  ControllerExtraConfig:
    nova::api::pci_alias:
      -  name: a1
         vendor_id: '1b73'
         product_id: '1100'
EOF

Enable IOMMU in Linux by adding intel_iommu=on to the kernel parameters. Create a template using GRUB for the first boot:

[stack@lab-director ~]$ cat << EOF > /home/stack/templates/first-boot-env.yaml 
resource_registry:
  OS::TripleO::NodeUserData: /home/stack/templates/firstboot/first-boot.yaml
[stack@lab-director ~]$ cat << EOF > /home/stack/templates/firstboot/first-boot.yaml 
heat_template_version: 2014-10-16

resources:
  userdata:
    type: OS::Heat::MultipartMime
    properties:
      parts:
      - config: {get_resource: compute_kernel_args}


  # Logs can be checked on /var/log/cloud-init.log on the overcloud node
  compute_kernel_args:
    type: OS::Heat::SoftwareConfig
    properties:
      config: |
        #!/bin/bash
        set -x

        # Set grub parameters
        if hostname | grep compute >/dev/null
        then
                sed -i.orig 's/quiet"$/quiet intel_iommu=on iommu=pt"/' /etc/default/grub
                grub2-mkconfig -o /etc/grub2.cfg
                systemctl stop os-collect-config.service
                /sbin/reboot
        fi

outputs:
  OS::stack_id:
    value: {get_resource: userdata}
EOF

Include these three templates into the deploy command and launch the deployment:

[stack@lab-director ~]$ cat << EOF >  ~/overcloud-deploy.sh 
#!/bin/bash
time openstack overcloud deploy \
--templates /usr/share/openstack-tripleo-heat-templates \
-e /home/stack/templates/node-info.yaml \
-e /home/stack/templates/custom-domain.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/templates/network-environment.yaml \
-e /home/stack/templates/first-boot-env.yaml \
-e /home/stack/templates/overcloud_images.yaml \
-e /home/stack/templates/controller-params.yaml \
-e /home/stack/templates/compute-params.yaml \
--timeout 60
EOF

Deploy:

[stack@lab-director ~]$ ~/overcloud-deploy.sh 
...
2018-06-18 23:27:55Z [overcloud.AllNodesDeploySteps.ControllerPostConfig]: CREATE_COMPLETE  state changed
2018-06-18 23:27:55Z [overcloud.AllNodesDeploySteps.BlockStoragePostConfig]: CREATE_COMPLETE  state changed
2018-06-18 23:27:55Z [overcloud.AllNodesDeploySteps.ComputePostConfig]: CREATE_COMPLETE  state changed
2018-06-18 23:27:55Z [overcloud.AllNodesDeploySteps.ObjectStoragePostConfig]: CREATE_IN_PROGRESS  state changed
2018-06-18 23:27:56Z [overcloud.AllNodesDeploySteps.CephStoragePostConfig]: CREATE_IN_PROGRESS  state changed
2018-06-18 23:27:56Z [overcloud.AllNodesDeploySteps.ObjectStoragePostConfig]: CREATE_COMPLETE  state changed
2018-06-18 23:27:56Z [overcloud.AllNodesDeploySteps.CephStoragePostConfig]: CREATE_COMPLETE  state changed
2018-06-18 23:27:56Z [overcloud.AllNodesDeploySteps]: CREATE_COMPLETE  Stack CREATE completed successfully
2018-06-18 23:27:56Z [overcloud.AllNodesDeploySteps]: CREATE_COMPLETE  state changed
2018-06-18 23:27:56Z [overcloud]: CREATE_COMPLETE  Stack CREATE completed successfully

 Stack overcloud CREATE_COMPLETE

Host 172.16.5.55 not found in /home/stack/.ssh/known_hosts
Started Mistral Workflow tripleo.deployment.v1.get_horizon_url. Execution ID: 1f3f9fe8-9470-4f49-8c9c-bc773ed10e3f
Overcloud Endpoint: http://172.16.5.55:5000/
Overcloud Horizon Dashboard URL: http://172.16.5.55:80/dashboard
Overcloud rc file: /home/stack/overcloudrc
Overcloud Deployed
(undercloud) [stack@lab-director ~]$

Check configuration generated by director

Check nova configuration generated on the compute nodes:

[root@overcloud-compute-0 ~]$ sudo docker inspect nova_compute | grep MergedDir
                "MergedDir": "/var/lib/docker/overlay2/74e4b55ed5d046506ad5469a54cf36378634d858c70368dd089b67b0347c4d52/merged",

[root@overcloud-compute-0 ~]$ sudo grep 1b73 /var/lib/docker/overlay2/74e4b55ed5d046506ad5469a54cf36378634d858c70368dd089b67b0347c4d52/merged/etc/nova/nova.conf
passthrough_whitelist={"product_id":"1100","vendor_id":"1b73"}

Check nova configuration on the controller node:

[heat-admin@overcloud-controller-0 ~]$ sudo docker inspect nova_api | grep MergedDir
                "MergedDir": "/var/lib/docker/overlay2/480028668bd319f8b48e950234343d021f5dd4057717b667ddf61b3e79fc95d3/merged",

[heat-admin@overcloud-controller-0 ~]$ sudo grep 1b73 /var/lib/docker/overlay2/480028668bd319f8b48e950234343d021f5dd4057717b667ddf61b3e79fc95d3/merged/etc/nova/nova.conf
alias={"name":"a1","product_id":"1100","vendor_id":"1b73"}

Validate DMAR and IOMMU logs in the compute nodes:

[root@overcloud-compute-0 heat-admin]# dmesg | grep -e DMAR -e IOMMU
[    0.000000] ACPI: DMAR 00000000f1de4a80 003B4 (v01 HP     ProLiant 00000001   \xffffffd2? 0000162E)
[    0.000000] DMAR: IOMMU enabled
[    0.028740] DMAR: Host address width 39
[    0.028742] DMAR: DRHD base: 0x000000fed90000 flags: 0x1
[    0.028748] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap c9008020660262 ecap f010da
[    0.028749] DMAR: RMRR base: 0x000000f1ffd000 end: 0x000000f1ffffff
[    0.028751] DMAR: RMRR base: 0x000000f1ff6000 end: 0x000000f1ffcfff
[    0.028752] DMAR: RMRR base: 0x000000f1f93000 end: 0x000000f1f94fff
[    0.028753] DMAR: RMRR base: 0x000000f1f8f000 end: 0x000000f1f92fff
[    0.028754] DMAR: RMRR base: 0x000000f1f7f000 end: 0x000000f1f8efff
[    0.028755] DMAR: RMRR base: 0x000000f1f7e000 end: 0x000000f1f7efff
[    0.028756] DMAR: RMRR base: 0x000000000f4000 end: 0x000000000f4fff
[    0.028757] DMAR: RMRR base: 0x000000000e8000 end: 0x000000000e8fff
[    0.028758] DMAR: RMRR base: 0x000000f1dee000 end: 0x000000f1deefff
[    0.028761] DMAR-IR: IOAPIC id 8 under DRHD base  0xfed90000 IOMMU 0
[    0.028762] DMAR-IR: HPET id 0 under DRHD base 0xfed90000
[    0.028763] DMAR-IR: x2apic is disabled because BIOS sets x2apic opt out bit.
[    0.028763] DMAR-IR: Use 'intremap=no_x2apic_optout' to override the BIOS setting.
[    0.028894] DMAR-IR: Enabled IRQ remapping in xapic mode
[    0.690271] DMAR: No ATSR found
[    0.694262] DMAR: dmar0: Using Queued invalidation
[    0.694429] DMAR: Hardware identity mapping for device 0000:00:00.0
[    0.694431] DMAR: Hardware identity mapping for device 0000:00:01.0
[    0.694433] DMAR: Hardware identity mapping for device 0000:00:06.0
[    0.694435] DMAR: Hardware identity mapping for device 0000:00:1a.0
[    0.694436] DMAR: Hardware identity mapping for device 0000:00:1c.0
[    0.694438] DMAR: Hardware identity mapping for device 0000:00:1c.4
[    0.694439] DMAR: Hardware identity mapping for device 0000:00:1c.6
[    0.694441] DMAR: Hardware identity mapping for device 0000:00:1c.7
[    0.694443] DMAR: Hardware identity mapping for device 0000:00:1d.0
[    0.694445] DMAR: Hardware identity mapping for device 0000:00:1f.0
[    0.694450] DMAR: Hardware identity mapping for device 0000:07:00.0
[    0.694456] DMAR: Hardware identity mapping for device 0000:04:00.0
[    0.694461] DMAR: Hardware identity mapping for device 0000:01:00.1
[    0.694463] DMAR: Hardware identity mapping for device 0000:01:00.4
[    0.694464] DMAR: Setting RMRR:
[    0.694617] DMAR: Setting identity map for device 0000:01:00.0 [0xf1dee000 - 0xf1deefff]
[    0.694775] DMAR: Setting identity map for device 0000:01:00.2 [0xf1dee000 - 0xf1deefff]
[    0.694784] DMAR: Ignoring identity map for HW passthrough device 0000:01:00.4 [0xf1dee000 - 0xf1deefff]
[    0.694931] DMAR: Setting identity map for device 0000:00:1f.2 [0xe8000 - 0xe8fff]
[    0.694939] DMAR: Ignoring identity map for HW passthrough device 0000:07:00.0 [0xe8000 - 0xe8fff]
[    0.695088] DMAR: Setting identity map for device 0000:03:00.0 [0xe8000 - 0xe8fff]
[    0.695242] DMAR: Setting identity map for device 0000:03:00.1 [0xe8000 - 0xe8fff]
[    0.695250] DMAR: Ignoring identity map for HW passthrough device 0000:04:00.0 [0xe8000 - 0xe8fff]
[    0.695251] DMAR: Setting identity map for device 0000:01:00.0 [0xe8000 - 0xe8fff]
[    0.695259] DMAR: Setting identity map for device 0000:01:00.2 [0xe8000 - 0xe8fff]
[    0.695267] DMAR: Setting identity map for device 0000:00:1f.2 [0xf4000 - 0xf4fff]
[    0.695269] DMAR: Ignoring identity map for HW passthrough device 0000:07:00.0 [0xf4000 - 0xf4fff]
[    0.695271] DMAR: Setting identity map for device 0000:03:00.0 [0xf4000 - 0xf4fff]
[    0.695272] DMAR: Setting identity map for device 0000:03:00.1 [0xf4000 - 0xf4fff]
[    0.695274] DMAR: Ignoring identity map for HW passthrough device 0000:04:00.0 [0xf4000 - 0xf4fff]
[    0.695275] DMAR: Setting identity map for device 0000:01:00.0 [0xf4000 - 0xf4fff]
[    0.695276] DMAR: Setting identity map for device 0000:01:00.2 [0xf4000 - 0xf4fff]
[    0.695278] DMAR: Setting identity map for device 0000:00:1f.2 [0xf1f7e000 - 0xf1f7efff]
[    0.695285] DMAR: Ignoring identity map for HW passthrough device 0000:07:00.0 [0xf1f7e000 - 0xf1f7efff]
[    0.695286] DMAR: Setting identity map for device 0000:03:00.0 [0xf1f7e000 - 0xf1f7efff]
[    0.695294] DMAR: Setting identity map for device 0000:03:00.1 [0xf1f7e000 - 0xf1f7efff]
[    0.695301] DMAR: Ignoring identity map for HW passthrough device 0000:04:00.0 [0xf1f7e000 - 0xf1f7efff]
[    0.695302] DMAR: Setting identity map for device 0000:01:00.0 [0xf1f7e000 - 0xf1f7efff]
[    0.695307] DMAR: Setting identity map for device 0000:01:00.2 [0xf1f7e000 - 0xf1f7efff]
[    0.695312] DMAR: Setting identity map for device 0000:00:1f.2 [0xf1f7f000 - 0xf1f8efff]
[    0.695313] DMAR: Ignoring identity map for HW passthrough device 0000:07:00.0 [0xf1f7f000 - 0xf1f8efff]
[    0.695315] DMAR: Setting identity map for device 0000:03:00.0 [0xf1f7f000 - 0xf1f8efff]
[    0.695316] DMAR: Setting identity map for device 0000:03:00.1 [0xf1f7f000 - 0xf1f8efff]
[    0.695318] DMAR: Ignoring identity map for HW passthrough device 0000:04:00.0 [0xf1f7f000 - 0xf1f8efff]
[    0.695319] DMAR: Setting identity map for device 0000:01:00.0 [0xf1f7f000 - 0xf1f8efff]
[    0.695321] DMAR: Setting identity map for device 0000:01:00.2 [0xf1f7f000 - 0xf1f8efff]
[    0.695323] DMAR: Setting identity map for device 0000:00:1f.2 [0xf1f8f000 - 0xf1f92fff]
[    0.695325] DMAR: Ignoring identity map for HW passthrough device 0000:07:00.0 [0xf1f8f000 - 0xf1f92fff]
[    0.695326] DMAR: Setting identity map for device 0000:03:00.0 [0xf1f8f000 - 0xf1f92fff]
[    0.695327] DMAR: Setting identity map for device 0000:03:00.1 [0xf1f8f000 - 0xf1f92fff]
[    0.695329] DMAR: Ignoring identity map for HW passthrough device 0000:04:00.0 [0xf1f8f000 - 0xf1f92fff]
[    0.695330] DMAR: Setting identity map for device 0000:01:00.0 [0xf1f8f000 - 0xf1f92fff]
[    0.695332] DMAR: Setting identity map for device 0000:01:00.2 [0xf1f8f000 - 0xf1f92fff]
[    0.695333] DMAR: Setting identity map for device 0000:00:1f.2 [0xf1f93000 - 0xf1f94fff]
[    0.695335] DMAR: Ignoring identity map for HW passthrough device 0000:07:00.0 [0xf1f93000 - 0xf1f94fff]
[    0.695336] DMAR: Setting identity map for device 0000:03:00.0 [0xf1f93000 - 0xf1f94fff]
[    0.695337] DMAR: Setting identity map for device 0000:03:00.1 [0xf1f93000 - 0xf1f94fff]
[    0.695339] DMAR: Ignoring identity map for HW passthrough device 0000:04:00.0 [0xf1f93000 - 0xf1f94fff]
[    0.695340] DMAR: Setting identity map for device 0000:01:00.0 [0xf1f93000 - 0xf1f94fff]
[    0.695342] DMAR: Setting identity map for device 0000:01:00.2 [0xf1f93000 - 0xf1f94fff]
[    0.695344] DMAR: Setting identity map for device 0000:01:00.0 [0xf1ff6000 - 0xf1ffcfff]
[    0.695345] DMAR: Setting identity map for device 0000:01:00.2 [0xf1ff6000 - 0xf1ffcfff]
[    0.695347] DMAR: Ignoring identity map for HW passthrough device 0000:01:00.4 [0xf1ff6000 - 0xf1ffcfff]
[    0.695348] DMAR: Ignoring identity map for HW passthrough device 0000:00:1a.0 [0xf1ffd000 - 0xf1ffffff]
[    0.695349] DMAR: Ignoring identity map for HW passthrough device 0000:00:1d.0 [0xf1ffd000 - 0xf1ffffff]
[    0.695352] DMAR: Prepare 0-16MiB unity mapping for LPC
[    0.695353] DMAR: Ignoring identity map for HW passthrough device 0000:00:1f.0 [0x0 - 0xffffff]
[    0.695354] DMAR: Intel(R) Virtualization Technology for Directed I/O
[    0.742301] DMAR: 32bit 0000:00:1a.0 uses non-identity mapping
[    0.742445] DMAR: Setting identity map for device 0000:00:1a.0 [0xf1ffd000 - 0xf1ffffff]
[    0.751812] DMAR: 32bit 0000:00:1d.0 uses non-identity mapping
[    0.751956] DMAR: Setting identity map for device 0000:00:1d.0 [0xf1ffd000 - 0xf1ffffff]
[    0.761819] DMAR: 32bit 0000:01:00.4 uses non-identity mapping
[    0.761961] DMAR: Setting identity map for device 0000:01:00.4 [0xf1dee000 - 0xf1deefff]
[    0.761970] DMAR: Setting identity map for device 0000:01:00.4 [0xf1ff6000 - 0xf1ffcfff]

Testing

(overcloud) [stack@lab-director ~]$ source overcloudrc_myproject

Create a flavor without PCI Passtrough USB controller access:

(overcloud) [stack@lab-director ~]$ openstack flavor create --ram 2048 --disk 10 --vcpus 1 m1.small
+----------------------------+--------------------------------------+
| Field                      | Value                                |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                |
| OS-FLV-EXT-DATA:ephemeral  | 0                                    |
| disk                       | 10                                   |
| id                         | 03ad81da-d93c-42d0-b099-c48b550a5507 |
| name                       | m1.small                             |
| os-flavor-access:is_public | True                                 |
| properties                 |                                      |
| ram                        | 20148                                |
| rxtx_factor                | 1.0                                  |
| swap                       |                                      |
| vcpus                      | 1                                    |
+----------------------------+--------------------------------------+

Create a flavour with PCI Passtrough USB controller access:

(overcloud) [stack@lab-director ~]$ openstack flavor create --ram 2048 --disk 10 --vcpus 1 m1.small-usb
+----------------------------+--------------------------------------+
| Field                      | Value                                |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                |
| OS-FLV-EXT-DATA:ephemeral  | 0                                    |
| disk                       | 10                                   |
| id                         | acd58d32-d34d-4687-b6bb-9c088c3f7913 |
| name                       | m1.small-usb                         |
| os-flavor-access:is_public | True                                 |
| properties                 |                                      |
| ram                        | 20148                                |
| rxtx_factor                | 1.0                                  |
| swap                       |                                      |
| vcpus                      | 1                                    |
+----------------------------+--------------------------------------+

Update the USB passthrough flavor:

(overcloud) [stack@lab-director ~]$ openstack flavor show m1.small-usb
+----------------------------+--------------------------------------+
| Field                      | Value                                |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                |
| OS-FLV-EXT-DATA:ephemeral  | 0                                    |
| access_project_ids         | None                                 |
| disk                       | 10                                   |
| id                         | ac7e9fdd-e8d8-47d8-9585-e7d466302d19 |
| name                       | m1.small-usb                         |
| os-flavor-access:is_public | True                                 |
| properties                 |                                      |
| ram                        | 2048                                 |
| rxtx_factor                | 1.0                                  |
| swap                       |                                      |
| vcpus                      | 1                                    |
+----------------------------+--------------------------------------+

(overcloud) [stack@lab-director ~]$ openstack flavor set m1.small-usb --property "pci_passthrough:alias"="a1:1"

(overcloud) [stack@lab-director ~]$ openstack flavor show m1.small-usb
+----------------------------+--------------------------------------+
| Field                      | Value                                |
+----------------------------+--------------------------------------+
| OS-FLV-DISABLED:disabled   | False                                |
| OS-FLV-EXT-DATA:ephemeral  | 0                                    |
| access_project_ids         | None                                 |
| disk                       | 10                                   |
| id                         | ac7e9fdd-e8d8-47d8-9585-e7d466302d19 |
| name                       | m1.small-usb                         |
| os-flavor-access:is_public | True                                 |
| properties                 | pci_passthrough:alias='a1:1'         |
| ram                        | 2048                                 |
| rxtx_factor                | 1.0                                  |
| swap                       |                                      |
| vcpus                      | 1                                    |
+----------------------------+--------------------------------------+

Create an instance with PCI passthrough:

(overcloud) [stack@lab-director ~]$ openstack server create --flavor m1.small-usb --image rhel75 --security-group web --nic net-id=internal0 --key-name lambda instance0        
+-------------------------------------+-----------------------------------------------------+
| Field                               | Value                                               |
+-------------------------------------+-----------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                              |
| OS-EXT-AZ:availability_zone         |                                                     |
| OS-EXT-SRV-ATTR:host                | None                                                |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None                                                |
| OS-EXT-SRV-ATTR:instance_name       |                                                     |
| OS-EXT-STS:power_state              | NOSTATE                                             |
| OS-EXT-STS:task_state               | scheduling                                          |
| OS-EXT-STS:vm_state                 | building                                            |
| OS-SRV-USG:launched_at              | None                                                |
| OS-SRV-USG:terminated_at            | None                                                |
| accessIPv4                          |                                                     |
| accessIPv6                          |                                                     |
| addresses                           |                                                     |
| adminPass                           | nkKJ2BP4d7Fw                                        |
| config_drive                        |                                                     |
| created                             | 2018-06-29T11:18:40Z                                |
| flavor                              | m1.small-usb (ac7e9fdd-e8d8-47d8-9585-e7d466302d19) |
| hostId                              |                                                     |
| id                                  | d865e835-f009-4e8b-b582-0b49e30c77c6                |
| image                               | rhel75 (8ff93daa-5088-45c9-a3e1-61ab55867b63)       |
| key_name                            | lambda                                              |
| name                                | instance0                                           |
| progress                            | 0                                                   |
| project_id                          | 6163570b88e54038897a97e7be1f12d0                    |
| properties                          |                                                     |
| security_groups                     | name='a2bfd621-6872-4d98-ac33-12c481b3378c'         |
| status                              | BUILD                                               |
| updated                             | 2018-06-29T11:18:40Z                                |
| user_id                             | 602cf7eb3cbd43b19b665e9c3a11af38                    |
| volumes_attached                    |                                                     |
+-------------------------------------+-----------------------------------------------------+

List instances:

(overcloud) [stack@lab-director ~]$ openstack server list
+--------------------------------------+-----------+--------+------------------------------------+--------+--------------+
| ID                                   | Name      | Status | Networks                           | Image  | Flavor       |
+--------------------------------------+-----------+--------+------------------------------------+--------+--------------+
| d865e835-f009-4e8b-b582-0b49e30c77c6 | instance0 | ACTIVE | internal0=172.31.0.4, 172.16.5.155 | rhel75 | m1.small-usb |
+--------------------------------------+-----------+--------+------------------------------------+--------+--------------+

Attach a floating Ip to the instance:

(overcloud) [stack@lab-director ~]$ openstack server add floating ip instance0 172.16.5.155   

Test to ping the instance:

[root@overcloud-controller-0 ~]# ping 172.16.5.155
PING 172.16.5.155 (172.16.5.155) 56(84) bytes of data.
64 bytes from 172.16.5.155: icmp_seq=1 ttl=63 time=1.41 ms
64 bytes from 172.16.5.155: icmp_seq=2 ttl=63 time=2.08 ms
64 bytes from 172.16.5.155: icmp_seq=3 ttl=63 time=2.19 ms
64 bytes from 172.16.5.155: icmp_seq=4 ttl=63 time=1.78 ms

SSH to the instance:

[root@overcloud-controller-0 ~]# ssh cloud-user@172.16.5.155
The authenticity of host '172.16.5.155 (172.16.5.155)' can't be established.
ECDSA key fingerprint is SHA256:F0nJbJrJOsDH8oQvV0k7b0+uKVSnLJ8+HRfLI7qkcgI.
ECDSA key fingerprint is MD5:7d:ea:00:c6:b0:7e:b3:7b:b1:0e:5d:ef:2c:3d:fd:b4.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '172.16.5.155' (ECDSA) to the list of known hosts.
[cloud-user@instance0 ~]$ uptime
 07:23:40 up 4 min,  1 user,  load average: 0.05, 0.13, 0.06

Install pciutils and usbutils tools in the instance:

[cloud-user@instance0 ~]$ yum install -y pciutils libusbx usbutils

Check PCI status:

[root@instance0 ~]# lspci -nn
00:00.0 Host bridge [0600]: Intel Corporation 440FX - 82441FX PMC [Natoma] [8086:1237] (rev 02)
00:01.0 ISA bridge [0601]: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] [8086:7000]
00:01.1 IDE interface [0101]: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] [8086:7010]
00:01.2 USB controller [0c03]: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] [8086:7020] (rev 01)
00:01.3 Bridge [0680]: Intel Corporation 82371AB/EB/MB PIIX4 ACPI [8086:7113] (rev 03)
00:02.0 VGA compatible controller [0300]: Cirrus Logic GD 5446 [1013:00b8]
00:03.0 Ethernet controller [0200]: Red Hat, Inc. Virtio network device [1af4:1000]
00:04.0 SCSI storage controller [0100]: Red Hat, Inc. Virtio block device [1af4:1001]
00:05.0 USB controller [0c03]: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100] (rev 10)
00:06.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon [1af4:1002]

Check USB status:

[root@instance0 tmp]# lsusb
Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd 
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

Check USB hubs:

[root@instance0 tmp]# lsusb -v | grep -E '\<(Bus|iProduct|bDeviceClass|bDeviceProtocol)' 2>/dev/null
Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
  bDeviceClass            9 Hub
  bDeviceProtocol         3 
  iProduct                2 xHCI Host Controller
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
  bDeviceClass            9 Hub
  bDeviceProtocol         1 Single TT
  iProduct                2 xHCI Host Controller
Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd 
  bDeviceClass            0 (Defined at Interface level)
  bDeviceProtocol         0 
  iProduct                3 QEMU USB Tablet
      (Bus Powered)
  bDeviceClass            0 (Defined at Interface level)
  bDeviceProtocol         0 
  (Bus Powered)
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
  bDeviceClass            9 Hub
  bDeviceProtocol         0 Full speed (or root) hub
  iProduct                2 UHCI Host Controller

Plug the USB Sandisk key inside the compute node.

We can see a Sandisk key in the USB device list:

[root@instance0 ~]# lsusb
Bus 003 Device 003: ID 0781:5580 SanDisk Corp. SDCZ80 Flash Drive
Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 0627:0001 Adomax Technology Co., Ltd 
Bus 001 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub

The device is seen with the fdisk command inside the instance:

[root@instance0 tmp]# fdisk -l /dev/sda

Disk /dev/sda: 31.4 GB, 31406948352 bytes, 61341696 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk label type: dos
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sda1   *           2    61341695    30670847    b  W95 FAT32
[root@instance0 tmp]# fdisk -l | grep Disk
Disk /dev/vda: 100.7 GB, 10737418240 bytes, 20971520 sectors
Disk label type: dos
Disk identifier: 0x000ac719
Disk /dev/sda: 31.4 GB, 31406948352 bytes, 61341696 sectors
Disk label type: dos
Disk identifier: 0x00000000

Mount the USB key inside the instance:

[root@instance0 tmp]# mount -t vfat /dev/sda1 /mnt/sda1/

[root@instance0 tmp]# df -h /mnt/sda1
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda1        30G  1.6M   30G   1% /mnt/sda1

Write a file:

cat << EOF > /mnt/sda1/test_file_osp_13_2018.txt 
This is a text written for a test
EOF

Check the file:

[root@instance0 ~]# ls /mnt/sda1/
test_file_osp_13_2018.txt

Unmount the key:

[root@instance0 ~]# umount /mnt/sda1/

On the compute host, no USB device are listed:

[root@overcloud-compute-0 tmp]# lsusb
Bus 002 Device 003: ID 0424:2660 Standard Microsystems Corp. Hub
Bus 002 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0001 Linux Foundation 1.1 root hub
Bus 007 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 006 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 8087:0024 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

But still PCI device visible:

[root@overcloud-compute-0 tmp]# lspci -nn | grep Fresco
07:00.0 USB controller [0c03]: Fresco Logic FL1100 USB 3.0 Host Controller [1b73:1100] (rev 10)

With admin rights, we can use OS-EXT-SRV-ATTR to find the libvirt name of the instance:

(overcloud) [stack@lab-director ~]$ openstack server show instance0 | grep instance_name 
| OS-EXT-SRV-ATTR:instance_name       | instance-00000005     -

On the compute node we can find the running instance:

[heat-admin@overcloud-compute-0 ~]$ sudo virsh list
 Id    Name                           State
----------------------------------------------------
 1     instance-00000005              running    

vfio prepared by OpenStack can be seen in the libvirt configuration:

[heat-admin@overcloud-compute-0 tmp]$ sudo virsh dumpxml instance-00000005 | xmllint --xpath '//hostdev' -
<hostdev mode="subsystem" type="pci" managed="yes">
      <driver name="vfio"/>
      <source>
        <address domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
      </source>
      <alias name="hostdev0"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x05" function="0x0"/>
</hostdev>

To compare check the default instance configuration, create an instance without PCI Passtrough USB controller access:

(overcloud) [stack@lab-director images]$ openstack server create --flavor m1.small --image rhel75 --security-group web --nic net-id=internal0 --key-name lambda instance0
+-------------------------------------+-------------------------------------------------+
| Field                               | Value                                           |
+-------------------------------------+-------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                          |
| OS-EXT-AZ:availability_zone         |                                                 |
| OS-EXT-SRV-ATTR:host                | None                                            |
| OS-EXT-SRV-ATTR:hypervisor_hostname | None                                            |
| OS-EXT-SRV-ATTR:instance_name       |                                                 |
| OS-EXT-STS:power_state              | NOSTATE                                         |
| OS-EXT-STS:task_state               | scheduling                                      |
| OS-EXT-STS:vm_state                 | building                                        |
| OS-SRV-USG:launched_at              | None                                            |
| OS-SRV-USG:terminated_at            | None                                            |
| accessIPv4                          |                                                 |
| accessIPv6                          |                                                 |
| addresses                           |                                                 |
| adminPass                           | FJ7KaJ6cLuc7                                    |
| config_drive                        |                                                 |
| created                             | 2018-06-28T19:53:14Z                            |
| flavor                              | m1.small (8733812d-eacd-4a8d-9bcb-c920e9f583da) |
| hostId                              |                                                 |
| id                                  | 89a8b073-876f-4a77-9389-8537c7ce5c6a            |
| image                               | rhel75 (63b8eac0-095e-4d6a-b137-93f6b72e5c81)   |
| key_name                            | lambda                                          |
| name                                | instance0                                       |
| progress                            | 0                                               |
| project_id                          | c9d4227db1f54805881e81843abe7555                |
| properties                          |                                                 |
| security_groups                     | name='6f203816-7543-4218-bdbc-f0be9be6436f'     |
| status                              | BUILD                                           |
| updated                             | 2018-06-28T19:53:14Z                            |
| user_id                             | 9a6ad6a8e3c54f24b5eef74c3b2f4eb4                |
| volumes_attached                    |                                                 |
+-------------------------------------+-------------------------------------------------+

Check default PCI devices without USB controller:

[root@instance0 tmp]# lspci -nn
00:00.0 Host bridge [0600]: Intel Corporation 440FX - 82441FX PMC [Natoma] [8086:1237] (rev 02)
00:01.0 ISA bridge [0601]: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II] [8086:7000]
00:01.1 IDE interface [0101]: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II] [8086:7010]
00:01.2 USB controller [0c03]: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] [8086:7020] (rev 01)
00:01.3 Bridge [0680]: Intel Corporation 82371AB/EB/MB PIIX4 ACPI [8086:7113] (rev 03)
00:02.0 VGA compatible controller [0300]: Cirrus Logic GD 5446 [1013:00b8]
00:03.0 Ethernet controller [0200]: Red Hat, Inc. Virtio network device [1af4:1000]
00:04.0 SCSI storage controller [0100]: Red Hat, Inc. Virtio block device [1af4:1001]
00:05.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon [1af4:1002]