Mailing List Archive

bug suspcion and proposed modification when xen-pciback failed to map an irq (-19) to a domU
Hello,

I would like to paravirtualize several instances of a minimal Linux solution with xen. Guest OS have a PCI passthrough access on the ethernet interfaces.
I use a pvops kernel 4.4.122 , xen 4.11.2 and an x86 platform. My hardware platform doesn't support IOMMU but I doesn't need hard segmentation between OS with this platform.

I got a bug when the Ethernet driver configures the MSI for a domU from a domU.
pciback 0000:00:14.3: xen map irq failed -19 for 1 domain

Please find the bug description and maybe a solution :

1) dom1 is configured to use 0000:00:14.3
in file /etc/xen/dom1 :
pci=['00:14.3,permissive=1,seize=1,msitranslate=1']
and kernel bootparam include "iommu=soft"

2) xen-pciback is loaded in dom0 :
> modprobe xen-pciback passthrough=1 hide='(0000:00:14.3)'
> xl pci-assignable-list
0000:00:14.3

(xen-pciback is not loaded in dom1)

3) Xen debug terminal print that all the PCI devices are linked to dom0 before and after xen-pciback probing
...
(XEN) 0000:00:14.3 - dom 0 - node -1 - MSIs < >
...

4) no error are detected when I start dom1
> xl create /etc/xen/dom1
or with
> xl pci-attach dom1 00:14.3

5) For each Ethernet device I see the following call to the hypervisor
- xc_domain_iomem_permission
- xc_domain_ioport_permission
- xc_domain_iomem_permission
- xc_physdev_map_pirq
- xc_domain_irq_permission
- xc_assign_device

none of them update the pdev->domain or the pdev->domain->domain_id of the pdev element linked with the bfd 0000:00:14.3 (devfn 163)

6) Xen debug terminal prints again that all the PCI devices are linked to dom0
...
(XEN) 0000:00:14.3 - dom 0 - node -1 - MSIs < >
...

I guess that this is normal because Dom0 will allow dom1 to configure the device (IRQ & DMA) thanks to xen-pciback. Could you confirm me this assertion ?

7) When the Ethernet driver (igb) of dom1 configure MSI. xen failed to map MSI irqs because map_domain_pirq call pci_get_pdev_by_domain with domain_id 1 and the device is linked with domain_id 0
> modprobe igb
igb: Copyright (c) 2007-2014 Intel Corporation.
igb 0000:00:14.3: enabling device (0000 -> 0002)
igb 0000:00:14.3: Xen PCI mapped GSI23 to IRQ25
en_initdom_setup_msi_irqs:xen map irq map_irq.domid 3, map_irq.type 3 map_irq.index -1 , map_irq.pirq -1
xen_initdom_setup_msi_irqs:xen map irq map_irq.bus 0, map_irq.devfn 163
pciback 0000:00:14.3: PCI_CAP_ID_MSIX
pci_seg_supported

There,HYPERVISOR_physdev_op and PHYSDEVOP_map_pirq call map_domain_pirq :
(XEN) map_domain_pirq--> call pci_get_pdev_by_domain
(XEN) pci_get_pdev_by_domain-->
(XEN) domain->domain_id 1
(XEN) seg 0 bus 0 devfn 163
...

Unfortunately the device is already mapped to domain 0 and pci_get_pdev_by_domain don't found the device ... So map_domain_pirq & HYPERVISOR_physdev_op return -19 :
pciback 0000:00:14.3: xen map irq failed -19 for 2 domain

I don't find any hypervisor syscall able to assign my device to a domU. xc_assign_device is cancelled because iommu is disabled.
libxl_pci.c:1137:do_pci_add: Domain 3:do_pci_add --> call xc_assign_device -> hypervisor call arch_do_domctl--> iommu_do_domctl--> CANCELLED

To solve my issue , I just need to replace in map_domain_pirq in xen/arch/x86/irq.c
pdev = pci_get_pdev_by_domain(d, msi->seg, msi->bus, msi->devfn);
by
pdev = pci_get_pdev(msi->seg, msi->bus, msi->devfn);

With this "fix" in Xen, dom1 can use the ethernet device and its MSI that I see in /proc/interrupt

I use a customized cross-compilation of xen and xen-tools for a customized mininal linux solution thanks to buildroot. This solution has been built from scratch. So I'm not sure that all the xen components work perfectly. However, everything seem to work with my modifications (CPU, ethernet, memory, disk, interrupt ...)

So is it really a bug ? or just a configuration error ? maybe a parameter has been forgotten in /etc/xen/domU ?

Thank you for your help
Re: bug suspcion and proposed modification when xen-pciback failed to map an irq (-19) to a domU [ In reply to ]
On 29.11.2019 09:58, DOZ, MARC (ext) wrote:
> Hello,
>
> I would like to paravirtualize several instances of a minimal Linux solution with xen. Guest OS have a PCI passthrough access on the ethernet interfaces.
> I use a pvops kernel 4.4.122 , xen 4.11.2 and an x86 platform. My hardware platform doesn't support IOMMU but I doesn't need hard segmentation between OS with this platform.
>
> I got a bug when the Ethernet driver configures the MSI for a domU from a domU.
> pciback 0000:00:14.3: xen map irq failed -19 for 1 domain
>
> Please find the bug description and maybe a solution :
>
> 1) dom1 is configured to use 0000:00:14.3
> in file /etc/xen/dom1 :
> pci=['00:14.3,permissive=1,seize=1,msitranslate=1']
> and kernel bootparam include "iommu=soft"
>
> 2) xen-pciback is loaded in dom0 :
>> modprobe xen-pciback passthrough=1 hide='(0000:00:14.3)'
>> xl pci-assignable-list
> 0000:00:14.3
>
> (xen-pciback is not loaded in dom1)
>
> 3) Xen debug terminal print that all the PCI devices are linked to dom0 before and after xen-pciback probing
> ...
> (XEN) 0000:00:14.3 - dom 0 - node -1 - MSIs < >
> ...
>
> 4) no error are detected when I start dom1

I think if you retry with current master or staging, you'll find this
changed.

>> xl create /etc/xen/dom1
> or with
>> xl pci-attach dom1 00:14.3
>
> 5) For each Ethernet device I see the following call to the hypervisor
> - xc_domain_iomem_permission
> - xc_domain_ioport_permission
> - xc_domain_iomem_permission
> - xc_physdev_map_pirq
> - xc_domain_irq_permission
> - xc_assign_device
>
> none of them update the pdev->domain or the pdev->domain->domain_id of the pdev element linked with the bfd 0000:00:14.3 (devfn 163)
>
> 6) Xen debug terminal prints again that all the PCI devices are linked to dom0
> ...
> (XEN) 0000:00:14.3 - dom 0 - node -1 - MSIs < >
> ...
>
> I guess that this is normal because Dom0 will allow dom1 to configure the device (IRQ & DMA) thanks to xen-pciback. Could you confirm me this assertion ?
>
> 7) When the Ethernet driver (igb) of dom1 configure MSI. xen failed to map MSI irqs because map_domain_pirq call pci_get_pdev_by_domain with domain_id 1 and the device is linked with domain_id 0
>> modprobe igb
> igb: Copyright (c) 2007-2014 Intel Corporation.
> igb 0000:00:14.3: enabling device (0000 -> 0002)
> igb 0000:00:14.3: Xen PCI mapped GSI23 to IRQ25
> en_initdom_setup_msi_irqs:xen map irq map_irq.domid 3, map_irq.type 3 map_irq.index -1 , map_irq.pirq -1
> xen_initdom_setup_msi_irqs:xen map irq map_irq.bus 0, map_irq.devfn 163
> pciback 0000:00:14.3: PCI_CAP_ID_MSIX
> pci_seg_supported
>
> There,HYPERVISOR_physdev_op and PHYSDEVOP_map_pirq call map_domain_pirq :
> (XEN) map_domain_pirq--> call pci_get_pdev_by_domain
> (XEN) pci_get_pdev_by_domain-->
> (XEN) domain->domain_id 1
> (XEN) seg 0 bus 0 devfn 163
> ...
>
> Unfortunately the device is already mapped to domain 0 and pci_get_pdev_by_domain don't found the device ... So map_domain_pirq & HYPERVISOR_physdev_op return -19 :
> pciback 0000:00:14.3: xen map irq failed -19 for 2 domain
>
> I don't find any hypervisor syscall able to assign my device to a domU.
> xc_assign_device is cancelled because iommu is disabled.
> libxl_pci.c:1137:do_pci_add: Domain 3:do_pci_add --> call xc_assign_device
> -> hypervisor call arch_do_domctl--> iommu_do_domctl--> CANCELLED

And intentionally so.

> To solve my issue , I just need to replace in map_domain_pirq in xen/arch/x86/irq.c
> pdev = pci_get_pdev_by_domain(d, msi->seg, msi->bus, msi->devfn);
> by
> pdev = pci_get_pdev(msi->seg, msi->bus, msi->devfn);
>
> With this "fix" in Xen, dom1 can use the ethernet device and its MSI that I see in /proc/interrupt

Except that this is not a "fix", but the introduction of a security
vulnerability (permitting interrupt setup on un-owned devices). See
XSA-237, which actually changed it in the opposite direction of
what you're proposing.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: bug suspcion and proposed modification when xen-pciback failed to map an irq (-19) to a domU [ In reply to ]
>Except that this is not a "fix", but the introduction of a security vulnerability (permitting interrupt setup on un-owned devices). See XSA-237, which actually changed it in the opposite direction of what you're proposing.

Ok, I found it :
https://xenbits.xen.org/xsa/xsa237-4.5/0001-x86-dont-allow-MSI-pIRQ-mapping-on-unowned-device.patch

"MSI setup should be permitted only for existing devices owned by the respective guest"

But how to change the owner of my device or update the pdev->domain->domain_id ?


_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: bug suspcion and proposed modification when xen-pciback failed to map an irq (-19) to a domU [ In reply to ]
On 29.11.2019 13:34, DOZ, MARC (ext) wrote:
>
>> Except that this is not a "fix", but the introduction of a security vulnerability (permitting interrupt setup on un-owned devices). See XSA-237, which actually changed it in the opposite direction of what you're proposing.
>
> Ok, I found it :
> https://xenbits.xen.org/xsa/xsa237-4.5/0001-x86-dont-allow-MSI-pIRQ-mapping-on-unowned-device.patch
>
> "MSI setup should be permitted only for existing devices owned by the respective guest"
>
> But how to change the owner of my device or update the pdev->domain->domain_id ?

With the code as is and without an IOMMU there's no pre-cooked
way to, I'm afraid. You could try granting the guest access to
MMIO and IRQ "manually" (there are guest config file options
for this), but I take it you'll be in trouble if (as iirc you've
said) the device / driver want to use MSI.

Jan

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: bug suspcion and proposed modification when xen-pciback failed to map an irq (-19) to a domU [ In reply to ]
Ok,

I have a trusted software to localy configure the ethernet device assignment. I will probably add a "pre-cooked way" to share the configuration to the hypervisor and allow the MSI configuration from a device only with only one granted domain.

Thank you very much for the help

-----Original Message-----
From: Jan Beulich <jbeulich@suse.com>
Sent: Friday, November 29, 2019 2:32 PM
To: DOZ, MARC (ext) <marc.doz.external@atos.net>
Cc: xen-devel@lists.xenproject.org
Subject: Re: [Xen-devel] bug suspcion and proposed modification when xen-pciback failed to map an irq (-19) to a domU

On 29.11.2019 13:34, DOZ, MARC (ext) wrote:
>
>> Except that this is not a "fix", but the introduction of a security vulnerability (permitting interrupt setup on un-owned devices). See XSA-237, which actually changed it in the opposite direction of what you're proposing.
>
> Ok, I found it :
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fxenb
> its.xen.org%2Fxsa%2Fxsa237-4.5%2F0001-x86-dont-allow-MSI-pIRQ-mapping-
> on-unowned-device.patch&amp;data=02%7C01%7Cmarc.doz.external%40atos.ne
> t%7Cddc18189b78d47e0165d08d774d09a4a%7C33440fc6b7c7412cbb730e70b0198d5
> a%7C0%7C0%7C637106311594585817&amp;sdata=1EaYn7PE6n2JZxldEciBla7QBWBRW
> jZUugtEgmCnuZ4%3D&amp;reserved=0
>
> "MSI setup should be permitted only for existing devices owned by the respective guest"
>
> But how to change the owner of my device or update the pdev->domain->domain_id ?

With the code as is and without an IOMMU there's no pre-cooked way to, I'm afraid. You could try granting the guest access to MMIO and IRQ "manually" (there are guest config file options for this), but I take it you'll be in trouble if (as iirc you've
said) the device / driver want to use MSI.

Jan
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel