Mailing List Archive

pygrub, installed with Python 3, doesn't boot Xen DomU-s
This follows on from

pygrub gives "raise RuntimeError("Unable to find partition containing kernel")"

https://lists.xenproject.org/archives/html/xen-devel/2019-05/msg01589.html

and for some reason I submitted my latest findings onto the grub-devel list,
only to be reminded that pygrub is part of Xen, not Grub, so reposting here:


OK, so I am trying to build an LFS (Linux From Scratch) system that
will serve as a basic Xen Dom0.

You can see the way things have been compiled, including my UEFI-aware
Grub, here (Very much a WIP)

http://youvegotbuckleys.org.nz/LFS/LFS-BOOK.html


Suffice it to say that the LFS Grub boots either of the two
UEFI enties (non-Xen and Dom0) I have so as to bring up the
system.


I also have a couple of VBD-backed DomU-s that I can bring up,
using the pygrub from either an Ubuntu 1404, or a Centos 6.10
with Xen4Centos, Dom0.

When I come to try and have my LFS Xen's pygrub boot the VBD-backed
DomU, I get errors of the form


# cat /mnt/var/log/xen/bootloader.4.log
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /boot/grub/grub.cfg
Traceback (most recent call last):
File "/usr/lib/xen/bin/pygrub", line 928, in <module>
raise RuntimeError("Unable to find partition containing kernel")
RuntimeError: Unable to find partition containing kernel

I originally thought that this might be down to my using a Xen
source from the Xen Git repo that was in beyond 4.12 (which I
needed so to get a Xen that was happy with just Python3. See
https://lists.xenproject.org/archives/html/xen-devel/2019-04/msg00996.html
) and had prhaps had tripped over a regression, as that error used
to be a problem with Xen (if you go back far enough).


However I have since tracked down my problem to the pygrub, in
that if I point pygrub to the start of the partition in the VBD
that has the grub.cfg in it, I see the following failure to parse
the config file:


bash-5.0# /usr/lib/xen/bin/pygrub --debug --offset=1048576
--list-entries /dev/vg_xen_vbds/lv_4g_02
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /boot/grub/grub.cfg
Traceback (most recent call last):
File "/usr/lib/xen/bin/pygrub", line 907, in <module>
chosencfg = run_grub(file, entry, fs, incfg["args"])
File "/usr/lib/xen/bin/pygrub", line 625, in run_grub
g = Grub(file, fs)
File "/usr/lib/xen/bin/pygrub", line 249, in __init__
self.read_config(file, fs)
File "/usr/lib/xen/bin/pygrub", line 460, in read_config
self.cf.parse(buf)
File "/usr/lib/python3.7/site-packages/grub/GrubConf.py", line 376, in parse
lines = buf.split("\n")
TypeError: a bytes-like object is required, not 'str'


whereas if I do the same with the pygrub from the CentOs6.10/Xen4Centos
environment, I see


/usr/bin/pygrub --debug --offset=1048576 --list-entries
/dev/vg_xen_vbds/lv_4g_02
title: Ubuntu
root: None
kernel: /boot/vmlinuz-4.4.0-31-generic
args: root=/dev/xvda1 ro quiet splash $vt_handoff
initrd: /boot/initrd.img-4.4.0-31-generic
title: Ubuntu, with Linux 4.4.0-31-generic
root: None
kernel: /boot/vmlinuz-4.4.0-31-generic
args: root=UUID=5556a819-ced8-4864-9e7f-73792570703e ro quiet
splash $vt_handoff
initrd: /boot/initrd.img-4.4.0-31-generic
title: Ubuntu, with Linux 4.4.0-31-generic (recovery mode)
root: None
kernel: /boot/vmlinuz-4.4.0-31-generic
args: root=UUID=5556a819-ced8-4864-9e7f-73792570703e ro recovery nomodeset
initrd: /boot/initrd.img-4.4.0-31-generic
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /boot/grub/grub.cfg
INFO:root:Ignored directive if
INFO:root:Ignored directive load_env
INFO:root:Ignored directive fi
INFO:root:Ignored directive if
WARNING:root:grub2's saved_entry/next_entry not supported
INFO:root:Ignored directive save_env
...


and it goes on and suceeds.

Ok, so then I went hunting around for the cause of the error
I was seeing on the LFS system and got pointed towards the
cause being a Pythin2->3 issue, with the way 3 now differentiates
between strings and bytes.


FWIW, I tried making some changes to the GrubConf.py file but only
seemed to see more worms coming out of each can that I opened. The
first of my attempted changes was to try and decode the buffer but
that seems to have moved me off onto another error again, whilst the
second line of attack, where I tried to leave the buffer as "bytes"
whilst altering the operations on it to be "byte-aware" only
compilacted things further.

The odd things for me is that the same pygrub that can boot my
LFS system doesn't appear to be failing.

I present the chnages that I made and the resulting errors from
my attempts to point pygrub at the partition with the grub.cfg
file on here:

bash-5.0# /usr/lib/xen/bin/pygrub --debug --offset=1048576
--list-entries /dev/vg_xen_vbds/lv_4g_02
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /boot/grub/grub.cfg
Traceback (most recent call last):
File "/usr/lib/xen/bin/pygrub", line 907, in <module>
chosencfg = run_grub(file, entry, fs, incfg["args"])
File "/usr/lib/xen/bin/pygrub", line 625, in run_grub
g = Grub(file, fs)
File "/usr/lib/xen/bin/pygrub", line 249, in __init__
self.read_config(file, fs)
File "/usr/lib/xen/bin/pygrub", line 460, in read_config
self.cf.parse(buf)
File "/usr/lib/python3.7/site-packages/grub/GrubConf.py", line 376, in parse
lines = buf.split("\n")
TypeError: a bytes-like object is required, not 'str'

bash-5.0# vim /usr/lib/python3.7/site-packages/grub/GrubConf.py +376

bash-5.0# diff GrubConf.py /usr/lib/python3.7/site-packages/grub/GrubConf.py
376c376
< lines = buf.split("\n")
---
> lines = buf.decode().split("\n")


bash-5.0# /usr/lib/xen/bin/pygrub --debug --offset=1048576
--list-entries /dev/vg_xen_vbds/lv_4g_02
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /boot/grub/grub.cfg
INFO:root:Ignored directive if
INFO:root:Ignored directive load_env
INFO:root:Ignored directive fi
INFO:root:Ignored directive if
WARNING:root:grub2's saved_entry/next_entry not supported
Traceback (most recent call last):
File "/usr/lib/xen/bin/pygrub", line 907, in <module>
chosencfg = run_grub(file, entry, fs, incfg["args"])
File "/usr/lib/xen/bin/pygrub", line 625, in run_grub
g = Grub(file, fs)
File "/usr/lib/xen/bin/pygrub", line 249, in __init__
self.read_config(file, fs)
File "/usr/lib/xen/bin/pygrub", line 460, in read_config
self.cf.parse(buf)
File "/usr/lib/python3.7/site-packages/grub/GrubConf.py", line 440, in parse
setattr(self, self.commands[com], arg_strip)
File "/usr/lib/python3.7/site-packages/grub/GrubConf.py", line 238,
in _set_default
if self._default < 0:
TypeError: '<' not supported between instances of 'str' and 'int'


Hmmm?

So now try going back and try keeping things as "bytes"
for as long as possible


bash-5.0# vim /usr/lib/python3.7/site-packages/grub/GrubConf.py +376

bash-5.0# diff GrubConf.py /usr/lib/python3.7/site-packages/grub/GrubConf.py
376c376
< lines = buf.split("\n")
---
> lines = buf.split(b"\n")

bash-5.0# /usr/lib/xen/bin/pygrub --debug --offset=1048576
--list-entries /dev/vg_xen_vbds/lv_4g_02
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /boot/grub/grub.cfg
Traceback (most recent call last):
File "/usr/lib/xen/bin/pygrub", line 907, in <module>
chosencfg = run_grub(file, entry, fs, incfg["args"])
File "/usr/lib/xen/bin/pygrub", line 625, in run_grub
g = Grub(file, fs)
File "/usr/lib/xen/bin/pygrub", line 249, in __init__
self.read_config(file, fs)
File "/usr/lib/xen/bin/pygrub", line 460, in read_config
self.cf.parse(buf)
File "/usr/lib/python3.7/site-packages/grub/GrubConf.py", line 388, in parse
if l.startswith('#'):
TypeError: startswith first arg must be bytes or a tuple of bytes, not str


bash-5.0# vim /usr/lib/python3.7/site-packages/grub/GrubConf.py +388

bash-5.0# diff GrubConf.py /usr/lib/python3.7/site-packages/grub/GrubConf.py
376c376
< lines = buf.split("\n")
---
> lines = buf.split(b"\n")
388c388
< if l.startswith('#'):
---
> if l.startswith(b'#'):


bash-5.0# /usr/lib/xen/bin/pygrub --debug --offset=1048576
--list-entries /dev/vg_xen_vbds/lv_4g_02
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /boot/grub/grub.cfg
Traceback (most recent call last):
File "/usr/lib/xen/bin/pygrub", line 907, in <module>
chosencfg = run_grub(file, entry, fs, incfg["args"])
File "/usr/lib/xen/bin/pygrub", line 625, in run_grub
g = Grub(file, fs)
File "/usr/lib/xen/bin/pygrub", line 249, in __init__
self.read_config(file, fs)
File "/usr/lib/xen/bin/pygrub", line 460, in read_config
self.cf.parse(buf)
File "/usr/lib/python3.7/site-packages/grub/GrubConf.py", line 392, in parse
if l.startswith('function'):
TypeError: startswith first arg must be bytes or a tuple of bytes, not str


bash-5.0# vim /usr/lib/python3.7/site-packages/grub/GrubConf.py +392


bash-5.0# diff GrubConf.py /usr/lib/python3.7/site-packages/grub/GrubConf.py
376c376
< lines = buf.split("\n")
---
> lines = buf.split(b"\n")
388c388
< if l.startswith('#'):
---
> if l.startswith(b'#'):
392c392
< if l.startswith('function'):
---
> if l.startswith(b'function'):

bash-5.0# /usr/lib/xen/bin/pygrub --debug --offset=1048576
--list-entries /dev/vg_xen_vbds/lv_4g_02
Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /boot/grub/grub.cfg
Traceback (most recent call last):
File "/usr/lib/xen/bin/pygrub", line 907, in <module>
chosencfg = run_grub(file, entry, fs, incfg["args"])
File "/usr/lib/xen/bin/pygrub", line 625, in run_grub
g = Grub(file, fs)
File "/usr/lib/xen/bin/pygrub", line 249, in __init__
self.read_config(file, fs)
File "/usr/lib/xen/bin/pygrub", line 460, in read_config
self.cf.parse(buf)
File "/usr/lib/python3.7/site-packages/grub/GrubConf.py", line 401, in parse
title_match = re.match('^menuentry ["\'](.*?)["\'] (.*){', l)
File "/usr/lib/python3.7/re.py", line 173, in match
return _compile(pattern, flags).match(string)
TypeError: cannot use a string pattern on a bytes-like object


Hmmm?

This looks like something a bit harder to transpose into
a "bytes" paradigm.

Any ideas as to why my pythin3 Grub works in some cases but not
in others?

Kevin Buckley.

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: pygrub, installed with Python 3, doesn't boot Xen DomU-s [ In reply to ]
On 22/07/2019 03:57, Kevin Buckley wrote:
> This follows on from
>
> pygrub gives "raise RuntimeError("Unable to find partition containing kernel")"
>
> https://lists.xenproject.org/archives/html/xen-devel/2019-05/msg01589.html
>
> and for some reason I submitted my latest findings onto the grub-devel list,
> only to be reminded that pygrub is part of Xen, not Grub, so reposting here:
>
>
> OK, so I am trying to build an LFS (Linux From Scratch) system that
> will serve as a basic Xen Dom0.
>
> You can see the way things have been compiled, including my UEFI-aware
> Grub, here (Very much a WIP)
>
> http://youvegotbuckleys.org.nz/LFS/LFS-BOOK.html

Having a quick read through, I think the pkgcfg issue mentioned was
fixed by
https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=a0f981451509bf83af153038d35079c4e62dc1cf

If you pull up to a newer version of master, it ought to work as you expect.

>
>
> Suffice it to say that the LFS Grub boots either of the two
> UEFI enties (non-Xen and Dom0) I have so as to bring up the
> system.
>
>
> I also have a couple of VBD-backed DomU-s that I can bring up,
> using the pygrub from either an Ubuntu 1404, or a Centos 6.10
> with Xen4Centos, Dom0.
>
> When I come to try and have my LFS Xen's pygrub boot the VBD-backed
> DomU, I get errors of the form
>
>
> # cat /mnt/var/log/xen/bootloader.4.log
> Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /boot/grub/grub.cfg
> Traceback (most recent call last):
> File "/usr/lib/xen/bin/pygrub", line 928, in <module>
> raise RuntimeError("Unable to find partition containing kernel")
> RuntimeError: Unable to find partition containing kernel
>
> I originally thought that this might be down to my using a Xen
> source from the Xen Git repo that was in beyond 4.12 (which I
> needed so to get a Xen that was happy with just Python3. See
> https://lists.xenproject.org/archives/html/xen-devel/2019-04/msg00996.html
> ) and had prhaps had tripped over a regression, as that error used
> to be a problem with Xen (if you go back far enough).
>
>
> However I have since tracked down my problem to the pygrub, in
> that if I point pygrub to the start of the partition in the VBD
> that has the grub.cfg in it, I see the following failure to parse
> the config file:
>
>
> bash-5.0# /usr/lib/xen/bin/pygrub --debug --offset=1048576
> --list-entries /dev/vg_xen_vbds/lv_4g_02
> Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /boot/grub/grub.cfg
> Traceback (most recent call last):
> File "/usr/lib/xen/bin/pygrub", line 907, in <module>
> chosencfg = run_grub(file, entry, fs, incfg["args"])
> File "/usr/lib/xen/bin/pygrub", line 625, in run_grub
> g = Grub(file, fs)
> File "/usr/lib/xen/bin/pygrub", line 249, in __init__
> self.read_config(file, fs)
> File "/usr/lib/xen/bin/pygrub", line 460, in read_config
> self.cf.parse(buf)
> File "/usr/lib/python3.7/site-packages/grub/GrubConf.py", line 376, in parse
> lines = buf.split("\n")
> TypeError: a bytes-like object is required, not 'str'
>
>
> whereas if I do the same with the pygrub from the CentOs6.10/Xen4Centos
> environment, I see
>
>
> /usr/bin/pygrub --debug --offset=1048576 --list-entries
> /dev/vg_xen_vbds/lv_4g_02
> title: Ubuntu
> root: None
> kernel: /boot/vmlinuz-4.4.0-31-generic
> args: root=/dev/xvda1 ro quiet splash $vt_handoff
> initrd: /boot/initrd.img-4.4.0-31-generic
> title: Ubuntu, with Linux 4.4.0-31-generic
> root: None
> kernel: /boot/vmlinuz-4.4.0-31-generic
> args: root=UUID=5556a819-ced8-4864-9e7f-73792570703e ro quiet
> splash $vt_handoff
> initrd: /boot/initrd.img-4.4.0-31-generic
> title: Ubuntu, with Linux 4.4.0-31-generic (recovery mode)
> root: None
> kernel: /boot/vmlinuz-4.4.0-31-generic
> args: root=UUID=5556a819-ced8-4864-9e7f-73792570703e ro recovery nomodeset
> initrd: /boot/initrd.img-4.4.0-31-generic
> Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /boot/grub/grub.cfg
> INFO:root:Ignored directive if
> INFO:root:Ignored directive load_env
> INFO:root:Ignored directive fi
> INFO:root:Ignored directive if
> WARNING:root:grub2's saved_entry/next_entry not supported
> INFO:root:Ignored directive save_env
> ...
>
>
> and it goes on and suceeds.
>
> Ok, so then I went hunting around for the cause of the error
> I was seeing on the LFS system and got pointed towards the
> cause being a Pythin2->3 issue, with the way 3 now differentiates
> between strings and bytes.

Yes - this looks like a Py 2/3 compatibility issue.  This particular one
is related to
https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=ff915c8cacc264ae1380d51fea07267b8308d7ba

However, I can't explain why python is complaining at str.split(). 
split() is a string operation, not a bytes operation.

~Andrew

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: pygrub, installed with Python 3, doesn't boot Xen DomU-s [ In reply to ]
On Mon, 22 Jul 2019, Andrew Cooper wrote:

> On 22/07/2019 03:57, Kevin Buckley wrote:
>> bash-5.0# /usr/lib/xen/bin/pygrub --debug --offset=1048576
>> --list-entries /dev/vg_xen_vbds/lv_4g_02
>> Using <class 'grub.GrubConf.Grub2ConfigFile'> to parse /boot/grub/grub.cfg
>> Traceback (most recent call last):
>> File "/usr/lib/xen/bin/pygrub", line 907, in <module>
>> chosencfg = run_grub(file, entry, fs, incfg["args"])
>> File "/usr/lib/xen/bin/pygrub", line 625, in run_grub
>> g = Grub(file, fs)
>> File "/usr/lib/xen/bin/pygrub", line 249, in __init__
>> self.read_config(file, fs)
>> File "/usr/lib/xen/bin/pygrub", line 460, in read_config
>> self.cf.parse(buf)
>> File "/usr/lib/python3.7/site-packages/grub/GrubConf.py", line 376, in parse
>> lines = buf.split("\n")
>> TypeError: a bytes-like object is required, not 'str'
>>
>> ...
>>
>> Ok, so then I went hunting around for the cause of the error
>> I was seeing on the LFS system and got pointed towards the
>> cause being a Pythin2->3 issue, with the way 3 now differentiates
>> between strings and bytes.
>
> Yes - this looks like a Py 2/3 compatibility issue.  This particular one
> is related to
> https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=ff915c8cacc264ae1380d51fea07267b8308d7ba
>
> However, I can't explain why python is complaining at str.split(). 
> split() is a string operation, not a bytes operation.

It looks like you are missing the above patch and I think you also need
https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=485079e816788d70169f45579e1f5a8f909dc1b3
https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=767ba397d34848c7e0c4e9cdfc5efa4e0cb61442
and perhaps
https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=a57a1b26ec0ae31f924cf2bbcf479637d007be44
if you build with ocaml. Actually the first of these 3 patches is optional
as it is just an error message change but should allow the second patch to
apply cleanly.

Michael Young
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel
Re: pygrub, installed with Python 3, doesn't boot Xen DomU-s [ In reply to ]
On Tue, 23 Jul 2019 at 03:43, YOUNG, MICHAEL A. <m.a.young@durham.ac.uk> wrote:

> > Yes - this looks like a Py 2/3 compatibility issue. This particular one
> > is related to
> > https://xenbits.xen.org/gitweb/?p=xen.git;a=commitdiff;h=ff915c8cacc264ae1380d51fea07267b8308d7ba
> >
> > However, I can't explain why python is complaining at str.split().
> > split() is a string operation, not a bytes operation.
>
> It looks like you are missing the above patch and I think you also need
> https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=485079e816788d70169f45579e1f5a8f909dc1b3
> https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=767ba397d34848c7e0c4e9cdfc5efa4e0cb61442
> and perhaps
> https://xenbits.xen.org/gitweb/?p=xen.git;a=commit;h=a57a1b26ec0ae31f924cf2bbcf479637d007be44
> if you build with ocaml. Actually the first of these 3 patches is optional
> as it is just an error message change but should allow the second patch to
> apply cleanly.
>
> Michael Young

Just to say that, over the weekend, I rebuilt things using Xen commit f7f7cd5
and I now have a Xen environment that boots my existing Dom0s.

Thanks for all the pointers to the ongoing work as regards pythin2->3.

Kevin

_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xenproject.org
https://lists.xenproject.org/mailman/listinfo/xen-devel