Mailing List Archive

failed to spawn?
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1




I would really appreciate if someone can point me to the right
direction. I already spent few hours with this but can't seem to resolve
the issue. I have checked all the virtualization/libvirtd options in the
bios and the node. All seems to be ok there. My hunch is that my glance
setup is not correct (until today, I didn't realize that there was a
/etc/glance/*.conf).

I am using RHEL 6.1

from failed to spawn it goes to shutdown.

This is from the nova-compute.log from the compute node:


2011-07-13 22:36:23,168 nova.compute.manager: AUDIT
[KIO6M5FGWOQXGS0EVPW8 sharif myproject] instance 32: starting...
2011-07-13 22:36:24,795 nova: called setup_basic_filtering in nwfilter
2011-07-13 22:36:24,795 nova: ensuring static filters
2011-07-13 22:36:25,076 nova.virt.libvirt_conn: instance
instance-00000020: Creating image
2011-07-13 22:36:25,095 nova.exception: Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File
"/usr/lib/python2.6/site-packages/nova/exception.py", line 87, in _wrap
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/connection.py", line
590, in spawn
(nova.exception): TRACE: block_device_mapping=block_device_mapping)
(nova.exception): TRACE: File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/connection.py", line
845, in _create_image
(nova.exception): TRACE: size=size)
(nova.exception): TRACE: File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/connection.py", line
751, in _cache_image
(nova.exception): TRACE: call_if_not_exists(base, fn, *args, **kwargs)
(nova.exception): TRACE: File
"/usr/lib/python2.6/site-packages/nova/utils.py", line 605, in inner
(nova.exception): TRACE: retval = f(*args, **kwargs)
(nova.exception): TRACE: File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/connection.py", line
749, in call_if_not_exists
(nova.exception): TRACE: fn(target=base, *args, **kwargs)
(nova.exception): TRACE: File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/connection.py", line
762, in _fetch_image
(nova.exception): TRACE: images.fetch(image_id, target, user, project)
(nova.exception): TRACE: File
"/usr/lib/python2.6/site-packages/nova/virt/images.py", line 44, in fetch
(nova.exception): TRACE: metadata = image_service.get(elevated,
image_id, image_file)
(nova.exception): TRACE: File
"/usr/lib/python2.6/site-packages/nova/image/glance.py", line 139, in get
(nova.exception): TRACE: image_meta, image_chunks =
self.client.get_image(image_id)
(nova.exception): TRACE: File
"/usr/lib/python2.6/site-packages/glance/client.py", line 224, in get_image
(nova.exception): TRACE: res = self.do_request("GET", "/images/%s" %
image_id)
(nova.exception): TRACE: File
"/usr/lib/python2.6/site-packages/glance/client.py", line 167, in do_request
(nova.exception): TRACE: "server. Got error: %s" % e)
(nova.exception): TRACE: ClientConnectionError: Unable to connect to
server. Got error: [Errno 111] ECONNREFUSED
(nova.exception): TRACE:
2011-07-13 22:36:25,096 nova.compute.manager: Instance '32' failed to
spawn. Is virtualization enabled in the BIOS? Details: Unable to connect
to server. Got error: [Errno 111] ECONNREFUSED
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 323, in
_run_instance
(nova.compute.manager): TRACE:
block_device_mapping=block_device_mapping)
(nova.compute.manager): TRACE: File
"/usr/lib/python2.6/site-packages/nova/exception.py", line 93, in _wrap
(nova.compute.manager): TRACE: raise Error(str(e))
(nova.compute.manager): TRACE: Error: Unable to connect to server. Got
error: [Errno 111] ECONNREFUSED
(nova.compute.manager): TRACE:
2011-07-13 22:37:02,254 nova.compute.manager: Updating host status
2011-07-13 22:37:02,280 nova.compute.manager: Found instance
'instance-00000020' in DB but no VM. State=8, so setting state to shutoff.
2011-07-13 22:37:02,280 nova.compute.manager: DB/VM state mismatch.
Changing state from '8' to '5'


thanks. let me know if you need more information about the setup here.

- --sharif
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJOHlkBAAoJEACffes9SivF/DcIAJNOiQbsdmzP7tT8CSjIkYHc
zqchpLCIq/W9ypGNg4ISNQnyWdoBIUmSxfbKleA1kVq/bkzihV8JjEQ4qcyGGkzR
7TqY2+tipA8nm3UD289ll/7cX3QvZFhdX/qbMpi0AePvHtQTTUs5iGUOQ1KtGbHT
Scp5G/GofYjakZgWoDFyAlbYQH6FRF4mCKZ3rVO9Ir7sU1Xm1FD52SMQzVqb7+9H
6h2NbYaI+4Zh/ZqnAJG5HRRk3j+blJq5vhAkmSRSIV9copxUbqJ9a9DAYyFWCMD6
blZMMq2JZiwP6+3TOE4kwVanNagvCDERTHYr5E1V1ifASswGnKthQIELRpRWVQ4=
=5WGU
-----END PGP SIGNATURE-----
failed to spawn? [ In reply to ]
Where's your glance server located? On the same node ? Can you telnet the
glance API port to see if connects ? TCP 9292 , if that fail it seems like
your glance service is down ...

Regards
Lele

On Wed, Jul 13, 2011 at 11:48 PM, Sharif Islam <islamsh at indiana.edu> wrote:

> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
>
>
>
> I would really appreciate if someone can point me to the right
> direction. I already spent few hours with this but can't seem to resolve
> the issue. I have checked all the virtualization/libvirtd options in the
> bios and the node. All seems to be ok there. My hunch is that my glance
> setup is not correct (until today, I didn't realize that there was a
> /etc/glance/*.conf).
>
> I am using RHEL 6.1
>
> from failed to spawn it goes to shutdown.
>
> This is from the nova-compute.log from the compute node:
>
>
> 2011-07-13 22:36:23,168 nova.compute.manager: AUDIT
> [KIO6M5FGWOQXGS0EVPW8 sharif myproject] instance 32: starting...
> 2011-07-13 22:36:24,795 nova: called setup_basic_filtering in nwfilter
> 2011-07-13 22:36:24,795 nova: ensuring static filters
> 2011-07-13 22:36:25,076 nova.virt.libvirt_conn: instance
> instance-00000020: Creating image
> 2011-07-13 22:36:25,095 nova.exception: Uncaught exception
> (nova.exception): TRACE: Traceback (most recent call last):
> (nova.exception): TRACE: File
> "/usr/lib/python2.6/site-packages/nova/exception.py", line 87, in _wrap
> (nova.exception): TRACE: return f(*args, **kw)
> (nova.exception): TRACE: File
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/connection.py", line
> 590, in spawn
> (nova.exception): TRACE: block_device_mapping=block_device_mapping)
> (nova.exception): TRACE: File
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/connection.py", line
> 845, in _create_image
> (nova.exception): TRACE: size=size)
> (nova.exception): TRACE: File
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/connection.py", line
> 751, in _cache_image
> (nova.exception): TRACE: call_if_not_exists(base, fn, *args, **kwargs)
> (nova.exception): TRACE: File
> "/usr/lib/python2.6/site-packages/nova/utils.py", line 605, in inner
> (nova.exception): TRACE: retval = f(*args, **kwargs)
> (nova.exception): TRACE: File
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/connection.py", line
> 749, in call_if_not_exists
> (nova.exception): TRACE: fn(target=base, *args, **kwargs)
> (nova.exception): TRACE: File
> "/usr/lib/python2.6/site-packages/nova/virt/libvirt/connection.py", line
> 762, in _fetch_image
> (nova.exception): TRACE: images.fetch(image_id, target, user, project)
> (nova.exception): TRACE: File
> "/usr/lib/python2.6/site-packages/nova/virt/images.py", line 44, in fetch
> (nova.exception): TRACE: metadata = image_service.get(elevated,
> image_id, image_file)
> (nova.exception): TRACE: File
> "/usr/lib/python2.6/site-packages/nova/image/glance.py", line 139, in get
> (nova.exception): TRACE: image_meta, image_chunks =
> self.client.get_image(image_id)
> (nova.exception): TRACE: File
> "/usr/lib/python2.6/site-packages/glance/client.py", line 224, in get_image
> (nova.exception): TRACE: res = self.do_request("GET", "/images/%s" %
> image_id)
> (nova.exception): TRACE: File
> "/usr/lib/python2.6/site-packages/glance/client.py", line 167, in
> do_request
> (nova.exception): TRACE: "server. Got error: %s" % e)
> (nova.exception): TRACE: ClientConnectionError: Unable to connect to
> server. Got error: [Errno 111] ECONNREFUSED
> (nova.exception): TRACE:
> 2011-07-13 22:36:25,096 nova.compute.manager: Instance '32' failed to
> spawn. Is virtualization enabled in the BIOS? Details: Unable to connect
> to server. Got error: [Errno 111] ECONNREFUSED
> (nova.compute.manager): TRACE: Traceback (most recent call last):
> (nova.compute.manager): TRACE: File
> "/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 323, in
> _run_instance
> (nova.compute.manager): TRACE:
> block_device_mapping=block_device_mapping)
> (nova.compute.manager): TRACE: File
> "/usr/lib/python2.6/site-packages/nova/exception.py", line 93, in _wrap
> (nova.compute.manager): TRACE: raise Error(str(e))
> (nova.compute.manager): TRACE: Error: Unable to connect to server. Got
> error: [Errno 111] ECONNREFUSED
> (nova.compute.manager): TRACE:
> 2011-07-13 22:37:02,254 nova.compute.manager: Updating host status
> 2011-07-13 22:37:02,280 nova.compute.manager: Found instance
> 'instance-00000020' in DB but no VM. State=8, so setting state to shutoff.
> 2011-07-13 22:37:02,280 nova.compute.manager: DB/VM state mismatch.
> Changing state from '8' to '5'
>
>
> thanks. let me know if you need more information about the setup here.
>
> - --sharif
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.9 (Darwin)
> Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
>
> iQEcBAEBAgAGBQJOHlkBAAoJEACffes9SivF/DcIAJNOiQbsdmzP7tT8CSjIkYHc
> zqchpLCIq/W9ypGNg4ISNQnyWdoBIUmSxfbKleA1kVq/bkzihV8JjEQ4qcyGGkzR
> 7TqY2+tipA8nm3UD289ll/7cX3QvZFhdX/qbMpi0AePvHtQTTUs5iGUOQ1KtGbHT
> Scp5G/GofYjakZgWoDFyAlbYQH6FRF4mCKZ3rVO9Ir7sU1Xm1FD52SMQzVqb7+9H
> 6h2NbYaI+4Zh/ZqnAJG5HRRk3j+blJq5vhAkmSRSIV9copxUbqJ9a9DAYyFWCMD6
> blZMMq2JZiwP6+3TOE4kwVanNagvCDERTHYr5E1V1ifASswGnKthQIELRpRWVQ4=
> =5WGU
> -----END PGP SIGNATURE-----
> _______________________________________________
> Openstack-operators mailing list
> Openstack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20110714/1b3dae4f/attachment.html>
failed to spawn? [ In reply to ]
On 07/14/2011 08:20 AM, Leandro Reox wrote:
> Where's your glance server located? On the same node ? Can you telnet
> the glance API port to see if connects ? TCP 9292 ,


Yes. I can telnet to both 9292 and 9191 from the compute nodes.


here the services running in the controller node (which has glance api
and registry running):

# ps -eaf|grep glance
glance 989 1 0 09:35 ? 00:00:00 /usr/bin/python
/usr/bin/glance-api -d -v --log-config=/etc/glance/logging-api.conf
glance 1006 1 0 09:35 ? 00:00:00 /usr/bin/python
/usr/bin/glance-registry -d -v
--log-config=/etc/glance/logging-registry.conf




The REST API works as well:

{"images": [.{"name": null, "container_format": "ami", "disk_format":
"ami", "checksum": "b7207b3f7d2c512ce835835008d74560", "id": 7, "size":
1476395008}, {"name": null, "container_format": "aki", "disk_format":
"aki", "checksum": "65912c22ce14a2886cbe2bf40243df78", "id": 6, "size":
4407632}, {"name": null, "container_format": "ari", "disk_format":
"ari", "checksum": "8106433c49662a8b84c47c5f411294bf", "id": 5, "size":
91708}]}

However, in the controller node, this happens:

# glance details
Failed to show details. Got error:
Unable to connect to server. Got error: [Errno 111] ECONNREFUSE


I used euca-bundle-image to register and upload images, didn't use
glance. Could that be the issue here?

# euca-describe-images
IMAGE ami-00000007
ubuntu_bucket/maverick-server-uec-amd64.img.manifest.xml
available public i386 machine aki-00000006
IMAGE aki-00000006
ubuntu_bucket/maverick-server-uec-amd64-vmlinuz-virtual.manifest.xml
available public i386 kernel
IMAGE ari-00000005
ramdisk-bucket/maverick-server-uec-amd64-loader.manifest.xml
available public x86_64 ramdisk


--sharif


> if that fail it
> seems like your glance service is down ...
>
___
failed to spawn? [ In reply to ]
I was able to fix my glance config. So that is now working. However, i
am still getting the failed to spawn error.

this is from the controller:

# glance index
Found 3 public images...
ID Name Disk Format
Container Format Size
---------------- ------------------------------ --------------------
-------------------- --------------
20 None ami ami
25165824
19 None ari ari
5882349
18 None aki aki
4404752
[root at i26 ~]# glance details
Found 3 public images...
================================================================================
URI: http://0.0.0.0/images/20
Id: 20
Public: Yes
Name: None
Status: active
Size: 25165824
Location: file:///var/lib/glance/images/20
Disk format: ami
Container format: ami
Property 'image_location':
ramdisk-bucket/ttylinux-uec-amd64-12.1_2.6.35-22_1.img.manifest.xml
Property 'image_state': available
Property 'kernel_id': 18
Property 'ramdisk_id': 19
Property 'architecture': x86_64
Property 'project_id': fgadmin
================================================================================
URI: http://0.0.0.0/images/19
Id: 19
Public: Yes
Name: None
Status: active
Size: 5882349
Location: file:///var/lib/glance/images/19
Disk format: ari
Container format: ari
Property 'image_location':
ramdisk-bucket/ttylinux-uec-amd64-12.1_2.6.35-22_1-initrd.manifest.xml
Property 'image_state': available
Property 'project_id': fgadmin
Property 'architecture': x86_64
================================================================================
URI: http://0.0.0.0/images/18
Id: 18
Public: Yes
Name: None
Status: active
Size: 4404752
Location: file:///var/lib/glance/images/18
Disk format: aki
Container format: aki
Property 'image_location':
kernel-bucket/ttylinux-uec-amd64-12.1_2.6.35-22_1-vmlinuz.manifest.xml
Property 'image_state': available
Property 'project_id': fgadmin
Property 'architecture': x86_64
================================================================================



from the compute node log:

2011-07-14 19:43:55,585 nova.compute.manager: AUDIT
[A8UEQDDOMY0QIXJW8H4G sharif fgadmin] instance 112: starting...
2011-07-14 19:43:56,403 nova: called setup_basic_filtering in nwfilter
2011-07-14 19:43:56,403 nova: ensuring static filters
2011-07-14 19:43:56,571 nova.virt.libvirt_conn: instance
instance-00000070: Creating image
2011-07-14 19:43:56,586 nova.exception: Uncaught exception
(nova.exception): TRACE: Traceback (most recent call last):
(nova.exception): TRACE: File
"/usr/lib/python2.6/site-packages/nova/exception.py", line 87, in _wrap
(nova.exception): TRACE: return f(*args, **kw)
(nova.exception): TRACE: File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/connection.py", line
590, in spawn
(nova.exception): TRACE: block_device_mapping=block_device_mapping)
(nova.exception): TRACE: File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/connection.py", line
817, in _create_image
(nova.exception): TRACE: project=project)
(nova.exception): TRACE: File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/connection.py", line
751, in _cache_image
(nova.exception): TRACE: call_if_not_exists(base, fn, *args, **kwargs)
(nova.exception): TRACE: File
"/usr/lib/python2.6/site-packages/nova/utils.py", line 605, in inner
(nova.exception): TRACE: retval = f(*args, **kwargs)
(nova.exception): TRACE: File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/connection.py", line
749, in call_if_not_exists
(nova.exception): TRACE: fn(target=base, *args, **kwargs)
(nova.exception): TRACE: File
"/usr/lib/python2.6/site-packages/nova/virt/libvirt/connection.py", line
762, in _fetch_image
(nova.exception): TRACE: images.fetch(image_id, target, user, project)
(nova.exception): TRACE: File
"/usr/lib/python2.6/site-packages/nova/virt/images.py", line 44, in fetch
(nova.exception): TRACE: metadata = image_service.get(elevated,
image_id, image_file)
(nova.exception): TRACE: File
"/usr/lib/python2.6/site-packages/nova/image/glance.py", line 139, in get
(nova.exception): TRACE: image_meta, image_chunks =
self.client.get_image(image_id)
(nova.exception): TRACE: File
"/usr/lib/python2.6/site-packages/glance/client.py", line 224, in get_image
(nova.exception): TRACE: res = self.do_request("GET", "/images/%s" %
image_id)
(nova.exception): TRACE: File
"/usr/lib/python2.6/site-packages/glance/client.py", line 167, in do_request
(nova.exception): TRACE: "server. Got error: %s" % e)
(nova.exception): TRACE: ClientConnectionError: Unable to connect to
server. Got error: [Errno 111] ECONNREFUSED
(nova.exception): TRACE:
2011-07-14 19:43:56,587 nova.compute.manager: Instance '112' failed to
spawn. Is virtualization enabled in the BIOS? Details: Unable to connect
to server. Got error: [Errno 111] ECONNREFUSED
(nova.compute.manager): TRACE: Traceback (most recent call last):
(nova.compute.manager): TRACE: File
"/usr/lib/python2.6/site-packages/nova/compute/manager.py", line 323, in
_run_instance
(nova.compute.manager): TRACE:
block_device_mapping=block_device_mapping)
(nova.compute.manager): TRACE: File
"/usr/lib/python2.6/site-packages/nova/exception.py", line 93, in _wrap
(nova.compute.manager): TRACE: raise Error(str(e))
(nova.compute.manager): TRACE: Error: Unable to connect to server. Got
error: [Errno 111] ECONNREFUSED
(nova.compute.manager): TRACE:
2011-07-14 19:44:01,069 nova.compute.manager: Updating host status
2011-07-14 19:44:01,096 nova.compute.manager: Found instance
'instance-00000070' in DB but no VM. State=8, so setting state to shutoff.
2011-07-14 19:44:01,096 nova.compute.manager: DB/VM state mismatch.
Changing state from '8' to '5'


I can telnet to the glance server fine:

[root at i38 ~]# telnet xx.xx.xxxxx 9191
Trying xx.xx.xxxxx..
Connected to xx.xx.xxxxx.
Escape character is '^]'.
TEST
<head>
<title>Error response</title>
</head>
<body>
<h1>Error response</h1>
<p>Error code 400.
<p>Message: Bad request syntax ('TEST').
<p>Error code explanation: 400 = Bad request syntax or unsupported method.
</body>
Connection closed by foreign host.



--sharif




On 07/14/2011 08:20 AM, Leandro Reox wrote:
> Where's your glance server located? On the same node ? Can you telnet
> the glance API port to see if connects ? TCP 9292 , if that fail it
> seems like your glance service is down ...
>
> Regards
> Lele
>
failed to spawn? [ In reply to ]
> I was able to fix my glance config. So that is now working. However, i
> am still getting the failed to spawn error.

Sharif,

Thank you very much for cooperation on IRC yesterday.
We run into an issue also.
You need to upgrade to last packages (yum update) to fix that.

Or simply fix your /etc/nova/nova.conf according to our blog post:
http://openstackgd.wordpress.com/2011/07/14/glance-flags-for-nova/

Our technical blog is also mirrored on http://planet.openstack.org

> ramdisk-bucket/ttylinux-uec-amd64-12.1_2.6.35-22_1.img.manifest.xml

You probably can run into an issue using Ubuntu-based image here.
Especially if you need to inject networking settings into an instance.

Right now code working this way:
RHEL-based OpenStack (Grid Dynamics build) is trying to inject to /etc/sysconfig/networking-scripts/ifcfg-eth0 using libguestfs and Ubuntu-based OpenStack is trying to inject to /etc/network/interfaces using qemu-nbd.

That basically means you need RHEL-based OpenStack Nova to run RHEL-based images (RHEL, CentOS, Fedora) and Ubuntu-based OpenStack Nova to run Ubuntu- and Debian-based images.

We plan to implement instance's OS at the runtime utilizing libguestfs features and inject configs to proper place.

Andrey Brindeyev
Grid Dynamics, Development Manager
abrindeyev at griddynamics.com