OpenStack has a service called metadata which is implemented by either the nova-api service or the nova-api-metadata service. This metadata service enables your Virtual Machine instances to retrieve instance-specific information on a link-local IPv4 address 169.254.169.254.
The instance sends an HTTP request the link-local IP address. A metadata service agent will get the request and proxy it to a Nova service, and gives the response back to the VM instance.
Metadata service allows your instances to get additional configuration information such as:
- Public hostname
- Public IP
- SSH public key
- cloud-init script
- Random seed
- user-data provided to nova boot invocation
- Static routing information
The nova-api-metadata service is only used in multi-host OpenStack deployment as it retrieves instance-specific metadata. For single node OpenStack installations, the nova-api service is commonly used.
Configure metadata service in Nova API
Check your Nova configuration file if Metadata service is enabled.
$ sudo vim /etc/nova/nova.conf
metadata_listen=0.0.0.0
metadata_listen_port=8775
service_metadata_proxy=True
metadata_proxy_shared_secret=ecf4fcffd196495hIf you make any modifications, a restart of nova-api service must be performed.
sudo systemctl restart openstack-nova-api.serviceTo check the status of Nova API service, run the following commands.
systemctl status openstack-nova-api.serviceYou can also see what’s listening on port 8775, this should be nova-api.
$ sudo ss -tunelp |grep 8775
tcp LISTEN 0 128 0.0.0.0:8775 0.0.0.0:* users:(("nova-api",pid=49816,fd=12),("nova-api",pid=49816,fd=10),("nova-api",pid=49814,fd=12),("nova-api",pid=49814,fd=10),("nova-api",pid=49813,fd=12),("nova-api",pid=49813,fd=10),("nova-api",pid=49811,fd=12),("nova-api",pid=49811,fd=10),("nova-api",pid=49810,fd=12),("nova-api",pid=49810,fd=10),("nova-api",pid=49809,fd=12),("nova-api",pid=49809,fd=10),("nova-api",pid=49808,fd=12),("nova-api",pid=49808,fd=10),("nova-api",pid=49807,fd=12),("nova-api",pid=49807,fd=10),("nova-api",pid=49806,fd=12),("nova-api",pid=49806,fd=10),("nova-api",pid=49805,fd=12),("nova-api",pid=49805,fd=10),("nova-api",pid=49804,fd=12),("nova-api",pid=49804,fd=10),("nova-api",pid=49803,fd=12),("nova-api",pid=49803,fd=10),("nova-api",pid=49784,fd=10)) uid:162 ino:1491838 sk:4002 cgroup:/system.slice/openstack-nova-api.service <->Check instance logs if cannot get to 169.254.169.254
We need to start our instance on OpenStack and check its logs if there is failed connection to http://169.254.169.254/openstack when it tries to get cloud-init data.
Use the openstack console log show command to see your instance console boot logs. Rocky-Linux-9 should be replaced with your instance name.
openstack console log show Rocky-Linux-9From our output we can clearly see “Failed to establish a new connection: [Errno 113] No route to host’))”)] during request to http://169.254.169.254/openstack, raising last exception“.
....
[ 9.286330] cloud-init[793]: ci-info: +++++++++++++++++++Route IPv6 info+++++++++++++++++++
[ 9.287283] cloud-init[793]: ci-info: +-------+-------------+---------+-----------+-------+
[ 9.288233] cloud-init[793]: ci-info: | Route | Destination | Gateway | Interface | Flags |
[ 9.289184] cloud-init[793]: ci-info: +-------+-------------+---------+-----------+-------+
[ 9.290080] cloud-init[793]: ci-info: | 1 | fe80::/64 | :: | eth0 | U |
[ 9.290978] cloud-init[793]: ci-info: | 3 | multicast | :: | eth0 | U |
[ 9.291868] cloud-init[793]: ci-info: +-------+-------------+---------+-----------+-------+
[ 12.518923] cloud-init[793]: 2024-06-19 22:08:37,674 - url_helper.py[WARNING]: Exception(s) [UrlError("HTTPConnectionPool(host='fe80::a9fe:a9fe', port=80): Max retries exceeded with url: /openstack (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fbe9fe5ea90>: Failed to establish a new connection: [Errno 22] Invalid argument'))"), UrlError("HTTPConnectionPool(host='169.254.169.254', port=80): Max retries exceeded with url: /openstack (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fbe9ecf0640>: Failed to establish a new connection: [Errno 113] No route to host'))")] during request to http://169.254.169.254/openstack, raising last exception
[ 12.525218] cloud-init[793]: 2024-06-19 22:08:37,675 - url_helper.py[ERROR]: Timed out, no response from urls: ['http://[fe80::a9fe:a9fe]/openstack', 'http://169.254.169.254/openstack']
[ 12.526960] cloud-init[793]: 2024-06-19 22:08:37,675 - util.py[WARNING]: No active metadata service found
[ 14.341877] cloud-init[793]: Generating public/private rsa key pair.
[ 14.342775] cloud-init[793]: Your identification has been saved in /etc/ssh/ssh_host_rsa_key
[ 14.343794] cloud-init[793]: Your public key has been saved in /etc/ssh/ssh_host_rsa_key.pub
[ 14.344947] cloud-init[793]: The key fingerprint is:
[ 14.345572] cloud-init[793]: SHA256:tKbcYg40AUytwHNSbLeMk44y87xQYdPPYLgfCL+3XDs root@host-188-40-183-240
[ 14.346677] cloud-init[793]: The key's randomart image is:
[ 14.347371] cloud-init[793]: +---[RSA 3072]----+
[ 14.347965] cloud-init[793]: |.o=o |
....Fix failed connection to http://169.254.169.254/openstack
To fix this issue, we need to configure Neutron DHCP server to assist with providing metadata support on isolated networks. The DHCP server will append specific host routes to DHCP request.
Open Neutron DHCP agent configuration file for editing.
sudo vim /etc/neutron/dhcp_agent.iniSet both of the parameters below to True.
enable_isolated_metadata=True
enable_metadata_network=TrueBy enabling metadata network we allow neutron to serve metadata requests coming
from an instance connected to Neutron router and network whose CIDR 169.254.169.254/16 (or larger prefix) VM instances will now reach
169.254.169.254 through a router.
After saving the changes we need to restart neutron-dhcp-agent service.
sudo systemctl restart neutron-dhcp-agent.serviceConfirm the status, it may fail to start if any misconfigurations in the file are detected.
$ systemctl status neutron-dhcp-agent.service
● neutron-dhcp-agent.service - OpenStack Neutron DHCP Agent
Loaded: loaded (/usr/lib/systemd/system/neutron-dhcp-agent.service; enabled; preset: disabled)
Active: active (running) since Thu 2024-06-20 01:13:51 EAT; 24s ago
Main PID: 50860 (neutron-dhcp-ag)
Tasks: 1 (limit: 1645073)
Memory: 118.4M
CPU: 2.078s
CGroup: /system.slice/neutron-dhcp-agent.service
└─50860 "neutron-dhcp-agent (/usr/bin/python3 -s /usr/bin/neutron-dhcp-agent --config-file /usr/share/neutron/neutron-dist.conf --config-file /etc/neutron/neutron.conf --config-file /e>
Jun 20 01:13:51 osp1.hosting.cloudspinx.com systemd[1]: Started OpenStack Neutron DHCP Agent.
Jun 20 01:13:52 osp1.hosting.cloudspinx.com sudo[50865]: neutron : PWD=/ ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf privsep-helper --config-file /usr/share/neutron/>
Jun 20 01:13:54 osp1.hosting.cloudspinx.com sudo[50880]: neutron : PWD=/ ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf privsep-helper --config-file /usr/share/neutron/>
Jun 20 01:13:56 osp1.hosting.cloudspinx.com sudo[50914]: neutron : PWD=/ ; USER=root ; COMMAND=/usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.confLet’s restart our instance on OpenStack.
openstack server reboot Rocky-Linux-9After making the changes we can now see Cloud-init is running.
...
[[0;32m OK [0m] Started [0;1;39mD-Bus System Message Bus[0m.
[[0;32m OK [0m] Reached target [0;1;39mBasic System[0m.
Starting [0;1;39mNTP client/server[0m...
Starting [0;1;39mInitial cloud-init job (pre-networking)[0m...
Starting [0;1;39mRestore /run/initramfs on shutdown[0m...
[[0;32m OK [0m] Started [0;1;39mirqbalance daemon[0m.
[[0;32m OK [0m] Started [0;1;39mHardware RNG Entropy Gatherer Daemon[0m.
[[0;32m OK [0m] Reached target [0;1;39msshd-keygen.target[0m.
[[0;32m OK [0m] Reached target [0;1;39mUser and Group Name Lookups[0m.
Starting [0;1;39mUser Login Management[0m...
[[0;32m OK [0m] Finished [0;1;39mRestore /run/initramfs on shutdown[0m.
[[0;32m OK [0m] Started [0;1;39mNTP client/server[0m.
[[0;32m OK [0m] Started [0;1;39mUser Login Management[0m.
[ 4.848100] cloud-init[698]: Cloud-init v. 23.4-7.el9_4.0.1 running 'init-local' at Wed, 19 Jun 2024 22:17:06 +0000. Up 4.82 seconds.You can also query any information specific to 169.254.169.254.
$ openstack console log show Rocky-Linux-9|grep 169.254.169.254
[ 15.810447] cloud-init[784]: ci-info: | 1 | 169.254.169.254 | 188.40.183.226 | 255.255.255.255 | eth0 | UGH |OpenStack can now inject data such as SSH Public key into the instance and you should be able to login using SSH private key.
$ ssh rocky@192.168.20.22
Warning: Permanently added '192.168.20.22' (ED25519) to the list of known hosts.
Enter passphrase for key '/Users/jmutai/.ssh/id_rsa':
[rocky@rocky-linux-9 ~]$We can check routes
$ ip route
default via 192.168.20.1 dev eth0 proto dhcp src 192.168.20.22 metric 100
169.254.169.254 via 192.168.20.20 dev eth0 proto dhcp src 192.168.20.22 metric 100
192.168.20.0/24 dev eth0 proto kernel scope link src 192.168.20.22 metric 100To retrieve a list OpenStack metadata API supported versions, use a GET request:
[rocky@rocky-linux-9 ~]$ curl http://169.254.169.254/openstack
2012-08-10
2013-04-04
2013-10-17
2015-10-15
2016-06-30
2016-10-06
2017-02-22
2018-08-27
2020-10-14
latestFor EC2-compatible metadata API versions supported run:
[rocky@rocky-linux-9 ~]$ curl http://169.254.169.254
1.0
2007-01-19
2007-03-01
2007-08-29
2007-10-10
2007-12-15
2008-02-01
2008-09-01
2009-04-04
latestMetadata from the OpenStack API is distributed in JSON format. To retrieve the metadata, make a GET request given URL using curl.
curl http://169.254.169.254/openstack/latest/meta_data.jsonTo get pretty output use JSON PP.
sudo dnf -y install perl-JSON-PP
curl http://169.254.169.254/openstack/latest/meta_data.json| json_ppYour instance also retrieve user data (passed as the user_data parameter in the API call or by the --user_data flag in the openstack server create command)
curl http://169.254.169.254/openstack/latest/user_data
curl http://169.254.169.254/openstack/2020-10-14/user_dataCloudSpinx Engineers are always available to help you with any infrastructure related issues. You can chat with us now and we will be happy to engage for any projects – from design, to implementation to diagnosis and issues resolution.
