One of the main features that I’ve always been impressed with ACI is its capabilities to integrate with popular virtualization domains- VMWare, OpenStack, Microsoft SCVMM, Kubernetes, etc. There have been quite a lot of resources covering ACI integration with VMWare in great details. You can find the lab guide in any Cisco DC specialization courses such as DCVAI / DCAC9K, and many other sources you can google for yourself. However, there are not as many for OpenStack. This tutorial is based on Cisco’s official installation guide, adapted to a newer OpenStack release (Queens) on Ubuntu 18.04 Bionic. I hope it will be helpful for those who are seeking a quick and easy way to setup an OpenStack environment with ACI integration. In the future posts, I may write about some key benefits, features, and mechanisms of this ACI-OpenStack integration.
The installation steps will be different depending on your OpenStack distributions and your choice of automated installation tool. If you want to learn more about methods of automated OpenStack installation (e.g. Triple-O, Kola-Ansible, etc.), you might also want to check out the “OpenStack Installation and Deployment” course on Udemy.
In our lab, we will be using MAAS/Juju to automate the installation process. You can refer to my previous series for the guide. The overall topology is as follows:
We will have 7 machines which have been commissioned in the MAAS cloud and are in the “Ready” state on MAAS:
- 03 bare metals UCS C200 servers (m0, m2, m3). Each server has a dedicated NIC (es1f1) connecting to ACI fabric.
- 04 virtual machines (m1, m4, m5, m6)
We will consume these machines to deploy OpenStack applications using Juju later on.
Getting the integration plugin
First, we need to refer to the ACI virtualization compatibility matrix, available here. We will need to check the OpenStack release, and current ACI version to get the correct plugin software. For example, in our setup:
- OpenStack and OS: Canonical Queens with Ubuntu 18.04 Bionic
- Current ACI version: 4.2(2f)
- The recommended plugin would be 4.2(3.20191220), which we can download from Cisco Software Central here.
Then, we will need access and permissions to download the necessary files:
- Debian Bionic packages of ACI Plugin for OpenStack Queens (as we will install OpenStack Queens on Ubuntu 18.04).
- Juju charms for ACI plugins (as we will use Juju for the automated installation)
Preparing local apt repository, charms and configuration files
We will need a local apt repo that the machines can access during the installation. I use
nginx for this, but there are many other ways to get it done. We need to extract the Debian Bionic packages (first file that we downloaded) and put on the repo server. Take note of the repo URL (e.g.
deb http://172.16.9.2 bionic main) because we will need to refer to it in the configuration file.
Untar and extract the Juju charms (second file we downloaded) on the Juju client (e.g.
Prepare the ACI fabric and Juju configuration file. We will need to do the following:
- Create a physical domain on ACI and bind it with an AAEP (e.g. OPENSTACK_AAEP)
- Enable Infra VLAN on the AAEP (as we will be installing OpenStack integration in Opflex mode, where the controllers and compute nodes will have IP addresses assigned from the VTEP pool via DHCP on APIC).
We need to put the configured AAEP name in the Juju configuration file. Please refer to my full juju-aci-queens-bionic.yaml configuration for further details.
Deploying OpenStack with ACI integration using Juju
If you prefer a full fledged OpenStack setup with HA, you can refer to OpenStack documentation. In my lab, the machines will be deployed with the OpenStack core functions as follows:
- Machine m0 (bare metal): OpenStack controller
cephintegration) – install from charmstore
neutron-api-plugin-aci) – install from local charms
- On LXD containers:
nova-cloud-controller– install from charmstore
- Machine m1 (virtual machine): OpenStack dashboard (Horizon)
openstack-dashboard– install from charmstore
- Machine m2, m3: OpenStack Compute Nodes
neutron-aci-opflex) – install from local charms
- Machine m4, m5, m6: CEPH nodes
ceph– install from charmstores
First, we need to create a Juju model and add machines:
juju add-model aci-openstack juju add-machine --constraints tags=m0 juju add-machine --constraints tags=m1 juju add-machine --constraints tags=m2 juju add-machine --constraints tags=m3
At this point, Juju will deploy the necessary Ubuntu 18.04 OS on those machines. You can check the status with
When Juju finishes the OS installation, the machine state will be Deployed. In the next step, we will deploy the charms onto those machines.
CONFIG=/home/administrator/juju-aci-queens-bionic.yaml juju deploy --config=$CONFIG --to lxd:0 cs:mysql juju deploy --to lxd:0 rabbitmq-server juju deploy --config=$CONFIG --to lxd:0 keystone juju add-relation keystone:shared-db mysql:shared-db juju deploy --config=$CONFIG --to 0 glance juju add-relation glance:identity-service keystone:identity-service juju add-relation glance:shared-db mysql:shared-db juju deploy --config=$CONFIG nova-cloud-controller --to lxd:0 juju add-relation nova-cloud-controller:amqp rabbitmq-server:amqp juju add-relation nova-cloud-controller:identity-service keystone:identity-service juju add-relation nova-cloud-controller:image-service glance:image-service juju add-relation nova-cloud-controller:shared-db mysql:shared-db juju deploy --config=$CONFIG --to 0 cinder juju add-relation cinder:image-service glance:image-service juju add-relation glance:cinder-volume-service cinder:cinder-volume-service juju add-relation mysql cinder juju add-relation cinder rabbitmq-server juju add-relation cinder nova-cloud-controller juju add-relation cinder keystone juju deploy --config=$CONFIG nova-compute --to 2 juju add-relation nova-compute:amqp rabbitmq-server:amqp juju add-relation nova-compute:cloud-compute nova-cloud-controller:cloud-compute juju add-relation nova-compute:image-service glance:image-service juju deploy --config=$CONFIG openstack-dashboard --to 1 juju add-relation openstack-dashboard:identity-service keystone:identity-service juju deploy /home/administrator/aci-openstack/charms/charm-neutron-gateway --config=$CONFIG neutron-gateway --to 0 juju add-relation neutron-gateway:amqp rabbitmq-server:amqp juju deploy /home/administrator/aci-openstack/charms/charm-neutron-api --config=$CONFIG neutron-api --to 0 juju add-relation neutron-api:amqp rabbitmq-server:amqp juju add-relation neutron-api:identity-service keystone:identity-service juju add-relation neutron-api:shared-db mysql:shared-db juju deploy /home/administrator/aci-openstack/charms/charm-neutron-api-plugin-aci --config=$CONFIG neutron-api-plugin-aci juju add-relation neutron-api-plugin-aci:amqp rabbitmq-server:amqp juju add-relation neutron-api-plugin-aci:shared-db mysql:shared-db juju deploy /home/administrator/aci-openstack/charms/charm-neutron-aci-opflex --config=$CONFIG neutron-aci-opflex juju add-relation neutron-aci-opflex:neutron-plugin nova-compute:neutron-plugin juju add-relation nova-cloud-controller:neutron-api neutron-api:neutron-api juju add-relation nova-cloud-controller:quantum-network-service neutron-gateway:quantum-network-service juju add-relation neutron-gateway:neutron-plugin-api neutron-api:neutron-plugin-api juju add-relation nova-cloud-controller:quantum-network-service neutron-gateway:quantum-network-service juju add-relation nova-cloud-controller:neutron-api neutron-api:neutron-api juju add-relation neutron-gateway:neutron-plugin-api neutron-api:neutron-plugin-api juju add-relation neutron-api-plugin-aci neutron-api juju add-relation neutron-aci-opflex:neutron-plugin-api neutron-api:neutron-plugin-api juju add-relation neutron-aci-opflex:amqp rabbitmq-server:amqp juju add-relation neutron-aci-opflex:quantum-network-service neutron-gateway:quantum-network-service
You can watch the installation process using
watch -n 2 juju status
Later on, compute nodes (e.g. machine 3) can be added using
juju add-unit. The aci plugin subordinate charm will automatically be deployed, but you might need to restart
juju add-unit nova-compute --to 3
Deploying CEPH backend for Cinder and Glance
We can follow the similar steps to have a 3-node CEPH cluster (on m4, m5, m6) ready for Cinder and Glance.
Configuration file: refer to https://jaas.ai/ceph. My example ceph.yaml:
cat ~/aci-openstack/ceph.yaml ceph: fsid: 36931d5e-84f0-46f7-ba9c-1e14c0267f03 monitor-secret: AQB701xeuR7yFRAAOdcPWINWH/BXYDpnq+9SRg== osd-devices: /dev/sdb
We can deploy the
ceph cluster, the
cinder-ceph sub-ordinate for
cinder and add the needed relations :
juju deploy -n 3 --config ceph.yaml ceph juju deploy cinder-ceph juju add-relation cinder-ceph cinder juju add-relation cinder-ceph ceph juju add-relation glance ceph
The installation should finish with all green Active status, hopefully.
The installation might go wrong by different reasons. You should always check juju log by SSH to the node (host or LXD containers) and
tail -f the log. For example:
# SSH to the bare metal host: juju ssh 0 # or any linux containers: juju ssh 0/lxd/0 # or application: juju ssh neutron-api/0
tail -f /var/log/juju/unit-neutron-api-plugin-aci-0.log 2020-03-02 07:48:37 DEBUG amqp-relation-changed File "/var/lib/juju/agents/unit-neutron-api-plugin-a ci-0/charm/hooks/amqp-relation-changed", line 108, in amqp_changed 2020-03-02 07:48:37 DEBUG amqp-relation-changed aci_db_setup() 2020-03-02 07:48:37 DEBUG amqp-relation-changed File "/var/lib/juju/agents/unit-neutron-api-plugin-a ci-0/charm/hooks/aci_utils.py", line 175, in aci_db_setup 2020-03-02 07:48:37 DEBUG amqp-relation-changed _neutron_gbp_db_manage() 2020-03-02 07:48:37 DEBUG amqp-relation-changed File "/var/lib/juju/agents/unit-neutron-api-plugin-a ci-0/charm/hooks/aci_utils.py", line 124, in _neutron_gbp_db_manage 2020-03-02 07:48:37 DEBUG amqp-relation-changed subprocess.check_output(cmd) 2020-03-02 07:48:37 DEBUG amqp-relation-changed File "/usr/lib/python3.6/subprocess.py", line 356, i n check_output 2020-03-02 07:48:37 DEBUG amqp-relation-changed **kwargs).stdout 2020-03-02 07:48:37 DEBUG amqp-relation-changed File "/usr/lib/python3.6/subprocess.py", line 438, i n run 2020-03-02 07:48:37 DEBUG amqp-relation-changed output=stdout, stderr=stderr) 2020-03-02 07:48:37 DEBUG amqp-relation-changed subprocess.CalledProcessError: Command '['gbp-db-manag e', '--config-file', '/etc/neutron/neutron.conf', 'upgrade', 'head']' returned non-zero exit status 1. 2020-03-02 07:48:37 ERROR juju.worker.uniter.operation runhook.go:132 hook "amqp-relation-changed" fai led: exit status 1
We can reconfig the charms using
juju config and try to resolve the charm failure
juju config neutron-api-plugin-aci --file <path_to_config_file> juju resolve neutron-api-plugin-aci/0
The integration at-first-glance
When the plugin is successfully installed, we can see that ACI has created an OpenStack VMM domain, and detected the hypervisors (compute nodes). The VTEP IP addresses assigned to the compute nodes are also shown. They are the IP addresses that the nodes get from APIC (DHCP Server).
ACI creates a separate tenant (name prj_<project_id> for each OpenStack project, with its on networking constructs (VRF, BD, EPG, etc.)
If we drill down into one tenant (e.g. first_project), we can see ACI has created an Application Profile named OpenStack, but without any EPG and no BD/VRF yet.
Those networking constructs will be defined automatically when we spin up an instance, assign network/subnet, create OpenStack routers, etc. But that will be under scope of the next post. We will look at some key benefits, features, and mechanisms of this ACI-OpenStack integration. So stay tuned!