Apache CloudStack Cloud Operation
This document presents the concept of cloud computing environments, under
the context of private clouds, and an introduction to the Apache CloudStack
cloud orchestration platform, focusing on its operation process.
Last update: January 9, 2026
Revision: a861aa100534f4f28acdbb7f732a7becb500075c
Contents
1 Introduction 22
1.1 Private cloud platforms . . . . . . . . . . . . . . . . . . . . . . . . . 23
1.1.1 Basic functionalities . . . . . . . . . . . . . . . . . . . . . . . 23
1.2 Apache CloudStack history . . . . . . . . . . . . . . . . . . . . . . . 25
2 Apache CloudStack basic concepts 26
2.1 Control Plane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.2 Compute Plane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
2.3 Data Plane . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28
2.4 Logical organization of Apache CloudStack . . . . . . . . . . . . . . 28
2.5 Apache CloudStack components . . . . . . . . . . . . . . . . . . . . 29
3 Apache CloudStack functionalities 31
3.1 Home dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.2 VM settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3.2.1 General settings . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.2.2 Miscellaneous settings for internal usage . . . . . . . . . . . 33
3.2.3 VM settings specific for KVM . . . . . . . . . . . . . . . . . . 33
3.2.4 VM settings specific for VMware . . . . . . . . . . . . . . . . 34
3.2.5 Settings specific for XenServer internal use . . . . . . . . . . 34
3.2.6 Settings specific for Mac OSX - Guest . . . . . . . . . . . . . 34
3.2.7 Global settings for VMs . . . . . . . . . . . . . . . . . . . . . 34
3.2.7.1 User access restriction . . . . . . . . . . . . . . . . 34
3.2.7.2 Extra settings metadata . . . . . . . . . . . . . . . . 35
3.2.7.3 VM statistics retention . . . . . . . . . . . . . . . . 36
3.3 Volume management . . . . . . . . . . . . . . . . . . . . . . . . . . 38
3.3.1 Volume migration with stopped VM (cold migration) . . . . 38
3.3.2 Volume migration with running VMs (hot migration) . . . . 39
3.3.3 Volume migration via CLI . . . . . . . . . . . . . . . . . . . . 40
2
3.3.4 Importing a volume to another zone or account . . . . . . . 42
3.4 Virtual router . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.4.1 Stopping the virtual router . . . . . . . . . . . . . . . . . . . 45
3.4.2 Restarting the virtual router . . . . . . . . . . . . . . . . . . 47
3.4.3 Destroying a virtual router . . . . . . . . . . . . . . . . . . . 47
3.4.4 Default offering definition for virtual router . . . . . . . . . 48
3.5 Public IPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.5.1 Public IP reserved for system VMs . . . . . . . . . . . . . . . 50
3.6 IPv6 support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50
3.6.1 Isolated networks and VPC tiers . . . . . . . . . . . . . . . . 50
3.6.2 Shared networks . . . . . . . . . . . . . . . . . . . . . . . . . 56
3.7 Host, storage and network tags . . . . . . . . . . . . . . . . . . . . . 58
3.7.1 Host tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
3.7.2 Storage tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
3.7.3 Network tags . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
3.7.4 Flexible tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
3.8 Managing instance deployment based on their operating system 67
3.8.1 Hosts Preference OS . . . . . . . . . . . . . . . . . . . . . . . 67
3.8.2 Flexible guest OS . . . . . . . . . . . . . . . . . . . . . . . . . 69
3.9 Snapshots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
3.9.1 Snapshot settings . . . . . . . . . . . . . . . . . . . . . . . . . 70
3.10 Event and alert audit . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
3.10.1 Alert e-mails . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72
3.10.2 Searching and filtering alerts . . . . . . . . . . . . . . . . . . 72
3.10.3 Removing or archiving alerts . . . . . . . . . . . . . . . . . . 73
3.10.4 Event and alert removal automation . . . . . . . . . . . . . 73
3.11 Storage management within Apache CloudStack . . . . . . . . . . 74
3.11.1 Primary storage . . . . . . . . . . . . . . . . . . . . . . . . . . 74
3.11.1.1 Adding a primary storage . . . . . . . . . . . . . . . 75
3
3.11.1.2 Disabling a primary storage . . . . . . . . . . . . . 78
3.11.1.3 Maintenance mode for primary storage . . . . . . 79
3.11.1.4 Behaviour after restarting hosts . . . . . . . . . . . 79
3.11.1.5 Local storage usage . . . . . . . . . . . . . . . . . . 80
3.11.2 Secondary storage . . . . . . . . . . . . . . . . . . . . . . . . 85
3.11.2.1 Adding secondary storages . . . . . . . . . . . . . . 86
3.11.2.2 Data migration between secondary storages . . . 87
3.11.2.3 Read-only mode for secondary storage . . . . . . 88
3.11.2.4 Read-write mode for secondary storage . . . . . . 89
3.11.2.5 Secondary storage removal . . . . . . . . . . . . . 90
3.12 Resource allocation for secondary storage . . . . . . . . . . . . . . 90
4 Service Offerings 94
4.1 Compute offerings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
4.2 Disk Offerings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
4.2.1 Limiting IOPS e BPS in disk offerings . . . . . . . . . . . . . . 101
4.3 Network offerings - throttling . . . . . . . . . . . . . . . . . . . . . . 105
4.4 System offerings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 106
4.4.0.1 Creating system offerings . . . . . . . . . . . . . . 107
4.4.0.2 Editing system offerings . . . . . . . . . . . . . . . 110
4.4.0.3 Removing system offerings . . . . . . . . . . . . . . 111
4.4.0.4 Changing the system offering of a system VMs . . 112
4.5 Backup offerings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 112
4.5.0.1 Enabling backup offerings . . . . . . . . . . . . . . 113
4.5.0.2 Importing backup offerings . . . . . . . . . . . . . 115
4.5.0.3 Backup offering removal . . . . . . . . . . . . . . . 116
4.5.0.4 Using backup offerings . . . . . . . . . . . . . . . . 116
5 Apache CloudStack settings 119
5.1 Settings scopes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
4
5.2 Global settings that control the primary storage usage . . . . . . . 120
5.3 Settings to limit resources . . . . . . . . . . . . . . . . . . . . . . . . 122
5.4 Settings that control Kubernetes usage . . . . . . . . . . . . . . . . 122
5.4.1 Enabling Kubernetes integration . . . . . . . . . . . . . . . . 122
5.4.2 Kubernetes clusters creation . . . . . . . . . . . . . . . . . . 123
6 UI customization 124
6.1 Changing logo and other elements . . . . . . . . . . . . . . . . . . . 124
6.2 Changing logo when resizing the page . . . . . . . . . . . . . . . . . 126
6.3 Theme management . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
6.3.1 Theme creation . . . . . . . . . . . . . . . . . . . . . . . . . . 128
6.3.2 Theme list . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
6.3.3 Updating a theme . . . . . . . . . . . . . . . . . . . . . . . . 131
6.3.3.1 CSS field . . . . . . . . . . . . . . . . . . . . . . . . . 132
6.3.3.2 JSON settings . . . . . . . . . . . . . . . . . . . . . . 132
6.3.4 Removing a theme . . . . . . . . . . . . . . . . . . . . . . . . 134
6.3.5 Common UI customization examples . . . . . . . . . . . . . 134
6.3.5.1 Creating themes with external stylization file . . . 134
6.3.5.2 Notes about style conflicts . . . . . . . . . . . . . . 134
6.3.5.3 Adding fonts . . . . . . . . . . . . . . . . . . . . . . 135
6.3.5.4 Using CSS variables . . . . . . . . . . . . . . . . . . 135
6.3.5.5 Login page . . . . . . . . . . . . . . . . . . . . . . . 135
6.3.5.6 Header stylization . . . . . . . . . . . . . . . . . . . 137
6.3.5.7 Sidebar stylization . . . . . . . . . . . . . . . . . . . 138
6.3.5.8 Cards and dashboard graphs stylization . . . . . . 140
6.3.5.9 Listings and links stylization . . . . . . . . . . . . . 141
6.4 Redirection to external links . . . . . . . . . . . . . . . . . . . . . . . 142
7 Resources consumption accounting 145
7.1 Usage Server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
5
7.1.1 Usage Server setup . . . . . . . . . . . . . . . . . . . . . . . . 145
7.1.2 Usage type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
7.1.3 Usage records . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
7.2 Quota . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
7.2.1 Quota setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . 154
7.2.2 Tariffs management . . . . . . . . . . . . . . . . . . . . . . . 156
7.2.2.1 Creating tariffs . . . . . . . . . . . . . . . . . . . . . 157
7.2.2.2 Tariff details . . . . . . . . . . . . . . . . . . . . . . 159
7.2.2.3 Editing tariffs . . . . . . . . . . . . . . . . . . . . . . 159
7.2.2.4 Removing tariffs . . . . . . . . . . . . . . . . . . . . 160
7.2.3 Activation rules . . . . . . . . . . . . . . . . . . . . . . . . . . 160
7.2.3.1 Default presets for all resource types . . . . . . . . 163
7.2.3.2 Presets for the RUNNING_VM type . . . . . . . . . 165
7.2.3.3 Presets for the ALLOCATED_VM type . . . . . . . . 166
7.2.3.4 Presets for the VOLUME type . . . . . . . . . . . . 167
7.2.3.5 Presets for the TEMPLATE and ISO type . . . . . . 167
7.2.3.6 Presets for the SNAPSHOT type . . . . . . . . . . . 168
7.2.3.7 Presets for the NETWORK_OFFERING type . . . . . 169
7.2.3.8 Presets for the VM_SNAPSHOT type . . . . . . . . 169
7.2.3.9 Presets for the BACKUP type . . . . . . . . . . . . . 169
7.2.3.10 Presets for the NETWORK_USAGE type . . . . . . . 170
7.2.3.11 Presets for the BACKUP_OBJECT type . . . . . . . . 170
7.2.3.12 Verifying presets via API . . . . . . . . . . . . . . . 170
7.2.3.13 Presets for the other resources . . . . . . . . . . . 171
7.2.3.14 Expressions examples . . . . . . . . . . . . . . . . . 171
7.2.4 Credits management . . . . . . . . . . . . . . . . . . . . . . . 174
7.2.4.1 Adding/removing credits . . . . . . . . . . . . . . . 175
7.2.5 Active accounts . . . . . . . . . . . . . . . . . . . . . . . . . . 176
7.2.6 Managing e-mail templates from Quota . . . . . . . . . . . 177
6
7.2.7 Notes about using the Quota plugin . . . . . . . . . . . . . . 179
8 Operation 180
8.1 Debugging issues and troubleshooting process . . . . . . . . . . . 180
8.1.1 Debugging via logs . . . . . . . . . . . . . . . . . . . . . . . . 180
8.1.2 Debugging via web interface . . . . . . . . . . . . . . . . . . 180
8.1.3 Debugging network problems . . . . . . . . . . . . . . . . . 181
8.1.4 Log files path . . . . . . . . . . . . . . . . . . . . . . . . . . . 181
8.1.5 Log level increase on Management Servers and KVM Agents182
8.1.6 Troubleshooting process . . . . . . . . . . . . . . . . . . . . 185
8.2 Host/VM failover . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
8.2.1 Host HA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
8.2.2 VM HA . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
8.3 HA’s relation to heartbeat storage connection loss . . . . . . . . . 197
8.4 Apache CloudStack services management . . . . . . . . . . . . . . 198
8.4.1 Managing the cloudstack-management . . . . . . . . . . . . 199
8.4.2 Managing cloudstack-agent (for KVM hypervisor) . . . . . . 200
8.4.3 Managing cloudstack-usage . . . . . . . . . . . . . . . . . . . 201
8.5 System VMs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
8.5.1 Console Proxy Virtual Machine . . . . . . . . . . . . . . . . . 203
8.5.2 Secondary Storage Virtual Machine . . . . . . . . . . . . . . 203
8.5.3 Virtual Router . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
8.5.3.1 Virtual Router health checks . . . . . . . . . . . . . . 204
8.5.4 Accessing the system VMs . . . . . . . . . . . . . . . . . . . . 207
8.5.5 Changing the system VMs password . . . . . . . . . . . . . . 208
8.5.6 URL for CPVM and SSVM consumption . . . . . . . . . . . . 209
8.6 Enabling VMs computational resources increase . . . . . . . . . . 209
8.7 Overprovisioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
8.8 Updating the Apache CloudStack . . . . . . . . . . . . . . . . . . . . 213
8.8.1 Major versions updates . . . . . . . . . . . . . . . . . . . . . 213
7
8.8.2 Updates within the same major version . . . . . . . . . . . 215
8.9 SSL certificate update in the environment (nginx and ACS) . . . . . 215
8.9.1 Root and intermediary certificates extraction . . . . . . . . 215
8.9.2 Key conversion to PKCS#8: . . . . . . . . . . . . . . . . . . . 216
8.9.3 Adding certificates in nginx . . . . . . . . . . . . . . . . . . . 216
8.9.4 Adding certificates in the Apache CloudStack . . . . . . . . 216
8.10 SSH key pairs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
9 KVM hypervisor 226
9.1 KVM installation and CloudStack Agent . . . . . . . . . . . . . . . . 227
9.2 KVM and CloudStack Agent setup . . . . . . . . . . . . . . . . . . . 228
9.3 KVM operation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 230
9.4 KVM’s CPU topology . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
9.5 CPU control with KVM . . . . . . . . . . . . . . . . . . . . . . . . . . 232
10 VMware hypervisor 235
10.1 Creating datastores . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
10.2 Installing the ESXi hosts . . . . . . . . . . . . . . . . . . . . . . . . . 236
10.2.1 Default ESXi hosts installation . . . . . . . . . . . . . . . . . 237
10.2.2 ESXi hosts basic settings . . . . . . . . . . . . . . . . . . . . . 240
10.2.3 Advanced ESXi hosts settings . . . . . . . . . . . . . . . . . . 245
10.2.3.1 Adding new IPs . . . . . . . . . . . . . . . . . . . . . 245
10.2.3.2 Adding new datastores . . . . . . . . . . . . . . . . 248
10.2.3.3 Adding the license key . . . . . . . . . . . . . . . . 250
10.3 vCenter installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
10.3.1 Adding license key . . . . . . . . . . . . . . . . . . . . . . . . 261
10.3.2 Adding multiple ESXi hosts . . . . . . . . . . . . . . . . . . . 263
10.3.3 Removing the Linux graphical interface . . . . . . . . . . . . 267
10.4 Adding VMware cluster in Apache CloudStack . . . . . . . . . . . . 268
10.5 Problems when adding a VMware cluster . . . . . . . . . . . . . . . 273
8
List of Figures
1 Cloud computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2 Example of a fault tolerant Apache CloudStack architecture . . . . 26
3 Logical organization of Apache CloudStack . . . . . . . . . . . . . . 29
4 Home dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
5 VM settings tab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32
6 Global settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
7 Example for the query return with the total size for the vm _ s ta t s
table . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
8 Migrating volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
9 Confirming operation . . . . . . . . . . . . . . . . . . . . . . . . . . 39
10 Migrating the VM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
11 Configuring the migration . . . . . . . . . . . . . . . . . . . . . . . . 40
12 Getting to the virtual routers tab . . . . . . . . . . . . . . . . . . . . 44
13 Verifying if the VR was correctly created . . . . . . . . . . . . . . . . 44
14 Browsing to the virtual routers . . . . . . . . . . . . . . . . . . . . . 45
15 Stopping the virtual router . . . . . . . . . . . . . . . . . . . . . . . 45
16 Confirming operation . . . . . . . . . . . . . . . . . . . . . . . . . . 46
17 Confirming that the virtual router stopped . . . . . . . . . . . . . . 46
18 Restarting the virtual router . . . . . . . . . . . . . . . . . . . . . . . 47
19 Destroying the virtual router . . . . . . . . . . . . . . . . . . . . . . 47
20 Confirming operation . . . . . . . . . . . . . . . . . . . . . . . . . . 47
21 Confirming virtual router removal . . . . . . . . . . . . . . . . . . . 48
22 Steps to define the default offering for the virtual router . . . . . . 49
23 Beginning to add the IPv6 interval for the public network . . . . . 51
24 Adding an IPv6 interval for the public network . . . . . . . . . . . . 51
25 A /52 prefix allows 4096 IPv6 subnetworks in /64 block . . . . . . . 51
26 Beginning to add the IPv6 interval for the guest network . . . . . . 52
10
27 Adding an IPv6 interval for the guest network . . . . . . . . . . . . 52
28 Creating VPC offering with IPv6 support . . . . . . . . . . . . . . . . 53
29 Creating offering for VPC tiers with IPv6 support . . . . . . . . . . 53
30 VM’s IPv6 autoconfiguration validation (via SLAAC) . . . . . . . . . 54
31 Exhibition via UI for the routes that must be added to the edge
router . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
32 Exhibition via API for the routes that must be added to the edge
router . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
33 Creating an isolated network offering with IPv6 support . . . . . . 55
34 VM’s IPv6 autoconfiguration validation (via SLAAC) . . . . . . . . . 55
35 Exhibition via UI for the routes that must be added to the edge
router . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
36 Exhibition via API for the routes that must be added to the edge
router . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 56
37 Shared network creation from IPv4 info . . . . . . . . . . . . . . . . 57
38 Shared network creation from IPv6 info . . . . . . . . . . . . . . . . 57
39 Field for updating storage tags and host tags of a compute offering 58
40 Accessing the host editing . . . . . . . . . . . . . . . . . . . . . . . . 65
41 Creating the flexible tag in the host . . . . . . . . . . . . . . . . . . 65
42 Accessing the primary storage editing . . . . . . . . . . . . . . . . . 66
43 Creating the flexible tag in the primary storage . . . . . . . . . . . 67
44 Access the host being configured . . . . . . . . . . . . . . . . . . . . 68
45 Access the host’s editing form . . . . . . . . . . . . . . . . . . . . . 68
46 Host’s Preference OS configurations . . . . . . . . . . . . . . . . . . 69
47 Guest OS rule configuration for the host . . . . . . . . . . . . . . . 69
48 Accessing alerts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
49 Archiving or deleting an alert . . . . . . . . . . . . . . . . . . . . . . 73
50 Accessing the primary storage addition menu . . . . . . . . . . . . 76
51 Details for adding a primary storage . . . . . . . . . . . . . . . . . . 76
11
52 Adding a primary storage with NFS . . . . . . . . . . . . . . . . . . 77
53 Adding a primary storage with Shared Mount Point . . . . . . . . . 78
54 Disabling a primary storage . . . . . . . . . . . . . . . . . . . . . . . 79
55 Enabling the maintenance mode . . . . . . . . . . . . . . . . . . . . 79
56 Enabling the usage of local storage for system VMs . . . . . . . . . 80
57 Accessing the zone to be edited . . . . . . . . . . . . . . . . . . . . 81
58 Accessing the zone editing menu . . . . . . . . . . . . . . . . . . . . 81
59 Enabling the usage of local storage for user’s VMs . . . . . . . . . . 82
60 Adding a compute offering with highlight in their Storage Type . . 83
61 Adding a disk offering with highlight in the Storage Type . . . . . . 84
62 Cold volume migration to local storage . . . . . . . . . . . . . . . . 85
63 Adding a new secondary storage . . . . . . . . . . . . . . . . . . . . 87
64 Details while adding a new secondary storage . . . . . . . . . . . . 87
65 Migrating data between secondary storages . . . . . . . . . . . . . 88
66 Details for migrating data between secondary storages . . . . . . 88
67 Defining a secondary storage as read-only . . . . . . . . . . . . . . 89
68 Defining a secondary storage as read-write . . . . . . . . . . . . . . 89
69 Deleting a secondary storage . . . . . . . . . . . . . . . . . . . . . . 90
70 Confirming secondary storage removal . . . . . . . . . . . . . . . . 90
71 Starting to create a new disk offering . . . . . . . . . . . . . . . . . 96
72 Creating the new disk offering . . . . . . . . . . . . . . . . . . . . . 97
73 Starting to edit the disk offering . . . . . . . . . . . . . . . . . . . . 99
74 Editing a disk offering . . . . . . . . . . . . . . . . . . . . . . . . . . 99
75 Starting to update disk offering accesses . . . . . . . . . . . . . . . 99
76 Updating the disk offering accesses . . . . . . . . . . . . . . . . . . 100
77 Changing the disk offerings order . . . . . . . . . . . . . . . . . . . 100
78 Starting to delete a disk offering . . . . . . . . . . . . . . . . . . . . 100
79 Deleting a disk offering . . . . . . . . . . . . . . . . . . . . . . . . . . 100
80 Selecting a QoS type . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
12
81 Limiting a BPS ratio . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
82 Default system offerings . . . . . . . . . . . . . . . . . . . . . . . . . 107
83 Starting to create a new system offering . . . . . . . . . . . . . . . 108
84 Creating a new system offering - 1 - continues . . . . . . . . . . . . 109
85 Creating a new system offering - 2 . . . . . . . . . . . . . . . . . . . 109
86 Starting to edit the system offering . . . . . . . . . . . . . . . . . . 110
87 Editing the system offering . . . . . . . . . . . . . . . . . . . . . . . 110
88 Changing system offering order . . . . . . . . . . . . . . . . . . . . 111
89 Starting the system offering removal . . . . . . . . . . . . . . . . . 111
90 Removing a system offering . . . . . . . . . . . . . . . . . . . . . . . 111
91 Backup offering tab . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
92 Starting to import a new backup offering . . . . . . . . . . . . . . . 115
93 Importing a new backup offering . . . . . . . . . . . . . . . . . . . . 115
94 Starting backup offering removal . . . . . . . . . . . . . . . . . . . . 116
95 Removing the backup offering . . . . . . . . . . . . . . . . . . . . . 116
96 Starting to assign a VM to a backup offering . . . . . . . . . . . . . 116
97 Assigning a VM to a backup offering . . . . . . . . . . . . . . . . . . 117
98 Starting manual backup for a VM . . . . . . . . . . . . . . . . . . . . 117
99 Performing a manual backup for a VM . . . . . . . . . . . . . . . . 117
100 Starting backup scheduling . . . . . . . . . . . . . . . . . . . . . . . 117
101 Scheduling backups . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
102 Starting to remove backup offering from the VM . . . . . . . . . . 118
103 Removing backup offering from the VM . . . . . . . . . . . . . . . . 118
104 Global setting endpoint.url . . . . . . . . . . . . . . . . . . . . . . . 123
105 Customized banner . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
106 Customized footer . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
107 Whole logo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
108 Cut logo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
109 Mini logo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
13
110 Customized login page . . . . . . . . . . . . . . . . . . . . . . . . . . 137
111 Stylized header . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
112 Stylized sidebar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
113 Stylized closed sidebar . . . . . . . . . . . . . . . . . . . . . . . . . . 140
114 Stylized dashboard on root admin account . . . . . . . . . . . . . . 141
115 Stylized dashboard on user account . . . . . . . . . . . . . . . . . . 141
116 Stylized listing and links . . . . . . . . . . . . . . . . . . . . . . . . . 142
117 Only one defined element . . . . . . . . . . . . . . . . . . . . . . . . 143
118 More than one defined element . . . . . . . . . . . . . . . . . . . . 143
119 Link with an icon attribute . . . . . . . . . . . . . . . . . . . . . . . . 144
120 Accessing the settings . . . . . . . . . . . . . . . . . . . . . . . . . . 146
121 Editing the settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
122 Quota plugin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
123 List of active tariffs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
124 Listing filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
125 Tariff creation form . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
126 Tariff details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
127 Possible actions for active tariffs . . . . . . . . . . . . . . . . . . . . 159
128 Tariff editing form . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
129 Removing a tariff . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
130 List of active accounts . . . . . . . . . . . . . . . . . . . . . . . . . . 175
131 Form for adding/removing credits . . . . . . . . . . . . . . . . . . . 175
132 List of active accounts . . . . . . . . . . . . . . . . . . . . . . . . . . 176
133 Listing filters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177
134 List of e-mail templates from Quota . . . . . . . . . . . . . . . . . . 178
135 Editing the e-mail template . . . . . . . . . . . . . . . . . . . . . . . 178
136
Quota state for admin accounts and custom-account . . . . . . . . 179
137 VM created through the UI . . . . . . . . . . . . . . . . . . . . . . . 186
138 Identifying the command sent. . . . . . . . . . . . . . . . . . . . . . 186
14
139 Identifying the logid for the desired process. . . . . . . . . . . . . . 187
140 Console Proxy Virtual Machine . . . . . . . . . . . . . . . . . . . . . . 203
141 Secondary Storage Virtual Machine . . . . . . . . . . . . . . . . . . . 203
142 Virtual Router . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
143 Health checks display for a certain VR. . . . . . . . . . . . . . . . . . 205
144 Acessing the SSL certificate addition form via UI . . . . . . . . . . . 217
145 Adding SSL certificate via UI . . . . . . . . . . . . . . . . . . . . . . . 218
146 Accessing the SSH key pairs sections . . . . . . . . . . . . . . . . . . 219
147 Creating an SSH key pair . . . . . . . . . . . . . . . . . . . . . . . . . 220
148 SSH key pair automatic creation . . . . . . . . . . . . . . . . . . . . . 222
149 Creating the SSH key . . . . . . . . . . . . . . . . . . . . . . . . . . . 224
150 Creating instance and implementing the SSH key . . . . . . . . . . 225
151 KVM virtualization . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
152 Hypervisors comparison . . . . . . . . . . . . . . . . . . . . . . . . . 227
153 Output example for the command virsh schedinfo . . . . . . . . . 234
154 Initial ESXi installation screen . . . . . . . . . . . . . . . . . . . . . . 237
155 Accept terms of use . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
156 Choosing the disk to install the system . . . . . . . . . . . . . . . . 238
157 Selecting the keyboard layout . . . . . . . . . . . . . . . . . . . . . . 238
158 Creating the root password . . . . . . . . . . . . . . . . . . . . . . . 238
159 Potential warning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
160 Confirm installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
161 Host installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
162 Restarting the host . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
163 Initial login . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
164 The default user is root, and its password is the one set during
installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
165 Section Configure Management Network . . . . . . . . . . . . . . . . 241
166 Section IPv4 Configuration . . . . . . . . . . . . . . . . . . . . . . . . 242
15
167 Static IPv4 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 242
168 Applying changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243
169 Section Troubleshooting Options . . . . . . . . . . . . . . . . . . . . . 243
170 Shell access on the host . . . . . . . . . . . . . . . . . . . . . . . . . 244
171 SSH access on the host . . . . . . . . . . . . . . . . . . . . . . . . . . 244
172 URL for accessing the web interface . . . . . . . . . . . . . . . . . . 245
173 Login screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
174 Available NICs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
175 Virtual switches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 246
176 Configuring the new virtual switch . . . . . . . . . . . . . . . . . . . 247
177 VM Kernel NICs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247
178 Configuring the new VM Kernel NIC . . . . . . . . . . . . . . . . . . 248
179 Datastores . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 248
180 Datastore types supported. . . . . . . . . . . . . . . . . . . . . . . . 249
181 Configuring the new datastore . . . . . . . . . . . . . . . . . . . . . 249
182 Finish the datastore setup . . . . . . . . . . . . . . . . . . . . . . . . 250
183 Accessing the license . . . . . . . . . . . . . . . . . . . . . . . . . . . 250
184 Verifying if the license is valid . . . . . . . . . . . . . . . . . . . . . . 251
185 Adding the license . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251
186 Possible operation types . . . . . . . . . . . . . . . . . . . . . . . . . 253
187 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 253
188 Terms of use . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 254
189 Available deploy types . . . . . . . . . . . . . . . . . . . . . . . . . . 254
190 Adding ESXi host . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 255
191 Adding the root password . . . . . . . . . . . . . . . . . . . . . . . . 255
192 Infrastructure size . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
193 Available datastores . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
194 Configuring the vCenter network . . . . . . . . . . . . . . . . . . . . 257
195 Finish configuring the vCenter . . . . . . . . . . . . . . . . . . . . . 257
16
196 Start configuring the PSC . . . . . . . . . . . . . . . . . . . . . . . . 258
197 Applying appliance basic settings . . . . . . . . . . . . . . . . . . . . 258
198 Creating and setting up a new SSO . . . . . . . . . . . . . . . . . . . 259
199 VMware improvement program . . . . . . . . . . . . . . . . . . . . 259
200 Finishing the setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
201 Finishing the installation . . . . . . . . . . . . . . . . . . . . . . . . . 260
202 vSphere home screen . . . . . . . . . . . . . . . . . . . . . . . . . . 261
203 Licenses screen . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
204 Adding the license . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
205 Changing the license name . . . . . . . . . . . . . . . . . . . . . . . 262
206 Finish adding the license . . . . . . . . . . . . . . . . . . . . . . . . . 263
207 Starting to add a new datacenter/folder . . . . . . . . . . . . . . . . 264
208 Naming the new datacenter . . . . . . . . . . . . . . . . . . . . . . . 264
209 Adding a new host to the datacenter . . . . . . . . . . . . . . . . . 265
210 Adding the IP to the host . . . . . . . . . . . . . . . . . . . . . . . . . 265
211 User and password of the host . . . . . . . . . . . . . . . . . . . . . 265
212 Host details overview . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
213 Host license . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
214 Host lockdown mode . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
215 VM location . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
216 Final details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
217 Adding a VMware datacenter . . . . . . . . . . . . . . . . . . . . . . 268
218 Configuring a VMware datacenter in Apache CloudStack . . . . . . 269
219 Accessing the physical networks details in Apache CloudStack . . 269
220 Updating networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
221 Adding the VMware network mapping in the Apache CloudStack . 271
222 Configuring the VMware cluster in Apache CloudStack . . . . . . . 272
223 Tag error message on the UI when trying to add the vCluster . . . 273
224 vCenter zone selection . . . . . . . . . . . . . . . . . . . . . . . . . . 274
17
225 Removing the cloud.zone attribute . . . . . . . . . . . . . . . . . . . 275
226 Selecting and erasing the vCluster from the database . . . . . . . 276
227 Accessing the Tools view to import the VMs . . . . . . . . . . . . . . 277
228 Checking VMware cluster VMs . . . . . . . . . . . . . . . . . . . . . 277
229 Importing a VM to the Apache CloudStack . . . . . . . . . . . . . . 278
230 Details for VM import . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
231 VM successfully imported . . . . . . . . . . . . . . . . . . . . . . . . 279
18
List of Tables
1 VM general configurations . . . . . . . . . . . . . . . . . . . . . . . . 32
2 VM configurations for internal use . . . . . . . . . . . . . . . . . . . 33
3 KVM specific VM configurations . . . . . . . . . . . . . . . . . . . . . 33
4 VMware specific VM settings . . . . . . . . . . . . . . . . . . . . . . 34
5 VM settings specific for XenServer internal use . . . . . . . . . . . 34
6 VM settings specific for Mac OSX - Guest . . . . . . . . . . . . . . . 34
7 Host tags usage example . . . . . . . . . . . . . . . . . . . . . . . . 60
8 Storage tags usage example . . . . . . . . . . . . . . . . . . . . . . . 61
9 Network tags usage example . . . . . . . . . . . . . . . . . . . . . . 63
10 Snapshot settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
11 Events and alerts settings . . . . . . . . . . . . . . . . . . . . . . . . 72
12 Storage protocols supported by each hypervisor . . . . . . . . . . 74
13 Possible values for each resource available in the heuristicrules . 91
14 listSecondaryStorageSelectors API parameters . . . . . . . . . . . 92
15 createSecondaryStorageSelectors API parameters . . . . . . . . . 93
16 updateSecondaryStorageSelectors API parameters . . . . . . . . . 93
17 removeSecondaryStorageSelectors API parameters . . . . . . . . . 94
18 Comparison between test results using block size of 64 KiB and
128 KiB . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
19 Comparison between test results using BPS limits . . . . . . . . . . 104
20 System offerings settings for system VMs . . . . . . . . . . . . . . . 112
21 Backup settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
22 Backup settings for Veeam . . . . . . . . . . . . . . . . . . . . . . . 114
23 Cloudstack behavior settings in case of storage running out of space121
24 Resource limitation settings . . . . . . . . . . . . . . . . . . . . . . . 122
25 createGuiTheme parameters . . . . . . . . . . . . . . . . . . . . . . 129
26 listGuiThemes parameters . . . . . . . . . . . . . . . . . . . . . . . 130
19
27 updateGuiTheme parameters . . . . . . . . . . . . . . . . . . . . . . 131
28 jsonconfiguration attributes . . . . . . . . . . . . . . . . . . . . . . . 133
29 jsonconfiguration plugin attributes . . . . . . . . . . . . . . . . . . 133
30 Usage server settings . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
31 Usage types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150
32 Quota plugin settings . . . . . . . . . . . . . . . . . . . . . . . . . . . 155
33 Default presets for every resource type - Part 1 . . . . . . . . . . . 163
34 Default presets for every resource type - Part 2 . . . . . . . . . . . 164
35 Presets for the RUNNING_VM type . . . . . . . . . . . . . . . . . . . 165
36 Presets for the ALLOCATED_VM type . . . . . . . . . . . . . . . . . 166
37 Presets for the VOLUME type . . . . . . . . . . . . . . . . . . . . . . 167
38 Presets for the TEMPLATE and ISO types . . . . . . . . . . . . . . . 167
39 Presets for the SNAPSHOT type . . . . . . . . . . . . . . . . . . . . . 168
40 Presets for the NETWORK_OFFERING type . . . . . . . . . . . . . . 169
41 Presets for the VM_SNAPSHOT type . . . . . . . . . . . . . . . . . . 169
42 Presets for the BACKUP type . . . . . . . . . . . . . . . . . . . . . . 169
43 Presets for the NETWORK_USAGE type . . . . . . . . . . . . . . . . 170
44 Presets for the BACKUP_OBJECT type . . . . . . . . . . . . . . . . . 170
45 Logs hierarchy levels on Log4j . . . . . . . . . . . . . . . . . . . . . 183
46 Health check settings from virtual routers - Part 1 . . . . . . . . . . 206
47 Health check settings from virtual routers - Part 2 . . . . . . . . . . 207
48 System VM URL settings . . . . . . . . . . . . . . . . . . . . . . . . . 209
49 Overprovisioning settings . . . . . . . . . . . . . . . . . . . . . . . . 212
50 Use case examples for cpu.corespersocket . . . . . . . . . . . . . . 232
20
Notes
• This document does not cover all the topics, subjects and use cases for
Apache CloudStack. Therefore, in case you do not find what you are look-
ing for, create a review or inclusion solicitation through GitLab.
• This document is periodically updated. So, stay tuned for new updates.
21
1. Introduction
In the past few years, there was a huge growth in the usage of cloud comput-
ing (private, public and/or hybrid), intensifying opportunities in the digital ser-
vices market. To supply this demand, the infrastructure needed to create and
maintain this kind of environment grows constantly, directly impacting man-
agement and operation costs.
Cloud computing environments are complex and heterogeneous, possess-
ing dynamic variables and many components that must be intertwined and
managed. In order to efficiently manage this kind of environment it is neces-
sary to employ tools capable of orchestrating the infrastructure, helping during
implementation, maintaining and managing cloud services and systems.
Apache CloudStack and OpenStack are the main open-source alternatives
for creating private cloud environments, highlighted by their robustness, project
maturity and range of technologies and functionalities supported. Such charac-
teristics translate to hundreds of companies adopting these platforms around
the world, including: USP, UNICAMP, Dell, Deutsche Telekom, Apple, Planetel,
LWSA and many others.
SC Clouds comes in aid of this need, providing consultancy, support and
continued development to cloud infrastructure providers and companies that
have cloud technology as the main pillar for creating and providing services. In
partnership with clients, planning and execution are carried out for the cloud
computing environments that provide infrastructure as a service. Our profes-
sionals have full mastery with the Apache CloudStack and OpenStack platforms,
being responsible for multiple new features and bug fixes in the projects. Cur-
rently, SC Clouds helps with management, planning, implantation, support, bug
fixing and implementing new functionalities for companies in Germany, Brazil,
Italy, Mexico and Switzerland.
22
1.1. Private cloud platforms
Cloud orchestration platforms, such as Apache CloudStack and OpenStack,
are used to provide private, public and hybrid clouds to companies and insti-
tutions around the globe. They are capable of connecting different computa-
tional systems (storage, network and hypervisors), creating the abstraction for
infrastructure-as-a-service for the system’s users and administrators.
Both OpenStack and CloudStack are free open-source softwares for cloud
environment orchestration. Both share the goal of managing computational
resources, working with the virtualization concept to provide resources on de-
mand, focusing on the provision of infrastructure-as-a-service resources.
Figure 1: Cloud computing
1.1.1. Basic functionalities
Both CloudStack and OpenStack offer basic functionalities needed to provi-
sion cloud computing services. Some of the provided functionalities are:
• Computing services
– VM provisioning and management for their resources (networks, mem-
ory, CPU, disks, etc);
23
– Management of the images with which the VMs will be created;
– Support for multiple hypervisors, like KVM, XenServer and VMware.
• Network services
– Management and configuration for shared and dedicated networks;
– Creation of VLAN, VxLANS (overlay networks) and other topologies;
– Definition of access and routing policies;
– Qos, Load balancers and firewalls management;
• Storage services
– Disk management;
– Block and object based storage;
– Backups;
– SDS support (Software Defined Storage), e.g. Ceph/RBD;
• Resource monitoring services
– Event monitoring;
– Display available and used resources;
– Pricing and billing;
• Open REST APIs, standardized and documented, to manage all services,
their automation and integration with other tools;
• Authentication service with native LDAP integration;
• Support to SAML2 and OpenID Connect federated systems;
• Containers-as-a-service;
Within the basic functionalities for cloud orchestration systems, Apache Cloud-
Stack has limitations related to federated systems, since it only supports the
SAML2 protocol.
24
1.2. Apache CloudStack history
The CloudStack project started in 2008 as a startup project known as VMOps.
The company changed its name to Cloud.com and released a CloudStack stable
version in 2010, under the GNU GPLv3 license (General Public License version
3). In 2011, Cloud.com was acquired by Citrix and in 2012 the project was do-
nated to the Apache foundation. In the past few years, CloudStack has grown
and has become a reference orchestration tool, being used by many organiza-
tions, such as Globo, RNP, UNICAMP, USP, LWSA, CGEE, Apple, Dell, Nokia and
Disney.
25
2. Apache CloudStack basic concepts
This section introduces the basic concepts of Apache CloudStack, its logical
organization and the components that constitute the orchestrator.
Apache CloudStack’s architecture is designed to support both vertical scal-
ing (increasing a machine’s resources) and horizontal scaling (adding new ma-
chines, either Management Servers or CloudStack Agents).
Figure 2: Example of a fault tolerant Apache CloudStack architecture
A minimal Apache CloudStack implantation project requires a Control Plane,
a Compute Plane and a Data Plane; besides that, a network infrastructure is
also required.
2.1. Control Plane
The Control Plane refers to a layer composed of services responsible for
managing cloud available resources, such as physical servers, storages and net-
work switches. In addition, within the Control Plane is performed the configu-
ration and management of virtual networks, routing and load balancing. The
basic components required to implement the Control Plane include:
• Management Server: responsible for managing and orchestrating resources
available in the infrastructure, as well as providing REST APIs and a web
26
interface for management. It can be duplicated/replicated to provide a
high availability structure to the API and UI, however, it is necessary to
use a web proxy for this setup, and sticky sessions for the UI.
• Load balancer: CloudStack allows the use of a load balancer that provides
a virtual IP to perform load distribution between the Management Servers
as well as allowing the use of sticky sessions. The environment’s adminis-
trator is responsible for creating load balancer rules for the Management
Servers based on the environment’s needs.
• Database: CloudStack uses various settings that define how the cloud
will be managed. These settings are stored on a database (MariaDB or
MySQL), that can be handled by admin users
1
.
• Galera Cluster: In the context of database clustering, Galera Cluster is a
synchronous virtual replication multi-primary cluster. By offering the syn-
chronous replication functionality and allowing read and write in any clus-
ter node, Galera comes as a way to ensure fault tolerance and consistency
between all database nodes.
• Usage Server (optional): Besides the essential components for implanta-
tion, the Usage Server is an optional service responsible for processing
resource consumption events in the infrastructure, making it possible for
external agents to charge for these events. This topic will be detailed in
the section 7.1.
2.2. Compute Plane
The Compute Plane is responsible for sustaining the infrastructure offered
as a service, acting during allocation of computational resources and work load
1
We do not recommend performing changes in the database without the supervision of an
SC Clouds member or previous knowledge, since this kind of change may cause inconsistencies
in CloudStack’s processes and workflows.
27
distribution between available hosts in the environment. It handles VM cre-
ation, migration and monitoring. The physical servers assigned for this service
must support virtualization, as they will host the allocated VMs.
• CloudStack Agent: only necessary when using the KVM hypervisor, the
CloudStack Agent performs the connection between the Management Server
and KVM.
2.3. Data Plane
The Data Plane consists of elements responsible for storage management
(appliances or SDSs, when applicable, such as Ceph/RBD). The main concepts
regarding storages will be addressed in the section 3.11.
2.4. Logical organization of Apache CloudStack
The logical organization used by CloudStack can be split in:
• Region: broadest cloud implementation unit. Groups one or more zones.
• Zone: represents a single datacenter, composed of one or more pods and
secondary storages.
• Pod: represents a single rack on the datacenter.
• Cluster: consists of one or more homogeneous hosts that share the same
primary storage. Ideally, the processors of all hosts should belong to the
same family, aligned with the capabilities of the oldest generation.
• Host: a server where the user VMs are executed. When using KVM as the
hypervisor, it is also where the CloudStack Agent is installed.
• Primary Storage: operates at either host, cluster or zone level. Responsible
for storing the disks (also called volumes) utilized by VMs.
• Secondary Storage: operates at zone level, storing VM templates/ISOs/back-
ups/snapshots.
28
Figure 3: Logical organization of Apache CloudStack
Figure 3 illustrates the ACS hierarchy, in which the outermost component
is a region, that groups one or more zones. The zones represent a datacenter
composed of pods, which are composed by clusters, and clusters consist of
hosts, which function as active processing nodes.
2.5. Apache CloudStack components
The CloudStack structure is composed of the following components:
1. Management Servers;
2. Hosts;
3. Network : network that connects the ACS components. It is recommended
to make use of network segregation, physically or virtually (via broadcast
domain), to improve security;
4. Primary storage;
5. Secondary storage;
6. Virtual router: System VM which emulates a router, implementing the ser-
vices of guest networks. Making it possible, for example, to access the
internet;
29
7. Console proxy virtual machine: System VM responsible for providing console
access for any VM via web interface;
8. Secondary storage virtual machine: System VM responsible for managing
the zone’s secondary storages. Provides functions including download,
registry and upload for volumes, templates and ISOs.
30
3. Apache CloudStack functionalities
This section provides a brief overview of the main Apache CloudStack func-
tionalities related to environment operations (RootAdmin users), along with in-
structions on how to use them. For information regarding resource consump-
tion by end users, refer to the Apache CloudStack Cloud Usage document.
3.1. Home dashboard
This is the default Apache CloudStack dashboard, where the infrastructure
details are shown.
Figure 4: Home dashboard
3.2. VM settings
VMs have both general settings and hypervisor specific settings. It is recom-
mended for some settings to be kept, such as the ones for internal usage, in
order to prevent issues.
This section will only present settings available at operation level. For VM
settings available at usage level, check the Apache CloudStack usage documen-
tation.
To modify a VM’s settings, it is first necessary to stop it. Then, the VM settings
can be modified accessing the Settings tab.
31
Figure 5: VM settings tab
3.2.1. General settings
Setting Description.
rootdisksize Defines the root disk size for the VM. If a service offer-
ing has its root disk size configured, this parameter is
overwritten by the service offering.
rootDiskController Defines the disk controller used by the VM’s root disk.
dataDiskController Defines the disk controller used by the VM’s data disks.
nameonhypervisor Defines the VM’s name within the hypervisor.
Table 1: VM general configurations
32
3.2.2. Miscellaneous settings for internal usage
Setting Description
cpuOvercommitRatio Defines the ratio for CPU overcom-
mit, which allows for more CPU
cores to be allocated than the num-
ber physically available on the host
machine.
memoryOvercommitRatio Defines the ratio for memory over-
commit, which allows for more
memory to be allocated than the
amount physically available on the
host machine.
Message.ReservedCapacityFreed.Flag Internal flag used to indicate if the
reserved resources for a VM were re-
leased.
Table 2: VM configurations for internal use
3.2.3. VM settings specific for KVM
Setting Description
kvm.vnc.port Defines the VNC port used by the VM in the KVM environ-
ment.
kvm.vnc.address Defines the IP used to access the VM via NIC in a KVM
environment.
video.hardware Defines the virtual video adapter used by the VM.
video.ram Defines the amount of video RAM available for the VM.
io.policy Defines the policy used to deal with data I/O.
iothreads Allows the usage of specific threads for I/O.
Table 3: KVM specific VM configurations
When setting video.hardw are with the value virtio, Windows VMs will start
to utilize the VirtIO driver, permiting the usage of new console resolutions in
the VM. More information related to KVM can be found in Section 9.
33
3.2.4. VM settings specific for VMware
Setting Description
svga.vramSize Defines the amount of video memory available for
the VM.
nestedVirtualizationFlag Enables or disables nested virtualization within
the VM.
ramReservation Defines the minimum RAM amount allocated to
the VM.
Table 4: VMware specific VM settings
More information related to KVM can be found in the section 10.
3.2.5. Settings specific for XenServer internal use
Setting Description
hypervisortoolsversion Defines the hypervisor version used by the VM.
platform Defines the platform type in which the VM is being
executed, such as Linux or Windows.
timeoffset Defines the timezone for the VM.
Table 5: VM settings specific for XenServer internal use
3.2.6. Settings specific for Mac OSX - Guest
Setting Description
smc.present Enables or disables SMC support in the VM.
firmware Defines the firmware used by the VM.
Table 6: VM settings specific for Mac OSX - Guest
3.2.7. Global settings for VMs
Beyond the settings that individually affect VMs, Apache CloudStack pro-
vides some global settings related to the VMs, so that the environment may be
customized as a whole.
3.2.7.1. User access restriction
It is possible to restrict user type accounts from modifying certain settings in
the VMs. There are two restriction options, that are defined with the following
34
global settings:
• user.vm.denied.details: prevents adding, editing and visualizing settings;
• user.vm.readonly.details: prevents adding or editing settings, but they are
still visible for the users in the Settings tab.
Figure 6: Global settings
3.2.7.2. Extra settings metadata
It is possible to add extra properties to the deployment of a VM, as long as the
enable.additional.vm.configuration setting is enabled (it is necessary to restart
the Management Server for the changes to this setting to be applied).
The cloud administrator may define through the following global settings
which commands the users may add to their VMs, based on their respective
hypervisors.
• allow.additional.vm.configuration.list.kvm
Additional KVM settings must be provided in XML format, encoded as URL.
For example:
<memoryBacking> <hugepages/> </memoryBacking>
As URL:
%3CmemoryBacking%3E%0D%0A++%3Chugepages%2F%3E%0D%0A%3C%2FmemoryBacking%3E
35
• allow.additional.vm.configuration.list.vmware
Additional VMware settings are key=value pairs, encoded as URL. For ex-
ample:
hypervisor.cpuid.v0=FALSE
As URL:
hypervisor.cpuid.v0%3DFALSE
• allow.additional.vm.configuration.list.xenserver
Additional XenServer settings are vm-param-set command parameters,
in the form of key=value pairs, encoded as URL. For example:
HVM-boot-policy=
PV-bootloader=pygrub
PV-args=hvc0w
As URL:
HVM-boot-policy%3D%0APV-bootloader%3Dpygrub%0APV-args%3Dhvc0
3.2.7.3. VM statistics retention
VM statistics retention is governed through two global settings:
• vm.stats.interval: interval, in millisecond, between each data collection.
The default value for this setting is 60000 milliseconds, equivalent to 1
minute.
• vm.stats.max.retention.time: retention time, in minutes, for the data on
the database. The default value for this setting is 720 minutes, equivalent
to 12 hours.
Based on the value defined on these settings, it is possible to calculate the
maximum amount of statistical records within the database using the following
36
formula:
storageSpace =
(retention · 60000)
interval
· M Ss · V M s · recordSize
• storageSpace: space, in bytes, necessary to store the statistics from the
VMs;
• retention: value for the vm.stats.max.retention.time setting;
• interval: value for the vm.stats.interval setting;
• MSs: amount of Management Servers running in the environment;
• VMs: amount of VMs running in the environment;
• recordSize: estimated size, in bytes, for each record in the database.
Therefore, in an environment with 2 Management Servers and 1000 VMs,
in which the retention time is 10 minutes, the collection interval is 30 seconds
and the size for each record is 400 bytes, the maximum amount of records in
the database will be 16 MB:
(10 · 60000)
30000
· 2 · 1000 · 400 = 16000000B = 16MB
Furthermore, it is possible to track the physical growth of the table with the
following command in the database:
SELECT TABLE_NAME AS 'Table',
DATA_LENGTH AS 'Data size (B)',
INDEX_LENGTH AS 'Index size (B)',
ROUND((DATA_LENGTH + INDEX_LENGTH) / 1024) AS 'Total size (KiB)'
FROM information_schema.TABLES
WHERE TABLE_SCHEMA = 'cloud'
AND TABLE_NAME = 'vm_stats';
Figure 7: Example for the query return with the total size for the vm_stats table
37
Still, it should be highlighted that the MySQL database shows value changes
for each 16KiB of data. Therefore, considering that each record has a 400 bytes
size, changes in those values will be observed every 40 records, approximately.
3.3. Volume management
This section addresses operational processes related to volume manage-
ment within ACS
2
.
3.3.1. Volume migration with stopped VM (cold migration)
As default, this operation is only allowed for root admin users. It is important
to note that the VM’s host must have access to the volume’s destination storage.
Figure 8: Migrating volumes
2
Other processes related to volume management can be found in the Apache CloudStack
usage documentation.
38
Figure 9: Confirming operation
3.3.2. Volume migration with running VMs (hot migration)
As default, this operation is allowed only for root admin users. If the utilized
hypervisor is KVM, it is also necessary to migrate the VM to which the volume
belongs in order to perform hot volume migration. More details about the lim-
itations of hot migration are found in the following item.
Figure 10: Migrating the VM
39
Figure 11: Configuring the migration
3.3.3. Volume migration via CLI
There are two possible cases for volume migration: migrating volumes of
stopped VMs (cold migration) and migrating volumes of running VMs (hot mi-
gration).
In cold volume migration the volume is copied to the secondary storage and
then to the primary storage. While in hot migration, the volume is directly mi-
grated to the target primary storage, which makes it faster and cheaper for the
environment.
To perform a cold migration, just use the following command:
migrate volume volumeid=<volume_id>
storageid=<destination_storage_id>
On the other hand, for hot volume migration, some details are important to
40
note:
• If the hypervisor in use is VMware or XenServer, the command above can
be executed directly. However, when using KVM, it is necessary to perform
a VM migration between hosts at the same time. KVM provides a feature
called live block copy, which allows the migration of volumes from a run-
ning VM to another storage without requiring the VM itself to be migrated
to a different host. After the migration, the XML and the VM process are
updated to reference the new storage. This functionality, however, is not
supported by ACS currently. Consequently, running VMs on KVM cannot
migrate volumes independently through ACS. Therefore, volume migra-
tion between storages while the VM is active must occur in conjunction
with a host migration to ensure that all necessary updates are applied.
There is a known issue with XFS, in which it may have its partitions cor-
rupted during the hosts migration, making it necessary to execute a pro-
cess to recover these volumes;
• If the VM contains large data disks, the ACS timeout settings must be re-
viewed, as very large volumes may trigger a timeout and cause the migra-
tion process to fail;
• KVM has some timeout settings that deal with VM migrations. If the VM
possesses a huge amount of RAM, the migration process may fail.
The main global settings that affect migrations are:
• enable.storage.migration: Enables volume migrations;
• secstor age.max .migrat e.sessions: Maximum number of data copy tasks
that an SSVM may perform concurrently;
• secstorage.sess ion.max: Maximum number of requests that an SSVM
may have queued. When the value is exceed, a new SSVM is created.
41
Moreover, when the amount of requests is far from reaching this value
and there are inactive SSVMs, they are destroyed;
• storage.pool.max.waitseconds: Wait timeout for storage pool synchro-
nization operations;
• migratewait: VM migration timeout. After this time limit the migration is
cancelled;
• kvm.s torage.offl ine.migration.wait: Timeout for cold volume migration
(only applies to KVM), and
• kvm.storage.online.migration.wait: Timeout for hot volume migration (only
applies to KVM).
The KVM agent settings may be changed in the following file: /etc/cloud
stack-features/agent/agent.properties, with the main settings that affect the
migration process being:
• vm.migrate. pauseafter: Wait time for the completion of hot migrations.
After this limit, the VM is paused to complete the migration;
• vm.migrate.downtime: Time that the VM will stay paused to complete the
migration, and
• vm.migrate.speed: Migration speed (in MB/s). As default, ACS tries to es-
timate the network transfer speed.
To perform a volume migration, use the command:
migrate virtualmachinewithvolume virtualmachineid=<vm_id> hostid=<destination_host_id> \
migrateto[0].volume=<volume_id> migrateto[0].pool=<target_storage_id>
3.3.4. Importing a volume to another zone or account
There are some situations where it is necessary to import volumes to an-
other zone and/or account. Via UI, this can be achieved following these steps:
42
1. Acquire the volume download link from the Apache CloudStack usage doc-
ument
3
;
2. Copy and paste the URL on the volume upload URL field
4
;
3. Configure the volume to belong to the desired zone and/or account.
It is also possible to perform this procedure via API. For this, follow these
steps:
1. Use the following command to generate the URL:
extract volume zoneid=<zone_id> id=<volume_id> mode=HTTP_DOWNLOAD
2. To perform the importation, use the command
5
:
upload volume zoneid=<destination_zone_id> name=<volume_name> format=<volume_format> \
url=<previously_generated_URL> domainid=<destination_domain_id> account=<destination_account_id>
3.4. Virtual router
This section will exclusively address virtual router operations. For more in-
formation related to guest networks, access the Apache CloudStack usage doc-
umentation.
When a VM uses an isolated network and when a shared network is created,
Apache CloudStack automatically creates a virtual router for them.
3
More information may be found in the section Volume download in the Apache CloudStack
usage document.
4
More information may be found in the section Online volume upload via URL in the Apache
CloudStack usage document.
5
The parameters domainid and account are optional, however the ac c o u n t parameter de-
pends on the domainid.
43
Figure 12: Getting to the virtual routers tab
Figure 13: Verifying if the VR was correctly created
44
3.4.1. Stopping the virtual router
Figure 14: Browsing to the virtual routers
Figure 15: Stopping the virtual router
45
Figure 16: Confirming operation
Bear in mind that when the Force option is enabled, ACS will mark the router
VM as stopped, even if the stop command fails to execute on the host. This
option should only be used when it is certain that the router is already stopped
but ACS still reports it as running.
Figure 17: Confirming that the virtual router stopped
46
3.4.2. Restarting the virtual router
Figure 18: Restarting the virtual router
3.4.3. Destroying a virtual router
It is possible to destroy a VR, but it will be recreated when the network is
restarted. When destroying a VR all services provided by it will be interrupted,
therefore running VMs that utilize it may show errors and faults.
Figure 19: Destroying the virtual router
Figure 20: Confirming operation
47
Figure 21: Confirming virtual router removal
3.4.4. Default offering definition for virtual router
When creating a new network, there is a sequence of actions established by
CloudStack to define which offering (section 4) the VR will utilize, determined
through the following steps:
1. If a service offering is specified in the network offering used in the network
creation, it will be used in the VR creation. Otherwise, the next step will
be verified:
2. If the router.service.offering setting, at account level, defines the offering,
it will be used in the VR creation. Otherwise, the next step will be verified:
3. If the router.service.off ering global setting defines the offering, it will be
used in the VR creation. Otherwise, the next step will be verified:
4. The default system offering will be used (System Offering For Software Router
or System Offering For Software Router - Local Storage).
48
Figure 22: Steps to define the default offering for the virtual router
It is important to verify that if an already existing network is updated to uti-
lize a new network offering, or the network is restarted with clean up, the new
VRs will also follow the same steps to define which offering will be used.
49
3.5. Public IPs
This section will only approach topics relevant to public IPs at operation
level. For more information, access the Apache CloudStack Cloud usage docu-
mentation.
3.5.1. Public IP reserved for system VMs
CloudStack operates with reserved IPs for system VMs. They are not strict,
so if there are only public IPs reserved for system VMs free, ACS will use them
for user VMs. To change this behaviour and make sure that public IPs reserved
for system VMs are only available for them it is necessary to change the global
setting system.vm.public.ip.reservation.mode.strictness to true.
3.6. IPv6 support
ACS provides support for IPv6 in shared and isolated networks, as well as
VPC tiers. Currently, it is not possible to create any of the mentioned networks
using only IPv6, all of them need to be created with an IPv4 + IPv6 dual-stack.
The following subsections detail the requirements and limitations related to
shared networks and isolated networks/VPC tiers support.
3.6.1. Isolated networks and VPC tiers
Currently, ACS only supports IPv6 through the SLAAC configuration, not sup-
porting DHCPv6 stateless or stateful, making it necessary to manually add routes
on the edge router for each network created in the environment. The addition
of routes on the edge router should only be performed after the network is
created in ACS. Furthermore, public networks are allocated using a whole /64
block for IPv6, making it necessary for the prefix to be lower than or equal to /
64. The following steps are mandatory to enable IPv6 in isolated networks/VPC
tiers:
1. Enable the global setting ipv6.offering.enabled.
2. Add the IPv6 interval for the public network.
50
Figure 23: Beginning to add the IPv6 interval for the public network
Figure 24: Adding an IPv6 interval for the public network
Figure 25: A /52 prefix allows 4096 IPv6 subnetworks in /64 block
3. Add the IPv6 prefix, which should be equal to or lower than /64.
51
Figure 26: Beginning to add the IPv6 interval for the guest network
Figure 27: Adding an IPv6 interval for the guest network
Furthermore, it is necessary to create offerings for VPCs and their tiers, spec-
ifying both protocols; the same applies to isolated networks. Below is shown a
step-by-step to create tiers with IPv6 support.
1. Create VPC offering with IPv6 support.
52
Figure 28: Creating VPC offering with IPv6 support
2. Create network offering for tiers with IPv6 support.
Figure 29: Creating offering for VPC tiers with IPv6 support
53
3. The VM allocated to the network will use the SLAAC protocol, assigning an
unused IPv6 to it.
Figure 30: VM’s IPv6 autoconfiguration validation (via SLAAC)
4. In the edge router, it is necessary to add the informed routes, via UI or via
API.
Figure 31: Exhibition via UI for the routes that must be added to the edge router
Figure 32: Exhibition via API for the routes that must be added to the edge router
The process of creating isolated networks is similar, differing only during the
creation of the network offering.
54
1. Create an isolated network offering with IPv6 support.
Figure 33: Creating an isolated network offering with IPv6 support
2. The VM allocated to the network will use the SLAAC protocol, assigning an
unused IPv6 to it.
Figure 34: VM’s IPv6 autoconfiguration validation (via SLAAC)
3. In the edge router, it is necessary to add the informed routes, via UI or via
API.
55
Figure 35: Exhibition via UI for the routes that must be added to the edge router
Figure 36: Exhibition via API for the routes that must be added to the edge router
3.6.2. Shared networks
Unlike isolated networks and VPC tiers, it is not necessary to perform any
configuration within ACS to use shared networks with IPv6, being enough to
create a network with the shared type while specifying the gateway, CIDR and
IP range used by the network. These informations must be specified for both IP
protocol versions, since, currently, it is not possible to create a network using
only IPv6.
Shared networks management is performed directly through the network
router, being done separately from ACS. VMs will also use the SLAAC protocol
for IPv6 autoconfiguration, so, it is necessary for the gateway to have the Router
56
Advertisements service enabled, informing the IPv6 network prefix. With these
informations, the VM itself will obtain a valid IP within the network. Below is
shown an example for creating a dual-stack shared network.
Figure 37: Shared network creation from IPv4 info
Figure 38: Shared network creation from IPv6 info
57
3.7. Host, storage and network tags
Host tags and storage tags, despite their names, do not relate to resource
tags and are functionalities aimed to direct resource allocation, such as in which
host the VM deploy will be perfomed or in which storage the volume will be
created. Tags serve multiple purposes (such as directing volumes to a better
quality storage, based on the offering). The behaviour of tags varies depending
on the resource type.
Figure 39: Field for updating storage tags and host tags of a compute offering
3.7.1. Host tags
Host tags are responsible for directing VMs to compatible hosts. They are
validated with the host tags informed in the compute offerings (section 4.1) or
in the system offerings (section 4.4).
To explain the host tags’ behaviour some examples with two hosts (Host1
and Host2) will be presented:
1. Tags organization:
• Host1: h1
• Host2: h2
• Offering: h1
58
When creating a VM with the offering, the deploy will be perfomed in
Host1, since it has the tag compatible with the offering.
2. Tags organization:
• Host1: h1
• Host2: h2,h3
• Offering: h3
The hosts accept a tag list, using comma (,) as a separator. Therefore, in
this example, Host2 has the tags h2 and h3. When creating a VM with
the offering, the deploy will be performed in Host2, since it has the tag
compatible with the offering.
3. Tags organization:
• Host1: h1
• Host2: h2,h3
• Offering: h2,h3
Contrary to hosts, offerings do not accept a list of host tags, therefore in
this example, h2,h3 is only one tag in the offering. None of the hosts have
compatible tags, therefore, it will not be possible to deploy a VM with the
offering. However, when a host is manually selected CloudStack ignores
this behaviour and allows the deployment.
4. Tags organization:
• Host1: h1
• Host2: h2,h3
• Offering: (no tags)
When the offering does not have any tags, the VM deployment may be
performed in any host.
5. Tags organization:
59
• Host1: (no tags)
• Host2: h2
• Offering: h3
None of the hosts have compatible tags and the deployment for VMs with
this offering is not possible. However, CloudStack ignores this behaviour
when a host is manually selected and allows the deployment.
Example Host tags Offering tag Behaviour
1 Host1: h1
Host2: h2
h1 Deploy in Host1
2 Host1: h1
Host2: h2,h3
h3 Deploy in Host2
3 Host1: h1
Host2: h2,h3
h2,h3 The deploy will not be per-
fomed, unless one of the hosts
is selected
4 Host1: h1
Host2: h2,h3
(no tags) Deploy in any host
5 Host1: (no tags)
Host2: h2
h3 The deploy will not be per-
fomed, unless one of the hosts
is selected
Table 7: Host tags usage example
3.7.2. Storage tags
Storage tags are responsible for directing volumes to compatible primary
storages. They are validated with the storage tags informed in the disk offer-
ings
6
, compute offerings (section 4.1) or system offerings (section 4.4).
The following examples demonstrate the behavior of storage tags:
1. Tags organization:
• Storage: A
• Offering: A,B
6
Information about disk offerings can be checked in the Apache CloudStack usage docu-
mentation.
60
Both storage and offering accept a comma-separated (,) list of tags. There-
fore, in this example the offering has the tags A and B. In this example it
will not be possible to allocate the volume, because all offering tags need
to be present in the storage. Despite the storage having the tag A, it does
not have the tag B.
2. Tags organization:
• Storage: A,B,C,D,X
• Offering: A,B,C
In this example, it will be possible to allocate the volume, because all of
the offering tags are present in the storage.
3. Tags organization:
• Storage: A,B,C
• Offering: (no tags)
In this example, it will be possible to allocate the volume, because the
offering does not have any tag requirements.
4. Tags organization:
• Storage: (no tags)
• Offering: D,E
In this example, it will not be possible to allocate the volume, because the
storage does not have tags, therefore it does not fulfill the tag require-
ments of the offering.
Example Storage tags Offering tags Behaviour
1 A A,B The volume will not be allocated
2 A,B,C,D,X A,B,C The volume will be allocated
3 A,B,C (no tags) The volume will be allocated
4 (no tags) D,E The volume will not be allocated
Table 8: Storage tags usage example
61
In summary, if the offering has tags, the storage must have all the tags for
the volume allocation process to occur. If the offering does not have any tags,
the volume can be allocated, regardless of the storage having any tags.
3.7.3. Network tags
Network tags are responsible for directing the virtual networks to compat-
ible physical networks. They are validated with network tags informed in the
network offerings
7
.
The following examples demonstrate the behavior of the network tags:
1. Tags organization:
• Physical network: A
• Offering: A,B
Neither the physical network nor the offering accept lists for tags, there-
fore, there can only be one tag for each resource. In this case, the value
A,B set for the offering corresponds to a single tag. As the tags from the
offering and the physical network are different, it will not be possible to
direct the virtual network for this physical network.
2. Tags organization:
• Physical network: B
• Offering: B
In this example, both the physical network and the offering have the same
tag, therefore it will be possible to direct the virtual network to the physical
network.
3. Tags organization:
• Physical network: C
7
Information about network offerings can be found in the Apache CloudStack usage docu-
mentation.
62
• Offering: (no tags)
In this example, it will be possible to direct the virtual network to the phys-
ical network, because the offering does not have any tag requirements.
4. Tags organization:
• Physical network: (no tags)
• Offering: D
In this example, it will not be possible to direct the virtual network to the
physical network, because the physical network does not have any tags
and, consequently, does not fulfill the offering requirements.
Example Network tag Offering tag Behaviour
1 A A,B Will not be directed
2 B B Will be directed
3 C (no tags) Will be directed
4 (no tags) D Will not be directed
Table 9: Network tags usage example
In summary, directing the virtual network to the physical network will not be
possible if the offering has a different tag than the physical network. If the of-
fering does not have any tags, directing will be possible, regardless of whether
the physical network has tags.
3.7.4. Flexible tags
When defining tags for a resource (i.e. a host), the offerings with those tags
will be directed to this resource. However, offerings without tags can also be
directed to it. In a way that, even after adding tags to a resource with the pur-
pose of making it exclusive for some types of offerings, this exclusivity can be
ignored.
Furthermore, the default tag system only allows the user to inform a sim-
ple list of tags, without the possibility to create more complex rules, such as
verifying if the offering has certain pairs of tags.
63
To circumvent these situations, ACS allows hosts and storages to use tags
defined as JavaScript rules, known as flexible tags. With flexible tags, the role of
tags is inverted, instead of requiring the host or storage to have the same tag as
the offering to be selected, the offering must include the tag of the resource it
will be assigned to. This inversion makes it so that offerings without tags cannot
be directed to any resource. This way, operators may have finer control over
VMs and volumes directed within the environment.
The configuration of rules in the hosts is done through the updateHost API,
informing the rule in the hostta gs field. Still, the configuration of rules in the
storages is done through the updateStoragePool API, informing the rule in the
tags field. For the informed tag to be effectively interpreted as JavaScript, it is
necessary to declare the istagaru le parameter as true every time that one of
the listed APIs is used.
It is important to highlight that tags in compute offerings or disk offerings
are injected as a list. Therefore, when validating an offering with the tags A,B,
during processing, there will be a tags variable, in which tags[0] will be tag A
and tags[1] will be tag B.
Usage example for the updateHost API for updating a host with a tag using
JS:
update host id=3d7d8532-d0cf-476c-a36e-1b936d780abb istagarule=true hosttags="tags[0] == 'Premium'"
Via UI:
• Select the host and click on edit:
64
Figure 40: Accessing the host editing
• Create the flexible tag:
Figure 41: Creating the flexible tag in the host
Example for updateStoragePool API usage to update a primary storage with
a tag using JS:
65
update storagepool id=dc916dc7-cc1c-3f3d-905b-b198daf15a79 istagarule=true tags="tags.length == 0 ||
tags[0] === 'test'"
Via UI:
• Select the primary storage and click on edit:
Figure 42: Accessing the primary storage editing
• Create the flexible tag:
66
Figure 43: Creating the flexible tag in the primary storage
It is important to mention that flexible tags are not compatible with Quota
activation rules.
3.8. Managing instance deployment based on their operating system
Apache CloudStack provides two functionalities to direct instance allocation
based on their operating system: preference OS and flexible guest OS.
3.8.1. Hosts Preference OS
Preference OS for hosts is a configuration that lets the operator define a
prioritary operating system for a certain host. This way, hosts that have this
property configured as the same operating system as the VM, will have priority
during the allocation process. The opposite is also valid, hosts with the Prefer-
ence OS configuration set with a different operating system than the VM, will
have lower allocation priority, when compared to the others hosts in the envi-
ronment.
However, this functionality only defines the priority order during the host al-
location, still allowing VMs with disctinct operating systems from the host con-
figuration to be allocated in them. Therefore, it is important to emphasize that
67
this configuration does not make it possible to isolate a host for a determined
operating system.
To access this configuration through the UI:
Figure 44: Access the host being configured
Figure 45: Access the host’s editing form
68
Figure 46: Host’s Preference OS configurations
3.8.2. Flexible guest OS
The Flexible guest OS functionality allows the creation of rules, written in
JavaScript, that filter instances based on their operating systems during the al-
location process of a VM in a host. Thus, using the functionality it is possible to
dedicate a host for one or more operating systems.
To access the configuration through the UI, access the host’s editing form:
Figure 47: Guest OS rule configuration for the host
To define the rules, the variable v mGuestOs is provided and contains the
69
operating system of the instance being allocated, as a string. Thus, to dedicate
a host for Windows VMs, the following rule can be added:
vmGuestOs.toLowerCase().indexOf('windows') >= 0
To dedicate the host for more than one operating system the logical opera-
tor OR, || in JavaScript, can be used, as such:
// dedicar host para VMs Ubuntu e Debian
vmGuestOs.toLowerCase().indexOf('ubuntu') >= 0 ||
vmGuestOs.toLowerCase().indexOf('debian') >= 0
Also, it is possible to add rules that forbid allocating certain operating sys-
tems in the host, for example:
// host accepts only VMs that don't use Ubuntu
vmGuestOs.toLowerCase().indexOf('ubuntu') == -1
3.9. Snapshots
Snapshots are recovery points for a whole VM or just a single disk. They
can be used as a way to recover from failures or even to create templates and
new volumes. In some hypervisors, such as VMware and KVM, they behave like
backups.
In this section we will only approach topics at operation level. For informa-
tion related to the Snapshot usage, check the Apache CloudStack usage docu-
mentation.
3.9.1. Snapshot settings
The main snapshot settings are:
70
Setting Description Default value
max.account.snapshots Maximum number of
snapshots per account
20
max.domain.snapshots Maximum number of
snapshots per domain
Infinite
max.project.snapshots Maximum number of
snapshots per project
20
vmsnapshot.max Maximum number of
snapshots for a VM
10
Table 10: Snapshot settings
3.10. Event and alert audit
Events are generated whenever a user performs an action within the cloud
or when the state of a resource, virtual or physical, is changed. Policy based
events are called alerts by Apache CloudStack.
Using events, it is possible to monitor tasks, jobs and processes in Cloud-
Stack, including their possible errors.
The main event and alert configurations are:
71
Name Description Default
Value
event.purge.delay Days until a created event is removed.
The value 0 indicates that events are
never removed.
15
event.purge.interval Interval, in seconds, until the creation of
a job to remove old events
86400
∗
alert.purge.delay Days until a created alert is removed. The
value 0 indicates that alerts are never re-
moved.
0
alert.purge.interval Interval, in seconds, until the creation of
a job to remove old alerts
86400
alert.smtp.host Host in which the SMTP server is running
alert.smtp.password SMTP server password
alert.smtp.port SMTP server port
alert.smtp.username SMTP server user
alert.email.sender E-mail shown as issuer
alert.email.addresses E-mail receiver, which can be a comma
separated list
∗Value, in seconds, for a full day.
Table 11: Events and alerts settings
3.10.1. Alert e-mails
By default, alerts, as they are considered critical events, can be configured
to send e-mails to the cloud administrators. Possible scenarios where an alert
e-mail will be sent to the cloud administrators are:
• The Management Server has low CPU, memory or storage;
• The Management Server could not communicate with a host for at least
3 minutes;
• A host has low CPU, memory or storage.
3.10.2. Searching and filtering alerts
To view and search for alerts in the web interface you need to:
72
Figure 48: Accessing alerts
3.10.3. Removing or archiving alerts
When selecting the checkbox for an alert, it is possible to remove or archive
it:
Figure 49: Archiving or deletin g an alert
No matter which option is chosen, a pop-up will appear asking for confirma-
tion.
3.10.4. Event and alert removal automation
ACS provides ways to automate the removal of events and alerts through
the global settings event.purge.delay and alert.purge. delay, respectively. The
73
value for these settings stands for the number of days until an event/alert is
removed, since its creation date.
So, if the configuration is set with the value 5, 5 days after the events/alerts
are created they will be automatically removed by ACS. The default value for
event.purge.delay is 15 and for alert.purge.delay is 0. If they are set with the
value 0, events/alerts are never removed.
It is advised to increase event retention to make it possible to audit the en-
vironment in larger time periods, such as 90 or 180 days.
3.11. Storage management within Apache CloudStack
The main goal of this topic is to show in more details the purpose of each
storage type present in ACS, their respective scopes, protocols and providers.
3.11.1. Primary storage
The primary storage is used to store disks used by the hosted virtual ma-
chines. It can operate at host level, cluster level or zone level, with the possibil-
ity to add multiple primary storages to clusters and zones. At least one primary
storage per cluster must exist for the orchestrator to operate properly.
CloudStack was designed to work with a wide variety of storage systems,
also capable of using local disks in the hypervisor’s hosts, as long the selected
hypervisor offers support for this. The support for storage types for virtual
disks depends on the hypervisor used.
Storage type XenServer vSphere KVM
NFS Supports Supports Supports
iSCSI Supports Supports via
VMFS
Supports via
cluster file sys-
tem
Fiber Channel Supports via
storage reposito-
ries
Supports Supports via
cluster file sys-
tem
Local disk Supports Supports Supports
Table 12: Storage protocols supported by each hypervisor
74
ACS offers support for the following protocols:
• NFS
• Shared Mount Point
• RDB
• CLVM
• Gluster
• Linstor
• Custom
CloudStack allows administrators to manage the primary storage based on
the cloud environment’s needs. These can be added, enabled, disabled or set
in maintenance mode, where each of these states presents different effects on
the storage’s management.
3.11.1.1. Adding a primary storage
Before creating primary storages, it is necessary to define the storage type and
protocol used
8
. Next, access must be granted to the storage from the hosts,
validating read and write operations for each of them. To access the primary
storage addition menu:
8
The table 3.11.1 shows the storage types supported by each hypervisor.
75
Figure 50: Accessing the primary storage addition menu
Figure 51: Details for adding a primary storage
• Addition with the NFS protocol:
To operate with NFS, it is only necessary to inform the NFS server’s ad-
dress and the path exported from the server. With this protocol, it is not
necessary to mount the primary storage on the host manually, since ACS
will perform this procedure automatically.
76
Figure 52: Adding a primary storage with NFS
• Addition with Shared Mount Point protocol:
A shared mount point is a path from the local file system for each host
in a given cluster. The path must be the same for all hosts in the cluster,
such as /mnt/primary-st orage. ACS will not mount the primary storage
like when using NFS, therefore the operator must ensure that the storage
is properly made available.
77
Figure 53: Adding a primary storage with Shared Mount Point
3.11.1.2. Disabling a primary storage
When disabling a primary storage, running VMs that have volumes in this Stor-
age Pool will not be affected, however new volumes and VMS created will not
be allocated on it. The migration of volumes to a disabled primary storage is
only possible via API, as through UI the primary storage will be shown as Not
suitable. Yet, volumes present in a disabled primary storage can be migrated
out of it both through API and UI.
78
Figure 54: Disabling a primary storage
3.11.1.3. Maintenance mode for primary storage
When setting a primary storage in maintenance mode all VMs that have vol-
umes in it will be stopped and volumes may be migrated to other primary stor-
age in Up state. Furthermore, it will not be possible to allocate new VMs and
volumes in this primary storage. When starting a stopped VM that has its vol-
ume in a primary storage under maintenance, Apache CloudStack will automat-
ically migrate the volume to an appropriate primary storage. It is important to
notice that the CloudStack Agent will not unmount the primary storage under
maintenance.
Figure 55: Enabling the maintenance mode
3.11.1.4. Behaviour after restarting hosts
If the CloudStack Agent is restarted and the primary storage is either disabled
or in maintenance mode, it will not be automatically mounted. In case the state
is changed after restarting the CloudStack Agent, the primary storage will have
one of the following behaviours, depending on the initial state:
79
• Maintenance mode: If the primary storage leaves the maintenance mode,
it will be mounted again.
• Disabled: If the primary storage is enabled, it will be necessary to restart
the CloudStack Agent for it to be mounted again. If it is in the user’s in-
terest, they can enable the setting mount.disabl ed.storage.pool, which
makes it so that disabled storage pools are automatically mounted in case
of host reboot.
3.11.1.5. Local storage usage
In the global settings, there is a parameter called system.vm.use.local.storage,
that indicates if system VMs (CPVM, SSVM, VR) may use local or shared disks.
When the value of this parameter is set to true, the system VM’s data will be
preserved in an available local disk and if there are no enabled local disks, the
Management Server will show an error message for insufficient capacity during
system VM initialization.
Figure 56: Enabling the usage of local storage for system VMs
80
To utilize local storage for user’s VMs, it is necessary to enable this feature
for the zone. This can be achieved by accessing the zone editing menu, and
enabling the option Enable local storage for user VMs.
Figure 57: Accessing the zone to be edited
Figure 58: Accessing the zone editing menu
81
Figure 59: Enabling the usage of local storage for user’s VMs
After making changes to system.vm.use.local.storage and Enable loca l sto
rage for user V Ms settings it is, necessary to restart the Management Servers
services. The section 8.4.1 exemplifies this process in more detail.
For a new instance to make use of local disks, it is necessary, in addition
to enabling the setting that allows this behaviour in the zone, that the service
offerings used for creating this instance are also set to utilize local disks and
that there are local disks enabled. Otherwise, an error message of insufficient
capacity will be displayed from the Management Server, since there is no avail-
able resource to allocate the instances on. Service offerings can not be edited
directly, therefore it is necessary for them be created with the local storage op-
tion checked. The images below show where to find this option when creating
a new offering.
82
Figure 60: Adding a compute offering with highlight in their Storage Type
83
Figure 61: Adding a disk offering with highlight in the Storage Type
With the option Ena ble local st orage for us er VMs enabled, available local
storages will be listed under the "Infrastructure → Primary storage" menu. To
migrate stopped user VMs’ volumes to local storage, it is necessary to perform
cold migration of the volumes, selecting the available local storages, as the im-
age below shows. However, to migrate running user VMs, it is necessary to
perform hot migration of the volumes. Both processes are exemplified in more
detail in the 3.3 section.
84
Figure 62: Cold volume migration to local storage
For system VMs and virtual routers it is possible to perform the migration
via the migrateVirtualMachineWithVolume API, informing the VM ID, destiny
host ID, volume ID and destiny storage ID. For example:
> migrate virtualmachinewithvolume virtualmachineid=<VM-id> hostid=<host-id>
migrateto[0].volume=<volume-id> migrateto[0].pool=<target-storage-id>
To get the ID from the system VM’s volumes it is necessary to use the listVolum
es API, informing the listall and listsystemvms parameters to true. For example:
> list volumes zoneid=<zone-id> listall=true listsystemvms=true
Therefore, it is necessary for the settings to be properly configured for both sys-
tem VMs and user VMs to utilize the local storage feature, otherwise, there will
be errors for insufficient capacity when allocating resources for new instances.
3.11.2. Secondary storage
The secondary storage items are available for all hosts in the same hierar-
chical level of this storage, which is defined zone wise. This storage type is used
to store:
85
• Templates: Operating systems’ images that may be used to initialize in-
stances and can include additional configuration information, such as in-
stalled applications.
• ISOs: disk images containing data or bootable midia for operating sys-
tems.
• Snapshots: saved copies of a VM’s data (either the whole VM or a specific
volume) that may be used for data recovery or for creating new models.
Each zone has at least one secondary storage, which is shared between all
service points in the zone. CloudStack offers the functionality to automatically
replicate the second storage through zones in a way that is fault tolerant. ACS
was also designed to operate with any scalable secondary storage system. The
only requirement is that the secondary storage system supports the NFS pro-
tocol.
CloudStack provides plugins that allow storing objects from OpenStack Ob-
ject Storage swift.openstack.org and from Amazon Simple Storage Service (S3).
When using these storage plugins, it is necessary to configure the Swift or S3
storage and then setup an NFS secondary storage in each zone. An NFS storage
must be kept in each zone, because most hypervisors can not directly mount S3
storages. This NFS storage acts in each zone as a setup zone which all templates
and other secondary storage data travels through before they are forwarded
to Swift or S3. Swift or S3 storages act as environment wide resources, making
templates and other data available for any zone in the cloud.
The following operations are available for secondary storage management:
3.11.2.1. Adding secondary storages
When creating a new zone, the first secondary storage is created as part of the
process. If needed, the operator may create a new secondary storage.
86
Figure 63: Adding a new secondary storage
Figure 64: Details while adding a new secondary storage
3.11.2.2. Data migration between secondary storages
CloudStack allows migrating data from a secondary storage to another, while
choosing between two migration policies:
• Complete: migrates all data from one secondary storage to another;
• Balance: only migrates a portion of the data, keeping the storages bal-
anced.
87
Figure 65: Migrating data between secondary storages
Figure 66: Details for migrating data between secondary storages
3.11.2.3. Read-only mode for secondary storage
It is possible to define a read-only secondary storage, preventing any additional
templates, ISOs and snapshots to be stored in it.
88
Figure 67: Defining a secondary storage as read-only
3.11.2.4. Read-write mode for secondary storage
If the read-only mode is selected, the option to reactivate the read-write mode
becomes available, having to reactivate it in order to perform new store oper-
ations in the secondary storage.
Figure 68: Defining a secondary storage as read-write
89
3.11.2.5. Secondary storage removal
Lastly, it is also possible to delete secondary storages:
Figure 69: Deleting a secondary storage
Figure 70: Confirming secondary storage removal
3.12. Resource allocation for secondary storage
Apache CloudStack has a functionality called secondary storage selectors
that allows it to specify in which secondary storage templates, ISOs, volumes,
volume snapshots, and backups will be allocated. Currently, the only way to
use this resource is sending requests directly to the API through CloudMonkey.
The Secondary Storage Selectors, through the heuristicrule parameter, use a
conditional block that must be written in JavaScript as an activation rule. When
specifying an activation rule, the secondary storage attributes will always be
90
present for any kind of resource; on the other hand, snapshot, ISO, template,
volume, and backups attributes will only be available for their respective types
based on the purp ose parameter from the createSecond aryStorageSe lector s
API.
The table below presents possible values for each available resource:
Resource Attributes
Secondary Storage id, usedDiskSize, totalDis
kSize, protocol
Snapshot size, hypervisorType
ISO/TEMPLATE format, hypervisorType
Volume size, format
Table 13: Possible values for each resource available in the heuristicrules
To carry out the usage of this feature, firstly, the operator needs to create
the selector via the createSeco ndaryStorageSele c tor API. The created selector
will specify the resource type (ISO, snapshot, template or volume), for which
the activation rule will be validated when the zone allocation is performed, and
the zone where the rule will be applied. It is important to highlight that it is only
possible to have one activation rule for each type within the same zone. Listed
below are some use cases:
1. Allocating a resource type for a specific secondary storage:
function findStorageWithSpecificId(pool) {
return pool.id === '7432f961-c602-4e8e-8580-2496ffbbc45d';
}
secondaryStorages.filter(findStorageWithSpecificId)[0].id
2. Dedicating storage pools for a specific template format;
function directToDedicatedQCOW2Pool(pool) {
return pool.id === '7432f961-c602-4e8e-8580-2496ffbbc45d';
}
function directToDedicatedVHDPool(pool) {
return pool.id === '1ea0109a-299d-4e37-8460-3e9823f9f25c';
}
if (template.format === 'QCOW2') {
secondaryStorages.filter(directToDedicatedQCOW2Pool)[0].id
} else if (template.format === 'VHD') {
secondaryStorages.filter(directToDedicatedVHDPool)[0].id
}
91
3. Directing a volume snapshot with KVM to a specific secondary storage;
if (snapshot.hypervisorType === 'KVM') {
'7432f961-c602-4e8e-8580-2496ffbbc45d';
}
4. Directing resources for a specific domain
if (account.domain.id == '52d83793-26de-11ec-8dcf-5254005dcdac') {
'1ea0109a-299d-4e37-8460-3e9823f9f25c'
} else if (account.domain.id == 'c1186146-5ceb-4901-94a1-dd1d24bd849d') {
'7432f961-c602-4e8e-8580-2496ffbbc45d'
}
The Secondary Storage Selectors feature has 4 APIs: l istSecondaryStorageS
electors, cr eateSecondaryStorageSelectors,updateSeco ndaryStorageSelectors
and removeSecondaryStorageSelectors.
• The listSecondaryStorageSelectors API has the function to show all the
Secondary Storage Selectors available and has the following parameters:
Parameter Description Obligatory Default value
zoneid Id from the zone used in
the selectors search
Yes -
purpose Type from the object
used in the search. Valid
options are ISO, SNAP-
SHOT, TEMPLATE and
VOLUME
Yes -
showRemoved Specify if removed selec-
tors shall be listed
No false
Table 14: listSecondaryStorageSelectors API parameters
• listSecondaryStorageSelectors API usage example
list secondarystorageselectors zoneid=<zone id> purpose=<object type>
• The createSecondarySto rag eSelectors API’s function is to create a Sec-
ondary Storage Selector and it has the following parameters:
92
Parameter Description Obligatory
name Name for selector identi-
fication
Yes
description Selector description Yes
zoneid Zone where the selector
will be active
Yes
purpose Object type for which
the selector will operate.
Valid options are ISO,
SNAPSHOT, TEMPLATE and
VOLUME
Yes
heuristicRule The rule specified in
JavaScript that will direct
the object to a specific
secondary storage; the
rule must always return a
text containing the UUID
for a valid secondary
storage
Yes
Table 15: createSecondaryStorageSelectors API parameters
• createSecondaryStorageSelectors API usage example
create secondarystorageselector name=<selector name> description=<description> zoneid=<zone id>
heuristicrule=<activation rule> purpose=<object type>
• The upd ateSecondaryStorageSelectors API’s function is to update a Sec-
ondary Storage Selector and it has the following parameters:
Parameter Description Obligatory
id Selector identifier of the
secondary storage
Yes
heuristicRule The new rule specified in
JavaScript; the rule must
always return a text con-
taining the UUID for a
valid secondary storage
Yes
Table 16: updateSecondaryStorageSelectors API parameters
93
• updateSecondaryStorageSelectors API usage example
update secondarystorageselector id=<selector id> heuristicrule=<activation rule>
• The removeSecondaryStorageSelectors API’s function is to remove a Sec-
ondary Storage Selector and it has the following parameters:
Parameter Description Obligatory
id identifier of the selector
to be removed
Yes
Table 17: removeSecondaryStorageSelectors API parameters
• removeSecondaryStorageSelectors API usage example
remove secondarystorageselector id=<selector id>
4. Service Offerings
CloudStack is a cloud orchestration platform that enables centralized man-
agement of virtualized infrastructure environments. To create virtual machines
(VMs), it is necessary to define their specifications, such as CPU core count,
memory size, disk capacity, and other parameters. In CloudStack, these spec-
ifications are organized into service offerings, which standardize VM configu-
rations. This approach streamlines instance provisioning, makes resource us-
age easier to track, and, when required, allows the application of consumption-
based billing policies.
4.1. Compute offerings
Compute offerings are offerings related to the computional resources of a
VM, such as CPU and RAM. A compute offering also has a disk offering, that
defines the characteristics for the VM root disk (when it is created based on a
template).
There are some settings that limit some aspects from compute offerings,
such as:
94
Configuração Descrição Valor
Padrão
vm.serviceoffering.cpu.cores.max Maximum amount of CPU
cores when creating offerings
or VMs. 0 means boundless.
0
vm.serviceoffering.ram.size.max Maximum amount of RAM
when creating offerings or
VMs. 0 means boundless.
0
For information related to the usage of Compute Offerings, check the Apache
CloudStack usage documentation.
4.2. Disk Offerings
Disk offerings are service offerings related to storage resources of a VM. Vir-
tual machines created from a template will utilize disk offerings to create extra
disks, while those created with ISOs will need a disk offering to create their root
disk.
To create a disk offering:
95
Figure 71: Starting to create a new disk offering
Clicking on the button Add Disk Offering will open a form to fill the informa-
tion:
96
Figure 72: Creating the new disk offering
CloudStack has three types of provisioning:
• Thin Provisioning: disk of this type will be allocated with the minimum size
necessary and may grow accordingly as needed, up to their limit.
• Sparse Provisioning: disks of this type will start with the size defined in the
offering or during the disk creation; however, they may contain old data.
Their creation will be fast because old data is not erased or overwritten,
but it will be necessary to do that before writing new data. The perfor-
mance of the first writes for this disk type will be slow.
• Fat Provisioning: disks of this type will start with the size defined in the
offering or during the disk creation and all disk space will be overwritten,
97
erasing any data within the blocks. When compared with sparse provision-
ing, disks of this type will take longer to be created because of the data
cleaning process; however the performance for their first writes will be
faster.
The Quality of Service (QoS) is not implemented by the offerings by default,
however it is possible to utilize a QoS from a hypervisor, in which read and
write rates done by the hypervisor, or storage, are determined, defining the
minimum and maximum IOPS for storages. To utilize a hypervisor QoS it is
necessary that they support this funcionality. For some hypervisors some def-
initions may not be implemented; thus, it is necessary to validate them before
applying in production environments.
Storage tags are used to assign volumes to compatible storages (see section
3.7.2).
There are settings that limit some aspects of the disk offerings:
Configuração Descrição Valor
Padrão
custom.diskoffering.size.min Minimum size (in GB) for customiz-
able disk offerings.
1
custom.diskoffering.size.max Maximum size (in GB) for customiz-
able disk offerings.
1024
After created, only a few attributes of the disk offering may be changed.
• Edit:
98
Figure 73: Starting to edit the disk offering
Figure 74: Editing a disk offering
• Update access:
Figure 75: Starting to update disk offering accesses
99
Figure 76: Updating the disk offering accesses
• Edit order:
Figure 77: Changing the disk offerings order
To delete a disk offering, access it and click the trash can icon:
Figure 78: Starting to delete a disk offering
Figure 79: Deleting a disk offering
100
If there are volumes using a disk offering, they will keep working normally,
as well as VM migration and restart processes, because CloudStack copies the
disk offering characteristics to the disk. However, it will not be possible to create
new volumes using it.
More information may be found on the official documentation.
4.2.1. Limiting IOPS e BPS in disk offerings
During the creation of a disk offering, it is possible to limit the volume IOPS
and BPS rates through the QoS type option.
Figure 80: Selecting a QoS type
When determining limits for volume IOPS and BPS rates, it is important to
be careful with the following details:
• Control over the size of I/O operations:
101
When an operator sets a limit for IOPS rate, all I/O operations, regardless
of size, are treated similarly, and, consequently, users may exploit this to
circumvent the limits set. To prevent this, the operator needs to specify
the block size for each operation. Assuming that the IOPS limit for a VM is
1000, the hypervisor will be limiting for 1000 I/O operations per second,
independently of the size of the IO request. Thus, requests with bigger
block sizes will benefit from this. To illustrate this, some tests with the fio
tool were performed.
A VM was deployed with a disk offering limiting both its read and write
IOPS to 1000. The fio command was executed with a block size of 64 KiB
and then executed again but with a block size of 128 KiB. Although the to-
tal amount of IO operations were different in those executions, both per-
formed approximately 1000 IOPS. This happens because of the execution
time necessary for each one. Using a bigger writing block, double the read
and write speed was achieved, even though both executions were limited
to 1000 IOPS. The table bellow shows a briefly comparison between the
values discussed.
Block
Size
Write
(IOPS)
Write
(Mib/s)
Read
(IOPS)
Read
(Mib/s)
64 KiB 1008 63 337 21.1
128 KiB 1016 127 346 43.3
Table 18: Comparison between test results using block size of 64 KiB and 128 KiB
ACS currently does not externalize the operation block size, however, it
is possible to achieve the same objective setting out a read and write
amount in BPS (bytesreadrate and byteswriterate parameters).
102
Figure 81: Limiting a BPS ratio
Considering the same use case above, specifying the diskBytesReadRate
to 104857600 (100 MiB) and diskBytesWriteRate to 31457280 (30 MiB), the
usage for block size of 128 KiB would be limited by these values. There-
fore, when a BPS and IOPS limit is defined, the VM will be limited by the
first limit reached. In this case, when using a block size of 128 KiB, the BPS
limit will be reached first and, consequently, the VM will never reach 1000
IOPS. The BPS limitation for the block size of 64 KiB will have no effect, be-
cause the IOPS limit will be reached first. The table bellow briefly shows
the data mentioned.
103
Block
Size
Write
(IOPS)
Write
(Mib/s)
Read
(IOPS)
Read
(Mib/s)
64 KiB 1008 63 337 21.1
128 KiB 710 88.8 242 30.3
Table 19: Comparison between test results using BPS limits
Considering the situation above, it is recommended using the parameters
diskBytesReadRate and diskBytesWriteRate together with the IOPS limits
to limit volume read and write speeds.
• I/O bursts;
A burst is when the read and write limits are exceed for some time to al-
low a greater performance during intense activities. This functionality is
present in ACS, however, it is only available through the APIs
9
(createSe
rviceOffering and createDiskOfferin g). The parameters that allow config-
uring this functionality are:
9
For more details on how to send requests directly to the API ??
104
Parâmetro Descrição
bytesreadratemax Defines the maximum
reading speed for the
burst in BPS
bytesreadratemaxlength Defines the maximum
amount of time for the
BPS read burst in sec-
onds
byteswritemax Defines the maximum
writing speed for the
burst in BPS
byteswritemaxlength Define the maximum
amount of time for the
BPS write burst in sec-
onds
iopsreadratemax Defines the maximum
reading speed for the
burst in IOPS
iopsreadratemaxlength Defines the maximum
amount of time for the
IOPS read burst in sec-
onds
iopswriteratemax Defines the maximum
writing speed for the
burst in IOPS
iopswritemaxlength Define the maximum
amount of time for the
IOPS write burst in sec-
onds
4.3. Network offerings - throttling
Process of controlling network access and bandwidth based on certain rules.
This behaviour is controlled by CloudStack in the guest networks based on net-
work rates parameters (default transfer rates, measured in Mbps, allowed in
a guest network). This parameter determines maximum limits for network us-
age; if the current usage is higher than the defined limits, the access will not be
granted.
It is possible to control network jamming and accounts using the network
105
beyond the defined limits through bandwidth limitations. The network rate for
your cloud can be adjusted in the following way:
• Network offering
• Service offering
• Global settings
If the network rate is defined as NULL in the service offering, the value set
in the global setting vm.network.throttling.rate will be applied. If the value is
defined as NULL for the service offering, the value set in the global setting net
work.throttling.rate will be considered.
For public networks, either storage or predefined management, the network
rate is defined as 0, which implies that they have unlimited bandwidth as de-
fault. Now for guest networks, the network rate is defined as NULL.
The network rate defined for a network offering that is used by a specific
network within CloudStack, is used for the traffic modelling policies for a port
group, i.e., VRs connected to this port group and, consequently, the network
instance also connected. However, an instance that was implemented with a
compute offering that defines a network rate, will be connected to the port
group imposed for such traffic modelling policy, in which the network rate will
be used.
4.4. System offerings
System offerings are similar to compute offerings but intended for system
VMs: console proxy VMs, secondary storage VMs and virtual routers. Cloud-
Stack comes with some offerings intended for each of those VMs by default.
However, the root admin user may create new offerings and change which of
them one of these machines is using.
An example would be when a guest network is created, in which the virtual
router uses the system offering as their network offering. To satisfy a new net-
work offering it is possible to define resources that will be suppplied by the
106
virtual router. This way, all VRs in this network will utilize this new service of-
fering.
Figure 82: Default system offerings
4.4.0.1. Creating system offerings
Adding a new system offering:
107
Figure 83: Starting to create a new system offering
By clicking on the Add System Service Offering button, a form for inserting
the information will open:
108
Figure 84: Creating a new system offering - 1 - continues
Via UI, the option System VM Type shows only the following types: Domain
router, Console proxy and Se condary storage VM. However, via API it is also
possible to create offerings of the internalloadbalancervm type.
Figure 85: Creating a new system offering - 2
• Host tags will be used to direct the system VMs to compatible hosts (check
109
section 3.7.1).
• Storage tags will be used to direct volumes to compatible storages (check
section 3.7.2).
4.4.0.2. Editing system offerings
After the creation of a system offering only a few attributes can be changed.
• Editing:
Figure 86: Starting to edit the system offering
Figure 87: Editing the system offering
• Change order:
110
Figure 88: Changing system offering order
4.4.0.3. Removing system offerings
To remove a system offering, access it and click the trash can icon (default
offerings can not be removed):
Figure 89: Starting the system offering removal
Figure 90: Removing a system offering
If there are system VMs running under the system offering they will keep
working normally, as CloudStack copies their system offering characteristics to
the system VMs.
111
4.4.0.4. Changing the system offering of a system VMs
It is possible to change the offering that a system VM uses with the changeSer
viceForSystemVm API. With the root admin logged in on CloudMonkey
1011
:
stop systemvm id=<system_vm_id>
change serviceforsystemvm id=<system_vm_id> serviceofferingid=<system_offering_id>
start systemvm id=<system_vm_id>
However, if the VM is recreated it will use the default offering again. To
change the system offering used by a system VM when created it is necessary
to indicate the offering uuid in the global settings. The null value indicates that
the default CloudStack offering will be used.
Setting Description Value
Deafult
consoleproxy.service.offering uuid of the offering used by console
proxy VMs by default.
null
internallbvm.service.offering uuid of the offering used by internal
load balancer VMs by default.
null
router.service.offering uuid of the offering used by virtual
routers by default
null
secstorage.service.offering uuid of the offering used by sec-
ondary storage VMs by default.
null
Table 20: System offerings settings for system VMs
After changing the values and restarting the Management Servers, destroy-
ing the system VMs will cause CloudStack to recreate them with the new offer-
ings.
4.5. Backup offerings
These offerings are intended for importing backup offerings from providers
outside CloudStack. Currently, the only supported provider is Veeam, for VMware.
10
To change the service offering, it is necessary for the system VM to be stopped. When
turning it on it will already be using the new offering.
11
When stopping the system VM it will automatically start after some time. To keep it shut
down, it is necessary to disable the zone.
112
CloudStack has a fake provider (Dummy) just for sufficing logical requirements,
however, when offerings are imported from this provider nothing will actually
happen.
4.5.0.1. Enabling backup offerings
To enable the backup offerings feature, it is necessary to define the backup.fr
amework.enabled global setting to true. Besides, there are also other settings
available for this feature:
Setting Description Value
Deafult
backup.framework.enabled Defines if the backup offering
feature is enabled or not.
false
backup.framework.provider.plugin Defines which provider will be
used.
dummy
backup.framework.sync.interval Defines the interval, in sec-
onds, for task synchronization
between CloudStack and the
provider.
300
Table 21: Backup settings
The settings backup.framework.enabled and backup.framework.provider.p
lugin work within the zone scope.
There are also specific settings for Veeam providers:
113
Setting Description Default Value
backup.plugin.veeam.password Password for
Veeam access.
backup.plugin.veeam.request
.timeout
Timeout, in seconds,
for Veeam requests.
300
backup.plugin.veeam.url Veeam URL. https://localhost:9398/api/
backup.plugin.veeam.username Username for
Veeam access.
administrator
backup.plugin.veeam.validate
.ssl
If set to true, will
validate the SSL cer-
tificate in https/ssl
connection with the
Veeam API.
false
Table 22: Backup settings for Veeam
Furthermore, the Veeam server needs to have SSH installed and running on
port 22.
It is also important to highlight that when using Veeam, its templates should
not be used if they contain special characters. The reason being that using
Regex to space such characters causes errors on Veeam.
The following characters are considered special: blank space, \, #, $, (), *, +,
., ?, [], ^, {}, | and accented letters. The characters “-" and “_" are not considered
special characters and can be used normally.
With the backup.framework.enabled global setting set to true, when restart-
ing CloudStack a new sub-tab will show up under Service Offerings:
Figure 91: Backup offering tab
114
4.5.0.2. Importing backup offerings
To import a backup offering:
Figure 92: Starting to import a new backup offering
When clicking on the Import Backup Offering button a form will show up to
fill the information needed:
Figure 93: Importing a new backup offering
115
The values shown in the External Id field are related to the provider config-
ured for the selected zone. If the Allow User Driven Backups field is checked,
the user may schedule backups or perform them manually.
4.5.0.3. Backup offering removal
It is not possible to edit a backup offering, only remove it:
Figure 94: Starting backup offering removal
Figure 95: Removing the backup offering
It is not possible to remove a backup offering if it is already being used by a
VM.
4.5.0.4. Using backup offerings
To use a backup offering, just select a VM and:
Figure 96: Starting to assign a VM to a backup offering
116
Figure 97: Assigning a VM to a backup offering
To perform a manual backup you just need to select the Start Backup option:
Figure 98: Starting manual backup for a VM
Figure 99: Performing a manual backup for a VM
To schedule a backup, select the Configure Backup Schedule option:
Figure 100: Starting backup scheduling
117
5. Apache CloudStack settings
This section presents concepts related to the ACS settings, as well as settings
that are frequently changed for environment customization.
5.1. Settings scopes
One of Apache CloudStack’s main characteristics is its flexibility regarding
environment configuration. This is achieved, in part, through the use of settings
scopes, which are setting groups that affect CloudStack’s behaviour in different
levels. This approach provides greater flexibility in ACS configuration and helps
optimize performance and efficiency for the system as a whole.
ACS provides settings in the following scopes: Global, Zone, Cluster, Domai
n, Account, ManagementServer, StoragePool and ImageStore.
The Global scope is the broadest one, embracing all the environment’s set-
tings. The settings defined under this scope are applied to all of the cloud subdi-
visions, allowing the user to uniformly customize the settings for the whole en-
vironment. Similarly, other scopes indicate that settings belonging to them are
applied to all resources within their categories. For example, when the value
for a setting within the
Accoun t scope is changed, this change will only affect
the current account. In turn, if the change was made in the Domain scope, the
change will affect all members within that domain.
Beyond setting scopes, there are other settings that are useful during the
ACS customization process.
• enable.a cco unt .settings.for.domain: Allows settings under the account
scope to also be visible at domain level. Furthermore, if an account setting
is set by the administrator, that will be the value considered. Otherwise,
the value from the domain scope that will be considered. If the value un-
der the domain scope is not set, the value from the global setting will be
selected. If this setting is set to false, if a setting at account level is not set
119
by the administrator, the value considered will be the one from the global
setting.
• enable.domain.settings.for.child.domain: if enabled, this setting allows
child domains (subdomains) to inherit the settings defined by their par-
ent domain. For example, if a parent domain defines a certain setting
to limit the CPU usage to 50%, all child domains will inherit this setting,
unless they have a specific setting that overwrites the one from its par-
ent domain. This eases domain and account configuration and makes
the process more efficient, allowing administrators to define settings at a
higher level and inherit them to lower levels. As default, this setting comes
disabled.
5.2. Global settings that control the primary storage usage
The following global settings manage the CloudStack behaviour when a stor-
age is close to running out of storage capacity:
120
Setting Description Default
Value
cluster.storage.allocated.capacity.notificationthreshold Value between 0
and 1 to indicate
the storage allo-
cation percentage
that will trigger
sending alerts for
low available stor-
age space.
0.75
cluster.storage.capacity.notificationthreshold Value between 0
and 1 to indicate
the storage usage
percentage that
will trigger sending
alerts for low avail-
able storage space.
0.75
pool.storage.allocated.capacity.disablethreshold Value between 0
and 1 to indicate
the storage allo-
cation percentage
that will trigger it to
disable itself.
0.85
pool.storage.capacity.disablethreshold Value between 0
and 1 to indicate
the storage usage
percentage that
will trigger it to
disable itself.
0.85
Table 23: Cloudstack behavior settings in case of storage running out of space
More information about primary storage management can be found in 3.11.1.
121
5.3. Settings to limit resources
Setting Description
max.account.public.ips Maximum number of public IPs for an ac-
count
max.account.snapshots Maximum number of snapshots for an
account
max.account.templates Maximum number of templates for an ac-
count
max.account.user.vms Maximum number of VMs for an account
max.account.volumes Maximum number of disks for an ac-
count
max.template.iso.size Maximum size for a template or ISO (GB)
max.volume.size.gb Maximum size for disks (GB)
max.account.cpus Maximum number of CPUs for an ac-
count
max.account.ram Maximum amount of RAM (MB) for an ac-
count
max.account.primary.storage Maximum space (GB) in the primary stor-
age that may be used by an account
max.account.secondary.storage Maximum space (GB) in the secondary
storage that may be used by an account
max.project.cpus Maximum number of CPUs for a project
max.project.ram Maximum amount of RAM (MB) for a
project
max.project.primary.storage Maximum space (GB) in the primary stor-
age that may be used by a project
max.project.secondary.storage Maximum space (GB) in the secondary
storage that may be used by a project
Table 24: Resource limitation settings
5.4. Settings that control Kubernetes usage
This section will only approach global settings necessary for using Kuber-
netes. For information related to the usage of Kubernetes, access the Apache
CloudStack usage documentation.
5.4.1. Enabling Kubernetes integration
Firstly, its important to pay attention to the versions used by both Cloud-
Stack and Kubernetes. New Kubernetes versions are released more frequently
122
than CloudStack versions, therefore new Kubernetes versions may not properly
work with the current CloudStack version.
The Kubernetes integration is disabled by default. To enable it, access the
global settings and change the setting cloud.kubernetes.service.enabled to true
and then restart the Management Server.
As soon as integration is enabled, the new APIs will be available and new
Kubernetes tabs will show up in the UI.
5.4.2. Kubernetes clusters creation
To create a Kubernetes cluster, the Management Server must have access
to the public IPs from the networks used for virtual machine provisioning in the
cluster. This is accomplished by configuring the global parameter endpoint.url
o match the domain used for accessing ACS, as demonstrated in the following
example:
Figure 104: Global setting endpoint.url
For a new network to be created when none is selected during the creation
of the Kubernetes cluster, the global setting cloud.kubernetes.cluster.network
.offering must be defined, set with the desired offering to be used as default.
123
6. UI customization
This section demonstrates how to customize the Apache CloudStack web
interface in different ways and preferences.
6.1. Changing logo and other elements
Note: this is the legacy GUI customization model. We recommend using a
theme management system (Section 6.3) for greater customization and better
management.
It is possible to change/customize some aspects from the CloudStack web
interface, including the logo, banner, page title and footers. This describes the
processes for customizing the logo and banner. Before any changes are made,
it is advisible to perform a backup from the folder containing the following set-
tings, in case it is necessary to revert the changes made.
In order to customize logos and banner, follow this procedure:
1. Login into the Management Server via ssh:
user@machine:~$ ssh username@management-server-ip
# A practical example:
user@machine:~$ ssh root@192.168.103.168
2. Browse to the directory: /usr/share/clou d st a c k -m a n ag e m e nt / w eb a p p /a s
sets:
user@machine:~$ cd /usr/share/cloudstack-management/webapp/assets
# In case a "Permission denied" error occurs, execute the command:
user@machine:~$ sudo su
# And then browse to the CloudStack logs directory.
3. The images used by CloudStack may be found in this directory. If you
want to change the logo or banner, just change the files logo.svg and ban
ner.svg, respectively and update them in the config.json settings file. The
procedure of changing these images consists of uploading the client’s logo
and banner to the Management Server via scp.
4. Repeat this process in each existent Management Server in the infrastruc-
ture.
124
5. If any procedure on the Management Servers results in a "Permission de-
nied" error, execute the command:
user@machine:~$ sudo su
Then repeat the process that caused the error.
Some notes:
• There is a file named cloud.ico in the directory /usr/sh are/cloudstack-m
anagement/webapp, responsible for the icon shown in the browser tab,
that can also be changed.
• In this same directory, there is a file named c on f i g. j s on, that can be used
to change the page title, as well as the footers and other details.
• The CloudStack UI has a cache period. If changes are not visible it is rec-
ommended to clear the cache and check again if the changes made were
applied.
• As the CloudStack web interface, at least until the writting of this docu-
ment, is relatively new and is still under development, it may be necessary
to repeat this procedure when the current CloudStack version is updated.
An example of CloudStack customization:
{
"apiBase": "/client/api",
"docBase": "http://docs.cloudstack.apache.org/en/latest",
"appTitle": "SC Clouds",
"footer": "Welcome to the portal with custom aspects",
"logo": "assets/logo.svg",
"banner": "assets/banner.svg",
"error": {
"403": "assets/403.png",
"404": "assets/404.png",
"500": "assets/500.png"
},
// ...
}
125
Figure 105: Customized banner
Figure 106: Customized footer
6.2. Changing logo when resizing the page
When the user resizes the page to some extent, or clicks to collapse the side
menu, the environment logo set is cut to adjust to the menu size.
The default ACS logo is designed so that when cut it becomes sort of a "mini
logo", other logos, however, may not present this behaviour as shown below:
Figure 107: Whole logo
126
Figure 108: Cut logo
To customize the logo change behaviour the settings in the file /usr/ shar
e/cloudstack-management /webapp/config.json must be edited in each of the
Management Servers of the environment.
Properties related to this improvement (mini logo) are shown below. Notice
that other properties in this file were omitted with [...] to avoid cluttering the
document:
{
[...]
"logo": "assets/logo.svg",
"minilogo": "assets/mini-logo.svg",
[...]
"theme": {
"@logo-background-color": "#ffffff",
"@mini-logo-background-color": "#ffffff",
[...]
"@logo-width": "256px",
"@logo-height": "64px",
"@mini-logo-width": "80px",
"@mini-logo-height": "64px",
}
}
Figure 109: Mini logo
Note: To apply the effects the page’s cache must be refreshed.
127
6.3. Theme management
Themes are an alternative for UI customization in CloudStack. They have
distinct functionalities and purposes, designed, but not limited, to answer the
needs from cloud providers that wish to implement a resale model and pro-
vide a White Label cloud system. It is possible to manage themes at different
application levels, allowing greater customization freedom.
This functionality allows managing themes at account, domain and internet
common name level. It is possible to setup CSS rules and attributes in JSON for-
mat that will be used to load the theme dynamically. If no theme is registered,
the GUI will use the current environment settings.
Listed below are the APIs used to manage themes:
6.3.1. Theme creation
The API createGuiTheme is only accessible for root admin accounts and al-
lows themes to be created at different scopes. This API has the following pa-
rameters:
Parameter Description Obligatory Default
value
name Name to identify the theme. Yes -
description Theme description. No null
css CSS imported by the GUI to cor-
respond to the theme access set-
tings.
No null
jsonconfiguration JSON containing the settings to
be imported by the GUI if they
match the theme access settings.
More details about the JSON set-
tings may be found in the Section
6.3.3.2.
No null
commonnames Comma-separated set of internet
common names that have access
to the theme.
No null
128
Parameter Description Obligatory Default
value
domainids Comma-separated list of do-
main IDs that have access to the
theme.
No null
accountsids Comma-separated list of account
IDs that access to the theme.
No null
ispublic Defines if the theme is accessible
to anyone, when only the commo
nnames is informed. If the doma
inids or accountids are informed,
it is considered false.
No true
Table 25: createGuiTheme parameters
If c ommonnames, domainids and accountids are not informed, the theme
used will be the default; only one default theme can exist and it is automati-
cally public. If there is no corresponding theme, the GUI will use the current
environment settings as fallback.
Despite the fields css and jsonconfiguration not being mandatory, it is nec-
essary for at least one of these fields to be informed in order to create the
theme.
Subsection 6.3.5 exemplifies theme creation and common interface cus-
tomizations.
6.3.2. Theme list
The listG uiThemes API is accessible to any user and searches for themes
based on the parameters and the requesting user’s access. It has the following
parameters:
129
Parameter Description Obligatory Default
value
id Theme ID. No null
name Theme name. No null
commonname Filtered internet common name. No null
domainid Filtered domain ID. No null
accountid Filtered account ID. No null
listall If used, lists all themes. No false
listremoved If used, lists removed themes. No false
ispublic If used, lists public themes. By de-
fault, all will be listed.
No true
listonlydefaulttheme Lists only the default theme. If set
to true, then all other parameters
will be ignored.
No false
Table 26: listGuiThemes parameters
To allow the theme to be shown on the login page, it is possible to call this API
without authentication; however there are limitations to this use case. An unau-
thenticated call to the listGuiThemes API always retrieves the default theme or
the latest active public theme that corresponds to the internet common name
requested to the API. In addition to that, all API parameters are ignored.
One possible way to perform queries is shown in the example below:
(admin) > list guithemes listall=true
{
"count": 1,
"guiThemes": [
{
"created": "2023-09-15T16:20:18+0000",
"css": ".layout.ant-layout .header {background: purple;}",
"id": "a9a05158-2870-48b7-877a-626badde4b28",
"ispublic": true,
"jsonconfiguration": "{\"banner\":\
"https://res.cloudinary.com/sc-clouds/image/upload/v1645707938
/identidade/scclouds-logo_f56v2y.png\", \
"logo\":\"https://res.cloudinary.com/sc-clouds/image/upload/v1645707938
/identidade/scclouds-logo_f56v2y.png\", \"favicon\":\
"https://gitlab.scclouds.com.br/uploads/-/system/appearance/favicon
/1/scclouds-avatar.ico\"}",
"name": "example-theme"
}
]
}
130
6.3.3. Updating a theme
The updateGuiTheme API is only accessible for root admin accounts and
allows updating an already existing theme. It has the following parameters:
Parameter Description Obligatory Default
value
id ID from theme for update. Yes -
name Name to identify the theme. No null
description Theme description. No null
css CSS imported by the GUI when
the access settings match those
from the theme.
No null
jsonconfiguration JSON containing the settings to
be imported by the GUI if they
match the theme access settings.
More details about the JSON set-
tings may be found in the Section
6.3.3.2.
No null
commonnames Comma-separated set of internet
common names that can retrieve
the theme.
No null
domainids Comma-separated list of domain
IDs that can retrieve the theme.
No false
accountids Comma-separated ID set of
accounts that can retrieve the
theme.
No false
ispublic Boolean parameter that defines if
the theme access is open to any-
one, when only the commonnam
es is informed. If the domainids
or acc ountids are informed, it is
considered false.
No true
Table 27: updateGuiTheme parameters
Below is presented an update example for the theme listed in the section
6.3.2. We highlight that the update will overwrite the fields css and jsonconfigu-
ration, as can be seen in the API call return. Besides, to remove a parameter, it
is necessary to explicitly pass it as an empty string.
131
(admin) > update guitheme id=a9a05158-2870-48b7-877a-626badde4b28
css='.layout.ant-layout .header {background: orange;}'
{
"guiThemes": {
"created": "2023-09-15T16:20:18+0000",
"css": ".layout.ant-layout .header {background: orange;}",
"id": "a9a05158-2870-48b7-877a-626badde4b28",
"ispublic": true,
"jsonconfiguration": "{\"banner\":\"https://res.cloudinary.com/sc-clouds/image
/upload/v1645707938/identidade/scclouds-logo_f56v2y.png\", \
"logo\":\"https://res.cloudinary.com/sc-clouds/image/upload/v1645707938
/identidade/scclouds-logo_f56v2y.png\", \"favicon\":\
"https://gitlab.scclouds.com.br/uploads/-/system/appearance/favicon/1
/scclouds-avatar.ico\"}",
"name": "example-theme"
}
}
6.3.3.1. CSS field
In the CSS field it is possible to add customized styles to the CloudStack UI,
directly applied in the HTML tags or within the <style> tags, using the CSS lan-
guage. Another option is to use @import, which makes it possible to add a style
through a URL to the CSS file.
@import <url|string> <media-queries-list>
• url: A URL or string that represents the location of the resource to import.
The URL may be absolute or relative.
• media queries: Comma-separated list of midia queries that set the appli-
cation of the CSS rules defined in the given URL.
Using @import may facilitate adding and organizing themes, as ACS does
not have a mechanism to control CSS files, which limits the operator’s options.
For this purpose, importing a versioning system is simpler and easier, despite
minor disadvantages, such as needing to perform additional HTTP requests,
which have little to no performance impact.
6.3.3.2. JSON settings
The JSON settings follow the current ACS standard, config.json; however, limited
in some attributes:
132
Attribute Description.
appTitle Web page title.
favicon Web page favicon.
footer Web page footer.
loginFooter Web page login footer.
logo Local or external link for the image to be presented as the left
bar logo.
minilogo Local or external link for the image to be presented as the left
bar logo miniature, when collapsed.
banner Local or external link for the image to be presented as the login
background.
error.403 Local or external link for the image to be presented when re-
ceiving an error 403.
error.404 Local or external link for the image to be presented when re-
ceiving an error 404.
error.500 Local or external link for the image to be presented when re-
ceiving an error 500.
plugins Set of plugin objects.
Table 28: jsonconfiguration attributes
The plugins structure is as follows:
Attribute Description
name Plugin name.
path Link to the web page to add the plugin.
icon Icon for the plugin.
isExternalLink Determines if the plugin refers to an external link.
Table 29: jsonconfiguration plugin attributes
jsonconfiguration example:
{
"appTitle": "CloudStack",
"favicon": "cloud.ico",
"footer": "Licensed under the Apache License, Version 2.0.",
"loginFooter": "",
"logo": "assets/logo.svg",
"minilogo": "assets/mini-logo.svg",
"banner": "assets/banner.svg",
"error": {
"403": "assets/403.png",
"404": "assets/404.png",
"500": "assets/500.png"
},
"plugins": [
133
{
"name": "Apache CloudStack",
"path": "https://cloudstack.apache.org/",
"isExternalLink": "true",
"icon": "https://cloudstack.apache.org/images/favicon.ico"
}]
}
If any other attribute is specified, it will be ignored.
6.3.4. Removing a theme
The re moveGuiTh eme API is only accessible for root admin accounts and
allows them to remove themes. It is necessary to give the theme ID for removal,
as shown in the example below:
(admin) > remove guitheme id=a9a05158-2870-48b7-877a-626badde4b28
{
"success": true
}
6.3.5. Common UI customization examples
This topic aims to present the most common UI customization examples,
besides providing general tips to ease the stylization process.
6.3.5.1. Creating themes with external stylization file
Firstly, it is necessary to execute the cre ateGuiTheme API via CloudMonkey to
create the theme. As mentioned in the 6.3.3.1 topic, it is advisible for the CSS @i
mport at-rule to be used to configure the stylization. Through that, it is possible
to introduce CSS properties in a separated file instead of inserting them via
terminal.
create guitheme name="theme" css="@import url('<css-file-url>')"
6.3.5.2. Notes about style conflicts
When styling the ACS interface it is common for conflicts to appear between
the styles that are being inserted and the ones in use. They occur when two or
more selectors have conflicting styles applied to the same element.
To solve this issue it is necessary for the selector being inserted to have the
highest specificity possible. As a last resort the !important rule can be used.
134
Styles declared with it will be preferred over any other style. As shown in the
following examples, using this rule turns into a common practice.
6.3.5.3. Adding fonts
In CSS, fonts may be imported. For this example the font Poppins was utilized.
@import url("<font-url>");
* {
font-family: 'Poppins' , Courier !important;
}
6.3.5.4. Using CSS variables
It is recommended adding CSS variables to abstract property values and keep
consistency during stylization. For that, they can be inserted in the CSS file,
preferably in the :root selector. The following variables were used in this ex-
ample:
:root {
--main-green: #3E7C59;
--main-red: #ae0000;
--main-default: #327355;
--main-primary-light: #db4e2d22;
--main-primary-dark: #db4e2d;
--main-secondary-light: #E9782622;
--main-secondary-dark: #E97826;
--main-gray-light: #eee;
--main-linear-colora: #ddd;
--main-linear-colorb: #ddd;
--main-image-menu: url("<figure-url>");
--main-image-login: url("<figure-url>");
--main-image-icon: url("<figure-url>");
}
6.3.5.5. Login page
/* Login page logo */
.userLayout .user-layout-header {
min-height: 100px;
background-image: var(--main-image-login);
background-repeat: no-repeat;
background-size: 250px;
background-position: top center;
}
/* ACS logo hidden by default */
.userLayout .user-layout-header > img {
display: none;
}
135
/* Page background-color, excluding footer */
.ant-tabs-tab,
.user-layout {
background-color: var(--main-gray-light) !important;
}
/*
background-color applied in the HTML tag.
However, in practice, the property will be applied in the login page footer
*/
html {
background-color: var(--main-secondary-dark);
}
/* continuous border under the tabs */
.ant-tabs-top-bar .ant-tabs-nav-scroll {
border-bottom: 3px solid var(--main-primary-dark);
width: 90.5%;
margin: 0 auto;
}
/* Tabs title color ("Portal login", "Single sign-on", ...) */
.user-layout .ant-tabs-top-bar .ant-tabs-nav-scroll .ant-tabs-tab {
color: var(--main-primary-dark);
border-bottom: 0px solid var(--main-primary-dark);
}
/* active tab border-bottom */
.user-layout .ant-tabs-top-bar .ant-tabs-nav-scroll .ant-tabs-tab.ant-tabs-tab-active {
border-width: 3px;
}
/* Tab styles when :hover */
.user-layout .ant-tabs-top-bar .ant-tabs-nav-scroll .ant-tabs-tab:hover {
background-color: var(--main-secondary-dark) !important;
color: #fff;
border-width: 3px;
}
/* Necessary to remove the ACS default border */
.ant-tabs-ink-bar.ant-tabs-ink-bar-no-animated {
display: none !important;
}
/* All buttons, except for those that have the class .ant-btn-icon-only*/
button.ant-btn:not(.ant-btn-icon-only){
background-color: var(--main-primary-dark) !important;
color: #fff;
border: 0px;
}
/* :hover buttons */
button.ant-btn:not(.ant-btn-icon-only):hover {
background-color: var(--main-secondary-dark) !important;
color: #fff;
}
/* Inputs */
.ant-input-affix-wrapper,
.ant-input-affix-wrapper input {
background-color: var(--main-gray-light) !important;
}
/* Inputs placeholder */
.ant-input-search .ant-input-search.input-search input::placeholder {
136
color: #666;
}
Figure 110: Customized login page
6.3.5.6. Header stylization
/* Cores do header */ .layout.ant-layout .header {
background: linear-gradient(var(--main-linear-colora), var(--main-linear-colorb)) !important;
box-shadow: 0px 0px 10px #00000055;
}
/* Header icon */
.layout.ant-layout .header .user-menu > span *,
.layout.ant-layout .header .anticon-menu-unfold,
.layout.ant-layout .header .anticon-menu-fold {
color: #000 !important;
}
/* :hover header icons */
.layout.ant-layout .header .anticon-menu-unfold:hover,
.layout.ant-layout .header .anticon-menu-fold:hover,
.layout.ant-layout .header .user-menu > span:hover {
background-color: var(--main-primary-dark) !important;
color: #fff !important;
}
/* User menu icons */
.layout.ant-layout .header .user-menu > span:hover * {
color: #fff !important;
}
/* Dropdown triggers when active */
.layout.ant-layout .header .user-menu > span.ant-dropdown-open {
137
background-color: var(--main-primary-light) !important;
border-bottom: 5px solid var(--main-primary-dark) !important;
}
Figure 111: Stylized header
6.3.5.7. Sidebar stylization
/* background sidebar */
aside {
background: linear-gradient(var(--main-linear-colora), var(--main-linear-colorb)) !important;
}
.sider.light .ant-menu-light, .sider.light .ant-menu-submenu > .ant-menu {
background-color: transparent;
}
/* Navigation links colors */
.sider.light .ant-menu-light a,
.sider.light .ant-menu-submenu > .ant-menu-submenu-title {
color: #000 !important;
}
/* border-right for the active link */
.ant-menu-vertical .ant-menu-item::after,
.ant-menu-vertical-left .ant-menu-item::after,
.ant-menu-vertical-right .ant-menu-item::after,
.ant-menu-inline .ant-menu-item::after {
border-color: var(--main-primary-dark) !important;
}
/* Active link background */
.ant-menu:not(.ant-menu-horizontal) .ant-menu-item.ant-menu-item-selected {
background-color: var(--main-primary-light) !important;
}
/* background link :hover */
.ant-menu:not(.ant-menu-horizontal) .ant-menu-item:hover,
.ant-menu:not(.ant-menu-horizontal) .ant-menu-submenu div:hover {
background-color: var(--main-primary-dark) !important;
}
/* :hover icons color*/
.ant-menu:not(.ant-menu-horizontal) span.ant-menu-title-content:hover .custom-icon path {
fill: #fff !important;
}
/* :hover links color*/
.ant-menu:not(.ant-menu-horizontal) .ant-menu-item-active:hover a,
.ant-menu:not(.ant-menu-horizontal) .ant-menu-submenu span.ant-menu-title-content:hover{
color: #fff !important;
}
/* Sidebar logo */
.ant-layout-sider.light.ant-fixed-sidemenu > div > div{
height: 100px !important;
background-image: var(--main-image-menu);
138
background-repeat: no-repeat;
background-size: 200px;
background-position: 25px 25px;
margin-bottom: 20px;
}
/* Closed sidebar logo*/
.ant-layout-sider.light.ant-fixed-sidemenu.ant-layout-sider-collapsed > div > div{
height: 70px !important;
background-image: var(--main-image-icon);
background-repeat: no-repeat;
background-size: 30px;
background-position: 25px 20px;
}
/* Hide the ACS default logo */
.ant-layout-sider img {
display: none;
}
Figure 112: Stylized sidebar
139
Figure 113: Stylized closed sidebar
6.3.5.8. Cards and dashboard graphs stylization
/* Card "Running VMs" */
.usage-dashboard .ant-row .ant-col:nth-child(1) .ant-card.usage-dashboard-chart-card {
background-color: var(--main-green) !important;
}
/* Card "Stopped VMs" */
.usage-dashboard .ant-row .ant-col:nth-child(2) .ant-card.usage-dashboard-chart-card {
background-color: var(--main-red) !important;
}
/* Cards text: "RunningVMs" and "Stopped VMs" */
.usage-dashboard .ant-row .ant-col:nth-child(1) .ant-card.usage-dashboard-chart-card h2,
.usage-dashboard .ant-row .ant-col:nth-child(1) .ant-card.usage-dashboard-chart-card h3,
.usage-dashboard .ant-row .ant-col:nth-child(2) .ant-card.usage-dashboard-chart-card h2,
.usage-dashboard .ant-row .ant-col:nth-child(2) .ant-card.usage-dashboard-chart-card h3 {
color: #fff !important;
}
/* Graphs percentages */
.ant-progress-circle.ant-progress-status-normal .ant-progress-text {
color: var(--main-default) !important;
font-weight: bold;
}
/* Graph filling color */
.ant-progress.ant-progress-status-normal path.ant-progress-circle-path {
stroke: var(--main-default) !important;
}
/* Color for graph section without filling */
.ant-progress path.ant-progress-circle-trail{
stroke: var(--main-gray-light) !important;
}
140
Figure 114: Stylized dashboard on root admin account
Figure 115: Stylized dashboard on user account
6.3.5.9. Listings and links stylization
/* Links color */
a {
color: var(--main-primary-dark);
}
/* :hover link color */
a:hover {
color: var(--main-secondary-dark);
}
/* Dropdown links color */
.ant-dropdown .ant-dropdown-menu-item:hover {
141
background-color: var(--main-primary-light) !important;
}
/* Dropdown :hover links color */
.ant-dropdown .ant-dropdown-menu-item:hover * {
color: #000;
}
/* :hover listings items color*/
.ant-table-tbody > tr:hover > td {
background: var(--main-secondary-light) !important;
}
Figure 116: Stylized listing and links
6.4. Redirection to external links
There are two properties in the config.json settings file that define redirec-
tions to external links.
The property externalLink consists of a list containing values for the ti tle, li
nk and icon attributes.
"externalLinksIcon: "",
"externalLink": [
{
"title": "Cloudstack",
"link": "https://cloudstack.apache.org",
"icon": myIcons/cloudstach.png
},
{
"title": "Apache",
A list of possible resource limitations setting "link": "https://apache.org",
"icon": "https://www.apache.org/icons/apache.png"
}
]
The external link redirection feature is controlled by the following rules:
• if the properties are undefined, the redirection button will not be shown
to the user;
142
• the attribute link from the externa lLinks property is mandatory in this
context, and empty elements or those without the link property in the ex
ternalLinks property will not be considered;
• the t itle and i con properties are optional, and in the case that the icon
attribute is undefined, the default icon will be applied;
• when the title attribute is undefined, the link will be shown in its place.
Once the title, link and icon are defined, two possible behaviours can be
observed:
1. Only one element is defined: in this context a button will be shown, and
when clicked it will redirect the user to the link defined in the settings;
Figure 117: Only one defined element
2. More than one defined element: a dropdown list will be shown, containing
all the configured links.
Figure 118: More than one defined element
"externalLinksIcon: "",
"externalLinks": [
{
"title": "",
"link": "http://apache.org",
"icon": "https://www.apache.org/icons/apache.png"
},
{
"title": "Will not be displayed",
143
"link": "",
"icon": ""
},
{
"title": "CloudStack",
"link": "http://cloudstack.org",
"icon": "myIcons/cloudstach.png"
}
]
The externalLinksIcon property, also optional, defines an icon that will be
used to compose the button shown when more than one external link is in-
formed. In the example above this property was ommited, therefore, the de-
fault icon was displayed. It is also possible to use local images in the i con at-
tribute or in an external link.
Figure 119: Link with an icon attribute
More information about UI customization can be accessed in github.
144
7. Resources consumption accounting
The module for accounting computational resources consumption is subdi-
vided in two mechanisms: usage collection (Usage Server) and usage account-
ing (Quota), with the second acting complementarily to the first.
The Usage Server mechanism periodically performs the identification and
collection of usage data from the environment resources.
The Quota mechanism is a plugin that allows the management of a tariff
model over the computational resource consumption, guided by an one-to-
many relation, making it possible to define many tariffs for the same kind of
resource. Each tariff uses the type of resource, consumption volume and user
characteristics to evaluate and calculate the cost estimates.
7.1. Usage Server
The Usage Server (cloudstack-usage service) is an optional CloudStack com-
ponent, responsible for generating records over used resources in the infras-
tructure, with those records being saved in a separated database, called cloud
usage.
It is used to monitor resource consumption from users, allowing the im-
plementation of reports or billing services. It works collecting data from events
released from CloudStack and using this data to create resource usage reports.
7.1.1. Usage Server setup
The command to enable the Usage Server service is:
user@scclouds:~$ systemctl enable cloudstack-usage.service
user@scclouds:~$ systemctl start cloudstack-usage.service
After enabled, it is necessary to perform its setup within CloudStack:
• Accessing the CloudStack settings:
145
Figure 120: Accessing the settings
The main settings are:
146
Setting Description Default Value
enable.usage.server Enable the service false
publish.usage.events Enable publishing us-
age events
usage.timezone Timezone used by Us-
age Server
GMT
usage.sanity.check.interval Interval, in days, be-
tween error checks in
the Usage Server
usage.snapshot.virtualsize.select Analyses the virtual
size (true) or physical
size (false) from snap-
shots
false
usage.stats.job.aggregation.range Interval, in minutes,
that the data will be
aggregated.
1440
usage.stats.job.exec.time Time scheduled to
start the data aggrega-
tion job.
00:15
Table 30: Usage server settings
Figure 121: Editing the settings
As soon as the desired settings are saved and applied, it is necessary to
restart the Management Server and Usage Server services using the commands:
user@scclouds:~$ systemctl restart cloudstack-management.service
user@scclouds:~$ systemctl restart cloudstack-usage.service
7.1.2. Usage type
The Usage Server is utilized to monitor the consumption of various resources
and so, to differentiate records, each report is sent along with the usage type
147
parameter to indicate the type of resource accounted during the aggregation
period.
It is possible to find all usage types via API, as follows:
(admin) > list usagetypes
Type Name Description
1 RUNNING_VM Verifies the execution time for a VM dur-
ing the usage record period. If the VM
is updated during the period, you will re-
ceive a separated record for the updated
VM.
2 ALLOCATED_VM Verifies the total time interval between
the creation of a VM until its destruction.
3 IP_ADDRESS Shows the public IP address used by the
account.
4 NETWORK_BYTES_SENT Verifies the number of bytes sent from
the VMs from a certain account. Indi-
vidual traffic sent from each VM is not
tracked.
5 NETWORK_BYTES_RECEIVED Verifies the number of bytes received by
the VMs from a certain account. Indi-
vidual traffic received by each VM is not
tracked.
6 VOLUME Verifies the total time interval between
the creation of a disk volume until its de-
struction.
148
7 TEMPLATE Verifies the total time interval between
the creation of a template (created from a
snapshot or upload) until its destruction.
The template size is also returned.
8 ISO Verifies the total time interval between
the upload of an ISO until its destruction.
The ISO size is also returned.
9 SNAPSHOT Verifies the total time interval between
the creation of a snapshot until its de-
struction.
10 SECURITY_GROUP Verifies security groups usage.
11 LOAD_BALANCER_POLICY Verifies the total time interval between
the creation of a load balancer rule until
its removal. It does not track whether or
not a VM has used this rule.
12 PORT_FORWARDING_RULE Verifies the total time interval between
the creation of a port forwarding rule un-
til its removal.
13 NETWORK_OFFERING Verifies the total time interval between
the designation of a network offering to
a VM until its removal.
14 VPN_USERS Verifies the time interval between the cre-
ation of a VPN user until its removal.
21 VM_DISK_IO_READ Shows the amount of disk read opera-
tions from the VM.
22 VM_DISK_IO_WRITE Shows the amount of disk write opera-
tions from the VM.
149
23 VM_DISK_BYTES_READ Shows the amount of bytes read from the
VM disk.
24 VM_DISK_BYTES_WRITE Shows the amount of bytes written in the
VM disk.
25 VM_SNAPSHOT Shows the amount of storage used by the
VM snapshots.
26 VOLUME_SECONDARY Shows the amount of secondary storage
used by the VM volumes.
27 VM_SNAPSHOT_ON_PRIMARY Shows the amount of primary storage
used by the VM snapshots.
28 BACKUP Shows the amount of storage used by VM
backups.
29 VPC Verifies the total time interval between
the creation of a VPC until its destruction.
30 NETWORK Verifies the total time interval between
the creati a network until its destruction.
31 BACKUP_OBJECT Shows the storage used by backup ob-
jects.
Table 31: Usage types
A list of possible resource limiting settings may be accessed in 5.3.
7.1.3. Usage records
Usage Server collects resource usage statistics through event data gener-
ated by CloudStack and uses an aggregation job to generate reports. The time
the Usage records aggregation job runs for the first time each day is deter-
mined by the u sage.stats.job.exec.time global setting, and the generated Us-
age records records contain resource usage over a period of minutes defined
by the usage.stats.job.ag gregation.ra nge global setting. Since the generation
150
of Usage records can be confusing for some Usage types, this section aims to
explain each of them:
• RUNNING_VM: A Usage record will be generated for each VM execution
that occurred during the recording period, with the exception of system
VMs. Recording begins with the VM.START event and ends with the VM.
STOP event or the VM.DYNAMIC.SCALE event. After the VM.DYNAMIC.SC
ALE event, the recording done until the moment is saved in a record and
recording restarts for a new record.
• ALLOCATED_VM: A Usage record will be generated for each VM allocation
that occurred during the recording period, with the exception of system
VMs. Recording begins with the VM.CREATE event and ends with the V
M.DESTRO Y, VM.UPGRADE, or VM.D YNAMIC.SC ALE events. After the VM
.UP GRADE or VM.DYNAMIC.SCALE events, the recording done until the
moment is saved in a record and recording restarts for a new record.
• IP_ADDRESS: A Usage record will be generated for each public IP address
that was associated with a network or assigned to a VM during the record-
ing period, with the exception of system VMs. Recording begins with the
NET.IPASSIGN event and ends with the NET.IPRELEASE event.
• NETWORK_BYTES_SENT: A Usage record will be generated for each net-
work that sent bytes during the recording period. Each time the Usage
records aggregation job is executed, this number of bytes is recalculated.
• NETWORK_BYTES_RECEIVED: A Usage record will be generated for each
network that received bytes during the recording period. Each time the
Usage records aggregation job is executed, this number of bytes is recal-
culated.
• VOLUME: A Usage record will be generated for each disk volume that was
allocated in primary storage during the recording period. Recording be-
151
gins with the VOLUME. C R E A T E event and ends with the VOLUME.DELETE
or VOLUME.RESIZE event. After the VOLUME.RESIZE event, the recording
done until the moment is saved in a record and recording restarts for a
new record.
• TEMPLAT E: A Usage record will be generated for each template that ex-
isted in secondary storage during the recording period. Recording begins
with the TEMPLATE. CREATE or TEMPLATE.COPY event and ends with the
TEMPLATE.DELETE event.
• ISO: A Usage record will be generated for each ISO that existed on a sec-
ondary storage during the registration period. Recording begins with the
ISO.CREATE or ISO.COPY event and ends with the ISO.DELETE event.
• SNAPSHOT: A Usage record will be generated for each snapshot of a vol-
ume that existed on a secondary storage during the registration period.
Recording begins with the SNAPSHOT.CREATE event and ends with the S
NAPSHOT.DELETE event.
• LOAD_BALANCER_POLICY: A Usage record will be generated for each load
balancer policy used during the registration period. Recording begins with
the LB.CREATE event and ends with the LB.DELETE event.
• PORT_FORWARDING_RULE: A Usage record will be generated for each
port forwarding rule used during the registration period. Recording be-
gins with the NE T.RULEADD event and ends with the NET.RULEDELETE
event.
• NETWORK_OFFERING: A Usage record will be generated for each VM con-
nected to a network created with a network offering during the registra-
tion period. Recording begins with the NETWORK.OFFERING.CREATE or N
ETWORK.OFFERING. A SSIGN event and ends with the NE T WORK.OFFERIN
G.DELETE or NETWORK.OFFERING.REMOVE event.
152
• VPN_USERS: A Usage record will be generated for each VPN user that ex-
isted during the record period. Recording begins with the VPN.USER.ADD
event and ends with the VPN.USER.REMOVE event.
• VM_DISK_IO_READ and VM_DISK_BYTES_READ: A Usage record will be
generated for each disk in a VM that performed read operations during
the recording period. Each time the Usage records aggregation job is ex-
ecuted, this number of bytes is recalculated.
• VM_DISK_IO_WRITE and VM_DISK_BYTES_WRITE: A Usage record will be
generated for each disk in a VM that performed write operations during
the recording period. Each time the Usage records aggregation job is ex-
ecuted, this number of bytes is recalculated.
• VM_SNAPSHOT: A Usage record will be generated for each volume saved
by a snapshot of a VM during the registration period. Recording begins
with the VMSNAPSHOT.CREATE event and ends with the VMSNAPSHOT.D
ELETE event.
• VOLUME_SECONDARY: A Usage record will be generated for each volume
that was allocated to secondary storage during the registration period.
Recording begins with the VOLUME.UPLOAD event and ends with the VOL
UME.DELETE or VOLUME.CREATE event (since the volume will be migrated
to primary storage).
• VM_SNAPSHOT_ON_PRIMARY: A Usage record will be generated for each
VM snapshot that existed on the primary storage during the registration
period. Recording begins with the VMSNAPSHOT.ON_PRIMARY event and
ends with the VMSNAPSHOT.OFF_PRIMARY event.
• BACKUP: A Usage record will be generated for each backup offering that
was associated with a VM during the registration period. Recording begins
153
with the BACKUP.OFFERING.ASSIGN event and ends with the BACKUP.OF
FERING.REMOVE event.
• VPC: A Usage record will be generated for each VPC that existed during
the recording period. Recording begins with the VPC.CREATE event and
ends with the VPC.DELETE event.
• NETWORK: A Usage record will be generated for each network on which
VMs were running during the recording period. Recording begins with the
NETWORK.CREATE event and ends with the NETWORK.DELETE event.
• BACKUP_OBJECT: A Usage record will be generated for each backup that
existed during the recording period. Recording begins with the BACKUP.
CREATE event and ends with the BACKUP.DELETE event.
7.2. Quota
The Quota plugin is a service that extends the Usage Server functionalities,
making it possible to assign monetary value to computational resources con-
sumption in the control reports.
7.2.1. Quota setup
The main settings for this service are:
154
Name Description Default
Value
quota.currency.symbol Currency symbol used to
measure the resource us-
age.
$
quota.enable.service Enable the Quota plugin. false
quota.statement.period Interval in which the Quota
details are sent via e-mail,
with possible values being:
bimonthly(0), monthly(1),
quarterly(2), half-yearly(3)
and annual(4).
1
quota.usage.smtp.connection.timeout Connection timeout with the
SMTP server.
60
quota.usage.smtp.host Host that holds the SMTP
server.
quota.usage.smtp.password SMTP server password.
quota.usage.smtp.port SMTP server port.
quota.usage.smtp.sender Issuing e-mail.
quota.usage.smtp.useAuth Use authentication in the
SMTP server.
quota.usage.smtp.user SMTP server user.
quota.enable.enforcement Makes resource manipula-
tion unavailable for the ac-
count when it reaches the
Quota limit.
false
Table 32: Quota plugin settings
After changing these settings, it is necessary to restart the Management
Server and the Usage Server to apply them:
user@scclouds:~$ systemctl restart cloudstack-management.service
user@scclouds:~$ systemctl restart cloudstack-usage.service
After this restart, the Quota menu will be available in the UI:
155
Figure 122: Quota plugin
From this menu it will be possible to manage tariffs, credits, Quota e-mail
templates and visualize reports.
7.2.2. Tariffs management
In the Tariff submenu it is possible to visualize and manage the Quota tariffs.
When accessing the menu, a list of active system tariffs will be shown:
Figure 123: List of active tariffs
It is also possible to list already removed tariffs, just by selecting the Remo
ved option in the filter, or All to list everything:
156
Figure 124: Listing filters
The operator may create new tariffs or edit/remove already existing ones.
7.2.2.1. Creating tariffs
To create a new tariff, use the Create Quota tarif f button, which will open the
following form:
157
Figure 125: Tariff creation form
The operator must obligatorily inform the Name, Usage ty pe and Tariff val
ue fields. The remaining fields are optional.
In the Processing period field, when choosing the Monthly option, a new
field named Ex e c ute on will appear, where it will be necessary to add a day of
the month between 1-28, indicating the date in which the tariff will be monthly
processed.
It is possible to set rules to define when a tariff must be applied. The docu-
mentation about activation rules may be found in the Activation rules section.
158
7.2.2.2. Tariff details
When selecting a tariff in the listing, it is possible to visualize its details and any
actions that may be executed:
Figure 126: Tariff details
7.2.2.3. Editing tariffs
After creating tariffs, the operator may change some of its information:
Figure 127: Possible actions for active tariffs
159
Figure 128: Tariff editing form
The changes made in the tariff will only have an effect on processings made
after the changes.
7.2.2.4. Removing tariffs
If necessary, the operator may remove tariffs to prevent them from being in-
cluded in future Quota processings:
Figure 129: Removing a tariff
7.2.3. Activation rules
Activation rules are logical expressions used to define which tariffs are ap-
plied based on which resources are being used, possibly including specific tar-
160
iffs for specific clients.
Through activation rules it is possible to use resource tags as a way to iden-
tify different resource types within the same category and then apply unique
tariffs for each one (for example, apply a greater tariff if the storage is of SSD
type).
Due to its simplicity, both in terms of understanding and writing, compared
to other programming languages, JavaScript was chosen for the creation of
the activation rules. Therefore, the expressions must be written specifically
in JavaScript ECMAScript 5.1 code and need to follow, besides the language
syntax, the following rules:
• The expression processing engine is only instantiated once per processing
cycle, therefore, reserved words such as const, var or let must not be used
to declare variables when creating activation rules, as this will cause an I
dentifier has already been declared error and it will not be possible to
process such rules. Instead of using const a = 1;, for example, a = 1; should
be used.
• ACS expects the return from these expressions to be a boolean value (t
rue/false) or a numeric value. It will deduce the result type applying the
following rules:
– if the result is a number, such as 1, 2.5 and so on, the result will be
used as the tariff value, instead of the value defined in the Tariff valu
e field;
– if the result is not a number, ACS will try to convert it to a boolean.
If the result is true, the value set in the Tarif f value field will be used
in the calculation. Otherwise, if the result is false or if the expression
does not result in a valid boolean, the tariff will not be applied in the
calculation;
161
– if the tariff does not have an expression to be evaluated or if the ex-
pression is empty, the tariff will always be applied.
Some variables will be pre-created (referred as preset throughout the text)
in the expressions context to provide greater flexibility to the operators. Each
resource type will have a series of presets corresponding to its characteristics:
162
7.2.3.1. Default presets for all resource types
Variable Description
account.id uuid of the account that owns the resource.
account.name name of the account that owns the resource.
account.role.id uuid of the role from the account that owns the
resource (if it exists).
account.role.name name of the role from the account that owns the
resource (if it exists).
account.role.type type of the role from the account that owns the
resource (if it exists).
domain.id uuid of the domain of the account that owns the
resource.
domain.name name of the domain of the account that owns the
resource (if it exists).
domain.path path of the domain of the account that owns the
resource.
project.id uuid of the project that owns the resource (if it
exists).
project.name name of the project that owns the resource (if it
exists).
resourceType type of resource.
value.accountResources List containing the remaining of the account re-
sources of the same type that are valid in the
same period that is being processed.
zone.id uuid of the zone of the account that owns the re-
source.
zone.name name of the domain of the account that owns the
resource.
processedData.id uuid of the resource.
processedData.name name of the resource.
processedData.startDate Start date from the processed data.
processedData.endDate End date from the processed data.
Table 33: Default presets for every resource type - Part 1
163
Variable Description
processedData.usageValue usage from the resources during the pe-
riod.
processedData.aggregatedTariffs Aggregation from all tariffs applied by
Quota over the resource during the pe-
riod.
processedData.tariffs.value Tariff value.
processedData.tariffs.id uuid of the tariff.
processedData.tariffs.name name of the tariff.
lastTariffs List of objects containing id and value at-
tributes for previous tariffs.
Table 34: Default presets for every resource type - Part 2
164
7.2.3.2. Presets for the RUNNING_VM type
Variable Description
value.host.id uuid of the host where the VM is
running.
value.host.name name of the host where the VM is
running.
value.host.tags List of tags of the host where the
VM is running.
Example: ["a", "b"].
value.host.isTagARule Boolean indicating if the tag used
is a rule.
value.id uuid of the VM.
value.name name of the VM.
value.osName name of the OS in the VM.
value.computeOffering.customized Boolean indicating if the compute
offering of the VM is customizable.
value.computeOffering.id uuid of the compute offering of
the VM.
value.computeOffering.name name of the compute offering of
the VM.
value.computingResources.cpuNumber Current number of vCPUs in the
VM.
value.computingResources.cpuSpeed Current CPU speed in the VM (in
MHz).
value.computingResources.memory Current amount of memory in the
VM (in MiB).
value.tags VM tags, in the format key:value.
Example: {"a":"b", "c":"d"}.
value.template.id uuid of the VM template.
value.template.name name of the VM template.
value.hypervisorType type of hypervisor of the VM.
Table 35: Presets for the RUNNING_VM type
165
7.2.3.3. Presets for the ALLOCATED_VM type
Variable Description
value.id uuid of the VM.
value.name name of the VM.
value.osName name of the OS in the VM.
value.computeOffering.customized Boolean indicating if the compute of-
fering of the VM is customizable.
value.computeOffering.id uuid of the compute offering of the
VM.
value.computeOffering.name name of the compute offering of the
VM.
value.tags VM tags, in the format key:value. Ex-
ample: {"a":"b", "c":"d"}.
value.template.id uuid of the VM template.
value.template.name name of the VM template.
value.hypervisorType type of hypervisor of the VM.
Table 36: Presets for the ALLOCATED_VM type
166
7.2.3.4. Presets for the VOLUME type
Variable Description
value.diskOffering.id uuid of the disk offering of the volume.
value.diskOffering.name name of the disk offering of the volume.
value.id uuid of the volume.
value.name name of the volume.
value.provisioningType Resource provisioning type. Values for this set-
ting may be: thin, sparse or fat.
value.storage.id uuid of the storage where the volume is located.
value.storage.isTagARule Boolean indicating if the tag used is a rule.
value.storage.name name of the storage where the volume is located.
value.storage.scope scope of the storage where the volume is located.
Values for this setting may be: ZONE or CLUSTE
R.
value.storage.tags List of tags of the storage where the volume is
located. Example: ["a", "b"].
value.tags Volume tags, in the format key:value. Example: {
"a":"b", "c":"d"}.
value.size size of the volume (in MiB).
value.volumeFormat Volume format. Values for this setting may be: R
AW, VHD, VHDX, OVA and QCOW2.
Table 37: Presets for the VOLUME type
7.2.3.5. Presets for the TEMPLATE and ISO type
Variable Description
value.id uuid of the template/ISO.
value.name name of the template/ISO.
value.osName name of the template’s/ISO’s OS.
value.tags Template/ISO tags, in the format k ey:value. Example: {"a":"
b", "c":"d"}.
value.size size of the template/ISO (in MiB).
Table 38: Presets for the TEMPLATE and ISO types
167
7.2.3.6. Presets for the SNAPSHOT type
Variable Description
value.id uuid of the snapshot.
value.name name of the snapshot.
value.size size of the snapshot (in MiB).
value.snapshotType type of snapshot. Values for this setting may be:
MANUAL, HOURLY, DAILY, WEEKLY or MONTHLY.
value.storage.id uuid of the storage where the snapshot is lo-
cated.
value.storage.isTagARule Boolean indicating if the tag used is a rule.
value.storage.name name of the storage where the snapshot is lo-
cated.
value.storage.scope scope of the storage where the snapshot is lo-
cated. Values for this setting may be: ZONE or
CLUSTER.
value.storage.tags List of tags of the storage where the snapshot is
located. Example: ["a", "b"].
value.tags Snapshot tags, in the format key:value. Example:
{"a":"b", "c":"d"}.
value.hypervisorType hypervisor in which the resource was deployed.
Values for this setting may be: XenServer, KVM, V
Mware, Hy per-V, BareMetal, Ovm, Ovm3 and L X
C.
Table 39: Presets for the SNAPSHOT type
Notes:
• If the global setting snapshot.backup.to.secondary is set to false, the value
for the presets value. storage.id and value.storage.name will be from the
primary storage. Otherwise, they will be from the secondary storage.
• Hosts or storages using the flexible tags feature will have the isTagARule
variable set to true and will have their tag variable empty.
• If the global setting snapshot.backup.to.secondary is set to false, the value
168
for the presets value.sto rage.scope and value.storage.tags will be from
the primary storage. Otherwise, they will not exist.
7.2.3.7. Presets for the NETWORK_OFFERING type
Variable Description
value.id uuid of the network offering.
value.name name of the network offering.
value.tag tag of the network offering.
Table 40: Presets for the NETWORK_OFFERING type
7.2.3.8. Presets for the VM_SNAPSHOT type
Variable Description
value.id uuid of the VM snapshot.
value.name name of the VM snapshot.
value.tags VM snapshot tags, in the format key :value. Exam-
ple: {"a":"b", "c":"d"}.
value.vmSnapshotType type of the VM snapshot. Values for this setting
may be:
Disk or DiskAndMemory.
value.hypervisorType Hypervisor in which the resource was deployed.
Values for this setting may be: XenServer, KVM, V
Mware, Hyper-V, BareMetal, Ovm, Ovm3 and LXC.
Table 41: Presets for the VM_SNAPSHOT type
7.2.3.9. Presets for the BACKUP type
Variable Description
value.size size of the backup.
value.virtualSize virtual size of the VM.
value.backupOffering.id uuid of the backup offering.
value.backupOffering.name name of the backup offering.
value.backupOffering.externalId external id of the backup offering.
Table 42: Presets for the BACKUP type
Notes:
169
• The measurement unit of the presets value.size and value.virtualSize varies
for each backup provider. For example, values informed by Veeam are in
bytes.
7.2.3.10. Presets for the NETWORK_USAGE type
Variable Description
value.id uuid of the network.
value.name name of the network.
value.state state of the network. Values for this setting may be: Allocated,
Configured, Implementing, Implemented, Shutdown and De-
stroyed.
Table 43: Presets for the NETWORK_USAGE type
7.2.3.11. Presets for the BACKUP_OBJECT type
Variable Description
value.id uuid of the backup object.
value.name name of the backup object.
value.size size of the resource, in MiB.
value.virtualSize virtual size of the backup.
value.backupOffering.id uuid of the backup offering.
value.backupOffering.name name of the backup offering.
value.backupOffering.externalId external id of the backup offering.
value.virtualMachine.id uuid of the VM.
value.virtualMachine.name name of the VM.
Table 44: Presets for the BACKUP_OBJECT type
7.2.3.12. Verifying presets via API
An API call may be done to verify all the presets for each resource, but only
admin users have access to this command. For that, just choose a usageType
to check the variable names and descriptions, as shown below:
(admin) > quota listpresetvariables usagetype=<usageType_id>
170
7.2.3.13. Presets for the other resources
The specific presets for each resource type were addressed based on use cases,
therefore, not all resource properties are present, similarly, not all resources
have specific presets, only the default presets. Other presets may be added as
use cases appear.
7.2.3.14. Expressions examples
As described at the beginning of this section, activation rules are logical expres-
sions written in JavaScript, created to address specific use cases. It is advisible
for the environment administrator to create their own expressions based on
their needs, paying attention to the rules described at the beginning of this
section and to the language syntax. The following examples provide demon-
strations of what can be achieved using activation rules.
1. Apply a tariff to a single account (available to all resource types):
if (account.id == 'b29e84da-ed2e-47dc-9785-49231de8ff07') {
true
} else {
false
}
Or simply:
account.id == 'b29e84da-ed2e-47dc-9785-49231de8ff07'
2. Apply a tariff if the account currently has more than 20 resources of the
same type (available for all resource types):
value.accountResources.filter(resource =>
resource.domainId == 'b5ea6ffb-fa80-455e-8b38-c9b7e3900cfd'
).length > 20
171
3. Return the tariff value based on the amount of resources that the account
currently owns (available for all resource types)
12
:
resourcesLength = value.accountResources.filter(resource =>
resource.domainId == 'b5ea6ffb-fa80-455e-8b38-c9b7e3900cfd'
).length
if (resourcesLength > 40) {
20
} else if (resourcesLength > 10) {
25
} else {
30
}
4. Apply the tariff for a certain OS (available for the RUNNING_VM and AL-
LOCATED_VM):
['Windows 10 (32-bit)',
'Windows 10 (64-bit)',
'Windows 2000 Advanced Server'].indexOf(value.osName) !== -1
5. Storage tags validation (available for VOLUME and SNAPSHOT):
value.storage.tags.indexOf('SSD') !== -1
&& value.storage.tags.indexOf('NVME') !== -1
6. Host tags validation (available for RUNNING_VM):
value.host.tags.indexOf('CPU platinum') !== -1
12
If the value is not informed in the else, the expression may result in undefined and the
tariff will not be applied.
172
7. Public IPs validation
13
. If the first public IP is free of charge for the user, it
is possible to prevent charging source NAT IPs (available for IP ADDRESS):
resourceType !== 'SourceNat'
8. Return the tariff value if the storage in use is of HDD type (available for
VOLUME and SNAPSHOT):
useHdd = false
if (value.storage) {
for (i = 0; i < value.storage.tags.length; i++) {
if (value.storage.tags[i].indexOf('hdd') !== -1) {
useHdd = true
break
}
}
}
if (useHdd) {
0.3
} else {
0
}
9. For a more complex example, the following expression represents the li-
censing cost for Windows OS and it has some peculiarities (available for
ALLOCATED_VM):
• The amount charged will be based on the number of vCPUs assigned
to the VM;
• A minimum of 4 vCPUs are charged;
13
Public IPs are connected to the VPC or isolated networks (not directly to the user VM). Every
first public IP in a VPC or isolate network is a source NAT; if the public IP is allocated by the user,
the preset resourceType will be null.
173
• For more than 4 vCPUs it is charged incrementally for vCPUs pairs
(2). In other words, if the number of vCPUs is odd, the charge will be
rounded up to an even value. For example, the charge for 5 vCPUs is
the same as for 6 vCPUs, for 7 vCPUs is the same as 8, and so on;
• The charge will be monthly.
TOTAL_CORES_PER_PACKAGES = 2;
MININUM_NUMBER_OF_PACKAGES_WINDOWS_LICENSES = 4;
OPERATING_SYSTEM_NAME = "windows";
windows_operating_system_monthly_price = 36;
calculate_number_of_license_packages = function (vcpus) {
return (vcpus + vcpus%TOTAL_CORES_PER_PACKAGES)/
TOTAL_CORES_PER_PACKAGES;
};
if (value.osName.toLocaleLowerCase().
indexOf(OPERATING_SYSTEM_NAME) >= 0) {
calculate_number_of_license_packages
(value.computingResources.cpuNumber) *
windows_operating_system_monthly_price
} else {
0
}
7.2.4. Credits management
In the Summa ry sub-menu it is possible to check the Quota reports and
manage the accounts’ credits.
174
Figure 130: List of active accounts
7.2.4.1. Adding/removing credits
To add/remove credits from an account, use the Add credits button, which will
open the following form:
Figure 131: Form for adding/removing credits
Notes:
175
• To remove credits from an account, use the - operator before the tariff
value, in the Value field.
• The Min Balance field indicates the minimum limit for the account bal-
ance.
• Checking the Enforce Quota field will make accounts that reach their limits
have their balances blocked.
7.2.5. Active accounts
In the Summary sub-menu it is shown, by default, the current state of active
accounts (last balance + credits):
Figure 132: List of active accounts
It is also possible to list already removed accounts, just by selecting the filter
option Removed accounts, or All to list everything:
176
Figure 133: Listing filters
Via CloudMonkey, this query may be performed in the following way:
(admin) > quota summary account=admin domainid=52d83793-26de-11ec-8dcf-5254005dcdac listall=true
{
"count": 1,
"summary": [
{
"account": "admin",
"accountid": "af16aaed-26de-11ec-8dcf-5254005dcdac",
"accountremoved": false,
"balance": 124.49,
"currency": "$",
"domain": "/",
"domainid": "52d83793-26de-11ec-8dcf-5254005dcdac",
"domainremoved": false,
"enddate": "2023-10-06T12:00:00+0000",
"quota": 8.33871072,
"quotaenabled": true,
"startdate": "2023-10-01T12:00:00+0000",
"state": "ENABLED"
}
]
}
7.2.6. Managing e-mail templates from Quota
It is possible to define notification templates for the Quota mechanism, which
will be sent to users based on four pre-defined situations. For each user it is
possible to define when the notifications will be sent.
In the Email Templ ate sub-menu it is possible to visualize and manage the
e-mail templates that the Quota plugin will send:
177
Figure 134: List of e-mail templates from Quota
By default, the Quota plugin will send e-mails to an account in the following
scenarios:
• Low credits;
• No credits;
• Credits added;
• Balance in use.
Each scenario has a corresponding email template that will be sent. To edit
one of these templates, just select it in the listing:
Figure 135: Editing the e-mail template
178
7.2.7. Notes about using the Quota plugin
1. To add an account to the Quota plugin processing it is necessary to add
credits to it at least once. Furthermore, the account level setting quota.
account.enabled must be defined as true. The Quota state for a specific
account may be accessed through the Summary sub-menu in the Quota
state column.
Figure 136: Quota state for admin accounts and custom-account
2. For accounts with credits balance lower than zero to become blocked the
global setting quota.enable.enfo rce me nt must be set to true and the op-
tion Enforce Quota must be used when adding credits to the account.
3. A blocked account still has access to its resources that are already allo-
cated. It can deallocate them but can neither allocate new resources nor
reallocate old ones. In other words, if the user has its account blocked,
they can destroy or stop their VMs but can’t start/restart them. If any is-
sue occurs with the VM (like a shutdown caused by lack of CPU or RAM)
the user will have to acquire more credits to restart the VM.
4. The check for account credits, and subsequent block (when applicable) is
performed in a interval defined by the variable usage.stats.job.
aggregation.range, with default value being once a day.
179
8. Operation
This section presents some of the basic operations that operators must be
familiar with for troubleshooting issues in ACS.
8.1. Debugging issues and troubleshooting process
There are several situations where problems may occur in ACS. This section
provides recommendations for identifying the origin of most problems, as it is
frequently possible to solve them without the need to open a new issue.
8.1.1. Debugging via logs
When encountering an error, the first step in troubleshooting is often to
check the log files of the components involved in the actions that triggered it.
Reading logs frequently reveals the source of most problems. Furthermore,
if the problem is not treatable in the operation context, sending the logs rele-
vant to the problem simplifies and speeds up their resolution when opening an
issue. Section 8.1.4 describes how to find the log files.
Eventually, errors happen because of resource shortage. For example, an
error when creating a virtual router may be caused by the shortage of available
IPs. This kind of situation is fixed by increasing the current amount of resources
or recycling other resources that are no longer used.
8.1.2. Debugging via web interface
When errors occur through the web interface, it is possible to verify which
API commands are currently being executed, which facilitates searching for the
error in the logs, as the commands being called are known. The steps for this
kind of debugging are:
1. Open the development tools menu in your browser (F12 is usually the
shortcut);
2. Select the network tab;
3. In parallel, open the Management Server’s log file, using the command ta
il -f;
180
4. Perform the action that causes the error in the web interface. You will see
all calls being made in the Network tab;
5. Select one of your interest;
6. In the GET section of the Network tab, the API command being executed
will be visible;
By doing so, it is possible to follow the execution flow on the logs and debug
the problem.
8.1.3. Debugging network problems
Sometimes, the routes of some ACS components are changed, which may
prevent proper communication between certain components. The command i
p route is the main way to detect any changes in the routes, ensuring that the
IPs are coherent with the adopted topology. Alternatively, the a rp command
can be used to verify if the packets are travelling through the right interface.
The SC Clouds team provides a document with the defined topology, that may
be consulted to verify if the IPs are correct.
8.1.4. Log files path
The log file from the Management Server can be found in the following path:
• /var/log/cloudstack/management/management-server.log
If the Usage Server is being used, its log file may be found in the following
path:
• /var/log/cloudstack/usage/usage.log
If the hypervisor used is KVM, a CloudStack Agent will exist. Its log file can
be found in the following path:
• /var/log/cloudstack/agent/agent.log
The logs for the system VMs can be found in the following path:
• /var/log/cloud.log
• /var/log/cloud/cloud.out
181
8.1.5. Log level increase on Management Servers and KVM Agents
This section describes how to configure Log4j
14
in a way that prevents the
loss of important ACS logs. Log4j organizes the logs in the following way:
1. FATAL: Errors that cause a system stop (crash). It is important for this type
of message to be reported!
2. ERROR: Errors that interrupt a task, however these do not cause a com-
plete system stop. Generally it is also important to report this type of
message;
3. WARN: Warnings that do not represent errors that happened but indicate
events that may cause them in the future;
4. INFO: Information relevant to the performed action. Messages of this
type describe normal system events, not errors;
5. DEBUG: Information that details the perfomed action;
6. TRACE: Detailed step by step of the performed action, being of a finer level
than the DEBUG level;
ACS logs have some known categorization and clarity issues. For example,
there are logs that should be registered at INFO or ERROR level but are regis-
tered at DEBUG level. The SC Clouds team has been working on improvements
on these and others aspects referent to ACS logs. However, until this task is
completed, it is necessary for the Log4j to be adequately configured (at DEBUG
level) to ease the troubleshooting processes.
It is possible to configure Log4j to register only logs of certain types, this way
Log4j will register all messages of that type and those that are more severe. The
adopted hierarchy is:
14
Tool used by ACS to register logs.
182
Setting Log levels registered
OFF No log registry.
FATAL FATAL
ERROR FATAL and ERROR
WARN FATAL, ERROR and WARN
INFO FATAL, ERROR, WARN and INFO
DEBUG FATAL, ERROR, WARN, INFO and DEBUG
TRACE FATAL, ERROR, WARN, INFO, DEBUG and TRACE
ALL FATAL, ERROR, WARN, INFO, DEBUG and TRACE
Table 45: Logs hierarchy levels on Log4j
It is possible to configure the appender
15
to accept a log level from the ones
shown above. The current default setting for the Management Servers is DEB
UG; for agents and system VMs the default is INFO. Therefore, the appender
will only register logs of the specified level or of greater severity, which may be
insufficient to detect or comprehend certain problems. If the hypervisor used is
KVM, it is recommended to change this setting from INFO to DEBUG throughout
agents.
For that, edit the files shown below in the respective hosts using a text editor:
• In the agent:
sudo vim /etc/cloudstack/agent/log4j-cloud.xml
• In the Management Server:
sudo vim /etc/cloudstack/management/log4j-cloud.xml
• In the system VMs:
sudo vim /usr/local/cloud/systemvm/conf/log4j-cloud.xml
15
Part of the tool responsible for delivering the logs to their destination.
183
If the hypervisor used is VMware, the process to access the system VMs and
edit the logj4 settings file is different. The section Accessing the System VMs
shows this process in more detail.
Change the following setting:
...
<appender name="FILE" class="org.apache.log4j.rolling.RollingFileAppender">
...
<param name="Threshold" value="INFO"/>
...
To:
...
<appender name="FILE" class="org.apache.log4j.rolling.RollingFileAppender">
...
<param name="Threshold" value="DEBUG"/>
...
To limit the log sources, Log4j allows the creation of categories, defining
from where the logs will be collected, as well as the accepted level. It is impor-
tant to take note that the XML that defines the Log4j settings is serially read,
therefore the order of the categories is important, since if we want to limit the
logs from a package while keeping logs in a less restrictive level, we will first
need to define the less restrictive level for the wanted section and them define
the restriction level for the package as a whole.
With this in mind, we recommend adding the following category to the agents
XML:
<category name="org.apache.cloudstack">
<priority value="DEBUG"/>
</category>
This category must be above org.apache, for the reason previously explained.
However, if instead of creating the category, the category org.apache was edited
to DEBUG level, an imense flood of unimportant log messages would be cre-
ated.
In the management servers this category already exists, but it is poorly po-
sitioned, therefore it must be moved so it is above org.apache.
It is also necessary to change the priority value of the category com.clou d
184
to DEBUG level:
<category name="com.cloud">
<priority value="DEBUG"/>
</category>
Finally, it is necessary to change the root category setting, because it dictates
the maximum level that the logger accepts, and without this alteration all the
changes made previously will not take effect. The default for this setting is INFO:
<root>
<level value="INFO"/>
<appender-ref ref="CONSOLE"/>
<appender-ref ref="FILE"/>
</root>
It is sufficient to change the level value from INFO to DEBUG.
8.1.6. Troubleshooting process
Analyzing CloudStack logs is extremaly useful and can be a key point during
investigations, since they contain information related to all steps of ACS pro-
cesses, including errors, warnings, and additional messages.
For an efficient log analysis in ACS it is necessary to filter the entries of in-
terest, this is achieved by identifying the log id or job id related to them.
If the analysis is made based on the Management Server logs and there is
more than one Management Server in use, it is necessary to search through
all of them. The reason for this is that command executions may be shared
between them.
To better illustrate the filtering and analysis process, we will present an ex-
ample of the troubleshooting process. In this example we will accompany the
creation of a new VM, but the steps used may also be applied to investigate
multiple occurences in CloudStack.
To begin the analysis, a VM was created via UI and the steps below were
followed:
185
Figure 137: VM created th rough the UI
1. To identify the log id for the process under investigation, it is first neces-
sary to locate a log entry containing this information. This can be achieved
by tracing the HTTP requests forwarded to the back-end by the UI when
performing certain user-requested actions. By checking the details of a
HTTP request it is possible to identify the command sent. We will use this
command to search for log entries:
Figure 138: Identifying the command sent.
To perform this search it is necessary to access the location where the log
186
files are stored and execute the command
16
:
grep -r "<UI command>" ./
In the example shown, the command used was:
grep -r "command=deployVirtualMachine&response=json" ./
Thus, we can identify some entries containing the searched command. In
them it is possible to visualize the desired log id. In this example the ID is:
logid:ef4bf833.
Figure 139: Identifying the logid for the desired process.
2. Then, we can filter the desired information using the log id:
grep -r "<logid>" ./
All log entries containing this log id will be returned.
./management-server.log:2022-03-15 12:38:40,334 DEBUG [c.c.a.ApiServlet]
(qtp1603198149-17:ctx-430e157a) (logid:ef4bf833) ===START=== 172.16.71.7 -- POST
command=deployVirtualMachine&response=json
[...]
./management-server.log:2022-03-15 12:38:41,471 DEBUG [c.c.v.UserVmManagerImpl]
(qtp1603198149-17:ctx-430e157a ctx-91e054dc) (logid:ef4bf833) Allocating in the DB for vm
./management-server.log:2022-03-15 12:38:41,545 INFO
[c.c.v.VirtualMachineManagerImpl] (qtp1603198149-17:ctx-430e157a ctx-91e054dc)
(logid:ef4bf833) allocating virtual machine from
template:b81aab72-2c37-466e-b53f-bdaf398322fa with hostname:i-2-119-VM and 1 networks
./management-server.log:2022-03-15 12:38:41,555 DEBUG
[c.c.v.VirtualMachineManagerImpl] (qtp1603198149-17:ctx-430e157a ctx-91e054dc)
(logid:ef4bf833) Allocating entries for VM: VM instance {id: "119", name:
"i-2-119-VM", uuid: "13fe0bce-a240-48e4-9d5b-081359fcf422", type="User"}
./management-server.log:2022-03-15 12:38:41,560 DEBUG
[c.c.v.VirtualMachineManagerImpl] (qtp1603198149-17:ctx-430e157a ctx-91e054dc)
(logid:ef4bf833) Allocating nics for VM instance {id: "119", name: "i-2-119-VM",
uuid: "13fe0bce-a240-48e4-9d5b-081359fcf422", type="User"}
./management-server.log:2022-03-15 12:38:41,568 DEBUG
[o.a.c.e.o.NetworkOrchestrator] (qtp1603198149-17:ctx-430e157a ctx-91e054dc)
(logid:ef4bf833) Allocating nic for vm VM instance {id: "119", name: "i-2-119-VM",
uuid: "13fe0bce-a240-48e4-9d5b-081359fcf422", type="User"} in network
Ntwk[211|Guest|11] with requested profile NicProfile[0-0-null-null-null]
./management-server.log:2022-03-15 12:38:41,653 DEBUG [c.c.n.NetworkModelImpl]
(qtp1603198149-17:ctx-430e157a ctx-91e054dc) (logid:ef4bf833) Service SecurityGroup
is not supported in the network id=211
./management-server.log:2022-03-15 12:38:41,660 DEBUG
16
It is alo possible to replace "./" in the command with the log folder’s full path. That way it
is possible to execute it without accessing the directory.
187
[c.c.v.VirtualMachineManagerImpl] (qtp1603198149-17:ctx-430e157a ctx-91e054dc)
(logid:ef4bf833) Allocating disks for VM instance {id: "119", name: "i-2-119-VM",
uuid: "13fe0bce-a240-48e4-9d5b-081359fcf422", type="User"}
./management-server.log:2022-03-15 12:38:41,660 INFO [o.a.c.e.o.VolumeOrchestrator]
(qtp1603198149-17:ctx-430e157a ctx-91e054dc) (logid:ef4bf833) adding disk object
ROOT-119 to i-2-119-VM
./management-server.log:2022-03-15 12:38:41,700 DEBUG
[c.c.r.ResourceLimitManagerImpl] (qtp1603198149-17:ctx-430e157a ctx-91e054dc)
(logid:ef4bf833) Updating resource Type = volume count for Account = 2 Operation =
increasing Amount = 1
./management-server.log:2022-03-15 12:38:41,729 DEBUG
[c.c.r.ResourceLimitManagerImpl] (qtp1603198149-17:ctx-430e157a ctx-91e054dc)
(logid:ef4bf833) Updating resource Type = primary_storage count for Account = 2
Operation = increasing Amount = (50.00 MB) 52428800
./management-server.log:2022-03-15 12:38:41,962 DEBUG
[c.c.v.VirtualMachineManagerImpl] (qtp1603198149-17:ctx-430e157a ctx-91e054dc)
(logid:ef4bf833) Allocation completed for VM: VM instance {id: "119", name:
"i-2-119-VM", uuid: "13fe0bce-a240-48e4-9d5b-081359fcf422", type="User"}
./management-server.log:2022-03-15 12:38:41,962 DEBUG [c.c.v.UserVmManagerImpl]
(qtp1603198149-17:ctx-430e157a ctx-91e054dc) (logid:ef4bf833) Successfully
allocated DB entry for VM instance {id: "119", name: "i-2-119-VM", uuid:
"13fe0bce-a240-48e4-9d5b-081359fcf422", type="User"}
[...]
We can see above the time when the command deployVirtualMachine was
received (2022-0 3-1 5 12:38:40,334), and also various entries related to
resource allocation for the VM creation.
./management-server.log:2022-03-15 12:38:42,646 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(qtp1603198149-17:ctx-430e157a ctx-91e054dc) (logid:ef4bf833) submit async job-848, details:
AsyncJobVO {id:848, userId: 2, accountId: 2, instanceType: VirtualMachine, instanceId: 119, cmd
:org.apache.cloudstack.api.command.admin.vm.DeployVMCmdByAdmin, cmdInfo:
{"iptonetworklist[0].networkid":"f8ce6644-6478-4770-84a7-834bd8a717a2","boottype":
"BIOS","httpmethod":"POST","templateid":"b81aab72-2c37-466e-b53f-bdaf398322fa","ctxAccountId":
"2","uuid":"13fe0bce-a240-48e4-9d5b-081359fcf422","cmdEventType":"VM.CREATE","startvm":
"true","bootmode":"LEGACY","serviceofferingid":"ab647165-7a0a-4984-8452-7bfceb036528",
"response":"json","ctxUserId":"2","zoneid":"8b2ceb16-a2f2-40ea-8968-9e08984bdb17",
"ctxStartEventId":"1499","id":"119","ctxDetails":"{\"interface com.cloud.dc.DataCenter\":
\"8b2ceb16-a2f2-40ea-8968-9e08984bdb17\",\"interface com.cloud.template.VirtualMachineTemplate
\": \"b81aab72-2c37-466e-b53f-bdaf398322fa\",\"interface com.cloud.offering.ServiceOffering\"
:\"ab647165-7a0a-4984-8452-7bfceb036528\",
\"interface com.cloud.vm.VirtualMachine\":\"13fe0bce-a240-48e4-9d5b-081359fcf422\"}",
"affinitygroupids":""}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
resultCode: 0, result: null, initMsid: 90520745551922, completeMsid: null,
lastUpdated: null, lastPolled: null, created: null, removed: null}
./management-server.log:2022-03-15 12:38:42,651 DEBUG [c.c.a.ApiServlet]
(qtp1603198149-17:ctx-430e157a ctx-91e054dc) (logid:ef4bf833) ===END===
172.16.71.7 -- POST command=deployVirtualMachine&response=json
We can also see above the time when the command was finished.
(20 22-03-15 12:38:42,651). It is important to highlight that the request
processing shown until now generates a job, which can be processed by
any ACS Management Server from the cloud environment. Filtering en-
tries that contain the id from this job allows us to obtain more information
related to the VM creation process (the same applies for others processes
188
in ACS). In the second to last log entry displayed we can see that the job
was sent at 2022-03-15 12 :38:42,646 and identify the job id that the re-
quest generated: job-848.
3. Then, the logs containing the job id found must be searched using the
command:
grep -r "<job id>" ./
Using the information returned, it is possible to identify the ID from the
thread that processed the job:
[...]
./management-server.log:2022-03-15 12:38:42,647 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-5:ctx-c454c9f5 job-848) (logid:2f55511b) Executing AsyncJobVO
{id:848, userId: 2, accountId: 2, instanceType: VirtualMachine, instanceId: 119, cmd:
org.apache.cloudstack.api.command.admin.vm.DeployVMCmdByAdmin, cmdInfo:
{"iptonetworklist[0].networkid":"f8ce6644-6478-4770-84a7-834bd8a717a2","boottype":
"BIOS","httpmethod":"POST","templateid":"b81aab72-2c37-466e-b53f-bdaf398322fa",
"ctxAccountId":"2","uuid":"13fe0bce-a240-48e4-9d5b-081359fcf422","cmdEventType":
"VM.CREATE","startvm":"true","bootmode":"LEGACY","serviceofferingid":
"ab647165-7a0a-4984-8452-7bfceb036528", "response":"json","ctxUserId":"2","zoneid":
"8b2ceb16-a2f2-40ea-8968-9e08984bdb17","ctxStartEventId":"1499","id":"119","ctxDetails":
"{\"interface com.cloud.dc.DataCenter\":\"8b2ceb16-a2f2-40ea-8968-9e08984bdb17\",
\"interface com.cloud.template.VirtualMachineTemplate\":\"b81aab72-2c37-466e-b53f-bdaf398322fa
\", \"interface com.cloud.offering.ServiceOffering\":\"ab647165-7a0a-4984-8452-7bfceb036528\",
\"interface com.cloud.vm.VirtualMachine\":\"13fe0bce-a240-48e4-9d5b-081359fcf422\"}",
"affinitygroupids":""}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
resultCode: 0, result: null, initMsid: 90520745551922, completeMsid: null,
lastUpdated: null, lastPolled: null, created: null, removed: null}
[...]
4. Finally, after the thread ID is identified as (logid:2f5551 1b), it is impor-
tant to serch the logs for this ID, as it will return more entries than when
searching for the job id, allowing for a more thorough analysis process.
The command used is:
grep -r "<thread id>" ./
This process provides multiple details related to the VM creation process.
Below, we first see the logs associated to cluster identification, followed by
the ones identifying the host where the VM may be allocated. The existing
resources in the available options are compared to those defined during
the creation of the VM. This process is repeated for other hosts, and, if a
match occurs, they are added to a list of potential hosts. This process can
189
be observed in the last log entry shown below, where host 31 is added to
the aforementioned list.
./management-server.log:2022-03-15 12:38:43,984 DEBUG [c.c.d.FirstFitPlanner]
(API-Job-Executor-5:ctx-c454c9f5 job-848 ctx-20458e85) (logid:2f55511b) Listing
clusters in order of aggregate capacity, that have (at least one host with) enough
CPU and RAM capacity under this Zone: 1
./management-server.log:2022-03-15 12:38:44,003 DEBUG [c.c.d.FirstFitPlanner]
(API-Job-Executor-5:ctx-c454c9f5 job-848 ctx-20458e85) (logid:2f55511b) Removing
from the clusterId list these clusters from avoid set: []
./management-server.log:2022-03-15 12:38:44,054 DEBUG
[c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-5:ctx-c454c9f5 job-848
ctx-20458e85) (logid:2f55511b) Checking resources in Cluster: 1 under Pod: 1
./management-server.log:2022-03-15 12:38:44,078 DEBUG
[c.c.a.m.a.i.FirstFitAllocator] (API-Job-Executor-5:ctx-c454c9f5 job-848
ctx-20458e85 FirstFitRoutingAllocator) (logid:2f55511b) Looking for hosts in dc: 1
pod:1 cluster:1
./management-server.log:2022-03-15 12:38:44,091 DEBUG
[c.c.a.m.a.i.FirstFitAllocator] (API-Job-Executor-5:ctx-c454c9f5 job-848
ctx-20458e85 FirstFitRoutingAllocator) (logid:2f55511b) FirstFitAllocator has 3
hosts to check for allocation: [Host {"id": "31", "name": "cloudstack-lab-host-3",
"uuid": "2fd584d8-9fc3-4666-98f8-17f3e43e4348", "type"="Routing"}, Host {"id":
"30", "name": "cloudstack-lab-host-2", "uuid":
"3d7d8532-d0cf-476c-a36e-1b936d780abb", "type"="Routing"}, Host {"id": "29",
"name": "cloudstack-lab-host-1", "uuid": "11662552-8221-4081-92b5-a3f2c852754a",
"type"="Routing"}]
./management-server.log:2022-03-15 12:38:44,145 DEBUG
[c.c.a.m.a.i.FirstFitAllocator] (API-Job-Executor-5:ctx-c454c9f5 job-848
ctx-20458e85 FirstFitRoutingAllocator) (logid:2f55511b) Looking for speed=500Mhz, Ram=512 MB
./management-server.log:2022-03-15 12:38:44,146 DEBUG [c.c.c.CapacityManagerImpl]
(API-Job-Executor-5:ctx-c454c9f5 job-848 ctx-20458e85 FirstFitRoutingAllocator)
(logid:2f55511b) Host {id: 31, name: cloudstack-lab-host-3, uuid:
2fd584d8-9fc3-4666-98f8-17f3e43e4348} is KVM hypervisor type, no max guest limit check needed
./management-server.log:2022-03-15 12:38:44,173 DEBUG [c.c.c.CapacityManagerImpl]
(API-Job-Executor-5:ctx-c454c9f5 job-848 ctx-20458e85 FirstFitRoutingAllocator)
(logid:2f55511b) Host: 31 has cpu capability (cpu:6, speed:2394) to support
requested CPU: 1 and requested speed: 500
./management-server.log:2022-03-15 12:38:44,189 DEBUG [c.c.c.CapacityManagerImpl]
(API-Job-Executor-5:ctx-c454c9f5 job-848 ctx-20458e85 FirstFitRoutingAllocator)
(logid:2f55511b) Hosts's actual total CPU: 14364 and CPU after applying
overprovisioning: 14364
./management-server.log:2022-03-15 12:38:44,189 DEBUG [c.c.c.CapacityManagerImpl]
(API-Job-Executor-5:ctx-c454c9f5 job-848 ctx-20458e85 FirstFitRoutingAllocator)
(logid:2f55511b) Free RAM: (14.38 GB) 15444189184 , Requested RAM: (512.00 MB) 536870912
./management-server.log:2022-03-15 12:38:44,190 DEBUG [c.c.c.CapacityManagerImpl]
(API-Job-Executor-5:ctx-c454c9f5 job-848 ctx-20458e85 FirstFitRoutingAllocator)
(logid:2f55511b) Host has enough CPU and RAM available
./management-server.log:2022-03-15 12:38:44,191 DEBUG [c.c.c.CapacityManagerImpl]
(API-Job-Executor-5:ctx-c454c9f5 job-848 ctx-20458e85 FirstFitRoutingAllocator)
(logid:2f55511b) STATS: Can alloc MEM from host: 31, used: (256.00 MB) 268435456,
reserved: (0 bytes) 0, total: (14.63 GB) 15712624640; requested mem: (512.00 MB)
536870912, alloc_from_last_host?: false , considerReservedCapacity?: true
./management-server.log:2022-03-15 12:38:44,191 DEBUG
[c.c.a.m.a.i.FirstFitAllocator] (API-Job-Executor-5:ctx-c454c9f5 job-848
ctx-20458e85 FirstFitRoutingAllocator) (logid:2f55511b) Found a suitable host,
adding to list: 31
In the following entries it is possible to verify that a search for a compatible
storage is made, as well as a verification of potential hosts with access to
the storage.
190
./management-server.log:2022-03-15 12:38:45,069 DEBUG [c.c.s.StorageManagerImpl]
(API-Job-Executor-5:ctx-c454c9f5 job-848 ctx-20458e85) (logid:2f55511b) Found
storage pool storage-1-iscsi of type SharedMountPoint
./management-server.log:2022-03-15 12:38:45,069 DEBUG [c.c.s.StorageManagerImpl]
(API-Job-Executor-5:ctx-c454c9f5 job-848 ctx-20458e85) (logid:2f55511b) Total
capacity of the pool storage-1-iscsi with ID 5 is (50.00 GB) 53687091200
./management-server.log:2022-03-15 12:38:45,079 DEBUG [c.c.s.StorageManagerImpl]
(API-Job-Executor-5:ctx-c454c9f5 job-848 ctx-20458e85) (logid:2f55511b) Checking
pool: 5 for storage allocation , maxSize : (50.00 GB) 53687091200,
totalAllocatedSize : (9.20 GB) 9878175856, askingSize : (50.00 MB) 52428800,
allocated disable threshold: 0.85
./management-server.log:2022-03-15 12:38:45,090 DEBUG
[c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-5:ctx-c454c9f5 job-848
ctx-20458e85) (logid:2f55511b) Trying to find a potenial host and associated
storage pools from the suitable host/pool lists for this VM
./management-server.log:2022-03-15 12:38:45,093 DEBUG
[c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-5:ctx-c454c9f5 job-848
ctx-20458e85) (logid:2f55511b) Checking if host: 31 can access any suitable storage
pool for volume: ROOT
./management-server.log:2022-03-15 12:38:45,096 DEBUG
[c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-5:ctx-c454c9f5 job-848
ctx-20458e85) (logid:2f55511b) Host: 31 can access pool: 5
After all these verifications are performed, a host and a storage compati-
ble with the VM are identified in the log entries bellow:
./management-server.log:2022-03-15 12:38:45,100 DEBUG
[c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-5:ctx-c454c9f5 job-848
ctx-20458e85) (logid:2f55511b) Found a potential host id: 31 name:
cloudstack-lab-host-3 and associated storage pools for this VM
./management-server.log:2022-03-15 12:38:45,103 DEBUG
[c.c.d.DeploymentPlanningManagerImpl] (API-Job-Executor-5:ctx-c454c9f5 job-848
ctx-20458e85) (logid:2f55511b) Returning Deployment Destination:
Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] :
Dest[Zone(1)-Pod(1)-Cluster(1)-Host(31)-Storage(Volume(117|ROOT-->Pool(5))]
[...]
./management-server.log:2022-03-15 12:38:59,918 DEBUG
[c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-11:ctx-b5892741
job-848/job-849 ctx-b5ac54d3) (logid:2f55511b) Start completed for VM, VM instance
{id: "119", name: "i-2-119-VM", uuid: "13fe0bce-a240-48e4-9d5b-081359fcf422",
type="User"}
./management-server.log:2022-03-15 12:38:59,923 DEBUG [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-11:ctx-b5892741 job-848/job-849 ctx-b5ac54d3) (logid:2f55511b)
Done executing VM work job:
com.cloud.vm.VmWorkStart{"dcId":1,"podId":1,"clusterId":1,"hostId":31,"rawParams":
{"VmPassword":"rO0ABXQADnNhdmVkX3Bhc3N3b3Jk"},"userId":2,"accountId":2,"vmId":119,
"handlerName":"VirtualMachineManagerImpl"}
[...]
Finally we can see the moment when the process was finished by ACS (20
22-03-15 12:39:00,827).
[...]
./management-server.log:2022-03-15 12:39:00,585 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-5:ctx-c454c9f5 job-848
ctx-20458e85) (logid:2f55511b) Publish async job-848 complete on message bus
./management-server.log:2022-03-15 12:39:00,585 DEBUG
[o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-5:ctx-c454c9f5 job-848
ctx-20458e85) (logid:2f55511b) Wake up jobs related to job-848
./management-server.log:2022-03-15 12:39:00,585 DEBUG
[o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-5:ctx-c454c9f5 job-848
191
ctx-20458e85) (logid:2f55511b) Update db status for job-848
./management-server.log:2022-03-15 12:39:00,593 DEBUG
[o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-5:ctx-c454c9f5 job-848
ctx-20458e85) (logid:2f55511b) Wake up jobs joined with job-848 and disjoin all
subjobs created from job- 848
./management-server.log:2022-03-15 12:39:00,827 DEBUG
[o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-5:ctx-c454c9f5 job-848)
(logid:2f55511b) Done executing
org.apache.cloudstack.api.command.admin.vm.DeployVMCmdByAdmin for job-848
./management-server.log:2022-03-15 12:39:00,827 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-
Executor-5:ctx-c454c9f5 job-848) (logid:2f55511b) Remove job-848 from job monitoring
Therefore, as shown above, ACS log analysis was used to investigate the
VM’s creation, following the steps performed throughout the process. These
same principles applied here may be adapted and utilized to explore any event.
In CloudStack it is pretty common to employ this kind of analysis when in-
vestigating errors, faults, etc. The following command can be used to search
for problematic logs.
grep -i -E 'exception|unable|fail|invalid|leak|warn|error' ./
This analysis process provides a huge amount of information, making the
troubleshooting process very efficient.
8.2. Host/VM failover
ACS has a functionality called HA (High Availability), which, truly, applies the
failover concept, for hosts or for VMs in KVM.
17
The nomenclature used in ACS is wrong, as the concept of High Availabil-
ity dictates that when one of the environment nodes fails, the user does not
perceive this failure, because there is a replica in another node of the environ-
ment, ready to be used. However, that is not the ACS’s HA behaviour, as the
user VM’s are stopped and then recreated in another host, causing interrup-
tions. This behaviour is known as failover, and not HA.
Note: The SC Clouds team does not recommend the utilization of ACS’s HA
functionality, as enabling it requires deep platform and environment knowl-
edge, as well as intense monitoring of the environment as a whole and caution
17
when using another hypervisor, as VMware and XenServer, the HA process is executed on
the hypervisor side, external to CloudStack.
192
with operations regarding CloudStack’s Agents. It is necessary to validate its
multiple configurations, as any minor unavailability (in network, storage, etc)
can result in the activation of the VM failover process. The SC Clouds team has
already mapped issues to mature ACS’s failover functionality, in order to turn
it into a more robust functionality.
The following sections present the two “HA” modes supported by ACS.
8.2.1. Host HA
In order to work with the host HA feature, it is necessary for the Manage-
ment Servers to establish connection to the host’s OOBM (Out-of-band Man-
agement), and have OOBM configured and enabled in ACS. ACS monitors the
KVM Agents states and, in case of connection loss, starts processes to recover
the host. If the processes fail to recover the host, ACS flags the host as Do w
n (calling another shutdown via the OOBM interface), and VMs running in the
host are restarted in another available host.
Therefore, the operation team must always be aware of this behaviour when
executing procedures. Otherwise, the process of failover may be activated in
an inopportune moment and cause issues to user VMs, such as deploying the
same VM twice, or causing unnecessary downtime to the end user.
To perform operations, such as updating the Agent, it is necessary to put
the hosts in maintenance mode (causing the host’s evacuation, so the Agent’s
stop will not trigger the failover process).
Still, it is necessary to adjust the value of some global configurations, in order
to adequate ACS’s failover procedure to the operation’s procedure and achieve
the desired outcome. Below, are listed the configurations that impact the host
HA process:
193
Configuration Description Default
value
kvm.ha.activity.check.interval interval, in seconds, between ver-
ifications
60
kvm.ha.activity.check.max.
attempts
maximum number of verifica-
tions made before choosing
whether to recover or to degrade
the host
10
kvm.ha.recover.wait.period max period, in seconds, to wait
for the recovery of a resource
300
kvm.ha.recover.failure.
threshold
maximum number of recovery
attempts made before consider-
ing a host as degraded. The
counter is reset to zero when an
attempt succeeds
1
kvm.ha.activity.check.timeout max period, in seconds, to wait
for a verification to be completed
60
kvm.ha.degraded.max.period max period, in seconds, to only
attempt health checks in a de-
graded host, before executing
further operations
300
The kvm.ha.recover.fai lure.threshold configuration defines the limit of “re-
covery” attempts that will be executed. A recovery attempt is a reboot com-
mand sent to the host; because of the feature design, this results in multiple
host state changes, leading to a certain delay before executing the necessary
procedures to reestablish VMs’ services. Thus, it is recommended to set this
configuration value to 0, so no operation (reboot operation) will be issued and
the host will be identified as Down immediately.
Below, is an example of a possible configuration set, to achieve a period of
60 to 90 seconds to start the failover process:
194
Configuration Value
kvm.ha.activity.check.interval 15
kvm.ha.activity.check.max.attempts 1
kvm.ha.activity.check.timeout 15
kvm.ha.degraded.max.period 15
kvm.ha.recover.failure.threshold 0
kvm.ha.recover.wait.period 15
With the configurations updated, it is necessary to enable “HA” for each one
of the hosts. This can be achieved by accessing the Infrastructu re -> Ho sts
menu. By default, host’s HA is disabled for all hosts.
8.2.2. VM HA
VM HA comes as an alternative to the host’s HA. With this method, if the
VM is unexpectedly stopped, externally to ACS (because of a internal shutdown
command
18
, VM crash or other ways) and the Management Server still has
connection to the host’s Agent where the VM was running, it will be restarted
automatically in the same host.
In order to prevent the Management Server from losing access to the Agent,
it is necessary to have at least one NFS primary storage in the environment, so
the Agents can execute the heartbeat
19
to this storage.
If the storage is only used for the heartbeat process, it does not require high
capacity. The VM’s HA process is only executed if the VM’s compute offering has
this functionality enabled, or if the force.ha configuration is enabled.
This way, the failover process can be triggered whenever a VM or Agent
stops unexpectedly:
• ACS records the VMs’ status periodically and triggers the failover process
18
if a VM has the “HA” enabled, shuting the VM down from within the SO will trigger ACS to
reboot it again. Thus, to truly shut it down, it is necessary to do it through ACS.
19
during this process, if the Agent is unable to write on the heartbeat file, there is a behaviour
that can cause the host’s reboot. This behaviour is guided by the reboot.host.and.alert.manag
ement.on.heartbeat.timeout property, defined on the agent.properties file in each Agent, and
is disabled, by default, in all the SC Clouds’ deployments.
195
if there were no new VM records in more time than the defined in the
ping.interval configuration, multiplied by 2. Therefore, shutting down a
VM, with HA enabled, directly in the OS, will cause the VM’s restart, as ACS
does not know that the shutdown was intentional. In a scenario where
the configuration ping.interval is defined as 60 (default value), a VM crash
would activate the failover process after about 2 minutes.
• ACS sends, periodically, a “ping” to the environment Agents. If no re-
sponse is received in a period greater than the defined in the ping.inte
rval, multiplied by ping.timeout, ACS then sends a command to the other
hosts in the cluster, in order to investigate the host’s state. The hosts
validate the heartbeat file and, if they identify that the target host is not
updating the file, define it as inactive. Then, ACS starts the failover pro-
cess based on the neighbors discovery. In a scenario where the ping.int
erval configuration is set to 30 and the ping.timeout configuration is set
to 1.5, it would take 45 (p ing.timeout * 1.5) + 70 (commands hardcoded
timeout
20
) = 115 seconds to start the failover process.
If the Agent is stopped, it sends a signal to the Management Servers. So,
ACS will not try to “ping” the Agent and will not validate the host’s VM states;
that way, if something happens (as a host shutdown or a VM crash) while the
Agent is stopped, ACS will not take action.
Another point regarding this approach is that if a host loses access to the
storage, the failover process can be activated. After reestablishing the host con-
nection to the storage, as there is no stonith/fencing process in this scenario,
there is a possibility that the VM is still running in the target host, and still be
started in a new host, which may lead to block corruption.
20
we identified that the CloudStack’s community introduced a bug in the 4.19.0.0 version,
where one of the health check commands is using the wait configuration value as the timeout,
being that 30 minutes by default. Thus, the failover process will take more time to start. An
issue was mapped internally to fix this behaviour.
196
Even if the failover process is executed correctly, it is recommended that the
operation team executes validations to VMs that went through this process, as
there can be filesystem corruptions to the VM because of the host’s sudden
failure.
All the environment VRs and system VMs have VM HA enabled by default;
HA is disabled by default for the other VMs.
8.3. HA’s relation to heartbeat storage connection loss
If there is any unavailability to access the NFS storage, a series of effects
can occur. Independently of the HA configuration, each process that tries to
execute operations to this storage will remain blocked, halting the I/O until the
storage is available again.
This implies that some Agent’s worker threads can be stuck, as ACS period-
ically executes statistic collection both from the primary storage, and from the
volumes of VMs allocated in it. In the same way, operations such as enabling
the storage’s maintenance mode or starting and stoping VMs related to this
storage can result in threads blockage.
Depending on the intensity of operations, all the Agent’s worker threads
may be blocked, compromising a great part of the VMs’ functionalities, such as
their start, stop or snapshot creation.
Simultaneously, the hosts will continue to try to execute the heartbeat to
the NFS storage, but will always reach the timeout. When the property reboo
t.host.and.alert.man agement.on. heartbeat.t imeout is defined as true in the a
gent .properti es file, the host is automatically restarted after five consecutive
heartbeat failures (value adjustable by the property kvm.heartbeat.update.ma
x.tries in the same file).
Regarding the VMs, those with HA enabled and executing in the unavailable
NFS primary storage will not be restarted, as their process will remain active,
but they will be frozen if try to execute any I/O operation. VMs with HA en-
abled and executing in others primary storages will not be affected. Addition-
197
ally, when a HA enabled VM is shutdown internally while its host is active, its
restart will be executed normally, as the heartbeat file is not validated while the
host is active.
If a host fails, ACS then starts the investigation process. As the NFS primary
storage is unavailable, the neighbor hosts that try to read the heartbeat file
will remain stuck, waiting for I/O. After reaching the investigation timeout of
each neighbor, the degraded host is marked as disconnected, and ACS tries to
restart all of its VMs in another hosts, using the shared storage, including the
unavailable NFS primary storage.
That means that the VM HA process will continue to present the same be-
haviour. However, some platform operations will start to be stuck indefinitely.
To prevent the Agent’s worker threads blockage and reestablish the nor-
mal environment operation, it is necessary to remove the NFS primary storage.
This happens because, even when the primary storage is disabled or in main-
tenance mode, ACS still tries to execute some operations on it. As these oper-
ations result in blockages due to the storage unavailability, its exclusion must
be made manually in the database. It is important that, if an environment nor-
malization is required, the procedure must be performed together with the SC
Clouds team.
Actually, we are mapping some operations that present blockage, to imple-
ment, in the future, a functionality that will permit its expiration. Furthermore,
we have mapped an issue internally to allow operators to define which NFS pri-
mary storages should be used for Agent’s heartbeat. At the moment, Agents
execute the heartbeat in all of the environment NFS mounted storages.
8.4. Apache CloudStack services management
This section will present some basic operations related to managing cloudstack-
management, cloudstack-agent (only for KVM) and cloudstack-usage.
198
8.4.1. Managing the cloudstack-management
In some work flows, such as in updates, it is necessary for the Management
Servers to be stopped and started again for changes to be applied. In others,
such as in setting changes, just restarting them is enough. For such procedures,
follow these steps:
• Start: after this, the Management Server will start loading the modules. It
will only be available after the modules are loaded.
– To start the Management Server service, execute the command:
systemctl start cloudstack-management.service
– After it is started, it is possible to verify the service status by running
the command:
systemctl status cloudstack-management.service
– It is possible to track the modules loading in the logs:
tail -f /var/log/cloudstack/management/management-server.log
• Stop: after perfomed, if there is only one Management Server instance, it
will not be possible to interact with the orchestrator.
– To stop the Management Server service, execute the command:
systemctl stop cloudstack-management.service
– After the service stop it is possible to verify its status using the com-
mand:
systemctl status cloudstack-management.service
• Restart: stops and then starts the Management Server. Shares the same
behaviour as "Start" in relation to loading modules.
– To restart the Management Server service, execute the command:
199
systemctl restart cloudstack-management.service
– After the restart it is possible to verify the service status by running
the command:
systemctl status cloudstack-management.service
– It is possible to track the modules loading in logs:
tail -f /var/log/cloudstack/management/management-server.log
This process should not be executed simultaneously on all Management
Servers, since if all of them are stopped it will not be possible to use Cloud-
Stack.
8.4.2. Managing cloudstack-agent (for KVM hypervisor)
In clusters that use KVM as hypervisor, there is a component called Cloud-
Stack Agent. In some work flows, such as updates, it is necessary to stop and
start them again for the changes to be applied. In other cases, such as in setting
changes, just restarting them is enough. If the host HA feature is enabled, it is
recommended to disable it before performing the stop and restart. To perform
these procedures, follow these steps:
• Start: after this, the CloudStack Agent will try to connect to the Manage-
ment Server.
– To start the CloudStack Agent service, execute the command:
systemctl start cloudstack-agent.service
– After it starts, it is possible to verify the service status by running the
command:
systemctl status cloudstack-agent.service
– It is possible to track the CloudStack Agent initialization in logs:
tail -f /var/log/cloudstack/agent/agent.log
200
• Stop: when performed, all virtual machines in the host will keep working
normally, however CloudStack will not be able to manage them or create
new instances.
– To stop the CloudStack Agent service, execute the command:
systemctl stop cloudstack-agent.service
– After the service stops, it is possible to verify its status by running the
command:
systemctl status cloudstack-agent.service
• Restart: stops and then starts the CloudStack Agent. Shares the same
behaviour related to the connection to CloudStack as "Start".
– To restart the CloudStack Agent service, execute the command:
systemctl restart cloudstack-agent.service
– After the restart, it is possible to verify the service’s status by running
the command:
systemctl status cloudstack-agent.service
– It is possible to track the CloudStack Agent initialization in logs:
tail -f /var/log/cloudstack/agent/agent.log
8.4.3. Managing cloudstack-usage
Sometimes, such as when changing Usage Server settings, it is necessary
to restart it for the changes to be applied. For such procedures, follow these
steps:
• Start:
– To start the Usage Server service, execute the command:
systemctl start cloudstack-usage.service
201
– After it starts, it is possible to verify the service status by running the
command:
systemctl status cloudstack-usage.service
• Stop:
– To stop the Usage Server service, execute the command:
systemctl stop cloudstack-usage.service
– After the service stops it is possible to verify its status using the com-
mand:
systemctl status cloudstack-usage.service
• Restart: stops and then starts the Usage Server.
– To restart the Usage Server service, execute the command:
systemctl restart cloudstack-usage.service
– After it restarts, it is possible to verify the service’s status by running
the command:
systemctl status cloudstack-usage.service
8.5. System VMs
Within the Apache CloudStack architecture, system VMs are instances cre-
ated and managed by ACS to support the provisioning of certain cloud services.
Each of these instances is assigned a public IP.
By default, they only have the user root, set with the password password. It
is possible to configure ACS to define a new password when booting the sys-
tem VM. The section 8.5.5 describes in details how to perform this process.
However, it is not possible to access the system VMs via SSH using user and
password. More information in 8.5.4.
There are three types of system VMs:
Console Proxy Virtual Machine(CPVM), Secondary Storage Virtual Machine(SSVM)
and Virtual Router(VR).
202
8.5.1. Console Proxy Virtual Machine
The CPVMs, identified as v-<id>-VM within ACS, grant access to the console
of VMs managed by CloudStack. They work based on Virtual Network Com-
puting (VNC), which is a system for graphic user interface sharing, that uses a
protocol called Remote Frame Buffer (RFB) to transmit mouse and keyboard
inputs from a computer to another by sending screen updates through a net-
work. The user connects to the CPVM, which then perfoms a proxy connection
with the selected VM, allowing access to their console remotely.
Figure 140: Console Proxy Virtual Machine
8.5.2. Secondary Storage Virtual Machine
The SSVMs, identified as s-<id>-V M within ACS, make it possible to register
templates and ISOs, and also download and upload templates and ISOs.
Figure 141: Secondary Storage Virtual Machine
8.5.3. Virtual Router
The VRs, identified within ACS as r-<id>-VM, are interfaces between the VMs
and the world, managing networks and allowing the implementation of VPNs,
203
firewall and port-forwarding rules, among others features. Each isolated net-
work, shared network or VPC created in ACS will have a VR.
Figure 142: Virtual Router
8.5.3.1. Virtual Router health checks
CloudStack offers a feature to verify the integrity of virtual routers. Theses
health checks are divided in basic and advanced, to make it possible to set
different execution intervals, with the advanced being more computationally
expensive, and therefore, executed less frequently.
Aside from periodical tests, it is also possible to force a verification round
through the API getRouterHealthCheckResults, as long as the global setting ro
uter.health.checks.enabled is enabled.
204
Figure 143: Health checks display for a certain VR.
The points that are tested are:
• Connectivity between the virtual router and Management Servers;
• Connectivity to the virtual router interface gateways;
• Free disk space;
• CPU and memory usage;
• Service status for SSH, DNS, HAProxy and HTTP.
Also, for the advanced tests:
• Virtual router version;
• Correspondence between the DHCP/DNS settings and the ACS metadata;
205
• Correspondence between the HAProxy settings and the ACS metadata;
• Correspondence between the iptables rules and the ACS metadata.
There is no need to manually set the iptables, as this process can be executed
both on the API and the UI. Editing and updating the tables will be handled by
CloudStack.
The health checks can be adjusted through the following global settings:
Name Description Value
Default
router.alerts.check.interval Interval, in seconds, to verify VR
alerts.
1800
router.health.checks.enabled Enables or disables the VR health
checks.
true
router.health.checks.basic.
interval
Interval, in minutes, between ba-
sic health checks executions. If
set to 0, no tests will be sched-
uled.
3
router.health.checks.advanced.
interval
Interval, in minutes, between ad-
vanced health checks executions.
If set to 0, no tests will be sched-
uled.
10
router.health.checks.config.
refresh.interval
Interval, in minutes, between set-
tings updates from the Manage-
ment Server for the VR health
checks.
∗
10
router.health.checks.results.
fetch.interval
Interval, in minutes, between re-
sult updates from the Manage-
ment Server for health checks
in each update, the Manage-
ment Servers evaluate the need
to recreate the VR based on the
route r.health.checks.failur es.to.r
ecreate.vr setting.
∗
10
Table 46: Health check settings from virtual routers - Part 1
206
Name Description Value
Default
router.health.checks.failures.
to.recreate.vr
Failures during health checks de-
fined by this setting causes the VR
to restart. If empty, the restart
will not be performed for any
health checks failures.
router.health.checks.to.exclude Determines which health checks
must be ignored when executing
scheduled verifications. Comma
separated list containing script
names from the /root/health_ch
ecks/ folder.
router.health.checks.free.disk.
space.threshold
Minimum free disk space on the
VR, in MB, before resulting in a
fault.
100
router.health.checks.max.
cpu.usage.threshold
Maximum CPU usage for the VR,
in percentage, before resulting in
a fault.
100
router.health.checks.max.
memory.usage.threshold
Maximum memory utilization for
the VR, in percentage, before re-
sulting in a fault.
100
Table 47: Health check settings from virtual routers - Part 2
∗ This value must be sufficiently greater than router.health.checks.(basic/advance d).i nte rv
al to provide enough time to generate results between each update.
8.5.4. Accessing the system VMs
There are three ways to access the System VMs via SSH, depending on which
hypervisor is used or the machine distribution:
1. If the used hypervisor is KVM or XenServer, the access is made through
the host in which the System VM is running:
ssh -i /root/.ssh/id_rsa.cloud -p 3922 root@<systemvm_internal_ip>
2. If the used hypervisor is KVM there is also an alias to the command above,
facilitating its use:
cloudstack-ssh <systemvm_internal_ip>
207
3. Finally, if the hypervisor is VMware it is necessary to access them through
the Management Server using the command:
ssh root@<systemvm_internal_ip> -i ~cloud/.ssh/id_rsa
8.5.5. Changing the system VMs password
For security reasons, it might be interesting to change the system VMs’ pass-
words. It is possible to configure ACS to create a random password and apply
it to the system VMs’ root user when they are started. The following steps can
be executed to change the password ACS injects into the system VMs.
1. Change the global setting system.vm.random.password to true;
2. Restart the Management Servers, so they load the new value and create
a new global setting called system.vm.password, with a random value;
3. Recreate the system VMs, so ACS can inject the new password.
The random password generated by ACS will be accessible through the global
setting system.vm.passw ord and will be encrypted. To decrypt the password,
one of the following commands can be executed when the Management Server
is running:
1. In case ACS’s version being used precedes 4.18:
java -classpath /usr/share/cloudstack-common/lib/jasypt-1.9.3.jar org.jasypt.intf.cli.
JasyptPBEStringDecryptionCLI input="<data-to-be-decrypted>" password=$(cat /etc/cloudstack
/management/key) verbose="false"
2. In case ACS’s version being used is equal to 4.18 or later:
java -classpath /usr/share/cloudstack-common/lib/cloudstack-utils.jar com.cloud.utils.crypt.
EncryptionCLI -i "<data-to-be-decrypted>" -p=$(cat /etc/cloudstack/management/key) --
encryptorversion V2 -d
After defining the password for the first time, ACS will not change it any-
more. The password can be gathered through the global setting system.vm
.passwor d and, due to the encryption process, it will appear with a different
value for every request; however, the password will always be the same in the
database. Furthermore, it is possible to change the global setting system.vm
208
.passwo rd, passing the new password as plain text; ACS will be in charge of
encrypting it.
8.5.6. URL for CPVM and SSVM consumption
After perfoming downloads or uploads of templates, volumes or ISOs or
accessing a VM console, the services will be provided via URL. These URLs are
defined based on the following global settings.
Setting Description
consoleproxy.url.domain Domain used by the console
proxy VMs.
secstorage.ssl.cert.domain Domain used by secondary stor-
age VMs.
Table 48: System VM URL settings
Based on these settings, the URL for consuming the System VMs services
may have three formats:
• Empty: the public IP of the System VM is used; for example, 172.16.200.100.
• Static (demo.com.br): the domain is used; in this example, demo.com.br.
This approach works when there is only one CPVM or SSVM. When there
are more than one, it is necessary to discriminate in which VM the access
will occur using the dynamic format.
• Dynamic (*.demo.com.br): the public Ip of the VM is used in union with
the domain; for example, 172-16-200-100.demo.com.br.
8.6. Enabling VMs computational resources increase
Despite the functionality being supported for the VMware, XenServer and
KVM hypervisors, this section will only demonstrate how to setup the environ-
ment with the KVM hypervisor in order to enable changes to computational
resources (RAM and CPU) with the VM running. In the context of the KVM hy-
pervisor, this functionality exists with some caveats:
209
• The VM must be configured as dynamically scalable;
• The compute offering type of the VM must be either custom constrained
or custom unconstrained;
• The VM’s operating system must support memory and CPU hot (un)plug,
meaning, it must have the functionality to allocate and remove those phys-
ical resources during runtime.
21
• It is not possible to change either the vGPU or CPU families of the VM while
it is running, as operating systems commonly do not support this kind of
substitution while the VM is running.
• It is not possible to reduce the VM’s resources directly, only increase. How-
ever, if a VM downgrade needs to be performed, the user may stop it and
change its compute offering to a lesser offering.
To activate this feature it is necessary to change the global setting enable.d
ynamic.scale.vm, which has the default value false, to true.
8.7. Overprovisioning
Overprovisioning is the feature of offering more virtual resources than the
amount physically available. This exists due to how CloudStack accounts used
resources and how hypervisors deal with memory and CPU.
Let’s consider a situation where there is a 100Gb (100%) storage and a VM
will be created with a disk offering of 20Gb (20%). CloudStack considers that this
VM’s disk will be consuming 20Gb (20%) and the storage will remain with 80Gb
(80%) free. However, for most cases (depending on the Provisioning Type of
the disk offering) the disks will be allocated dynamically (created with minimum
size possible and then increased on demand until reaching defined limits) Thus,
CloudStack will account for 20Gb (20%), but the disk will have a variable size
21
By default, both Linux and Windows kernels support this functionality.
210
between 0Gb and 20Gb. In many cases the offered disk is not fully consumed
by the user, which leaves open space for others disks to be allocated.
CloudStack accounts 20Gb (20%) for the VM disk and 80Gb (80%) of free
storage, however the real value (for this example situation) is 10Gb (10%) in
use for the VM’s disk and 90Gb (90%) of free storage. In the end, there will be
free storage space but it will not be used because CloudStack considers it as
allocated.
In the situation described above it is possible to configure CloudStack to
consider the existence of more resources than there are physically. Setting it
to 2x, the total storage would be 200Gb (100%) and 20Gb (10%) would be used,
leaving 180Gb (90%) of free storage available.
For memory, KVM offers the KSM feature, that performs memory blocks
sharing. Thus, the more homogeneous the workloads are (VMs), the greater
the gain in memory optimization.
In a situation where 10 VMs exist with the same disk settings, operating sys-
tem, CPU and 2Gb of memory, for example, the memory consumption calcu-
lation would result in 20Gb, however, due to KSM, many memory blocks are
shared and the final result will be lower, which will make it possible to perform
memory overprovisioning.
Currently, there are 5 settings for overprovisioning:
211
Setting Description Value
Deafult
cpu.overprovisioning.factor Value used as the over-
provisioning factor for CPU.
CPU available after calcula-
tions will be current * fac-
tor .
1.0
mem.overprovisioning.factor Value used as the overpro-
visioning factor for mem-
ory. Memory available after
calculations will be current
* factor .
1.0
storage.overprovisioning.factor Value used as the overpro-
visioning factor for storage.
Storage available after cal-
culations will be current *
factor .
2.0
vm.min.cpu.speed.equals.cpu.speed.divided.
by.cpu.overprovisioning.factor
When overprovisioning
is utilized, ACS informs
a minimum CPU value,
with this value calculated
as CPU/overprovisioning,
regardless of whether the
service offering is scal-
able. This setting controls
if the behaviour will be
performed.
true
vm.min.memory.equals.memory.divided.by.
mem.overprovisioning.factor
When overprovisioning
is utilized, ACS informs
a minimum RAM value,
with this value calculated
as RAM/overprovisioning,
regardless of whether the
service offering is scal-
able. This setting controls
if the behaviour will be
performed.
true
Table 49: Overprovisioning settings
For both vm.min.cpu.speed.equals.cpu.speed.divided.by.cpu.overprovision
ing.factor and vm.min.memory.equals.memory.divided.by.mem.overprovision
212
ing.factor settings the default value is true, which keeps the existing behaviour,
however we recommend changing it to false, because we consider that the cal-
culation made by CloudStack on the deployment and migration of VMs to be
incoherent.
Except for the storage.overprovisioning.factor setting, which operates at
zone level, all other settings operate at cluster level.
Regarding CPU and memory overprovisioning, after changing the settings
values, the VMs must be shut down and started again for CloudStack to calcu-
late the proper allocated value.
8.8. Updating the Apache CloudStack
The LTS version of Apache CloudStack is periodically updated. This section
will address the steps needed to perfom updates. The changelog, specific re-
quirements and link for the release packages can be found in the Gitlab project,
in the section Deployments > Releases.
8.8.1. Major versions updates
After fulfilling the requirements and downloading the files, these are the
steps for updating the Management Server between major versions:
1. Stop the Management Server service:
systemctl stop cloudstack-management
2. Stop the Usage Server service:
systemctl stop cloudstack-usage
3. Dump the cloud database:
mysqldump -u root -p cloud > cloud-backup-before-update-`date '%Y-%m-%d-%H:%M:%S.%3N'`.sql
4. Dump the cloud_usage database:
mysqldump -u root -p cloud_usage > \
cloud_usage-backup-before-update-`date +'%Y-%m-%d-%H:%M:%S.%3N'`.sqlr
5. Install the new ACS Management Server packages (downloaded files will
have a version suffix):
213
• For .deb:
apt install cloudstack-common~bionic_all.deb cloud-stack-management~bionic_all.deb \
cloudstack-usage~bionic_all.deb cloudstack-ui~bionic_all.deb
• For .rpm:
yum install cloudstack-common.1.el7.centos.x86_64.rpm \
cloudstack-management.1.el7.centos.x86_64.rpm \
cloudstack-usage.1.el7.centos.x86_64.rpm cloudstack-ui.1.el7.centos.x86_64.rpm
6. Start the Management Server:
systemctl start cloudstack-management
7. Start the Usage Server:
systemctl start cloudstack-usage
If the hypervisor in use is KVM, follow these steps to update the CloudStack
Agents:
1. Stop the CloudStack Agent service:
systemctl stop cloudstack-agent
2. Install the new CloudStack Agent packages (downloaded files will have a
version suffix):
• For .deb:
apt install cloudstack-common~bionic_all.deb cloudstack-agent~bionic_all.deb
• For .rpm:
yum install cloudstack-common.1.el7.centos.x86_64.rpm \
cloudstack-agent.1.el7.centos.x86_64.rpm
3. Start the CloudStack Agent:
systemctl start cloudstack-agent
To finish the Management Server update:
8. Force restart the system VMs to apply the new template when deploying
them. Instructions on how to force restart the system VMs are found in
the official documentation;
214
9. Restart the Management Server service:
systemctl restart cloudstack-management
8.8.2. Updates within the same major version
To install an update within the same major version, it is crucial to carefully
read the release notes to verify if the updated is related to the Management
Server or to the CloudStack Agent. When the update is related to the Man-
agement Server, just follow through the steps 1 to 7 from Management Server
update. In turn, when the update is related to the CloudStack Agent, follow the
steps described in updating the KVM agent.
It is always important to check the Gitlab release notes, because, in addition
to important information about requirements and changes, they may also in-
clude notes about the version, which must be carefully read before proceding
with the specific ACS version update process.
8.9. SSL certificate update in the environment (nginx and ACS)
The following steps introduce the required actions to update the SSL certifi-
cates in the nginx and upload them to ACS.
8.9.1. Root and intermediary certificates extraction
Note: This step is necessary if the client only has the final certificate.
To perform the certificate’s extraction it is necessary to divide the final cer-
tificate in two files: one containing the certificate itself and another containing
the private key. After this, the intermediary certificate is extracted with the fol-
lowing commands:
user@scclouds :~ $ openssl x509 -in <domain>.crt -text -noout | grep -i 'issuer'
Issuer: C = US, ST = TX, L = Houston, O = "cPanel, Inc.", CN = "cPanel, Inc.
Certification Authority"
CA Issuers - URI:http://crt.comodoca.com/cPanelIncCertificationAuthority.crt
user@scclouds :~ $ curl -o caIntermediate
http://crt.comodoca.com/cPanelIncCertificationAuthority.crt
user@scclouds :~ $ open x509 -inform DER -in caIntermediate -outform PEM -out
caIntermediate.crt
With the intermediary certificate extracted, it is necessary to extract the root
certificate:
215
user@scclouds :~ $ openssl x509 -in caIntermediate.crt -text -noout | grep -i 'issuer'
Issuer: C = GB, ST = Greater Manchester, L = Salford, O = COMODO CA Limited, CN =
COMODO RSA Certification Authority
CA Issuers - URI:http://crt.comodoca.com/COMODORSAAddTrustCA.crt
user@scclouds :~ $ curl -o caRoot http://crt.comodoca.com/COMODORSAAddTrustCA.crt
user@scclouds :~ $ openssl x509 -inform DER -in caRoot -outform PEM -out caRoot.crt
8.9.2. Key conversion to PKCS#8:
user@scclouds :~$ openssl pkcs8 -topk8 -inform PEM -outform PEM -nocrypt -in
<domain>.key -out <domain>.pkcs8.key
8.9.3. Adding certificates in nginx
It is necessary to change the files /etc/nginx/certiïň Ącates/dom ain.c om.br
.cr t and /etc/nginx/certi ficates/domai n.com.br.pem with the contents of the
certificate and the desired private key. Then, just restart the nginx service with
the command:
systemctl restart nginx.service
8.9.4. Adding certificates in the Apache CloudStack
The certificate addition may be done via UI or via CloudMonkey:
Via UI:
216
Figure 144: Acessing the SSL certificate addition form via UI
217
Figure 145: Adding SSL certificate via UI
Via CloudMonkey the certificates are added with theuploadCustomCertificate
API.
• Root certificate
In a terminal, call the uploadCustomCertificate API using the parameters id
=1, name=root, domainsuffix and certificate. The certificate will be directly
loaded from the file through the command cat:
cloudmonkey upload customcertificate id=1 name=root domainsuffix="domain.com.br"
certificate="$(cat <root certificate's absolute path>)"
• Intermediary Certificates
In a terminal, do an API call to uploadCustomCertificate using the parame-
ters id, name, domainsuffix and certificate:
cloudmonkey upload customcertificate id=n name="intermediate<n-1>" domainsuffix="domain.com.br"
certificate="$(cat <intermediary certificate's absolute path>)"
218
This step must be repeated for each intermediary certificate, increasing
the n variable by 1 on each operation.
Note: the intermediary certificate’s name is n-1, because the first inter-
mediary certificate begins with ID 2 (root is the 1st).
• Website certificate
In a terminal, call the uploadCustomCertificate API using the parameters i
d, domainsuffix, certificate and privatekey:
cloudmonkey upload customcertificate id=n domainsuffix="domain.com.br"
certificate="$(cat <website certificate's absolute path>)" privatekey="$(cat <certificate's private key absolute path>)"
After sending the request with the privatek ey in the final certificate, ACS
will automatically restart the system VMs, both via API and via UI.
8.10. SSH key pairs
ACS has a functionality that allows injecting pre-defined SSH keys on the
VMs. To create them, go to the Compute section, then click SSH key pairs and
finally click Create a SSH Key Pair.
Figure 146: Accessing the SSH key pairs sections
219
Figure 147: Creating an SSH key pair
When creating an SSH key pair, it is necessary to take caution with the fol-
lowing details:
220
• For this feature to be available, the cloudinit
22
must be configured in the
VM.
• If the operator does not specify a public key, CloudStack will automatically
generate a key pair. The generated private key will immediately be avail-
able and will not be stored by CloudStack. For that, it is recommended for
the operator to download the given file.
22
For information regarding cloudinit refer to the Apache CloudStack Cloud usage documen-
tation
221
Figure 148: SSH key pair automatic creation
• If a domain is not specified during the creation, the key pair will be created
within the domain of the logged user.
• When specifying an account in the a c count field it is necessary to inform
the domainid for the domain where it belongs.
There are two possibilities when creating SSH key pairs via CloudMonkey:
222
• If the operator wants the keys to be generated automatically:
create sshkeypair name=<key_name> domainid=<domain_id> account=<account_name>
projectid=<project_id>
• If the operator wants to define their own key pair:
register sshkeypair name=<key_name> publickey=<key_value> domainid=<domain_id>
account=<account_name> projectid=<project_id>
After creating the key pair follow these steps to inject the public key in a VM:
223
Figure 149: Creating the SSH key
224
Figure 150: Creating instance and implementing the SSH key
225
9. KVM hypervisor
Kernel-based Virtual Machine (KVM) is an open source virtualization technol-
ogy under the GPL license that allows transforming the Linux OS into a hyper-
visor, turning it into a bare-metal hypervisor, making it possible for the host to
execute VMs, also known as guests.
KVM is responsible for managing the guest’s access to the host’s CPU and
RAM. To emulate other VM components, such as disks, graphic cards and USBs,
QEMU is used.
Figure 151: KVM virtualization
Designed for cloud solutions, KVM has some advantages over other hyper-
visors. Such as giving direct access to each host, without the need for a central-
izing element, such as vCenter from VMware or the master node in XenServer.
Facilitating their management for cloud orchestrators, such as OpenStack and
CloudStack, while simultaneously providing better performance.
The main pros and cons between the hypervisors supported by ACS are:
226
Figure 152: Hypervisors comparison
Another tool used alongside KVM is Libvirt, a set of softwares that ease vir-
tual machines management, including an API, a daemon (libvirtd) and a CLI
(virsh). Libvirt simply acts as a facilitator and must not be confused with KVM
itself.
9.1. KVM installation and CloudStack Agent
The first step is to install the following softwares: NFS
23
, NTP
24
, QEMU, KVM
and Libvirt:
23
NFS is the protocol used for storage.
24
NTP is necessary to synchronize server clocks in the cloud.
227
apt-get update
apt-get install nfs-common
apt-get install openntpd
apt-get install qemu-kvm libvirt-bin python3-libvirt virtinst bridge-utils cpu-checker
Then, download the latest release package of ACS (these may be found in
the section Deployments > Releases in the GitLab project). Next, install the
CloudStack Agent (there will be a version suffix on the downloaded files):
• For .deb:
apt install cloudstack-common_<acs-version>-scclouds~bionic_all.deb \
cloudstack-agent_<acs-version>-scclouds~bionic_all.deb
• For .rpm:
yum install cloudstack-common-<acs-version>-scclouds.1.el7.centos.x86_64.rpm \
cloudstack-agent-<acs-version>-scclouds.1.el7.centos.x86_64.rpm
9.2. KVM and CloudStack Agent setup
CloudStack allows configuring the CPU model exposed to KVM instances.
For that, edit the file /etc/cloudstack/agent/agent.properties, setting guest.cpu
.mode, guest.cpu.model and guest.cpu.features.
There are three possible values for guest.cpu.mode:
1. custom: Allows the operator to customize the CPU model. In this case,
the model must be specified in the guest.cpu.model setting. For example:
guest.cpu.mode=custom
guest.cpu.model=SandyBridge
The available models list for a certain architecture can be obtained through
the command:
virsh cpu-models <architecture>
For example:
$ virsh cpu-models
x86_64
pentium
pentium2
pentium3
pentiumpro
coreduo
228
n270
core2duo
qemu32
kvm32
cpu64-rhel5
cpu64-rhel6
qemu64
(...)
Furthermore, it should be highlighted that the file /usr/share/libvi r t / cp u _
map.xml contains a list of all CPU models and flags supported.
2. host-model: Here Libvirt will identify which model in /usr /share/libvir t/c
pu_map/ is more compatible with the host’s CPU model. This option of-
fers good performance, maintaining the possibility for migrations to hosts
with the same CPU architecture.
3. host-passthrough: Libvirt will communicate to KVM the exact character-
istics from the host’s CPU. The difference between this and host-model is
that instead of pairing CPU flags, all the CPU details from the host are
paired. This offers better performance, but at a cost related to the mi-
gration: the guest may only be migrated to a host with the excact same
CPU.
The g uest.cpu.features setting, on the other hand, represents CPU flags to
be applied, separated with spaces. For example:
guest.cpu.mode=custom
guest.cpu.model=SandyBridge
guest.cpu.features=aes mmx avx
In order to configure Libvirt, edit the file found in /etc/ libvirt/libvirtd.conf.
Here is a configuration example using the TCP protocol, without authentication
and using default ports:
listen_tls=0
listen_tcp=1
listen_addr = "0.0.0.0"
mdns_adv = 0
auth_tcp="none"
tcp_port="16509"
tls_port="16514"
auth_tls="none"
229
KVM also requires communication with several other components via net-
work. Therefore, the following TCP ports should be open in the firewall:
1. 22 (ssh)
2. 16509 (libvirt)
3. 16514 (libvirt tls)
4. 8250 (ACS system VMs)
5. 5900-6100 (VNC)
6. 49152-49216 (migration traffic)
9.3. KVM operation
Libvirt offers an array of features to facilitate the KVM operation. For ex-
ample, using Libvirt it is possible to describe the VMs using XML, making VM
manual creation and verification easier. To obtain the XML from a running VM,
execute the command:
virsh dumpxml <instance> >dump.xml
The resulting file should look similar to this:
<domain type='kvm' id='108'>
<name>r-1896-VM</name>
<uuid>c85d71db-3ce9-4c6f-98d3-b22f7e84058d</uuid>
<description>Debian GNU/Linux 5.0 (64-bit)</description>
<memory unit='KiB'>262144</memory>
<currentMemory unit='KiB'>262144</currentMemory>
<vcpu placement='static'>1</vcpu>
(...)
<sysinfo type='smbios'>
<system>
<entry name='manufacturer'>Apache Software Foundation</entry>
<entry name='product'>CloudStack KVM Hypervisor</entry>
<entry name='uuid'>c85d71db-3ce9-4c6f-98d3-b22f7e84058d</entry>
</system>
</sysinfo>
<os>
<type arch='x86_64' machine='pc-i440fx-2.11'>hvm</type>
(...)
If you want to create a new VM from an XML, run the command:
virsh create <VM_XML>
To edit a VM’s settings, the following command can be used:
virsh edit <instance>
230
It should, however, be noted that most modifications will only be applied
after restarting the VM.
Moreover, if XenServer is used, the following Libvirt commands are equiva-
lent:
virsh list = xe vm-list
virsh start = xe vm-start
virsh shutdown = xe vm-shutdown
Finally, if the user finds it necessary, it is possible to use the Virtual Machine
Manager, which is a graphic interface for working with Libvirt.
9.4. KVM’s CPU topology
When defining the CPU for a VM, it is possible to define the amount o vCPUs
that the VM will start with (current amount) and how many it can scale up to
without stopping (maximum amount). When working with fixed offerings (Sec-
tion 4.1), both values are equal and in the VM’s XML there is only the definition
for the current amount. The example below exhibits the definition of a fixed
offering with 4 vCPUs:
<vcpu placement='static'>4</vcpu>
When working with custom offerings (Section 4.1), the current amount and
the maximum amount are different, they are defined in the XML through the
property current and the tag value vcpu, respectively. Example of the XML def-
inition for a VM using a custom offering with initial value of 4 vCPUs and maxi-
mum value of 8:
<vcpu placement='static' current='4'>8</vcpu>
When defining the CPU topology for a VM in KVM, it is possible to inform the
quantity of sockets, cores and threads. ACS keeps the thread count fixed at 1
and derives the sockets and cores quantities based on the amount of vCPUs.
The multiplication of the quantity of sockets, cores and threads should result
in the maximum quantity of vCPUs that a VM may have (value of the tag vcpu).
As default, ACS derives the sockets quantity seeking to divide the core amount
by 6 or 4.
231
Example 1 : In a scenario with a custom offering with 4 vCPUs initially and
a maximum of 24, ACS tries to divide the maximum amount (24) firstly by 6. If
the remainder is 0, it uses the quotient and the divisor to define the topology;
in this case, the topology would be defined with 4 sockets and 6 cores each.
Example 2: On the other hand, in a scenario with a custom offering starting
with 4 vCPUs and a maximum of 16, ACS tries to divide the maximum amount
(16) firstly by 6. As the remainder is not 0, it tries the division by 4 instead;
then, the quotient and divisor are used to define the topology; which in this
case would be defined with 4 sockets and 4 cores each. If the remainder for
both divisions are not 0, the topology is not defined.
If the definition of the topology through the division by 6 or 4 is not adequate
for the context, it is also possible to overwrite this behaviour using the setting
cpu.corespersocket for each VM, that will replace the divisor for the definition
process.
Some usage examples for the cpu.corespersocket setting:
Example cpu.corespersocket max_vCPUs Result
1 12 24 2 sockets with 12 cores each
2 8 32 4 sockets with 8 cores each
3 1 12 12 sockets with 1 core each
4 2 20 10 sockets with 2 cores each
Table 50: Use case examples for cpu.corespersocket
9.5. CPU control with KVM
In CloudStack, the CPU control of the VMs in KVM can be done using the
following Libvirt parameters:
• shares:
Weight that indicates the priority for each vCPU to access the host’s CPU
when access concurrence occurs. For example, if all VMs have the shares
parameter defined as 1000, all will have the same priority. However, if a
VM has this value defined as 2000 while the others have the value defined
232
as 1000 for this setting, the VM with value 2000 will have twice the priority
for accessing the CPU. The interval accepted by this parameter depends
on the cgroups version used in the kernel of the current operating system
of the host. When using cgroups on version 1, the value of the flag shar
es must be between 2 and 262144; while in version 2, the interval range
goes from 1 to 10000.
• period:
Specifies the time interval (in microseconds) in which each vCPU must re-
spect the threshold set by the quota parameter. The value must be in the
range of 1000 to 1000000 microseconds.
• quota:
Defines the CPU time limit for each vCPU within the interval defined by pe-
riod. Both period and quota must be used together to enforce a CPU limit
on the VM’s vCPUs. Valid values range from 1000 to 17,592,186,044,415
microseconds or negative values. If quota is greater than period, a vCPU
may consume CPU time equivalent to more than one physical core. A
negative quota indicates no CPU limit, allowing unrestricted vCPU usage.
To list the values for these parameters in a specific instance, the following
command can be used:
virsh schedinfo <VM>
233
Figure 153: Output example for the command virsh schedinfo
The value of these parameters can be altered via ACS during the creation of
the service offering by enabling CPU cap, which allows you to limit the amount
of CPU usage for a VM, regardless of the amount available on the server.
With this parameter enabled, CloudStack calculates cpuQuotaPercentage =
Vm.MaxSpeed
Host.MaxSpeed
, and then the quota value is calculated by multiplying cpuQuo
taPercentage by the variable DEFAULT_PERIOD that has a fixed value of 10.000
in CloudStack. It’s worth to highlight that the quota parameter has a minimum
value of 1000, if the multiplication mentioned above returns a value lower than
that (equivalent to 10% of the DEFAULT_ PERIOD), CloudStack will replace the
result of the multiplication with the minimum value.
The period value is calculated in the following manner: period =
quota
cpuQuotaPercentage
,
this value is then compared with MAX_PERIOD, which has the value of 100000
0. If the division result is bigger than the value of this variable, period will be
defined as MAX_PERIOD, otherwise, the result of the division will be considered.
The value of shares is determined using the values of vm.speed, vm.minSpe
ed and vm.cpus, that are defined in the service offering. If the value of vm.minS
peed is null, the parameter value will be defined as shares = vm.cpus×vm.speed.
Otherwise, if vm.minSpeed has a defined value, then the parameter will be
calculated in the following manner: shares = vm.cpus × vm.minSpeed.
234
10. VMware hypervisor
This section describes how to integrate the VMware hypervisor into the Apache
CloudStack infrastructure.
The installation and setup of the ESXi hosts, as well as the vCenter, here
shown, were performed in a local laboratory setup using the KVM hypervisor,
therefore, when using this section as a guide for an installation on real hosts,
some procedures may differ.
Furthermore, the installed versions were:
• VMWare ESXi 6.5.0 Build 14320405
• Ubuntu 20.04.2 LTS
10.1. Creating datastores
Due to the use of a local laboratory environment, a storage VM was already
available. This VM employs the NFS protocol, with two additional paths ex-
ported to serve as primary datastores for the ESXi hosts and as a datastore for
vCenter deployment.
The new paths created were:
• Primary Datastore: /mnt/vmware-primary-storage
• Deploy Datastore: /mnt/vmware-deply-vcenter
These paths were also added to the settings file /etc/exports:
/mnt/vmware-primary-storage 192.168.31.0/24(rw,sync,no\_subtree\_check,no\_root\_squash)
/mnt/vmware-deply-vcenter 192.168.31.0/24(rw,sync,no\_subtree\_check,no\_root\_squash)
Next, the NFS service was restarted with the command:
systemctl restart nfs-server.service
235
10.2. Installing the ESXi hosts
The VMware installer 6.5 was utilized and the created VMs had five NICs, as
listed below:
• Public network: used to access the hosts via external networks. This net-
work was added to the laboratory to facilitate setting up the hosts, how-
ever its usage must be carefully considered in production environments,
keeping in mind that the ESXi hosts will require access to the external net-
work to search and apply updates to VMware.
• Management network: used for the communication with ACS.
• Primary storage network: used for the communication with the primary
datastore.
• Secondary storage network: used for the communication with the sec-
ondary datastore.
• Guest network: used for the VMs lateral communication.
The command used, in KVM, was:
virt-install --name esxi1 --vcpus 2 --memory 11000 --disk size=20 --graphics \
vnc,port=9999,listen=0.0.0.0 --network network:public --network bridge:br-management \
--network bridge:br-pri-storage --network bridge:br-sec-storage --network bridge:br-guests --cdrom \
~/isos/VMware-VMvisor-Installer-201908001-14320405.x86_64.iso
25
Next, the setup of the hosts was executed using Remmina, only following
through the default installation, described in the subsection below.
25
Here it is important to highlight that a VM with 10 GB RAM was created because the VMware
vCenter has limitations when adding ESXi hosts, as it blocks adding hosts with less than 10GB
RAM. This limitation, besides being noted in version 6.5 of VMware, can still be present in its
more recent versions.
236
10.2.1. Default ESXi hosts installation
1. At the beginning of the ESXi host installation process, press Enter to pro-
ceed:
Figure 154: Initial ESXi installation screen
2. After reading and agreeing, accept the terms by pressing F11:
Figure 155: Accept terms of use
3. A scan will be performed on the existing disks on the host. In the host
used for this document, only one disk was used for the installation. Envi-
ronments with multiple disks require attention when choosing the correct
disk:
237
Figure 156: Choosing the disk to install the system
4. Select the keyboard layout. In this example, the brazilian layout was used,
but the standard keyboard is from the USA:
Figure 157: Selecting the keyboard layout
5. Create a password for the root user. By default, the ESXi installation does
not allow creating new users:
Figure 158: Creating the root password
238
6. This warning occurred because the installation was performed within a
VM, in case this happens in a production environment it is possible to
proceed with the installation by pressing Enter:
Figure 159: Potential warning
7. Confirm the installation by pressing F11:
Figure 160: Confirm installation
8. The installation process will then start:
Figure 161: Host installation
9. After the process is finished, it will be necessary to restart the host, by
pressing Enter:
239
Figure 162: Restarting the host
10.2.2. ESXi hosts basic settings
After finishing the installation, it is recommended to log in to the hosts and
configure the following settings:
• Logging in to the host by pressing F2:
Figure 163: Initial login
240
Figure 164: The default user is root, and its password is the one set during installation
• Setting up static IP:
Go to the Configure Management Network section.
Figure 165: Section Configure Management Network
Then, go to the IPv4 Configuration section.
241
Figure 166: Section IPv4 Configuration
Enable the use of static IPv4. Here it is possible to change the IP used.
Figure 167: Static IPv4
By pressing ESC to go back, you will be asked to confirm changes. Accept
them.
242
Figure 168: Applying changes
• Enable SSH and shell access:
Access the Troubleshooting Options section.
Figure 169: Section Troubleshooting Options
Then, enable shell access on the host by pressing Enter.
243
Figure 170: Shell access on the host
Also, enable SSH access on the host by pressing Enter.
Figure 171: SSH access on the host
244
10.2.3. Advanced ESXi hosts settings
By default, ESXi hosts only have one IP, even if it has multiple NICs, in addi-
tion to the need to have their licenses manually activated, therefore, it is nec-
essary to perform some advanced settings that are only available in the web
interface provided by the hosts.
The link to access these web interfaces can be found on their login screen:
Figure 172: URL for accessing the web interface
Log in using the password created during the setup process.
Figure 173: Login screen
10.2.3.1. Adding new IPs
1. Go to Networking → Physical NICs to visualize the available NICs:
245
Figure 174: Available NICs
2. Go to Networking → Virtual switches and click in Add standard virtual
switch:
Figure 175: Virtual switches
3. Configure the new switch:
246
Figure 176: Configuring the new virtual switch
Repeat the steps 2 and 3 in case the host has more NICs.
4. Go to Networking → VMkernel NICs and click Add VMkernel NIC:
Figure 177: VM Kernel NICs
5. Configure the new VMkernel NIC:
247
Figure 182: Finish the datastore setup
5. Repeat this process for all datastores that must be added to the current
host.
10.2.3.3. Adding the license key
1. Go to Manage → Licensing and click Assign license:
Figure 183: Accessing the license
2. Type in your license and verify it:
250
Figure 184: Verifying if the license is valid
3. After validating the license, add it:
Figure 185: Adding the license
Repeat this process for all ESXi hosts in your infrastructure.
10.3. vCenter installation
Installing vCenter required creating an Ubuntu VM with a graphical inter-
face. Even though VMware’s documentation mentions the possibility of in-
stalling vCenter on Linux hosts through the cli-installer, in practice all attempts
failed with only a generic error message. After considerable research, the workaround
was to perform the installation on Linux with a graphical interface and, once
complete, removing it to save resources.
The steps followed to install vCenter were:
1. Linux graphical interface installation:
251
• Update the Ubuntu repositories:
apt update &&apt upgrade
27
• lightdm installation:
apt install lightdm
• Desktop environment installation
28
:
apt install ubuntu-desktop
• (OPTIONAL) Restart the host:
reboot
2. Copy the vCenter .ISO to the machine:
scp VMware-VCSA-all-6.5.0-18711281.iso user@vcenter-IP:~/
3. Mount the ISO
29
.
4. Access the files on the mounted ISO, browse to the directory
ISO-MOUNT-POINT/vcsa-ui-installer/lin64/ and execute the installer script
30
:
5. In the graphical interface, execute the script from the previous step and
the vCenter installation will start:
27
The upgrade is optional.
28
Other desktop environments may be used in this step, such as XFCE, Mate ,etc.
29
This can be made through command line or graphical interface.
30
This can be made through command line or graphical interface.
252
7. Accept the terms of use:
Figure 188: Terms of use
8. Select the type of structure where the deploy will be made:
Figure 189: Available deploy types
9. Add at least one ESXi host:
254
Figure 190: Adding ESXi host
A warning may occur mentioning the host certificate validity. Just ignore
it.
10. Define the root password in vCenter:
Figure 191: Adding the root password
11. Choose the size of the infrastructure where the deploy will be made:
255
Figure 192: Infrastructure size
12. Select one of the host datastores:
Figure 193: Available datastores
13. Configure the network used by vCenter:
256
Figure 194: Configuring the vCenter network
14. Confirm installation info and finish the process:
Figure 195: Finish configuring the vCenter
The vCenter installation process will begin, which may take several min-
utes to finish. If any error occurs, it is recommended to change the net-
work used by vCenter so that it uses DHCP instead of static IPv4 and repeat
the installation process.
257
15. Now, start configuring the PSC:
Figure 196: Start configuring the PSC
16. Either enable or disable SSH and define the synchronization mode that
will be used, based on your preferences:
Figure 197: Applying appliance basic settings
17. Create and setup the new SSO:
258
Figure 198: Creating and setting up a new SSO
18. Opt in, or not, to the VMware improvement program:
Figure 199: VMware improvement program
19. Review the settings applied and, if everything is appropriate, proceed with
the installation:
259
Figure 200: Finishing the setup
20. This process will take a few minutes and, after finishing, will allow access-
ing the vCenter web interface in the address provided at the end of the
installation:
Figure 201: Finishing the installation
The default login is administrator@<domain-defined-during-installation>,
with the same password defined during the installation.
After signing in, the following screen will be shown:
260
Figure 202: vSphere home screen
10.3.1. Adding license key
1. Go to Menu → Administration → Licensing → Licenses and click Add
New Licenses:
Figure 203: Licenses screen
2. Type in the license:
261
Figure 204: Adding the license
3. Rename the license:
Figure 205: Changing the license name
4. Finish adding the license:
262
Figure 206: Finish adding the license
10.3.2. Adding multiple ESXi hosts
In this document, the vCenter deploy was performed in one of the ESXi
hosts, therefore it will only be possible to add a second host. In a production
environment, it is recommended for the vCenter to be deployed in a dedicated
VM.
With that being said, to add ESXi hosts to vCenter, it is necessary to follow
these steps:
1. Create a datacenter:
• In the sidebar, right click photon-machine.public (or similar) and then
select New Datacenter:
263
Figure 207: Starting to add a new datacenter/folder
• Name the new datacenter:
Figure 208: Naming the new datacenter
2. Create a new cluster: right click the new datacenter created and then click
New Cluster. The procedure is the same as adding a datacenter.
3. With the new cluster created, right click it and then click Add Host:
264
Figure 209: Adding a new host to the datacenter
4. Inform the IP or hostname of the new host:
Figure 210: Adding the IP to the host
5. Inform user and password for this host:
Figure 211: User and password of the host
If any warning about the certificate comes up, accept it.
6. Confirm the host details:
265
Figure 212: Host details overview
7. Choose/confirm the host license:
Figure 213: Host license
8. Choose the host lockdown mode:
Figure 214: Host lockdown mode
9. Confirm the location:
266
Figure 215: VM location
10. Confirm the final details:
Figure 216: Final details
Then repeat these steps for each desired available host.
10.3.3. Removing the Linux graphical interface
Once the installation process is finished, it is no longer necessary to use a
graphical interface in Linux. The steps to remove it are:
• Removing lightdm:
apt remove lightdm
• Removing the desktop environment:
apt remove ubuntu-desktop
• Removing possible orphan packages from the graphical interface removal:
apt autoremove
267
• (OPTIONAL) Restart the host:
reboot
10.4. Adding VMware cluster in Apache CloudStack
To add a VMware cluster in Apache CloudStack it is necessary for ACS to have
access to the vCenter used via management network. Furthermore, depending
on the structure implemented in VMware, it may be necessary to create, in
VMware, new switches to the guest and storage networks used by ACS. Finally,
it may also be necessary to add, in ACS, the datastores used by VMware.
Once the VMware structure deployment is complete, it must be added as a
cluster in ACS. To do this, follow these steps:
1. Log in to ACS, browse to Infrastructure → Zones and access the zone
where the VMware cluster will be located.
2. In the zone, click Add VMware datacenter:
Figure 217: Adding a VMware datacenter
3. Configure the datacenter:
268
Figure 218: Configuring a VMware datacenter in Apache CloudStack
4. Still in the zone, browse to the Physical Network tab:
Figure 219: Accessing the physical networks details in Apache CloudStack
5. Access each of the networks and under their details go to the Traffic Types
tab and click Update Traffic Labels:
269
Figure 220: Updating networks
6. Update the VMware hypervisor label:
The standard syntax for this is:
[[“vSwitch Name/dvSwitch/EthernetPortProfile”][,“VLAN ID”[,“vSwitch
Type”]]]
Possible examples:
• Empty;
• dvSwitch0;
• dvSwitch0,200;
• dvSwitch1,300,vmwaredvs;
• myEthernetPortProfile„nexusdvs;
• dvSwitch0„vmwaredvs;
• vSwitch0„vmwaresvs
Where each field is:
• vSwitch Name/dvSwitch/EthernetPortProfile:
Virtual switch name in the vCenter (distributed or not), its default val-
ues depend on the virtual switch type:
270
– vSwitch0: If the switch is of the VMware vNetwork Standard virtual
switch type;
– dvSwitch0: If the switch is of the VMware vNetwork Distributed vir-
tual switch type and
– epp0: If the switch is of the Cisco Nexus 1000v Distributed virtual
switch type
• VLAN ID: VLAN ID in case it exists. Must only be used in public net-
works.
• vSwitch type: the possible values are:
– vmwaresvs: When using VMware vNetwork Standard virtual switch.
– vmwaredvs: When using VMware vNetwork distributed virtual switch.
– nexusdvs: When using Cisco Nexus 1000v distributed virtual switch.
Figure 221: Adding the VMware network mapping in the Apache CloudStack
271
7. With the network properly updated, browse to the Cluster section, click
Add Cluster and select the VMware hypervisor. This will enable a new set
of options, where the vCenter IP and other data must be supplied, such
as Datacenter name, user and password:
Figure 222: Configuring the VMware cluster in Apache CloudStack
8. Finally, it may be necessary to change the value for some global settings,
such as:
(a) vmware.management.portgroup;
(b) vmware.service.console.
It is important to emphasize that the Datacenter, either Cluster or Datastore,
in VMware must have the same name as in ACS, since the name is used during
the communication between them.
272
10.5. Problems when adding a VMware cluster
When adding a VMware cluster for the first time in a zone, ACS includes
a customizable tag in vCenter indicating that it is already under ACS manage-
ment. However, if any error occurs during the addition process of the vCenter,
the tag will not be removed, resulting in an error when the next addition is at-
tempted, since a vCenter that already has a tag can not be re-added to the ACS
management.
In the discussed case, the following error message is shown:
Figure 223: Tag error message on the UI when trying to add the vCluster
The process of removing a tag in order to add a vCluster is the following:
1. Access vCenter, select the zone, click Summary and under the section Cus-
273
tom Attributes click Edit....
Figure 224: vCenter zone selection
2. Select the attribute cloud.zone, then click the X button above to delete the
attribute and click OK when prompted.
274
Figure 225: Removing the cloud.zone attribute
3. Removing the VMware zone registry from the table vmware_data_center.
275
Figure 226: Selecting and erasing the vCluster from the database
After the previous steps, it is necessary to recreate the zone or add the clus-
ters and hosts manually.
More details about adding VMware to CloudStack may be found in the Apache
CloudStack official documentation.
10.6. Importing VMware VM to Apache CloudStack
Once the VMware cluster is correctly added to the ACS infrastructure, it will
be possible to import (either via UI or via API) the existing VMs in VMware to be
managed by ACS.
10.6.1. Importing VMs via UI
The new UI, with an existing VMware cluster in the ACS infrastructure, will
show a new view called Tools. By accessing this view, it is possible to import
VMs from vCenter to ACS:
• Access the Tools view:
276
Figure 227: Accessing the Tools view to import the VMs
• In this view, select the VMware zone, pod and cluster that were recently
added. After this, existing VMware VMs that are not managed by ACS will
be listed, as well as the VMs managed by ACS:
Figure 228: Checking VMware cluster VMs
• To import VMs, select them and then click the Import Instance button:
277
Figure 229: Importing a VM to the Apache CloudStack
• A pop-up will show up requiring some details, fill them accordingly with
your needs:
Figure 230: Details for VM import
• At the end of the procedure, if successful, the VM will be shown in the
Managed Instances column:
278
Figure 231: VM successfully imported
10.6.2. Importing VMs via API
To perform a import via CloudMonkey, the importUnmanagedInstance API
will be used:
• This API will import a VM from VMware to CloudStack. In the API call the
cluster ID in which the VM will be inserted must be informed, as well as
its name under VMware and the service offering that it will represent. It
is also possible to map disk offerings, network offerings, account and do-
main, and other properties.
Examples:
import unmanagedinstance clusterid=55d19ae9-a26d-43e3-9cb4-83d989cc882c name=test1
serviceofferingid=7fd97738-57b9-4240-9680-3dca279fa9a6
import unmanagedinstance clusterid=55d19ae9-a26d-43e3-9cb4-83d989cc882c name=test2
serviceofferingid=7fd97738-57b9-4240-9680-3dca279fa9a6 datadiskofferinglist[0].disk=disk1
datadiskofferinglist[0].diskOffering=1ef110db-1247-463b-b2ed-14ba1582c075
datadiskofferinglist[1].disk=disk2 datadiskofferinglist[1].diskOffering=07b97faf-740f-4c71-aae5-
b8a42e465617
import unmanagedinstance clusterid=55d19ae9-a26d-43e3-9cb4-83d989cc882c name=test3
serviceofferingid=7fd97738-57b9-4240-9680-3dca279fa9a6 datadiskofferinglist[0].disk=disk1
datadiskofferinglist[0].diskOffering=1ef110db-1247-463b-b2ed-14ba1582c075
nicNetworkList[0].nic="nic1" nicNetworkList[0].network=b9683956-8e1d-4128-8063-d3a0d8a4ceae
• The disks are informed in the form of a vector in the option datadiskof-
feringlist. The value of the disk must match the disk name under the VM
279
settings, which can be accessed in its details in vCenter. Only the VM’s
extra disks must be informed, because the root volume is automatically
imported (CloudStack always considers that the first disk retrieved from
the VM as the root). The disk option is the name of the disk in the VM
and the diskOffering is the ID for a compatible disk offering (can not be
lower than the disk capacity). When importing a VM with multiple disks, it
is possible to provide a list of disk to be imported, for example:
import unmanagedinstance name=test-unmanaged ... datadiskofferinglist[0].disk=51-2000
datadiskofferinglist[0].diskOffering=e9eb069f-2946-4e26-8d81-07aebba868d1
datadiskofferinglist[1].disk=52-2000 datadiskofferinglist[0].diskOffering=...
• The networks are informed through a vector in the nicNetworkList option,
where the nic option is the ethernet port and the network option is the
network UUID. If the VM network, in VMware, has a VLAN, it is necessary
for the informed network in ACS to have the same VLAN. It is also possible
to inform the IPs for each network, through the nicIpAddressList option,
where the nic option is the ethernet port and the ip4Address is the reserved
IP in CloudStack.
• When importing a VM from VMware to ACS it is important to be careful to
use a service offering that does not surpass the memory limitations from
the VMware VM. More details about this issue can be found in this link.
280
11. Conclusion
This document addressed, in general, the main concepts and recurring doubts
regarding Apache CloudStack and auxiliary tools. Both are vast and complex,
therefore it is not possible to adress them in their entirety, however, in case
of any doubts or suggestions for improvement in this documentation, an issue
can be created on GitLab.
281
Appendices
Appendix A. Terminology
• Infrastructure-as-a-Service:
Infrastructure-as-a-Service, also known as IaaS, consists of offering com-
putational resources, such as processing, storage and network access, on
demand, usually while applying pricing to the usage of these resources.
• Hypervisor:
Software responsible for the creation and execution of virtual machines
(VMs). A hypervisor allows a host machine to provide support to several
guest VMs, virtually sharing its resources, such as memory, storage and
processing, thus allowing better usage of the available resources.
• VM:
Acronym for Virtual Machine. It is a software that emulates a real com-
puter. Also known as guest, it is created on another computer, known as
host, and uses part of its resources. The advantages of using VMs are:
– Allows using different operating systems in a single computer.
– Extremely easy to manage and maintain.
– VMs can be easily created or replicated with an operating system pre-
viously installed and configured.
– VMs can also be easily migrated from one host to another without
data loss.
• Virtual network card:
Commonly referred to as NIC, VNIC or VIF. It is a software that fulfills the
role of a network card in a virtual system.
282
