Apache CloudStack Cloud Usage

This document presents the concept of cloud computing environments, under

the context of private clouds, and an introduction to the Apache CloudStack

cloud orchestration platform, focusing on its usage process.

Last update: January 9, 2026

Revision: a861aa100534f4f28acdbb7f732a7becb500075c

Contents

1 Introduction 26

1.1 Private cloud platforms . . . . . . . . . . . . . . . . . . . . . . . . . 27

1.1.1 Basic functionalities . . . . . . . . . . . . . . . . . . . . . . . 27

1.2 Apache CloudStack history . . . . . . . . . . . . . . . . . . . . . . . 29

2 Apache CloudStack functionalities 30

2.1 Home dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

2.2 Managing access to the cloud . . . . . . . . . . . . . . . . . . . . . . 30

2.2.1 Account . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32

2.2.1.1 Adding an account . . . . . . . . . . . . . . . . . . . 33

2.2.1.2 Visualizing account details . . . . . . . . . . . . . . 34

2.2.1.3 Removing an account . . . . . . . . . . . . . . . . . 35

2.2.1.4 Conﬁguring account accesses . . . . . . . . . . . . 36

2.2.1.5 Disabling an account . . . . . . . . . . . . . . . . . 37

2.2.1.6 Blocking an account . . . . . . . . . . . . . . . . . . 38

2.2.1.7 API access restriction by IP . . . . . . . . . . . . . . 39

2.2.2 Roles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39

2.2.2.1 Adding a new role . . . . . . . . . . . . . . . . . . . 40

2.2.2.2 Visualizing role details . . . . . . . . . . . . . . . . . 41

2.2.2.3 Customizing a role . . . . . . . . . . . . . . . . . . . 42

2.2.2.4 Removing a role . . . . . . . . . . . . . . . . . . . . 42

2.2.3 Domains . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44

2.2.3.1 Adding a domain . . . . . . . . . . . . . . . . . . . . 44

2.2.3.2 Conﬁguring the domain . . . . . . . . . . . . . . . . 46

2.2.3.3 Deﬁning the domain accesses . . . . . . . . . . . . 46

2.2.3.4 Restructuring the domain tree . . . . . . . . . . . . 47

2.2.4 Users . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48

2.2.4.1 Adding a user . . . . . . . . . . . . . . . . . . . . . . 48

2.2.4.2 Editing a user . . . . . . . . . . . . . . . . . . . . . . 50

2.2.4.3 Deleting a user . . . . . . . . . . . . . . . . . . . . . 51

2.2.4.4 Disabling/Enabling a user . . . . . . . . . . . . . . . 52

2.2.4.5 Password policies . . . . . . . . . . . . . . . . . . . 53

2.2.4.6 Disabling the user for entering wrong password . 55

2.2.4.7 Managing user API keys . . . . . . . . . . . . . . . . 55

2.2.4.8 API keys usage in CloudMonkey . . . . . . . . . . . 57

2.2.4.9 Two factors authentication . . . . . . . . . . . . . . 57

2.2.4.10 Timezone . . . . . . . . . . . . . . . . . . . . . . . . 62

2.2.5 Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 63

2.2.5.1 Adding a project . . . . . . . . . . . . . . . . . . . . 63

2.2.5.2 Interaction between domain admins and project

admins with project roles . . . . . . . . . . . . . . . 68

2.2.5.3 Adding accounts and users to the project . . . . . 68

2.2.5.4 Editing a project . . . . . . . . . . . . . . . . . . . . 71

2.2.5.5 Suspending a project . . . . . . . . . . . . . . . . . 72

2.2.5.6 Deleting a project . . . . . . . . . . . . . . . . . . . 73

2.2.6 Notes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

2.3 VM management . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

2.3.1 Adding a VM . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

2.3.1.1 Adding a VM as root admin . . . . . . . . . . . . . . 75

2.3.1.2 Adding a VM as admin . . . . . . . . . . . . . . . . 77

2.3.1.3 Adding a VM as user . . . . . . . . . . . . . . . . . . 80

2.3.1.4 Creating a VM with UEFI initialization . . . . . . . . 82

2.3.2 Removing a VM . . . . . . . . . . . . . . . . . . . . . . . . . . 86

2.3.3 Starting and stopping VMs . . . . . . . . . . . . . . . . . . . 87

2.3.4 Restarting a VM . . . . . . . . . . . . . . . . . . . . . . . . . . 87

2.3.5 Accessing VMs via web interface . . . . . . . . . . . . . . . . 88

2.3.6 Increasing computational resources from a VM . . . . . . . 89

2.3.7 Assigning a VM to another account . . . . . . . . . . . . . . 92

2.3.8 Adding a user data script to the VM . . . . . . . . . . . . . . 93

2.3.9 VM settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

2.3.9.1 General settings . . . . . . . . . . . . . . . . . . . . 100

2.3.9.2 Settings for VMs with custom compute oﬀering . 100

2.3.9.3 Miscellaneous settings for internal usage . . . . . 100

2.3.9.4 Settings for importing VMs with NIC, disk and cus-

tom parameters for custom compute oﬀerings . . 101

2.3.9.5 Adding VM settings via API . . . . . . . . . . . . . . 101

2.3.10 Limit on network consumption for user VMs’ NICs . . . . . 101

2.4 Managing volumes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102

2.4.1 Operations in data disks and root disks . . . . . . . . . . . . 103

2.4.1.1 Viewing disks and volumes . . . . . . . . . . . . . . 103

2.4.1.2 Creating a volume . . . . . . . . . . . . . . . . . . . 104

2.4.1.3 Uploading a local volume . . . . . . . . . . . . . . . 105

2.4.1.4 Uploading a local volume using cURL . . . . . . . . 106

2.4.1.5 Volume upload/register via URL . . . . . . . . . . . 107

2.4.1.6 Adding a volume to a VM . . . . . . . . . . . . . . . 108

2.4.1.7 Removing a volume from a VM . . . . . . . . . . . 109

2.4.1.8 Removing and adding a ROOT volume . . . . . . . 109

2.4.1.9 Resizing a data disk volume . . . . . . . . . . . . . 110

2.4.1.10 Volume download . . . . . . . . . . . . . . . . . . . 111

2.4.1.11 Recurring snapshots . . . . . . . . . . . . . . . . . . 112

2.4.1.12 Removing a volume . . . . . . . . . . . . . . . . . . 113

2.5 Guest networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

2.5.1 Isolated networks . . . . . . . . . . . . . . . . . . . . . . . . . 113

2.5.1.1 Egress rules . . . . . . . . . . . . . . . . . . . . . . . 114

2.5.2 Shared networks . . . . . . . . . . . . . . . . . . . . . . . . . 115

2.5.3 L2 network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117

2.5.4 Guest network operations . . . . . . . . . . . . . . . . . . . . 118

2.5.4.1 Adding a guest network . . . . . . . . . . . . . . . . 119

2.5.4.2 Acessing details of a guest network . . . . . . . . . 119

2.5.4.3 Removing networks . . . . . . . . . . . . . . . . . . 119

2.5.4.4 Editing a network . . . . . . . . . . . . . . . . . . . 120

2.5.4.5 Restarting a network . . . . . . . . . . . . . . . . . 120

2.5.4.6 Network permissions . . . . . . . . . . . . . . . . . 121

2.6 Public IPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123

2.6.1 Operations with public IPs . . . . . . . . . . . . . . . . . . . 124

2.6.1.1 Firewall setup . . . . . . . . . . . . . . . . . . . . . . 124

2.6.1.2 Enabling static NAT . . . . . . . . . . . . . . . . . . 125

2.6.1.3 Enabling port forwarding . . . . . . . . . . . . . . . 126

2.6.1.4 Enabling load balancing . . . . . . . . . . . . . . . . 128

2.6.2 Public IP acquisition . . . . . . . . . . . . . . . . . . . . . . . 129

2.6.3 Public IP Quarantine . . . . . . . . . . . . . . . . . . . . . . . 132

2.7 Virtual Private Cloud (VPC) . . . . . . . . . . . . . . . . . . . . . . . . 133

2.7.1 VPC operations . . . . . . . . . . . . . . . . . . . . . . . . . . 133

2.7.1.1 Adding a VPC . . . . . . . . . . . . . . . . . . . . . . 133

2.7.1.2 Restarting a VPC . . . . . . . . . . . . . . . . . . . . 135

2.7.1.3 Removing a VPC . . . . . . . . . . . . . . . . . . . . 136

2.7.1.4 Public IP acquisition for a VPC . . . . . . . . . . . . 137

2.7.2 Tiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

2.7.2.1 Adding tiers to a VPC . . . . . . . . . . . . . . . . . 139

2.7.2.2 Removing tiers from a VPC: . . . . . . . . . . . . . 141

2.7.3 Network ACL . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

2.7.4 Domain VPC . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145

2.7.4.1 Allocating public IPs in domain VPCs . . . . . . . . 146

2.8 Virtual Private Network (VPN) . . . . . . . . . . . . . . . . . . . . . . 147

2.8.1 VPN types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147

2.8.2 Operations with VPNs . . . . . . . . . . . . . . . . . . . . . . 147

2.8.2.1 Creating a VPN . . . . . . . . . . . . . . . . . . . . . 147

2.8.2.2 Disabling a VPN for a VPC . . . . . . . . . . . . . . . 150

2.8.2.3 Managing users . . . . . . . . . . . . . . . . . . . . 151

2.8.2.4 Adding a user . . . . . . . . . . . . . . . . . . . . . . 151

2.8.2.5 Deﬁning maximum VPN users per account . . . . 152

2.8.2.6 Removing a user . . . . . . . . . . . . . . . . . . . . 152

2.8.2.7 Connecting to the VPN via Linux . . . . . . . . . . . 152

2.8.2.8 Connecting to the VPN via Windows . . . . . . . . 155

2.8.3 S2S (Site-to-Site) VPNs . . . . . . . . . . . . . . . . . . . . . . 159

2.9 Signing HTTP requests . . . . . . . . . . . . . . . . . . . . . . . . . . 164

2.10 Templates and ISOs . . . . . . . . . . . . . . . . . . . . . . . . . . . 170

2.10.1 Operations with templates/ISOs . . . . . . . . . . . . . . . . 170

2.10.1.1 Viewing . . . . . . . . . . . . . . . . . . . . . . . . . 171

2.10.1.2 Listing . . . . . . . . . . . . . . . . . . . . . . . . . . 173

2.10.1.3 Registry . . . . . . . . . . . . . . . . . . . . . . . . . 174

2.10.1.4 Upload . . . . . . . . . . . . . . . . . . . . . . . . . . 177

2.10.1.5 Editing . . . . . . . . . . . . . . . . . . . . . . . . . . 179

2.10.1.6 Editing template/ISO permissions . . . . . . . . . . 181

2.10.1.7 Editing the template sharing . . . . . . . . . . . . . 182

2.10.1.8 Removing a template/ISO . . . . . . . . . . . . . . . 183

2.10.1.9 Creating a template based on an existing VM . . . 184

2.11 Resource Tags . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186

2.12 Aﬃnity Groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187

2.12.1 Aﬃnity groups types . . . . . . . . . . . . . . . . . . . . . . . 187

2.12.2 Aﬃnity groups operations . . . . . . . . . . . . . . . . . . . . 188

2.12.2.1 Adding an aﬃnity group . . . . . . . . . . . . . . . 188

2.12.2.2 Adding a VM to an aﬃnity group . . . . . . . . . . 189

2.12.2.3 Viewing VMs in the aﬃnity group . . . . . . . . . . 190

2.12.2.4 Removing an aﬃnity group . . . . . . . . . . . . . . 191

2.13 Autoscale VM groups . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191

2.13.1 Creating an autoscale VM group . . . . . . . . . . . . . . . . . 192

2.13.1.1 Setting up a scale-down policy . . . . . . . . . . . . 197

2.13.1.2 Conﬁguring autoscale VM group details . . . . . . 197

2.13.1.3 Group creation . . . . . . . . . . . . . . . . . . . . . 198

2.13.2 Testing the autoscaling . . . . . . . . . . . . . . . . . . . . . 199

2.13.3 Editing an autoscale VM group . . . . . . . . . . . . . . . . . 200

2.13.4 Removing an autoscale VM group . . . . . . . . . . . . . . . 203

2.14 Kubernetes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204

2.14.1 Kubernetes versions supported . . . . . . . . . . . . . . . . 205

2.14.2 Adding a Kubernetes ISO to Apache CloudStack . . . . . . . 205

2.14.3 Removing a Kubernetes ISO . . . . . . . . . . . . . . . . . . 207

2.14.4 Kubernetes clusters . . . . . . . . . . . . . . . . . . . . . . . 208

2.14.4.1 Creating Kubernetes clusters . . . . . . . . . . . . 209

2.14.4.2 Kubernetes cluster scaling . . . . . . . . . . . . . . 211

2.14.4.3 Upgrading a Kubernetes cluster . . . . . . . . . . . 212

2.14.5 Kubernetes access . . . . . . . . . . . . . . . . . . . . . . . . 212

2.14.6 Generating a token for the dashboard . . . . . . . . . . . . 214

2.15 Snapshots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214

2.15.1 Snapshot types . . . . . . . . . . . . . . . . . . . . . . . . . . 214

2.15.1.1 Volume/Disk snapshot . . . . . . . . . . . . . . . . 214

2.15.1.2 VM snapshot . . . . . . . . . . . . . . . . . . . . . . 215

2.15.2 Snapshot operations . . . . . . . . . . . . . . . . . . . . . . . 215

2.15.2.1 Volume snapshot . . . . . . . . . . . . . . . . . . . 216

2.15.2.2 Vm snapshot . . . . . . . . . . . . . . . . . . . . . . 217

2.15.2.3 Viewing created snapshots . . . . . . . . . . . . . . 218

2.15.2.4 Creating a template from a volume snapshot . . . 219

2.15.2.5 Creating a volume from a volume snapshot . . . . 220

2.15.2.6 Reverting a volume snapshot . . . . . . . . . . . . 221

2.15.2.7 Removing a volume snapshot . . . . . . . . . . . . 222

2.15.2.8 Creating a volume snapshot from a VM snapshot 222

2.15.2.9 Reverting a VM snapshot . . . . . . . . . . . . . . . 224

2.15.2.10Removing a VM snapshot . . . . . . . . . . . . . . . 224

2.15.2.11Conﬁguring recurrent snapshots . . . . . . . . . . 225

2.16 Events . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225

2.16.1 Event types . . . . . . . . . . . . . . . . . . . . . . . . . . . . 225

2.16.2 Searching and ﬁltering events . . . . . . . . . . . . . . . . . 225

2.16.3 Removing or archiving events . . . . . . . . . . . . . . . . . 226

2.17 User interface (UI) . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227

2.17.1 Using the browser’s timezone . . . . . . . . . . . . . . . . . 227

2.17.2 UI documentation . . . . . . . . . . . . . . . . . . . . . . . . 227

2.18 Service oﬀerings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228

2.18.1 Disk oﬀerings . . . . . . . . . . . . . . . . . . . . . . . . . . . 228

2.18.1.1 Creating a disk oﬀering . . . . . . . . . . . . . . . . 228

2.18.1.2 Editing a disk oﬀering . . . . . . . . . . . . . . . . . 232

2.18.1.3 Removing a disk oﬀering . . . . . . . . . . . . . . . 234

2.18.2 Compute oﬀerings . . . . . . . . . . . . . . . . . . . . . . . . 235

2.18.2.1 Creating a compute oﬀering . . . . . . . . . . . . . 235

2.18.2.2 Editing a compute oﬀering . . . . . . . . . . . . . . 240

2.18.2.3 Removing a compute oﬀering . . . . . . . . . . . . 241

2.18.2.4 Changing the compute oﬀering of a VM . . . . . . 242

2.18.3 Network oﬀerings . . . . . . . . . . . . . . . . . . . . . . . . 244

2.18.3.1 Creating a network oﬀering . . . . . . . . . . . . . 244

2.18.3.2 Editing a network oﬀering . . . . . . . . . . . . . . 248

2.18.3.3 Removing a network oﬀering . . . . . . . . . . . . 250

2.18.4 VPC oﬀering . . . . . . . . . . . . . . . . . . . . . . . . . . . . 251

2.18.4.1 Creating a VPC oﬀering . . . . . . . . . . . . . . . . 251

2.18.4.2 Editing a VPC oﬀering . . . . . . . . . . . . . . . . . 252

2.18.4.3 Removing a VPC oﬀering . . . . . . . . . . . . . . . 254

3 CloudMonkey, API and others 256

3.1 Installation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256

3.2 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256

3.3 Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257

4 Resources consumption accounting 262

4.1 Usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262

4.1.1 Resource usage report . . . . . . . . . . . . . . . . . . . . . . 262

4.2 Quota . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263

4.2.1 Tariﬀs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263

4.2.2 Reports . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263

4.2.2.1 Credits . . . . . . . . . . . . . . . . . . . . . . . . . . 264

4.2.2.2 Resource usage . . . . . . . . . . . . . . . . . . . . 265

4.2.2.3 Balance . . . . . . . . . . . . . . . . . . . . . . . . . 268

5 Cloud-init 271

5.1 Initialization steps . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271

5.2 Execution frequency . . . . . . . . . . . . . . . . . . . . . . . . . . . 272

5.3 Known datasources . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273

5.4 Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273

5.5 Creating a customized image . . . . . . . . . . . . . . . . . . . . . . 274

5.6 Cloud-init in Windows . . . . . . . . . . . . . . . . . . . . . . . . . . 277

5.7 Troubleshooting cloud-init . . . . . . . . . . . . . . . . . . . . . . . . 277

5.8 Force password change during ﬁrst login . . . . . . . . . . . . . . . 277

5.9 Self incrementing disk size during boot . . . . . . . . . . . . . . . . 278

5.10 Interactions with user-data . . . . . . . . . . . . . . . . . . . . . . . 279

6 Conclusion 280

Appendix A Terminology 281

List of Figures

1 Cloud computing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27

2 Home dashboard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30

3 Relational diagram between accounts, domains, projects and users. 32

4 Browsing to account adding page . . . . . . . . . . . . . . . . . . . 33

5 Adding a new account . . . . . . . . . . . . . . . . . . . . . . . . . . 34

6 Browsing to the account details page . . . . . . . . . . . . . . . . . 34

7 Checking account details . . . . . . . . . . . . . . . . . . . . . . . . . 35

8 Removing an account . . . . . . . . . . . . . . . . . . . . . . . . . . 35

9 Conﬁrming the account removal . . . . . . . . . . . . . . . . . . . . 36

10 Editing limits for an account . . . . . . . . . . . . . . . . . . . . . . . 36

11 Editing the account settings . . . . . . . . . . . . . . . . . . . . . . . 37

12 Disabling an account . . . . . . . . . . . . . . . . . . . . . . . . . . . 37

13 Conﬁrming disabling the account . . . . . . . . . . . . . . . . . . . 38

14 Blocking an account . . . . . . . . . . . . . . . . . . . . . . . . . . . 38

15 Conﬁrming an account’s block . . . . . . . . . . . . . . . . . . . . . 38

16 Deﬁning two IPv4 ranges and one IPv6 range . . . . . . . . . . . . 39

17 Starting to create a new role . . . . . . . . . . . . . . . . . . . . . . 40

18 Creating the new role . . . . . . . . . . . . . . . . . . . . . . . . . . 41

19 Accessing the newly created role . . . . . . . . . . . . . . . . . . . . 41

20 Customizing the role’s rules . . . . . . . . . . . . . . . . . . . . . . . 42

21 Starting to edit a role . . . . . . . . . . . . . . . . . . . . . . . . . . . 42

22 Editing the role . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

23 Removing a role . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43

24 Conﬁrming the role removal . . . . . . . . . . . . . . . . . . . . . . 43

25 Accessing the domains page . . . . . . . . . . . . . . . . . . . . . . 44

26 Creating a new domain . . . . . . . . . . . . . . . . . . . . . . . . . 45

27 Deﬁning the new domain’s name . . . . . . . . . . . . . . . . . . . . 45

28 Visualizing the created domain . . . . . . . . . . . . . . . . . . . . . 45

29 Selecting the created domain . . . . . . . . . . . . . . . . . . . . . . 46

30 Starting to edit the new domain . . . . . . . . . . . . . . . . . . . . 46

31 Editing the domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47

32 Browsing to the accounts page . . . . . . . . . . . . . . . . . . . . . 49

33 Select the account that the new user will be part of . . . . . . . . . 49

34 Check the users from the account . . . . . . . . . . . . . . . . . . . 50

35 Adding a new user . . . . . . . . . . . . . . . . . . . . . . . . . . . . 50

36 Deﬁne the new user’s informations and save the changes . . . . . 50

37 Selecting the user to edit it . . . . . . . . . . . . . . . . . . . . . . . 51

38 Starting to edit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

39 Editing the user . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51

40 Deleting a user . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

41 Conﬁrm the user removal . . . . . . . . . . . . . . . . . . . . . . . . 52

42 Disabling a user . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52

43 Selecting the user to be enabled again . . . . . . . . . . . . . . . . 53

44 Enabling a user . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53

45 Finding these settings . . . . . . . . . . . . . . . . . . . . . . . . . . 54

46 Message displayed when trying to log in to a disabled user . . . . 55

47 Generating access keys for a user . . . . . . . . . . . . . . . . . . . 56

48 Conﬁrmation message when generating access keys . . . . . . . . 56

49 Conﬁguring two factor authentication . . . . . . . . . . . . . . . . . 58

50 Two factor authentication provider options. . . . . . . . . . . . . . 58

51 Conﬁguring the two factor authentication with TOTP. . . . . . . . . 59

52 Veriﬁcation with TOTP during login. . . . . . . . . . . . . . . . . . . 59

53 Conﬁguring the two factor authentication with static PIN. . . . . . 60

54 Veriﬁcation using static PIN duing login. . . . . . . . . . . . . . . . . 60

55 Deactivating two factor authentication. . . . . . . . . . . . . . . . . 61

56 Conﬁguring the two factor authentication on login. . . . . . . . . . 61

57 Example of how the time zone aﬀects the Apache CloudStack in-

terface . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62

58 Editing the user’s timezone . . . . . . . . . . . . . . . . . . . . . . . 62

59 Activating local timezone functionality . . . . . . . . . . . . . . . . . 63

60 Starting to create a new project . . . . . . . . . . . . . . . . . . . . . 64

61 Creating a new project . . . . . . . . . . . . . . . . . . . . . . . . . . 64

62 Resource limits for a project . . . . . . . . . . . . . . . . . . . . . . . 65

63 Changing resource limits from a project . . . . . . . . . . . . . . . . 66

64 Starting to create a project role . . . . . . . . . . . . . . . . . . . . . 67

65 Creating a project role . . . . . . . . . . . . . . . . . . . . . . . . . . 67

66 Deﬁning the project role rules . . . . . . . . . . . . . . . . . . . . . 68

67 Starting to add members to the project . . . . . . . . . . . . . . . . 68

68 Adding an account to the project . . . . . . . . . . . . . . . . . . . . 69

69 Adding a user to the project . . . . . . . . . . . . . . . . . . . . . . . 69

70 Selecting the view . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71

71 Starting to edit a project . . . . . . . . . . . . . . . . . . . . . . . . . 71

72 Editing the project . . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

73 Starting to suspend a project . . . . . . . . . . . . . . . . . . . . . . 72

74 Suspending a project . . . . . . . . . . . . . . . . . . . . . . . . . . . 72

75 Starting the project removal . . . . . . . . . . . . . . . . . . . . . . . 73

76 Removing the project . . . . . . . . . . . . . . . . . . . . . . . . . . . 73

77 Adding a VM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74

78 Conﬁguring assignment, infrastructure, and VM template/ISO as

root admin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75

79 Conﬁguring compute oﬀering and disk oﬀering as root admin . . 76

80 Conﬁguring network, SSH key and advanced options as root admin 76

81 Conﬁguring VM details as root admin . . . . . . . . . . . . . . . . . 77

82 Conﬁguring assignment, infrastructure, and VM template/ISO as

admin . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 77

83 Conﬁguring compute oﬀering and disk oﬀering as admin . . . . . 78

84 Conﬁguring network, SSH key and advanced options as admin . . 79

85 Conﬁguring VM details as admin . . . . . . . . . . . . . . . . . . . . 79

86 Conﬁguring zone, template/ISO and compute oﬀering of the VM

as user . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 80

87 Conﬁguring disk oﬀering and network as user . . . . . . . . . . . . 81

88 Conﬁguring network, SSH key and advanced options as user . . . 81

89 Conﬁguring VM details as user . . . . . . . . . . . . . . . . . . . . . 82

90 Viewing initialization options . . . . . . . . . . . . . . . . . . . . . . 84

91 Initialization options in a VM deploy form . . . . . . . . . . . . . . . 84

92 VM’s ﬁnal settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85

93 Choosing the VM to destroy . . . . . . . . . . . . . . . . . . . . . . . 86

94 Conﬁrming VM destruction . . . . . . . . . . . . . . . . . . . . . . . 87

95 Stopping and starting a VM . . . . . . . . . . . . . . . . . . . . . . . 87

96 Choosing the VM to restart . . . . . . . . . . . . . . . . . . . . . . . 87

97 Restarting the VM . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88

98 Buttons for a VM console access and URL copying . . . . . . . . . 88

99 Return from the createConsoleEndpoint API . . . . . . . . . . . . . 89

100 Enabling dynamic scaling on a VM . . . . . . . . . . . . . . . . . . . 90

101 Conﬁguring dynamic scaling in a template . . . . . . . . . . . . . . 90

102 Choosing the VM that will have its account migrated . . . . . . . . 93

103 Migrating the VM to another account . . . . . . . . . . . . . . . . . 93

104 Creating a stored user data . . . . . . . . . . . . . . . . . . . . . . . 95

105 Stored user data creation formulary . . . . . . . . . . . . . . . . . . 95

106 Selecting a user data script to see its details . . . . . . . . . . . . . 96

107 User data script details . . . . . . . . . . . . . . . . . . . . . . . . . . 96

108 Adding a stored user data to a VM . . . . . . . . . . . . . . . . . . . 97

109 Adding a user data script during VM’s creation . . . . . . . . . . . . 98

110 Updating user data script in the VM edition tab . . . . . . . . . . . 98

111 Verifying the user data script’s functionality . . . . . . . . . . . . . 99

112 VM settings tab . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99

113 Visualizing all the volumes . . . . . . . . . . . . . . . . . . . . . . . . 103

114 Creating a new volume . . . . . . . . . . . . . . . . . . . . . . . . . . 104

115 Conﬁguring the new volume . . . . . . . . . . . . . . . . . . . . . . 104

116 Verifying the new volume . . . . . . . . . . . . . . . . . . . . . . . . 104

117 Uploading a local volume . . . . . . . . . . . . . . . . . . . . . . . . 105

118 Performing the upload . . . . . . . . . . . . . . . . . . . . . . . . . . 105

119 Uploading a volume . . . . . . . . . . . . . . . . . . . . . . . . . . . 107

120 Conﬁguring the volume upload . . . . . . . . . . . . . . . . . . . . . 107

121 Accessing the volume details . . . . . . . . . . . . . . . . . . . . . . 108

122 Adding the volume to a VM . . . . . . . . . . . . . . . . . . . . . . . 108

123 Conﬁguring the VM that will receive the new volume . . . . . . . . 109

124 Removing the volume from a VM . . . . . . . . . . . . . . . . . . . . 109

125 Conﬁrming the operation . . . . . . . . . . . . . . . . . . . . . . . . 109

126 Resizing a volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 110

127 Conﬁguring the resizing . . . . . . . . . . . . . . . . . . . . . . . . . 111

128 Downloading a volume . . . . . . . . . . . . . . . . . . . . . . . . . . 111

129 Conﬁrming the download . . . . . . . . . . . . . . . . . . . . . . . . 111

130 Automating volume snapshot creation . . . . . . . . . . . . . . . . 112

131 Conﬁguring the snapshot automation . . . . . . . . . . . . . . . . . 112

132 Removing a volume . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113

133 Conﬁrming the operation . . . . . . . . . . . . . . . . . . . . . . . . 113

134 Isolated network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114

135 Isolated network created . . . . . . . . . . . . . . . . . . . . . . . . 114

136 Conﬁguring egress rules . . . . . . . . . . . . . . . . . . . . . . . . . 115

137 Creating a shared network - Part 1 . . . . . . . . . . . . . . . . . . . 116

138 Creating a shared network - Part 2 . . . . . . . . . . . . . . . . . . . 117

139 Shared network created . . . . . . . . . . . . . . . . . . . . . . . . . 117

140 Creating a L2 network . . . . . . . . . . . . . . . . . . . . . . . . . . 118

141 L2 network created . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118

142 Browsing to the guest network . . . . . . . . . . . . . . . . . . . . . 119

143 Accessing the details for a guest network . . . . . . . . . . . . . . . 119

144 Removing a network . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

145 Conﬁrming the network removal . . . . . . . . . . . . . . . . . . . . 120

146 Editing a network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

147 Restarting a network . . . . . . . . . . . . . . . . . . . . . . . . . . . 120

148 Accessing the client-network network. . . . . . . . . . . . . . . . . . 121

149 Adding an account with Add network permission. . . . . . . . . . . 122

150 After adding the network permission. . . . . . . . . . . . . . . . . . 123

151 Accessing the public IP . . . . . . . . . . . . . . . . . . . . . . . . . . 123

152 Viewing the public IP details . . . . . . . . . . . . . . . . . . . . . . . 124

153 Setting up the ﬁrewall . . . . . . . . . . . . . . . . . . . . . . . . . . 124

154 Enabling static NAT . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

155 Conﬁguring the IP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 125

156 Verifying the ﬁnal settings . . . . . . . . . . . . . . . . . . . . . . . . 126

157 Accessing and setting up port forwarding in the ports tab . . . . . 127

158 Choosing the VM that will have a port mapped . . . . . . . . . . . 127

159 Example for port forwarding setup with several VMs . . . . . . . . 127

160 Accessing and setting up the ports for load balancing . . . . . . . 128

161 Adding the VMs that will be managed by the load balancer . . . . 128

162 Created load balancer . . . . . . . . . . . . . . . . . . . . . . . . . . 129

163 Removing the public IP . . . . . . . . . . . . . . . . . . . . . . . . . . 129

164 Accessing the network details . . . . . . . . . . . . . . . . . . . . . . 130

165 Access the network public IP . . . . . . . . . . . . . . . . . . . . . . 130

166 Acquire a new IP . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130

167 List of available IPs . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

168 New IP acquired . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131

169 Creating a new VPC . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133

170 Conﬁguring a new VPC . . . . . . . . . . . . . . . . . . . . . . . . . . 134

171 Verifying if the VPC was correctly created . . . . . . . . . . . . . . . 134

172 Restarting a VPC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

173 Options for VPC restart . . . . . . . . . . . . . . . . . . . . . . . . . 136

174 Removing a VPC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136

175 Conﬁrming the operation . . . . . . . . . . . . . . . . . . . . . . . . 137

176 Possible error while trying to remove a VPC . . . . . . . . . . . . . 137

177 Accessing the VPC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137

178 Accessing the public IPs of the VPC . . . . . . . . . . . . . . . . . . . 138

179 Choosing a public IP . . . . . . . . . . . . . . . . . . . . . . . . . . . 138

180 Accessing the created VPC . . . . . . . . . . . . . . . . . . . . . . . . 139

181 Verifying the VPC details . . . . . . . . . . . . . . . . . . . . . . . . . 139

182 Adding a new tier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139

183 Creating a new tier . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140

184 Browsing to tiers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

185 Removing a tier . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141

186 Conﬁrming the tier removal . . . . . . . . . . . . . . . . . . . . . . . 141

187 Adding a new ACL . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

188 Creating a new ACL . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142

189 Browsing to the created ACL . . . . . . . . . . . . . . . . . . . . . . 143

190 Browsing to new ACL rule creation . . . . . . . . . . . . . . . . . . . 143

191 Adding rule to allow traﬃc coming in through the port 22 from

the TCP protocol (SSH) . . . . . . . . . . . . . . . . . . . . . . . . . . 144

192 Adding rule to allow traﬃc going out through the port 22 from

the TCP protocol (SSH) . . . . . . . . . . . . . . . . . . . . . . . . . . 145

193 Selecting an account to own the tier . . . . . . . . . . . . . . . . . . 146

194 Selecting the account in the public IP acquisition form . . . . . . . 147

195 Selecting the VPC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148

196 VPC details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148

197 Public IP details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149

198 Enabling the VPN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149

199 VPN enabled . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

200 Desabling a VPN . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 150

201 Managing VPN users . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

202 Starting to add a VPN user . . . . . . . . . . . . . . . . . . . . . . . . 151

203 Adding a VPN user . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151

204 Removing a VPN user . . . . . . . . . . . . . . . . . . . . . . . . . . . 152

205 Starting to add a VPN on Linux . . . . . . . . . . . . . . . . . . . . . 153

206 Adding a VPN on Linux . . . . . . . . . . . . . . . . . . . . . . . . . . 153

207 Informing the IPSec pre-shared key - Part 1 . . . . . . . . . . . . . . 154

208 Informing the IPSec pre-shared key - Part 2 . . . . . . . . . . . . . . 154

209 Disabling the EAP protocol - Part 1 . . . . . . . . . . . . . . . . . . . 154

210 Disabling the EAP protocol - Part 2 . . . . . . . . . . . . . . . . . . . 155

211 Adding a VPN on Windows . . . . . . . . . . . . . . . . . . . . . . . 155

212 Starting to add a VPN on Windows . . . . . . . . . . . . . . . . . . . 156

213 Finishing to add a VPN on Windows . . . . . . . . . . . . . . . . . . 156

214 Changing the PPP settings . . . . . . . . . . . . . . . . . . . . . . . . 157

215 Enabling the MS-CHAP v2 protocol . . . . . . . . . . . . . . . . . . . 158

216 Disabling IPv6 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158

217 Disabling the use of default gateway in the remote network . . . . 159

218 Browsing to the VPN customer gateway tab . . . . . . . . . . . . . 159

219 Creating a VPN customer gateway . . . . . . . . . . . . . . . . . . . 160

220 Editing a VPN customer gateway . . . . . . . . . . . . . . . . . . . . 161

221 Deleting a VPN customer gateway . . . . . . . . . . . . . . . . . . . 161

222 Browsing to the VPN gateway under the VPC details . . . . . . . . 162

223 Creating a VPN gateway . . . . . . . . . . . . . . . . . . . . . . . . . 162

224 Browsing to the VPN connection under the VPC details . . . . . . . 163

225 Creating a VPN S2S connection . . . . . . . . . . . . . . . . . . . . . 163

226 Restarting a VPN S2S connection . . . . . . . . . . . . . . . . . . . . 164

227 Removing a VPN S2S connection . . . . . . . . . . . . . . . . . . . . 164

228 Generating keys. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165

229 Viewing templates and ISOs . . . . . . . . . . . . . . . . . . . . . . . 171

230 Accessing the template details . . . . . . . . . . . . . . . . . . . . . 171

231 Template details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 172

232 Accessing the ISO details . . . . . . . . . . . . . . . . . . . . . . . . . 172

233 ISO details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173

234 Listing ﬁlters for ISOs and templates. . . . . . . . . . . . . . . . . . 174

235 Registering the template . . . . . . . . . . . . . . . . . . . . . . . . . 175

236 Conﬁguring the template . . . . . . . . . . . . . . . . . . . . . . . . 175

237 Registering the ISO . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176

238 Conﬁguring the ISO . . . . . . . . . . . . . . . . . . . . . . . . . . . . 177

239 Uploading the template . . . . . . . . . . . . . . . . . . . . . . . . . 177

240 Conﬁguring the template . . . . . . . . . . . . . . . . . . . . . . . . 178

241 Uploading the ISO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178

242 Conﬁguring the ISO . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179

243 Editing the template’s details . . . . . . . . . . . . . . . . . . . . . . 179

244 Editing the template . . . . . . . . . . . . . . . . . . . . . . . . . . . 180

245 Editing the ISO’s details . . . . . . . . . . . . . . . . . . . . . . . . . 180

246 Editing the ISO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181

247 Editing the template’s permissions . . . . . . . . . . . . . . . . . . . 181

248 Editing the ISO’s permissions . . . . . . . . . . . . . . . . . . . . . . 181

249 Editing permissions . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182

250 Editing the template’s sharing . . . . . . . . . . . . . . . . . . . . . . 182

251 Editing the ISO’s sharing . . . . . . . . . . . . . . . . . . . . . . . . . 183

252 Editing the sharing options . . . . . . . . . . . . . . . . . . . . . . . 183

253 Deleting a template . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184

254 Deleting an ISO . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184

255 Browsing to the VM . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

256 Selecting a volume from the VM . . . . . . . . . . . . . . . . . . . . 185

257 Creating a template . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185

258 Conﬁguring a template . . . . . . . . . . . . . . . . . . . . . . . . . . 186

259 Resource tags from a given component . . . . . . . . . . . . . . . . 187

260 Browsing to the aﬃnity groups . . . . . . . . . . . . . . . . . . . . . 188

261 Creating an aﬃnity group . . . . . . . . . . . . . . . . . . . . . . . . 188

262 Aﬃnity group created . . . . . . . . . . . . . . . . . . . . . . . . . . 189

263 Accessing the VM that will be added to the group . . . . . . . . . . 189

264 Moving a VM to an aﬃnity group . . . . . . . . . . . . . . . . . . . . 189

265 Selecting the group . . . . . . . . . . . . . . . . . . . . . . . . . . . . 190

266 Accessing the group . . . . . . . . . . . . . . . . . . . . . . . . . . . 190

267 Viewing the VMs in the group . . . . . . . . . . . . . . . . . . . . . . 190

268 List of VMs in the group . . . . . . . . . . . . . . . . . . . . . . . . . 191

269 Removing the group . . . . . . . . . . . . . . . . . . . . . . . . . . . 191

270 Creating an isolated network . . . . . . . . . . . . . . . . . . . . . . 193

271 Accessing the Public IP addresses tab . . . . . . . . . . . . . . . . . . 193

272 Acquiring a new public IP . . . . . . . . . . . . . . . . . . . . . . . . 194

273 Adding a load balancing rule . . . . . . . . . . . . . . . . . . . . . . 194

274 Selecting the isolated network . . . . . . . . . . . . . . . . . . . . . 195

275 Selecting the load balancing rule . . . . . . . . . . . . . . . . . . . . 195

276 Setting up the scale-up policy . . . . . . . . . . . . . . . . . . . . . . 196

277 Setting up a scale-down policy . . . . . . . . . . . . . . . . . . . . . 197

278 Conﬁguring the autoscale VM group details . . . . . . . . . . . . . 198

279 Visualizing the instances from an autoscale VM group . . . . . . . 198

280 Instances from the autoscale VM group . . . . . . . . . . . . . . . . 199

281 Instances in the autoscale VM group after the scale-up . . . . . . . 199

282 Instances in the autoscale VM group after the scale-down . . . . . 200

283 Disabling the autoscale VM group . . . . . . . . . . . . . . . . . . . 200

284 Conﬁrming the operation . . . . . . . . . . . . . . . . . . . . . . . . 200

285 Editing group details and VM deploy parameters . . . . . . . . . . 201

286 Dialog screen for editing group details and VM deploy parameters 201

287 Editing the group scale-up policy . . . . . . . . . . . . . . . . . . . . 202

288 Editing the group scale-down policy . . . . . . . . . . . . . . . . . . 202

289 Enabling the autoscale VM group . . . . . . . . . . . . . . . . . . . . 203

290 Conﬁrming the operation . . . . . . . . . . . . . . . . . . . . . . . . 203

291 Removing the autoscale VM group . . . . . . . . . . . . . . . . . . . 203

292 Conﬁrming the operation . . . . . . . . . . . . . . . . . . . . . . . . 204

293 Form for adding a Kubernetes version . . . . . . . . . . . . . . . . 207

294 Deleting Kubernetes ISOs via UI . . . . . . . . . . . . . . . . . . . . 208

295 Conﬁrming the removal of a Kubernetes ISO . . . . . . . . . . . . . 208

296 Form for creating a Kubernetes cluster . . . . . . . . . . . . . . . . 210

297 Form for scaling a Kubernetes cluster . . . . . . . . . . . . . . . . . 211

298 Form for upgrading a Kubernetes cluster . . . . . . . . . . . . . . . 212

299 Accessing Kubernetes . . . . . . . . . . . . . . . . . . . . . . . . . . 213

300 Accessing the VM details . . . . . . . . . . . . . . . . . . . . . . . . . 216

301 Taking a snapshot of the volume . . . . . . . . . . . . . . . . . . . . 216

302 Choosing which volume to take a snapshot . . . . . . . . . . . . . . 216

303 Conﬁguring the snapshot . . . . . . . . . . . . . . . . . . . . . . . . 217

304 Taking a snapshot of the VM . . . . . . . . . . . . . . . . . . . . . . 217

305 Conﬁguring the snapshot . . . . . . . . . . . . . . . . . . . . . . . . 217

306 Accessing the snapshots . . . . . . . . . . . . . . . . . . . . . . . . . 218

307 Accessing the volume snapshots . . . . . . . . . . . . . . . . . . . . 218

308 Accessing the VM snapshots . . . . . . . . . . . . . . . . . . . . . . 218

309 Accessing the snapshot details . . . . . . . . . . . . . . . . . . . . . 219

310 Creating a template . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219

311 Conﬁguring the template information . . . . . . . . . . . . . . . . . 220

312 Creating a new volume . . . . . . . . . . . . . . . . . . . . . . . . . . 220

313 Conﬁguring the new volume . . . . . . . . . . . . . . . . . . . . . . 221

314 Reverting to the snapshot . . . . . . . . . . . . . . . . . . . . . . . . 221

315 Conﬁrming the operation . . . . . . . . . . . . . . . . . . . . . . . . 221

316 Removing a volume snapshot . . . . . . . . . . . . . . . . . . . . . . 222

317 Conﬁrming the removal . . . . . . . . . . . . . . . . . . . . . . . . . 222

318 Accessing the VM snapshot details . . . . . . . . . . . . . . . . . . . 222

319 Creating a volume snapshot from a VM snapshot . . . . . . . . . . 223

320 Conﬁguring which volume will be extracted . . . . . . . . . . . . . 223

321 Creating a volume snapshot . . . . . . . . . . . . . . . . . . . . . . . 223

322 Reverting the snapshot . . . . . . . . . . . . . . . . . . . . . . . . . 224

323 Conﬁrming the operation . . . . . . . . . . . . . . . . . . . . . . . . 224

324 Removing a VM snapshot . . . . . . . . . . . . . . . . . . . . . . . . 224

325 Conﬁrming the operation . . . . . . . . . . . . . . . . . . . . . . . . 225

326 Browsing through the events . . . . . . . . . . . . . . . . . . . . . . 226

327 Archiving or removing an event . . . . . . . . . . . . . . . . . . . . . 226

328 Enabling the setting . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227

329 Example of pop-ups from help icons . . . . . . . . . . . . . . . . . . 228

330 Starting to create a new disk oﬀering . . . . . . . . . . . . . . . . . 229

331 Creating a new disk oﬀering . . . . . . . . . . . . . . . . . . . . . . . 230

332 Starting to edit the disk oﬀering . . . . . . . . . . . . . . . . . . . . 232

333 Editing the disk oﬀering . . . . . . . . . . . . . . . . . . . . . . . . . 233

334 Starting to update the disk oﬀering’s access . . . . . . . . . . . . . 233

335 Updating the disk oﬀering’s access . . . . . . . . . . . . . . . . . . . 233

336 Changing the order of disk oﬀerings . . . . . . . . . . . . . . . . . . 234

337 Starting to remove the disk oﬀering . . . . . . . . . . . . . . . . . . 234

338 Removing the disk oﬀering . . . . . . . . . . . . . . . . . . . . . . . 234

339 Starting to create a new compute oﬀering . . . . . . . . . . . . . . 236

340 Creating a new compute oﬀering - 1 - continues . . . . . . . . . . . 237

341 Starting to edit the compute oﬀering . . . . . . . . . . . . . . . . . 240

342 Editing the compute oﬀering . . . . . . . . . . . . . . . . . . . . . . 240

343 Starting to update the compute oﬀering’s access . . . . . . . . . . 241

344 Updating the compute oﬀering’s access . . . . . . . . . . . . . . . . 241

345 Changing the compute oﬀering’s order . . . . . . . . . . . . . . . . 241

346 Starting the compute oﬀering removal . . . . . . . . . . . . . . . . 241

347 Removing the compute oﬀering . . . . . . . . . . . . . . . . . . . . 242

348 Starting to change the compute oﬀering . . . . . . . . . . . . . . . 242

349 Changing the compute oﬀering . . . . . . . . . . . . . . . . . . . . . 243

350 Changing the compute oﬀering . . . . . . . . . . . . . . . . . . . . . 243

351 Starting to create a network oﬀering . . . . . . . . . . . . . . . . . . 245

352 Creating a new network oﬀering - 1 - continues . . . . . . . . . . . 245

353 Creating a new network oﬀering - 2 . . . . . . . . . . . . . . . . . . 247

354 Starting to edit the network oﬀering . . . . . . . . . . . . . . . . . . 248

355 Editing the network oﬀering . . . . . . . . . . . . . . . . . . . . . . . 248

356 Starting to disable the network oﬀering . . . . . . . . . . . . . . . . 249

357 Disabling the network oﬀering . . . . . . . . . . . . . . . . . . . . . 249

358 Starting to update the network oﬀering’s access . . . . . . . . . . . 249

359 Updating the network oﬀering’s access . . . . . . . . . . . . . . . . 249

360 Changing the order of network oﬀerings . . . . . . . . . . . . . . . 250

361 Starting the network oﬀering removal . . . . . . . . . . . . . . . . . 250

362 Removing the network oﬀering . . . . . . . . . . . . . . . . . . . . . 250

363 Starting to create a new VPC oﬀering . . . . . . . . . . . . . . . . . 251

364 Creating a new VPC oﬀering . . . . . . . . . . . . . . . . . . . . . . . 252

365 Starting to edit a VPC oﬀering . . . . . . . . . . . . . . . . . . . . . . 253

366 Editing a VPC oﬀering . . . . . . . . . . . . . . . . . . . . . . . . . . 253

367 Starting to disable the VPC oﬀering . . . . . . . . . . . . . . . . . . 253

368 Disabling the VPC oﬀering . . . . . . . . . . . . . . . . . . . . . . . . 253

369 Starting to update the VPC oﬀering’s access . . . . . . . . . . . . . 254

370 Updating the VPC oﬀering’s access . . . . . . . . . . . . . . . . . . . 254

371 Changing the order of VPC oﬀerings . . . . . . . . . . . . . . . . . . 254

372 Starting the VPC oﬀering removal . . . . . . . . . . . . . . . . . . . 254

373 Removing the VPC oﬀering . . . . . . . . . . . . . . . . . . . . . . . 255

374 Using autocomplete . . . . . . . . . . . . . . . . . . . . . . . . . . . 258

375 Checking the autocomplete for a speciﬁc command . . . . . . . . 258

376 Checking the autocomplete for a speciﬁc API . . . . . . . . . . . . . 258

377 Using the -h ﬂag to get help . . . . . . . . . . . . . . . . . . . . . . . 259

378 Performing the API call . . . . . . . . . . . . . . . . . . . . . . . . . . 260

379 Deploying a VM via CloudMonkey . . . . . . . . . . . . . . . . . . . 261

380 Active tariﬀs listing . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263

381 Tariﬀs details . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263

382 Accessing details and reports from an account . . . . . . . . . . . 264

383 Credits listing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264

384 Resource usage listing . . . . . . . . . . . . . . . . . . . . . . . . . . 266

385 Detailed listing for resource usage . . . . . . . . . . . . . . . . . . 267

386 Quota balance listing . . . . . . . . . . . . . . . . . . . . . . . . . . . 269

List of Tables

1 2FA settings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58

2 Project creation for regular users settings . . . . . . . . . . . . . . 63

3 Resource limit settings for projects . . . . . . . . . . . . . . . . . . 66

4 Project invites settings . . . . . . . . . . . . . . . . . . . . . . . . . . 70

5 E-mail sending service settings . . . . . . . . . . . . . . . . . . . . . 70

6 General VM settings . . . . . . . . . . . . . . . . . . . . . . . . . . . 100

7 Settings for VMs with custom compute oﬀering . . . . . . . . . . . 100

8 Miscellaneous settings for internal usage . . . . . . . . . . . . . . . 100

9 Settings for importing VMs with NIC, disk and custom parameters

for custom compute oﬀerings . . . . . . . . . . . . . . . . . . . . . 101

10 Maximum amount of volumes supported by each hypervisor . . . 110

11 IPSec pre-shared key size settings . . . . . . . . . . . . . . . . . . . 150

12 Setting maximum VPN users per account . . . . . . . . . . . . . . . 152

13 API calls expiration time parameters . . . . . . . . . . . . . . . . . . 169

14 ACS-K8s compatibility matrix . . . . . . . . . . . . . . . . . . . . . . 205

Notes

• This document does not cover all the topics, subjects and use cases for

Apache CloudStack. Therefore, in case you do not ﬁnd what you are look-

ing for, create a review or inclusion solicitation through GitLab.

• This document is periodically updated. So, stay tuned for new updates.

1. Introduction

In the past few years, there was a huge growth in the usage of cloud comput-

ing (private, public and/or hybrid), intensifying opportunities in the digital ser-

vices market. To supply this demand, the infrastructure needed to create and

maintain this kind of environment grows constantly, directly impacting man-

agement and operation costs.

Cloud computing environments are complex and heterogeneous, possess-

ing dynamic variables and many components that must be intertwined and

managed. In order to eﬃciently manage this kind of environment it is neces-

sary to employ tools capable of orchestrating the infrastructure, helping during

implementation, maintaining and managing cloud services and systems.

Apache CloudStack and OpenStack are the main open-source alternatives

for creating private cloud environments, highlighted by their robustness, project

maturity and range of technologies and functionalities supported. Such charac-

teristics translate to hundreds of companies adopting these platforms around

the world, including: USP, UNICAMP, Dell, Deutsche Telekom, Apple, Planetel,

LWSA and many others.

SC Clouds comes in aid of this need, providing consultancy, support and

continued development to cloud infrastructure providers and companies that

have cloud technology as the main pillar for creating and providing services. In

partnership with clients, planning and execution are carried out for the cloud

computing environments that provide infrastructure as a service. Our profes-

sionals have full mastery with the Apache CloudStack and OpenStack platforms,

being responsible for multiple new features and bug ﬁxes in the projects. Cur-

rently, SC Clouds helps with management, planning, implantation, support, bug

ﬁxing and implementing new functionalities for companies in Germany, Brazil,

Italy, Mexico and Switzerland.

1.1. Private cloud platforms

Cloud orchestration platforms, such as Apache CloudStack and OpenStack,

are used to provide private, public and hybrid clouds to companies and insti-

tutions around the globe. They are capable of connecting diﬀerent computa-

tional systems (storage, network and hypervisors), creating the abstraction for

infrastructure-as-a-service for the system’s users and administrators.

Both OpenStack and CloudStack are free open-source softwares for cloud

environment orchestration. Both share the goal of managing computational

resources, working with the virtualization concept to provide resources on de-

mand, focusing on the provision of infrastructure-as-a-service resources.

Figure 1: Cloud computing

1.1.1. Basic functionalities

Both CloudStack and OpenStack oﬀer basic functionalities needed to provi-

sion cloud computing services. Some of the provided functionalities are:

• Computing services

– VM provisioning and management for their resources (networks, mem-

ory, CPU, disks, etc);

– Management of the images with which the VMs will be created;

– Support for multiple hypervisors, like KVM, XenServer and VMware.

• Network services

– Management and conﬁguration for shared and dedicated networks;

– Creation of VLAN, VxLANS (overlay networks) and other topologies;

– Deﬁnition of access and routing policies;

– Qos, Load balancers and ﬁrewalls management;

• Storage services

– Disk management;

– Block and object based storage;

– Backups;

– SDS support (Software Deﬁned Storage), e.g. Ceph/RBD;

• Resource monitoring services

– Event monitoring;

– Display available and used resources;

– Pricing and billing;

• Open REST APIs, standardized and documented, to manage all services,

their automation and integration with other tools;

• Authentication service with native LDAP integration;

• Support to SAML2 and OpenID Connect federated systems;

• Containers-as-a-service;

Within the basic functionalities for cloud orchestration systems, Apache Cloud-

Stack has limitations related to federated systems, since it only supports the

SAML2 protocol.

1.2. Apache CloudStack history

The CloudStack project started in 2008 as a startup project known as VMOps.

The company changed its name to Cloud.com and released a CloudStack stable

version in 2010, under the GNU GPLv3 license (General Public License version

3). In 2011, Cloud.com was acquired by Citrix and in 2012 the project was do-

nated to the Apache foundation. In the past few years, CloudStack has grown

and has become a reference orchestration tool, being used by many organiza-

tions, such as Globo, RNP, UNICAMP, USP, LWSA, CGEE, Apple, Dell, Nokia and

Disney.

2. Apache CloudStack functionalities

This section will brieﬂy present the main Apache CloudSack functionalities,

along with instructions on their use.

2.1. Home dashboard

This is the default dashboard in Apache CloudStack, where it is possible to

visualize the amount of resources consumed by the user.

Figure 2: Home dashboard

2.2. Managing access to the cloud

CloudStack allows the cloud administrator to precisely deﬁne and manage

which resources are available and to whom, through domains, accounts, users,

projects, and roles. To do this, it is ﬁrst important to understand these con-

cepts.

• Account: an account represents a client of the cloud service provider,

representing one or multiple users. An account may deﬁne, for example,

a department in a company or a college within a university, grouping the

users from within these sectors.

• User: The user, in turn, is a member of the account. It belongs to a single

account and has access to resources based on the account’s type, receiv-

ing the same privileges as the account that holds it. They are an alias for

the account.

• Domain: a domain may contain several accounts that, usually, have a

logical relation between them. They can be used to limit the resources

that will be provided to the users within the domain.

• Project: projects are means to organize accounts and resources. The

users may be grouped in projects to share resources such as VMs, snap-

shots, templates and IP addresses. Resources created in a project belong

solely to it, not to an account, and can only by used within it. Members of

the project can visualize and manage the virtual resources created by any

account within the project.

• Roles: deﬁned at account and/or project level, they are sets of rules that

deﬁne which permissions accounts (and consequently the users of that

account) within the environment will have, such as creating, listing and

removing resources.

The following image shows a relational diagram between accounts, domains,

projects and users.

Figure 3: Relational diagram between accounts, domains, projects and users.

2.2.1. Account

Represents a client that uses the cloud services. In CloudStack, an account

may be used by more than one user, to the point that a whole organization

may use the same account. It can be divided in three basic types for accessing

resources, with those types being deﬁned when creating the role:

• Admin: has access to all resources within the environment, including in-

frastructure elements and platform settings. These accounts can also be

called root admin.

• DomainAdmin: has access to all elements allocated within its domain,

including user management.

• ResourceAdmin: has access to all elements allocated within its domain,

except for user management.

• User: only has access to elements allocated in its account.

The set of permissions for each account type is deﬁned through roles; how-

ever, some actions, as well as the scope of the visualized resources, depend on

the account type.

An account cannot have its type changed after creation; however, its per-

missions can be modiﬁed.

The following operations can be performed on accounts:

2.2.1.1. Adding an account

Figure 4: Browsing to account adding page

Figure 5: Adding a new account

2.2.1.2. Visualizing account details

Figure 6: Browsing to the account details page

Figure 7: Checking account details

All operations detailed below may be performed on this page.

2.2.1.3. Removing an account

Figure 8: Removing an account

When attempting to delete an account, a pop-up will ask for conﬁrmation. Dur-

ing the removal process, all users and resources (VMs, snapshots, templates,

volumes, etc) associated with the account will be removed aswell.

Figure 9: Conﬁrming the account removal

2.2.1.4. Conﬁguring account accesses

There are two ways to setup an account’s accesses:

• Edit the account limits: this is the most basic way to conﬁgure the ac-

count’s accesses, where the system administrator deﬁnes resource limits

for the account.

Figure 10: Editing limits for an account

• Editing the account settings: contain options more advanced to conﬁgure

its accesses.

Figure 11: Editing the account settings

2.2.1.5. Disabling an account

This option prevents the account’s users from accessing and managing any re-

source, while also automatically shuting down all VMs created by them. It can

be re-enabled later.

Figure 12: Disabling an account

Figure 13: Conﬁrming disabling the account

2.2.1.6. Blocking an account

Similar to the option of disabling an account, however it just prevents the user

from creating and managing new resources. Already existing resources are not

aﬀected.

Figure 14: Blocking an account

Figure 15: Conﬁrming an account’s block

2.2.1.7. API access restriction by IP

Within the global settings, of accounts or domains

, it is possible to change

the parameter api.allowed. sourc e.cid r.lis t to deﬁne a list (comma separated)

containing the only IPv4 and IPv6 addesses allowed to access the CloudStack

API. The default value is 0.0.0.0/0,::/0, allowing the access from any IP.

If there is any discrepancy between settings, account settings hold priority.

The global setting api.source.cidr.checks.enabled must be set to true to use

this function.

Figure 16: Deﬁning two IPv4 ranges and one IPv6 range

ACS uses the header X-Forwar ded-For from the HTTP request to identify

the source IP from API requests. This allows requests made by the users to be

audited and the IP restrictions mentioned above to be enforced. This header

can be changed by the requester, therefore it is necessary to setup ﬁrewalls

and proxies to intermediate the communication with ACS in a way that this is

identiﬁed and the header passed to ACS contains the real source IP from the

requester.

2.2.2. Roles

Roles represent sets of permissions that accounts have within the cloud in-

frastructure. Beyond permission sets, as presented in Section 2.2.1, roles de-

ﬁne the account types, which indicate the scope for resource management. The

account types are:

• Admin: has access to all resources from the environment, including in-

frastructure elements and platform settings. These accounts are also called

If the global setting enable.account.settings.for.domain is true.

root admin.

• DomainAdmin: has access to all elements allocated within its domain,

including user management.

• ResourceAdmin: has access to all elements allocated within its domain,

except for user management.

• User: has access only to elements allocated in its account.

CloudStack provides default roles, which cannot be edited or removed. How-

ever, it is possible to create custom roles, with complete access. The procedure

to perform this action is the following:

2.2.2.1. Adding a new role

1. Browse to Roles and add a new role.

Figure 17: Starting to create a new role

2. Add the basic settings for the new role:

Figure 18: Creating the new role

The ﬁeld Based on is mandatory. It can refer to already existing types or

roles. It is used to determine which account type may use this role. Exam-

ple: a role of Admin type can only be allocated to Admin type accounts.

Regarding the two available options:

• Type: deﬁnes which account type can use this role.

• Role: uses an existing role to deﬁne which account type can use this

role and also copies the rules from the chosen role.

2.2.2.2. Visualizing role details

1. Browse to the details of the created role.

Figure 19: Accessing the newly created role

2.2.2.3. Customizing a role

1. Customize this role’s rules.

Figure 20: Customizing the role’s rules

To utilize this new role it is necessary to create a new account using it.

2.2.2.4. Removing a role

1. It is also possible to edit and remove roles, except in a situation that will

soon be covered.

Figure 21: Starting to edit a role

Figure 22: Editing the role

Figure 23: Removing a role

Figure 24: Conﬁrming the role removal

Some important notes:

• Roles already associated with an account can not be edited or removed,

making it necessary to remove the account ﬁrst.

• A role may be used by many accounts, but an account can only have one

role.

• Despite it not being possible to edit or delete roles associated with an

account, rules that compose that role can still be added, removed and

edited.

2.2.3. Domains

Aggregation of several accounts, usually used to unify accounts with diﬀer-

ent roles, but that are part of the same organization. Domains provide means

to isolate and limit access to cloud resources to the accounts that compose

it. The standard CloudStack architecture already has a default domain, called

ROOT, which is "parent" to all other domains, while also having complete access

to the cloud resources.

• The ROOT domain:

The ROOT domain is considered the default domain. It can be neither

deleted nor have its cloud resources accesses reduced. Only users from

accounts using the root admin role can have access to this domain. From

this domain all the others are conﬁgured.

Figure 25: Accessing the domains page

2.2.3.1. Adding a domain

• When adding a new domain, it will be created as a subdomain (child)

from the current domain. In the example presented there is only one

domain. However, within more complex organization, containing multi-

ple domains, it is necessary to pay attention to the hierarchy during the

creation process.

Figure 26: Creating a new domain

Figure 27: Deﬁning the new domain’s name

Figure 28: Visualizing the created domain

Figure 29: Selecting the created domain

2.2.3.2. Conﬁguring the domain

• Conﬁguring the new domain:

By default, new domains inherit ressource access privileges from the par-

ent domain. However, these accesses may be changed.

Figure 30: Starting to edit the new domain

2.2.3.3. Deﬁning the domain accesses

• Deﬁning the accesses for the new domain:

By default, the value -1 indicates that there are no limits, i.e., accounts in

this domain can use all the resources available in the cloud. The value 0

indicates that they can not use that resource. Any value greater than 0

deﬁnes the maximum amount that the accounts from that domain can

utilize.

Figure 31: Editing the domain

With the new domain created and conﬁgured, the next step is to add ac-

counts to it. Currently, it is not possible to migrate an account from a domain

to another. Therefore, for structures where domains manage the cloud, it is

necessary to create them before creating the accounts.

2.2.3.4. Restructuring the domain tree

To restructure the domain tree, the moveDomain API can be used

move domain domainid=<domain_id> parentdomainid=<destination_parent_domain_id>

It is important to highlight that for the domain to be moved, it is necessary

to follow these restrictions:

• Both the domain and the parent domain must be active.

The parameter parentdomainid is optional. If not informed, the domain will be moved to

the parent domain ROOT.

• The new parent domain can not be a subdomain from the domain itself.

• A domain with the same name as the domain being moved must not exist

in the new parent domain.

• The new parent domain must have access to all resources that the domain

and its subdomains utilize (shared networks and aﬃnity groups).

2.2.4. Users

Represents a user that belongs to an account and has a unique username

within the domain. Users within the same domain can communicate with each

other and see other users actions, but can not access users from others do-

mains. They have the privileges given to the account and act within a deﬁned

domain. Depending on the account that the user is part of, it will have diﬀerent

privileges within the domain. For example:

• Users without admin privileges can only use resources available inside the

domain, respecting the limitations from the account that it is part of and

also limitations from the role from the accout used.

• Admin users can manage the resources, in addition to also manage ac-

counts and users within the domain.

It is possible to add several users to the same account, each of them having

its name and password. It is also possible to edit or delete users previously

created. However, these two operations are only allowed to admin users.

2.2.4.1. Adding a user

The steps to add a new user are:

1. Go to Accounts.

Figure 32: Browsing to the accounts page

2. Select to which account the user will be added.

Figure 33: Select the account that the new user will be part of

3. Browse to the users page from this account.

Figure 34: Check the users from the account

4. Add a new user.

Figure 35: Adding a new user

Figure 36: Deﬁne the new user’s informations and save the changes

2.2.4.2. Editing a user

To edit a user:

1. Repeat steps 1, 2 and 3 previously explained, but now select the account

which has the user to be edited.

2. Select the user to be edited.

Figure 37: Selecting the user to edit it

3. Click edit.

Figure 38: Starting to edit

4. Perform the desired changes and save.

Figure 39: Editing the user

2.2.4.3. Deleting a user

To delete a user:

1. Repeat all the steps (2) to get to the user details page.

2. Delete the user.

Figure 40: Deleting a user

Figure 41: Conﬁrm the user removal

2.2.4.4. Disabling/Enabling a user

It is also possible to disable a user, preventing it from accessing and managing

any resource.

Figure 42: Disabling a user

To give the user’s access back it is necessary for an administrator of the

domain to enable the disabled user.

Figure 43: Selecting the user to be enabled again

Figure 44: Enabling a user

2.2.4.5. Password policies

To improve the security for the cloud environment’s users, the operator can

enable password policies (rules that the password must follow), which can be

accessed in the global settings. It is worth highlighting that these rules are only

imposed to user passwords, other passwords, such as those from VMs, are not

aﬀected.

Currently there are 7 password policy options available:

• password.policy.allowPasswordToContainUsername: indicates if the pass-

word may contain the user name.

• password.policy.minimum.digits: minimum quantity of digits in the pass-

word.

• password.policy.minimum.length: minimum password length.

• password.pol icy.minimum.lowercase .letters: minimum quantity of low-

ercase letters in the password.

• password.pol icy.minimum. special.char acters: minimum quantity of spe-

cial characters in the password.

• password.policy.minimu m.uppercase.l etters: minimum quantity of up-

percase letters in the password.

• password.policy.regex: regular expression that the password must follow.

Figure 45: Finding these settings

By default, there are no restrictions for password creation, all the settings

mentioned above are deﬁned in the broadest way possible. It is up to the oper-

ator to identify the need to implement these security measures and setup the

password policies accordingly.

2.2.4.6. Disabling the user for entering wrong password

ACS has a functionality to disable the user for perfoming too many incorrect lo-

gin attempts. By default, it is necessary for the user to enter a wrong password

5 times in a row to be disabled. This value can be changed through the global

setting incorrect.login.attempts.allowed.

To enable it again, follow the steps described in section Disabling/Enabling

a user.

Figure 46: Message displayed when trying to log in to a disabled user

2.2.4.7. Managing user API keys

API keys are access credentials given as a way to authorize the use of speciﬁc

functionalities from one or more APIs. They often act as a unique identiﬁer for

users. In CloudStack they may be used to perform operations and integrations

(often automatically), access usage data and speciﬁc CloudStack development,

test and integration tools.

ACS allows users to generate their own keys and administrators to create

and update access keys for their users. Some of the operations possible for

them are:

1. registerUserKeys: This command’s input is the user ID for which the ac-

cess keys will be generated, and as output shows the API key and the se-

cret key registered for the ID informed.

2. getUserKeys: Returns the registered keys for the user ID informed on the

input.

3. updateUser: In the context of access keys, the command updateUser al-

lows user keys to be updated. To do this, provide the user ID and both u

serapikey and usersecretkey parameters.

To generate keys through the UI, it is necessary to access the edit page of

the user to whom the keys will be generated. Follow the steps described in

Editing a user and click the button located in the upper right corner as shown

in the image 2.2.4.7.

Figure 47: Generating access keys for a user

After clicking the button, a window requesting to conﬁrm the action will

show up.

Figure 48: Conﬁrmation message when generating access keys

Select conﬁrm for the key pair to be generated and registered for this user.

2.2.4.8. API keys usage in CloudMonkey

To utilize access keys in CloudMonkey, you can conﬁgure them directly in the

settings ﬁle

. Just ﬁll the apikey and secretkey parameters adequately in the ﬁle

and save it. The next time CloudMonkey starts, it will use the conﬁgured keys.

asyncblock = true

timeout = 1800

output = json

verifycert = true

profile = mylab

[mylab]

url = http://my-local-lab/client/api

domain = /

apikey = IsBXfV_Km9sgbc3zYsf6fiia2z19qCnB80S7quN2lSy0SvD97A

secretkey = crmgT-n820MGl8EYZ37yRvnjDjfQyV8FWHBb5EmFxN5JiqZg1ZHRk6vsNw

Another way to utilize the keys when using CloudMonkey is to manually set

the parameters. For that, open CloudMonkey and execute the following com-

mands:

(mylab) > set username your-username

(mylab) > set domain your-domain

(mylab) > set apikey YOUR-API-KEY

(mylab) > set url https://your-lab/client/api

(mylab) > set secretkey YOUR-SECRET-KEY

After ﬁnishing these steps execute a sync command to verify if the changes

took the desired eﬀect.

(mylab) > sync

Discovered 623 APIs

2.2.4.9. Two factors authentication

CloudStack supports two factor authentication (2FA), the second factor utilized

can be the Google Authenticator, any authenticator with TOTP support or a static

PIN.

This extra security layer can be setup for a domain or globally by the admin-

istrator through the following settings:

This ﬁle is normally found in the folder home/.cmk/config or in

home/snap/cloudmonkey/curr e nt/.cmk/config depending on the way it was installed.

Setting Description Value

Deafult

enable.user.2fa Determines if two factor authen-

tication is enabled.

false

mandate.user.2fa Determines if two factor authen-

tication is mandatory for the

users.

false

user.2fa.default.provider Default two factor authentication

provider.

totp

Table 1: 2FA settings

When enabling two factor authentication, users may setup the 2FA accessing

their proﬁles.

Figure 49: Conﬁguring two factor authentication

In the pop-up, the user must select one of the providers and perform the

respective conﬁguration.

Figure 50: Two factor authentication provider options.

When choosing Google Authenticator or some other authenticator that sup-

ports the TOTP algorithm, it is necessary to setup the account on the respective

application by scanning the QR code or using the conﬁguration key provided by

CloudStack. After setting up, the user must provide the code generated by the

application to perfom the login.

Figure 51: Conﬁguring the two factor authentication with TOTP.

Figure 52: Veriﬁcation with TOTP during login.

If the static PIN was chosen, the user will have to provide the code generated

by CloudStack during the 2FA setup when logging in.

Figure 53: Conﬁguring the two factor authentication with static PIN.

Figure 54: Veriﬁcation using static PIN duing login.

The user can deactivate the 2FA feature at any time in their proﬁle

If the setting manda t e.user.2fa is true, the user will have to conﬁgure again the authenti-

cation during the next login.

Figure 55: Deactivating two factor authentication.

Furthermore, the administrator can also make the use two factor authen-

tication mandatory through the ma ndate.user.2fa setting. This way, users will

have to setup the authentication on their next login.

Figure 56: Conﬁguring the two factor authentication on login.

If it is necessary to make 2FA mandatory for a single user, it is possible to

use the updateUser API:

(mylab) > update user id=<user_id> mandate2fa=true

If a user loses access to the authentication application or forgets the PIN,

they will need to contact the system administrator to disable the two factor

authentication. It can be deactivated accessing the user proﬁle and selecting

the option Disable User Two Factor Authentication as previously explained.

If the administrator themself loses the two factor, they can call the API setu

pUserTwoFactorAuthentication using secretkey and apikey

(mylab) > setup usertwofactorauthentication userid=<admin_user_id> enable=false

If the administrator does not have secretkey and apikey conﬁgured, it is

necessary to contact the support to recover the account.

2.2.4.10. Timezone

When creating a new user, it is necessary to deﬁne a timezone for it (as in sec-

tion Adding a new user). This parameter is responsible for the times displayed

in various pages in the Apache CloudStack interface, making the time informa-

tion in accordance with the chosen timezone.

Figure 57: Example of how the time zone aﬀects the Apache CloudStack interface

It is possible to edit the user’s current timezone. For that, follow the steps

in Editing a user.

Figure 58: Editing the user’s timezone

When using secretkey and apikey, the 2FA is ignored.

By default, the Apache CloudStack interface will show the time accordingly

to the chosen timezone. However, it is possible to change the local timezone

without editing the user. For that, just activate the option "Use local timezone"

as shown in the image below:

Figure 59: Activating local timezone functionality

2.2.5. Projects

Projects are used to organize resources and group accounts and users. By

default, both admin and regular users may create a project, with the creator au-

tomatically becoming its administrator. It is possible to change this behaviour

through the setting allow.user.create.projects:

Setting Description Value

Deafult

allow.user.create.projects Deﬁnes if a regular user can cre-

ate a project.

true

Table 2: Project creation for regular users settings

2.2.5.1. Adding a project

To create a project:

Figure 60: Starting to create a new project

Clicking the New Project button will display a form to ﬁll in the information:

Figure 61: Creating a new project

Projects have resource allocation limits:

Figure 62: Resource limits for a project

These limits are inherited from the global settings:

Setting Description Value

Deafult

max.project.cpus Maximum number of CPU cores

that a project can use.

max.project.memory Maximum amount of memory (in

MiB) that a project can use.

40960

max.project.networks Maximum number of networks

that can be created to a project.

max.project.primary.storage Maximum amount of primary

storage (in GiB) that a project can

use.

200

max.project.public.ips Maximum number of IPs that a

project can consume.

max.project.secondary.storage Maximum amount of secondary

storage (in GiB) that a project can

use.

400

Setting Description Value

Deafult

max.project.snapshots Maximum number of snapshots

that can be created for a project.

max.project.templates Maximum number of templates

that can be created for a project.

max.project.user.vms Maximum number of user VMs

that can be deployed for the

project.

max.project.volumes Maximum number of volumes

that can be created for a project.

max.project.vpcs Maximum number of VPCs that

can be created for a project.

Table 3: Resource limit settings for projects

However, they can be edited for each project:

Figure 63: Changing resource limits from a project

Projects also have roles, which are a subset of account role permissions.

Project roles are, by nature, restrictive, meaning they are used to restrict per-

missions in the project’s context and, for this reason, can not be used to grant

new permissions. Project roles have priority over the account’s roles. If an ac-

count is added to a project without being assigned a project role, the account’s

role will take eﬀect.

As an example, if an account has the permission to create networks, and is

added to a project without being assigned a project role, the network creation

permission would be kept. However, if it has the network creation permission,

and is added to a project assigned to a role that restricts this permission, the

creation of networks would not be permitted in the project’s context.

Figure 64: Starting to create a project role

Figure 65: Creating a project role

Figure 66: Deﬁning the project role rules

2.2.5.2. Interaction between domain admins and project admins with project

roles

Besides the project roles restrictive nature, if the account’s role is of the domain

admin type, it will have access to all the projects under their domain, regardless

of its project role.

Also, if the account is the project’s admin, the user will maintain the account

role permissions, regardless of the project role restrictions.

2.2.5.3. Adding accounts and users to the project

To add accounts and users to the project, just ﬁll the form and send an invite:

Figure 67: Starting to add members to the project

Figure 68: Adding an account to the project

Figure 69: Adding a user to the project

After ﬁnishing the addition, the account or user will be part of the project,

however it is possible to change this behaviour to make it necessary for them to

conﬁrm the invite, through e-mail, before being eﬀectively added to the project.

For that, the following settings are available:

Setting Description Value

Deafult

project.invite.required Deﬁnes if the account or user

needs to accept the invite to the

project.

false

project.invite.timeout Time (in seconds) for invite expi-

ration.

86400

Table 4: Project invites settings

To make it possible to send the conﬁrmation e-mails, it is necessary to setup

the e-mail sending service for projects:

Setting Description Value

Deafult

project.email.sender Address in the From header of

the conﬁrmation e-mail.

null

project.smtp.enabledSecurity

Protocols

Security protocols for send-

ing e-mails, separated with

spaces. Supported protocols:

SSLv2Hello; SSLv3; TLSv1;

TLSv1.1; TLSv1.2; TLSv1.3.

project.smtp.host SMTP server address for send-

ing e-mails.

null

project.smtp.password Password for authentication in

the SMTP server (only if the set-

ting project.smtp.useAuth is set to

true)

null/false

project.smtp.port Port that the SMTP is listening to. 465

project.smtp.useAuth Deﬁnes if authentication will be

used to send e-mails.

null

project.smtp.username User for authentication on the

SMTP server (only if the set-

ting project.smtp.useAuth is set to

true)

null

project.smtp.useStartTLS Deﬁnes if StartTLS will be used

to protect the connection (only if

the setting project.smtp.useAuth

is set to true).

false

Table 5: E-mail sending service settings

An account or user may be in multiples projects simultaneously. For it to

be possible to consume and visualize the project resources, it’s necessary to

change the visualization:

Figure 70: Selecting the view

Asides from the functionalities mentioned, it is also possible to:

2.2.5.4. Editing a project

• Edit:

Figure 71: Starting to edit a project

Figure 72: Editing the project

2.2.5.5. Suspending a project

• Suspend:

Figure 73: Starting to suspend a project

Figure 74: Suspending a project

If a project has resources while it is suspended, its VMs will be stopped

and its VRs may be destroyed.

2.2.5.6. Deleting a project

• And delete:

Figure 75: Starting the project removal

Figure 76: Removing the project

If a project has resources when it is removed, they will be destroyed.

More information in the oﬃcial documentation.

2.2.6. Notes

There are situations where a user has a domain with multiple accounts, with

diﬀerent roles, that will need to share resources. Based on the CloudStack ar-

chitecture, there are two alternatives for sharing resources between diﬀerent

accounts:

• Through projects, in which the accounts and users that will share resources

will be added. It is also possible to create diﬀerent project roles, individu-

ally limiting the access to shared resources.

• Creating a role for accounts, based on the DomainAdmin type. This role will

operate at the same level as the default DomainAdmin, however it will be

possible to limit access to the created DomainAdmin role through access

rules.

It is important to notice that a project can only include accounts from the

same domain in which it was created. It is not possible for accounts from dif-

ferent domains to be added to the same project.

2.3. VM management

The following operations can be carried out for VMs:

2.3.1. Adding a VM

Figure 77: Adding a VM

To add a VM, it is necessary to specify some settings, which vary depending

on the account type of the user. It is also importante to ensure that the VM’s

hardware requirementes are fullﬁled, including CPU, RAM and network. To de-

ploy a VM based on an ISO, Apache CloudStack will use the data disk to boot

the OS. This way, the data disk will become the root disk afterwards. Thus, only

after the VM is created will it be possible to allocate a data disk, since the initial

volume becomes the root disk for the operating system implementation.

2.3.1.1. Adding a VM as root admin

Figure 78: Conﬁguring assignment, infrastructure, and VM template/ISO as root admin

Figure 79: Conﬁguring compute oﬀering and disk oﬀering as root admin

Figure 80: Conﬁguring network, SSH key and advanced options as root admin

Figure 81: Conﬁguring VM details as root admin

2.3.1.2. Adding a VM as admin

Figure 82: Conﬁguring assignment, infrastructure, and VM template/ISO as admin

Figure 83: Conﬁguring compute oﬀering and disk oﬀering as admin

Figure 84: Conﬁguring network, SSH key and advanced options as admin

Figure 85: Conﬁguring VM details as admin

2.3.1.3. Adding a VM as user

Figure 86: Conﬁguring zone, template/ISO and compute oﬀering of the VM as user

Figure 87: Conﬁguring disk oﬀering and network as user

Figure 88: Conﬁguring network, SSH key and advanced options as user

Figure 89: Conﬁguring VM details as user

2.3.1.4. Creating a VM with UEFI initialization

The UEFI (Uniﬁed Extensible Firmware Interface) allows the limitations imposed by

the BIOS to be surpassed. The diﬀerence between these two initialization types

is that UEFI stores all the data regarding the initialization in a .efi ﬁle instead of

storing it in ﬁrmware. Here are some of the advantages of using UEFI compared

to BIOS:

• Supports unit sizes up to 9 zetabytes (BIOS only allows up to 2,2 terabytes);

• Faster initialization time;

• Oﬀers better security with "Secure Boot", preventing the machine from

booting from unauthorized applications;

• Runs in 32 or 64 bits (with mouse support for browsing), while BIOS oper-

ates in 16 bits (with browsing limited to the keyboard)

Libvirt conventionally uses BIOS as default ﬁrmware to start the guest VMs.

For it to start VMs with UEFI, it is necessary to install the ovmf packages, avail-

able on most Linux repositories, in the host where the VMs will be deployed.

Verify whether the package is present in the host with the following command:

dpkg -s ovmf. If the package is not found, follow these steps:

• 1. For Debian based systems, execute:

sudo apt install ovmf

• 2. For RHEL based systems, it is necessary to install the edk2-ovmf pack-

age using the following command:

sudo dnf install edk2-ovmf

sudo yum install edk2-ovmf

• 3. For Arch Linux systems:

sudo pacman -S edk2-ovmf

For the second step it is necessary for the VM’s template to support UEFI

initialization.

When setting up a VM, ensure that CloudStack only allocates it to UEFI-

enabled hosts, as deﬁned in the database. Setup the virtual machine as needed,

adjusting the XML deﬁnitions when using KVM to ensure UEFI compatibility.

Figure 90: Viewing initialization options

Figure 91: Initialization options in a VM deploy form

During the VM creation process, it is possible to check the details of the ﬁnal

settings for the VM before creating it:

Figure 92: VM’s ﬁnal settings

2.3.2. Removing a VM

It is possible to perform VM removals via CloudMonkey or via UI. When per-

foming these action, pay attention to the expung e parameter. By default, it is

set to false, allowing a VM removed by an administrator to be recovered later.

If it is set to true, the VM is completely removed and can not be retrieved after-

wards.

Through the API, it is possible to use the de stroyVirtualMachine command

to place the VM in the D estroyed, state, marking it for permanent removal af-

ter the period deﬁned by the global setting ex p u n ge.delay. CloudStack makes

purge checks based on the interval deﬁned in the global setting expunge.inter

val. The API can be assisted by the expunge parameter. It is worth highlighting

that Admin type roles will always be able to inform such parameter; accounts

with other roles will only have access to it if the setting a ll ow .u se r. ex pu ng e. re

cover.vm (at account level) is set to true.

(localcloud) > destroy virtualmachine id=<VM_id> expunge=<False or True>

CloudStack also provides the expungeVirtualMachine API, which can perma-

nently remove VMs in the Destroyed, Expunging, or Error state.

(localcloud) > expunge virtualmachine id=<VM_id>

Both procedures may be done via UI, as shown next:

Figure 93: Choosing the VM to destroy

Figure 94: Conﬁrming VM destruction

2.3.3. Starting and stopping VMs

Figure 95: Stopping and starting a VM

2.3.4. Restarting a VM

To restart a VM, it must be running.

Figure 96: Choosing the VM to restart

Figure 97: Restarting the VM

2.3.5. Accessing VMs via web interface

ACS provides access to VM consoles through the web browser, using a UI

button that launches the console in a new tab. However, for security reasons,

console access is restricted to one session at a time. If a console is already in

use, a second console for the same VM cannot be opened, and the following

error is displayed: Failed to connect to server / access token has expired. A but-

ton is provided next to the console to copy its URL to the clipboard for sharing

or later access.

Figure 98: Buttons for a VM console access and URL copying

Via CloudMonkey it is possible to access the VM console through the create

ConsoleEndpoint API by providing the VM’s ID as parameter, which returns the

console access URL for the web browser.

To obtain the access URL via CloudMonkey, execute the following command:

$ create consoleendpoint virtualmachineid=<VM_id>

Figure 99: Return from the createConsoleEndpoint API

2.3.6. Increasing computational resources from a VM

Despite this functionality also being supported by the VMware and XenServer

hypervisors, this section will only use KVM when demonstrating how to prepare

the environment to enable changes to computational resources (RAM and CPU)

with the VM running. In the context of the KVM hypervisor, this functionality ex-

ists with some caveats:

• The VM must be conﬁgured as dynamically scalable;

• The compute oﬀering type of the VM must be either custom constrained

or custom unconstrained;

• The VM’s operating system must support dynamic scale for resources;

• neither the vGPU nor the CPU families of a VM can be modiﬁed during run-

tine, as most operating systems do not support this type of substitution

with the VM running

• It is not possible to reduce the VM’s resources, only increase.

It is necessary for the global setting enable.dynamic.scale.vm, which is set

to false by default, to be set to true.

To conﬁgure the desired VMs it is necessary for it to be stopped. The neces-

sary deﬁnitions for dynamic scalling on KVM are only informed on the Document

Object Model (DOM) in the VM deploy. Thus, it is not possible to changes those

deﬁnitions while the VM is running.

Already existing VMs that are stopped can be conﬁgured as dynamically scal-

able:

Figure 100: Enabling dynamic scaling on a VM

It is also possible to setup templates so that the VMs created with them are

dynamically scalable since their creation.

Figure 101: Conﬁguring dynamic scaling in a template

Lastly, to increase the computational resources of a VM, just use the follow-

ing API command via CloudMonkey:

scale virtualmachine id=<VM_id> serviceofferingid=<compute_offering_id>

details[0].memory=<new_memory_amount> details[0].cpuNumber=<new_CPU_quantity>

details[0].cpuSpeed=<new_CPU_speed>

For example:

scale virtualmachine id=13661a05-6640-48d3-938f-2ba054356d0b serviceofferingid=daa3225c-caa8-4a37-a534-cf53635aaaf9

details[0].memory=512 details[0].cpuNumber=2

details[0].cpuSpeed=1000

Some important notes about the command’s execution:

• In the API, the memory values (details[0].memory) and the number of

CPUs (details[0] .cpuNumbe r) are mandatory. If there is no intention to

change the amount of memory, just inform the current value, and the

same applies for number of CPUs;

• The memory value (details[0].memory) informed in the API must be the

current value + a multiple of 128.

• If the VM has a compute oﬀering of custom unconstrained type, the value

for CPU speed (details[0].cpuSpeed) is also mandatory;

Depending on the VM’s operating system, it may be necessary to activate the

CPUs or RAM for them to become usable. On Ubuntu, it is possible to activate

the CPU in the following way:

1. Become the root user:

sudo su

Activate the CPU with the following command, changing the CPU number

to the desired:

echo 1 > /sys/devices/system/cpu/cpu<CPU_NUMBER>/online

2.3. Repeat those steps for the other CPUs that need to be activated.

On CentOS and RHEL the following procedure can be used to activate RAM:

1. Use this command to check for deactivated RAMs:

grep line /sys/devices/system/memory/*/state

2. If there is any deactivated RAM, activate it with the command;

echo online >/sys/devices/system/memory/memory[number]/state

It is also possible to increase the computational resources of a VM through

the CloudStack UI, as explained in section 2.18.2.4.

2.3.7. Assigning a VM to another account

When using root or domain admin users it is possible to assign a VM from

one account to another. Root adm ins can assign VMs to accounts in any do-

main, while domain admins can only assign VMs to accounts within the same

domain and its subdomains.

As an example, consider the following domains structure:

ROOT

-> domain1

-> subdomain1

-> subdomain2

-> domain2

-> subdomain1

A domain admin from the domain1 can reassign VMs from/to accounts

within the domains RO OT/domain1, ROO T/domain1/subdoma in1 and ROO T

/domain1 /subdomai n2. However, that user will not be able to reassign VMs

to/from accounts within the domain ROOT/domain2 and its subdomains.

This process also has some important caveats:

• The VM must be stopped;

• The VM cannot have snapshots;

• The account to which the VM is being migrated must have access to the

networks that the VM uses. It is also important to consider that it is not

possible to migrate a network between accounts;

• If the default NIC is connected to an isolated network and the new account

has more than one isolated network, it will be necessary to specify the

network;

• The IPs from the VM cannot have Port Forwarding/Load Balancing rules or

be Static Nat.

Figure 102: Choosing the VM that will have its account migrated

Figure 103: Migrating the VM to another account

It is also possible to perfom this process using CloudMonkey, described in

section 3, through the following command:

(localcloud) > assign virtualmachine virtualmachineid=<vm-id> domainid=<domain-id>

account=<account-name>

2.3.8. Adding a user data script to the VM

The user data script is a script that executes on VM boot, and can be added

during it is creation or afterwards, by updating its deﬁnition. Its assignment

may be done in two ways, either through CloudMonkey or through the UI.

Via CloudMonkey, the script must be assigned encoded in base64 format

(because of CloudMonkey’s architecture design, a base64 script may have up

to 32768 characters, or, 32KB size), with the userdata= parameter, followed by

the script encoded.

Example of assigning a user data script via CloudMonkey:

(localcloud) > deploy/update virtualmachine [...] userdata=IyEvYmluL2Jhc2gKCmVjaG8gInVzZXJkYXR

hIHRlc3QgLSAkKGRhdGUpIiA+IC90bXAvdXNlcmRhdGE=

The script above decoded:

#!/bin/bash

echo "userdata test - $(date)" > /tmp/userdata

Notice that [...] was only used to improve the visualization of the user data

script parameter, removing all other mandatory parameters.

The updateVirtualMachine API used in example the above, besides changing

the boot settings, can also be used to change other settings, such as memory

values, number of CPUs and CPU speed. Therefore, to prevent other settings

from being changed by the ﬁnal user, it is necessary to add some values in the

global setting user.vm.denied.details.

These values are the settings names, comma separated. For example, to

block a user from changing the settings for memory and CPU, the setting must

have the following value: cpuNumber, cpuSpee d, memory. It is important to

highlight that the default value for this setting is: rootdisksize, cpu O v e r c o m m i

tRatio, memoryOvercommitRatio, Message.ReservedCapacityFreed.Flag.

Via UI, the user data script may be added manually, through the text ﬁeld

available at the Manual Userdata Entr y tab, or by selecting a pre-conﬁgured

script, available at the Stored userdata tab. When added through the text ﬁeld,

the base64 encoding is not needed, because CloudStack encodes the script

when its sent to the back-end.

If the script was introduced as a stored user data, the imposed limit is 3069

characters, which results in 4092 characters after encoding. The Following steps

are necessary to create a stored user data:

Figure 104: Creating a stored user data

The creation formulary will be displayed:

Figure 105: Stored user data creation formulary

The following ﬁelds will be available:

• Name: The name of your user data script.

• Userdata: The content of your user data script, containing variables or

commands.

• Userdata parameters: A comma-separated list of the variables created.

Only user created ones are necessary.

• Domain: Optional. Informs to which domains this user data script will be

available.

• Account: Optional. If a domain is selected, an account can be assigned to

the script.

After the script’s creation, it is possible to access its details:

Figure 106: Selecting a user data script to see its details

Figure 107: User data script details

Currently, it is impossible to edit user data script after creation, making it

necessary to delete it and create a new one. A script can not be deleted while

there are VMs that implement it, so, it is necessary to stop the VM and remove

the user data script by selecting the No thanks option, in the Reset Userdata o

n Instance VM functionality.

Despite not being able to edit a created user data script, it is possible to

edit the VM’s user data script by editing the instance itself. This will not apply

any change to other VMs that implement it nor the script itself. Also, VMs that

implement a user data script that was modiﬁed are still considered users of the

script, preventing it from being deleted.

To assign a stored user data to a VM during its creation, the following steps

must be performed:

Figure 108: Adding a stored user data to a VM

If the Manual userdata entry is preferred, the applied limit is 24567 charac-

ters (24KB size), reaching 32768 characters (32KB size) after encoding, Cloud-

Stack’s maximum supported size. The addition process involves performing

steps 1 and 2 from the previous example, and then entering the script content

in the text box, as shown in the example below:

Figure 109: Adding a user data script during VM’s creation

The user data script can be edited in the following way:

Figure 110: Updating user data script in the VM edition tab

With the user data script added, its functionality can be veriﬁed as shown in

the following screenshot:

Figure 111: Verifying the user data script’s functionality

2.3.9. VM settings

To change a VM’s settings, it is necessary to stop it ﬁrst. Then, the VM settings

may be modiﬁed by accessing the Settings tab.

Figure 112: VM settings tab

2.3.9.1. General settings

Setting Description.

keyboard Deﬁnes the keyboard layout in the VM.

cpu.corespersocket Deﬁnes the number of CPU cores per socket.

nicAdapter Deﬁnes the virtual network adapter used by the VM.

Table 6: General VM settings

2.3.9.2. Settings for VMs with custom compute oﬀering

Setting Description

cpuNumber Number of CPU cores assigned to the VM.

cpuSpeed Clock speed of the processor assigned to the VM.

memory Amount of RAM assigned to the VM.

Table 7: Settings for VMs with custom compute oﬀering

2.3.9.3. Miscellaneous settings for internal usage

Setting Description

deployvm Used during deploy.

SSH.PublicKey Public SSH key used to authenticate the VM access.

password Password used to authenticate the VM access.

Encrypted.Password Encrypted password used to authenticate the VM ac-

cess.

conﬁgDriveLocation Location where the conﬁguration unit from the VM

will be mounted.

Table 8: Miscellaneous settings for internal usage

100

2.3.9.4. Settings for importing VMs with NIC, disk and custom parameters

for custom compute oﬀerings

Setting Description

nic Virtual network adapter used by the VM.

network Virtual network used by the VM.

ip4Address IPv4 address assigned to the VM.

ip6Address IPv6 address assigned to the VM.

disk Virtual disk used by the VM.

diskOﬀering Virtual disk oﬀering used by the VM.

conﬁgurationId Id from the conﬁguration used to deploy an existing VM in

a new host machine.

Table 9: Settings for importing VMs with NIC, disk and custom parameters for custom compute

oﬀerings

2.3.9.5. Adding VM settings via API

Not all VM settings are available through the UI. However, it is possible to change

or add them through the updateVirtualMachine API, as shown below:

update virtualmachine id=<VM_id> details[0].<property1>=<value1>

details[0].<property2>=<value2> <other parameters>

When executing the updateVirtualMachine command to update the VM set-

tings, only the settings passed through the command will be assigned to the

VM. The current settings will be removed if they are not included in it.

(admin) > update virtualmachine id=565f5741-315e-47ea-816b-3f4d98b49553

details[0].keyboard=us details[0].cpu.corespersocket=1

details[0].Message.ReservedCapacityFreed.Flag=true details[0].video.hardware=vga

details[0].io.policy=threads details[0].iothreads=enabled

It is possible for settings passed to be mispelled or not exist, in these cases

the settings will not be applied to the VM and will be ignored. However, they

will remain visible on the VM settings tab.

2.3.10. Limit on network consumption for user VMs’ NICs

The limitation on guest networks consumption for user VMs is applied diﬀer-

ently for the default NIC and the others NICs. When applied to the consumption

of a default NIC, ACS will use the limit speciﬁed on the VM’s compute oﬀering. If

101

the limit has not been speciﬁed on the oﬀering, ACS will use the value deﬁned

on the vm.network.throttling.rate global conﬁguration. For the other NICs, the

consumption limit will be deﬁned by the speciﬁed value on the network oﬀering

of their respective guest networks. Again, when the oﬀering does not specify a

limit, ACS applies the value from the network.throttli ng.rate global conﬁgura-

tion.

2.4. Managing volumes

There are two types of volumes used by the VMs:

• Root Disk: The disk that contains the "kernel" or "base" of the operating

system from a virtual machine is known as root disk. In Unix operating

systems, this partition is usually designated as "/". Every virtual machine

includes a single root disk, with no possibility of having two or more.

• Data Disk: Data disk unit that expands the available storage capacity for a

virtual machine. A VM may have several data disks attached to it; however,

the maximum number is determined by the hypervisor.

The diﬀerence between these two types is their location within the VM. A

volume that is labeled as Root Disk can become a Data disk and vice versa.

Besides having diﬀerent types, volumes can also be in the following states:

• Ready: Volumes in this state already exist in the primary storage and may

be either detached and ready to be assigned to a VM or already in use by

a VM, with their current usage percentage available for monitoring.

• Allocated: Represents a volume that does not physically exist yet, however

its metadata is already stored in CloudStack.

• Up lo a d e d: After uploading the volume, it is stored in the secondary stor-

age. It will only be copied to the primary storage when assigned to a VM.

102

In this state, the volume may only be attached to a VM as a DATADISK

• Mi g ra t i n g: Transactional state that indicates that the volume is being mi-

grated between storages.

• Destroy: A volume in this state can still be recovered or be permanently

deleted; in this state the volume cannot be attached to a VM.

• Expunging: Transactional state that indicates that the volume already went

through the Destroy state, but it is still in the permanent deletion process.

Disks generated by a hypervisor, such as KVM, are not compatible with other

hypervisors due to the creation and recognition of speciﬁc disk formats by each

of them.

2.4.1. Operations in data disks and root disks

This subsection presents some of the operations available for root and data

disks. The section 2.18.1 has more details about operations with root disks.

2.4.1.1. Viewing disks and volumes

Figure 113: Visualizing all the volumes

For more information, check the section 2.4.1.6

103

2.4.1.2. Creating a volume

Figure 114: Creating a new volume

Figure 115: Conﬁguring the new volume

Figure 116: Verifying the new volume

All volumes created will have the Data Disk type.

104

2.4.1.3. Uploading a local volume

Figure 117: Uploading a local volume

Figure 118: Performing the upload

105

To upload a volume, the disk oﬀering 2.18.1 must have the custom disk type,

since, during the upload, the disk size is unknown.

2.4.1.4. Uploading a local volume using cURL

By using CloudMonkey and cURL via terminal, it is also possible to upload local

volumes. For this purpose, follow these steps:

1. In CloudMonkey, use the following command

to obtain the parameters

used for next step:

get uploadparamsforvolume format=<volume_format> name=<name> zoneid=<zone_UUID>

Command output example:

{

"getuploadparams": {

"expires": "2022-02-07T15:43:12.591Z",

"id": "41af6492-8695-4b08-b04b-170edfd4f092",

"metadata": "XsUP46nagJgEdN0ZVLHEBMFxJysHre6RtCtGh7+/4/nrK3LvslGsct44QcjIH0sheCig

N9NTn/MElyuv8uxx3V8cA1pc/3KFFZAv9Ycn3XWXBKdUQsKfYe3h/NLnzPv4klpPU35o4h8qjTi+BTTPT

hotakSWWVZybe0gcYKXZGVOHeZOLqL3PMsGC+TumrSFcvW79smfytdMAebNG2nhekVmf8Lwbqdhn0G9Cl

0Szv4NAl4ZIJValMBz+qL84Z/GDEw/V1Nq1EYVqQpoyftlA9ZkhPiitLGeQgJ7UzyHaoMG3vCu0EQgBDe

IgyoJdVRpkw23BR35WTvK2HSIijjOkm/fQJJbMzwBAdLRN1wW+4bPhzLQfc+Jf/ruVk1BDEA3RRDQLp5o

VDfqveV/fj6fpnkjUmyJJ3trDlitb5i/ZnqPGt+2VeJu/UJzNUtDorV/9KVUVf7SEHTL9B154ds1RNAiK

CLM3mfo",

"postURL": "https://172.16.200.99/upload/41af6492-8695-4b08-b04b-170edfd4f092",

"signature": "A8ElmOHx3gAC5xA43b0Kwa97qac="

}

2. Via terminal, use the curl command to upload the volume:

curl -X POST "<postURL>" -H "X-signature:<signature>" -H "X-metadata:<metadata>" \

-H "X-expires:<expires>" -H "Expect:" \

-F "file=@<path_to_the_file_for_upload>" -v --insecure

It is important to notice that the Expect header must be empty, otherwise

cURL will wait for a signal from the SSVM to perform the upload, which in turn

will cause a timeout, as the SSVM does not implement this function.

The operation for generating a template using a volume is described in the

page 184.

Only mandatory parameters are shown. There are other parameters that allow ﬁner con-

trol over how the volume will be created. For example, the parameter account allows the cre-

ated volume to assigned to an account (in this case, the domainid parameter must also be

informed).

106

2.4.1.5. Volume upload/register via URL

Figure 119: Uploading a volume

Figure 120: Conﬁguring the volume upload

107

2.4.1.6. Adding a volume to a VM

Figure 121: Accessing the volume details

Figure 122: Adding the volume to a VM

108

Figure 123: Conﬁguring the VM that will receive the new volume

2.4.1.7. Removing a volume from a VM

Figure 124: Removing the volume from a VM

Figure 125: Conﬁrming the operation

2.4.1.8. Removing and adding a ROOT volume

Through the CLI it is possible to both add or remove ROOT volumes, provided

that the VM is stopped. To remove a volume, use the following command:

109

detach volume id=<volume_id>

To add a volume as ROOT, use the following command, deﬁning its de v i c e i

d as 0:

attach volume id=<volume_id> virtualmachineid=<VM_id> deviceid=0

The X deﬁnes the volume position within the devices list. The ROOT volume

is always in the position 0. When adding a volume that is not ROOT, the param-

eter may be ommited or changed to another number. The maximum amount

of volumes supported depends on the hypervisor used:

Hypervisor Amount of volumes

KVM 32

VMWare 13

XenServer 6

Table 10: Maximum amount of volumes supported by each hypervisor

2.4.1.9. Resizing a data disk volume

This option is available for volumes created by a user, unless they were created

via upload.

Figure 126: Resizing a volume

110

Figure 127: Conﬁguring the resizing

2.4.1.10. Volume download

It is important to notice that downloading ROOT volumes is only possible if the

VM is stopped.

Figure 128: Downloading a volume

Figure 129: Conﬁrming the download

111

2.4.1.11. Recurring snapshots

Automates the process of creating volume snapshots. More information about

snapshot functionality and use can be found in Section 2.15.

Figure 130: Automating volume snapshot creation

Figure 131: Conﬁguring the snapshot automation

112

2.4.1.12. Removing a volume

Figure 132: Removing a volume

Figure 133: Conﬁrming the operation

2.5. Guest networks

Guest networks are responsible for the VMs’ internal communication, either

between each other or with their respective gateways. There are three network

types, each allowing for diﬀerent services according to the available network

oﬀerings; more information in 2.18.3).

2.5.1. Isolated networks

It encapsulates the private traﬃc between VMs in the same network, using

a virtual router (VR) to manage the network and as the public gateway. The

isolated network can be conﬁgured through its egress rules and its public IP

features. More information about public IPs can be found in 2.6.

113

Figure 134: Isolated network

Figure 135: Isolated network created

2.5.1.1. Egress rules

The isolated network’s egress rules can be conﬁgured in the respective tab,

found in the network details.

114

Figure 136: Conﬁguring egress rules

• Source CIDR: Address range from the local network to receive access per-

mission.

• Destination CIDR: Address range external to the local network that will be

accessible.

• Protocol: TCP, UDP, ICMP or all.

• Start port and End port: First and last port to be accessed.

2.5.2. Shared networks

It allows for diﬀering degrees of visibility. Usually, shared networks also

have a dedicated VR, but it doesn’t manage the public traﬃc and it doesn’t have

public IPs. As such, these networks are often used for connecting: multiple

isolated networks; VMs from diﬀerent accounts; and/or using a public gateway

that is external to the platform, just like an L2 network, but without giving up

on features like static DHCP and cloud-init.

The scope of a shared network can receive one of the following values, de-

pending on the network owner’s permissions.

• Account: visible only to a certain account’s users, in the same way as an

isolated network;

115

• Domain: visible to all accounts in the speciﬁed domain, with the option to

also allow access to its subdomains;

• Project: although the option may be present in the UI, currently it’s not

possible to create a shared network with the scope of a Project;

• All: all users in the platform have access to networks with this scope.

Figure 137: Creating a shared network - Part 1

116

Figure 138: Creating a shared network - Part 2

Figure 139: Shared network created

2.5.3. L2 network

It allows network services to be provided and managed by equipments ex-

ternal to CloudStack. As a result, there is not a virtual router and VMs deployed

in this type of network will get their IP addresses and other services through

117

these external equipments.

Figure 140: Creating a L2 network

Figure 141: L2 network created

2.5.4. Guest network operations

Some of the possible operations are:

118

2.5.4.1. Adding a guest network

Figure 142: Browsing to the guest network

2.5.4.2. Acessing details of a guest network

Figure 143: Accessing the details for a guest network

2.5.4.3. Removing networks

It is only possible to remove a network if there are no VMs using it. Otherwise,

it will be necessary to remove the VMs from the network before continuing.

119

Figure 144: Removing a network

Figure 145: Conﬁrming the network removal

2.5.4.4. Editing a network

Figure 146: Editing a network

2.5.4.5. Restarting a network

This option is only available to isolated and shared networks.

Figure 147: Restarting a network

120

2.5.4.6. Network permissions

By default, when creating an isolated or L2 network in ACS, it will be visible

only to the network owner’s account. However, it is possible to allow for other

accounts and projects in the same domain to access the network, through net-

work permissions.

This demonstration contains a domain called client, two accounts, client and

client-users, and a network called client-network.

• When accessing the Guest networks tab with the client-users account, no

network was visible.

To allow the account to access the network, it will be necessary to create a

network permission in the cl ient-networ k network and add the account client

-users.

• Access the Guest networks menu, select the network and click Add netw

ork permissions

Figure 148: Accessing the client-network network.

121

• Then, add the account in the network permission

Figure 149: Adding an account with Add network permission.

• After adding the account, the network wil be visible.

122

Figure 150: After adding the network permission.

2.6. Public IPs

The operations available for public IPs can be accessed by opening the de-

tails of the desired public IP:

Figure 151: Accessing the public IP

123

Figure 152: Viewing the public IP details

2.6.1. Operations with public IPs

2.6.1.1. Firewall setup

A public IP within a guest networks can have its ﬁrewall conﬁgured to receive

communications.

Figure 153: Setting up the ﬁrewall

• Source CIDR: Address range outside the network that will receive access

permission.

• Protocol: TCP, UDP or ICMP

124

• Start port and End port: Deﬁne the range of ports (from ﬁrst to last) open

to the source CIDR address range.

2.6.1.2. Enabling static NAT

When enabling Static NAT, it is possible to reserve a public IP for a VM.

Figure 154: Enabling static NAT

Figure 155: Conﬁguring the IP

125

Figure 156: Verifying the ﬁnal settings

With that, the VM becomes visible and accessible to the external network.

2.6.1.3. Enabling port forwarding

When enabling the port mapping it is possible to allocate speciﬁc ports for the

public IP to "redirect" the traﬃc to another port from a given VM.

126

Figure 157: Accessing and setting up port forwarding in the ports tab

Figure 158: Choosing the VM that will have a port mapped

Figure 159: Example for port forwarding setup with several VMs

The greatest beneﬁt of using port forwarding is turning multiple VMs acces-

sible to the external network through a single IP, in addition to allow controling

127

which ports from the VMs may be accessible from the external network.

2.6.1.4. Enabling load balancing

Enabling this option makes it possible to perform load balancing operations

between many VMs through a single public IP, following the rules deﬁned by

the user.

Figure 160: Accessing and setting up the ports for load balancing

Figure 161: Adding the VMs that will be managed by the load balancer

128

Figure 162: Created load balancer

Once the load balancer is created, it is possible to add more complex rules

to it.

It is important to emphasize that all the operations demonstrated here can

be reversed simply by deleting the settings added to the public IP, or by remov-

ing the public IP entirely.

Figure 163: Removing the public IP

2.6.2. Public IP acquisition

To perform operations with public IPs in a network, it is necessary for a pub-

lic IP to be allocated to that network. By default, networks managed by ACS

have at least one IP called Source NAT, that can not be removed.

Guest networks with conserve mode set to false in their network oﬀering

(section 2.18.3) and VPC tiers do not support more than one service with the

same public IP. They can only have one of the functions among Source Nat, Static

Nat, Port Forwarding and Load Balancing.

Therefore, the procedure to acquire a new public IP in a network is:

129

1. Browse to the desired network:

Figure 164: Accessing the network details

2. Browse to the public IPs tab:

Figure 165: Access the network public IP

3. Acquire a new public IP for the network:

Figure 166: Acquire a new IP

130

Figure 167: List of available IPs

Figure 168: New IP acquired

It is also important to highlight that the available public IPs depend on

three factors:

(a) Public IP range allocated during the Zone creation in ACS;

131

(b) Quantity of public IPs still available

of dedicated IPs, in addition to the global setting use . s y st e m . p ub l i c . i

ps is set to true.

2.6.3. Public IP Quarantine

On ACS, when a public IP is released (via UI or the disassoci ateIpAddress

API) by a user, it can be allocated (via UI or the associateIpAdd re s s API) by any

other user in the cloud. It is possible to restrict the public IP relocation through

the public.ip. address .quarantine.duration global setting, whose default value

is 0 (in minutes). When this global setting is enabled (value diﬀerent than 0), by

releasing a public IP, it is moved to the quarantine and only the previous owner

can relocate it. Root admins and domain admins can use the following APIs to

manage the quarantined IPs:

• removeQuarantinedIp: removes a public IP from the quarantine, using as

parameters i d (the public IP’s ID to be removed) or ip address (the public

IP itself to be removed), and removalreason (the removal reason, as a non

empty string).

• updateQuarantinedIp: updates a public IP’s quarantine end date, using

as parameters id (the public IP’s ID to have its end date updated) and end

date (the quarantine end date).

• l istQuarantinedIps: lists the public IPs in quarantine, showing the ID, the

creation and end dates, the public IP itself, the previous owner’s account

ID and account name, the early removal reason, the removal date and the

account’s ID who removed it.

They were not allocated to other networks.

132

2.7. Virtual Private Cloud (VPC)

A VPC is a set of isolated guest networks, each called a tier. All tiers of a VPC

are managed by a single virtual router that manages both the public traﬃc and

the traﬃc between tiers, according to the deﬁned ACL. In this way, it’s possible

to establish a virtual network topology, emulating an actual physical network.

Here are the operations available when using VPCs and its components:

2.7.1. VPC operations

2.7.1.1. Adding a VPC

Figure 169: Creating a new VPC

133

Figure 170: Conﬁguring a new VPC

Figure 171: Verifying if the VPC was correctly created

Here it is important to highlight some details:

• CIDR: deﬁnes the range for possible IPs in this virtual network. Tiers allo-

cated in this network must be within this range. Also, a VPC is not acces-

sible out of the zone it was created in.

• VPC Oﬀering: The available options are:

134

– Default VPC Oﬀering:

Services supported: VPN, DNS, Static Nat, Network ACL, User Data,

Source Nat, Lb,Port Forwarding, DHCP. Creates a virtual router but

does not make it redundant.

– Default VPC Oﬀering with Netscaler:

Similar to the Default VPC Oﬀering, however, since it uses Netscaler,

adds the possibility to perform external load balancing.

– Redundant VPC Oﬀering:

Similar to the Default VPC Oﬀering, however it creates a redundant

virtual router.

By default, when creating a VPC, Apache CloudStack automatically creates a

virtual router.

2.7.1.2. Restarting a VPC

It is possible to restart a VPC, either to apply more complex changes or to debug

issues. When perfoming this operation, it is important to keep in mind that the

VMs that utilize the VPC may lose connection during the restart process.

Figure 172: Restarting a VPC

135

Figure 173: Options for VPC restart

The available options are:

• Clean up: will try to clean old network elements, if any exists.

• Make redundant: will create a second virtual router on one of the infras-

tructure hosts, preferably on a diﬀerent host than the one running the

ﬁrst virtual router. This option is only available if the VPC is not redun-

dant.

2.7.1.3. Removing a VPC

When removing a VPC, it is necessary to remove the tiers that are part of it. The

virtual routers, however, will be automatically deleted.

Figure 174: Removing a VPC

136

Figure 175: Conﬁrming the operation

Figure 176: Possible error while trying to remove a VPC

A detailed description on how to remove tiers from a VPC can be found in

the page 141. Once the tiers of the VPC are properly removed, just repeat the

removal process.

2.7.1.4. Public IP acquisition for a VPC

These are the steps to acquire a public IP:

Figure 177: Accessing the VPC

137

Figure 178: Accessing the public IPs of the VPC

Figure 179: Choosing a public IP

The performed steps just allocated a public IP for the VPC, but did not link

them with any VM. Available operations for public IPs are described in section

2.6.

2.7.2. Tiers

A tier works like an isolated network, as seen in section 2.18.3), within the

VPC.

138

2.7.2.1. Adding tiers to a VPC

Figure 180: Accessing the created VPC

Figure 181: Verifying the VPC details

Figure 182: Adding a new tier

139

Figure 183: Creating a new tier

It is important to highlight that it is not possible to have two tiers with overlap-

ping CIDRs in the same VPC.

Once the tier has been added to the VPC, it can be selected during VM cre-

ation just like any other guest network.

140

2.7.2.2. Removing tiers from a VPC:

Figure 184: Browsing to tiers

Figure 185: Removing a tier

Figure 186: Conﬁrming the tier removal

Before removing a tier, it’s necessary to remove all VMs from it

This process is described in page 86.

141

2.7.3. Network ACL

An ACL (or Network ACL/ACL list) is a set of rules that allow or block traﬃc

passing through tiers. These rules may be used to grant or restrict communica-

tion between the Internet and the tiers, and also between the tiers themselves.

Basically, ACL functions as a mix between the egress rules from a guest net-

work and the ﬁrewall of a public IP, but designed for tiers. Each tier may only

have one ACL, but an ACL may be used by multiple tiers.

By default, CloudStack has two options for ACLs that ca not be changed: de

fault_allow, which allows all entrance and exit traﬃc, and default_deny, which

blocks all traﬃc. To add a custom rule set, an ACL must be created:

Figure 187: Adding a new ACL

Figure 188: Creating a new ACL

142

Figure 189: Browsing to the created ACL

Figure 190: Browsing to new ACL rule creation

To add an ACL rule, the following ﬁelds must be ﬁlled:

• #Rule:

Rule number. Rules are evaluated in ascending order, from lowest to

highest. If this ﬁeld is left empty, it will be automatically deﬁned (auto-

incremented).

• CIDR list:

Deﬁnes the range of IPs for which this rule will be applied.

• Action:

Deﬁnes if the rule is a permission or a restriction.

• Protocol:

Available options are: TCP, UDP, ICMP, All and Protocol Number. This pro-

vides great ﬂexibility when creating rules.

143

• Traﬃc Type:

Deﬁnes the type of traﬃc, either inbound or outbound.

• Description:

Rule description.

Example of a rule setup to ensure the SSH command works properly:

Figure 191: Adding rule to allow traﬃc coming in through the port 22 from the TCP protocol

(SSH)

144

Figure 192: Adding rule to allow traﬃc going out through the port 22 from the TCP protocol

(SSH)

Thus, when creating a new tier in that VPC, it is possible to use the newly

created ACL.

At the time of this writing, ACLs created through the CloudStack UI were re-

stricted to tiers in the same VPC where they were created. However, when using

the createNetworkACLList API and leaving the vpcId parameter blank, root ad-

min accounts can also create global ACLs. These global ACLs, like the default

ones (default_allow and default_deny), are available for all VPCs. Example:

create networkacllist name=<acl_name>

2.7.4. Domain VPC

Domain VPC is a feature that allows a VPC to be managed by an account

while the tiers for that same VPC can be created for distinct accounts, making

it possible to share resources.

When using Domain VPCs, several accounts can use the same VPC while

consuming a single VR; each tier will stay isolated via broadcast domain.

145

To utilize this functionality, just select the account to own the tier being cre-

ated:

Figure 193: Selecting an account to own the tier

For the tier creation to be possible, both the account creating the tier and

the account that owns the VPC must have access to the account that will own

tier. The account creating the tier must also have access to the account that

owns the VPC.

2.7.4.1. Allocating public IPs in domain VPCs

If a public IP needs to be allocated to a domain VPC, it must be acquired to the

account that owns the tier that will use it, and not to the account that owns the

domain VPC. If acquired by the account that owns the domain VPC, only that

account will be able to use it; accounts that own tiers in the VPC will not. The

account is selected in the public IP acquisition form.

146

Figure 194: Selecting the account in the public IP acquisition form

2.8. Virtual Private Network (VPN)

2.8.1. VPN types

CloudStack allows creating VPNs for users to access their VMs, oﬀering two

possible types:

• C2S (Client-to-Site) VPN: Allows the user to create safe connections with the

VPC, enabling them to connect to their VMs without the need to assign

public IPs for them. CloudStack uses the IPSec protocol to establish the

connections. It establishes a single connection between two public IPs,

therefore, only one user at a time will be able to connect to the VPC when

using the same public IP.

• S2S (Site-to-Site) VPNs: Allows the user to create safe connections between

a datacenter and the VPC, for example, connecting an oﬃce network to

the cloud.

2.8.2. Operations with VPNs

2.8.2.1. Creating a VPN

• To create a VPN, ﬁrstly it is necessary to have a VPC (section 2.7) with an of-

fering (section 2.18.4) that supports the VPN service. If one already exists,

simply add it:

147

Figure 195: Selecting the VPC

• The VPC details will include several tabs, the one concerning the User VPN

is Public IP Addresses. The remaining tabs, Private Gateway, VPN Gate-

way and VPN Connection are related to S2S VPNs.

Figure 196: VPC details

• Information about VPNs will be located in the Public IP marked as source-

NAT.

• To conﬁgure the VPN, browse to the VPN tab from the Public IP:

148

Figure 197: Public IP details

• In this tab the Enable Remote Access VPN button can be used to enable the

VPN for that VPC:

Figure 198: Enabling the VPN

To enable the VPN, the following UDP ports must be open:

1. 500 (IKE)

2. 1701 (L2TP)

3. 4500 (NAT-T)

• Then, CloudStack will enable the VPN for the VPC and generate a pre-

shared key to be used in the connections with IPSec, as shown in the fol-

lowing example:

149

– VPN gateway: 192.168.100.52

– IPSec pre-shared key: kuNbGHMBvODUxztzUXtyDz7q

Figure 199: VPN enabled

• The IPSec pre-shared key size can be changed through the global settings:

Name Description Default

value

remote.access.vpn.psk.length Size of the shared IPSec key

(min: 8, max: 256)

Table 11: IPSec pre-shared key size settings

2.8.2.2. Disabling a VPN for a VPC

After it is enabled, it is possible to disable a VPN for a VPC:

Figure 200: Desabling a VPN

150

2.8.2.3. Managing users

To manage the users, browse to the VPN users:

Figure 201: Managing VPN users

2.8.2.4. Adding a user

To add a user to a VPN, click the Add VPN user button and ﬁll the form:

Figure 202: Starting to add a VPN user

Figure 203: Adding a VPN user

151

2.8.2.5. Deﬁning maximum VPN users per account

It is possible to set the maximum quantity of VPN users for each account

through the global settings:

Name Description Default

value

remote.access.vpn.user.limit Maximum number of VPN

users per account

Table 12: Setting maximum VPN users per account

2.8.2.6. Removing a user

After created, a user may not be edited, only removed:

Figure 204: Removing a VPN user

After conﬁguring the VPN and adding a user, the connection may be es-

tablished.

2.8.2.7. Connecting to the VPN via Linux

In the network settings, a new VPN must be added:

152

Figure 205: Starting to add a VPN on Linux

The protocol used by CloudStack is the L2TP.

In the form, the VPN’s gateway, the user and the registered password must

be informed.

Figure 206: Adding a VPN on Linux

The IPSec pre-shared key must be informed:

153

Figure 207: Informing the IPSec pre-shared key - Part 1

Figure 208: Informing the IPSec pre-shared key - Part 2

Also, the EAP protocol must be disabled, since CloudStack does not support

it:

Figure 209: Disabling the EAP protocol - Part 1

154

Figure 210: Disabling the EAP protocol - Part 2

After adding the VPN, the connection can be established.

2.8.2.8. Connecting to the VPN via Windows

Under the network and internet settings, browse to the VPN tab and add a new

VPN connection:

Figure 211: Adding a VPN on Windows

155

The following form will pop up:

Figure 212: Starting to add a VPN on Windows

Important settings:

• Name or server address: enter the VPN gateway.

• VPN type: select the option L2TP/IPSec with pre-shared key.

• Pre-shared key: inform the IPSec pre-shared key.

Figure 213: Finishing to add a VPN on Windows

156

In the ﬁeld Input information type, select the option Username and pass-

word and then ﬁll the form’s ﬁelds with the user and password previously reg-

istered.

After adding the VPN, it is necessary to change some settings:

1. Go to Network connections and access the VPN adapter properties.

2. Options → Properties → PPP settings → and check all options.

Figure 214: Changing the PPP settings

3. Security → Allow these protocols → Microsoft CHAP Version 2.

157

Figure 215: Enabling the MS-CHAP v2 protocol

4. Networking → Disable IPv6.

Figure 216: Disabling IPv6

5. Right button IPv4 → Properties → Advanced → Ip Settings →

Disable "Use default gateway on remote network".

158

Figure 217: Disabling the use of default gateway in the remote network

Finally, establish the connection with the VPN.

2.8.3. S2S (Site-to-Site) VPNs

To setup a S2S VPN connection, it is necessary to create a VPN customer

gateway:

Figure 218: Browsing to the VPN customer gateway tab

The following creation form will be displayed:

159

Figure 219: Creating a VPN customer gateway

Just inform the gateway, the CIDR (separating multiple entries with commas,

if necessary) and the client’s IPSec pre-shared key. Additional security settings

can be adjusted in the remaining ﬁelds if needed. When using multiple CIDRs,

activating the Split connections option may be necessary to avoid errors.

If the IKEv1 protocol was selected, the connection must be conﬁgured through

160

strongSwan’s Unity plugin, which is external to ACS. For IKEv2, the behavior de-

pends on the peer’s subnet support. For example, AWS allows multiple CIDRs

but has issues with multiple connections, whereas Fortinet does not support

multiple CIDRs in one connection and instead requires enabling Split Connecti

ons.

After creating a VPN customer gateway, it is possible to edit it:

Figure 220: Editing a VPN customer gateway

If the VPN customer gateway is not associated with any connection, it can

also be removed:

Figure 221: Deleting a VPN customer gateway

The next step is creating a VPN gateway to the VPC that will be part of the

VPN S2S connection. In the VPC details, browse to the VPN gateway tab and

search for the Create site-to-site VPN gateway button:

161

Figure 222: Browsing to the VPN gateway under the VPC details

After clicking the button, CloudStack will provide an IP address that will be

the VPN gateway for the VPC.

Figure 223: Creating a VPN gateway

Lastly, the VPN S2S connection to the VPC must be created. Still on the VPC

details, go to the VPN connection tab and search for the Create site-to-site VPN

connection button:

162

Figure 224: Browsing to the VPN connection under the VPC details

The following form will open:

Figure 225: Creating a VPN S2S connection

Inform the VPN customer gateway and, if connecting two VPCs, check the

option Passive in the VPC that will not start the connection.

After creation it is possible to restart the connection:

163

Figure 226: Restarting a VPN S2S connection

It is also possible to remove it:

Figure 227: Removing a VPN S2S connection

2.9. Signing HTTP requests

The CloudStack API requires a signing process for HTTP requests to ensure

proper authentication and authorization of the operator. This process uses API

keys and secret keys, provided by ACS, which are included in the request body.

• Generating keys via UI:

164

Figure 228: Generating keys.

• Generating keys via CloudMonkey:

• Checking keys via CloudMonkey

165

get userkeys id=<user_id> filter=<desired_filters>

The request sent to the CloudStack API can be either GET or POST and must

include the following body:

• CloudStack main URL, for example:

http://example.com:8080/client/api

• The command that will be executed, for example: listUsers

• Parameters required by the command

• The API key that will be used to identify the user

• The signature used to authenticate the user

• Request example

http://localhost:8080/client/api? command=<command>&

<parameter(s)>&

apiKey=<api_key>&

signature=<signature>

All request will have the same format: URL + Command + Signature; to gen-

erate the signature, follow these steps:

1. Separate each of the request’s ﬁelds with a & and apply URL encode

2. Make sure that the command strings are completely in lowercase, for ex-

ample:

apikey=mivr6x7u6bn_sdahobpjnejpgest35exq-jb8cg20yi3yaxxcgpyuairmfi_ejtvwz0nukkjbpmy3y2bcikwfq&

command=deployvirtualmachine&

diskofferingid=1&serviceofferingid=1&

templateid=2&zoneid=4

3. Then, it is necessary to calculate the command string hash using the SHA-

1 hashing algorithm with the user’s secret key;

4. Lastly, encode the resulting key in Base64, the result will be something

similar to the following example:

166

http://localhost:8080/client/api?

command=deployVirtualMachine&serviceOfferingId=1&

diskOfferingId=1&

templateId=2&

zoneId=4&

apiKey=miVr6X7u6bN_sdahOBpjNejPgEsT35eXq-jB8CG20YI3yaxXcgpyuaIRmFI_EJTVwZ0nUkkJbPmY3y2bciKwFQ&

signature=Lxx1DM40AjcXU%2FcaiK8RAP0O1

To demonstrate the request signing process in practice, a step-by-step guide

is provided below:

1. Import the modules needed:

import urllib2

import urllib

import hashlib

import hmac

import base64

2. Specify the ACS endpoint, commands to be executed and the user keys:

baseurl='http://localhost:8080/client/api?'

request={}

request['command']='listUsers'

request['response']='json'

request['apikey']= '<apikey>'

secretkey= '<secretkey>'

3. Create the request string:

request_str='&'.join(['='.join([k,urllib.quote_plus(request[k])]) for k in request.keys()])

4. Start the signature encryption process with HMAC, Base64 and URL en-

conding:

sig_str=

'&'.join(['='.join([k.lower(),urllib.quote_plus(request[k]).lower().replace('+','%20')])

for k in sorted(request.iterkeys())])

sig_str

'apikey=<apikey>&

command=listusers&

response=json'

sig=hmac.new(secretkey,sig_str,hashlib.sha1)

print(sig) # <hmac.HMAC instance at 0x10d91d680>

sig=hmac.new(secretkey,sig_str,hashlib.sha1).digest()

print(sig) # "M:]x0exafxfbx8fxf2yxf1px91x1ex89x8axa1x05xc4Axdb"

sig=base64.encodestring(hmac.new(secretkey,sig_str,hashlib.sha1).digest())

print(sig) # "TTpdDq/7j/J58XCRHomKoQXEQds=n"

sig=base64.encodestring(hmac.new(secretkey,sig_str,hashlib.sha1).digest()).strip()

print(sig) # 'TTpdDq/7j/J58XCRHomKoQXEQds='

sig=urllib.quote_plus(base64.encodestring(hmac.new(secretkey,sig_str,hashlib.sha1).digest())

.strip())

167

5. Lastly, assemble the string and send the request:

req=baseurl+request_str+'&signature='+sig

print(req) # 'http://localhost:8080/client/api?

apikey=plgWJfZK4gyS3mOMTVmjUVg-X-jlWlnfaUJ9GAbBbf9EdM-kAYMmAiLqzzq1ElZLYq_u38zCm0bewzGUdP66mg&

command=listUsers&

response=json&

signature=TTpdDq%2F7j%2FJ58XCRHomKoQXEQds%3D'

res=urllib2.urlopen(req)

res.read()

'{ "listusersresponse" :

{ "count":3 ,"user" :

[

{"id":"7ed6d5da-93b2-4545-a502-23d20b48ef2a",

"username":"admin",

"firstname":"admin",

"lastname":"cloud",

"created":"2012-07-05T12:18:27-0700",

"state":"enabled",

"account":"admin",

"accounttype":1,

"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77",

"domain":"ROOT",

"apikey":

"plgWJfZK4gyS3mOMTVmjUVg-X-jlWlnfaUJ9GAbBbf9EdM-kAYMmAiLqzzq1ElZLYq_u38zCm0bewzGUdP66mg",

"secretkey":

"VDaACYb0LV9eNjTetIOElcVQkvJck_J_QljX_FcHRj87ZKiy0z0ty0ZsYBkoXkY9b7eq1EhwJaw7FF3akA3KBQ",

"accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"},

{"id":"1fea6418-5576-4989-a21e-4790787bbee3",

"username":"runseb",

"firstname":"foobar",

"lastname":"goa",

"email":"joe@smith.com",

"created":"2013-04-10T16:52:06-0700",

"state":"enabled",

"account":"admin",

"accounttype":1,

"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77",

"domain":"ROOT",

"apikey":

"Xhsb3MewjJQaXXMszRcLvQI9_NPy_UcbDj1QXikkVbDC9MDSPwWdtZ1bUY1H7JBEYTtDDLY3yuchCeW778GkBA",

"secretkey":

"gIsgmi8C5YwxMHjX5o51pSe0kqs6JnKriw0jJBLceY5bgnfzKjL4aM6ctJX-i1ddQIHJLbLJDK9MRzsKk6xZ_w",

"accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"},

{"id":"52f65396-183c-4473-883f-a37e7bb93967",

"username":"toto",

"firstname":"john",

"lastname":"smith",

"email":"john@smith.com",

"created":"2013-04-23T04:27:22-0700",

"state":"enabled",

"account":"admin",

"accounttype":1,

"domainid":"8a111e58-e155-4482-93ce-84efff3c7c77",

"domain":"ROOT",

"apikey":

"THaA6fFWS_OmvU8od201omxFC8yKNL_Hc5ZCS77LFCJsRzSx48JyZucbUul6XYbEg-ZyXMl_wuEpECzK-wKnow",

"secretkey":

"O5ywpqJorAsEBKR_5jEvrtGHfWL1Y_j1E4Z_iCr8OKCYcsPIOdVcfzjJQ8YqK0a5EzSpoRrjOFiLsG0hQrYnDA",

"accountid":"7548ac03-af1d-4c1c-9064-2f3e2c0eda0d"} ] } }'

It is important to mention that it is also possible to deﬁne an expiration time

168

for the API calls. The server will monitor the time frame speciﬁed and then

reject all requests sent after the expiration period. To enable this feature it is

necessary to add the following parameters to the request:

Parameter Value Description

signatureVersion 3 If this parameter is

not equals to 3 or

is empty, it will be

ignored in the re-

quest.

expires YYYY-MM-DDThh:mm:ssZ(ISO 8601) Speciﬁes the date

for the signature

on the request to

expire.

Table 13: API calls expiration time parameters

Usage example:

• expires=2011-10-10T12:00:00+0530

• Example for a request with expiration time:

http://<IPAddress>:8080/client/api?

command=listZones&

signatureVersion=3&

expires=2011-10-10T12:00:00+0530&

apiKey=miVr6X7u6bN_sdahOBpjNejPgEsT35eXq-jB8CG20YI3yaxXcgpyuaIRmFI_EJTVwZ0nUkkJbPmY3y2bciKwFQ&

signature= Lxx1DM40AjcXU%2FcaiK8RAP0O1hU%3D

169

2.10. Templates and ISOs

Templates are reusable images that serve as a model or standard to cre-

ate VMs. Intrinsically, a template is an image from a virtual disk that contains

its basic conﬁgurations, such as operating system, users, default passwords,

and even advanced conﬁgurations, such as softwares previously installed and

conﬁgured. It is worth highlighting that templates depend on the hypervisor

used.

During the template creation, the user must deﬁne if it will be private or pub-

lic. When set to public, the template will be available to all users that can access

the zone where the template was stored. The new template will become visi-

ble in the Templates menu when the template creation process is ﬁnished, i.e.,

when ACS ﬁnishes its download and preparation. The template then becomes

available during VM creation.

ISOs are images from optical disks, like a CD, DVD or Blu-ray. It is a ﬁle with

an exact copy, or image, of the whole optical disk content. These images are

frequently used to distribute huge programs or operating systems, allowing the

user to burn the image on a physical disk or mount it as a virtual disk in their

system to access its content without the need of a physical disk.

Just like templates, ISOs may be set as public or private. An important detail

about ISOs is that they do not depend on speciﬁc hypervisors, so, the same

image may be mounted in diﬀerent hypervisors, unlike templates. VMs created

with this resource must be conﬁgured since their installation.

Similarly to templates, ISOs can be stored and provided with certain privacy

settings. During its registration, the user will label the image as bootable or non

bootable. Bootable images are the ones that contain an operating system.

2.10.1. Operations with templates/ISOs

Apache CloudStack oﬀers many operation options for templates and ISOs,

such as:

170

2.10.1.1. Viewing

Figure 229: Viewing templates and ISOs

• Template:

Figure 230: Accessing the template details

171

Figure 231: Template details

• ISO:

Figure 232: Accessing the ISO details

172

Figure 233: ISO details

2.10.1.2. Listing

By default, the graphical interface sends the all ﬁlter (which lists all registered

elements in the environment) when the request is sent by either a root admin

or domain admin account. For user type accounts the request is sent with the

self ﬁlter (which only lists ISOs and templates that were registered by the user).

These ﬁlters may be changed by clicking the text ﬁeld next to the refresh button.

The following ﬁlters are available to use:

• featured: Returns elements marked as featured;

• self : Returns elements registered by the user that sent the request;

• selfexecutable: Similar to the previous ﬁlter but includes only images that

may be used to deploy new VMs;

173

• sharedexecutable: Returns elements created by other users that have

been shared with the user making the request;

• executable: Returns the elements that the requesting user registered as

well as elements labeled as public and that can be used to deploy new

VMs;

• community: Returns the elements labeled as public that were not marked

as featured;

• all: Returns all elements from the environment (can only be used by ad-

ministrators)

Figure 234: Listing ﬁlters for ISOs and templates.

2.10.1.3. Registry

• Template:

174

Figure 235: Registering the template

Figure 236: Conﬁguring the template

175

– Direct download: if enabled, the template will be downloaded di-

rectly to primary storage when the ﬁrst VM is created. It remains

cached until all VMs using it are destroyed; after that, it is removed.

A new download will only occur if no VMs using the template remain.

Templates conﬁgured with this option will have the Bypas s e d Secon

dary Storage state, indicating that they were directly downloaded to

the primary storage and will never be stored in the secondary stor-

age.

• ISO:

Figure 237: Registering the ISO

176

Figure 238: Conﬁguring the ISO

It is necessary to indicate the type of operating system from the template/ISO,

as this information will be used by the hypervisor. If the chosen operating sys-

tem does not match the template/ISO, some inconsistencies may appear during

the VMs creation.

2.10.1.4. Upload

• Template:

Figure 239: Uploading the template

177

Figure 240: Conﬁguring the template

• ISO:

Figure 241: Uploading the ISO

178

Figure 242: Conﬁguring the ISO

2.10.1.5. Editing

Browse to the template/ISO details.

• Template:

Figure 243: Editing the template’s details

179

Figure 244: Editing the template

• ISO:

Figure 245: Editing the ISO’s details

180

Figure 246: Editing the ISO

2.10.1.6. Editing template/ISO permissions

Changes to the permissions of templates/ISOs may be done either via UI or

API. Via API, the updateTemplatePerm i s s i o n s and updateIso P e r m issions com-

mands enable adding, removing, or fully redeﬁning template/ISO permissions.

• Template:

Figure 247: Editing the template’s permissions

• ISO:

Figure 248: Editing the ISO’s permissions

181

Figure 249: Editing permissions

The options for editing permissions are the same for both templates and

ISOs. Operations available:

• Add:

Used to add the template/ISO to a given account or project.

• Remove:

Used to remove the template/ISO from a given account or project.

• Reset:

Reset all the template/ISO permissions to default settings.

2.10.1.7. Editing the template sharing

Browse to the template/ISO details.

• Template:

Figure 250: Editing the template’s sharing

182

• ISO:

Figure 251: Editing the ISO’s sharing

Figure 252: Editing the sharing options

The global setting allow.u ser.view.all.domain.ac counts, which is set to fals

e by default, prevents regular users from viewing all accounts in their domain.

As a result, to share a template or ISO, they must know the exact name of the

target account, which is entered in a dedicated ﬁeld.

The options for editing sharing details are the same for both templates and

ISOs.

2.10.1.8. Removing a template/ISO

Templates/ISOs can be removed, but there are some caveats. The default be-

haviour for CloudStack is to refuse removing a template/ISO when there are

VMs using it.

183

After removing templates/ISOs, VM instances that use that image in the

same zone will not be aﬀected and will keep executing. However, the removed

template/ISO will not be available for new instances.

• Template:

Figure 253: Deleting a template

• ISO:

Figure 254: Deleting an ISO

2.10.1.9. Creating a template based on an existing VM

To create a template based on an existing VM, it needs to be in the Stopped

state:

184

Figure 255: Browsing to the VM

Figure 256: Selecting a volume from the VM

Figure 257: Creating a template

185

Figure 258: Conﬁguring a template

2.11. Resource Tags

Tags, or resource tags, are key:value pairs that store metadata about re-

sources, to categorize them.

This feature can be very useful for ﬁltering lists and for custom CloudStack

Usage reports (such as Quota activation rules, section 4).

To add or remove tags:

• Via UI:

186

Figure 259: Resource tags from a given component

• Via CloudMonkey: For this, the listTags, createTags and deleteTags APIs

can be used. In the deleteTags API, the parameter tags[0] can be ommited

to remove all tags from the resource.

A host, storage pool or oﬀering may also have resource tags (which should

not be confused with host tags or storage tags).

2.12. Aﬃnity Groups

Aﬃnity groups are a functionality that allows users to deﬁne whether specifc

VMs should run on the same host or be separated. This can optimize perfor-

mance by lowering latency or improve fault tolerance so that if one host fails,

VMs on the remaining hosts are still available.

Aﬃnity groups should not be mistaken for Instance groups, which are only

meant to help users organize their VMs and have no eﬀect on host placement.

2.12.1. Aﬃnity groups types

Before version 4.18, only the host aﬃnity and host anti-aﬃnity types were

available, corresponding to host aﬃnity (strict) and host anti-aﬃnity (strict). Since

version 4.18, CloudStack supports four aﬃnity group types:

• Host aﬃnity (strict): speciﬁes that the VMs must always run on the same

host. However, if the host does not have enough capacity to run all the

VMs from the group, an error occurs during the creation of new instances.

• Host anti-aﬃnity (strict): speciﬁes that the VMs must always run on dif-

ferent hosts. Similarly, if there are not enough hosts for all VMs, an error

will occur when creating new instances.

187

• Host aﬃnity (non-strict): speciﬁes that the VMs should run on the same

host whenever possible. If the host does not have enough capacity, new

instances will be created in other hosts.

• Host anti-aﬃnity (non-strict): speciﬁes that the VMs should run on diﬀer-

ent hosts whenever possible. If there are not enough hosts, new instances

will be created in a host that already has instances running.

2.12.2. Aﬃnity groups operations

2.12.2.1. Adding an aﬃnity group

Figure 260: Browsing to the aﬃnity groups

Figure 261: Creating an aﬃnity group

188

Figure 262: Aﬃnity group created

2.12.2.2. Adding a VM to an aﬃnity group

The VM must be stopped for the option to add it to an aﬃnity group to be

available.

Figure 263: Accessing the VM that will be added to the group

Figure 264: Moving a VM to an aﬃnity group

189

Figure 265: Selecting the group

2.12.2.3. Viewing VMs in the aﬃnity group

Figure 266: Accessing the group

Figure 267: Viewing the VMs in the group

190

Figure 268: List of VMs in the group

2.12.2.4. Removing an aﬃnity group

Figure 269: Removing the group

2.13. Autoscale VM groups

The autoscale VM groups feature allows the creation of VM groups that will

automatically scale instances up or down, depending on resource demand.

This functionality can be used, for example, to create more machines that pro-

vide a service when existing ones are under heavy load, or to remove them

when demand decreases, both actions are executed automatically, without the

need of intervention from an operator. This ensures more eﬃcient use of com-

putational resources and, consequently, reduces costs.

191

This automatic instance management works by deﬁning two sets of rules:

one to determine when new VMs should be created, and another to determine

when they should be removed. For example, it is possible to deﬁne that new

machines will be added when the average CPU usage of all instances is greater

than 70% and that they will be removed when it is lower than 40%.

To perform this management, CloudStack monitors usage statistics for the

instances provided by the hypervisor

. Furthermore, it uses a load balancer to

distribute the usage between the VMs. Therefore, these groups must associ-

ated to an isolated network, or to a VPC with the Load balancer service enabled

and conﬁgured.

2.13.1. Creating an autoscale VM group

Before creating an autoscale VM group it is necessary to create an isolated

network or a VPC and setup the load balancing service. The setup process of

an isolated network is exempliﬁed below:

1. Create an isolated network:

The autoscale VM groups functionality is supported by the KVM, VMware and XenServer

hypervisors.

192

Figure 270: Creating an isolated network

2. Access the Public IP addresses tab and acquire a new public IP for the

network:

Figure 271: Accessing the Public IP addresses tab

193

Figure 272: Acquiring a new public IP

3. In the public IP, access the Load balancing tab and add a load balancing

rule

Figure 273: Adding a load balancing rule

With the isolated network setup complete, it is possible to proceed to the

autoscale VM group creation menu:

1. Conﬁgure the group’s VMs through zone, template, compute oﬀering and

data disk settings.

The Autoscale parameter indicates that the rule will be used by an autoscale VM group.

194

2. Select the isolated network created:

Figure 274: Selecting the isolated network

3. Select the load balancing rule created:

Figure 275: Selecting the load balancing rule

4. Setup the scale-up policy:

The scale-up policy is a set of rules that determine when new machines

must be added to the group. It has the following settings:

• Condition: rules that deﬁne when new VMs must be added to the

group. When more than one condition is conﬁgured, all must be true

for the scale-up to happen;

• Duration: speciﬁes, in seconds, for how long the conditions must be

met for the scale-up to occur;

• Quiet Time: period, in seconds, after VMs are added that the scale-up

policy will not provision additional instances.

195

Each condition has the following parameters:

• Counter: statistic from the group’s VMs that must be monitored. The

counters available are:

– VM CPU - average percentage: average CPU usage per VM, as a

percentage (0–100);

– VM Memory - average percentage: average memory usage per VM,

as a percentage (0–100);

– Public Network - Mbps received per vm: average amount of data

received per VM, in Mbps;

– Public Network - Mbps transmited per vm: average amount of data

sent per VM, in Mbps;

– Load Balancer - average connections per vm: average number of

active connections per VM.

• Operator: operator used to compare the values;

• Threshold: value to compare with the counter.

Figure 276: Setting up the scale-up policy

In the image, for example, a scale-up policy was created to add new VMs

when the current CPU usage averaged over 10% for 30 seconds.

196

2.13.1.1. Setting up a scale-down policy

The scale-down policy follows the same principle as the scale-up policy,

but determines when VMs must be removed.

Figure 277: Setting up a scale-down policy

In the image, a scale-down policy was deﬁned to remove VMs when the

current CPU usage averaged under 10% for 60 seconds.

2.13.1.2. Conﬁguring autoscale VM group details

• The settings are the following:

– Name: group name;

– Expunge VM grace period: amount of time, in seconds, waited af-

ter a scale-down before removing more VMs. This wait time helps

guarantee that all pending sessions and transactions were ﬁn-

ished.

– Max members: maximum number of VMs in the group;

– Min members: minimum number of VMs in the group;

– Polling interval: interval, in seconds, that the conditions are veri-

ﬁed.

197

Figure 278: Conﬁguring the autoscale VM group details

2.13.1.3. Group creation

After creating an autoscale VMs group, the instances in it may be visualized

in the following way:

Figure 279: Visualizing the instances from an autoscale VM group

198

Figure 280: Instances from the autoscale VM group

2.13.2. Testing the autoscaling

With the instances created, it is possible to test the autoscaling functionality.

To force an increase in CPU usage and, consequently, a scale-up, the following

command may be executed inside the instances:

for i in 1 2 3 4; do while : ; do : ; done &done

This command will create four processes with an inﬁnite loop. Each loop

generates a 100% usage in one of the CPU cores.

After a few minutes, the scale-up will be perfomed and the group will reach

the maximum amount of instances.

Figure 281: Instances in the autoscale VM group after the scale-up

To perform the scale-down, the processes shall be ﬁnished:

sudo killall -9 bash

Then, after a few minutes, the number of instances will be reduced to the

minimum amount.

199

Figure 282: Instances in the autoscale VM group after the scale-down

2.13.3. Editing an autoscale VM group

To edit an autoscale VM group it is necessary to disable it ﬁrst:

Figure 283: Disabling the autoscale VM group

Figure 284: Conﬁrming the operation

200

While the group is disabled, VMs will keep running normally. However, nei-

ther scale-up nor scale-down will be performed based on the demand.

After it is disabled, the group’s details, VM deploy parameters and scale-up

and scale-down policies can be edited:

Figure 285: Editing group details and VM deploy parameters

Figure 286: Dialog screen for editing group details and VM deploy parameters

201

Figure 287: Editing the group scale-up policy

Figure 288: Editing the group scale-down policy

After the changes are done, the group can be enabled again:

202

Figure 289: Enabling the autoscale VM group

Figure 290: Conﬁrming the operation

2.13.4. Removing an autoscale VM group

It is also possible to remove an autoscale VM group:

Figure 291: Removing the autoscale VM group

203

Figure 292: Conﬁrming the operation

2.14. Kubernetes

The Kubernetes Service plugin adds Kubernetes integration to CloudStack.

This plugin is disabled by default and only admin users can enable it through

the global setting cloud.kubernetes.service.enabled. It allows users to execute

services in containers using Kubernetes clusters.

Since ACS version 4.16, the Kubernetes Service plugin uses the default sys-

tem VM template for the Kubernetes cluster implantation. Moreover, an ISO

is used to install the Kubernetes binaries in the cluster nodes, consequently,

a separate ISO must be provided to CloudStack for every desired Kubernetes

version.

To deploy and conﬁgure the Kubernetes cluster nodes, the plugin utilizes the

kubeadm tool from Kubernetes. kubeadm is a command line tool used to easily

provide a Kubernetes cluster for physical servers, clouds or virtual machines.

Internally, the control nodes start a Kubernetes cluster using the command k

ubeadm init, with a custom token, and the worker nodes join this Kubernetes

cluster using the command kubeadm join with the same token. More informa-

tion about the kubeadm tool can be found in the oﬃcial documentation. The

Weave Net CNI plugin is used to create a virtual network for the clusters and

allow the communication between nodes. More information about the Weave

Net CNI plugin can be found in the oﬃcial documentation.

The plugin allows creating Kubernetes clusters using either the UI or the

API. Both of them provide functionalities to list, remove, update, stop and start

204

these clusters, as well as increase or decrease the numbers of nodes in them.

2.14.1. Kubernetes versions supported

ACS allows users to keep multiple Kubernetes versions. To upload a Kuber-

netes version it is necessary to create an ISO containing these ﬁles:

1. Kubernetes binary ﬁles

2. CNI binary ﬁles

3. CRICTL binary ﬁles

4. Network settings ﬁles from Weave

5. Dashboard settings ﬁles from Kubernetes

2.14.2. Adding a Kubernetes ISO to Apache CloudStack

The community provides some out-of-the-box ISOs. The tests and exam-

ples executed in this document were performed using the ISO containing the

version 1.23.3. However, given the fast release cycle of the Kubernetes ISOs, it

is recommended to use the latest version. It is important to notice that opera-

tions over Kubernetes ISOs are only available to root admin users.

The matrix below outlines the ISOs’ compatibility with ACS:

K8s

ACS

4.18.0

< 1.23.3

1.24.0

1.25.0

1.26.*

1.27.*

1.28.4

Table 14: ACS-K8s compatibility matrix

Since version 4.16, Kubernetes started using the SystemVM (Debian) tem-

plate for cluster deployment, and the SSH user changed to cloud.

205

When creating a cluster, although ACS allows names containing uppercase

letters for its resources, Kubernetes only permits lowercase names. Therefore,

when creating Kubernetes clusters, it is necessary to name the cluster only us-

ing lowercase characters. Otherwise, when trying to perform operations that

demand communication with CloudStack, Kubernetes will not ﬁnd the cluster

nodes.

Through the addKubernetesSupportedVer sion API it is possible to add an

Kubernetes ISO to ACS.

add kubernetessupportedversion name=example mincpunumber=2 minmemory=2048 semanticversion=1.23.3 \

url=<ISO_download_URL> zoneid=<zone_id>

It is also possible to perfom the same operation via UI, accessing the Kuber-

netes ISOs page, in the Images submenu; clicking the Add Kuberne t es version

button will display the following form:

206

Figure 293: Form for adding a Kubernetes version

2.14.3. Removing a Kubernetes ISO

Through the deleteKubernetesSupportedVersion API it is possible to re-

move a Kubernetes ISO from ACS.

delete kubernetessupportedversion id=<ISO_id> filter=<optional>

The same operation can also be executed via UI, accessing the Kubernetes

ISOs page, in the Images submenu; clicking the ISO to delete; and then clicking

the delete button.

207

Figure 294: Deleting Kubernetes ISOs via UI

Figure 295: Conﬁrming the removal of a Kubernetes ISO

2.14.4. Kubernetes clusters

A Kubernetes cluster consists of two main components: the control node

and the worker nodes. The control node is responsible for organizing and man-

aging the cluster, with the option to replicate it for improved system reliability.

Worker nodes are the nodes where the applications run. The nodes may be

VMs or physical machines, which contain pods, the smallest unit within a Ku-

bernetes cluster. They work as creation templates for a container type.

The Kubernetes Service plugin provides functionalities to create and man-

age Kubernetes clusters, simplifying container deployments without requiring

manual setup of Kubernetes on each node. It automatically allocates the re-

208

quired number of virtual machines according to the deﬁned cluster size, with

binaries corresponding to the chosen Kubernetes version. Furthermore, the

service provides features to upgrade clusters and to increase or decrease the

number of nodes, allowing updates to newer minor or patch versions of Kuber-

netes while scaling as needed.

2.14.4.1. Creating Kubernetes clusters

The plugin provides functionalities to create Kubernetes clusters for shared or

isolated networks in CloudStack.

The createKubernetesCluster API allows the creation of Kubernetes clusters

and its usage is illustrated below.

create KubernetesCluster name=example description=example zoneid=<zone_id> \

kubernetesversionid=<Kubernetes_ISO_id> serviceofferingid=<compute_offering_id> \

size=<number_of_work_nodes>

The same operation is available via UI, accessing the Kubernetes page, in the

Compute submenu; clicking the Creat e Kube rnetes cluster button will display

the following form:

209

Figure 296: Form for creating a Kubernetes cluster

There are certain details that require attention when creating Kubernetes

clusters:

• Currently there can only be one Kubernetes cluster per network.

• The Kubernetes ISOs, when created, have minimum resource require-

ment, such as CPU cores and allocated memory. For this reason, a com-

pute oﬀering that fullﬁlls all the requirementes must be present in the

environment.

• When creating a Kubernetes cluster, at least two VMs will be instantiated

with the selected compute oﬀering: one for the control node and one as

a worker node. This number increases based on the cluster size deﬁned

210

during the cluster’s creation. Therefore, it is worth highlighting that the

resource usage will be greater than the minimum requirements.

• To access the nodes via SSH it is necessary to provide an SSH keys pair dur-

ing the Kubernetes cluster creation. Furthermore, the user for connecting

via SSH is cloud

2.14.4.2. Kubernetes cluster scaling

The scaleKubernetesCluster API allows scaling a Kubernetes cluster, increasing

or decreasing the number of worker nodes in it, or changing the compute oﬀer-

ing used for the cluster. However, the compute oﬀering can only be increased.

An API usage example is shown below.

scale KubernetesCluster id=<Kubernetes_cluster_id> size=<new_cluster_size> \

serviceofferingid=<compute_offering_id>

The same API can be used via UI, as shown below.

Figure 297: Form for scaling a Kubernetes cluster

Until ACS version 4.16, the user for connecting was core.

211

2.14.4.3. Upgrading a Kubernetes cluster

The upg radeKubernete sCluster API allows upgrading the minor or patch ver-

sion of a Kubernetes cluster in execution. For that, it is necessary for an ISO

with the desired version to be already available in ACS.

upgrade kubernetescluster id=<Kubernetes_cluster_id> kubernetesversionid=<Kubernetes_version_id>

The same API can be used via UI, as shown below.

Figure 298: Form for upgrading a Kubernetes cluster

It is important to notice that the Kubernetes version upgrade can only be

performed between patch versions from the current minor version or, between

a minor version and the next minor; that is, versions jumping is not allowed.

This restriction also applies to ACS.

2.14.5. Kubernetes access

While the Kubernetes cluster is created, an user interface is automatically

generated for the cluster. The Kubernetes documentation, available in this link,

shows more information about the Kubernetes dashboard. The UI provides an

Access tab with information on how to access Kubernetes, including a step-by-

step guide.

212

Figure 299: Accessing Kubernetes

To provide dashboard access to the cluster, the user must ﬁrst run a local

proxy using kubectl and the kubeconﬁg ﬁle. For security reasons, logins based

on kubeconﬁg are not allowed. To connect, an access token must be generated

and used with kubectl. The proxy can be started with the command below,

specifying the correct kubeconﬁg ﬁle path:

kubectl --kubeconfig /custom/path/kube.config proxy

As soon as the proxy is running, the dashboard is accessible by clicking the

available link. The token used for login can be obtained using the following

command:

kubectl --kubeconfig /custom/path/kube.config describe secret $(kubectl --kubeconfig

213

/custom/path/kube.config get secrets -n kubernetes-dashboard |

grep kubernetes-dashboard-token | awk '{print $1}') -n kubernetes-dashboard

2.14.6. Generating a token for the dashboard

A default user token can not be used to access the Kubernetes dashboard.

To access it, a token needs to be created for the kubernetes-dashboard user in

the kubernet e s-dashboard namespace and be informed when logging in. The

token can be created using the following command:

kubectl --kubeconfig <kube.conf-file-path> create token kubernetes-dashboard -n kubernetes-dashboard

2.15. Snapshots

Snapshots are recovery points for a whole VM or just a single disk. They

can be used as a way to recover from failures or even to create templates and

new volumes. In some hypervisors, such as VMware and KVM, they behave like

backups.

2.15.1. Snapshot types

There are two types of snapshots:

2.15.1.1. Volume/Disk snapshot

Takes a snapshot of the VM’s disks

. The cloud administrator can deﬁne a

limit for how many snapshots a user may have. Users may create volumes and

templates based on a snapshot.

It is possible to create snapshots manually or automatically, through set-

tings. They are created in the Primary Storage and, if the global setting sna ps

hot.backup.to.s e condary is set to true

, they are backed up to the Secondary

Storage, where they will be kept until their removal.

The section 2.4 provides more detailed information about VM disks.

We recommend always having this setting set to true

214

2.15.1.2. VM snapshot

Takes a snapshot similar to the volume snapshot, however this type of snap-

shot has options to include the memory/CPU states of the VM. This procedure

can facilitate the fast recovery of a VM

. This type of snapshot supports incre-

mental snapshots. Starting from the ﬁrst snapshot of a VM, the subsequent

ones will save their diﬀerences in a delta ﬁle

There are some limitations regarding snapshots:

• Changes to volumes, CPU or memory of a VM that has snapshots may

cause errors, either when performing these changes or when trying to

restore the VM to the snapshot.

• It is not possible to save a copy of two volumes of a VM on the same snap-

shot.

• It is not possible to perform a snapshot of a VM and a volume at the same

time.

• Only snapshots created through CloudStack are managed by it.

2.15.2. Snapshot operations

The operations that are available for snapshots are:

As long as no changes are made to the memory/CPU of the VM.

This function depends on the hypervisor used, so, for more information check the hyper-

visor’s documentation.

215

2.15.2.1. Volume snapshot

Figure 300: Accessing the VM details

Figure 301: Taking a snapshot of the volume

Figure 302: Choosing which volume to take a snapshot

216

Figure 303: Conﬁguring the snapshot

2.15.2.2. Vm snapshot

Figure 304: Taking a snapshot of the VM

Figure 305: Conﬁguring the snapshot

217

2.15.2.3. Viewing created snapshots

Figure 306: Accessing the snapshots

Figure 307: Accessing the volume snapshots

Figure 308: Accessing the VM snapshots

218

2.15.2.4. Creating a template from a volume snapshot

Figure 309: Accessing the snapshot details

Figure 310: Creating a template

219

Figure 311: Conﬁguring the template information

2.15.2.5. Creating a volume from a volume snapshot

Figure 312: Creating a new volume

220

Figure 313: Conﬁguring the new volume

2.15.2.6. Reverting a volume snapshot

Figure 314: Reverting to the snapshot

Figure 315: Conﬁrming the operation

221

2.15.2.7. Removing a volume snapshot

Figure 316: Removing a volume snapshot

Figure 317: Conﬁrming the removal

2.15.2.8. Creating a volume snapshot from a VM snapshot

Figure 318: Accessing the VM snapshot details

222

Figure 319: Creating a volume snapshot from a VM snapshot

Figure 320: Conﬁguring which volume will be extracted

Figure 321: Creating a volume snapshot

223

2.15.2.9. Reverting a VM snapshot

Figure 322: Reverting the snapshot

Figure 323: Conﬁrming the operation

2.15.2.10. Removing a VM snapshot

Figure 324: Removing a VM snapshot

224

Figure 325: Conﬁrming the operation

2.15.2.11. Conﬁguring recurrent snapshots

This operation is described in page 112.

2.16. Events

Events are generated when a user performs an action in the cloud or when

the state of a resource, either virtual or physical, is changed.

Using events, it is possible to monitor tasks, jobs and processes in Cloud-

Stack, including potential errors.

2.16.1. Event types

The existing event types are:

• INFO: informs that a monitored process executed correctly.

• WARN: informs that a monitored process encountered a problem, though

it was not terminal

• ERROR: informs that a terminal error occurred in a monitored process.

2.16.2. Searching and ﬁltering events

To visualize and search through the events in the web interface, the follow-

ing steps can be executed:

225

Figure 326: Browsing through the events

The web interface does not support ﬁltering events by date.

2.16.3. Removing or archiving events

When selecting an event’s checkbox, it is also possible to remove or archive

them:

Figure 327: Archiving or removing an event

Regardless of the action selected, a conﬁrmation screen will pop-up.

226

2.17. User interface (UI)

The purpose of this section is to give additional details about the Apache

CloudStack UI, as well as tips to simplify its usage.

2.17.1. Using the browser’s timezone

All dates managed by ACS are stored in standard UTC within the database,

and when queried through the API, they are also returned in UTC. To improve

user experience, the GUI provides the Use local timezone option, which ensures

that dates are displayed to the user adjusted to the browser’s local timezone.

Figure 328: Enabling the setting

A usage example for this option is when ﬁltering VM statistics. With the

option Use local time zone disabled, a user in Brasilia’s timezone(UTC-3), for

instance, would need to enter time intervals in UTC rather than their own to

obtain the desired statistics. Thus, if that user tries to check a VM’s statistics

and sets the time interval from 12:00h to 17:00h without enabling the option,

the statistics corresponding to the period from 9:00h to 14:00h will be returned,

as the value would be interpreted as UTC. With the option enabled, the UI auto-

matically handles the conversion to the browser’s timezone, letting users query

and visualize statistics without requiring manual timezone conversions.

This option also aﬀects tariﬀ manipulation and Quota reports.

2.17.2. UI documentation

In the CloudStack web interface, most pages feature small help icons. Hov-

ering over them reveals short descriptions of the corresponding components,

which can be quite useful.

227

There are also question mark (?) icons that can be clicked to open a new

browser tab with the corresponding section of the oﬃcial CloudStack docu-

mentation for that component.

Figure 329: Example of pop-ups from help icons

2.18. Service oﬀerings

To create VMs, hypervisors require the speciﬁcation of their characteristics,

such as CPU, memory size, and disk allocation, among other deﬁnitions. Cloud-

Stack groups and standardizes these VM characteristics in the form of service

oﬀerings, in order to ease VM speciﬁcation, resource usage monitoring and, if

applicable, charge their usage. CloudStack separates service oﬀerings into two

categories:

• Disk oﬀerings

• Compute oﬀerings

2.18.1. Disk oﬀerings

Disk oﬀerings are service oﬀerings destined to storage resources from a VM.

Virtual machines created based on a template will use disk oﬀerings to create

extra disks, while those created with ISOs will need a disk oﬀering to create the

root disk.

2.18.1.1. Creating a disk oﬀering

To create a disk oﬀering:

228

Figure 330: Starting to create a new disk oﬀering

Clicking the A dd Disk Offering button opens a form where the necessary

information can be entered:

229

Figure 331: Creating a new disk oﬀering

230

This is a quick description about some of the form’s ﬁelds:

• Provisioning type:

1. Thin Provisioning: disks of this tipe will be allocated with the minimum

necessary size and will be able to grow accordingly to their need, until

the deﬁned limit is reached.

2. Sparse Provisioning: disks of this type are created with the size deﬁned

in the oﬀering, but they may contain residual data. Their creation is

fast, since this data is not erased or overwritten. However, it must be

cleared before new data can be written. The ﬁrst write operations on

this type of disk will therefore have lower performance.

3. Fat Provisioning: disks of this type will start allocated with the size es-

tablished in the oﬀering and all space used will be overwritten, eras-

ing any data present on the blocks. Their creation will be slower when

compared to sparse provisioning, due to the cleaning of old data,

however, their performance during the ﬁrst writes will be better.

• Disk Size Strictness: When set to true, the size of the volume cannot be

changed.

• QoS Type: deﬁnes the QoS policy for the disks. This functionality is only

available if supported by the hypervisor or storage.

1. None: By default, oﬀerings do not implement it.

2. Hypervisor: Deﬁnes the rate of write and read by the hypervisor. De-

pending on the hypervisor, some QoS deﬁnitions may not be imple-

mented, therefore, it is necessary to validate them before applying

to the production environment.

3. Storage: Deﬁnes minimum and maximum IOPS values for the stor-

age.

231

• Hypervisor Snapshot Reserve: Allows reserving additional space for snap-

shots beyond the data disk. The reserved space is calculated as the cho-

sen percentage plus the data disk size. This does not apply to KVM.

• Storage Tags: Storage tags will be used to direct the volumes to compatible

storages.

• regarding the access:

1. Public: Deﬁnes if the service oﬀering will be available to all domains

or if its use will be restricted.

2. Domain: This option will only show up when Public is unchecked. This

option determines the domains that will use the oﬀering.

3. Zone: Deﬁnes in which zones the oﬀering will be available.

2.18.1.2. Editing a disk oﬀering

After creation, only a few attributes of the disk oﬀering can be altered.

• Edit:

Figure 332: Starting to edit the disk oﬀering

232

Figure 333: Editing the disk oﬀering

• Update access:

Figure 334: Starting to update the disk oﬀering’s access

Figure 335: Updating the disk oﬀering’s access

• Change order:

233

Figure 336: Changing the order of disk oﬀerings

2.18.1.3. Removing a disk oﬀering

To remove a disk oﬀering, just access it and click the trash can icon:

Figure 337: Starting to remove the disk oﬀering

Figure 338: Removing the disk oﬀering

If there are volumes using the disk oﬀering, they will keep working normally,

as well as the VM’s migration and restarting processes, as CloudStack copies the

disk oﬀering characteristics to the disk, however it will not be possible to create

new volumes using it.

More information in the oﬃcial documentation.

234

2.18.2. Compute oﬀerings

Compute oﬀerings are oﬀerings dedicated to the computational resources

of a VM, like CPU and RAM. A compute oﬀering also contains a disk oﬀering,

which deﬁnes the characteristics of the VM’s root disk (when created based on

a template).

When creating a compute oﬀering, it will be associated to a disk oﬀering.

This association can occur in two ways. One of them is through disk speciﬁ-

cations, storage type and tags, which may be included directly in the compute

oﬀering. The second way is to assign a disk oﬀering to a compute oﬀering, this

causes the disk oﬀering to be used by the root disk during the instance cre-

ation. It is worth to emphasize that users can choose a diﬀerent disk oﬀering

for the root disk during the VM creation.

2.18.2.1. Creating a compute oﬀering

To create a compute oﬀering:

235

Figure 339: Starting to create a new compute oﬀering

Clicking the Add Compute Offeri ng button opens a form where the neces-

sary information can be entered:

236

Figure 340: Creating a new compute oﬀering - 1 - continues

237

Explanations about some of the form’s ﬁelds:

• Compute Oﬀering Type: alternatives that deﬁne the level of autonomy the

end user has when creating a disk oﬀering. Three options are available:

1. Fixed oﬀerings: oﬀerings with ﬁxed values for CPU (cores and speed)

and memory. All VMs created with this oﬀering will have the same

conﬁgurations.

2. Custom constrained oﬀerings: in this type of oﬀering, the adminis-

trator deﬁnes lower and upper limits for each resource. During VM

creation, the user must select values within the established limits for

each parameter.

3. Custom unconstrained oﬀerings: unlimited oﬀerings. Any value may

be informed during the VM creation, provided that it is within the

values available for use.

• CPU cores: total number of cores dedicated to the VM using this service

oﬀering. If the Custom constrained offerings option is selected, it is neces-

sary to determine the minimum and maximum quantity of cores during

the oﬀering’s creation. If the Custom unconstrained option is selected,

this ﬁeld will not be shown.

• CPU (in MHz): determines the core’s speed. This deﬁnition is only utilized

if the CPU limit is active. If the Cus tom Unconstrained option is selected,

this ﬁeld will not be shown.

• Memory (in MB): deﬁnes the amount of memory, in megabytes, that the

system will allocate to the VM. If the Custom con strained option is se-

lected, it is necessary to determine the minimum and maximum amount

of memory during the oﬀering’s creation. If the Custom Unconstrained

option is selected, this ﬁeld will not be shown, and the values must be

informed during the VM creation process.

238

• Host tags: these will be used to direct VMs to compatible hosts.

• Network Rate: deﬁnes the data transfer rate, in Mbps.

• Oﬀer HA: If selected, the VM will be monitored and, in case of host failure,

restarted in another available host.

• Dynamic Scaling Enabled: If selected, the CPU and memory of the in-

stance may scale dynamically.

• CPU Cap: limits CPU usage even if there are available resources.

• Volatile: VMs created with a volatile compute oﬀering will have their root

disks restored to their initial state every time they are booted.

• Deployment Planner: Algorithm used by CloudStack to choose in which

host to deploy the VMs created with this service oﬀering

1. First Fit: selects the ﬁrst host that has enough capacity to support

the instance requirements.

2. User Dispersing: uniformly disperses the instances from a single ac-

count in diﬀerent clusters or pods.

3. User Concentrated: concentrates instances from a single account in

the same pod.

4. Implicit Dedication: will deploy instances that are dedicated to a do-

main or speciﬁc account. If you choose this planner, it will be also

necessary to deﬁne a value for the planner mode parameter.

5. Bare Metal: it is used with a bare metal host.

• Planner Mode: This option will only be available if implicit dedication is

selected. The planner mode will determine how the instances will be de-

ployed in the private infrastructure.

1. Strict: A host will not be shared between diﬀerent accounts.

239

2. Preferred: The instances will be deployed in a dedicated infrastruc-

ture whenever possible. Otherwise, the instance may be deployed in

a shared infrastructure.

• GPU: Assigns a physical GPU, or part of it, to an instance. This makes

graphical applications executable in the VM. A list of available GPUs is dis-

played.

2.18.2.2. Editing a compute oﬀering

After creation, only a few attributes of the compute oﬀering can be altered.

• Edit:

Figure 341: Starting to edit the compute oﬀering

Figure 342: Editing the compute oﬀering

• Updating the access:

240

Figure 343: Starting to update the compute oﬀering’s access

Figure 344: Updating the compute oﬀering’s access

• Changing order:

Figure 345: Changing the compute oﬀering’s order

2.18.2.3. Removing a compute oﬀering

To remove a compute oﬀering, just access it and click the trash can icon:

Figure 346: Starting the compute oﬀering removal

241

Figure 347: Removing the compute oﬀering

If there are VMs running under the compute oﬀering they will keep working

normally, as CloudStack copies their compute oﬀering characteristics to the VM,

although it will not be possible to create new ones with it. When live scaling,

the VM that uses that oﬀering will need to exchange it for another one.

2.18.2.4. Changing the compute oﬀering of a VM

It is also possible to change the compute oﬀering used by an existing VM. There

are two ways to perform this change: via UI and via CloudMonkey, and both

ways require the VM to be stopped.

• Via UI:

Figure 348: Starting to change the compute oﬀering

242

Figure 349: Changing the compute oﬀering

Figure 350: Changing the compute oﬀering

After performing these steps the compute oﬀering of the VM will be up-

dated.

• Via CloudMonkey: It is possible to use the APIs scaleVirtualMachine or

changeServiceForVirtualMachine

. In this situation, both share the same

behaviour and will have the same result. Just use the command:

scale virtualmachine id=<VM_id> serviceofferingid=<compute_offering_id>

The syntax used by the APIs is the same, changing only the API call from scale vi r t u a l m a c

hine to change serviceforvirtualmachine.

243

When this command is successfully executed, its result will display several

details about the modiﬁed VM, including the newly assigned compute of-

fering.

{

"virtualmachine": {

"account": "admin",

"affinitygroup": [],

"cpunumber": 1,

"cpuspeed": 1000,

"created": "2022-03-08T17:06:23+0000",

"details": {

"Message.ReservedCapacityFreed.Flag": "true",

"rootDiskController": "osdefault"

"displayname": "VM-ea2c34dd-a8a9-4b6a-b6c4-9404e034f3b7",

...

"serviceofferingid": "ab647165-7a0a-4984-8452-7bfceb036528",

"serviceofferingname": "Small Instance",

"state": "Stopped",

"tags": [],

"templatedisplaytext": "Macchinina - Light SO",

"templateid": "b81aab72-2c37-466e-b53f-bdaf398322fa",

"templatename": "Macchinina",

"userid": "af1818b9-26de-11ec-8dcf-5254005dcdac",

"username": "admin",

"zoneid": "8b2ceb16-a2f2-40ea-8968-9e08984bdb17",

"zonename": "sc-floripa-01"

}

When performing changes to a VM’s compute oﬀering, it is important to

consider its host and storage tags, as well as the tags of the target compute

oﬀering. This is because only the compatible compute oﬀerings will be shown

as options for VM scaling, i.e., only compute oﬀerings with the same tags as the

VM or those without any tag restrictions.

More information in the oﬃcial documentation.

2.18.3. Network oﬀerings

Network oﬀerings are oﬀerings destined for network resources, such as

DNS, DHCP, ﬁrewall, load balancing, among other services.

2.18.3.1. Creating a network oﬀering

To create a network oﬀering:

244

Figure 351: Starting to create a network oﬀering

Clicking the Add Network Offering button opens a formulary where the nec-

essary information can be entered:

Figure 352: Creating a new network oﬀering - 1 - continues

It is possible to create three diﬀerent types of networks:

245

• Isolated: Networks designated for a single account.

• L2: Networks without services. When using this type of network, the ﬁnal

user must either handle IP management or use static IPs. At the logical

level, this network is limited to a single account. However, if multiple ac-

counts need to use it, a shared network with no services can be created,

eﬀectively simulating an L2 network (when creating a network with this of-

fering, CloudStack requires the IPv4 ﬁelds to be ﬁlled, but the values will

not be used and, therefore, any input is acceptable.).

• Shared: Networks that may be shared between accounts within the same

domain.

The Specify VLAN options allows the cloud administrator to assign a spe-

ciﬁc VLAN to the created network. It is important that this speciﬁc VLAN is not

within the range available for dynamic allocation, as deﬁned in the physical

network for guest traﬃc (accessible through the Physical Network tab of the

corresponding zone). If not selected, CloudStack will automatically assign one

from the available pool when the network moves to the Implemented state,

and it will be removed once the network is no longer in use (Allocated state).

For L2 networks, VLANs are always automatically assigned.

Networks with network oﬀerings marked as Persistent will be implemented

without the need to add a VM to the network.

246

Figure 353: Creating a new network oﬀering - 2

• With the Internet Protocol parameter, it is possible to specify which IP

version the oﬀering will support. CloudStack oﬀers two options: IPv4 (de-

fault) and Dual Stack. Dual Stack oﬀerings support both IPv4 and IPv6.

• The options Promiscuous Mode, MAC Address Changes and Forged Transmits

are destined to use with Nested Virtualization, on VMware.

• The Supported Services are the services available for selection. Each ser-

vice has a conﬁguration, making it possible to choose a provider for each

of them.

• From the compute oﬀering, it is possible to inform the service oﬀering

that the VR will use. To enable this feature, at least one of the following

services must be included in the list of Supported Services: VPN, DHCP,

DNS, Firewall, LB, UserData, SourceNat, StaticNat or PortForwarding.

• Networks with Conserve Mode will only allocate resources when the ﬁrst

247

VM is created.

• It is possible to select the Enable network oﬀering option so that the cre-

ated network oﬀering becomes available for use. Otherwise, it must be

enabled before being used.

2.18.3.2. Editing a network oﬀering

After creation, only a few attributes of the network oﬀering can be altered.

• Edit:

Figure 354: Starting to edit the network oﬀering

Figure 355: Editing the network oﬀering

• Enable/Disable the network oﬀering:

248

Figure 356: Starting to disable the network oﬀering

Figure 357: Disabling the network oﬀering

If there are any networks using the oﬀering when it is disabled, those net-

works will keep working normally, however, it will not be possible to create

new networks using it.

• Updating access:

Figure 358: Starting to update the network oﬀering’s access

Figure 359: Updating the network oﬀering’s access

249

• Changing order:

Figure 360: Changing the order of network oﬀerings

2.18.3.3. Removing a network oﬀering

To remove a network oﬀering, just access it and click the trash can icon:

Figure 361: Starting the network oﬀering removal

Figure 362: Removing the network oﬀering

If the network oﬀering is being used by networks, or if it is a default oﬀering,

it cannot be removed, however, it can still be disabled.

More information in the oﬃcial documentation.

250

2.18.4. VPC oﬀering

VPC oﬀerings are oﬀerings destined for VPCs (more information about VPCs

in section 2.7). They are similar to network oﬀerings (section 2.18.3).

2.18.4.1. Creating a VPC oﬀering

To create a VPC oﬀering:

Figure 363: Starting to create a new VPC oﬀering

Clicking the Add VPC Offering button opens a formulary where the neces-

sary information can be entered:

251

Figure 364: Creating a new VPC oﬀering

The Supported Services are the services available for selection. Each service

has a conﬁguration, making it possible to choose a provider for each of them.

For most services only the VpcVirtualRouter provider is available.

The Compute oﬀering ﬁeld is explained in Section 2.18.3.

It is possible to select the Enable VPC oﬀering option so that the created VPC

oﬀering becomes available for use. Otherwise, it must be enabled before being

used.

2.18.4.2. Editing a VPC oﬀering

After creation, only a few attributes of the VPC oﬀering can be altered.

• Edit:

252

Figure 365: Starting to edit a VPC oﬀering

Figure 366: Editing a VPC oﬀering

• Enable/Disable the oﬀering:

Figure 367: Starting to disable the VPC oﬀering

Figure 368: Disabling the VPC oﬀering

If there are any VPCs using the oﬀering when it is disabled, those VPCs

will keep working normally, however, it will not be possible to create new

VPCs using it.

253

• Updating access:

Figure 369: Starting to update the VPC oﬀering’s access

Figure 370: Updating the VPC oﬀering’s access

• Changing order:

Figure 371: Changing the order of VPC oﬀerings

2.18.4.3. Removing a VPC oﬀering

To remove a VPC oﬀering, just access it and click the trash can icon:

Figure 372: Starting the VPC oﬀering removal

254

Figure 373: Removing the VPC oﬀering

If the VPC oﬀering is being used by VPCs, or if it is a default oﬀering, it cannot

be removed, however, it can still be disabled.

255

3. CloudMonkey, API and others

CloudMonkey is a command line tool (Command Line Interface - CLI) written

in Go and used to interact with the CloudStack API. It is an open-source project,

that can be accessed through its github.

The steps to install, setup and use CloudMonkey are shown in the following

subsections.

3.1. Installation

CloudMonkey has two distinct installation methods, one for Linux/Mac users

and other for Windows users.

• Linux/Mac:

It is necessary to download the latest release available, selecting the binary

appropriate for your platform, then, copying it to a directory included in

the $PATH variable, such as /usr/local/bin. Via terminal:

sudo wget <file link> -O /usr/local/bin/cmk sudo

chmod +x /usr/local/bin/cmk

• Windows:

In the releases link previously mentioned, select the executable (.exe) cor-

responding to your platform and copy it to a directory included in the Win-

dows executables path, such as C:\Windows\System32\. Then, verify that

cmk.exe runs correctly.

3.2. Setup

By default, CloudMonkey will create an initial proﬁle with some default Cloud-

Stack credentials. To setup CloudMonkey, use the command set:

$ cmk

Apache CloudStack CloudMonkey 6.1.0

Report issues: https://github.com/apache/cloudstack-cloudmonkey/issues

(localcloud) > set url http://myprovider.cloud/client/api

(localcloud) > set username myusername

(localcloud) > set password mypassword

(localcloud) > sync Discovered 622 APIs

256

To create a new proﬁle, the same set can be used:

(localcloud) > set profile newprofile

(newprofile) > set url http://myprovider.cloud/client/api

(newprofile) > set username myusername

(newprofile) > set password mypassword

(newprofile) > sync Discovered 622 APIs

It may also be necessary to disable the setting for SSL certiﬁcate veriﬁcation

when using HTTP instead of HTTPS:

(localcloud) > set verifycert false

When logging in with another user, it is recommended to always use the sy

nc command, to verify which APIs the current user has access to. To return to

the previous user, just execute the command:

(newprofile) > set profile localcloud

3.3. Usage

Since CloudMonkey communicates with the CloudStack API, using it requires

a basic understanding of the performed operations and which APIs will be needed.

For that CloudMonkey has some facilitators:

• Verifying available APIs:

(localcloud) > list apis

{

"api": [

{

"description": "Creates VPC offering",

"isasync": true,

"name": "createVPCOffering",

"params": [

{

"description": "desired service capabilities as part of vpc offering",

"length": 255,

"name": "servicecapabilitylist",

"required": false,

"since": "4.4",

"type": "map"

...

}

"count": 622

}

257

• Autocomplete: pressing the TAB key provides a list of available command

suggestions.

Figure 374: Using autocomplete

Figure 375: Checking the autocomplete for a speciﬁc command

Figure 376: Checking the autocomplete for a speciﬁc API

258

• Details about an API: it is possible to obtain a detailed description of a

speciﬁc API with the -h ﬂag:

Figure 377: Using the -h ﬂag to get help

Or through the help:

(localcloud) > help createAccount

Once the needed procedures are known, just execute the API call via Cloud-

Monkey:

259

Figure 378: Performing the API call

260

Figure 379: Deploying a VM via CloudMonkey

For more detailed information about CloudMonkey’s usage, check the project

wiki.

261

4. Resources consumption accounting

The module for accounting computational resources consumption is subdi-

vided in two mechanisms: usage collection (Usage) and use accounting (Quota),

with the second acting complementarily to the ﬁrst.

The Usage mechanism periodically collects data about resource consump-

tion in the environment. The Quota mechanism is a plugin that allows the im-

plementation of a tariﬀ model for computational resource consumption man-

agement.

This section describes how to access the resource usage reports from Usage,

and the active tariﬀs and reports from Quota

4.1. Usage

4.1.1. Resource usage report

The resource usage reports can be accessed through the list U s a g e R e c o r d s

API, as shown next:

(admin) > list usagerecords startdate='yyyy-MM-dd'T'HH:mm:ssZ' enddate='yyyy-MM-dd'T'HH:mm:ssZ'

This command will return a list containing several entries with the following

structure:

...

{

"account": "<Account name>",

"accountid": "<Account id>",

"description": "<Report description>",

"domain": "<Domain name>",

"domainid": "<Domain ID>",

"enddate": "yyyy-MM-ddTHH:59:59+0000",

"rawusage": "Depends on the usagetype",

"startdate": "yyyy-MM-ddTHH:00:00+0000",

"tags": [<Set of tags>],

"usage": "<Depends on the usagetype>",

"usagetype": <Usage type>,

"zoneid": "<Zone ID>"

...

When an event is sent to CloudStack, it will not be immediately visible in the

API request tables, instead the tables will be periodically updated, based on the

time period deﬁned by the operator in the global settings.

262

4.2. Quota

4.2.1. Tariﬀs

To visualize active tariﬀs for the user’s account:

Figure 380: Active tariﬀs listing

Additional information regarding the tariﬀs, such as name, description and

value, can be accessed in the details page for each tariﬀ.

Figure 381: Tariﬀs details

4.2.2. Reports

By selecting the Summary submenu and then selecting the desired account,

it is possible to visualize details and Quota reports related to that account.

263

Figure 382: Accessing details and reports from an account

4.2.2.1. Credits

To visualize the credit addition/removal history (who added, how much was

added and the date), browse to the Credits tab:

Figure 383: Credits listing

This process may also be performed via CloudMonkey. Example:

(admin) > quota creditslist accountid=af16aaed-26de-11ec-8dcf-5254005dcdac domainid=52d83793-26de-11ec

-8dcf-5254005dcdac

264

{

"count": 3,

"credit": [

{

"credit": -192.87,

"creditedon": "2023-10-04T13:05:13+0000",

"creditorid": "af16aaed-26de-11ec-8dcf-5254005dcdac",

"creditorname": "admin",

"currency": "$",

"postingdate": "2023-10-04T13:05:06+0000"

{

"credit": 350,

"creditedon": "2023-10-04T13:20:24+0000",

"creditorid": "af16aaed-26de-11ec-8dcf-5254005dcdac",

"creditorname": "admin",

"currency": "$",

"postingdate": "2023-10-04T13:20:24+0000"

{

"credit": -25,

"creditedon": "2023-10-05T15:56:58+0000",

"creditorid": "af16aaed-26de-11ec-8dcf-5254005dcdac",

"creditorname": "admin",

"currency": "$",

"postingdate": "2023-10-05T15:56:58+0000"

}

]

}

4.2.2.2. Resource usage

The Quota usage tab shows information regarding resource usage during the

selected period:

265

Figure 384: Resource usage listing

Via CloudMonkey, the query must be performed as shown in the example:

(admin) > quota statement account=admin domainid=52d83793-26de-11ec-8dcf-5254005dcdac startdate=2023-1

0-01T00:00:00+0000 enddate=2023-10-05T13:55:01+0000

{

"statement": {

"account": "admin",

"accountid": "af16aaed-26de-11ec-8dcf-5254005dcdac",

"currency": "$",

"domain": "52d83793-26de-11ec-8dcf-5254005dcdac",

"enddate": "2023-10-05T13:55:01+0000",

"quotausage": [

{

"name": "RUNNING_VM",

"quota": 0,

"type": 1,

"unit": "Compute*Month"

{

"name": "ALLOCATED_VM",

"quota": 195,

"type": 2,

"unit": "Compute*Month"

...other resources...

{

"name": "NETWORK",

"quota": 0,

"type": 30,

"unit": "Compute*Month"

266

{

"name": "BACKUP_OBJECT",

"quota": 0,

"type": 31,

"unit": "GB*Month"

}

"startdate": "2023-10-01T00:00:00+0000",

"totalquota": 202.64381816

}

You can also execute a detailed consumption query:

Figure 385: Detailed listing for resource usage

Via CloudMonkey:

(admin) > quota statement account=admin domainid=52d83793-26de-11ec-8dcf-5254005

dcdac startdate=2023-10-04T00:00:00+0000 enddate=2023-10-05T13:55:01+0000

showresources=true type=6

{

"statement": {

"account": "admin",

"accountid": "af16aaed-26de-11ec-8dcf-5254005dcdac",

"currency": "$",

267

"domain": "52d83793-26de-11ec-8dcf-5254005dcdac",

"enddate": "2023-10-05T13:55:01+0000",

"quotausage": [

{

"name": "VOLUME",

"quota": 7.64381816,

"resources": [

{

"displayname": "usage",

"quotaconsumed": 4.805108,

"removed": false,

"resourceid": "f710ed0f-c234-4c31-89ae-de23c9d8779f"

{

"displayname": "DATA-220",

"quotaconsumed": 2.83871016,

"removed": false,

"resourceid": "e4727094-2571-4ebb-aca7-5cc97778fde0"

}

"type": 6,

"unit": "GB*Month"

}

"startdate": "2023-10-04T00:00:00+0000",

"totalquota": 7.64381816

}

4.2.2.3. Balance

The Daily Quota balance tab presents daily balance data (credits + resource

usage) for the selected period. The balance is calculated daily and represents

the sum of all previous data, with the current balance always referring to the

last day’s balance.

Example: Suppose a resource generates a dailly usage cost of $100,00. At

the end of the ﬁrst day, the balance will be -$100,00. At the end of the second

day, the balance will be -$200,00. At the end of the third day, the balance will

be -$300,00.If the search is ﬁltered for the period between the third and ﬁfth

day, the balances shown will be -$300,00 (third day), -$400,00 (fourth day) and

-$500,00 (ﬁfth day).

268

Figure 386: Quota balance listing

Via CloudMonkey:

(lab) > quota balance account=oliver domainid=7efa7a0c-8f43-4564-b8cc-d709424dc69a startdate=2022-04-1

0 enddate=2022-04-16

{

"balance": {

"currency": "$",

"dailybalances":

[

{

"balance": 99.67,

"date": "2022-04-10T00:00:00+0000"

{

"balance": 99.6,

"date": "2022-04-11T00:00:00+0000"

{

"balance": 99.53,

"date": "2022-04-12T00:00:00+0000"

{

"balance": 99.47,

"date": "2022-04-13T00:00:00+0000"

{

"balance": 94.23,

"date": "2022-04-14T00:00:00+0000"

{

"balance": 88.96,

"date": "2022-04-15T23:59:59+0000"

269

{

"balance": 83.7,

"date": "2022-04-16T23:59:59+0000"

}

]

}

270

5. Cloud-init

cloud-init is an open source software created to automate the initialization

and conﬁguration of operating systems during boot, especially in cloud-based

virtual machines. Its functionalities include password management, SSH key in-

jection, and user metadata injection. It is supported by major cloud orchestra-

tion platforms such as Azure, Google Compute Engine, OpenStack, and Cloud-

Stack.

5.1. Initialization steps

Cloud-init has ﬁve initialization steps, which are:

• Generator: only available for systems that use systemd, this is the ﬁrst

step executed and deﬁnes whether cloud-init should run during boot. To

disable its execution during OS boot, the /etc/cloud/cloud-init.disabled ﬁle

must exist.

• Local: step that locates the local datasources and applies the network set-

tings on the system.

• Network: this step is executed after the Local step and requires network

settings to be working properly. All modules informed in the cloud-init-

modules section of the /etc/cloud/cloud.cfg ﬁle will be processed here.

This step performs the initial setup of the virtual machine, such as setting

the hostname and executing the mount command.

• Conﬁg: this step is executed after the Network step and during it all mod-

ules informed in the section cloud-conﬁg-modules from the ﬁle /etc/cloud/

cloud.cfg will be processed. It is designated to independent settings mod-

ules, that do not aﬀect other modules, such as changing user password,

location and time zone of the operating system.

271

• Final: last step to be executed during the operating system boot. All mod-

ules informed in the section cloud-ﬁnal-modules from the ﬁle /etc/cloud

/cloud.cfg will be processed in it. It is designated for processes that are

usually executed by the user, such as package installation and update and

script execution.

If the user wants their own scripts to be executed, they need to be added

to the /var/lib/cloud/scripts folder, which has three subdirectories, repre-

senting their execution frequency:

– per-boot: scripts in this directory will be executed every time the op-

erating system is booted.

– per-instance: scripts under this directory will only be executed during

the ﬁrst boot of the operating system. Additional information can be

found in section 5.2.

– per-once: scripts inside this directory will only be executed once, even

if the operating system is changed. Additional information can be

found in section 5.2.

5.2. Execution frequency

Cloud-init has three possible execution frequencies for modules and scripts:

• per-boot/always: executes every boot.

• per-instance: only executes during the operating system’s ﬁrst boot. To

determine whether it is the ﬁrst boot, cloud-init stores a cache for future

validations. The existence of this cache implies that cloud-init has already

been executed, and therefore this is not the ﬁrst boot. This behaviour may

cause inconsistencies when an image is created based on that instance.

To address this, cloud-init adopts a default behaviour, called check, which

validates the instance ID stored in the cache. If they diﬀer, cloud-init con-

siders it the ﬁrst boot. There is also a trust behaviour, which skips the

272

instance ID validation. The instance ID usually is deﬁned by the orches-

trator when a new virtual machine is created. It is also possible to clear

the cache manually to redeﬁne the ﬁrst boot.

• per-once: only executes the ﬁrst time that the operating system boots,

even if the instance ID changes. This behaviour is similar to the trust be-

haviour from the per-instance frequency. Resources set to this frequency

can be executed again by manually clearing the cache and rebooting the

operating system.

To manually clear cloud-init’s cache, just execute the command:

cloud-init clean

5.3. Known datasources

Cloud-init supports integration with major cloud orchestrators, such as Azure,

Google Compute Engine, OpenStack and CloudStack (the complete list is avail-

able at known-sources). Each orchestrator operates in a unique way and has

their own settings. In CloudStack’s case, for example, the virtual machine must

be connected to a network that has a virtual router, as it is from this router that

cloud-init retrieves user information and metadata.

5.4. Modules

Cloud-init has several functionalities, such as password management, SSH

key injection and user metadata injection (the complete list is available at mod-

ules). If the user requires a functionality not provided by cloud-init, they can still

create custom scripts (more information in section 5.1). Each script must begin

with a line indicating which program should execute it. For example:

#!/bin/bash

Bear in mind that the the modules have diﬀerent execution frequency. Mod-

ules like ssh and set-passwords execute, by default, at per-instance frequency,

i.e., will only be executed during the ﬁrst boot of the operating system. If this

273

frequency needs to be altered, cloud-init allows it to be overwritten in the fol-

lowing way:

...

cloud_init_modules:

...

- update_etc_hosts

- ca-certs

- rsyslog

- users-groups

- [ ssh, always ]

...

5.5. Creating a customized image

It is possible to create a customized image via CloudStack by following these

steps:

• Create a VM from an ISO, as described in sections 2.10 and 2.3.

• With the operating system installed and setup, install the cloud-init pack-

age.

• By default, cloud-init attempts to retrieve data from all available data-

sources. If one fails, it tries the next. However, it is possible to limit the

datasources that cloud-init will utilize. As this image will be designated

to CloudStack, the datasources will be limited to it. To do so, create a ﬁle

named 99_cloudstack.cfg under /etc/cloud/cloud.cfg.d/ with the following

content:

datasource:

CloudStack: {}

None: {}

datasource_list: [CloudStack, None]

This setting will limit the datasources to CloudStack and None. If the data

is not found through CloudStack, no attempt will be made to retrieve it

from any other datasource. Other settings may also be created in the

same directory and cloud-init will read them in alphabetical order, how-

ever, by default, they will be deﬁned in the /etc/cloud/cloud.cfg ﬁle.

274

• In the /etc/cloud/cloud.cfg ﬁle, within the users section, users can be de-

ﬁned for creation. By default, there is already a default entry, which ref-

erences the default_user entry in the system_info section. The default_user

values can be edited to match the desired user.

• In the /etc/cloud/cloud.cfg ﬁle, when installing the cloud-init package, sev-

eral modules come pre-deﬁned. It is possible to change their properties

as well as add or remove modules whenever needed. More information

about modules in section 5.4.

To change the host name it is necessary to use the modules set_hostname,

update_hostname and update_etc_hosts. To inject SSH keys it is necessary to

use the ssh module. To update the default user’s password it is necessary

to use the module set-passwords.

Here is an example of module setup and all the steps to achieve this:

# The modules that run in the 'init' stage

cloud_init_modules:

- migrator

- seed_random

- bootcmd

- write-files

- growpart

- resizefs

- disk_setup

- mounts

- set_hostname

- update_hostname

- update_etc_hosts

- ca-certs

- rsyslog

- users-groups

- [ssh, always]

# The modules that run in the 'config' stage

cloud_config_modules:

# Emit the cloud config ready event

# this can be used by upstart jobs for 'start on cloud-config'.

- emit_upstart

- snap

- ssh-import-id

- locale

- [set-passwords, always]

- grub-dpkg

- apt-pipelining

- apt-configure

- ubuntu-advantage

- ntp

- timezone

- disable-ec2-metadata

275

- runcmd

- byobu

# The modules that run in the 'final' stage

cloud_final_modules:

- package-update-upgrade-install

- fan

- landscape

- lxd

- ubuntu-drivers

- chef

- reset_rmc

- refresh_rmc_and_interface

- rightscale_userdata

- scripts-per-boot

- ssh-authkey-fingerprints

- keys-to-console

- phone-home

- power-state-change

• When executing the ssh module, cloud-init will also generate new public

keys for the host. If the user saves their ﬁngerprints, they might receive

the following message and become unable to connect until the issue is

ﬁxed:

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @

@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@

IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!

Someone could be eavesdropping on you right now (man-in-the-middle attack)!

It is also possible that a host key has just been changed.

The fingerprint for the ECDSA key sent by the remote host is

...

To prevent this situation from occuring every time the user reboots the

VM, it is possible to add settings to cloud-init so that it does not generate

new public keys when executing the ssh module. For that, a ﬁle named

49_hostkeys.cfg must be created at the /etc/clou d/cloud.cfg.d/ directory,

containing the following content:

ssh_deletekeys: false

• To quickly validate the cloud-init settings, it is possible to shut down the

virtual machine, alter its name in CloudStack and restart it. As soon as it

starts, its hostname should match the name deﬁned on CloudStack.

276

• With the settings validated, turn oﬀ the virtual machine and create a tem-

plate based on the volume, as described in section 2.10. The template will

have cloud-init set up and ready for use.

5.6. Cloud-init in Windows

cloud-init is a solution developed for Linux, however there is an equivalent

software for Windows called cloudbase-init. It also has integration with Cloud-

Stack, with support to, but not limited to hostname change, password change

and SSH keys injection.

5.7. Troubleshooting cloud-init

If an error occurs during the execution of cloud-init modules, it is possible

to begin troubleshooting through the logs recorded in the ﬁles /var/log/cloud-i

nit*.

5.8. Force password change during ﬁrst login

When cloud-init injects the default user password, it may be interesting, for

security sake, to force its change during the ﬁrst login.

For that, it is necessary to create a user directive in the VM that has cloud-init

installed. Just follow these steps:

1. Access the VM containing cloud-init;

2. create the user directive ﬁle:

sudo touch /etc/cloud/cloud.cfg/reset_password.cfg

3. add the following commands to expire password within the created ﬁle

(in <default-user> the default user from the VM must be informed):

#cloud-config

runcmd:

- echo "Default user password expired to force password change during first access"

- passwd --e <default-user>

4. change the access ﬁle so that it becomes an executable:

sudo chmod +x /var/lib/cloud/scripts/per-instance/reset_password.sh

277

After these steps, just create a template based on this VM, as shown in sec-

tion 2.10. Then, all VMs created from this template will force the user to change

the password during their ﬁrst login.

5.9. Self incrementing disk size during boot

The following processes were tested by the SC Clouds team with the Ubuntu

20.04 LTS operating system, Cloud.init version 21.4, LVM version 2.03.07. We

do not guarantee full functionality on other systems, libraries and com-

ponents. Use this section only as an example.

It is possible to automatically resize the disk during boot using a script exe-

cuted by Cloud-init with the per-boot frequency. For this, the last partition of the

device must be conﬁgured as a volume group (VG) containing a logical volume

(LV).

1. Access the VM containing cloud-init;

2. Create a script ﬁle:

sudo touch /var/lib/cloud/scripts/per-boot/resizelvm.sh

3. The following script expands the LV with the whole block available at the

end of the disk, assuming that:

(a) The device is /dev/vda

(b) The partition containing the VG is the 3

#!/bin/bash

DEVICE="/dev/vda"

LVPARTITION="3"

LVPATH="/dev/vg0/lv0"

growpart "$DEVICE" "$LVPARTITION"

pvresize "$DEVICE""$LVPARTITION"

lvextend -l +100%Free "$LVPATH"

resize2fs "$LVPATH"

4. Grant execution permission for this script:

sudo chmod +x /var/lib/cloud/scripts/per-boot/resizelvm.sh

278

With that, the instance will verify if there is enough available space to expand

the LVM during the boot process and, if there is, the expansion will be per-

formed allocating available blocks of the partition pre-conﬁgured in the script.

5.10. Interactions with user-data

It is important to keep in mind the interaction between cloud-init and the

user-data script from the VM. The user-data executes after cloud-init. There-

fore, if cloud-init deﬁnes the password for the users:

chpasswd:

expire: false

users:

- name: ubuntu

password: cloudinit

type: text

- name: test

password: cloudinit

type: text

And the user-data script also does that:

#!/bin/bash

echo "ubuntu:userdata" > passwords

cat passwords | chpasswd

rm passwords

The settings from cloud-init will be overwritten by those from user-data. In

the example, the password for the user ubuntu will be userdata, and for the

user test will be cloudinit.

279

6. Conclusion

This document addressed, in general, the main concepts and recurring doubts

regarding Apache CloudStack and auxiliary tools. Both are vast and complex,

therefore it is not possible to adress them in their entirety, however, in case

of any doubts or suggestions for improvement in this documentation, an issue

can be created on GitLab.

280

Appendices

Appendix A. Terminology

• Infrastructure-as-a-Service:

Infrastructure-as-a-Service, also known as IaaS, consists of oﬀering com-

putational resources, such as processing, storage and network access, on

demand, usually while applying pricing to the usage of these resources.

• Hypervisor:

Software responsible for the creation and execution of virtual machines

(VMs). A hypervisor allows a host machine to provide support to several

guest VMs, virtually sharing its resources, such as memory, storage and

processing, thus allowing better usage of the available resources.

• VM:

Acronym for Virtual Machine. It is a software that emulates a real com-

puter. Also known as guest, it is created on another computer, known as

host, and uses part of its resources. The advantages of using VMs are:

– Allows using diﬀerent operating systems in a single computer.

– Extremely easy to manage and maintain.

– VMs can be easily created or replicated with an operating system pre-

viously installed and conﬁgured.

– VMs can also be easily migrated from one host to another without

data loss.

• Virtual network card:

Commonly referred to as NIC, VNIC or VIF. It is a software that fulﬁlls the

role of a network card in a virtual system.

281