vRA 6 distributed setup problems

by | Jun 14, 2016 | VMware, vRealize, vRealize Automation

This week I’m at the SDDC consulting training at the VMware EMEA HQ in Staines. There is a really full program with presentations and labs about the VMware SDDC portfolio. Products that will be covered in the training are:

  • vRealize Automation
  • vRealize Orchestrator
  • VMware NSX
  • VMware SRM

But the most important focus this week is the integration between all VMware products and 3th party products like InfoBlox and Service Now.

We started yesterday with the installation of a distributed vRealize Automation 6 environment. After clicking thru 281 pages of instruction the installation was finished. Some people in the class had problems with the lab base environment because of time out errors. The reason was a slow network connection not just slow but really really slow…

Dropped Packages

The lab environment consists of virtualized ESXi hosts and is using NSX for the networking part. In NSX there is some bug (or should I say undocumented feature ;-)) that cause lots of packet drops when using virtualized ESXi hosts and NSX. The solution to work around is to create DRS rules to keep some of the VMs (the ones you are working on) together on a virtualized ESXi host so all network traffic is kept locally. I think it’s also possible you experience the same slow connection if you are doing the VMware Hands On Labs because the setup is probably the same.

Strange names

Today when booting up my lab again I had the issue that the infrastructure tab had a strange name. The name was changed in:  com.vmware.csp.component.iaas.proxy.provider@csp.places.iaas.label instead of just Infrastructure. All underlying tabs had the same problems. If you know where to click everything is still working, but it doesn’t feel good.

vRA6-comvmwarecspcomponent

The solution to this problem is to just reboot some nodes of the vRA installation. But wait, which of the 10 servers do need a reboot? The answer is nearly all of them. The boot order for the complete stack is:

  1. Microsoft SQL Database server
  2. Identity appliance
  3. vRealize appliance 1
  4. vRealize appliance 2
  5. IAAS webserver 1 & 2 (vRealize webportal and ModelManagerData services)
  6. Application server 1 ( primary IAAS Manager and the DEM Orchestrator Server services)
  7. Application server 2 (secondary IAAS Manager and the DEM Orchestrator Server services)
  8. Proxy server 1 & 2 (DEM worker and Proxy Agent server services)

Rebooting from step 3 will resolve this issue. First shutdown all services in the reverse order and when you are at the vRealize appliance 1 just reboot this one. Wait till the VAMI is showing up in the console and then (and not earlier!) start the next one of the list. If the server is a Windows server give it some extra time to boot up all services.

If everything is restarted then you will see the normally names and tabs.

Infrastructure tab

VMware vCloud Air part 1

by | May 5, 2016 | VMware, vRealize

The last couple of weeks I’ve been doing research on the VMware vCloud Air for one of our customers. Our customers is looking at the vCloud Air solution for large pieces of their current infrastructure, and the big driver for this investigation is the fact that at first looks the prices for VMware vCloud Air look cheaper (way cheaper accually) in stead of building and migrating and their own Datacenters. But to get a clearer view of what the VMware vCloud Air , let’s dive in to vCloud Air to see if this is true and what vCloud Air is.

Vmware vCloud Air

First we’ll investigate what VMware is providing through their vCloud Air offering. To make sure we got everything right we’ll see what the VMware website tells us about their vCloud Air offering:

 

As you can see there are multiple ways the business could use the VMware vCloud Air solution as seen in picture above there are 6 different offerings:

  • Disaster Recovery
  • Virtual Private Cloud
  • Virtual Private Cloud OnDemand
  • vCloud Government Service
  • Object Storage
  • Dedicated Cloud

For this part we’re going to take a look at the Dedicated Cloud offering and we’re diving in to the benefits and positive points of the solution.

Welcome in the buzzword bingo?

There is to much buzzwords going around in this area that it is hard to keep track on what is what! What is the difference betweens private cloud, Infrastructure as a Service and Dedicated cloud and on-premises infrastructure? What does it mean for your company and where does fit in your IT environment?

Invisible IT?

For me the Invisible IT buzzwords is what most companies (I do business with) are really looking for…. Providing the IT resources instantly when the business needs it, and thereby being a business enabler is what IT should be all about.

In the the last couple of decades it often happened that when the business needed a new application to help the business growth, it could take months before the application could be used. With the birth, and adoption of virtualization most IT departments managed to cut this down to about a week. But that time is used to just implement the virtual servers needed for the application (with the right network and storage resources), and after this time the application still needs to implement and test the the application.

Welcome to the Cloud Era

With the introduction of cloud a lot of people were sceptic. But after a couple of years people use it all the time, and got used to the benefits of cloud computing. One of the biggest advantages of cloud computing is how fast it is to buy resources. Go to AWS, Azure, Google or whatever cloud providor you want and with a credit card and a few clicks your VM is running in minutes….

This is where most IT departments lost the battle (they think…). If a in house department still needs to wait weeks of even still months before they can really start developing, implementing and using the application they tend to run, and use, the public cloud quickly. They normally won’t think of the business impact of such a move, but on the other the project can deliver much quicker and that’s all that counts to them.

As Dilbert explained in the comic above there is a way for IT to use the on-premises resources as well as the public cloud to move be the business enabler IT needs to be.

Virtualization vs. Hybrid Cloud

It seems such a long time ago that virtualization needed to prove its place in the Datacenter. A lot of companies looked at the virtualization product and didn’t see it production ready, but after testing it in their test environments and seeing the benefits almost all companies testing also started using it in their production environments as well.

The same is seems to happen with the use of hybrid cloud, but it seems that the hybrid cloud adoption goes much faster. The way companies start using a hybrid cloud solution is lots of time driven by the fact that certain workloads already started their development in the public cloud, and the company would like to embed the posibilities the cloud provides. The Hybrid cloud is the combination of private (which could also be a traditional IT environment) and public cloud(s) which provides your company the best of both worlds. But to manage these clous, you’ll need the right tools.

 

Cloud Management Platform

To manage your comapnies Hybrid Cloud they’ll need a Cloud Management Platform. As already mentioned the CMP’s are Management portals that offer your business the management needed to provide the private and public IT services. It is important to know that although there are many CMP’s I have found any (yet) that offers the complete spectrum of private and public offerings, although they all offer REStful api support so you could create certain things yourself (if you have the development force to do so ;)) I’ll probably dive into a couple of the CMP’s at a later stage, but for now if you want to know more about CMP’s look at these:

There are many more, but for now it is more than enough to have some reading material during a couple of days 😉

VMware vRealize suite and vCloud Air

I started this post about the VMware vCloud Air solution, but in the end I didn’t really talk about it that much. I promise I’ll do more in depth in the next part but for now I want to focus a little more on VMware vRealize Suite and the vCloud Air products for building a VMware Hybrid cloud.

With a lot of companies that build their virtualization environment on the VMware vSphere product, it is an easy step to want to build their hybrid IT environment on this foundation. To do so, they can leverage the vRealize suite product to automate and orchestrate their current environment as well as the vCloud air solutions, and furthermore other cloud solutions like AWS, Azure and others.

For a lot of companies this would build the environment they need to be on the edge, while still maintaining a soltution build on the foundation they already had, keeping the knowledge they already have in house, and giving IT the power to become a business enabler again.

Conclusion

When I started this post I didn’t intend it to be this long, and that’s the main reason to stop puting more information in this single post. Where I started out with an introduction to VMware vCloud Air, it became much more, but that’s what blogging is all about (IMHO :D) I’ll be back with more information on vCloud Air, vRealize suite, CMP, and more…. But for now cheerio!

If you want to know more about this topic, I’ll be presenting at next TECHunplugged conference in London on 12/5/16. A one day event focused on cloud computing and IT infrastructure with an innovative formula combines a group of independent, insightful and well-recognized bloggers with disruptive technology vendors and end users who manage rich technology environments. Join us!

 

 

Upgrading vRealize Automation 7 to 7.0.1

by | Mar 29, 2016 | VMware, vRealize, vRealize Automation

This is a cross post from my Metis IT blogpost, which you can find here.

Last week VMware released a new version of vRealize Automation (vRA), version 7.0.1. In this version most of the version 7.0.0 bugs and issues are resolved. In the release notes you can find the list of all resolved issues. In this blog I will guide you through the upgrade process.

It is possible to upgrade to this new version from any supported vRealize Automation 6.2.x version and the latest 7.0 version. In this blog I will focus on an upgrade from version 7.0.0 to version 7.0.1. If you still use an earlier version of vRA you have to upgrade frist to version 6.2.x. The environment we will upgrade is a minimum deployment based on version 7.0.0.

The following steps are required for a successful upgrade of vRealize Automation.

  1. Backup your current installation
  2. Shut down vRealize Automation Windows services on your IAAS server
  3. Configure hardware resources
  4. Download and install upgrades to the vRA appliance
  5. Download and install updates for IAAS
  6. Post Upgrade Tasks

Backup your current installation

Before you start  the upgrade it is important to backup some components of the existing installation. If something goes wrong you can always go back to the current version.

Configuration file backup

First start with a backup of the vRA configuration files. This file can be backupped with the following steps:

  1. Login with ssh on the vRA appliance
  2. Make a copy of the following directories:
    • /etc/vcac/
    • /etc/vco/
    • /etc/apache2/
    • /etc/rabbitmq/

First create a directory backup.

mkdir /etc/backupconf

Copy now all directories to this folder:

cp -R /etc/vcac/ /etc/backupconf/

Perform these steps for each folder.

Database backup

Make a SQL backup of the vRA IAAS database. For the integrated postgres database it is enough to snapshot the complete vRA appliance.

  1. Login to the database server
  2. Open the MSSQL Management Console and login
  3. Click left on the vRA database and choose Tasks and choose Backup Up…
  4. Choose the location for the backup and click on OK.
  5. Wait for the completion of the backup.

 

Screenshots of the tenant configuration and users

If something goes wrong with the upgrade it could be possible that this configuration information is changed. For safety it is recommended to capture this information.

  1. Login as administrator to the vRA interface
  2. Make a Screenshot of your tenantsafb1
  3. And the Local Users of the tenantafb2
  4. And the Administratorsafb3

Backup any files you have customized

The vRA upgrade will possibly delete or modify all customized files. If you want to keep this files please backup them. In our environment we don’t use any customized files.

Create snapshot of the IAAS server

To take a snapshot of the IAAS server is the last step in the upgrade process.

  1. Shutdown the IAAS server and the vRA appliance in the correct order.
    1. Login to vCenter
    2. First select the IAAS VM and select shutdown guest. If the shutdown is complete select the vRA appliance and choose again for shutdown guest.
  2. Right-click on the IAAS VM and select Snapshots and Take Snapshot. Fill in the name of the snapshot and click on OK.
  3. Power On the IAAS VM

Disable the IAAS services

  1. Login on the IAAS server, open msc and stop the following services:
    1. All VMware vCloud Automation agents
    2. All VMware DEM workers
    3. All DEM orchestrator
    4. VMware vCloud Automation Center Service

afb4

Configure hardware resources of the vRA appliance

For the upgrade it is  necessary to extend the existing disks of the vRA appliance. But before we do this, create a copy of the existing vRA appliance.

  1. Right-click on the vRA appliance, select Clone and Clone to Virtual Machine
  2. Give the VM a unique name and select the resources for the new VM and click on Finish.
  3. Wait for completion.
  4. Right-click on the original VM and select Edit Settings.
  5. Extend the first disks (1) to 50GB and click OK.afb5
  6. Create a snapshot of the VM. Select the VM, click on Snapshots and click Take Snapshot.
  7. Wait for the snapshot.
  8. Power on the vRA VM.
  9. Wait for the machine to start
  10. SSH to the vRA VM and login with the root
  11. Execute the following commands to stop all vRA services:

Service vcac-server stop

Service vco-server stop

Service vpostgres stop

  1. Extend the Linux file system with the following commands:

Unmount swap table:
Swapoff –a

Delete existing partitions and create a 44GB root and 6GB swap partition. This command and the next command return an error about the kernel that is still active at this point. After a reboot at step 13 all changes will be active:
(echo d; echo 2; echo d; echo 1; echo n; echo p; echo ; echo ; echo ‘+44G’; echo n; echo p; echo ; echo ; echo ; echo w; echo p; echo q) | fdisk /dev/sda

Change the swap partition type:

(echo t; echo 2; echo 82; echo w; echo p; echo q) | fdisk /dev/sda

 Set disk 1 bootable:

(echo a; echo 1; echo w; echo p; echo q) | fdisk /dev/sda

 Register partition changes and format the new swap partition:

Partprobe

Mkswap /dev/sda2

Mount the swap partition:

Swapon –a

  1. Reboot the vRA partition
  2. When the appliance is started again login with SSH and resize the partiation table:

Resize2fs /dev/sda1

  1. Check the resize with command df -h

Install the vRA update

  1. Login on the management interface: https://vRAhostname:5480
  2. Click on the Services tab and check the services. All services should be registered except the iaas-service.

If everything is checked, click on the update tab. If not all services are running and you are using a proxy server, check this Vmware article: https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2144067

  1. Click on Check Updates. The new update will be displayed.afb6
  2. Click now on Install update and Click OK.
  3. The follow the installation you can check the following log files: /opt/vmware/var/log/vami/updatecli.log

/opt/vmware/var/log/vami/vami.log

/var/log/vmware/horizon/horizon.log

The most useful information can be found in the vami.log and updatecli.log. In these log files you can see the download progress and information about the upgrade status.

Use tail –f /opt/vmware/var/log/vami/* to show all log files

  1. Wait untill the update is finished.afb7
  2. If the upgrade is finished, reboot the appliance. Click on the System tab and click on

Upgrading the IAAS server components

The next step in this process is to upgrade the IAAS components. The IAAS installer will also upgrade the MSSQL database. In earlier upgrade processes it was needed to separately upgrade the database. To start the IAAS upgrade, follow the following steps:

  1. Open your favorite webbrowser and go to: https://vRAhostname:5480/installer
  2. Click the IAAS installer Save the prompted file. (Do not change the filename!)
  3. Open the installer and follow the wizard.
  4. Accept the license agreement and click on Next.
  5. Provide the Appliance Login Information. Click on Next.

afb8

  1. Choose for Upgrade. Click on Next.
  2. Provide the correct Service Account for the component services and the authentication information of the SQL server. Click on Next.
  3. Accept the certificate of the SSO Default Tenant and provide the SSO Administrator Credentials. Click on Next.
  4. Click now on Upgrade to start the upgrade.afb9
  5. Click on Next and finish to complete the IAAS upgrade.

Post upgrade tasks

After the IAAS upgrade first check the correct operation of the vRA appliance. Click on the infrastructure tab and click on endpoint. Verify the endpoint overview is correct. Next try to request a blueprint and check if everything will finish successful.

If everything is correct, the last step is the upgrade of the vRA agents on the OS templates. The new agents also contain some bug fixes. In our environment we use CentOS and Windows Operating Systems. We will first start with the upgrade of the CentOS agent followed by the Windows Agent.

CentOS agent

  1. Convert the CentOS template to a VM and boot the VM.
  2. Download the prepare_vra_template.sh script from the following location: https://vRAhostname.local:5480/service/software/download/prepare_vra_template.sh
  3. Allow execution of the script with:

chmod +x prepare_vra_template.sh

  1. Execute the script: ./prepare_vra_template.sh.
  2. Follow the wizard and provide the correct information. I choose for vSphere, no certificate check and the install Java.
  3. Wait for completion and shutdown the VM.
  4. Convert the VM back to a template.

Windows Agent

For the upgrade of the Windows Agent we will use the script made by Gary Coburn. He developed a script that will install all the needed components and the vRA agent on Windows. Thanks to my colleague Ronald van Vugt for this modification on this script because of newer java version. The original script is based on vRA version 7.0.0 which included version jre-1.8.0-66. The java version included in version 7.0.1 is newer, so a modification to the script is required.

  1. Download the original script from here or here. And open the script and search for the following line:
    $url=”https://” + $vRAurl + “:5480/service/software/download/jre-1.8.0_66-win64.zip”
  1. This line must be edited to:
    $url=”https://” + $vRAurl + “:5480/service/software/download/jre-1.8.0_72-win64.zip”
  1. If the script is edited run the script with the following parameters:

./prepare_vra_template.ps1 vra-hostname iaas-hostnamePasswordofDarwinUser

  1. The script will sometimes ask for confirmation.
  2. Wait till the installation is complete.
  3. Shutdown the VM and convert it again to a template.

Verify the installation

Now request some of your blueprints to verify the correct operation of the vRA appliance, IAAS server and the guest agents. If everything is OK, then it is time to delete the snapshots of the vRA appliance and IAAS server.

  1. Select the VM, choose for snapshots and Manage Snapshots
  2. Delete the snapshot you have made before installation.
  3. Do this for both VMs

Conclusion

Before executing this upgrade in a production environment it is recommended to plan the upgrade and verify that all dependencies will work after the upgrade. Also plan enough time for this upgrade, so you have the time to check and verify the installation.

Download here the upgrade procedure in PDF.