Reporting from the field Issue #2: Active Directory/DNS Changes post implementation of vSphere – Lessons Learned

I love working with customers.  It’s challenging and rewarding at the same time.  Sometimes it can be more challenging but how you rise above those challenges is where the rewards are really noticeable.   Working with a customer recently, they had some challenges around their current Domain.  The Domain consisted of a couple of Windows Server 2003 systems running a Windows 2000 Native Domain.  After my implementation, one of the recommendations I provided was a migration away from the Legacy 2k Domain and to modernize and re-architect their Domain/DNS Structure.

Fast forward a couple of weeks, and I started receiving emails about loss of access into their vSphere Infrastructure I had just installed.  The first question I always ask when I get on the phone in situations like these is “Did anything change?”  The answer to this question was  yes…which is typically what I find.  This customer took my advice and they did implement and upgrade with two new Windows 2008 R2 Domain Controllers and a change to a Windows 2008 R2 Native Domain.

Two things that popped in my head.  One, this must have broke Single Sign On as the Identity Source must not be valid.  The second, the Domain I joined for the vCSA is more than likely invalid.

I logged into the vSphere Web Client with an SSO admin account and when I went to look at the Identity Sources, I received this error:

 ‘alias’ value should not be empty

I immediately connected to the Admin Console of the vCSA: https://yourvCSAHostname or IP:5480

CapturevCSA

 

 

 

 

 

 

 

 

I then selected Configure SSO

CapturevCSA1

 

 

 

 

 

 

 

 

I noticed that the SSO Deployment Type was showing UNCONFIGURED.

I setup SSO again then moved on to rejoin Active Directory here:

CapturevCSA2

 

 

 

 

 

 

 

 

I then rebooted the vCSA logged back in as SSO Adminstrator, reconfigured the Identity Source, and removed and re-added my user permissions.

CapturevCSA3

 

 

 

 

 

These steps resolved the issues.  Customer could now connect and work within vCenter.

Lessons Learned: In this case I made my recommendations to my customer which they happily followed, however, I was not thorough in explaining the impact that this would have on their Shiny new VMware vSphere Infrastructure.  Of course, this should have been explained in detail to them and the upgrade should have been scheduled in cooperation with us.

Take away: Communication with the customer is key.  Point out risk, and always be upfront as to how changes can impact the environment.

SRM findings in the Lab

I wanted to learn how to setup and configure SRM for a PoC for a new contract at work.  After downloading the Docs from VMware and looking over the requirements, I went right to work.

The first phase of the testing was leveraging vSphere replication where the replication of the VM(s) is done between the Host and the VR server.  We’ll get to the VR server in a minute..let’s first talk about the setup.

First, let’s get the lab out of the way.  I added an additional Host as an acting “DR” site.  This host is very basic and runs limited VM’s. It’s only for emulating a failover site.  Here is the updated diagram with the additional “DR” host representing the Recovery Site and Recovery Cluster:

Image

This “DR” Host runs vCenter/SRM/vSphere Replication Server/vSphere Replication Mgmt Server, and of course, the recovered VM which is just a basic XP machine used for recovery testing only.(make sure VM tools is installed)

Image

The “PROD” site, runs many VM’s, however this also runs vCenter, of course, SRM/vSphere Replication MGMT Server.  Since i’m only concerned about testing one way DR at this point, I do not need a vSphere Replication Server at the Prod Site.

Image

Now, again, this is not going to be a tutorial on setting up an SRM Lab.  You can find plenty of those on other blogs and using the SRM documentation from VMware.  I’m not one for publishing what’s already out there if I can help it.

Back to the findings.  For SRM alone, there are several DB instances that would need to be setup.  You need a DB instance for SRM at both the PROD and RECOVERY sites.  You need a DB instance for the vSphere Replication Manager Server at both the PROD and RECOVERY sites.  Along with vCenter, you could easily have 6 DB instances supporting this.

Thanks to help from some very experienced folks from some of the boards and blogs I go to, I was able to resolve  two issues I ran into when setting this up.

1.  You need to make sure that you are either using FQDN and have DNS entries, of course, for all the connectivity or you use ALL IP.  Some of the appliances such as the VRMS like to put the IP in for vCenter and it’s imperative that you either choose IP for DB/SR/vCenter or all FQDN or else you will run into issues.  I chose IP as it was quicker, most likely not the correct approach in a production envioronment, however.

2.  Pay attention to the DB requirements for all these products.  The windows based products support ODBC and Windows Authentication, while the appliances require JDBC connections and do not support Active Directory Authentication so you’ll have to use a local DB account as the owner or else you’ll run into this:

Image

When you go through all the steps and have it all setup, you can right click on a RUNNING VM and select vSphere Replication:

Image

Choose your RPO:

Image

Now you can watch the SYNC status within SRM/vSphere Replication Tab:

Image

When that is complete create your Protection Group(s) and Recovery Plan(s) and now you are ready to test:

Image

And now you can see the VM up at the RECOVERY Site:

Image

There ya have it.  Testing a simple SRM vSPhere Replication in a lab!

vSphere Database Migration from SQL Express 2008 to SQL Server 2008 R2

When I first took over our vSphere environment from one of my former admins who left the company, I was still getting my feet wet with VMware products.  At that time I had very little experience with vSphere or VMware products in general.  My background was Microsoft, some Cisco, having just obtained the CCNA, and SAN,  and the virtual world experience was summed up in using Microsoft  Type 2 hypervisor technologies.

I knew that I had to get a handle on our environment.  Long story short, we actually started our journey with a couple of servers using Virtual Server.  As you can imagine, not the most robust virtual environment.  At that time, we were led to believe that we had a VMware skillset within our midst so on we went to rolling out VDI on 3.5.  I can tell you it was painful to say the least.  When the admin left, I was left with a few virtual servers running on a single machine running Virtual Server, while not tier one applications, still very important to the business, and a contract that was extremely unhappy with VDI performance.

After educating myself, building a lab etc, I conveyed to my management team that we needed to undo the mess that was made and do it right.  At that time vSphere 4.1 had just came out and it was all the rave…and now I see why.  We had very little time to get VDI out of the mess it was in and to start down our Server consolidation path.  Yes, we certainly were late to the party, but my view is that we just needed to show up to start seeing the benefits.

Right away, I transitioned our VDI environment to View 4.5 and vSphere 4.1.  I was still learning and still trying to retain all this information that was being thrown at me.  Unfortunately, while we got the infrastructure, storage, and network right, I missed one very important thing and that was having a centralized DB MGMT tier to run our vSphere INF DB for things such as vCenter, VUM, View, etc.  We were running SQL Express 2008 and after continuing to educate myself in reading the documents, obtaining certifications, etc, I realized that we were at risk not having a core DB for VMware Infrastructure Applications.

At that point a decision was made to create a centralized DB VM based on MS SQL Server 2008 R2.

Let me first start out by saying that I am not a DBA, however, after going through the process, it’s not that bad.  Of course this process doesn’t take you deep into Database MGMT, etc, but you will learn some things along the way and today, I have no problem setting up SQL for vCenter, VUM..etc.  It’s very easy as long as you make sure that you follow all the steps in the process and that you have some things figured out before diving in.  As you know, installing SQL Express with the standard vCenter install is a cakewalk, but it could steer you down the wrong path, and one that you should really should not consider in a Production environment unless you are running less than 10 hosts and 50 VMs.

Since there is information freely out there pertaining the steps in doing this type of migration, I won’t reiterate it here.  I’m not one for “reposting” whats been done or said by others.  One particular site that has a step by step directions, broken down into 4 sections is here:

http://boubchir.co.uk/vm-blog/migrating-vcenter-database-express-to-sql-2008-r2/

I first attempted this in my lab before I did this on our prod environment and my lab is a little quirky.  On of the things I had setup in my lab was I had an account on my Domain that I was using for my “service” account for VMware Inf Apps, but it was the same as my actual Domain Name.  My user account was CORE and my domain account was CORE.  When you get to the reinstallation of vCenter, vCenter will autopopulate your account, except it will not include the Domain name in front of it.  This did cause problems in vCenter 4.1 installation and users had to create a matching account locally on the vCenter server, install vCenter, then go back and change the service to start using the Domain service account.  Back to my issue, however, I found that you can’t use an account that is:

1.  The same name as your Domain Name, such in my case.

2.  The same name as your vCenter Server Host name.

I hope this helps those that run into this issue.  Use the directs from the blog I posted, but if you run into these particular issues, then make sure you remember these findings.

I also want to note that the vCenter 5 install has the same issues along with a requirement to have a Reverse Lookup Zone in your environment, which every Prod environment should have anyway.  After the account is populated it does a reverse lookup on the FQDN of the vCenter server and if a Reverse Lookup Zone or PTR record doesn’t exist it will also fail the install.

Just some findings..hope this helps!