SharePoint on Windows Azure – Part 2: Storage

March 8, 2013
SharePoint
This entry is part 1 of 4 in the series SharePoint on Windows Azure

In Part 1 of “SharePoint on Windows Azure” we gave a brief introduction and set up our basic tools to communicate with Windows Azure. There’s one more we’ll use that deals specifically with storage so let’s set that up now.

Storage Account Setup

First we need a storage account in Windows Azure. One may have been created when you signed up so let’s check first and if not, we’ll create one.

  1. In the Windows Azure management portal, click on “Storage” to see if a Storage Account exists. If one exists, skip to step # 6. Otherwise continue to step #2.
  2. To create a new Storage Account, click the New button at the bottom of the screen and click Data Services.
  3. Select Storage and Quick Create. Fill in a unique name for the URL.
  4. The Affinity Group may be blank, don’t worry if it is. Also, uncheck “Enable Geo-replication” if you want. We don’t need that for Dev/Test farms.
  5. Click Create Storage Account.
  6. Now we need to copy the key. Click on the Storage Account to get to the dashboard view.
  7. At the bottom, click “Manage Keys” and copy the Account Name and Primary Access Key somewhere temporary (like Notepad or OneNote). We’ll use these coming up soon.

Viewing Storage

There are several tools you can use to view Windows Azure Storage, I’ve put a link to a list of tools to explore Windows Azure storage in the resources section at the end. For now, we’ll use ClumsyLeaf CloudXPlorer (http://clumsyleaf.com/products/cloudxplorer), we’ll be using the freeware version, version 1). After you’ve downloaded and installed CloudXPlorer, we need to add a new account:

  1. Open CloudXPlorer and click File…, Accounts…, New…, Windows Azure account.
  2. Type in the Account Name you copied down in Step #7 of the earlier procedure.
  3. For Secret Key, type in the Primary Access Key you copied down in Step #7 of the earlier procedure.
  4. Check “Use SSL/TLS” and click OK. The account has been added, so close the Manage Accounts window if it’s still open.

Back in CloudXPlorer, you’ll see your account in a tree view. It’s probably empty now but you should have one container called “vhds.” This is where we can place our images and disks. Before we continue, we should spend a minute understanding the difference between an image and a disk.

Image vs. Disk

A disk is a vhd of a virtual machine that may or may not contain an operating system. When you attach this to a virtual machine in Windows Azure, it boots and expects everything to be ready. In contrast, an image is a vhd of virtual machine that has been sysprepped. It means, the machine will get a new SID (security identifier), new product key, and new Administrator credentials. This type of vhd can be used to deploy virtual machines, whereas a disk cannot.

What’s confusing is when researching how to upload a vhd, many people come to this article: https://www.windowsazure.com/en-us/manage/windows/common-tasks/upload-a-vhd/. In my opinion both the URL and the tile are inaccurately named, “Creating and Uploading a Virtual Hard Disk that Contains the Windows Server Operating System.” The instructions are good if you want to create an “image,” but not if you want to create a “disk” in Windows Azure. Windows Azure treats them differently. I uploaded a non-sysprepped vhd and attached it to a new VM. It tried to provision endlessly. In any case, that article *should* be titled “Creating and Uploading a Virtual Hard Disk that Contains the Windows Server Operating System as an Image,” and should link to another article for people that just want to upload a vhd as a disk.

Working with disks

Next, we’re going to upload a vhd. Unfortunately, there’s no support for vmdk’s or iso files (yet) so if you want your files in Windows Azure, you need to create a vhd. Let’s take a break and go over a few notes about vhd’s in Windows Azure. For a vhd to “work” it must meet these guidelines:

  • No dynamic disks. The vhd must be a fixed disk.
  • Maximum size must be under 127 GB.
  • No support for vhdx (need to verify this one)
  • Must be uploaded as a page blob (rather than a block blob).

If you create a vhd in Hyper-V for use in Windows Azure, make sure it’s a fixed disk and smaller than 127 GB. If you have existing dynamic disks you want to upload, you can use the “Edit Disk” feature in Hyper-V to convert it to a fixed disk. With that, let’s look at two upload tools.

The first is GUI-based, CloudXPlorer. To upload a vhd using CloudXplorer, DON’T CLICK THE UPLOAD BUTTON. If you look carefully, this is for block blobs. Also when you right-click, you’ll see an “Upload” link in the context menu – again for block blobs and as stated, we need to upload page blobs. The only way to upload a page blob is to right-click in a container or folder and click on “Upload page blob,” that’s the right option we want. I’ve pasted a link in the resources section called “Understanding Block blobs and Page blobs” if you want to read up on it. Once you click “Upload page blob,” you just select your .vhd file and away it goes.

The second tool is a called “CSUpload,” a command-line program included as part of the Windows Azure SDK that you downloaded in Part 1. I believe csupload has one major advantage, it’s multi-threaded so your uploads will go faster. CloudXPlorer is only muti-threaded for block blobs. To upload a .vhd using csupload:

  1. Open Windows Azure Command Prompt (in your Start Menu or Start Screen)
  2. Type in:

    csupload add-disk -destination “https://<storage account name>.blob.core.windows.net/vhds/MyDemoDisk.vhd” -Label “MyDemoDisk” -LiteralPath “<path to>\MyDemoDisk.vhd”

    Where <storage account name> is your Storage Account name, MyDemoDisk.vhd is the name of your vhd file and <path to> is the full path to the location of the vhd file.

    Note: “csupload add-disk” uploads the vhd, converts dynamic to fixed disks on the fly, and adds your disk to the repository. In contrast, the Windows Azure PowerShell version, “Add-AzureDisk” only adds the disk to the repository, after is uploaded. It doesn’t upload or convert dynamic disks to fixed disks.

  3. As stated earlier, the “-” character can get messed up when you copy and paste, so re-type it if it doesn’t work. For a 40GB file on a fast connection (5Mbps), I saw about 30 minute upload times.

You can use CloudXplorer to add containers and folders if you want to better organize your vhds. Just click on the parent object and then right-click in the content pane. Select “New” container or folder. In my screenshot, I’m in the “vhds” container so only New Folders are available. If I click on my account “wahids acct,” I’d see the New Container option.

Note: Windows Azure Management Portal doesn’t show tree views. So, all your disks will appear in one single view. The folders you create will all show up as “$$$.$$$”

 

 

OS Disks vs. Data Disks

In Windows Azure, OS Disks are those that contain a bootable operating system. A Data Disk is a non-bootable disk usually created to hold data (such as database files). In addition, all Windows Azure virtual machines get a “Temporary Storage” disk. The volumes are usually represented as follows:

Drive Letter

Type

Best Used For

C

OS Disk, System Drive

OS only

D

Temporary Storage, non-persistent drive

Pagefile, TempDB

E

Data Disk, not created by default or automatically

Databases, Program Files

 

When creating disks (or adding them) in Windows Azure, there’s two ways to specify that a disk is an OS Disk.

  1. First, in the Windows Azure management portal on the “Create Disk” dialog box you’ll put a check mark on the box “The VHD contains an operating system” and select the OS from the drop down box.
  2. If you’re using PowerShell, you’ll just specify the –OS parameter followed by either “Windows” or “Linux” during the Add-AzureDisk command.

By default, disks (either kind) in Windows Azure use caching to increase performance. This is great for the OS Disks because it leads to really fast boot times. However, on Data Disks, this isn’t ideal. We want to be assured our data is written immediately so we want to disable caching. We can do this by using the Set-AzureDataDisk –HostCaching “None” command for existing disks or as part of Add-AzureDataDisk when we’re creating new data disks.

Note: Changing the cache on a running VM will reboot the VM.

 

Some Best Practices

For the type of machines we’ll be using, I’m including some best practices I learned from a presentation by Corey Sanders (@CoreySandersWA) and included a link to his Channel 9 videos in the resources section.

SQL Servers: For the TempDB data and logs, keep them on the D: drive (the temporary, non-persistent disk) that Windows Azure gives us automatically. That drive is optimized for these kind of temporary data stores. Keep your actual database files on a separate data disk, even on multiple data disks. As those disks get used more often, Windows Azure optimizes performance. You may even get better performance by using separate storage accounts, just make sure they’re in the same region/Affinity Group as your VM.

Domain Controllers: Like any other applications we host, we shouldn’t put the database on the C: drive. If you’re setting up a DC from scratch, create a data disk and use that to store the Active Directory databases and folders when asked during dcpromo.

SharePoint: The same pattern applies here. Keep the data separate, install SharePoint on a data disk.

Finally, not really a best practice but just a note on storage costs. You pay for what you use, so creating a 700 GB data disk won’t cost you 700 GB worth of storage costs. As the drive fills up, you’ll pay for what you’re using. However, there is no “trim” functionality as of this writing. That means, even if you delete files, your costs won’t be reduced. One workaround that you can use in rare cases, such as if you know the data won’t grow again to fill up the space is to create a second disk and copy the data over. Use your new disk and delete the old one.

Conclusion

I’ll talk about creating a VM from scratch later in the series. For our purposes, I’m going to assume you have created a domain controller and SQL Server (and possible SharePoint servers) outside of Windows Azure (in Hyper-V). Upload all your vhds using csupload (you can write a batch file).

If you want to create your VMs from scratch in Windows Azure, you can still follow along in Part 3.

Next time, we’ll start building out our virtual network in Windows Azure. We won’t need our disks initially, but will want at least our domain controller to be ready.

Resources:

0 comments

SharePoint on Windows Azure Part 1: Introduction

February 27, 2013
SharePoint
This entry is part 1 of 4 in the series SharePoint on Windows Azure

For the past few months, I have been testing running SharePoint on Windows Azure. It’s a very intriguing proposition – enabling the full power and benefits of the cloud with SharePoint. By using Windows Azure, you get the ability to scale quickly and get up and running fast at a low cost. Plus, you don’t need to worry about the underlying infrastructure (as much), how reliable it is and if your information is redundant. I won’t go into a lot of detail on the benefits of the cloud or specifically Windows Azure but I’ll say this: You get to use a world-class datacenter where your data is copied 6 times in different geographic regions, where your compute and storage resources are constantly being optimized and load balanced, and where you can trust your data is secure.

There are some great resources that can help you get started deploying SharePoint on Windows Azure and I’ll list links at the end. However, I found myself referencing several different sources: blogs (mostly), MSDN articles, TechNet articles and internal communities for information. As a cloud newbie, the concepts were sometimes difficult to understand, the instructions sometimes conflicting or not applicable and the audience often seemed unclear (is this for developers? Do I really need to install Visual Studio?).

I’m writing this series to try and consolidate the information and present it from an ITPro’s perspective. My intention is to skip the theoretical discussions around using cloud, skip the developer-focused content and show you how to set up your cloud environment end to end the right way! Along the way, I’ll also show you some tools I use and highlight some things that tripped me up (lessons learned).

Getting Started

So let’s get started right away, we’ll review the architecture and concepts along the way. You’ll need a Windows Azure account to follow. Unfortunately, the 90-day free trial is limited in compute hours and storage, so is the MSDN Subscription benefits. However, I’ll show you scripts later on that will help you conserve your usage.

Once you’ve signed up, download the Windows Azure SDK. We’ll need this to interact with Windows Azure:

  1. Browse to http://www.windowsazure.com and click on “Develop” in the top bar.

  2. Under Development Centers, click the “other” link.

  3. Click the Windows Azure SDK Windows Installer.

  4. Install the SDK. When it’s finished, you should have a few new apps. Specifically, Windows Azure PowerShell and Windows Azure Command Prompt.

Now, if your Windows Azure account is ready to go, continue on. Otherwise, you’ll have to wait until your account is “activated.” You may have read articles on how to create a management certificate and import it. It sounded quite complicated to me, instead just follow these steps:

Note: In programs such as Microsoft Word, Adobe Reader and even Windows Notepad, the “-” symbol doesn’t transcribe properly to the command prompt. If you copy and paste commands from this file and get strange errors, re-type everything manually rather than using copy/paste.

  1. Open Windows Azure PowerShell and type in: Get-AzurePublishSettingsFile
  2. A browser window will open, you may be asked to login and then the browser will prompt you to open or save the *.publishsettings file. SAVE it, somewhere safe and secure. The cmdlets also downloads the management certificate for you.
  3. Back in Windows Azure PowerShell type in: Import-AzurePublishSettingsFile –PublishSettingsFile <full path to your .publishsettingsfile>
  4. Done. You’re now fully setup and ready to go. To test this, simply type in Get-AzureSubscription. As a final step, delete the *.publishsettingsfile file as it contains sensitive information.

    Note: You can also use the normal Windows PowerShell and import the Windows Azure cmdlets using “Import-Module Azure”

Next, we’ll need to also setup the Windows Azure Command Prompt in a similar fashion. We need the information from the certificate we just downloaded. The certificate you downloaded is real, you can find it in two places. First, login to the Windows Azure Portal and click on “Settings” on the left bar. You’ll find your management certificate here.

While you’re here, copy and paste the “Subscription Identifier” and the “Thumbprint,” we’ll use these next.

Second, open the Certificate Manager on your local machine (Win + R: certmgr.msc). You’ll see the same certificate under Personal\Certificates.

And you can verify it’s the same certificate by opening the certificate (double-click on it) and looking at the thumbprint in the details tab.
We can see that “FEC51664” is the first part of the thumbprint here as well on in the Windows Azure Settings page (your thumbprint will differ from mine, of course but it should match what’s on Windows Azure).

OK, so let’s setup our connection in the Windows Azure Command Prompt. Open the Windows Azure Command Prompt and type in the following:

csupload.exe set-connection “SubscriptionID=e3a…..95; CertificateThumbprint=cb……..94; ServiceManagementEndpoint=https://management.core.windows.net”

Replace the ID and Thumbprint with the ones you copied from the Windows Azure management portal earlier. You can close the command prompt if you want, the connection persists until you clear it using “csupload clear-connection.”

Note: If you copy and paste, you may get special characters. For example, in front of your SubscriptionID and an extra space or missing words for the ServiceManagementEndpoint. Check it carefully.

The last thing I want to you show is for times when you have more than one subscription in your account (as I do). We already looked at Get-AzureSubscription, this will list all your subscriptions but one of them will be default. To change which one is the default, simply execute the following:

Set-AzureSubscription –DefaultSubscription <“SubscriptionName”>


Wrap-Up

Next time we’ll start building some VM’s. If you’d like a head start and have the capability, go ahead and stand up a domain controller and SQL server in Hyper-V using the old .vhd (not .vhdx) format and fixed disks.

As promised, here are a few resources for further reading/viewing:

  

3 comments

SPC12 Presentations & Videos (for attendees)

November 20, 2012
SharePoint

Who has time to go to sessions at the SharePoint Conference? Not me, that’s for sure. Like last year, I kept a look out on a way to download all those presentations and videos. Here are a few options:

Option 1: @givenscj provided a really nice way to get both, PowerPoint Presentations and Videos. Read up on how to do it on his blog at http://blogs.architectingconnectedsystems.com/blogs/cjg/archive/2012/11/19/Download-all-SPC12-PowerPoints-and-Videos-_2D00_-PowerShell-Script.aspx

Option 2: @patricklamber provides another method using one of my favorite tools, Free Download Manager. Read up on this method on his blog at: http://nubo-corp.com/patrick-lamber/2012/11/how-do-we-download-all-spc12-videos-in-a-batch/

One note on this second option, it’s a bit of a manual process. I tried to automate some more pieces of it, so if you’re interested in that code, its located here: http://pastebin.com/060gbLFb

Both options work, I personally tested them. Both options will leave you with not-so-friendly-titles. Patrick Lamber has a fix (I haven’t tested but should work), follow the links to the updates on his blog.

Or, use my script (I created it before I knew Patrick was working on it). I think they both do the same thing, just a little differently. I am using a “sessioncsv.csv” file that has all the sessions and titles (normalized so they don’t have special characters). Then, I do the rename. You can download the csv file and Change-SPC12Filenames.ps1 by clicking on the links.

Session List: Some people just want a session list and then to go watch those on MySPC, or to download specific presentations. That’s the other reason for the CSV file above, it contains the session code, URL code and title of the session.

Enjoy!

0 comments

Renaming lots of files according to a spreadsheet

May 15, 2012
SharePoint

I don’t know about this blog post title, it doesn’t quite explain what I needed to do. Anyway.

Months and months ago, I downloaded all the SharePoint 2011 videos (using some PowerShell) and it worked great. However, the videos were all named with their session code, for example SPC201.wmv. See this post on how it was done: http://www.sharepointeer.com/blog/Lists/Posts/Post.aspx?ID=4

The file names are not very descriptive, but luckily a gentleman named Patrick Drews provided a session list in Excel that matched the code with the title. It looked like this:

Code Session
SPC201 Applying a Brand to your SharePoint Web Site

 

Armed with this, we should be able to rename the files so they’re descriptive. Here’s what I did.

The first problem is how to interact with this Excel spreadsheet. There are a lot of opinions on this, because natively, there’s no real easy way to do it. Many times, converting to CSV is the easiest. So, I found this function that does just that. It’s available from this site: http://jamesone111.wordpress.com/2011/02/07/how-to-read-excel-files-in-powershell/

Great! The next problem was to figure out the logic. First, lets define some variables:

$dir = "C:\SPC11"
$xlsessions = gci $dir\SessionList.xlsx | ConvertFrom-XLx -PassThru
$filenames = gci $dir\Videos | select -ExpandProperty Name

I end up with two arrays, $xlsessions which has the codes and session titles and $filenames which is just a list of the filenames. I need to match the filename with the session code for each file.

foreach ($filename in $filenames)    
        {    
        foreach ($xlsession in $xlsessions)
                {        
                If (($xlsession.CODE + ".wmv") -match $filename)            
                { #Do something }
                }
        }

This was actually the hardest part (for me anyway, because I’m a PowerShell n00b). Basically, we loops through each filename and each time we also loop through each of the session codes. If they match, then we’ll do something. Here’s what we’ll do:

{
$newname = $xlsession.CODE + " - " + $xlsession.SESSION + ".wmv"
write-host "Renaming $filename to $newname" -ForegroundColor Yellow
Rename-Item -Path $dir\Videos\$filename -NewName $dir\Videos\$newname
}

First, I’ve setup a variable called $newname that sets up the new filename for me. Then, I’m using Rename-Item to rename it. It seems simple enough, but took me a couple hours to get the logic right. Once I ran it, my files were renamed just as I wanted them in a matter of seconds. It would have taken hours upon hours to do this manually, cutting and pasting names from the Excel sheet into Windows Explorer. PowerShell saves the day again! Now take some time and save Live Writer….

This post was created with Windows Live Writer. Save this great tool by signing the petition here. Learn more by following #dontkillwlw on twitter.

0 comments

How To: Programmatically Delete a Folder from each site

April 30, 2012
SharePoint

In this post I’ll show you how to write a PowerShell script that will delete a single folder from multiple sites. In this case, there’s a folder called "Error Logs" in the Site Pages document library, as shown here:

image

This folder is generated by some 3rd party tool, which always creates the same folder name in the "Site Pages" Document Library of any site or sub-site. I’d like to simply delete it.

Disclaimer: The method I’m going to show you is just one way to do it and there may be a better way so feel free to write your suggestions in the comments.

So first, let’s just get some constant variables out of the way:

$url = "http://sharepoint"
$folder = "Error Logs"

$url is my portal URL and all the sites are underneath it. And $folder is the name of the folder I want to delete. The next thing I need to do is loop through each of my Site Collections (an SPSite object), here’s how that looks:

$sites = Get-SPSite -WebApplication $url | Where-Object {$_.Url -like "$url/sites/*"}
foreach ($site in $sites)
        {
        # Some functions here.
        }

So, I used Get-SPSite to get a list of the Site Collections and also used the | (pipe) symbol to only select sites that had a name like http://sharepoint/sites/

I’m trying to access a Document Library, which is inside a Site (an SPWeb object). So I’ll need to do a loop within a loop to go through every Site, in each Site Collection.

$sites = Get-SPSite -WebApplication $url | Where-Object {$_.Url -like "$url/sites/*"}
foreach ($site in $sites)
	{
	$siteid = $site | Select-Object -ExpandProperty Url
	$webs = Get-SPWeb -Site $siteid -Limit ALL
	foreach ($web in $webs)
		{
                # Some more functions here.
                }

What I’m doing here is is getting every Site Collection, selecting its "Url" and assigning it to $siteid. Then I use that to get every Site (SPWeb) in the Site Collection and assign it to $webs. Now I can iterate though all the Sites/SPWebs.

Next, I’ll need to grab the Document Library object, and delete the "Error Logs" folder:

$sites = Get-SPSite -WebApplication $url | Where-Object {$_.Url -like "$url/sites/*"}
foreach ($site in $sites)
	{
	$siteid = $site | Select-Object -ExpandProperty Url
	$webs = Get-SPWeb -Site $siteid -Limit ALL
	foreach ($web in $webs)
		{
                $library = $web.Folders["SitePages"]
		$library.SubFolders.Delete($folder)
                }

That’s all. Now, when you run this, it will work – but you’ll notice some errors, perhaps similar to this:

image

The reason for this is that not all of my sites have a Document Library called Site Pages. And even if they do, they may not all have a folder called "Error Logs." So, we need some better logic. We’ll use “If” to create that logic. 

	$webs = Get-SPWeb -Site $siteid -Limit ALL
	foreach ($web in $webs)
		{
		$library = $web.Folders["SitePages"]
		$allcontainers = $library.SubFolders | select Name
		Foreach ($container in $allcontainers)
			{
				If ($container -match $folder)
					{
					$library.SubFolders.Delete($folder)
					Write-Host "Deleted `"$folder`" from $web" -foregroundcolor Red
					}
				else {Write-Host "Finished checking $web." -foregroundcolor DarkGreen}
			}
		}

You’ll see here that I’ve created yet another loop! Now, I’m taking all the folder names from "SitePages" Document Library and looping through each one. I have an "If" statement that checks for a match and if it finds one, then and only then will it try to delete it. Otherwise, we just write a message stating we’ve checked the site.

Here’s the final, full script:

$url = "http://sharepoint"
$folder = "Error Logs"

$sites = Get-SPSite -WebApplication $url | Where-Object {$_.Url -like "$url/sites/*"}
foreach ($site in $sites)
	{
	$siteid = $site | Select-Object -ExpandProperty Url
	$webs = Get-SPWeb -Site $siteid -Limit ALL
	foreach ($web in $webs)
		{
		$library = $web.Folders["SitePages"]
		$allcontainers = $library.SubFolders | select Name
		Foreach ($container in $allcontainers)
			{
				If ($container -match $folder)
					{
					$library.SubFolders.Delete($folder)
					Write-Host "Deleted `"$folder`" from $web" -foregroundcolor Red
					}
				else {Write-Host "Finished checking $web." -foregroundcolor DarkGreen}
			}
		}
	}
Write-Host "Finished checking all sites."

Permissions problems are common when accessing Sites this way. I was a Farm Administrator and had rights over the content database for the Web Application I was working on. With the power of loops, you can take all sorts of actions on every Site in your SharePoint farm.

1 comment
Next Posts Previous Posts

Search

Categories

Archives